Support `appendpipe`command in PPL #4602

xinyual · 2025-10-20T07:50:38Z

Description

Add appendpipe command.
Appendpipe appends the result of the subpipeline to the search results. Unlike a subsearch, the subpipeline is not run first. The subpipeline is run when the search reaches the appendpipe command.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: xinyual <xinyual@amazon.com>

yuancu

Review in progress

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

Signed-off-by: xinyual <xinyual@amazon.com>

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLAppendPipeCommandIT.java

Signed-off-by: xinyual <xinyual@amazon.com>

yuancu

LGTM

xinyual · 2025-11-05T06:54:40Z

@Swiddis could you please approve again?

qianheng-aws · 2025-11-07T02:42:06Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

+    while (childNode.getChild() != null
+        && !childNode.getChild().isEmpty()
+        && !(childNode.getChild().getFirst() instanceof Values)) {
+      childNode = (UnresolvedPlan) childNode.getChild().getFirst();


Will there be more than 1 children in the subquery? Maybe adding assertion and throw exception to avoid this case if we don't support.

Signed-off-by: xinyual <xinyual@amazon.com>

* add demos Signed-off-by: xinyual <xinyual@amazon.com> * add missing column Signed-off-by: xinyual <xinyual@amazon.com> * add appendpipe poc Signed-off-by: xinyual <xinyual@amazon.com> * slighty change syntax Signed-off-by: xinyual <xinyual@amazon.com> * add unresolved plan Signed-off-by: xinyual <xinyual@amazon.com> * add IT Signed-off-by: xinyual <xinyual@amazon.com> * add tests Signed-off-by: xinyual <xinyual@amazon.com> * remove useless ut Signed-off-by: xinyual <xinyual@amazon.com> * fix conflict Signed-off-by: xinyual <xinyual@amazon.com> * remove useless code Signed-off-by: xinyual <xinyual@amazon.com> * remove useless code Signed-off-by: xinyual <xinyual@amazon.com> * remove useless code Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> * remove useless chaneg Signed-off-by: xinyual <xinyual@amazon.com> * add explain IT Signed-off-by: xinyual <xinyual@amazon.com> * fix IT Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> * add doc Signed-off-by: xinyual <xinyual@amazon.com> * optimize doc Signed-off-by: xinyual <xinyual@amazon.com> * add UT Signed-off-by: xinyual <xinyual@amazon.com> * fix IT due to performance change Signed-off-by: xinyual <xinyual@amazon.com> * add multiply children check Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com> (cherry picked from commit 3a3c8c8) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Support `appendpipe`command in PPL (#4602) * add demos Signed-off-by: xinyual <xinyual@amazon.com> * add missing column Signed-off-by: xinyual <xinyual@amazon.com> * add appendpipe poc Signed-off-by: xinyual <xinyual@amazon.com> * slighty change syntax Signed-off-by: xinyual <xinyual@amazon.com> * add unresolved plan Signed-off-by: xinyual <xinyual@amazon.com> * add IT Signed-off-by: xinyual <xinyual@amazon.com> * add tests Signed-off-by: xinyual <xinyual@amazon.com> * remove useless ut Signed-off-by: xinyual <xinyual@amazon.com> * fix conflict Signed-off-by: xinyual <xinyual@amazon.com> * remove useless code Signed-off-by: xinyual <xinyual@amazon.com> * remove useless code Signed-off-by: xinyual <xinyual@amazon.com> * remove useless code Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> * remove useless chaneg Signed-off-by: xinyual <xinyual@amazon.com> * add explain IT Signed-off-by: xinyual <xinyual@amazon.com> * fix IT Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> * add doc Signed-off-by: xinyual <xinyual@amazon.com> * optimize doc Signed-off-by: xinyual <xinyual@amazon.com> * add UT Signed-off-by: xinyual <xinyual@amazon.com> * fix IT due to performance change Signed-off-by: xinyual <xinyual@amazon.com> * add multiply children check Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com> * avoid to use get first Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com>

xinyual added 11 commits September 26, 2025 13:57

add demos

759aab4

Signed-off-by: xinyual <xinyual@amazon.com>

add missing column

00a1563

Signed-off-by: xinyual <xinyual@amazon.com>

add appendpipe poc

ddf26a4

Signed-off-by: xinyual <xinyual@amazon.com>

merge from main

7233111

Signed-off-by: xinyual <xinyual@amazon.com>

slighty change syntax

a8dd2ca

Signed-off-by: xinyual <xinyual@amazon.com>

add unresolved plan

c3549f5

Signed-off-by: xinyual <xinyual@amazon.com>

add IT

9dfc626

Signed-off-by: xinyual <xinyual@amazon.com>

Merge remote-tracking branch 'origin/main' into addappendPipe

a3e78a8

add tests

f88730f

Signed-off-by: xinyual <xinyual@amazon.com>

remove useless ut

4c281cc

Signed-off-by: xinyual <xinyual@amazon.com>

merge from main

70e368d

Signed-off-by: xinyual <xinyual@amazon.com>

xinyual changed the title ~~Addappend pipe~~ [WIP]: support appendpipecommand in PPL Oct 20, 2025

xinyual added 11 commits October 20, 2025 15:59

fix conflict

93eb5b3

Signed-off-by: xinyual <xinyual@amazon.com>

remove useless code

9148512

Signed-off-by: xinyual <xinyual@amazon.com>

remove useless code

345cdeb

Signed-off-by: xinyual <xinyual@amazon.com>

remove useless code

3d4d865

Signed-off-by: xinyual <xinyual@amazon.com>

apply spotless

00aba11

Signed-off-by: xinyual <xinyual@amazon.com>

remove useless chaneg

4c2b311

Signed-off-by: xinyual <xinyual@amazon.com>

add explain IT

8744cb2

Signed-off-by: xinyual <xinyual@amazon.com>

fix IT

c48fa34

Signed-off-by: xinyual <xinyual@amazon.com>

apply spotless

3a23979

Signed-off-by: xinyual <xinyual@amazon.com>

add doc

f934efa

Signed-off-by: xinyual <xinyual@amazon.com>

optimize doc

d71e145

Signed-off-by: xinyual <xinyual@amazon.com>

xinyual marked this pull request as ready for review October 23, 2025 05:28

xinyual requested review from YANG-DB, dai-chen, derek-ho, joshuali925, kavithacm and ps48 as code owners October 23, 2025 05:28

xinyual requested review from GumpacG, LantaoJin, RyanL1997, noCharger, qianheng-aws, ykmr1224 and yuancu as code owners October 23, 2025 05:28

yuancu added the enhancement New feature or request label Oct 31, 2025

yuancu changed the title ~~[WIP]: support appendpipecommand in PPL~~ Support appendpipecommand in PPL Oct 31, 2025

yuancu reviewed Oct 31, 2025

View reviewed changes

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java Show resolved Hide resolved

Swiddis previously approved these changes Oct 31, 2025

View reviewed changes

add UT

7664cd5

Signed-off-by: xinyual <xinyual@amazon.com>

xinyual dismissed Swiddis’s stale review via 7664cd5 November 3, 2025 02:19

yuancu reviewed Nov 3, 2025

View reviewed changes

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLAppendPipeCommandIT.java Show resolved Hide resolved

xinyual added 2 commits November 4, 2025 10:20

Merge remote-tracking branch 'origin/main' into addAppendpipe

1624912

fix IT due to performance change

fa2ec9e

Signed-off-by: xinyual <xinyual@amazon.com>

yuancu previously approved these changes Nov 5, 2025

View reviewed changes

qianheng-aws reviewed Nov 7, 2025

View reviewed changes

xinyual added 2 commits November 12, 2025 10:19

add multiply children check

fe1b23f

Signed-off-by: xinyual <xinyual@amazon.com>

merge from main

f1b3a81

Signed-off-by: xinyual <xinyual@amazon.com>

xinyual dismissed yuancu’s stale review via f1b3a81 November 12, 2025 02:22

qianheng-aws approved these changes Nov 12, 2025

View reviewed changes

yuancu approved these changes Nov 12, 2025

View reviewed changes

yuancu merged commit 3a3c8c8 into opensearch-project:main Nov 12, 2025
55 of 56 checks passed

yuancu added the backport 2.19-dev label Nov 12, 2025

opensearch-trigger-bot bot mentioned this pull request Nov 12, 2025

[Backport 2.19-dev] Support appendpipecommand in PPL #4785

Closed

xinyual mentioned this pull request Nov 12, 2025

[Backport 2.19-dev] Support appendpipe command in PPL #4787

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support `appendpipe`command in PPL #4602

Support `appendpipe`command in PPL #4602

Uh oh!

xinyual commented Oct 20, 2025 •

edited

Loading

Uh oh!

yuancu left a comment

Uh oh!

Uh oh!

Uh oh!

yuancu left a comment

Uh oh!

xinyual commented Nov 5, 2025

Uh oh!

qianheng-aws Nov 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Support appendpipecommand in PPL #4602

Support appendpipecommand in PPL #4602

Uh oh!

Conversation

xinyual commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

yuancu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yuancu left a comment

Choose a reason for hiding this comment

Uh oh!

xinyual commented Nov 5, 2025

Uh oh!

qianheng-aws Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Support `appendpipe`command in PPL #4602

Support `appendpipe`command in PPL #4602

xinyual commented Oct 20, 2025 •

edited

Loading