Mvexpand feature #4675

srikanthpadakanti · 2025-10-27T19:47:15Z

Description

This pull request adds native support for the mvexpand command in PPL to OpenSearch SQL, enabling users to expand multivalue fields (arrays) into separate rows directly within queries. This functionality is analogous to Splunk's mvexpand command and streamlines analytics, dashboarding, and data preparation involving arrays or multivalue fields.

Key features introduced:

Native mvexpand command for PPL queries to expand array fields into separate rows/events.
Optional limit parameter to restrict the number of expanded values per event/document.
Robust handling of empty/null arrays, large arrays (with memory/resource limits), and non-array fields.
Streaming/distributable execution for performance and scalability.
Comprehensive documentation and edge case coverage.
This feature makes OpenSearch SQL more powerful and user-friendly for log analytics, data exploration, and migration from platforms like Splunk.

Related Issues

Resolves #4439
#4439

Check List

[ X] New functionality includes testing.
[ X] New functionality has been documented.
[ X] New functionality has javadoc added.
[ X] New functionality has a user manual doc added.
[ X] New PPL command checklist all confirmed.
[ X] API changes companion pull request created.
[ X] Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

docs/category.json

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Signed-off-by: Srikanth Padakanti <srikanth29.9@gmail.com>

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

srikanthpadakanti · 2025-11-03T20:01:59Z

Hello @dai-chen There are some tests failed in Security Plugin IT - unrelated to my mvexpand changes.
@ykmr1224

ppl/src/main/antlr/OpenSearchPPLParser.g4

ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLMvExpandTest.java

docs/user/ppl/cmd/mvexpand.rst

dai-chen · 2025-11-04T19:45:16Z

docs/user/ppl/cmd/mvexpand.rst

+
+Example 5: Large Arrays and Memory Limits
+----------------------------------------
+If an array exceeds configured memory/resource limits, mvexpand returns an error.


Is this config already supported?

There is no mvexpand-specific config. Resource limits are enforced by the engine/cluster (e.g. circuit breakers and SQL/PPL query size/timeouts). Use mvexpand's limit data to avoid hitting those limits; if you want I can add references to the exact cluster/SQL settings for our release.

Let me add the same to the documentation

Do you mean SQL/PPL's circuit breaker or OpenSearch DSL's? I think our circuit breaker today only protect scan operator.

I clarified in the mvexpand docs that resource limits are enforced by OpenSearch node/SQL-PPL limits (no mvexpand-specific config) and documented using limit + prefiltering as the recommended mitigation;

https://docs.opensearch.org/1.0/search-plugins/ppl/settings/

Referring to this:
plugins.query.memory_limit | Set heap memory usage limit. If a query crosses this limit, it’s terminated.

I see. I recall in our V3 (Calcite-based) engine, only index scan operator is protected. Circuit breaking other operators are under research. I think probably we just keep "To avoid failures when expanding large arrays:" section which is very helpful.

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

ykmr1224 · 2025-11-06T23:15:04Z

#4675 (comment)
From this comment, it looks like an improvement for existing expand command.
Why don't we improve expand at the same time, and consider mvexpand as and alias of expand?

core/src/main/java/org/opensearch/sql/ast/tree/MvExpand.java

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

srikanthpadakanti · 2025-11-07T03:24:15Z

#4675 (comment) From this comment, it looks like an improvement for existing expand command. Why don't we improve expand at the same time, and consider mvexpand as and alias of expand?

Refactored as requested.
Improved the "expand" command and aliased the - "mvexpand" to it.

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

dai-chen

Thanks for the changes!

ykmr1224 · 2025-11-11T17:07:58Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

-    context.relBuilder.variable(correlVariable::set);
+  // New generic helper: builds Uncollect + Correlate using a provided left node (so caller
+  // can ensure left rowType is fixed).
+  private void buildUnnestForLeft(


I think we no longer need this method as the logic is unified. We might want to extract some portion of buildExpandRelNode, but we should extract with fewer parameters in that case.

ykmr1224 · 2025-11-11T17:11:56Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

  }

+  @Override
+  public RelNode visitMvExpand(MvExpand node, CalcitePlanContext context) {


nit: Let's move this method next to visitExpand as it is closely related.

ykmr1224 · 2025-11-11T17:27:26Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

+        arrayFieldRex = context.rexBuilder.makeInputRef(currentRowType, fld.getIndex());
+      } else {
+        throw new IllegalArgumentException(
+            "buildExpandRelNode: expected RexInputRef or resolvable field name: " + arrayFieldName);


Is this expected to be visible to the user? In that case, error message should be meaningful for the user.

ykmr1224 · 2025-11-11T17:32:49Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

+    } else {
+      RelDataType currentRowType = context.relBuilder.peek().getRowType();
+      RelDataTypeField fld = currentRowType.getField(arrayFieldName, false, false);


What kind of case falls into this branch? Which test case associated with this branch?

ykmr1224 · 2025-11-11T17:42:56Z

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMvExpandCommandIT.java

+    bulkInsert(
+        INDEX,
+        "{\"username\":\"happy\",\"skills\":[{\"name\":\"python\"},{\"name\":\"java\"},{\"name\":\"sql\"}]}",
+        "{\"username\":\"single\",\"skills\":[{\"name\":\"go\"}]}",
+        "{\"username\":\"empty\",\"skills\":[]}",
+        "{\"username\":\"nullskills\",\"skills\":null}",
+        "{\"username\":\"noskills\"}",
+        "{\"username\":\"missingattr\",\"skills\":[{\"name\":\"c\"},{\"level\":\"advanced\"}]}",
+        "{\"username\":\"complex\",\"skills\":[{\"name\":\"ml\",\"level\":\"expert\"},{\"name\":\"ai\"},{\"level\":\"novice\"}]}",
+        "{\"username\":\"duplicate\",\"skills\":[{\"name\":\"dup\"},{\"name\":\"dup\"}]}",
+        "{\"username\":\"large\",\"skills\":[{\"name\":\"s1\"},{\"name\":\"s2\"},{\"name\":\"s3\"},{\"name\":\"s4\"},{\"name\":\"s5\"},{\"name\":\"s6\"},{\"name\":\"s7\"},{\"name\":\"s8\"},{\"name\":\"s9\"},{\"name\":\"s10\"}]}");


Looks like some records are not used in tests. Should we add test cases or remove unused records?

Srikanth Padakanti and others added 9 commits October 27, 2025 10:44

Initial checkpoint - following calcite way and commented legacy way

445799f

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Removed the build.gradle dependency opensearch-common

427af0f

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Ready to submit this PR

83d7786

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Ready to submit this PR

7f6e127

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Ready to submit this PR

dcbf56b

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Add mvexpand.rst

a3c1384

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Add Tests

148ccc5

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Add the mvexpand.rst to the index.rst

f5a9e82

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Merge branch 'opensearch-project:main' into main

2cc60ad

Srikanth Padakanti added 3 commits October 30, 2025 15:46

Address the PR comments and fix the tests accordingly

825c52e

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Address the PR comments and fix the tests accordingly

dc76a55

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Add comment lines for buildUnnestForLeft

8319583

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

srikanthpadakanti commented Oct 30, 2025

View reviewed changes

docs/category.json Show resolved Hide resolved

srikanthpadakanti requested a review from dai-chen October 30, 2025 20:55

Srikanth Padakanti and others added 6 commits October 30, 2025 23:29

Fix the mvexpand.rst

6d87133

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Merge branch 'main' into main

b83ab21

Signed-off-by: Srikanth Padakanti <srikanth29.9@gmail.com>

Fix the failing test

c84703d

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Fix the failing test

de82b65

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Fix the failing test

6c6e0ec

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

Fix the failing test

069d52e

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

dai-chen reviewed Nov 4, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java Show resolved Hide resolved

srikanthpadakanti and others added 3 commits November 5, 2025 12:01

Merge branch 'opensearch-project:main' into main

2f3aeb6

Merge branch 'opensearch-project:main' into main

e584368

Address the PR comments

a41b081

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

srikanthpadakanti requested a review from dai-chen November 6, 2025 20:31

ykmr1224 reviewed Nov 6, 2025

View reviewed changes

Address the PR comments

bf018b7

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

srikanthpadakanti requested a review from ykmr1224 November 7, 2025 03:34

Merge branch 'opensearch-project:main' into main

c215243

dai-chen approved these changes Nov 11, 2025

View reviewed changes

ykmr1224 reviewed Nov 11, 2025

View reviewed changes

Mvexpand feature #4675

Are you sure you want to change the base?

Mvexpand feature #4675

Uh oh!

Conversation

srikanthpadakanti commented Oct 27, 2025

Description

Related Issues

Check List

Uh oh!

Uh oh!

srikanthpadakanti commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ykmr1224 commented Nov 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

srikanthpadakanti commented Nov 7, 2025

Uh oh!

dai-chen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ykmr1224 Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

srikanthpadakanti commented Nov 3, 2025 •

edited

Loading

ykmr1224 Nov 11, 2025 •

edited

Loading