POC #131168

n1v0lg · 2025-07-13T18:28:54Z

No description provided.

quux00 · 2025-07-14T18:05:37Z

server/src/main/java/org/elasticsearch/RewritableIndicesRequest.java

+
+    boolean checkRemote(List<RemoteClusterService.RemoteTag> tags);
+
+    record RewrittenIndexExpression(String original, List<String> rewritten) {}


I think we also need to check (and track) whether each component of the rewritten list is flat or qualified? Is that left for a later impl or do you think we don't need to track that here?

quux00 · 2025-07-14T18:09:00Z

server/src/main/java/org/elasticsearch/transport/RemoteClusterService.java

@@ -97,6 +98,33 @@ public final class RemoteClusterService extends RemoteClusterAware
        (ns, key) -> boolSetting(key, true, new RemoteConnectionEnabled<>(ns, key), Setting.Property.Dynamic, Setting.Property.NodeScope)
    );

+    public record RemoteTag(String key, String value) {


The local project also has tags, so my guess is that this doesn't belong in RemoteClusterService and shouldn't be called RemoteTag? Maybe MetadataTag or ProjectTag?

Yes, ProjectTag makes sense though we might also go broader and just call it metadata in case we also want it to cover project alias and such.

quux00 · 2025-07-14T18:30:59Z

server/src/main/java/org/elasticsearch/rest/action/search/RestSearchAction.java

+        if (routingTags != null) {
+            searchRequest.routingTags(
+                Arrays.stream(Strings.splitStringByCommaToArray(routingTags)).map(RemoteClusterService.RemoteTag::fromString).toList()
+            );


Can you give an example of the type/format of tags you are allowing here? And is there an implicit "ANDing" of those tags?

As a first impl that might be useful.

Of course to state the obvious (not a criticism)

it doesn't handle ES|QL if that is embedded in the query

it doesn't support the complex AND, OR, NOT and grouping logic we'll eventually have to implement
so later we'll need some additional constructs/interfaces around parsing that more complex scenario.

n1v0lg · 2025-07-15T13:08:01Z

...security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java

@@ -124,6 +132,11 @@ ResolvedIndices resolve(
        if (request instanceof IndicesRequest == false) {
            throw new IllegalStateException("Request [" + request + "] is not an Indices request, but should be.");
        }
+
+        if (request instanceof CrossProjectRequest rewritableIndicesRequest) {


This would also need to go in tryResolveWithoutWildcards to handle non-wildcard expressions

smalyshev · 2025-07-17T17:13:24Z

server/src/main/java/org/elasticsearch/CrossProjectAwareRequest.java

+     * Used to track a mapping from original expression (potentially flat-world) to canonicalized CCS expressions.
+     * e.g. for an original index expression `logs-*`, this would be:
+     * original=logs-*
+     * qualified=[(logs-*, _local), (my-remote:logs-*, my-remote)]


So we're storing the remote name twice here, as I understand? What's the reason for that?

Oh ya, we don't really need to -- I added it to clarify that that info will be available but in a production ready version of this, we wouldn't want to duplicate that info, you're right

smalyshev · 2025-07-17T17:18:13Z

server/src/main/java/org/elasticsearch/CrossProjectAwareRequest.java

+    /**
+     * Only called if cross-project rewriting (flat-world, linked project filtering) was applied
+     */
+    void qualified(List<QualifiedExpression> qualifiedExpressions);


Could we rename this method? This is a mutator, and QualifiedExpression below has qualified() which is a getter, and it creates some congnitive load to try and figure out which one is meant each time. Maybe something like updateIndices or something if we expect it to be used to update indices?

yeah we for sure want a better name -- I'm also wondering if this should go in the IndicesRequest.Replaceable interface which has an indices(...) method on it

That would require a new method like isCrossProjectAware() on the IndicesRequest.Replaceable interface -- not sure yet which approach I prefer

smalyshev · 2025-07-17T17:19:54Z

server/src/main/java/org/elasticsearch/CrossProjectAwareRequest.java

+    /**
+     * Can be used to determine if this should be processed in cross-project mode vs. stateful CCS.
+     */
+    boolean crossProjectModeEnabled();


What will be using this method? The javadoc describes what it does but it's unclear to me which parts needs to know that.

smalyshev · 2025-07-17T17:21:24Z

server/src/main/java/org/elasticsearch/action/fieldcaps/FieldCapabilitiesRequest.java

@@ -58,6 +64,7 @@ public final class FieldCapabilitiesRequest extends LegacyActionRequest implemen
    private QueryBuilder indexFilter;
    private Map<String, Object> runtimeFields = Collections.emptyMap();
    private Long nowInMillis;
+    private List<QualifiedExpression> qualifiedExpressions;


How this is meant to be used? We store the result of flat resolution in the indices anyway, so we essentially storing it twice. Not sure I understand why?

smalyshev · 2025-07-18T00:04:03Z

...re/src/main/java/org/elasticsearch/xpack/core/security/authz/CrossProjectTargetResolver.java

+import java.util.List;
+
+/**
+ * Resolves linked and authorized projects, if running in a cross-project environment.


What does "resolve" mean in this context?

All naming and Javadoc definitely to be massively improved

But basically it gets all the projects based on linking and authorization (and environment, i.e., somewhere the returns VOID).

smalyshev · 2025-07-18T00:07:49Z

...re/src/main/java/org/elasticsearch/xpack/core/security/authz/CrossProjectTargetResolver.java

+        }
+    }
+
+    record ResolvedProjects(List<String> projects) {


Why do we need this class and not just a list and VOID being the empty list?

Empty list would mean "no access to any project"

List.of("_local") -> just local, CPS enabled
List.of() -> no access (I do wonder how this would happen but it since it happens early on during authz code)
VOID -> this is not even a CPS-enabled environment

Are the two last cases different? I mean, if you have no access to any projects, you can't do any CPS things, so why would it matter if it's the CPS environment?

Because in the case of VOID the search should proceed but in the case of empty but not VOID, it should fail

The idea is to use VOID as a thing in core to toggle CPS behavior (and failing if a user has no access to any projects -- i.e. empty list -- is itself a CPS behavior)

This is an interesting question - if the user doesn't have an access to any projects, would we even get this far? Right now _query for example would reject it but we could change that. What exactly minimal permissions are needed for CPS-enabled endpoints? Is anybody allowed in case they have access to some other remotes? Are there any separate permissions for this scenario?
I assumed the user would always have access to local, and this is the case today - you can't run _query without local access, but this is considered a bug, so my assumption was wrong. But I am still wondering if having user that has no access to anything get this far and not being stopped by security layer earlier makes sense?

smalyshev · 2025-07-18T14:56:44Z

server/src/main/java/org/elasticsearch/action/fieldcaps/FieldCapabilitiesRequest.java

 import java.util.Map;
 import java.util.Objects;
 import java.util.Set;

-public final class FieldCapabilitiesRequest extends LegacyActionRequest implements IndicesRequest.Replaceable, ToXContentObject {
+public final class FieldCapabilitiesRequest extends LegacyActionRequest


We need some way to tell field caps not to do the rewrite, because when processing e.g. lookups we probably don't need that. Alternatively, we'd need to change lookup code so it qualifies all the indices.

Yes I think that's true but I want to think through how that actually looks, I don't know if we want just a boolean flag

smalyshev · 2025-07-18T15:34:45Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/PlannerUtils.java

        return indices.toArray(String[]::new);
    }

+    public static List<String> expressions(String indexPattern) {


I think we need to explore what's the deal with "original" indices and what is actually expected there - real original or with flat syntax resolved.

Yes definitely

n1v0lg added 19 commits June 24, 2025 22:49

poc

980723e

poc

4ff5e6e

Fix ups

3df7a1d

The missing commit

c21a0a9

Merge branch 'main' into poc-flat-world

5266376

Merge branch 'main' into poc-flat-world

650dd77

More

9d63d77

Moar

8eec08e

Also resolver

fe6b696

field caps

8dee748

Merge branch 'main' into poc-flat-world

f9d6407

SPI

be2ab99

SPI

710d789

Remote conns

3587e6a

More

26a49c0

Extract interface

2a45f5b

Inject authenticator

9d29d4f

Simplify

0ef2258

Scope

88104ce

n1v0lg self-assigned this Jul 13, 2025

n1v0lg added the >non-issue label Jul 13, 2025

elasticsearchmachine added v9.2.0 serverless-linked Added by automation, don't add manually labels Jul 13, 2025

n1v0lg added 3 commits July 14, 2025 13:16

Compile

035ffc1

Merge branch 'main' into poc-cps-e2e

f5cad57

Query routing

fe3dea0

quux00 reviewed Jul 14, 2025

View reviewed changes

Some clean up

1153dfa

WIP esql

3fd532e

idegtiarenko marked this pull request as ready for review July 15, 2025 09:00

idegtiarenko requested a review from a team as a code owner July 15, 2025 09:00

idegtiarenko marked this pull request as draft July 15, 2025 09:00

Fix interface

512222e

n1v0lg commented Jul 15, 2025

View reviewed changes

n1v0lg added 7 commits July 15, 2025 15:22

WIP resolver

b867e69

Only inject resolver

08d3e4e

Clean up

d36c3f6

esql patch by @idegtiarenko

d20a8c9

More context

32ed394

More still

67f5cdf

CrossProjectAware

e25d8b2

n1v0lg changed the title ~~poc~~ POC Jul 17, 2025

Javadoc

137195f

smalyshev reviewed Jul 18, 2025

View reviewed changes


		boolean checkRemote(List<RemoteClusterService.RemoteTag> tags);

		record RewrittenIndexExpression(String original, List<String> rewritten) {}

POC #131168

Are you sure you want to change the base?

POC #131168

Conversation

n1v0lg commented Jul 13, 2025

Uh oh!

quux00 Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

n1v0lg Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

quux00 Jul 14, 2025 •

edited

Loading

n1v0lg Jul 18, 2025 •

edited

Loading