Add explicit versions in the branches.json and use them for calculating unreleased versions for BwC tests #129637

jozala · 2025-06-18T14:36:41Z

Work in progress

Gradle task added to modify branches.json. Possibility to add, remove and update branches in the file with the version field added next to the branch.
After the change the file looks like this:

{
    "branches": [
        {
            "branch": "main",
            "version": "9.1.0"
        },
        {
            "branch": "9.0",
            "version": "9.0.3"
        },
        {
            "branch": "8.19",
            "version": "8.19.1"
        }
    ]
}

BwC test configuration has been changed to use changed branches.json file to calculate unreleased ES versions, instead of combining data from Version.java and branches.json.

breskeby · 2025-06-23T07:04:55Z

...internal/src/main/java/org/elasticsearch/gradle/internal/release/UpdateBranchesJsonTask.java

+
+    @TaskAction
+    public void executeTask() throws IOException {
+        File branchesFile = new File(Util.locateElasticsearchWorkspace(project.getGradle()), "branches.json");


referencing project here while executing task is deprecated. In this case I think we should just inject ProjectLayout instead of project in the ctor and then use layout.getSettingsDirectory().file('branches.json').asFile.

The Util.locateElasticsearchWorkspace is just a utility method only used to resolve files relateive from internal builds within this repository, which we don't need here as we never run this task anywhere else but in the :elasticsearch root build

Also we probably want to make this file an outputFile property of the task

I've now changed the way to reference branches.json file location using ProjectLayout and also made the branches file an @OutputFile property of the task.

breskeby · 2025-06-23T07:06:51Z

...internal/src/main/java/org/elasticsearch/gradle/internal/release/UpdateBranchesJsonTask.java

+    private final ObjectMapper objectMapper;
+    private final BranchesFileParser branchesFileParser;
+
+    @Nullable


Usually we do not use that annotation. In Gradle tasks, you usually mark properties that are optional as @optional and as @input

I've made all three parameters @Input and @Optional.

jozala · 2025-07-07T17:29:36Z

There is still a problem with GlobalBuildInfoPluginSpec failing because BwcVersions directly use static version from VersionProperties. It needs to be solved before merging to main.

jozala · 2025-07-24T08:23:18Z

...ols-internal/src/main/java/org/elasticsearch/gradle/internal/info/GlobalBuildInfoPlugin.java

+        project.getPlugins().apply(VersionPropertiesPlugin.class);
+        Properties versionProperties = (Properties) project.getExtensions().getByName(VERSIONS_EXT);
+        JavaVersion minimumCompilerVersion = JavaVersion.toVersion(versionProperties.get("minimumCompilerJava"));
+        JavaVersion minimumRuntimeVersion = JavaVersion.toVersion(versionProperties.get("minimumRuntimeJava"));
+
+        String bundledJdkVersion = versionProperties.getProperty("bundled_jdk");
+        String bundledJdkMajorVersion = bundledJdkVersion.split("[.+]")[0];
+        Version elasticsearchVersionProperty = Version.fromString(versionProperties.getProperty("elasticsearch"));


Not using static VersionProperties any more here as this make the class untestable.
This solves the problem described here: #129637 (comment)

When VersionProperties is static it is loaded only once per JVM. It means it cannot be changed in the tests. This is a problem, because we want to test that GlobalBuildInfoPlugin can calculate unreleased versions with BwcVersions for some fixed configuration. This was impossible when VersionProperties reads the real version.properties file which changes with every release.

jozala · 2025-07-24T08:28:12Z

...al/src/integTest/resources/org/elasticsearch/gradle/internal/fake_git/remote/settings.gradle

+include ":distribution:bwc:major1"
+include ":distribution:bwc:major2"
+include ":distribution:bwc:major3"
+include ":distribution:bwc:major4"
+include ":distribution:bwc:minor1"
+include ":distribution:bwc:minor2"
+include ":distribution:bwc:minor3"
+include ":distribution:bwc:minor4"


This is simplification of the naming convention.
Name minorN means previous minor on the same major where number N is a distance from the current.
Name majorN means previous major, still actively developed versions, where N is a distance from the current, omitting all the previous minor versions.

Related changes in other files across this PR.

jozala · 2025-07-24T08:31:23Z

build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/BwcVersions.java

+        Map<String, List<DevelopmentBranch>> bwcBranches = developmentBranches.stream()
+            .filter(developmentBranch -> developmentBranch.version().before(currentVersion))
+            .sorted(reverseOrder(comparing(DevelopmentBranch::version)))
+            .collect(Collectors.groupingBy(branch -> {
+                if (branch.version().getMajor() == currentVersion.getMajor()) {
+                    return "minor";
+                } else if (branch.version().getMajor() == currentVersion.getMajor() - 1) {
+                    return "major";
+                }
+                return "older";
+            }));
+
+        developmentBranches.stream()
+            .filter(branch -> branch.version().equals(currentVersion))
+            .findFirst()
+            .ifPresent(
+                developmentBranch -> result.put(
+                    currentVersion,
+                    new UnreleasedVersionInfo(currentVersion, developmentBranch.name(), ":distribution")
+                )
+            );

-        // The current version is always in development
-        String currentBranch = getBranchFor(currentVersion, developmentBranches);
-        result.put(currentVersion, new UnreleasedVersionInfo(currentVersion, currentBranch, ":distribution"));
-
-        // Check for an n.x branch as well
-        if (currentBranch.equals("main") && developmentBranches.stream().anyMatch(s -> s.endsWith(".x"))) {
-            // This should correspond to the latest new minor
-            Version version = versions.stream()
-                .sorted(Comparator.reverseOrder())
-                .filter(v -> v.getMajor() == (currentVersion.getMajor() - 1) && v.getRevision() == 0)
-                .findFirst()
-                .orElseThrow(() -> new IllegalStateException("Unable to determine development version for branch"));
-            String branch = getBranchFor(version, developmentBranches);
-            assert branch.equals(currentVersion.getMajor() - 1 + ".x") : "Expected branch does not match development branch";
-
-            result.put(version, new UnreleasedVersionInfo(version, branch, ":distribution:bwc:minor"));
+        List<DevelopmentBranch> previousMinorBranches = bwcBranches.getOrDefault("minor", Collections.emptyList());
+        for (int i = 0; i < previousMinorBranches.size(); i++) {
+            DevelopmentBranch previousMinorBranch = previousMinorBranches.get(i);
+            result.put(
+                previousMinorBranch.version(),
+                new UnreleasedVersionInfo(previousMinorBranch.version(), previousMinorBranch.name(), ":distribution:bwc:minor" + (i + 1))
+            );
        }

-        // Now handle all the feature freeze branches
-        List<String> featureFreezeBranches = developmentBranches.stream()
-            .filter(b -> Pattern.matches("[0-9]+\\.[0-9]+", b))
-            .sorted(reverseOrder(comparing(s -> Version.fromString(s, Version.Mode.RELAXED))))
-            .toList();
-
-        int bugfixCount = 0;
-        boolean existingStaged = false;
-        for (int i = 0; i < featureFreezeBranches.size(); i++) {
-            String branch = featureFreezeBranches.get(i);
-            Version version = versions.stream()
-                .sorted(Comparator.reverseOrder())
-                .filter(v -> v.toString().startsWith(branch))
-                .findFirst()
-                .orElse(null);
-
-            // If we don't know about this version we can ignore it
-            if (version == null) {
-                continue;
-            }
-
-            // If this is the current version we can ignore as we've already handled it
-            if (version.equals(currentVersion)) {
-                continue;
-            }
-
-            // We only maintain compatibility back one major so ignore anything older
-            if (currentVersion.getMajor() - version.getMajor() > 1) {
-                continue;
-            }
-
-            // This is the maintenance version
-            if (i == featureFreezeBranches.size() - 1) {
-                result.put(version, new UnreleasedVersionInfo(version, branch, ":distribution:bwc:maintenance"));
-            } else if (version.getRevision() == 0) { // This is the next staged minor
-                String project = existingStaged ? "staged2" : "staged";
-                result.put(version, new UnreleasedVersionInfo(version, branch, ":distribution:bwc:" + project));
-                existingStaged = true;
-            } else { // This is a bugfix
-                bugfixCount++;
-                String project = "bugfix" + (bugfixCount > 1 ? bugfixCount : "");
-                result.put(version, new UnreleasedVersionInfo(version, branch, ":distribution:bwc:" + project));
-            }
+        List<DevelopmentBranch> previousMajorBranches = bwcBranches.getOrDefault("major", Collections.emptyList());
+        for (int i = 0; i < previousMajorBranches.size(); i++) {
+            DevelopmentBranch previousMajorBranch = previousMajorBranches.get(i);
+            result.put(
+                previousMajorBranch.version(),
+                new UnreleasedVersionInfo(previousMajorBranch.version(), previousMajorBranch.name(), ":distribution:bwc:major" + (i + 1))
+            );


This is the main change. Now we use branches.json to get information about the unreleased versions instead of calculating it from Version.java.

jozala · 2025-07-24T08:34:30Z

...ols-internal/src/main/java/org/elasticsearch/gradle/internal/info/GlobalBuildInfoPlugin.java

+    private List<DevelopmentBranch> getDevelopmentBranches() {
+        String branchesFileLocation = project.getProviders()
+            .gradleProperty(BRANCHES_FILE_LOCATION_PROPERTY)
+            .getOrElse(DEFAULT_BRANCHES_FILE_URL);
+        LOGGER.info("Reading branches.json from {}", branchesFileLocation);
+        byte[] branchesBytes;
+        if (branchesFileLocation.startsWith("http")) {
+            try (InputStream in = URI.create(branchesFileLocation).toURL().openStream()) {
+                branchesBytes = in.readAllBytes();
+            } catch (IOException e) {
+                throw new UncheckedIOException("Failed to download branches.json from: " + branchesFileLocation, e);
+            }
+        } else {
+            try {
+                branchesBytes = Files.readAllBytes(new File(branchesFileLocation).toPath());
+            } catch (IOException e) {
+                throw new UncheckedIOException("Failed to read branches.json from: " + branchesFileLocation, e);
            }
-        } catch (IOException e) {
-            throw new UncheckedIOException(e);
        }

-        return branches;
+        return branchesFileParser.parse(branchesBytes);


It reads development branches (with current versions of ES on these branches) from main branch using raw GitHub link. This URL can be overridden with org.elasticsearch.build.branches-file-location Gradle property if needed.

# Conflicts: # branches.json

By default, it now reads branches.json from main branch of Elasticsearch (through GitHub raw link) . This can be overridden for testing or when there is a problem with remote call.

This change is needed to make GlobalBuildInfoPlugin testable. Previously it was using static VersionProperties which loaded values once for JVM so it was impossible to test with different values.

This file is needed now during the tests as it is no longer fully static during full JVM execution.

elasticsearchmachine · 2025-07-24T16:08:23Z

Pinging @elastic/es-delivery (Team:Delivery)

jozala added >non-issue v9.1.0 labels Jun 18, 2025

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Jun 18, 2025

jozala force-pushed the bwc-explicit-branches-version branch from 1e1a591 to d84fef9 Compare June 20, 2025 10:24

jozala self-assigned this Jun 20, 2025

breskeby reviewed Jun 23, 2025

View reviewed changes

breskeby approved these changes Jun 23, 2025

View reviewed changes

jozala force-pushed the bwc-explicit-branches-version branch from 4d569dc to 1e2fb07 Compare June 26, 2025 12:04

elasticsearchmachine added v9.2.0 and removed v9.1.0 labels Jun 26, 2025

jozala force-pushed the bwc-explicit-branches-version branch 2 times, most recently from 022297f to 4d3aa2f Compare July 3, 2025 09:33

jozala force-pushed the bwc-explicit-branches-version branch 2 times, most recently from e0f1357 to 2bf9688 Compare July 24, 2025 08:18

jozala commented Jul 24, 2025

View reviewed changes

jozala and others added 10 commits July 24, 2025 16:06

Initial work for having explicit active branch with version connection

c18c24c

# Conflicts: # branches.json

Basic implementation with single test passing

67e07c4

Tests for UpdateBranchesJsonTask

1c1a531

Allow changing branch.json location in GlobalBuildInfoPlugin

f6fedd5

By default, it now reads branches.json from main branch of Elasticsearch (through GitHub raw link) . This can be overridden for testing or when there is a problem with remote call.

Change branch.json location temporarily to point to the forked repo

588c2dd

[CI] Auto commit changes from spotless

d4c0bbc

Artificial Gradle projects for BwC tests added

c51044d

Tests corrected to be in line with new BwC test naming

79a9c0a

Register the task in the plugin

5206a31

Configure JSON formatting for branches.json

8f3d4c2

jozala added 6 commits July 24, 2025 16:06

Update manually branches.json to make it in line with current releases

cb0850c

Apply code review feedback

e2b9fbf

Fix the test to make it pass

42f7117

Change how version.properties is consumed in GlobalBuildInfoPlugin

202a485

This change is needed to make GlobalBuildInfoPlugin testable. Previously it was using static VersionProperties which loaded values once for JVM so it was impossible to test with different values.

Update branches.json after releases

c7f1f62

Add version.properties files for the tests

9dc8160

This file is needed now during the tests as it is no longer fully static during full JVM execution.

jozala force-pushed the bwc-explicit-branches-version branch from 8be969c to 9dc8160 Compare July 24, 2025 14:06

jozala marked this pull request as ready for review July 24, 2025 16:07

jozala requested a review from a team as a code owner July 24, 2025 16:07

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Jul 24, 2025

jozala added the :Delivery/Build Build or test infrastructure label Jul 24, 2025

elasticsearchmachine added the Team:Delivery Meta label for Delivery team label Jul 24, 2025

elasticsearchmachine removed the needs:triage Requires assignment of a team area label label Jul 24, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add explicit versions in the branches.json and use them for calculating unreleased versions for BwC tests #129637

Add explicit versions in the branches.json and use them for calculating unreleased versions for BwC tests #129637

jozala commented Jun 18, 2025

Uh oh!

breskeby Jun 23, 2025

Uh oh!

breskeby Jun 23, 2025

Uh oh!

jozala Jul 7, 2025

Uh oh!

breskeby Jun 23, 2025 •

edited

Loading

Uh oh!

jozala Jul 7, 2025

Uh oh!

jozala commented Jul 7, 2025

Uh oh!

jozala Jul 24, 2025

Uh oh!

jozala Jul 24, 2025 •

edited

Loading

Uh oh!

jozala Jul 24, 2025

Uh oh!

jozala Jul 24, 2025

Uh oh!

elasticsearchmachine commented Jul 24, 2025

Uh oh!

Uh oh!

Add explicit versions in the branches.json and use them for calculating unreleased versions for BwC tests #129637

Are you sure you want to change the base?

Add explicit versions in the branches.json and use them for calculating unreleased versions for BwC tests #129637

Conversation

jozala commented Jun 18, 2025

Uh oh!

breskeby Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

breskeby Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

jozala Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

breskeby Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jozala Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

jozala commented Jul 7, 2025

Uh oh!

jozala Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

jozala Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jozala Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

jozala Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Jul 24, 2025

Uh oh!

Uh oh!

breskeby Jun 23, 2025 •

edited

Loading

jozala Jul 24, 2025 •

edited

Loading