Source Build Validation legs are failing #114842

am11 · 2025-04-20T12:49:42Z

Source-Build (CentOS9) is failing with:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=1021233&view=logs&jobId=46d4bc78-82b7-5486-55ce-7714ebdf0ac0&j=527da66b-556b-54bb-107f-cdc8a55a2854&t=255f8d3e-16e8-5d6d-908e-7cc01450d912

    Determining projects to restore...
    Restored /__w/1/s/artifacts/sb/package-cache/microsoft.dotnet.arcade.sdk/10.0.0-beta.25216.2/tools/Tools.proj (in 10.25 sec).
    Determining projects to restore...
  /__w/1/s/.dotnet/sdk/10.0.100-preview.3.25201.16/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(108,5): error NETSDK1083: The specified RuntimeIdentifier 'centos.9-x64' is not recognized. See https://aka.ms/netsdk1083 for more information. [/__w/1/s/artifacts/sb/src/Build.proj]
  /__w/1/s/.dotnet/sdk/10.0.100-preview.3.25201.16/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(108,5): error NETSDK1203: Ahead-of-time compilation is not supported for the target runtime identifier 'centos.9-x64'. [/__w/1/s/artifacts/sb/src/Build.proj]
  /__w/1/s/.dotnet/sdk/10.0.100-preview.3.25201.16/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(108,5): error NETSDK1083: The specified RuntimeIdentifier 'centos.9-x64' is not recognized. See https://aka.ms/netsdk1083 for more information. [/__w/1/s/artifacts/sb/src/Build.proj]
  /__w/1/s/.dotnet/sdk/10.0.100-preview.3.25201.16/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(108,5): error NETSDK1203: Ahead-of-time compilation is not supported for the target runtime identifier 'centos.9-x64'. [/__w/1/s/artifacts/sb/src/Build.proj]
    Nothing to do. None of the projects specified contain packages to restore.

and Source-Build (NonexistentRID) - the purpose of this one is to validate that we don't take a dependency on non-portable RID's value and distro maintainer is free to choose the identifier prefix:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=1021233&view=logs&jobId=46d4bc78-82b7-5486-55ce-7714ebdf0ac0&j=46d4bc78-82b7-5486-55ce-7714ebdf0ac0&t=5940e102-b135-58b6-71a4-2593b90abf9d

    Determining projects to restore...
    Restored /__w/1/s/artifacts/sb/package-cache/microsoft.dotnet.arcade.sdk/10.0.0-beta.25216.2/tools/Tools.proj (in 10.83 sec).
    Determining projects to restore...
  /__w/1/s/.dotnet/sdk/10.0.100-preview.3.25201.16/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(108,5): error NETSDK1203: Ahead-of-time compilation is not supported for the target runtime identifier 'linux'. [/__w/1/s/artifacts/sb/src/Build.proj]
  /__w/1/s/.dotnet/sdk/10.0.100-preview.3.25201.16/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(108,5): error NETSDK1203: Ahead-of-time compilation is not supported for the target runtime identifier 'linux'. [/__w/1/s/artifacts/sb/src/Build.proj]
    Nothing to do. None of the projects specified contain packages to restore.

cc @jkoritzinsky, @tmds, last time they appeared passing was in December, then three months logs are missing and since then, all red: https://dev.azure.com/dnceng-public/public/_build?definitionId=129&_a=summary&branchFilter=331%2C331%2C331
it's the second circle:

The text was updated successfully, but these errors were encountered:

dotnet-policy-service · 2025-04-20T12:50:31Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

akoeplinger · 2025-04-22T09:56:09Z

FYI we'll be removing these legs soon: dotnet/source-build#5021

tmds · 2025-04-22T10:14:05Z

Our internal CI shows this as well across all .NET 10.0 jobs.

Build of April 13th (27b2548) was working, build of April 15 fails (b0c2d75).

It probably regressed in #113765.

cc @jkoritzinsky @ViktorHofer

tmds · 2025-04-22T10:14:33Z

cc @omajid

am11 · 2025-04-22T10:18:13Z

FYI we'll be removing these legs soon: dotnet/source-build#5021

It would be best to add their coverage to VMR before removing them from runtime. i.e. dotnet/source-build#4918 then dotnet/source-build#5021.

I think the ultimate test for RID agnostic-ness is NonexistentRID, if we can make it green we can be confident that there is no unintended inference from RID in the code. 🙂

tmds · 2025-04-22T10:24:57Z

I think the ultimate test for RID agnostic-ness is NonexistentRID, if we can make it green we can be confident that there is no unintended inference from RID in the code.

👍

In our internal CI we have a test that build the vmr with some random target rid (that doesn't match /etc/os-release). Then we take those assets to another system (which rid matches neither the initial system nor the target rid) and use them for a non-portable build.

ViktorHofer · 2025-04-22T10:47:57Z

@tmds would you mind filing an issue at https://github.com/dotnet/source-build to track adding a validation job that verifies that the VMR can be built with a random banana target rid?

tmds · 2025-04-22T12:23:30Z

It probably regressed in #113765.

With these changes, BaseOS gets used instead of PackageRID. This means it is required to pass BaseOS for non-portable builds to work. We are not doing this in our CI, and perhaps the CI legs that fail here are not doing it either.

src/libraries/System.Runtime.InteropServices/tests/TestAssets/NativeExports/NativeExports.csproj is missing the part that causes non-portable builds to use BaseOS for restore:

<AppHostRuntimeIdentifier Condition="'$(OutputRID)' != '$(NETCoreSdkRuntimeIdentifier)' and '$(BaseOS)' != '' ">$(BaseOS)</AppHostRuntimeIdentifier>

@ViktorHofer @jkoritzinsky should I make a PR to add this? When #114285 is merged, it is no longer needed.

tmds · 2025-04-22T12:59:24Z

@tmds would you mind filing an issue at https://github.com/dotnet/source-build to track adding a validation job that verifies that the VMR can be built with a random banana target rid?

I thought about this.

Previously .NET rids were determined at runtime from /etc/os-release while for recent versions the rid gets fixed at build time (TargetRid). All non-portable builds we do now are building an "unknown rid" (since the rid isn't part of Microsoft's portable SDKs).

If we mix up the Target RID with the build SDK RID in some places, cross-build CI will pick that up.

As a result, I think we have good coverage already with the vmr CI that is in place.

I assume this issue isn't picked up by that CI because it is in a tests project that the vmr doesn't restore.

am11 · 2025-05-03T12:27:15Z

Looks like it is regressed by --bootstrap change: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1032862&view=logs&jobId=2eda3965-9146-55d1-4226-c3d3f8432d8f&j=46d4bc78-82b7-5486-55ce-7714ebdf0ac0. BaseOS-derived RID graph is somehow not working?

    ILCompiler_publish -> /__w/1/s/artifacts/sb/src/artifacts/bin/ILCompiler_publish/x64/Release/ilc.dll
    Generating native code
    Unhandled exception. System.CommandLine.CommandLineException: Target OS 'banana' is not supported
       at System.CommandLine.Helpers.GetTargetOS(String token) in /_/src/coreclr/tools/Common/CommandLineHelpers.cs:line 84
       at System.CommandLine.Argument`1.<>c__DisplayClass8_0.<set_CustomParser>b__0(ArgumentResult argumentResult, Object& parsedValue)
       at System.CommandLine.Parsing.ArgumentResult.ValidateAndConvert(Boolean useValidators)
       at System.CommandLine.Parsing.CommandResult.ValidateOptions(Boolean completeValidation)
       at System.CommandLine.Parsing.CommandResult.Validate(Boolean completeValidation)
       at System.CommandLine.Parsing.ParseOperation.Validate()
       at System.CommandLine.Parsing.ParseOperation.Parse()
       at System.CommandLine.Parsing.CommandLineParser.Parse(Command command, IReadOnlyList`1 arguments, String rawInput, CommandLineConfiguration configuration)
       at System.CommandLine.CommandLineConfiguration.Invoke(String[] args)
       at ILCompiler.Program.Main(String[] args) in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 790
  /__w/1/s/artifacts/sb/src/artifacts/bin/coreclr/linux.x64.Release/build/Microsoft.NETCore.Native.targets(332,5): error MSB3073: The command ""/__w/1/s/artifacts/sb/src/artifacts/bin/coreclr/linux.x64.Release/x64/ilc/ilc" @"/__w/1/s/artifacts/sb/src/artifacts/obj/coreclr/ILCompiler_publish/linux.x64.Release/native/ilc.ilc.rsp"" exited with code 134. [/__w/1/s/artifacts/sb/src/src/coreclr/tools/aot/ILCompiler/ILCompiler_publish.csproj]
  
  Build FAILED.
  
  /__w/1/s/artifacts/sb/src/artifacts/bin/coreclr/linux.x64.Release/build/Microsoft.NETCore.Native.targets(332,5): error MSB3073: The command ""/__w/1/s/artifacts/sb/src/artifacts/bin/coreclr/linux.x64.Release/x64/ilc/ilc" @"/__w/1/s/artifacts/sb/src/artifacts/obj/coreclr/ILCompiler_publish/linux.x64.Release/native/ilc.ilc.rsp"" exited with code 134. [/__w/1/s/artifacts/sb/src/src/coreclr/tools/aot/ILCompiler/ILCompiler_publish.csproj]
      0 Warning(s)
      1 Error(s)

tmds · 2025-05-03T18:19:04Z

@am11 non-portable build was working after the changes made for #114285 (comment). I assume some later change regressed it.

am11 · 2025-05-03T18:25:31Z

I'm testing with ./build.sh --outputrid centos.9-arm64 --arch arm64 --os linux --bootstrap -p:BaseOS=linux-arm64 --portablebuild false -c release (on arm64 machine) but it fails during --use-bootstrap phase:

Cleaning up artifacts from bootstrap build...
  Determining projects to restore...
  Tool 'coverlet.console' (version '6.0.4') was restored. Available commands: coverlet
  Tool 'dotnet-reportgenerator-globaltool' (version '5.4.3') was restored. Available commands: reportgenerator
  Tool 'microsoft.dotnet.xharness.cli' (version '10.0.0-prerelease.25225.1') was restored. Available commands: xharness
  Tool 'microsoft.visualstudio.slngen.tool' (version '11.1.0') was restored. Available commands: slngen
  
  Restore was successful.
  All projects are up-to-date for restore.
  Determining projects to restore...
/runtime/.dotnet/sdk/10.0.100-preview.3.25201.16/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(108,5): error NETSDK1083: The specified RuntimeIdentifier 'centos.9-arm64' is not recognized. See https://aka.ms/netsdk1083 for more information. [/runtime/Build.proj]
...

Testing in the CI reveals different situation #115272. Trying to figure out the built arg differences..

tmds · 2025-05-03T19:45:25Z

The command from that comment fails against main.

am11 · 2025-05-03T20:06:40Z

Ah thanks. For some reason, my previous click scrolled to your next comment. Let me take a look.

dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Apr 20, 2025

dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Apr 20, 2025

am11 added area-Infrastructure and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Apr 20, 2025

github-project-automation bot added this to Runtime Infra Apr 20, 2025

tmds mentioned this issue Apr 23, 2025

Fix non-portable CI builds. #114949

Merged

jkoritzinsky closed this as completed in #114949 Apr 23, 2025

github-project-automation bot moved this to Done in Runtime Infra Apr 23, 2025

dotnet-policy-service bot removed the untriaged New issue has not been triaged by the area owner label Apr 23, 2025

This was referenced May 3, 2025

[main] Source code updates from dotnet/runtime dotnet/dotnet#365

Open

Fix source-build validation leg #115272

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Build Validation legs are failing #114842

Source Build Validation legs are failing #114842

am11 commented Apr 20, 2025 •

edited

Loading

dotnet-policy-service bot commented Apr 20, 2025

akoeplinger commented Apr 22, 2025

tmds commented Apr 22, 2025 •

edited

Loading

tmds commented Apr 22, 2025

am11 commented Apr 22, 2025

tmds commented Apr 22, 2025

ViktorHofer commented Apr 22, 2025

tmds commented Apr 22, 2025

tmds commented Apr 22, 2025

am11 commented May 3, 2025 •

edited

Loading

tmds commented May 3, 2025

am11 commented May 3, 2025

tmds commented May 3, 2025

am11 commented May 3, 2025

Source Build Validation legs are failing #114842

Source Build Validation legs are failing #114842

Comments

am11 commented Apr 20, 2025 • edited Loading

dotnet-policy-service bot commented Apr 20, 2025

akoeplinger commented Apr 22, 2025

tmds commented Apr 22, 2025 • edited Loading

tmds commented Apr 22, 2025

am11 commented Apr 22, 2025

tmds commented Apr 22, 2025

ViktorHofer commented Apr 22, 2025

tmds commented Apr 22, 2025

tmds commented Apr 22, 2025

am11 commented May 3, 2025 • edited Loading

tmds commented May 3, 2025

am11 commented May 3, 2025

tmds commented May 3, 2025

am11 commented May 3, 2025

am11 commented Apr 20, 2025 •

edited

Loading

tmds commented Apr 22, 2025 •

edited

Loading

am11 commented May 3, 2025 •

edited

Loading