3 times longer build time after upgrading to MAUI 8 SR9

[albyrock87]

Apple platform

iOS

Framework version

net8.0-*

Affected platform version

8.0.401

Description

I'm using ADO macos14 image (Mac-1727286443307) to build our MAUI app targeting iOS and Android.
After upgrading from MAUI 8.0.82 to 8.0.90 the iOS build time went from 12 minutes to 37 minutes with basically no changes to the code besides upgrading the MAUI package.

This unfortunately breaks the entire ADO pipeline because it goes over 60 minutes.

I have UseInterpreter set to true and no other special configuration set.

The majority of time seems to be spent after I see the message Optimizing assemblies for size. This process might take a while..

Steps to Reproduce

This is my azure-pipeline.yml:

- script: >
    dotnet publish $(projectPath)
    -f net8.0-ios
    -r ios-arm64
    -c Release
    -p:BuildIpa=True
    -p:CodesignProvision="$(APPLE_PROV_PROFILE_UUID)"
    -p:CodesignKey="$(APPLE_CERTIFICATE_SIGNING_IDENTITY)"

Did you find any workaround?

No response

Build logs

Using -v diag I've noticed the majority of time is spent in tasks like these:

2024-09-25T18:24:13.9042650Z                      Tool /Users/runner/.dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/8.0.8/Sdk/../tools/mono-aot-cross execution started with arguments: --path=/Users/runner/work/1/s/[Project.Name.Redacted]/obj/Release/net8.0-ios/ios-arm64/linked --aot=mtriple=arm64-ios,data-outfile=obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/BouncyCastle.Crypto.aotdata,static,asmonly,dedup-skip,direct-icalls,interp,full,readonly-value=ObjCRuntime.Runtime.Arch=i4/0,nodebug,dwarfdebug,llvm-path=/Users/runner/.dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/8.0.8/Sdk/../tools,outfile=obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/BouncyCastle.Crypto.dll.s,llvm-outfile=obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/BouncyCastle.Crypto.dll.llvm.o --debug --llvm -O=gsharedvt /Users/runner/work/1/s/[Project.Name.Redacted]/obj/Release/net8.0-ios/ios-arm64/linked/BouncyCastle.Crypto.dll
2024-09-25T18:24:13.9049710Z                               (TaskId:546)
2024-09-25T18:26:08.4803790Z                      Tool /Users/runner/.dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/8.0.8/Sdk/../tools/mono-aot-cross execution finished (exit code = 0).
2024-09-25T18:26:08.4949170Z                               (TaskId:546)
2024-09-25T18:26:08.4985480Z                      Mono Ahead of Time compiler - compiling assembly /Users/runner/work/1/s/[Project.Name.Redacted]/obj/Release/net8.0-ios/ios-arm64/linked/Microsoft.Maui.Controls.dll
2024-09-25T18:26:08.5244200Z                      AOTID 82E2F5B2-12E6-B763-8EB2-B5B559858F2F
2024-09-25T18:26:08.5311080Z                      Executing opt: "/Users/runner/.dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/8.0.8/Sdk/../tools/opt" -f -disable-tail-calls -passes="default<O2>,place-safepoints" -spp-all-backedges -o "obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/Microsoft.Maui.Controls.dll.s.opt.bc" "obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/Microsoft.Maui.Controls.dll.s.bc"
2024-09-25T18:26:08.5329660Z                      Executing llc: "/Users/runner/.dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/8.0.8/Sdk/../tools/llc"  -enable-implicit-null-checks -disable-fault-maps -march=aarch64 -asm-verbose=false -mtriple=arm64-ios -disable-gnu-eh-frame -enable-mono-eh-frame -mono-eh-frame-symbol=_mono_aot_Microsoft_Maui_Controls_eh_frame -disable-tail-calls -relocation-model=static -filetype=obj -o "obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/Microsoft.Maui.Controls.dll.llvm.o" "obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/Microsoft.Maui.Controls.dll.s.opt.bc"
2024-09-25T18:26:08.5339160Z                      Compiled: 14933/14933
2024-09-25T18:26:08.5348630Z                      Output file: 'obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/Microsoft.Maui.Controls.dll.s'.
2024-09-25T18:26:08.5362340Z                      Linking symbol: '_mono_aot_module_Microsoft_Maui_Controls_info'.
2024-09-25T18:26:08.5379710Z                      LLVM output file: 'obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/Microsoft.Maui.Controls.dll.llvm.o'.
2024-09-25T18:26:08.6108300Z                      JIT time: 5591 ms, Generation time: 150828 ms, Assembly+Link time: 0 ms.

2024-09-25T18:21:25.7283940Z                      Tool /Users/runner/.dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/8.0.8/Sdk/../tools/mono-aot-cross execution started with arguments: --path=/Users/runner/work/1/s/[Project.Name.Redacted]/obj/Release/net8.0-ios/ios-arm64/linked --aot=mtriple=arm64-ios,data-outfile=obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/PhoneNumbers.aotdata,static,asmonly,dedup-skip,direct-icalls,interp,full,readonly-value=ObjCRuntime.Runtime.Arch=i4/0,nodebug,dwarfdebug,llvm-path=/Users/runner/.dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/8.0.8/Sdk/../tools,outfile=obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/PhoneNumbers.dll.s,llvm-outfile=obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/PhoneNumbers.dll.llvm.o --debug --llvm -O=gsharedvt /Users/runner/work/1/s/[Project.Name.Redacted]/obj/Release/net8.0-ios/ios-arm64/linked/PhoneNumbers.dll
2024-09-25T18:21:25.7286320Z                               (TaskId:546)
2024-09-25T18:22:47.9847480Z                      Tool /Users/runner/.dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/8.0.8/Sdk/../tools/mono-aot-cross execution finished (exit code = 0).
2024-09-25T18:22:47.9849540Z                               (TaskId:546)
2024-09-25T18:22:47.9863890Z                      Mono Ahead of Time compiler - compiling assembly /Users/runner/work/1/s/[Project.Name.Redacted]/obj/Release/net8.0-ios/ios-arm64/linked/aot-instances.dll
2024-09-25T18:22:47.9866360Z                      AOTID 2FB68679-04E6-1118-039D-A0169676F9E6
2024-09-25T18:22:47.9867000Z                      Adding 68544 dedup-ed methods.
2024-09-25T18:22:47.9868810Z                      Executing opt: "/Users/runner/.dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/8.0.8/Sdk/../tools/opt" -f -disable-tail-calls -passes="default<O2>,place-safepoints" -spp-all-backedges -o "obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/aot-instances.dll.s.opt.bc" "obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/aot-instances.dll.s.bc"
2024-09-25T18:22:47.9872200Z                      Executing llc: "/Users/runner/.dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/8.0.8/Sdk/../tools/llc"  -enable-implicit-null-checks -disable-fault-maps -march=aarch64 -asm-verbose=false -mtriple=arm64-ios -disable-gnu-eh-frame -enable-mono-eh-frame -mono-eh-frame-symbol=_mono_aot_aot_instances_eh_frame -disable-tail-calls -relocation-model=static -filetype=obj -o "obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/aot-instances.dll.llvm.o" "obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/aot-instances.dll.s.opt.bc"
2024-09-25T18:22:47.9873540Z                      Compiled: 59471/59485
2024-09-25T18:22:47.9874410Z                      Output file: 'obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/aot-instances.dll.s'.
2024-09-25T18:22:47.9875230Z                      Linking symbol: '_mono_aot_module_aot_instances_info'.
2024-09-25T18:22:47.9876320Z                      LLVM output file: 'obj/Release/net8.0-ios/ios-arm64/nativelibraries/aot-output/arm64/aot-instances.dll.llvm.o'.
2024-09-25T18:22:47.9877510Z                      JIT time: 41698 ms, Generation time: 824892 ms, Assembly+Link time: 6 ms.

[rolfbjarne]

Can you get a binlog by adding -bl:msbuild.binlog to the build command:

- script: >
    dotnet publish $(projectPath)
    -f net8.0-ios
    -r ios-arm64
    -c Release
    -p:BuildIpa=True
    -p:CodesignProvision="$(APPLE_PROV_PROFILE_UUID)"
    -p:CodesignKey="$(APPLE_CERTIFICATE_SIGNING_IDENTITY)"
    -bl:msbuild.binlog

and then attach the resulting binlog here?

If at all possible, it would be great if you could get a binlog from a previous MAUI version, so that we can compare exactly where the time difference is.

[albyrock87]

@rolfbjarne I cannot upload a binlog here due to company policies, but I've investigated more and I've solved the issue, but in a strange way.

When upgrading to MAUI 8.0.90 I also updated my public library https://github.com/nalu-development/nalu from version 3.1.1 to 8.0.0 where I changed the Directory.Build.Props to use MauiVersion = 8.0.90 instead of the original 8.0.10.

So:

• I've reverted everything and just updated to my app MAUI 8.0.90
  • Result: Build time not changed (~20min)
• At this point it was clear the issue was somehow it Nalu library
• I changed the Directory.Build.Props in Nalu to use MauiVersion = 8.0.82
• I released a new 8.0.1 version of Nalu and upgraded the package dependency in my app
  • Result: Build time not changed (~20min)

This makes me think that somehow building a MAUI library into NuGet packages by using MAUI 8.0.90 is causing something strange in the output DLL.

I will try to create a simple how-to-repro repository and get back to you, in the mean time I hope the additional info rings some bell :)

[DDHSchmidt]

I would like to chime in with the increased build time. From 2 days ago to today, our build time went from 20-25 minutes to 50-55 minutes on average.
The main offender according to the binlog is the target "_AOTCompile"

We don't use <PublishAot> in our csproj, but <MtouchInterpreter>-all</MtouchInterpreter> together with <MtouchLink>SdkOnly</MtouchLink>

It's hard to compare with an exact build from some days ago since we had pinned neither workload versions, workload sets, etc.
The changes from the past days have also wrecked havoc on our build and it was a pita to get it running again at all...

@rolfbjarne I can privately message you the binlog or anything else you need to retrace this?

[rolfbjarne]

@DDHSchmidt you can send me the binlog at Rolf.Kvinge@microsoft.com if you want, I can have a look, although I'm not entirely sure I'll figure out what's going on without a binlog from a previous, fast build.

[rolfbjarne]

Thanks for the binlog, but nothing obvious (except that the AOTCompile takes a long time) stands out.

Assuming you're able to reproduce this locally, one idea might be to install the previous version of the workloads, get a binlog, install the new version and get a binlog again (see dotnet/maui#24819 (comment) for how to go back and forth).

[albyrock87]

Today I got 40 minutes even with 8.0.82, so at this point I think this is just an ADO agent performance issue.
Closing for now.

https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#recent-updates

macOS capacity is currently limited. Unlike Linux and Windows images, where our capacity is restrained by Azure's all up capacity, macOS capacity is constrained by the amount of hardware we have available. While we are working to make additional capacity available over Spring 2024, some jobs may experience delayed execution. Wherever possible, e.g. for jobs that do not create Apple ecosystem apps, customers should choose Linux or Windows images.