Write to cache when building a template

[djeebus]

Note

Adds optional NFS-backed caching when building templates and implements write-through local caching in storage, wiring flags and usage across orchestrator components with updated tests/mocks.

• Orchestrator build pipeline:
  • Pass featureFlags into build.NewBuilder and template server; use env SHARED_CHUNK_CACHE_PATH.
  • When flag feature_flags.BuildingFeatureFlagName is enabled, wrap template storage with storage.NewCachedProvider for build reads/writes.
  • Adjust rootfs size and layer executors to use the possibly-wrapped templateStorage.
• Storage cache (packages/shared/pkg/storage):
  • Implement write-through caching:
    • CachedFileObjectProvider.WriteFromFileSystem and Write now populate local chunk cache while writing to remote.
    • Async local cache cleanup on Delete and DeleteObjectsWithPrefix.
    • Concurrency-limited chunking for large files; tracing and metrics attributes added.
  • Helper utilities: error recording, safe removals; minor struct/param renames.
• Feature flags:
  • Add BuildingFeatureFlagName (use NFS for building templates).
• Tests and mocks:
  • Add/replace mockery-generated mocks in packages/shared/pkg/storage.
  • Expand cache tests (write-through, chunking, delete prefix); update to new mocks.
• Config:
  • Update .mockery.yaml to generate storage mocks in packages/shared/pkg/storage.

^{Written by Cursor Bugbot for commit b6a786c. This will update automatically on new commits. Configure here.}

[dobrac]

how about using errorgroup instead which has included concurrency limit by default?

[djeebus]

errgroup will cancel the context and goroutines if an error is encountered, which we don't want. If a single chunk fails for some reason, we still want all of the rest of the chunks to write to disk.

ready for a new review!

[ValentaTomas]

@djeebus Let's split to PR preparing for this change (refactor) and adding the caching behavior.

[djeebus]

@djeebus Let's split to PR preparing for this change (refactor) and adding the caching behavior.

OK, I've removed all the nice-to-haves, the rest will go in other PRs.

[djeebus]

There is currently a mismatch between reading/writing whole files and reading/writing chunked files. We need to do things:

1. Line things up so that the template writes the same files that the orchestrator expects to read
2. Find a way to throw errors if the function calls don't line up across structs

[djeebus]

This is blocked by #1361 ; will refactor after that gets merged.