[BUG] ContainerAppJob.StartAsync() Template Breaks Volume Mount Inheritance for Job Executions

[masj1a]

Library name and version

Azure.ResourceManager.AppContainers 1.4.0

Describe the bug

When programmatically triggering an Azure Container Apps Job execution using the .NET SDK's ContainerAppJobResource.StartAsync() method, and providing a ContainerAppJobExecutionTemplate to pass execution-specific parameters (such as Args or Env variables), the job execution fails to inherit or apply the volume mounts defined on its base ContainerAppJob resource.

This leads to a System.IO.DirectoryNotFoundException when the application inside the triggered job container attempts to access the mounted storage path.

The JobExecutionContainer class within ContainerAppJobExecutionTemplate does not expose properties to specify volumes or volume mounts, implying they should be inherited. However, our testing confirms that the act of providing any ContainerAppJobExecutionTemplate (even a minimal one with just Name, Image, Args, Env) causes the previously defined volume mounts to be ignored by the job execution.

In contrast:

 - Manually triggering the same job (via Azure portal/CLI) works correctly, and the volume is accessible and persistent.
 
 - Omitting the ContainerAppJobExecutionTemplate entirely from the StartAsync call (await containerAppJob.StartAsync(WaitUntil.Completed);) also works correctly, indicating full inheritance in that scenario.

This contradictory behavior points to a bug where programmatic job execution overrides vital base job properties (like volume mounts) even when not explicitly specified, which severely limits the ability to run parameterized jobs with persistent storage.

Expected behavior

When calling ContainerAppJobResource.StartAsync() with a ContainerAppJobExecutionTemplate, the triggered job execution should:

1. Inherit all properties (including VolumeMounts and Volumes) from the base ContainerAppJob definition.
2. Only the properties explicitly specified in the JobExecutionContainer within the provided ContainerAppJobExecutionTemplate should be overridden.
3. Alternatively, if overriding requires re-specifying volumes, the JobExecutionContainer should expose properties for VolumeMounts and Volumes to allow this.

In essence, providing a ContainerAppJobExecutionTemplate should allow for overrides of specific container properties, not a reset or exclusion of implicitly inherited ones like volume mounts. This would enable programmatic triggering of parameterized jobs that rely on persistent storage.

Actual behavior

When calling ContainerAppJobResource.StartAsync() and providing a ContainerAppJobExecutionTemplate (even if only specifying basic properties like Name, Image, Args, Env within the JobExecutionContainer):

1. The job execution fails to inherit or apply the VolumeMounts defined on the base ContainerAppJob resource.
2. The application inside the triggered job container, when attempting to access the mounted path (e.g., /mnt/data), logs a System.IO.DirectoryNotFoundException, indicating the volume is not mounted.
3. This occurs despite direct in-container shell tests and early Program.cs .NET code confirming that the volume is correctly mounted and accessible to other processes within the same container instance.
4. If the ContainerAppJobExecutionTemplate is entirely omitted from the StartAsync call (await containerAppJob.StartAsync(WaitUntil.Completed);), the job execution successfully inherits all properties (including VolumeMounts) and the volume is accessible. This confirms the bug is triggered specifically by the presence of the ContainerAppJobExecutionTemplate.

Reproduction Steps

To reproduce the issue, follow these step-by-step instructions. This guide outlines the setup of the Azure resources and the execution of the .NET applications that demonstrate the bug.

1. Azure Setup

Define your Azure Container App Job with the necessary environment, image, and crucial volume mount configuration.

• Prerequisites:

  • An Azure subscription.

  • Azure CLI installed and logged in (az login).

  • An Azure Container Registry (ACR) where you can push Docker images.

  • Ensure your Container Apps Environment's Managed Identity has Storage Account Contributor role on the Storage Account.

• Steps:
  To reproduce the issue, follow these step-by-step instructions. This guide outlines the setup of the Azure resources and the execution of the .NET applications that demonstrate the bug.

1. Azure Setup

Define your Azure Container App Job with the necessary environment, image, and crucial volume mount configuration.

• Prerequisites:

  • An Azure subscription.

  • Azure CLI installed and logged in (az login).

  • An Azure Container Registry (ACR) where you can push Docker images.

  • Ensure your Container Apps Environment's Managed Identity has Storage Account Contributor role on the Storage Account.

• Steps:

  1. Create an Azure Storage Account and File Share:

     • Create a Storage Account (e.g., myteststorage) and an Azure File Share within it (e.g., my-shared-storage-share).

  2. Create an Azure Container Apps Environment:

     • If you don't have one, create an ACA Environment (e.g., my-container-app-env in resource group my-resource-group).

  3. Link Azure Files Storage to ACA Environment:

     • In your ACA Environment (my-container-app-env), go to "Azure Files" (or "Storage") and link your my-shared-storage-share file share, giving it an internal name (e.g., my-internal-storage-name).

  4. Create the Azure Container App Job (Base Definition):

     • Create the job (e.g., my-unzip-job) using the Azure portal or CLI. This is where you define the crucial volume mount.

       # Example CLI command to create the job with volume mounts
       # Replace placeholders like <your-acr>, <your-env-name>, <your-rg>
       az containerapp job create \
         --name my-unzip-job \
         --resource-group my-resource-group \
         --environment my-container-app-env \
         --trigger-type Manual \
         --image myacr.azurecr.io/myworkerimage:latest \
         --cpu 0.25 --memory 0.5Gi \
         --replica-timeout 1800 \
         --volume-mounts name=my-internal-storage-name,path=/mnt/data \
         --volumes name=my-internal-storage-name,storage-type=AzureFile,storage-name=my-internal-storage-name # 'storage-name' here refers to the name you gave in ACA Env Azure Files

       (Note: storage-name refers to the internal name given to the Azure File Share linkage within the Container Apps Environment, e.g., "my-internal-storage-name".)

2. Prepare the .NET 8 Worker Application

This is the application code that will run inside the Azure Container App Job. It attempts to access the mounted volume and logs its findings.

• Project Structure: Assume your solution has a project (e.g., MyWorker.Unzip) that outputs a DLL (e.g., MyWorker.Unzip.dll).

• Dockerfile (Place in your project's root for building the image):

  # Use the official .NET 8.0 SDK image for building
  FROM [mcr.microsoft.com/dotnet/sdk:8.0](https://mcr.microsoft.com/dotnet/sdk:8.0) AS build
  WORKDIR /src
  
  # Copy csproj and restore as distinct layers
  COPY ["MyWorker.Unzip/*.csproj", "MyWorker.Unzip/"]
  RUN dotnet restore "MyWorker.Unzip/MyWorker.Unzip.csproj"
  
  # Copy the entire source code and build the app
  COPY . .
  WORKDIR /src/MyWorker.Unzip
  RUN dotnet publish -c Release -o out
  
  # Use the official .NET 8.0 runtime image for running
  FROM [mcr.microsoft.com/dotnet/aspnet:8.0](https://mcr.microsoft.com/dotnet/aspnet:8.0) AS runtime
  WORKDIR /app
  COPY --from=build /src/MyWorker.Unzip/out .
  
  # Entrypoint for your console application
  ENTRYPOINT ["dotnet", "MyWorker.Unzip.dll"]

• Program.cs (Main entry point of your worker app, simplified for reproduction):

  using Microsoft.Extensions.Logging;
  using Microsoft.Extensions.DependencyInjection;
  using System;
  using System.IO;
  using System.Threading.Tasks;
  
  class Program
  {
      static async Task Main(string[] args)
      {
          // Basic Logging Setup
          var services = new ServiceCollection();
          services.AddLogging(configure => configure.AddConsole(options =>
          {
              options.LogToStandardErrorThreshold = LogLevel.Error; // Send errors to stderr
          }));
          var serviceProvider = services.BuildServiceProvider();
          var logger = serviceProvider.GetRequiredService<ILogger<Program>>();
  
          // THE PATH THAT SHOULD BE MOUNTED BY AZURE CONTAINER APPS
          string mountedShareRoot = "/mnt/data"; // Ensure this matches your ACA Job's mount path
  
          logger.LogInformation($"[WorkerApp] Starting test. Running as: {Environment.UserName}");
          logger.LogInformation($"[WorkerApp] Application Current Directory: {Environment.CurrentDirectory}");
  
          // Attempt to list contents of root and the mount parent for diagnostic
          try {
              logger.LogInformation($"[WorkerApp] Contents of root (/): {string.Join(", ", Directory.GetDirectories("/"))}");
          } catch (Exception ex) { logger.LogError(ex, "[WorkerApp] Error listing root."); }
  
          // THIS IS THE CODE THAT FAILS WHEN PROGRAMMATICALLY TRIGGERED
          try {
              // This Directory.Exists check is expected to return FALSE in the failing scenario
              logger.LogInformation($"[WorkerApp] Checking mounted path: {mountedShareRoot}");
              if (!Directory.Exists(mountedShareRoot))
              {
                  logger.LogError($"[WorkerApp] DirectoryNotFoundException: Path '{mountedShareRoot}' does not exist (from app code).");
                  throw new DirectoryNotFoundException($"Path '{mountedShareRoot}' does not exist (from app code).");
              }
              else
              {
                  logger.LogInformation($"[WorkerApp] Path '{mountedShareRoot}' found. Attempting to write test file.");
                  string testFile = Path.Combine(mountedShareRoot, $"test_write_{DateTime.UtcNow.Ticks}.txt");
                  await System.IO.File.WriteAllTextAsync(testFile, "This should be written to the mount.");
                  logger.LogInformation($"[WorkerApp] Successfully wrote test file: {testFile}");
              }
          }
          catch (Exception ex)
          {
              logger.LogError(ex, "[WorkerApp] FATAL ERROR: File system test failed.");
              Environment.ExitCode = 1; // Indicate failure
          }
          finally
          {
              logger.LogInformation("[WorkerApp] Test finished.");
          }
      }
  }

• Build and Push Docker Image:

  • Build your .NET worker application into a Docker image.

  • Push this image to your Azure Container Registry (e.g., myacr.azurecr.io/myworkerimage:latest).

3. Prepare the Triggering Application (C# .NET SDK)

This is the separate application that will programmatically trigger the Azure Container App Job execution, demonstrating both the failing and working scenarios.

• Service Class: (Example of a service that would trigger the job)

  using Azure.ResourceManager;
  using Azure.ResourceManager.AppContainers;
  using Azure.ResourceManager.AppContainers.Models;
  using Azure.Identity;
  using Azure; // For ArmOperation, WaitUntil
  using System;
  using System.Collections.Generic;
  using System.Threading.Tasks;
  
  public class JobTriggerService
  {
      private readonly ArmClient _armClient;
      private readonly string _subscriptionId;
      private readonly string _resourceGroupName;
      private readonly string _containerAppJobName;
      private readonly string _workerImageName;
  
      public JobTriggerService(string subscriptionId, string resourceGroupName, string containerAppJobName, string workerImageName)
      {
          _subscriptionId = subscriptionId;
          _resourceGroupName = resourceGroupName;
          _containerAppJobName = containerAppJobName;
          _workerImageName = workerImageName;
  
          // Use DefaultAzureCredential (or ManagedIdentityCredential for Azure-hosted apps)
          var credentials = new DefaultAzureCredential();
          _armClient = new ArmClient(credentials, subscriptionId);
      }
  
      /// <summary>
      /// Programmatically triggers the Azure Container App Job with arguments and env vars.
      /// This is the method that exposes the bug: volume mounts are NOT inherited.
      /// </summary>
      /// <param name="jobIdToPass">A dynamic argument to pass to the job execution.</param>
      public async Task TriggerJobWithArgsAndVolumesIssue(int jobIdToPass)
      {
          ResourceIdentifier containerAppJobResourceId = ContainerAppJobResource.CreateResourceIdentifier(_subscriptionId, _resourceGroupName, _containerAppJobName);
          ContainerAppJobResource containerAppJob = _armClient.GetContainerAppJobResource(containerAppJobResourceId);
  
          // --- SCENARIO A: THIS IS THE FAILING SCENARIO ---
          // This template is used to pass 'Args' or 'Env' for a specific execution.
          // When this template is provided, it *overrides* the base job definition
          // and *excludes* volume mounts, leading to the DirectoryNotFoundException.
          ContainerAppJobExecutionTemplate templateWithOverrides = new ContainerAppJobExecutionTemplate()
          {
              Containers =
              {
                  new JobExecutionContainer()
                  {
                      Name = "my-unzip-container", // Must match container name in job definition
                      Image = _workerImageName, // Must match image in job definition (or be a valid override)
                      Args = { jobIdToPass.ToString() }, // Example: Passing Job ID as a dynamic argument
                      // Example: Adding an environment variable for this specific execution
                      // Env = { new ContainerAppEnvironmentVariable("MY_EXEC_VAR", "dynamic_value_" + jobIdToPass) },
  
                      // IMPORTANT: VolumeMounts and Volumes properties are NOT available on JobExecutionContainer
                      // in the current Azure.ResourceManager.AppContainers SDK.
                  }
              }
          };
  
          Console.WriteLine($"\n[TriggerApp] Attempting to trigger job '{_containerAppJobName}' with template (Args/Env included)...");
          try
          {
              // This is the call that leads to the DirectoryNotFoundException in the job container
              ArmOperation<ContainerAppJobExecutionBase> lro = await containerAppJob.StartAsync(WaitUntil.Completed, template: templateWithOverrides);
              ContainerAppJobExecutionBase results = lro.Value;
              Console.WriteLine($"[TriggerApp] Job execution triggered. Execution Name: {results.Name}, Status: {results.Status}");
              Console.WriteLine($"[TriggerApp] Please check logs for job '{results.Name}' in Azure Portal. Expected: DirectoryNotFoundException for /mnt/data.");
          }
          catch (Exception ex)
          {
              Console.Error.WriteLine($"[TriggerApp] Error triggering job with overrides: {ex.Message}");
              Console.Error.WriteLine($"[TriggerApp] Stack Trace: {ex.StackTrace}");
          }
      }
  
      /// <summary>
      /// Programmatically triggers the Azure Container App Job WITHOUT arguments or env vars.
      /// This is the WORKING scenario, demonstrating proper volume inheritance.
      /// </summary>
      public async Task TriggerJobWithoutOverrides()
      {
          ResourceIdentifier containerAppJobResourceId = ContainerAppJobResource.CreateResourceIdentifier(_subscriptionId, _resourceGroupName, _containerAppJobName);
          ContainerAppJobResource containerAppJob = _armClient.GetContainerAppJobResource(containerAppJobResourceId); // Corrected this line as it was GetContainerAppAppResource
  
          // --- SCENARIO B: THIS IS THE WORKING SCENARIO ---
          // Omitting the template entirely.
          // This allows the job execution to fully inherit ALL properties from the base job definition,
          // including volume mounts. However, no execution-specific args/env can be passed.
          Console.WriteLine($"\n[TriggerApp] Attempting to trigger job '{_containerAppJobName}' WITHOUT template (no Args/Env included)...");
          try
          {
              ArmOperation<ContainerAppJobExecutionBase> lro = await containerAppJob.StartAsync(WaitUntil.Completed); // No template parameter
              ContainerAppJobExecutionBase results = lro.Value;
              Console.WriteLine($"[TriggerApp] Job execution triggered. Execution Name: {results.Name}, Status: {results.Status}");
              Console.WriteLine($"[TriggerApp] Please check logs for job '{results.Name}' in Azure Portal. Expected: Successful volume access.");
          }
          catch (Exception ex)
          {
              Console.Error.WriteLine($"[TriggerApp] Error triggering job without overrides: {ex.Message}");
              Console.Error.WriteLine($"[TriggerApp] Stack Trace: {ex.StackTrace}");
          }
      }
  }

• Program.cs (Main entry point of your triggering application):

  using System;
  using System.Threading.Tasks;
  
  class Program
  {
      static async Task Main(string[] args)
      {
          // Replace with your actual Azure details
          string subscriptionId = Environment.GetEnvironmentVariable("AZURE_SUBSCRIPTION_ID") ?? "<YOUR_SUBSCRIPTION_ID>";
          string resourceGroupName = "my-resource-group";
          string containerAppJobName = "my-unzip-job";
          string workerImageName = "myacr.azurecr.io/myworkerimage:latest";
  
          var triggerService = new JobTriggerService(subscriptionId, resourceGroupName, containerAppJobName, workerImageName);
  
          // --- Run the FAILING scenario ---
          Console.WriteLine("--- Running SCENARIO A (Failing): Triggering with template ---");
          await triggerService.TriggerJobWithArgsAndVolumesIssue(12345);
  
          Console.WriteLine("\n--- Running SCENARIO B (Working): Triggering without template ---");
          await triggerService.TriggerJobWithoutOverrides();
  
          Console.WriteLine("\n[TriggerApp] Application finished.");
      }
  }

4. Verification

After running the triggering application, observe the logs in the Azure portal for your my-unzip-job.

• For SCENARIO A (Triggered with templateWithOverrides):

  • Expected Logs: The job container's logs (ContainerAppConsoleLogs_CL) should show a DirectoryNotFoundException for /mnt/data. No test_write_*.txt file will appear in your Azure File Share from this run.

• For SCENARIO B (Triggered without template parameter):

  • Expected Logs: The job container's logs should show successful access to /mnt/data and report Successfully wrote test file. A test_write_*.txt file will appear in your my-shared-storage-share Azure File Share.

This setup clearly demonstrates the bug where specifying any ContainerAppJobExecutionTemplate prevents volume mounts from being inherited, even when not explicitly attempting to override them.

Environment

No response

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @anthony-c-martin @calecarter @cemheren @j5lim @majastrz.