[BUG] ObjectDisposedException when mocking HttpResponseMessage content via a System.Net.Http.DelegatingHandler on the underlying HTTP client.

[kalbert312]

Library name and version

Azure.ResourceManager.Resources 1.6.0, Azure.ResourceManager 1.7.0, Azure.Core 1.34.0

Describe the bug

I'm encountering an ObjectDisposedException in our unit test environment that uses a DelegatingHandler to mock out response content. See the repro steps for relevant code snippets.

After debugging, I discovered the following:

• In Azure.Core.Pipeline.HttpClientTransport::ProcessAsync(HttpMessage message, bool async), the StringContent gets read as a MemoryStream and is passed to the SDK's response abstraction PipelineResponse which is an IDisposable that when disposed, will dispose the underlying HttpResponseMessage and its content.
• In Azure.Core.Pipeline.ResponseBodyPolicy, if the content stream is a non-seekable stream and message.BufferResponse is true, a setter message.Response.ContentStream = bufferedStream is called. This invokes the overridden setter in Azure.Core.Pipeline.PipelineResponse which nulls the Content on HttpResponseMessage
  • This step does not occur in the unit test setup because responseContentStream.CanSeek is true for a MemoryStream
• In Azure.ResourceManager.Resources.ArmDeploymentResource::UpdateAsync(...) the HttpMessage is disposed after CreateOrUpdateAtScopeAsync is done, which will dispose the StringContent on the response since it was not nulled by the previous point.
• The SDK then wraps the response with a ResourcesArmOperation which leads to the disposed exception.

Expected behavior

Be able to mock a long running operation's intermediate response at the HTTP layer via DelegatingHandler.

Actual behavior

System.ObjectDisposedException: Cannot access a closed Stream.
   at System.IO.__Error.StreamIsClosed()
   at System.IO.MemoryStream.get_Length()
   at Azure.Core.NextLinkOperationImplementation.IsFinalState(Response response, HeaderSource headerSource, Nullable`1& failureState, String& resourceLocation)
   at Azure.Core.NextLinkOperationImplementation.Create(HttpPipeline pipeline, RequestMethod requestMethod, Uri startRequestUri, Response response, OperationFinalStateVia finalStateVia, Boolean skipApiVersionOverride, String apiVersionOverrideValue)
   at Azure.Core.NextLinkOperationImplementation.Create[T](IOperationSource`1 operationSource, HttpPipeline pipeline, RequestMethod requestMethod, Uri startRequestUri, Response response, OperationFinalStateVia finalStateVia, Boolean skipApiVersionOverride, String apiVersionOverrideValue)
   at Azure.ResourceManager.Resources.ResourcesArmOperation`1..ctor(IOperationSource`1 source, ClientDiagnostics clientDiagnostics, HttpPipeline pipeline, Request request, Response response, OperationFinalStateVia finalStateVia, Boolean skipApiVersionOverride, String apiVersionOverrideValue)
   at Azure.ResourceManager.Resources.ArmDeploymentResource.<UpdateAsync>d__20.MoveNext()

Reproduction Steps

The mock delegating handler:

public class MockRequestHandler : DelegatingHandler
{
    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        // ...

        var response = new HttpResponseMessage();
        response.Content = new StringContent(/* mock json */);

        // it does NOT call base.SendAsync()

        // ...

        return Task.FromResult(response);
    }

The http client:

var webRequestHandler = new WebRequestHandler { AllowAutoRedirect = false };

var httpClient = new HttpClient(
    handler: HttpClientFactory.CreatePipeline(innerHandler: webRequestHandler, handlers: delegatingHandlers), // mock handler goes here
    disposeHandler: true);

The ArmClient is initialized as follows:

var armClientOptions = new ArmClientOptions
{
  // ...
  Transport = new HttpClientTransport(httpClient) // http client with the delegating handler
};

var armClient = new ArmClient(tokenCredential, default, armClientOptions);

The ArmClient call:

var armDeploymentSdkResource = armClient.GetArmDeploymentResource(/* ResourceIdentifier */);
var deploymentOperation = await armDeploymentSdkResource
    .UpdateAsync(WaitUntil.Started, deploymentRequestInput, this.CancellationToken);

Environment

Windows 11
System.Runtime.InteropServices.RuntimeInformation.FrameworkDescription = ".NET Framework 4.8.9167.0"
JetBrains Rider 2023.2.1

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jsquire]

//cc: @m-nash, @annelo-msft

I think this may turn out to be the same root cause as #38219, with what we currently suspect is the root cause discussed here.

[annelo-msft]

Thanks, @jsquire! I was planning to spend some time looking at #38219 today, so I'll look at this one as well while I'm doing that.

[annelo-msft]

@kalbert312, thanks for a really nice investigation and repro case! I have confirmed that this is the same issue as we're looking at in #38219. I'm going to close it as a duplicate, but I'm also tagging the other one as Azure.Core, and will try to turn around a fix soon. Thanks for reporting this!