Unable to handle TaskCanceledExceptions in HttpForwarder.SendAsync

[maurice-freitag]

Describe the bug

During load testing we see a lot of HTTP requests being cancelled, presumably by a client due to a timeout. Our API servers react to this by ignoring the corresponding Task-/OperationCanceledExceptions and setting the response code to 499. The gateway (implementing YARP) however still throws an unhandled Exception:

[2024-03-19 04:14:06] warn: Yarp.ReverseProxy.Forwarder.HttpForwarder[48]
RequestCanceled: The request was canceled before receiving a response.
System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
---> System.Net.Sockets.SocketException (125): Operation canceled
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.Metrics.MetricsHandler.SendAsyncWithMetrics(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Yarp.ReverseProxy.Forwarder.HttpForwarder.SendAsync(HttpContext context, String destinationPrefix, HttpMessageInvoker httpClient, ForwarderRequestConfig requestConfig, HttpTransformer transformer, CancellationToken cancellationToken)

This exception seems to be thrown outside of our middleware pipeline as we're unable to handle it. Ideally we would like to react the same way our API servers do, ignoring the exception, and setting the HTTP response code to 499.

Yarp setup:

  services.AddReverseProxy().LoadFromMemory(GetRoutes(), GetClusters());
...
  app.MapReverseProxy(x => x.Use(HandleSafelyAsync)).RequireAuthorization();

  public static async Task HandleSafelyAsync(HttpContext context, Func<Task> next)
  {
      try
      {
          await next();
      }
      catch (Exception ex)
      {
          // this is never called
          var logger = context.RequestServices.GetRequiredService<ILogger<ExceptionHandlingMiddleware>>();
          logger.LogError(ex, "An error occurred while processing '{Route}'. HTTP {StatusCode}, RequestAborted: {RequestAborted}", context.Request.Path, context.Response.StatusCode, context.RequestAborted.IsCancellationRequested);
      }
  }

We're using Yarp.ReverseProxy version 2.1.0.

[MihaZupan]

Looks like you ran into #2195.
These exceptions aren't uncaught (hence you don't see the catch triggering for them), but they are logged by YARP. Since cancellation errors are often outside of your control (e.g. client disconnects), this mainly leads to noise in logs, and #2195 tracks improving that.

[MihaZupan]

Closing as a duplicate of #2195