HttpServiceEndpointResolver leaked per HTTP handler build in Microsoft.Extensions.ServiceDiscovery

[epricer-polly]

Description

AddServiceDiscovery allocates a new HttpServiceEndpointResolver for every HTTP message handler that gets built. Because HttpClientFactory rebuilds handler chains on the normal handler-lifetime rotation (default 2 minutes), a fresh resolver is created periodically for the entire life of the process.

HttpServiceEndpointResolver is IAsyncDisposable and owns endpoint-refresh timers plus configuration change-token subscriptions, but ResolvingHttpDelegatingHandler wraps it and never disposes it. Each orphaned resolver stays rooted (by the runtime timer queue and the configuration root) and never becomes eligible for collection.

Its live timers keep firing, so the leak manifests as a slow, uptime-correlated CPU climb (and a steadily growing object count), not a memory blowup. I found this watching a new minimal service slowly but surely grow in CPU utilization over the course of days. Because the churn is driven by handler rotation rather than request volume, it does not reproduce under short load tests, but rather only under sustained uptime.

Reproduction Steps

using System.Reflection;
using Microsoft.Extensions.DependencyInjection;

var services = new ServiceCollection();
services.AddHttpClient("svc")
    .AddServiceDiscovery()
    .SetHandlerLifetime(TimeSpan.FromSeconds(1)); // short so the chain rebuilds quickly
await using var provider = services.BuildServiceProvider();

var factory = provider.GetRequiredService<IHttpMessageHandlerFactory>();
var resolvers = new HashSet<object>(ReferenceEqualityComparer.Instance);

for (var i = 0; i < 4; i++)
{
    var handler = factory.CreateHandler("svc"); // rebuilt after each lifetime expiry
    var resolving = FindHandler(handler, "ResolvingHttpDelegatingHandler");
    var resolver = resolving.GetType()
        .GetField("_resolver", BindingFlags.Instance | BindingFlags.NonPublic)!
        .GetValue(resolving)!;
    resolvers.Add(resolver);
    Console.WriteLine($"iteration {i}: distinct resolvers so far = {resolvers.Count}");
    await Task.Delay(TimeSpan.FromMilliseconds(1200)); // let the handler lifetime expire
}

Console.WriteLine($"TOTAL distinct HttpServiceEndpointResolver instances: {resolvers.Count}");

static HttpMessageHandler FindHandler(HttpMessageHandler handler, string typeName)
{
    for (var c = handler; c is not null; c = (c as DelegatingHandler)?.InnerHandler)
    {
        if (c.GetType().Name == typeName)
        {
            return c;
        }
    }

    throw new InvalidOperationException($"{typeName} not found in handler chain");
}

Output

iteration 0: distinct resolvers so far = 1
iteration 1: distinct resolvers so far = 2
iteration 2: distinct resolvers so far = 3
iteration 3: distinct resolvers so far = 4
TOTAL distinct HttpServiceEndpointResolver instances: 4

Expected behavior

Number of HttpServiceEndpointResolvers stays constant or constrained

Actual behavior

Number of HttpServiceEndpointResolvers grows over time

Regression?

Exists in 10.6.0 and 10.8.0 - Unknown if it exists prior.

Known Workarounds

Pin the handler lifetime so the chain (and its resolver) isn't rebuilt: ConfigureHttpClientDefaults(b => b.SetHandlerLifetime(Timeout.InfiniteTimeSpan)), paired with SocketsHttpHandler.PooledConnectionLifetime to preserve DNS refresh.

Configuration

Dotnet 10.0.302
Azure container apps
Not specific to that config

Other information

No response

[MihuBot]

I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):

• HttpServiceEndpointResolver timers leak on IHttpClientFactory handler rotation when using AddServiceDiscovery() microsoft/aspire#15673