2.13.10 suddenly giving errors

[liukonen]

using VALKEY

Unhandled exception. StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code.

at StackExchange.Redis.ConnectionMultiplexer.ConnectImpl(ConfigurationOptions configuration, TextWriter writer, Nullable`1 serverType, EndPointCollection endpoints) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 691

at StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 663

at StackExchange.Redis.ConnectionMultiplexer.Connect(String configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 641

at StocksRssFeed.Startup.ConfigureServices(IServiceCollection services)

at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)

at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)

at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)

at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services, Object instance)

[mgravell]

Well that's definitely not intentional! I've marked that build as hidden for now; will investigate.

[mgravell]

Question: what version are you upgrading from? I'm trying to but bookends on where I look for deltas.

[mgravell]

OK, I've had time to look, and I'm not seeing any fundamental connection problems with Valkey or any other backend - for example, testing locally against valkey_version:9.1.0: it works fine.

So right now I have no clue; I'm going to need your help!

Does the library fail for you when targeting a local Valkey server? for example docker container run --name valkey -p 6379:6379 -d valkey/valkey? (this spins up the current default valkey dockerhub image locally and exposes it to the host OS on port 6379). If that then fails to connect, it is probably something like assembly binding - we'd need to figure out what.

After that there's a few ways we can look into this:

1. pass a StringWriter (or similar) into the log parameter of Connect, so we can get verbose information about every step in connect and handshake; note that this may include server names etc, so you might want to eyeball it for anything privileged
2. you could enable RESP logging; this WILL contain any AUTH etc tokens, but provides the absolute most detail for any conversation between the endpoints

I've masked the release while I investigate, but: "works for meTM", so I'm going to need some help to get to the bottom of this.

Obviously don't post full logs here; I am available at marc.gravell@gmail.com and marc.gravell@redis.com

[liukonen]

Hi Marc,

Thanks — I ran the local Valkey repro exactly as suggested:

docker run --name valkey -p 6379:6379 -d valkey/valkey
Result
✔ Valkey is reachable on localhost:6379
✔ Connection works locally against the same container
✔ Same StackExchange.Redis version that fails in our Docker network scenario
✔ No failure reproduced in this local single-container setup

So:
local host repro works fine
Key difference vs failing environment
The failure only occurs when:

• App runs in Docker container
• Valkey runs in separate container
• Connected via Docker bridge network (service name DNS: valkey)

In that setup, latest StackExchange.Redis:

• Resolves endpoint ✔
• Begins connect ✔
• Waits for response ❌
• Times out after ~10s
• Repeats reconfiguration cycles
• Never completes handshake

Previous StackExchange.Redis version:

Same environment
Same network
Connects successfully immediately
Logging (as requested)

I captured verbose connect logs via StringWriter / Console.Out.

Key pattern:

BeginConnectAsync
Allowing 1 endpoint(s) 00:00:10 to respond...
Did not respond
Timeout 10000ms
WaitingForActivation
No primaries detected

No:

AUTH errors
TLS errors
protocol negotiation failures

Just silent non-response during handshake window.

RESP logging

I can enable RESP-level logging if needed, but current logs already show that the failure occurs before any meaningful protocol exchange completes.

Conclusion
Local single-container Valkey: ✔ works
Multi-container Docker network: ❌ fails (latest client only)
Prior StackExchange.Redis version: ✔ works in same multi-container setup

So the regression appears specific to container-to-container networking behavior in the latest client, not Valkey compatibility in general or single-node connectivity.

Attached the output of my docker logs. I dont see anything privileged

Also, I am running this in DotNet 8 on x64 hardware using ubuntu 24.04.4, and using microsoft Chisled repos as my base image

_stocksrssfeed_logs.txt

My "internal" network, where my containers live, is set up as

    "Name": "interlink",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv4": true,
    "EnableIPv6": false,
    "IPAM": {
        "Driver": "default",
        "Options": {},
    },
    "ConfigFrom": {
        "Network": ""
    },

[mgravell]

Ok, that's super interesting, thanks. Will look after the weekend (UK public holiday). If you have any specific docker config for that setup you can share, that might save some time and remove any false paths, but if not: I'll do what I can.

[wipiano]

Attempted to reproduce with a minimal repro: docker compose + small Program.cs using ConnectionMultiplexer.ConnectAsync("valkey", log).

compose.yaml:

services:
  consoleapp1:
    image: consoleapp1
    build:
      context: .
      dockerfile: ConsoleApp1/Dockerfile
    networks:
      - interlink
    depends_on:
      valkey:
        condition: service_started
  valkey:
    image: valkey/valkey
    networks:
      - interlink

networks:
  interlink:
    driver: bridge
    internal: true
    enable_ipv6: false

Findings:

• Initially reproduced the reported symptom (OnConnectedAsync init logged, never completes; 3x retry then RedisConnectionException) reliably.
• I tried socket.ConnectAsync(new DnsEndPoint("valkey", 6379)) without StackExchange.Redis but failed (SocketErrorCode is TryAgain and NativeErrorCode is 11) independent of TCP Keep-Alive.
• After compose.yaml changed (nicolaka/netshoot service was added, and recreate network bridges and services), the failure no longer reproduces - even with the exact same compose/Dockerfile/Program.cs.
• systemctl restart docker does not re-provoke it.
• Direct Dns.GetHostAddressesAsync("valkey") returns SocketError.TryAgain in the failing state, indicating the underlying issue is DNS resolution against docker's embedded DNS, not a TCP connect hang.

This may be not a regression in StackExchange.Redis. It looks like a transient state in docker's bridge networking / embedded DNS that I cannot reliably re-provoke now.

[wipiano]

Additional note: this was on a docker install that was less than a few hours old at the time of reproduction. After many compose up/down cycles + a daemon restart + OS reboot, I can no longer trigger the failure on the same host. This may further support a transient initialization-state hypothesis on docker's side rather than anything in StackExchange.Redis.

[liukonen]

Reproducible console app
If I migrate to dotnet 10, the code works, so it seems to be isolated to dotnet 8
https://github.com/liukonen/SR_Bug

[wipiano]

After more careful testing I realized I had switched my base image from mcr.microsoft.com/dotnet/runtime:8.0-noble-chiseled to mcr.microsoft.com/dotnet/runtime:10.0 (non-chiseled) at some point during my investigation.
The "I can't reproduce anymore" observations was made on the non-chiseled .NET 10 image.

Going back to 8.0-noble-chiseled, the failure reproduces reliably, and crucially the SE.Redis version does matter.

So this is a 2.13.10 regression, but it only surfaces on chiseled .NET 8 base images.

Sorry for the noise - I should have pinned the image tag from the start.

[wipiano]

I think the cause is in the .NET runtime layer, not StackExchange.Redis directly, but #3078 is the trigger because it added the SetSocketOption(SO_KEEPALIVE, ...) call.

On the .NET 8.0 chiseled image, calling Socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, [true/false]) on a socket constructed via the 2-arg new Socket(SocketType, ProtocolType) ctor before calling Socket.ConnectAsync(SAEA) with a DnsEndPoint causes Socket.ConnectAsync to immediately throw:

PlatformNotSupportedException: "Sockets on this platform are invalid for use after a failed connection attempt."

minimal PoC without StackExchange.Redis:

using System.Net;
using System.Net.Sockets;

using var stdout = Console.OpenStandardOutput();
using var sw = new StreamWriter(stdout);
sw.AutoFlush = true;

await Task.Delay(500); // wait valkey server start

await TestAsync(true, sw);
await TestAsync(false, sw);
await TestAsync(null, sw);

async Task TestAsync(bool? keepAlive, StreamWriter sw)
{
    using var socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
    if (keepAlive.HasValue)
    {
        socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, keepAlive.Value);
        sw.WriteLine("SO_KEEPALIVE={keepAlive}");
    }
    else
    {
        sw.WriteLine("No SetSocketOption call");
    }

    var tcs = new TaskCompletionSource();
    var args = new SocketAsyncEventArgs
    {
        RemoteEndPoint = new DnsEndPoint("valkey", 6379),
    };
    args.Completed += (_, _) => tcs.TrySetResult();

    bool pending;
    try
    {
        pending = socket.ConnectAsync(args);
    }
    catch (Exception ex)
    {
        Console.WriteLine($"ConnectAsync threw: {ex.GetType().Name}: {ex.Message}");
        return;
    }

    if (pending)
    {
        var done = await Task.WhenAny(tcs.Task, Task.Delay(TimeSpan.FromSeconds(10)));
        if (done != tcs.Task)
        {
            Console.WriteLine("TIMEOUT");
            return;
        }
    }
    
    Console.WriteLine($"SocketError={args.SocketError}");
}

await sw.FlushAsync();

Confirmed result matrix (chiseled .NET 8 / valkey on same internal bridge):

Pre-connect call	socket.ConnectAsync(SAEA{ DnsEndPoint})
SetSocketOption(KeepAlive, true)	throws PlatformNotSupportedException
SetSocketOption(KeepAlive, false)	throws PlatformNotSupportedException
no SetSocketOption call	succeeds (SocketError.Success)

I only tested SO_KEEPALIVE, not other socket options - the trigger may be narrower or broader than that.

Verified fix: moving the SetSocketOption(SOL_SOCKET, SO_KEEPALIVE, true) call from SocketManager.CreateSocket to after the Socket.ConnectAsync completes elimnates the PNSE and restores connectivity on chiseled .NET 8.

[wipiano]

Confirmed with a small repro that the problematic version of StackExchange.Redis does throw the PlatformNotSupportedException mentioned above, and that it gets silently swallowed:

AppDomain.CurrentDomain.FirstChanceException += (_, e) =>
{
    if (e.Exception is PlatformNotSupportedException pnse)
    {
        Console.WriteLine($"[FirstChance] {pnse.GetType().Name}: {pnse.Message}");
        Console.WriteLine(pnse.StackTrace);
        Console.WriteLine("------");
    }
};

using var mul = ConnectionMultiplexer.Connect("valkey");
Console.WriteLine(mul.GetDatabase().StringIncrement("test"));

Output:

consoleapp1-1  | [FirstChance] PlatformNotSupportedException: Sockets on this platform are invalid for use after a failed connection attempt.
consoleapp1-1  |    at System.Net.Sockets.Socket.ThrowMultiConnectNotSupported()
consoleapp1-1  | ------
consoleapp1-1  | [FirstChance] PlatformNotSupportedException: Sockets on this platform are invalid for use after a failed connection attempt.
consoleapp1-1  |    at System.Net.Sockets.Socket.ThrowMultiConnectNotSupported()
consoleapp1-1  | ------
consoleapp1-1  | Unhandled exception. StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code.
consoleapp1-1  |    at StackExchange.Redis.ConnectionMultiplexer.ConnectImpl(ConfigurationOptions configuration, TextWriter writer, Nullable`1 serverType, EndPointCollection endpoints) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 691
consoleapp1-1  |    at StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 663
consoleapp1-1  |    at StackExchange.Redis.ConnectionMultiplexer.Connect(String configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 641
consoleapp1-1  |    at Program.<Main>$(String[] args) in /src/ConsoleApp1/Program.cs:line 73
consoleapp1-1 exited with code 139

[mgravell]

This is super, thanks. Very much appreciated. I think you can see how if I was investigating from a cold start, even finding a failing setup would have been tricky, so: thanks.

What I propose, then:

1. Make TCP keep-alive an explicit option in config options
2. I'm tempted to default it to on for .NET 10+, else off
3. If not explicitly enabled (i.e. if on by default only), I'm tempted to automatically disable it if the first connect attempt fails

Thoughts on any of these very welcome!

[liukonen]

I will be migrating my app to dotnet 10 to resolve this. That said I want to thank both of you for the quick responses on this! I know the lts support of dotnet 8 expires soon, so this is kind of a sign I need to migrate up. Can't personally speak for anyone else who might be experiencing the issue, but I'd personally probably put a line in the sand on 2.13.1 as the last supported version for dotnet 8. But that's only my 2 cents.