Unable to initiate a connection to a Redis server from inside a Docker container

[zubialevich]

We are having problems doing a connection to a Redis server when we initiate the connection from inside a Linux Docker container. See code and exception below. Same code runs ok when run from a Linux server without Docker.

            ConnectionMultiplexer redis = ConnectionMultiplexer.Connect("redis-server:6379");

            try
            {
                IDatabase db = redis.GetDatabase(15);
                string key = "SOME_KEY";
                string value = db.StringGet(key);
                Console.WriteLine($"key: {key}, value: {value}");
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                throw;
            }

Unhandled Exception:
StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). SocketFailure (ReadSocketError/NotConnected, 0-read, 0-sent, last-recv: 0) on redis-server:6379/Interactive, Flushed, last: ECHO, origin: ReadFromPipe, outstanding: 7, last-read: 0s ago, last-write: 0s ago, unanswered-write: 0s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 7 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.0.513.63329
  at StackExchange.Redis.ConnectionMultiplexer.ConnectImpl (System.Object configuration, System.IO.TextWriter log) [0x00078] in <1656e54fa0554924a4a50ae9f4538aaf>:0
  at StackExchange.Redis.ConnectionMultiplexer.Connect (System.String configuration, System.IO.TextWriter log) [0x00005] in <1656e54fa0554924a4a50ae9f4538aaf>:0
  at WindowsService1.Service1.OnStart (System.String[] args) [0x00011] in <afaae550b2e74acc84faca60b060fe35>:0
  at WindowsService1.Service1.DoStart (System.String[] args) [0x00001] in <afaae550b2e74acc84faca60b060fe35>:0
  at (wrapper remoting-invoke-with-check) WindowsService1.Service1.DoStart(string[])
  at WindowsService1.Program.Main (System.String[] args) [0x00047] in <afaae550b2e74acc84faca60b060fe35>:0
[ERROR] FATAL UNHANDLED EXCEPTION: StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). SocketFailure (ReadSocketError/NotConnected, 0-read, 0-sent, last-recv: 0) on redis-server:6379/Interactive, Flushed, last: ECHO, origin: ReadFromPipe, outstanding: 7, last-read: 0s ago, last-write: 0s ago, unanswered-write: 0s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 7 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.0.513.63329
  at StackExchange.Redis.ConnectionMultiplexer.ConnectImpl (System.Object configuration, System.IO.TextWriter log) [0x00078] in <1656e54fa0554924a4a50ae9f4538aaf>:0
  at StackExchange.Redis.ConnectionMultiplexer.Connect (System.String configuration, System.IO.TextWriter log) [0x00005] in <1656e54fa0554924a4a50ae9f4538aaf>:0
  at WindowsService1.Service1.OnStart (System.String[] args) [0x00011] in <afaae550b2e74acc84faca60b060fe35>:0
  at WindowsService1.Service1.DoStart (System.String[] args) [0x00001] in <afaae550b2e74acc84faca60b060fe35>:0
  at (wrapper remoting-invoke-with-check) WindowsService1.Service1.DoStart(string[])
  at WindowsService1.Program.Main (System.String[] args) [0x00047] in <afaae550b2e74acc84faca60b060fe35>:0

We are using .NET 4.7.1, Ubuntu 18.04, Mono JIT compiler version 5.16.0.179 and Docker 18.09

Any ideas or workarounds?
Thanks.

[mgravell]

• on the machine (or container) that isn't working - does redis-cli -h redis-server work?
• does it work if you use IP address instead of host name?
• there is an optional TextWriter parameter to Connect - if you pass something in - does it say anything useful?

As an example of the last:

            var log = new StringWriter();
            ConnectionMultiplexer redis = ConnectionMultiplexer.Connect("redis-server:6379", log);
            try
            {
                IDatabase db = redis.GetDatabase(15);
                string key = "SOME_KEY";
                string value = db.StringGet(key);
                Console.WriteLine($"key: {key}, value: {value}");
            }
            catch (Exception ex)
            {
                Console.WriteLine(log.ToString());
                Console.WriteLine(ex.Message);
                throw;
            }

[zubialevich]

Thank you for the quick response!

on the machine (or container) that isn't working - does redis-cli -h redis-server work?

We've tested redis-cli and it works as expected from the inside the container.

does it work if you use IP address instead of host name?

Using directly the IP address of the Redis server works! Btw nslookup redis-server from the inside of the container works and resolves the hostname to the IP correctly.

there is an optional TextWriter parameter to Connect - if you pass something in - does it say anything useful?

Unfortunately, the added logs didn't provide any extra information in this case.

Looks like the container is able to get to the Redis server, and is also able to resolve the domain name.

[mgravell]

OK. There's an extra configuration option you can try - either in a config string or via the object model; "resolveDns". If you set that to *true*, it will use an explicit DNS resolution step before trying to connect. For example: "YourServer:Port,resolveDns=true" Can you give that a go?

…

On Thu, 15 Nov 2018, 07:12 Dzmitry Zubialevich ***@***.*** wrote: Thank you for the quick response! on the machine (or container) that isn't working - does redis-cli -h redis-server work? We've tested redis-cli and it works as expected from the inside the container. does it work if you use IP address instead of host name? Using directly the IP address of the Redis server *works*! Btw nslookup redis-server from the inside of the container works and resolves the hostname to the IP correctly. there is an optional TextWriter parameter to Connect - if you pass something in - does it say anything useful? Unfortunately, the added logs didn't provide any extra information in this case. Looks like the container is able to get to the Redis server, and is also able to resolve the domain name. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1002 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABDsO2W2zimL8hDwPoOLmU4t_haxWz5ks5uvRPNgaJpZM4Yduok> .

[zubialevich]

Thank you for the workaround!
Adding multiplexerConfig.ResolveDns = true; helped to solve the issue!

Do you know what is the underlying cause of this problem? Because DNS resolving seems to work, redis-cli also works, what does this property ResolveDns do?

[mgravell]

What we're controlling here is whether or not we trust the Socket implementation to do the right thing. Basically, the Socket API accepts an EndPoint, which can be an IPEndPoint, a DnsEndPoint, or a few others. Normally, we trust it to get it right, so we just describe the target in the simplest way - DnsEndPoint (meaning; a name and port). However, sometimes - depending on OS, framework, and a few other platform details, this doesn't work. In that case, we have opt-in code that invokes the DNS APIs *in our library code*, to fetch an IP address, which we then pass down as an IPEndPoint. It sounds tempting to say "just do that all the time if it works better", but it is a trade-off thing too - if we made that the default we get into problems with DNS cache invalidation, etc.

…

On Thu, 15 Nov 2018, 12:58 Dzmitry Zubialevich ***@***.*** wrote: Thank you for the workaround! Adding multiplexerConfig.ResolveDns = true; helped to solve the issue! Do you know what is the underlying cause of this problem? Because DNS resolving seems to work, redis-cli also works, what does this property ResolveDns do? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1002 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABDsPLIPHPo_Dm38_IXSLwaMVSuVgLKks5uvWTbgaJpZM4Yduok> .

[maggedotno]

Colleague here. Thank you so much for your help!

This is very interesting. There are a few other clients also running in mono, for example to a mysql and elasticsearch. It seems such a fundamental issue should affect many things, I mean the Socket API must be part of the mono implementation. I don't understand what makes StackExchange.Redis special in this case.

Do you have any theory on what may cause what looks like a bad selection of endpoint type in linux+mono+docker and at the same time not in linux+mono ?

[f2calv]

Thanks also :)
Adding ResolveDns fixed the internal docker issue however it then broke when testing outside of Docker/Docker-Compose, so used the below;

_configurationOptions.ResolveDns = bool.TryParse(Environment.GetEnvironmentVariable("DOTNET_RUNNING_IN_CONTAINER"), out var _);

[NickCraver]

Looks like we're good here - closing out to cleanup.