Running Under Docker Requires resetting proxy with http_proxy=""

[phantomdata]

Its late, and I wanted to make sure to get this logged in case I forgot. I haven't seen this with anything else I've Dockerized, but recently Dockerizing a bot of mine I ran into a unique issue:

ERROR -- : Failed to open TCP connection to : (getaddrinfo: Name or service not known) (Faraday::ConnectionFailed)

An hour or so of sleuthing through slack-ruby-bot, slack-ruby-client, faraday and finally ruby (2.3)'s own net/http library led me to an interesting realization; it was using an HTTP proxy. The problem was, the HTTP proxy specified by Docker was unrecognizable by Ruby's net libraries and consequently failed to determine its hostname... which unexpectedly didn't error out, but just returned nil for the entire proxy address. 🎉

Ultimately, I temporarily got around this by including http_proxy="" in my command to start my bot. This was Docker CE 17.06.1 on a Mac. I'm uncertain as to if it affects any other platforms.

I'll try to track down where this would properly go; but figured that if nothing else - dropping a line in this project's README would help someone out if they stumbled upon this later.

I admit, my Docker knowledge is not very advanced and I may be mis-interpreting something - but again; wanted this searchable somewhere in case anyone else ran into it. Feel free to close, I won't feel bad.

[dblock]

What's the proxy value specified?

[phantomdata]

Oh hai! docker.for.mac.localhost:63074 is the value that Docker sets.

$ export | grep proxy
=> declare -x http_proxy="docker.for.mac.localhost:63074"
=> declare -x no_proxy="localhost, 127/8, localhost, 127/8"

I suspect that the port changes depending on environmental factors. If it wasn't clear in my Issue; I am literally passing an empty value in to override the declare'd value.

$ http_proxy="" SLACK_API_TOKEN=super-secret bundle exec ruby ./cookiebot.rb

I'll get a chance to test if this happens on Docker in Linux either tonight or tomorrow night.

NOTE: I don't think its a bug in slack-ruby-bot; its just a very unhelpful message from Faraday that digs deep into a unique happenstance between Docker's usage of an HTTP proxy and net/http. More something worth documenting in the README or maybe checking during initialization for sanity's sake (is an environmental http_proxy host set? does it resolve to a valid hostname?).

[dblock]

What's the configuration in the bot? Are you setting proxy or websocket_proxy in code?

[phantomdata]

Nope. That was the weirdest part and what ultimately threw me for a gigantic loop. net/http was automatically picking up the environment variable that Docker had set for http_proxy and going "Oh, let me use that proxy.". In debugging, I reduced the bot down to:

#!/usr/bin/env ruby

require "slack-ruby-bot"

class CookieBot < SlackRubyBot::Bot; end

CookieBot.run

[dblock]

It's actually documented, https://docs.ruby-lang.org/en/2.0.0/Net/HTTP.html#class-Net::HTTP-label-Proxies.

I wouldn't assume that the problem is parsing and debug it. If it's a full proxy it should work, so I would try figuring out whether it's an SSL proxy, whether it supports websockets, etc., then potentially prefixing it with http:// or https:// to see if that helps. I am shooting in the dark though :)

[phantomdata]

Hehe, I was actually caught off guard because the proxy environment variable was set by Docker without my knowledge or intervention - hence having to track down what ERROR -- : Failed to open TCP connection to : (getaddrinfo: Name or service not known) (Faraday::ConnectionFailed) actually meant.

Ultimately, in tracing it down I found out that the specific value that Docker was setting it to resulted in net/http not being able to determine a hostname from the value and instead of throwing an error, it passes nil back up the call-stack which causes the "Name or service not known" (obviously, a hostname of nil won't resolve).

I guess, the crux of it is that I had no idea an http_proxy was being set in the brand new default environment and thus had no reason to suspect it was the cause of the issues. The error message that bubbles up the chain ends up being rather uninformative as to what's going wrong (an invalid http_proxy value was set in the environment; but the error message from Faraday simply says "name or service not known").

As soon as I unset that environment variable, everything was happy days. Its solved for me, but I was generally hoping this Issue might save someone else from having to figure out "Why doesn't this work on Docker for Mac?". :)

I guess; 3 possible outcomes:

• Update the README to include a note about this behavior (maybe if someone else could validate it happens for them as well?)
• Add a guard clause to suggest possible reasons for Faraday::ConnectionFailed exceptions (I'd be happy to open a PR, but it might not be for a bit)
• Close this issue and let Google serve as the next person's guide

(btw, thanks for being so responsive. I love your library and have gotten a great deal of use out of it over the last 6 months or so)

[dblock]

I think we should update the README around proxy configuration saying that the default is not nil, but the value of ENV['http_proxy'] and link to the documentation for net/http. Care to do that?

[phantomdata]

Sounds like a plan! If you don't mind, can you assign this issue to me?

[jasonwc]

Hey @phantomdata, I'm running into this same problem. I'm trying to get a bot dockerized and deployed to kubernetes. The image starts fine but I'm seeing the same 00:43:01 web.1 | E, [2017-12-28T00:43:01.545282 #16] ERROR -- : Failed to open TCP connection to slack.com:443 (getaddrinfo: Name or service not known) (Faraday::ConnectionFailed)

Did you ever get this resolved? I'm pretty new to docker/k8s and have a pretty basic docker file at the moment.

FROM ruby:2.3-onbuild
ENV LC_ALL C.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US.UTF-8
CMD ["bash", "-l", "-c", "/usr/src/app/script/start.sh"]

Where did you end up nilling out the proxy?

EDIT: My start script does something like this http_proxy="" bundle exec foreman start

[jasonwc]

I might be running into an issue with my cluster configuration actually... Might be unrelated.

[phantomdata]

👋 @jasonwc That's exactly what I did to get mine up and running (http_proxy="" bundle exec ...). Did your cluster configuration end up having something to do with it? If that container is able to run standard Unix network utilities (eg: ping) and can get out successfully; I'd start tracking down the value of that http_proxy variable in your program. Maybe Foreman or another lib (dot-env perhaps?) is getting in the way somehow.

Let us know what you find out. I'll probably update that README over the next few days, so if there's another reason for that Faraday::ConnectionFailed to confusingly popup; I'd love to see it added.

Thanks for reminding me about this issue, heh. It had totally slipped my mind to update the README.

[phantomdata]

@dblock I've got a PR awaiting review for this over at #174. Happy New Year! :)

[dblock]

Thanks for the docs @phantomdata, updated in c9ed355.

[jasonwc]

Thanks @phantomdata and @dblock!

It turned out that I was running into an issue with my Kubernetes cluster. I basically did what you had suggested @phantomdata and tried to ping Google and Slack and wasn't able to get out. I then realized that my kube-dns pods were in a CrashLoopBackOff cycle and that was causing my networking issues. I'll make sure to verify that the http_proxy issue isn't present in Linux once I get my cluster sorted, but as I was debugging I wasn't seeing anything in the env that suggested that it was setting it.

Regardless, this issue was helpful in giving me a few things to try so thanks for the README update! This library has been super helpful for our internal slackbot and I'm hoping to find a chance to contribute back soon. I've been tooling around on a blog post about combining this library, docker, and kubernetes to make a sweet sweet bot cluster for interfacing with a lot of different services.

Happy New Year to both of you and thanks again for your help!

[phantomdata]

Here here to the usefulness. Thanks for all the hard work @dblock and everyone else contributing! I'm glad you got the reason for your issues tracked down @jasonwc. Feel free to re-open this issue if you run into it once your cluster is working and my suggested fix doesn't resolve it for you.

The README has been updated with helpful instructions over in c9ed355, so I'm going to go ahead and close this out.

🎉

[ericwoodruff]

I came across this issue when investigating a similar error in a library that uses net/http so I wanted to leave a breadcrumb for others.

This error:
ERROR -- : Failed to open TCP connection to : (getaddrinfo: Name or service not known)

Is caused by http_proxy missing the http:// prefix. http_proxy="docker.for.mac.localhost:63074" in this case is not a valid proxy uri.