special DNS record for host.docker.internal + gateway.docker.internal #2348

0xbad0c0d3 · 2019-03-08T09:48:13Z

I was not able to find "magic" for mac and windows systems about 'host.docker.internal' and all solutions in the Net just workarounds. Here I am with my solution... Maybe it's not the best, but it works just fine.
Yes, this functionality can be implemented with dnsmasq, but not everyone can do it by themselves. That's why i find this PR useful

GordonTheTurtle · 2019-03-08T09:48:16Z

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "master" git@github.com:0xbad0c0d3/libnetwork.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

pbeza · 2019-03-22T14:01:27Z

When can we expect it to be merged into master branch?

nyetwurk · 2019-03-29T16:23:28Z

The way docker does DNS by default is fundamentally broken. It should always use dnsmasq and the host's DNS configuration by default (and not use 8.8.8.8). Each container should (by default) resolve DNS by querying the host, which should forward the request to its own resolver, and provide resolution for all of .docker.internal, including host.docker.internal (which should be configurable to allow containers named host). All other accessible containers should also resolve in .docker.internal

jtaylor100 · 2019-04-11T08:38:53Z

Any idea when this will be merged? 🙂

yinxing2008 · 2019-04-18T07:50:16Z

When will this nice feature be merged?

Akkadius · 2019-04-19T04:31:50Z

I have been watching this one, we have Linux members on the team that have this record broken which seems bizarre that Windows and Mac would have this record and not Linux

shahzadmasud · 2019-04-19T04:33:42Z

Dev team is using this one, and when pushing things on production - we are in a great challenging situation. Any anticipated fix date?

Bardioc1977 · 2019-04-19T08:34:47Z

Dev team is using this one, and when pushing things on production - we are in a great challenging situation. Any anticipated fix date?

Hello,

if you really need a feature like this before it has been merged, you can do something like this. In our Dockerfile, we override the entrypoint.sh with something like this:

#!/usr/bin/env bash
set -x
set -e

function fix_linux_internal_host() {
DOCKER_INTERNAL_HOST="host.docker.internal"
if ! grep $DOCKER_INTERNAL_HOST /etc/hosts > /dev/null ; then
DOCKER_INTERNAL_IP='/sbin/ip route | awk '/default/ { print $3 }' | awk '!seen[$0]++'
echo -e "$DOCKER_INTERNAL_IP\t$DOCKER_INTERNAL_HOST" | tee -a /etc/hosts > /dev/null
echo "Added $DOCKER_INTERNAL_HOST to hosts /etc/hosts"
fi
}

fix_linux_internal_host

/usr/local/bin/gosu jboss:jboss bash -c "/opt/jboss/wildfly/bin/standalone.sh $@"

Be aware, that you need to do this as root user, so the change to a specific user needs to be done within entrypoint.sh (in our Wildfly version via gosu).
Afterwards, you have a "host.docker.internal" available within your container.

nyetwurk · 2019-04-19T15:35:16Z

Again, that workaround is known, but not generally practical for complex deployments

Bardioc1977 · 2019-04-19T18:28:38Z

Again, that workaround is known, but not generally practical for complex deployments

I know that, thats why its called "workaround". But until the final solution is merged and available, it IS a possibility.

schildbach

General advice: if you restrict your PRs to the absolute bare minimum of changes necessary, you increase the chance of being merged. So never reformat any (otherwise untouched) code.

sandbox.go

schildbach · 2019-04-27T12:12:43Z

sandbox.go

@@ -133,7 +133,8 @@ type containerConfig struct {
 }

 const (
-	resolverIPSandbox = "127.0.0.11"
+	resolverIPSandbox  = "127.0.0.11"


The additional space is an unrelated change.

This space was produced by gofmt, are you sure it should be as you said?

It doesn't matter who or what produced it. It's a change that doesn't belong to the title of this PR. As such, it will distract any reviewers of this PR.

If i'll remove it -

🐳 vet 🐳 ineffassign 🐳 fmt sandbox.go 👹 please format Go code with 'gofmt -s -w' Makefile:155: recipe for target 'fmt' failed make: *** [fmt] Error 1 Makefile:114: recipe for target 'check' failed make: *** [check] Error 2 Exited with code 2

Another advice, though I don't know whether this applies to docker specifically, but you usually update the pull request, rather than adding more commits that fix the problem in the original pull request.
The reason is to keep the repository history clean and concise.
What I mean, it's better to have a commit that implements the desired functionality (or a logical part of it) without any unrelated changes such as code reformatting, and then another commit that does the reformatting. Rather than having a commit that implements the functionality including unrelated changes, than a commit that reverts the unrelated changes, and finally a commit that adds them back like here.

But to end on a more positive note - I really hope this will get merged soon (whatever version will be accepted by the devs). It's an incredibly useful functionality.

Totally agree with you about the history and will keep in mind for future
;)

0xbad0c0d3 · 2019-04-27T21:13:40Z

@schildbach, done

Akkadius · 2019-04-27T21:21:43Z

Let’s merge that bad boy! 😛

simpslandyy · 2019-05-01T19:44:46Z

Any day now for this merge! :D

thaJeztah · 2019-05-08T00:11:09Z

/cc @arkodg

mavenugo · 2019-05-09T04:25:24Z

@0xbad0c0d3 Sorry about the delayed response. couple of questions/comments :

I think this change will not have any effect for the default bridge (docker0) network since it doesn't use the built-in SD resolver. If we need this behavior for all the networks, then we will unfortunately have to tamper with the /etc/hosts of the container in the docker0 bridge networks.
What are the expectations of this fix in d4mac and d4win ? This will ofcourse return the gateway ip of the bridge which is local to the VM hosting the docker daemon. But wont directly point to the mac or win host. Will it satisfy the requirement ?

sandbox.go

arkodg · 2019-05-09T19:27:21Z

@0xbad0c0d3 have not tried this out but AFAIK this will break Docker Desktop (Mac and Windows) . Today this works on Desktop through a magic proxy called vpnkit running on OSX host ( ps -e | grep com.docker.vpnkit ) for more info which allows you a way to reach the OSX host via host.docker.internal (the DNS proxy in vpnkit sends this A record to the VM)
By adding a SRV Record in the DNS Server in the dockerd daemon that maps host.docker.internal to the Gateway-IP, containers running in a native Linux environment will be able to reach the Linux Host , but on MAC, containers will be sending traffic to the Linux VM and not the OSX host

If users were able to insert A records into dockerd's DNS server through cmdline/config options and so custom mappings such as host.docker.internal could be mapped to the Host IP / or interface, this functionality could be achieved (Note: this would not work for default docker0 bridge, since for this case, the host's resolv.conf is mounted so the nameserver does not point to dockerd). I will bring up this feature with the team .

0xbad0c0d3 · 2019-05-09T20:10:49Z

@0xbad0c0d3 have not tried this out but AFAIK this will break Docker Desktop (Mac and Windows) . Today this works on Desktop through a magic proxy called vpnkit running on OSX host ( ps -e | grep com.docker.vpnkit ) for more info which allows you a way to reach the OSX host via host.docker.internal (the DNS proxy in vpnkit sends this A record to the VM)
By adding a SRV Record in the DNS Server in the dockerd daemon that maps host.docker.internal to the Gateway-IP, containers running in a native Linux environment will be able to reach the Linux Host , but on MAC, containers will be sending traffic to the Linux VM and not the OSX host

If users were able to insert A records into dockerd's DNS server through cmdline/config options and so custom mappings such as host.docker.internal could be mapped to the Host IP / or interface, this functionality could be achieved (Note: this would not work for default docker0 bridge, since for this case, the host's resolv.conf is mounted so the nameserver does not point to dockerd). I will bring up this feature with the team .

I was thinking about the possibility to manage this feature with cmdline option, but, I am new in go, and do not have time to implement it, more because i could not test it on WIN/MAC systems... But, i think it's easy to detect OS family and apply my "fix" for linux only. Also, i realized that it would not work with docker0 because of the resolv.conf.

arkodg · 2019-05-10T04:06:22Z

@0xbad0c0d3 dockerd for both the scenarios (Mac and Linux) is running on Linux 😄 , for the Mac case it happens to be a VM .
https://docs.docker.com/engine/reference/commandline/run/#add-entries-to-container-hosts-file---add-host documents how you can use the --add-host argument to add entries to your container's /etc/hosts so something like docker run --add-host=host.docker.internal:172.17.0.1 --rm -it alpine ping host.docker.internal should help you reach the Linux host, but please do not add this argument while running containers on Mac or Windows (https://docs.docker.com/docker-for-mac/networking/)

0xbad0c0d3 · 2019-05-10T08:05:39Z

@0xbad0c0d3 dockerd for both the scenarios (Mac and Linux) is running on Linux , for the Mac case it happens to be a VM .
https://docs.docker.com/engine/reference/commandline/run/#add-entries-to-container-hosts-file---add-host documents how you can use the --add-host argument to add entries to your container's /etc/hosts so something like docker run --add-host=host.docker.internal:172.17.0.1 --rm -it alpine ping host.docker.internal should help you reach the Linux host, but please do not add this argument while running containers on Mac or Windows (https://docs.docker.com/docker-for-mac/networking/)

--add-host is requiring extra work, you have to configure interface first (add alias IP), then modify all your containers to be run with --add-host, or install dnsmasq, configure it...rather than just enjoying the feature "out of the box".

Signed-off-by: Daniil Skomarovskiy <0xbad0c0d3@gmail.com>

… - gofmt Signed-off-by: Daniil Skomarovskiy <0xbad0c0d3@gmail.com>

Signed-off-by: Daniil Skomarovskiy <0xbad0c0d3@gmail.com>

0xbad0c0d3 · 2019-09-26T12:43:30Z

@0xbad0c0d3 want to reiterate that this code will break Docker Desktop on Mac and Windows, what is needed is

a dockerd configuration flag called --dns-resolve-docker-host or something similar (such as --iptables) which needs to be added to https://github.com/moby/moby and enabled by default

Based on that flag we can programtically add a DNS entry (either in moby/moby or libnetowrk) that maps host.docker.internal to docker0's IP

Docker Desktop internally can then override this flag when creating a VM and spawning dockerd so that all these DNS requests go to its special proxy outside the VM

cc: @thaJeztah @justincormack @djs55

@arkodg I've implemented trough dns-opt, to switch the feature on --dns-opt resolve-docker-internal should be passed to dockerd command

arkodg · 2019-09-26T17:25:27Z

@0xbad0c0d3 according to https://docs.docker.com/v17.09/engine/userguide/networking/default_network/configure-dns/
--dns-opt is specific to resolve.conf, having a new flag in moby is the way to go IMHO

jwundrak · 2019-09-26T17:28:25Z

I can't understand, why this request is not merged. For us in an agency, we need a workaround for linux. Otherwise for WIndows or OSX, but this is the regular thing.

There are round abound more than 400 "likes" for this feature and there is nothing technical, why this can not merged.

So please merge this request.

0xbad0c0d3 · 2019-09-27T17:02:44Z

@0xbad0c0d3 according to https://docs.docker.com/v17.09/engine/userguide/networking/default_network/configure-dns/
--dns-opt is specific to resolve.conf, having a new flag in moby is the way to go IMHO

@arkodg Generally - you are right, but in other way it requires another PR directly to docker engine, or I am wrong?

arkodg · 2019-09-27T18:14:21Z

@0xbad0c0d3 I think this issue can be solved completely in moby with --dns-resolve-docker-host and ExtraHosts. I can look into this next week

arkodg · 2019-09-30T15:15:41Z

@0xbad0c0d3 thanks for highlighting this issue, I've raised a similar PR in moby moby/moby#40007 that solves this issue using the the below approach

a dockerd configuration flag called --dns-resolve-docker-host or something similar (such as --iptables) which needs to be added to https://github.com/moby/moby and enabled by default

Based on that flag we can programtically add a DNS entry (either in moby/moby or libnetowrk) that maps host.docker.internal to docker0's IP

Docker Desktop internally can then override this flag when creating a VM and spawning dockerd so that all these DNS requests go to its special proxy outside the VM

Vistrus · 2019-11-06T07:50:06Z

Any idea when this will be merged?

j0eii · 2019-11-17T11:08:34Z

need this, please add it

nazeefahmadmeer · 2019-12-01T00:45:22Z

MERGE MERGE MERGE!

shashank-sharma · 2019-12-17T08:02:46Z

Looks like it will take another 3-4 months to actually solve all the dependency problems

Akkadius · 2019-12-17T08:48:44Z

FYI for people who have been watching this like a hawk, dialogue has been going on here between members of the Moby team: moby/moby#40007

yilativs · 2020-01-14T11:22:34Z

is this scheduled?

arkodg · 2020-01-14T18:21:42Z

@0xbad0c0d3 can you please close this PR in favor of moby/moby#40007

omoustaouda · 2022-07-07T13:25:07Z

In brief:

for who came here looking for how to be able to use host.docker.internal on Linux:

this is needed in your docker-compose:

in the service definition:

    name: mycontainer

   ...

    working_dir: /webhome/lumen/api
    ports:
      - "3002:3002"
      
    ...


    extra_hosts:     <---- this new part is needed
      - "host.docker.internal:host-gateway"

Then, from your container your should be able to ping the host.

$ ping host.docker.internal
PING host.docker.internal (172.17.0.1) 56(84) bytes of data.
64 bytes from host.docker.internal (172.17.0.1): icmp_seq=1 ttl=64 time=0.080 ms
64 bytes from host.docker.internal (172.17.0.1): icmp_seq=2 ttl=64 time=0.096 ms
...

This should be working from Docker v. 20.10.

Thanks to everyone who worked on fixing it and who researched for solutions! 🙏☀️❤️

GordonTheTurtle added the dco/no label Mar 8, 2019

0xbad0c0d3 force-pushed the master branch from a9e5507 to 7f85a75 Compare March 8, 2019 09:51

GordonTheTurtle removed the dco/no label Mar 8, 2019

0xbad0c0d3 force-pushed the master branch from ac7573b to 23f1ffd Compare March 8, 2019 11:00

0xbad0c0d3 mentioned this pull request Mar 8, 2019

Support host.docker.internal DNS name to host docker/for-linux#264

Closed

3 tasks

0xbad0c0d3 force-pushed the master branch from 8af6b0f to 4fe4569 Compare March 12, 2019 15:10

flatmapthatshit approved these changes Mar 27, 2019

View reviewed changes

charleskorn mentioned this pull request Mar 27, 2019

Add support for local proxies batect/batect#10

Open

schildbach reviewed Apr 27, 2019

View reviewed changes

mavenugo reviewed May 9, 2019

View reviewed changes

sandbox.go Outdated Show resolved Hide resolved

0xbad0c0d3 changed the title ~~special DNS record for host.docker.internal~~ special DNS record for host.docker.internal + gateway.docker.internal May 10, 2019

0xbad0c0d3 added 6 commits September 26, 2019 12:16

special DNS record for host.docker.internal

aec58bd

Signed-off-by: Daniil Skomarovskiy <0xbad0c0d3@gmail.com>

special DNS record for host.docker.internal - unrelated changes

5dae7df

Signed-off-by: Daniil Skomarovskiy <0xbad0c0d3@gmail.com>

special DNS record for host.docker.internal - gofmt

be8abff

Signed-off-by: Daniil Skomarovskiy <0xbad0c0d3@gmail.com>

special DNS record for host.docker.internal + gateway.docker.internal

7fce6d6

Signed-off-by: Daniil Skomarovskiy <0xbad0c0d3@gmail.com>

special DNS record for host.docker.internal + gateway.docker.internal…

af67ba4

… - gofmt Signed-off-by: Daniil Skomarovskiy <0xbad0c0d3@gmail.com>

--dns-option "resolve-docker-internal" to enable the feature

186d22a

Signed-off-by: Daniil Skomarovskiy <0xbad0c0d3@gmail.com>

0xbad0c0d3 force-pushed the master branch from b728621 to 186d22a Compare September 26, 2019 09:19

0xbad0c0d3 closed this Jan 14, 2020

arkodg mentioned this pull request Jan 22, 2020

Support host.docker.internal in dockerd on Linux moby/moby#40007

Merged

ttpro1995 mentioned this pull request Jan 2, 2021

ECONNREFUSED when prepare on localhost postgresql database pantsel/konga#655

Closed

jkneubuh mentioned this pull request Mar 28, 2022

Provide clear guidance for debugging Java chaincode as a service hyperledger/fabric-samples#684

Closed

Zedo9 mentioned this pull request Feb 8, 2023

Unable to establish websocket connection when using a reverse proxy (selfhosted setup) supabase/realtime#472

Closed

jb2197 mentioned this pull request Mar 6, 2023

Dev update flood assessment agent cambridge-cares/TheWorldAvatar#637

Merged

plutack1 mentioned this pull request Sep 9, 2023

BUG : Can't access localhost:8080 on docker, <ip-address>:8080 produce cors error on fetching hiteshchoudhary/apihub#45

Closed

shankari mentioned this pull request Mar 17, 2024

MRE with working OCPP 2.0.1 security profile 1, 2 and 3 (all) EVerest/everest-demo#31

Merged

aindriu-aiven mentioned this pull request Jun 21, 2024

Fail to Approve Topic Request Aiven-Open/klaw#2490

Closed

special DNS record for host.docker.internal + gateway.docker.internal #2348

special DNS record for host.docker.internal + gateway.docker.internal #2348

Conversation

0xbad0c0d3 commented Mar 8, 2019 • edited Loading

GordonTheTurtle commented Mar 8, 2019

pbeza commented Mar 22, 2019

nyetwurk commented Mar 29, 2019 • edited Loading

jtaylor100 commented Apr 11, 2019

yinxing2008 commented Apr 18, 2019

Akkadius commented Apr 19, 2019

shahzadmasud commented Apr 19, 2019 • edited Loading

Bardioc1977 commented Apr 19, 2019

nyetwurk commented Apr 19, 2019

Bardioc1977 commented Apr 19, 2019

schildbach left a comment

Choose a reason for hiding this comment

schildbach Apr 27, 2019

Choose a reason for hiding this comment

0xbad0c0d3 May 2, 2019

Choose a reason for hiding this comment

schildbach May 2, 2019

Choose a reason for hiding this comment

0xbad0c0d3 May 2, 2019

Choose a reason for hiding this comment

stativ May 2, 2019

Choose a reason for hiding this comment

0xbad0c0d3 May 2, 2019

Choose a reason for hiding this comment

0xbad0c0d3 commented Apr 27, 2019

Akkadius commented Apr 27, 2019

simpslandyy commented May 1, 2019

thaJeztah commented May 8, 2019

mavenugo commented May 9, 2019

arkodg commented May 9, 2019

0xbad0c0d3 commented May 9, 2019 • edited Loading

arkodg commented May 10, 2019 • edited Loading

0xbad0c0d3 commented May 10, 2019

0xbad0c0d3 commented Sep 26, 2019

arkodg commented Sep 26, 2019

jwundrak commented Sep 26, 2019

0xbad0c0d3 commented Sep 27, 2019

arkodg commented Sep 27, 2019

arkodg commented Sep 30, 2019

Vistrus commented Nov 6, 2019

j0eii commented Nov 17, 2019

nazeefahmadmeer commented Dec 1, 2019

shashank-sharma commented Dec 17, 2019

Akkadius commented Dec 17, 2019

yilativs commented Jan 14, 2020

arkodg commented Jan 14, 2020

omoustaouda commented Jul 7, 2022 • edited Loading

0xbad0c0d3 commented Mar 8, 2019 •

edited

Loading

nyetwurk commented Mar 29, 2019 •

edited

Loading

shahzadmasud commented Apr 19, 2019 •

edited

Loading

0xbad0c0d3 commented May 9, 2019 •

edited

Loading

arkodg commented May 10, 2019 •

edited

Loading

omoustaouda commented Jul 7, 2022 •

edited

Loading