Private reachability for Public node

[b00f]

Reachability is not consistent information, and for some unknown reason, some public nodes with public IPs are shown as private [Example here].
Any idea why this happens? Probably, this may have some effects on other modules, which I believe should be taken care of.

[sukunrt]

The node may be behind a firewall. Is it possible to enable prometheus metrics?
https://github.com/libp2p/go-libp2p/tree/master/dashboards

[b00f]

The node may be behind a firewall.

Yes, we do have a firewall for security reasons, but we have enabled all the ports we are aware of.
Is there any specific port for this purpose that is not configurable by users?

Is it possible to enable Prometheus metrics?

Sure, I just enabled it, and I believe this is what you're looking for: Prometheus Metrics.

[sukunrt]

You need to open the port that the node is listening on. From the metrics, I see a lot of your requested dailbacks errored. I also see that in the last few hours, the node has not made any dial back requests, which is very strange because I see some peers in swarm. Maybe none of them support autonat protocol?
For further debugging you can enable autonat logs and see if there's anything suspicious there. That would be env GOLOG_LOG_LEVEL="autonat=debug"

If you're confident that the node is private, you can use https://github.com/libp2p/go-libp2p/blob/master/options.go#L349 ForceReachabilityPublic option.

[b00f]

Thanks for time to check and identify the issue.

Please consider these is not a test network and we can't easily make any changes for debugged purpose.
We update the nodes on official release. We try to add some logs to get more information in the next release.

[b00f]

You need to open the port that the node is listening on.

Which port?

[b00f]

Maybe none of them support autonat protocol?

Probably we have at least 6 (The nodes we have configured)

[2color]

You need to open the port that the node is listening on.

Which port?

The port is determined based on how you configure the libp2p host.

For example, this is how you listen on port 9000

go-libp2p/examples/libp2p-host/host.go

Lines 60 to 63 in 4e85c96

	libp2p.ListenAddrStrings(
	"/ip4/0.0.0.0/tcp/9000", // regular tcp connections
	"/ip4/0.0.0.0/udp/9000/quic-v1", // a UDP endpoint for the QUIC transport
	),

[melroy89]

Ps. for quic web transport and WebRTC you can't reuse the same port (in this example port 9000). So if you also want to listen for web transport and Web RTC requests, you will also need to define different ports.

Ps. Ps. like 2color mentioned, you will need to open both TCP & UDP protocols, since Quic is using UDP.

Issue can be closed right?

[b00f]

I'm not exactly sure of the reason, but I can confirm that this issue happened in the latest LibP2P versions. We never faced this problem before. (v0.36.5 or before)

Public nodes are still showing as private. Take a look here: https://bootstrap1.pactus.org/node

[melroy89]

OK, I'm not familiar with Pactus to be fully honest, however I did a simple telnet test on the IP + port, I got connection closed:

telnet 45.118.133.17 21888
Trying 45.118.133.17...
Connected to 45.118.133.17.
Escape character is '^]'.
Connection closed by foreign host.

I could imagine this kind of behaviour can lead to IPFS AutoNat to misjudge the network setup and keep set the reachability to private.

And like sukunrt said before, you can try to set an option to libp2p to force itself to be "Public" using ForceReachabilityPublic:

p2pHost, err := libp2p.New(
     libp2p.ForceReachabilityPublic(),
     ....,
)

For example. Did you already tried that in your code?

[b00f]

No, because that is a hack. It may cover the issue but not fix it.

[melroy89]

ok, and what about my telnet connection closed issue?

Furthermore, IPv6 address 2400:8901::f03c:94ff:fe10:5579 doesn't seem to have any port open whatsoever. In fact, the IPv6 IP can not even be pinged. Yes, my internet is Dual WAN (meaning I have IPv4 and IPv6 network), meaning I'm able to test both. And again IPv4 port (TCP) will close connection and IPv6 address doesn't even ping and also does not have port 21888 open at all...

Are you really sure your configured IPv6 correctly in your firewall?

[b00f]

Melror, thanks for the information provided! It's great to see that you are going to investigate this problem. Whatever the outcome, it will definitely help the LibP2P project, either by identifying a bug or improving the documentation to prevent future mistake by other projects.

---

I am not really good with networking tools and debugging.
Since we are dealing with a distributed system, we can analyze the system's overall behavior to gain a better understanding of the issue rather than relying on a specific node that might be misconfigured.

I wrote a Python script to check the reachability status of some nodes whose addresses we have (as bootstrappers). Most of them have not enabled their gRPC, but some have, so we were able to get some results.

python3 ./examples/pactus_reachability.py
Reachability status of bootstrap1.pactus.org:50051 is Private
Reachability status of bootstrap2.pactus.org:50051 is Private
Reachability status of bootstrap3.pactus.org:50051 is Public
Reachability status of bootstrap4.pactus.org:50051 is Public
Reachability status of pactus-bootstrap.sensifai.com:50051 is Private
Reachability status of 65.108.211.187:50051 is Private
Reachability status of 116.203.238.65:50051 is Private
Reachability status of node1.javad.dev:50051 is Private
Reachability status of pactus1.guanqian.me:50051 is Private
Reachability status of pactus2.guanqian.me:50051 is Private
Reachability status of 144.76.70.103:50051 is Private
Reachability status of 185.233.107.64:50051 is Private
Reachability status of 193.22.146.101:50051 is Private
Reachability status of 188.245.82.138:50051 is Private
Reachability status of 178.63.196.133:50051 is Private
Reachability status of 188.245.64.103:50051 is Private
Reachability status of 37.120.173.43:48844 is Private
Reachability status of 103.156.0.148:50051 is Private
Reachability status of 1.53.252.54:50051 is Private
Reachability status of 103.35.64.107:50051 is Public
Reachability status of 116.203.238.65:50051 is Private
Reachability status of 103.35.64.107:50051 is Public
Reachability status of 135.181.255.172:50051 is Public
Reachability status of 103.42.117.229:50051 is Private
Reachability status of 65.108.211.187:50051 is Private
Reachability status of 49.13.202.219:50051 is Unknown
Reachability status of 116.202.24.60:50051 is Private
Reachability status of 188.121.116.247:50051 is Private

See, most of them are private.
Here are some thoughts:

1. It could be a misconfiguration of the entire network, very unlikely, since we have no documentation on how to configure the server or anything about the hosting system. Meanwhile, they run Pactus on different machines. However, it’s possible; for example, not enabling the NAT server. We have opted to enable some. I'm not sure how we can verify if a node has enabled the NAT service or not.

2. It could be a mistake in the Pactus code. This is very likely, but it’s strange that it wasn’t observed in previous versions. We may have made a mistake in our code. If you could check, that would be helpful, or we may need to re-review the entire code to see if anything was missed.

3. It could be a bug in LibP2P. This is possible, but we need to confirm it first. Back to case number 2.

pactus_reachability.py:

# How to run:
# Install Pactus Python SDK: `pip install pactus-sdk`
# Execute the script:        `python3 pactus_reachability.py`
#

from pactus.rpc.network_pb2_grpc import NetworkStub
from pactus.rpc.network_pb2 import GetNodeInfoRequest
import grpc

public_node = [
    "bootstrap1.pactus.org:50051",
    "bootstrap2.pactus.org:50051",
    "bootstrap3.pactus.org:50051",
    "bootstrap4.pactus.org:50051",
    "pactus-bootstrap.sensifai.com:50051",
    "65.108.211.187:50051",
    "84.247.165.249:50051",
    "62.171.130.196:50051",
    "135.181.42.222:50051",
    "pactus-bootstrap.stakers.world:50051",
    "158.220.91.129:50051",
    "pactus-bootstrap.ionode.online:50051",
    "159.148.146.149:50051",
    "pactus-bootnode.stake.works:50051",
    "65.108.68.214:50051",
    "65.21.180.80:50051",
    "37.27.26.40:50051",
    "pactus-bootstrap.mflow.tech:50051",
    "5.161.222.154:50051",
    "65.21.155.128:50051",
    "pactus-bootstrap.teoviteovi.com:50051",
    "37.60.233.235:50051",
    "43.135.166.197:50051",
    "46.250.233.5:50051",
    "pactus-bootstrap.codeblocklabs.com:50051",
    "84.247.184.95:50051",
    "pactus-bootstrap.thd.io.vn:50051",
    "37.27.25.245:50051",
    "173.212.254.120:50051",
    "62.171.134.3:50051",
    "109.123.240.5:50051",
    "75.119.159.113:50051",
    "161.97.145.100:50051",
    "109.123.253.88:50051",
    "167.86.80.169:50051",
    "65.21.69.53:50051",
    "pactus-bootstrap.validator.wiki:50051",
    "157.90.22.45:50051",
    "65.21.152.153:50051",
    "128.140.103.90:50051",
    "185.216.75.142:50051",
    "85.190.246.251:50051",
    "45.134.226.48:50051",
    "154.38.187.40:50051",
    "194.163.164.10:50051",
    "167.86.116.61:50051",
    "46.196.214.35:50051",
    "42.117.247.180:50051",
    "38.242.225.172:50051",
    "167.86.121.11:50051",
    "194.163.189.255:50051",
    "37.60.245.47:50051",
    "84.247.172.23:50051",
    "95.216.222.135:50051",
    "65.109.143.78:50051",
    "116.203.238.65:50051",
    "49.13.27.37:50051",
    "37.60.241.89:50051",
    "195.201.112.164:50051",
    "94.72.120.50:50051",
    "95.111.242.225:50051",
    "pactus-bootstrap1.dezh.tech:50051",
    "5.75.183.248:50051",
    "node1.javad.dev:50051",
    "pactus1.guanqian.me:50051",
    "pactus2.guanqian.me:50051",
    "95.217.89.202:50051",
    "144.76.70.103:50051",
    "185.233.107.64:50051",
    "193.22.146.101:50051",
    "188.245.82.138:50051",
    "178.63.196.133:50051",
    "188.245.64.103:50051",
    "37.120.173.43:48844",
    "103.156.0.148:50051",
    "1.53.252.54:50051",
    "103.35.64.107:50051",
    "116.203.238.65:50051",
    "103.35.64.107:50051",
    "135.181.255.172:50051",
    "103.42.117.229:50051",
    "65.108.211.187:50051",
    "49.13.202.219:50051",
    "116.202.24.60:50051",
    "188.121.116.247:50051",
]


def main() -> None:
    for node in public_node:
        try:
            channel = grpc.insecure_channel(node)
            stub = NetworkStub(channel)
            req = GetNodeInfoRequest()
            res = stub.GetNodeInfo(req, timeout=3)

            print(f"Reachability status of {node} is {res.reachability}")
        except Exception as e:
            # print(f"Error: {e}")
            pass

if __name__ == "__main__":
    main()