fix two pubsub issues. #7394

Stebalien · 2020-05-29T22:59:30Z

Flood for first pubsub hop to ensure the message gets broadcasted even if some of our peers are overloaded.
Remove discovery logic. This is now handled by pubsub internally.

When publishing a value, flood. This ensures that our messages make it to the network even if some of our peers are bad and/or overloaded.

Pubsub handles this internally now.

Stebalien · 2020-05-30T01:25:49Z

@achingbrain this now floods on initial publish.

However, looking at the failure, publishing from go isn't the issue so flooding from go doesn't help.

Stebalien · 2020-05-30T01:26:03Z

I'm also not sure what is wrong with the ipns interop tests.

achingbrain · 2020-06-01T17:53:11Z

I've been trying to isolate the problem and I'm not getting very far. It's definitely the go/js interactions - go/go and js/js seem to pass.

The failures are intermittent, though appear on Linux the most frequently.

I've been running it with js-libp2p-gossipsub logging turned on, and the weird thing is from the JS end the log lines are identical between passing and failing test runs (if you ignore the peer IDs and transferred data):

Passing:

    publish from JS, subscribe on Go
initializing IPFS node at /tmp/go_ipfs_j-vToZ9-LY6h_G84V2MHP
generating 2048-bit RSA keypair...done
peer identity: QmRe416CaMu9R1EhiK5igXScUpoNfU13HgNGCgJ1F7jQzq
to get started, enter:

  ipfs cat /ipfs/QmQPeNsJPyVWPFDVHb77w8G42Fvo15z4bG2X8D2GhfbSXc/readme


ipfsd-ctl:utils Unable to open api file
Initializing daemon...
go-ipfs version: 0.5.1
Repo version: 9
System version: amd64/linux
Golang version: go1.13.10

Swarm listening on /ip4/127.0.0.1/tcp/41383
Swarm listening on /p2p-circuit
Swarm announcing /ip4/127.0.0.1/tcp/41383

API server listening on /ip4/127.0.0.1/tcp/33841
WebUI: http://127.0.0.1:33841/webui
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/34939
Daemon is ready

initializing ipfs node at /tmp/js_ipfs_xbZEPe5-82jiSP9yCHhnL
generating 2048-bit RSA keypair...
to get started, enter:

  jsipfs cat /ipfs/QmfGBRT6BbWJd7yUc2uYdaUZJBbnEFvTqehPFoSMQ6wgdr/readme


ipfsd-ctl:utils Unable to open api file
Initializing IPFS daemon...

js-ipfs version: 0.45.0
System version: x64/linux
Node.js version: 12.17.0

libp2p:transports adding TCP
libp2p:transports adding WebSockets
libp2p:transports adding WebRTCStar
libp2p:transports adding Circuit
libp2p libp2p is starting
libp2p:transports creating listener for TCP on /ip4/127.0.0.1/tcp/0
libp2p:tcp:listener Listening on { address: '127.0.0.1', family: 'IPv4', port: 40359 }
libp2p:gossipsub starting
libp2p:gossipsub started
libp2p libp2p has started

Swarm listening on /ip4/127.0.0.1/tcp/40359/p2p/QmYmW2uEQASEDTWsk14vv12PK9pLbdftv3fioEJqRD1xQ8

API listening on /ip4/127.0.0.1/tcp/39787/http

Gateway (read only) listening on /ip4/127.0.0.1/tcp/35293/http
Web UI available at http://127.0.0.1:39787/webui
Daemon is ready

libp2p:dialer 1 tokens request, returning 1, 149 remaining
libp2p:tcp dialing {"family":"ipv4","host":"127.0.0.1","transport":"tcp","port":41383}
libp2p:tcp connection opened {"family":"ipv4","host":"127.0.0.1","transport":"tcp","port":41383}
libp2p:tcp new outbound connection /ip4/127.0.0.1/tcp/41383/p2p/QmRe416CaMu9R1EhiK5igXScUpoNfU13HgNGCgJ1F7jQzq
libp2p:tcp outbound connection /ip4/127.0.0.1/tcp/41383/p2p/QmRe416CaMu9R1EhiK5igXScUpoNfU13HgNGCgJ1F7jQzq upgraded
libp2p:dialer token 0 released
libp2p:dialer dial succeeded to QmRe416CaMu9R1EhiK5igXScUpoNfU13HgNGCgJ1F7jQzq
libp2p:gossipsub connected QmRe416CaMu9R1EhiK5igXScUpoNfU13HgNGCgJ1F7jQzq
libp2p:gossipsub new peer QmRe416CaMu9R1EhiK5igXScUpoNfU13HgNGCgJ1F7jQzq
libp2p:identify received observed address of /ip4/127.0.0.1/tcp/44242
---> starting test
libp2p:gossipsub rpc from QmRe416CaMu9R1EhiK5igXScUpoNfU13HgNGCgJ1F7jQzq
libp2p:gossipsub rpc from QmRe416CaMu9R1EhiK5igXScUpoNfU13HgNGCgJ1F7jQzq
Found QmRe416CaMu9R1EhiK5igXScUpoNfU13HgNGCgJ1F7jQzq on attempt 2 in 1013ms
---> sending message
libp2p:gossipsub publish pubsub-ascii <Buffer 68 65 6c 6c 6f 20 77 6f 72 6c 64>

      ✓ should exchange ascii data (1038ms)
---> starting test
libp2p:gossipsub rpc from QmRe416CaMu9R1EhiK5igXScUpoNfU13HgNGCgJ1F7jQzq
Found QmRe416CaMu9R1EhiK5igXScUpoNfU13HgNGCgJ1F7jQzq on attempt 1 in 12ms
---> sending message
libp2p:gossipsub publish pubsub-non-ascii <Buffer e4 bd a0 e5 a5 bd e4 b8 96 e7 95 8c>

      ✓ should exchange non ascii data
---> starting test
libp2p:gossipsub rpc from QmRe416CaMu9R1EhiK5igXScUpoNfU13HgNGCgJ1F7jQzq
Found QmRe416CaMu9R1EhiK5igXScUpoNfU13HgNGCgJ1F7jQzq on attempt 1 in 7ms
---> sending message
libp2p:gossipsub publish pubsub-binary <Buffer a3 61 61 63 61 79 65 61 62 83 01 03 05 61 64 a1 64 66 66 66 66 f4 00 01 02 03 04 05 06 07 08 09>

      ✓ should exchange binary data

Failing:

    publish from JS, subscribe on Go
initializing IPFS node at /tmp/go_ipfs_J69F1HoBuDfvAQhcg96Yr
generating 2048-bit RSA keypair...done
peer identity: QmPUutxCBTtkAm8Tj1JCyhwkXeUiFc7dxnBsPxGKzhoWYZ
to get started, enter:

  ipfs cat /ipfs/QmQPeNsJPyVWPFDVHb77w8G42Fvo15z4bG2X8D2GhfbSXc/readme


ipfsd-ctl:utils Unable to open api file
Initializing daemon...
go-ipfs version: 0.5.1
Repo version: 9
System version: amd64/linux
Golang version: go1.13.10

Swarm listening on /ip4/127.0.0.1/tcp/40637
Swarm listening on /p2p-circuit
Swarm announcing /ip4/127.0.0.1/tcp/40637

API server listening on /ip4/127.0.0.1/tcp/33691
WebUI: http://127.0.0.1:33691/webui

Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/43721
Daemon is ready

initializing ipfs node at /tmp/js_ipfs_qUCk9N5yg3uJoLtPWYhW8
generating 2048-bit RSA keypair...
to get started, enter:

  jsipfs cat /ipfs/QmfGBRT6BbWJd7yUc2uYdaUZJBbnEFvTqehPFoSMQ6wgdr/readme


ipfsd-ctl:utils Unable to open api file
Initializing IPFS daemon...

js-ipfs version: 0.45.0

System version: x64/linux
Node.js version: 12.17.0

libp2p:transports adding TCP
libp2p:transports adding WebSockets
libp2p:transports adding WebRTCStar
libp2p:transports adding Circuit
libp2p libp2p is starting
libp2p:transports creating listener for TCP on /ip4/127.0.0.1/tcp/0
libp2p:tcp:listener Listening on { address: '127.0.0.1', family: 'IPv4', port: 44703 }
libp2p:gossipsub starting
libp2p:gossipsub started
libp2p libp2p has started

Swarm listening on /ip4/127.0.0.1/tcp/44703/p2p/QmQxkcEifHrJzVdR4HSQHCpmeLDEgMHtqpNqu41oG6sv64

API listening on /ip4/127.0.0.1/tcp/36767/http

Gateway (read only) listening on /ip4/127.0.0.1/tcp/36915/http
Web UI available at http://127.0.0.1:36767/webui
Daemon is ready

libp2p:dialer 1 tokens request, returning 1, 149 remaining
libp2p:tcp dialing {"family":"ipv4","host":"127.0.0.1","transport":"tcp","port":40637}
libp2p:tcp connection opened {"family":"ipv4","host":"127.0.0.1","transport":"tcp","port":40637}
libp2p:tcp new outbound connection /ip4/127.0.0.1/tcp/40637/p2p/QmPUutxCBTtkAm8Tj1JCyhwkXeUiFc7dxnBsPxGKzhoWYZ
libp2p:tcp outbound connection /ip4/127.0.0.1/tcp/40637/p2p/QmPUutxCBTtkAm8Tj1JCyhwkXeUiFc7dxnBsPxGKzhoWYZ upgraded
libp2p:dialer token 0 released
libp2p:dialer dial succeeded to QmPUutxCBTtkAm8Tj1JCyhwkXeUiFc7dxnBsPxGKzhoWYZ
libp2p:gossipsub connected QmPUutxCBTtkAm8Tj1JCyhwkXeUiFc7dxnBsPxGKzhoWYZ
libp2p:gossipsub new peer QmPUutxCBTtkAm8Tj1JCyhwkXeUiFc7dxnBsPxGKzhoWYZ
libp2p:identify received observed address of /ip4/127.0.0.1/tcp/33788
---> starting test
libp2p:gossipsub rpc from QmPUutxCBTtkAm8Tj1JCyhwkXeUiFc7dxnBsPxGKzhoWYZ
libp2p:gossipsub rpc from QmPUutxCBTtkAm8Tj1JCyhwkXeUiFc7dxnBsPxGKzhoWYZ
Found QmPUutxCBTtkAm8Tj1JCyhwkXeUiFc7dxnBsPxGKzhoWYZ on attempt 2 in 1018ms
---> sending message
libp2p:gossipsub publish pubsub-ascii <Buffer 68 65 6c 6c 6f 20 77 6f 72 6c 64>

      1) should exchange ascii data
---> starting test
libp2p:gossipsub rpc from QmPUutxCBTtkAm8Tj1JCyhwkXeUiFc7dxnBsPxGKzhoWYZ
Found QmPUutxCBTtkAm8Tj1JCyhwkXeUiFc7dxnBsPxGKzhoWYZ on attempt 1 in 12ms
---> sending message
libp2p:gossipsub publish pubsub-non-ascii <Buffer e4 bd a0 e5 a5 bd e4 b8 96 e7 95 8c>

      2) should exchange non ascii data
---> starting test
libp2p:gossipsub rpc from QmPUutxCBTtkAm8Tj1JCyhwkXeUiFc7dxnBsPxGKzhoWYZ
Found QmPUutxCBTtkAm8Tj1JCyhwkXeUiFc7dxnBsPxGKzhoWYZ on attempt 2 in 1016ms
---> sending message
libp2p:gossipsub publish pubsub-binary <Buffer a3 61 61 63 61 79 65 61 62 83 01 03 05 61 64 a1 64 66 66 66 66 f4 00 01 02 03 04 05 06 07 08 09>

      3) should exchange binary data

A weird thing is that Go sends two RPC messages to JS but only in the first test. Then again it does it in both the successful and the failed runs.

Maybe @vasco-santos @jacobheun or @wemeetagain know more about what's going on in the gossipsub module internally?

How can I get more logging out of the go-ipfs daemon?

Stebalien · 2020-06-01T19:24:10Z

@achingbrain you can run ipfs log level pubsub debug after starting the daemon.

aschmahmann

LGTM (modulo failing interop tests)

Stebalien · 2020-06-01T21:28:18Z

Merging as the interop tests are already broken. I've added a line to the 0.6.0 todo list to fix this issue.

Stebalien requested a review from aschmahmann May 29, 2020 23:00

Stebalien mentioned this pull request May 29, 2020

with gossipsub subscribers may not receive the first couple of messages libp2p/go-libp2p-pubsub#331

Open

Stebalien force-pushed the fix/pubsub branch from 71c4a53 to 780b261 Compare May 29, 2020 23:26

Stebalien added 2 commits May 29, 2020 16:42

fix(pubsub): flood publish

ac8a88d

When publishing a value, flood. This ensures that our messages make it to the network even if some of our peers are bad and/or overloaded.

fix: remove pubsub discovery hack

c58e3e4

Pubsub handles this internally now.

Stebalien force-pushed the fix/pubsub branch from 780b261 to c58e3e4 Compare May 29, 2020 23:42

aschmahmann approved these changes Jun 1, 2020

View reviewed changes

Stebalien merged commit cf68619 into master Jun 1, 2020

Stebalien added the need/backport label Jun 1, 2020

Stebalien added this to the go-ipfs 0.6 milestone Jun 1, 2020

Stebalien deleted the fix/pubsub branch June 1, 2020 21:29

vasco-santos mentioned this pull request Jun 2, 2020

upgrade dependencies. libp2p/go-libp2p-daemon#233

Merged

achingbrain mentioned this pull request Jun 2, 2020

Interop tests are failing against master #7390

Closed

Stebalien removed the need/backport label Jun 3, 2020

aschmahmann mentioned this pull request Sep 22, 2020

Release 0.7.0 #7560

Closed

72 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix two pubsub issues. #7394

fix two pubsub issues. #7394

Stebalien commented May 29, 2020

Stebalien commented May 30, 2020

Stebalien commented May 30, 2020

achingbrain commented Jun 1, 2020

Stebalien commented Jun 1, 2020

aschmahmann left a comment •

edited

Loading

Stebalien commented Jun 1, 2020

fix two pubsub issues. #7394

fix two pubsub issues. #7394

Conversation

Stebalien commented May 29, 2020

Stebalien commented May 30, 2020

Stebalien commented May 30, 2020

achingbrain commented Jun 1, 2020

Stebalien commented Jun 1, 2020

aschmahmann left a comment • edited Loading

Choose a reason for hiding this comment

Stebalien commented Jun 1, 2020

aschmahmann left a comment •

edited

Loading