Bring interfaces back here

[ghost]

I feel like once all implementation code is extracted, we could bring back the various interfaces (Conn, Swarm, Host, Peer, Transport, etc.) into a then-flat go-libp2p package. Every libp2p module would depend go-libp2p, but it'd be just a bunch of interfaces.

This would help making the mountain of abstractions more accessible.

package something

import (
  p2p "github.com/libp2p/go-libp2p"
)

// ...

[whyrusleeping]

I think i like this idea, but would need to draw out the dependency graph first to figure out how it would look in practice

[ghost]

Awesome -- yeah agreed about the dependency graph. The remaining extractions look something like this:

• p2p/discovery -- go-libp2p-mdns
• p2p/net/mock -- go-libp2p-mocknet
• p2p/protocol -- go-libp2p-ping, go-libp2p-identify, and relay can be killed
• p2p/host -- go-libp2p-basichost, go-libp2p-routedhost
• no idea about p2p/test

It's not actually that much left.

[ghost]

I've looked into this a bit more now, and it looks relatively staightforward. The annoying part will be updating all the packages depending on go-libp2p/p2p/ and the interface packages. Annoying, but still straightforward.

Though in addition to improving general accessibility, a handful of things can be simplified after the move :) For example we can remove a few slight duplications in the interfaces (namely in the various Conns and Listeners).

Any package which gets touched (likely all of them) should probably get a major version bump.

Extractions

• p2p/host
  • to go-libp2p-host (and move Host interface back into go-libp2p)
  • maybe let p2p/host/routed have its own package
• p2p/test
  • to go-libp2p-host (both reconnect and backpressure are integration tests for host/basic)
• p2p/net/mock
  • to go-libp2p-mocknet (maybe some testing-related packages can eventually be merged)
• p2p/protocol/identify
  • to go-libp2p-identify (WIP in Extract from go-libp2p go-libp2p-identify#1)
• p2p/protocol/ping
  • to go-libp2p-ping
• p2p/discovery/mdns
  • to go-libp2p-mdns
• examples
  • Can we have nested packages with gx? It's valuable to have this directory right here, but it's also valuable to cut the examples's dependencies out of the go-libp2p package. (Answer: it looks like we can, see go-peerstream/muxtest)
  • otherwise: to go-libp2p-examples

Packages to move back

• go-libp2p-host
  • Host interface
  • multistream semver utilities
• go-libp2p-interface-conn
  • Conn, PeerConn, Listener interfaces
  • utilities
• go-libp2p-interface-pnet
  • Protector interface
  • the ForcePrivateNetworks and EnvKey variables can probably be simplified after the move
• go-libp2p-net
  • Stream, StreamHandler, Conn, ConnHandler interfaces
  • Network, Listener, Dialer interfaces
  • Connectedness, Notifiee interfaces
• go-libp2p-peer
  • peer.ID type and utilities
• go-libp2p-protocol
  • protocol.ID type
• go-libp2p-peerstore
  • Peerstore, Addrbook, Keybook interfaces
  • an interface derived from PeerInfo
  • addr.Source, queue.PeerQueue interfaces
• go-libp2p-routing
  • ContentRouting, PeerRouting, IpfsRouting interfaces
  • ValueStore, PubkeyFetcher interfaces, RecvdVal type
  • utilities
• go-libp2p-secio
  • Session interface
• go-libp2p-swarm
  • Should there a Swarm interface?
• go-libp2p-transport
  • Transport, Listener, Dialer, Conn interfaces
  • fallback dialer utilities
• go-libp2p-stream-muxer
  • Stream, StreamHandler, Conn, Transport interfaces

After the move

Once we have only interfaces and tiny utilities in go-libp2p, we can optimize this further for developer happiness.

Candidates:

• go-libp2p-standard
  • a package which pulls in a "standard" libp2p stack, basically the kind of stack that ipfs uses, but without ipfs.
  • this would ideally be coordinated so that js-ipfs gets a similar package (I think this already exists), and the two are harmonized.
• testing packages
  • refactor the various testing-related packages
  • option: unified go-libp2p-test package
  • option: every impl package comes with test utilities for itself, e.g. mocknet would move into swarm or host

[Kubuxu]

I am not 100% sure about that, currently it is possible to updgrade one interface and do limited propagation. If we do bring all interfaces to go-libp2p, any update to interface will require propagation among all packages.

This means that first tooling around gx-workspace has to get awesome.

[ghost]

I am not 100% sure about that, currently it is possible to updgrade one interface and do limited propagation. If we do bring all interfaces to go-libp2p, any update to interface will require propagation among all packages.

Yeah, I think so too. Any change in the go-libp2p package would require an update of nearly all other go-libp2p-* packages. But I have a hunch it might actually turn out to be a good thing, because it's largely below the "UI surface". Within libp2p we can take care of it with tooling, and users of libp2p (ipfs included) will actually end up with smaller updates, since the surface of libp2p packages they directly depend on gets smaller.

This means that first tooling around gx-workspace has to get awesome.

Agreed 👍

[ghost]

This means that first tooling around gx-workspace has to get awesome.

Also @Kubuxu let's test ipfs-inactive/gx-workspace#3 more, it's promising.

[Kubuxu]

I tested it a bit and it still needs a lot of work. Libp2p currently is 29 packages, sorry but I can't really imagine doing all of that even with gx-wrokspace in its current state.

[hsanjuan]

From IRC:

13:29:03 <hsanjuan> lgierth: I'm not sure what to think about your libp2p proposal
13:29:22 <hsanjuan> it does not bother me to import modules (goimports does it)
13:29:29 <hsanjuan> and it is explicit
13:29:42 <hsanjuan> explicitness never hurts
13:30:10 <hsanjuan> i'll put this in the issue

So I'll leave it up to you, but for me personally it addresses a problem that does not exist or is not a big deal (for the amount of refactoring required).

[ghost]

Also from IRC. tl;dr let's scrap the interfaces move. Let's just have go-libp2p hold tested examples for everything.

<lgierth> ok scrap the go-libp2p interfaces move
<lgierth> there's a lot we can and should refactor but this kind of move is the wrong direction
<Kubuxu> what changed your mind?
<lgierth> the need for monolithic updates
<lgierth> and i just had a better idea :P
<lgierth> libp2p/go-libp2p would just hold examples, and acceptance tests for the examples
<lgierth> and be the source of the version for the whole suite of packages
<lgierth> packages can still be bumped independently like now, but it'd be a framing for "libp2p releases"
<lgierth> a new release of go-libp2p would for example contain: new examples+tests for new features, updated dependencies, and a changelog summarizing this
<lgierth> i think right now it's pretty hard for users of libp2p to determine when it's actually safe to update certain things, and in which combination
<lgierth> about the examples, they'd turn tests-as-example around: examples-as-tests
<lgierth> and fuck we just need tests for the examples. the echo example (!) broke so often in just the last few weeks
<lgierth> :)
<lgierth> :')
<lgierth> and i'd like to generate docs.libp2p.io from js and go code docs
<lgierth> with nice client-side fuzzy search
<Kubuxu> We can have also libp2p depend on "all" interfaces
<Kubuxu> which means that you don't have to have in your dependencies
<Kubuxu> just the go-libp2p
<lgierth> i think we'd have it depend on everything
<lgierth> because ideally the examples cover everything
<lgierth> and at the same time the examples could serve as integration tests for the whole thing
<lgierth> we don't really have those -- go-ipfs is kind of a test suite for libp2p

[daviddias]

I've looked into this a bit more now, and it looks relatively staight forward. The annoying part will be updating all the packages depending on go-libp2p/p2p/ and the interface packages. Annoying, but still straightforward

I believe it would only not be annoying if gx didn't exist but as soon as semver is a thing (with gx), circular deps are a pain unless I'm missing a special feature of gx??!

[ghost]

Yeah this whole thing wasn't though through very well