Key management on a decentralized network

[rngadam]

vpncloud is good stuff. seems to be a great start as a solution to build a decentralized data center.

Given this scenario:

• community of system administrators pooling their computers as nodes in a network
• each node joins the vpncloud network and subsequently (with VPN up) joins a microk8s cluster
• installation and setup of each node is decentralized (done by the node owner using Ansible script on is LAN before moving the node to the DMZ)
• we want to use public keys and private keys to authenticate between nodes

The challenge we have is to have each node share its public key generated at setup with every other node and have these nodes reload new nodes public keys before the node tries to join the vpn.

This is the solution we're shooting for right now:

does this approach make sense?

[dswd]

(moved here from issue)

This a very interesting and complex usecase you are describing. Scaling a key pair exchange is always challenging. Full blown PKIs normally include certificates that you can chain and thereby derive trust from a shared root key. VpnCloud does not support this (neither does tinc, afaik). There are no certificates, no cert chains and no inherited trust. I could add this but I am not sure I want to go down this rabbit hole (SSL is having security issues because of this feature).

Your setup seems fine as far as I can see. It is not clear who owns which component and whether there are different zones of trust. It looks a little bit odd to put private keys in a repo though. Did you know, you could derive the VpnCloud key pairs from passwords too? Not sure if that helps.

Your usecase is interesting. Let me know, if you have any blog post that describes the setup. It could be helpful for other users if I link to it in the documentation.

[bmullan]

Would the open source CoreDNS be useful in this? Their Plugin system in particular?

Although the following excerpt talks about a WireGuard plugin's use w/ CoreDNS.. could VpnCloud do the same thing using a wgsd-like CoreDNS plugin for VpnCloud?

wgsd is a CoreDNS plugin that serves WireGuard peer information via DNS-SD (RFC6763) semantics. This enables use cases such as:

Building a mesh of WireGuard peers from a central registry
Dynamic discovery of WireGuard Endpoint addressing (both IP address and port number)
NAT-to-NAT WireGuard connectivity where **[UDP hole punching](https://en.wikipedia.org/wiki/UDP_hole_punching)** is supported.

See this blog post for a deep dive on the underlying techniques and development thought.

Just thought I'd throw this out there...
Brian

[rngadam]

I also had originally thought certificate chains could help but on second thought, it adds complexity that's not needed by inserting a centralised issuing organization. Best to control access and authorization to the index of public keys (which is done through giving access to modifying the repository).

Private keys will be stored in the node owner private repo with symmetric encryption (along a number of secrets, such as private certificate keys). This is to ensure data safety in case the local deployment node "explodes".

I'll request our team to prepare English document.

At the moment we're working out of https://github.com/dravedev/devops but in French (our organization goal is to advance digital sovereignty for Quebec and a big part of it is making technology accessible in French).

short-term the work is:

• automation of the key generation to a file
• modifying the repository of public keys should lead to a webhook being called on every node to reload the keys from the repo

My thinking on trust is that only the owner of the node(s) should have the rights to set it up and join the VPN. Then, each node is joined to a cluster (microk8s) and it's at that cluster layer that other admins can manage and deploy apps on all nodes.

One issue we're facing is having to centrally manage IP on the VPN to define the mapping between private IP and internal/external node name. A DHCP like setup for reserved subnetworks with a federated internal DNS system would be needed to solve this.

[dswd]

@rngadam I tried to read up on your page as far as my limited French skills permit. It seems you are a group of citizens (lumber floaters ?!?) who want to promote open source software and open IT system to your local government. This is a great goal. I would be glad if my project could help you.

Distributed DHCP was another rabbit hole I didn't want to go down. However I start thinking this might be a useful addition. Maybe that is something that I can add to the agenda for 3.0.

[rngadam]

@rngadam I tried to read up on your page as far as my limited French skills permit. It seems you are a group of citizens (lumber floaters ?!?) who want to promote open source software and open IT system to your local government. This is a great goal. I would be glad if my project could help you.

Yes, and not just government but also small businesses and non-profits.

"Drave" (and the people, "Draveurs") was the process to transport wood logs using rivers. It used to be really central to our wood industry (with lumberjacks) as we were sourcing wood from far and wide in the province. It's a profession that's now disappeared but is still mythical to Quebeckers.

Distributed DHCP was another rabbit hole I didn't want to go down. However I start thinking this might be a useful addition. Maybe that is something that I can add to the agenda for 3.0.

Thinking about this: IP could be autoconfigured and we could use zeroconf to publish hostname as hostname.local (making it unnecessary to know the IP in advance)?

https://en.wikipedia.org/wiki/Zero-configuration_networking

Hosts on a network must be assigned IP addresses that uniquely identify them to other devices on the same network. On some networks there is a central authority that assigns these addresses as new devices are added. Mechanisms were introduced to handle this task automatically, and both IPv4 and IPv6 now include systems for address autoconfiguration, which allows a device to determine a safe address to use through simple mechanisms. For link-local addressing, IPv4 uses the special block 169.254.0.0/16 as described in RFC 3927 while IPv6 hosts use the prefix fe80::/10. More commonly addresses are assigned by a DHCP server, often built into common networking hardware like computer hosts or routers.

[jvtrudel]

Beacons already offer a format to publish network nodes information. Maybe I should have a closer look on that feature and first automate publication and synchronization of this data on github.

[rngadam]

new proposal

To follow up with this discussion, the original proposal was a bit complicated to implement so I came up with this alternative flow that takes advantage of having the users and their public key ssh access setup on each node:

downsides

the downside with this proposal compared to the original one is that instead of the community of node owners only having deployment access through the upper Kubernetes layer, they now have at least ssh access to every other node (although the commands could also be locked down here in authorized_keys to only allow creating a new peer config).

feature request

to make this work vpncloud would be to add the ability to read peer configurations from a directory /etc/vpncloud/peers (public key, dns name and port) - one file per peer. after the scp of the peer config we could also have ansible restart the remote vpncloud service.

as a temporary workaround, we could wrap vpncloud into a starting script to generate the config from this list of peers

more discussion

the other tweak would be to have each node(s) owner generate a single key per user (the private/public vpncloud key is the same for every node owned by that user).

Question: wouldn't there be a way to piggy back off ssh-keygen generated keys and use those instead as input to vpncloud? would that be interesting to allow reusing ssh keys within vpncloud?

[dswd]

Ok I think I now actually understood your requirements:

1. You want to deploy an arbitrary number of nodes using ansible (one script to deploy a node).
2. You want those nodes to find each others in a VPN.
3. You want to deploy the nodes without having to maintain a static list of all nodes.

(If I got it wrong, please tell me)

The central problem here is that new nodes somehow need to get their public keys into the config of old nodes.
I would create a script that runs as part of the ansible deployment and that puts the public key in a location shared and trusted by all nodes. This could be an etcd, a file in a shared folder (PVC or even Dropbox), or a Git repo, does not matter.
The important part is that the other nodes are able to get updates from this location (you do not want to give all other nodes SSH access, that does not sound good) either by polling regularly or, even better, by an inotify watcher.
Whever they detect a change, i.e. a new public key or peer entry, they will read all the public keys and peer addresses and create a new combined VpnCloud config and then trigger a service restart. I can help you write those scripts if you need.

On your feature request: I think this addition would be confusing to some users as it creates the impression, that the trusted key and the peer address defined in the same file are somehow associated which they are not. Any peer (even an unlisted one) can establish a connection with any trusted key. Defining them in pairs creates the false impression that a peer is somehow limited to a certain trusted key. If I really have to support a conf.d style config, I want the merging logic to be obvious and easy to understand.

Taking SSH keys for trust is a new idea, that I didn't have before. You could theoretically use SSH keys for that, if the keys are based on Curve25519. Unfortunately almost all systems right now still create RSA keys that VpnCloud can not use.
Of course, you can use SSH private key to derive a VpnCloud key pair in a repeatable way so that you don't need to store that key but then you still need to exchange the public keys as the VpnCloud public key can not be derived from the SSH public key. You can do so right now with vpncloud genkey -p $(sha1sum ~/.ssh/id_rsa | awk '{ print $1 }')

Please be aware, that from a security point of view, your solution is still centralized. Whoever runs and controls the Ansible scripts is in charge of the network. There is also no conceptual difference from this setup to one that uses a single shared password.
I would propose a more decentralized and easier to maintain solution:

• All nodes under the same control / circle of trust (i.e. an organization or a group of admins) share the same key pair.
• You can connect different node groups by explicitly trusting each other, i.e. manually exchange public keys and adding them to a static list in your ansible config. This is an opt-in. There is simple opt-out of this: just remove the trusted key again.
  Extending one group of nodes requires no modification of other nodes' configs. Peering with a new group requires you add the trusted nodes to all of your nodes, i.e. update the ansible config and push it to all nodes.
  This way, you can have non-transitive peerings where organization A peers with org B and org B peers with org C but A and C do not peer and their nodes will not connect.

[rngadam]

We actually had further discussions me and @jvtrudel and ended up in the same general area of thoughts as yours.

configurations

yes, you're right. combining trusted public key and peers information actually subtracts configuration flexibility to what you have now.

So better to have a way to load from /etc/vpncloud/peers and /etc/vpncloud/trustedkeys directories of configuration files (plain text file with one or more item to merge with the config).

I'd imagine that the CLI interface could be augmented to have --peer and --trusted-key accept directory paths.

ssh access to other nodes

this depends on what the organisational setup is (perhaps it's a mutualized tech team that is in charge of all the machines) but again, yes, if we can get away with NOT giving ssh access to other nodes that we don't own that is preferable.

shared store

that is definitely where we're going in what we can call "v3" of our setup. perhaps an ipfs or hypercore or some public hierarchical key/values store with change notification facilities where anyone can drop public keys and peers config but with a twist of having each conf file signed by gpg keypairs (DNSNAME.peer.sig, PUBLICKEYNAME.trustedkey.sig) and build up a parallel web of trust using gpg facilities.

conclusion

Beyond being able to load configurations from directory there should not be any change to vpncloud.

[rngadam]

I'm imagining this setup eventually:

[rngadam]

@dswd after some time in the wilderness of ipfs, figured out my solution was way more complicated than it needed to be.

@bmullan idea to look into how to work with DNS was the good one

Looking at the beacon infrastructure I also think that direction is more promising than text files loading since you already have everything to load continuously new known nodes.

What we have in mind:

• configure vpncloud on target node with ansible with --store /tmp/beacon.txt
• have ansible generate or fetch the node target /tmp/beacon.txt content (could this be generated on demand with a vpncloud beacon encode and tested with vpncloud beacon decode)?
• have ansible push to our DNS records TXT records with the beacon information
• have vpncloud on target node configured to retrieve beacon information from DNS

so our config would look like this:

beacon:
  store: "/tmp/beacon.txt"
  load: "|/usr/bin/dns-beacon.py load"

but... the beacon only contains ip addresses of peers and their ports.

https://github.com/dswd/vpncloud/blob/master/src/beacon.rs

...would there be a way to also stuff public keys in that stream?

It's a good compact format, but I'd suggest capnproto to make it a bit more extensible and versionable (while keeping it compact):

https://github.com/capnproto/capnproto-rust

or perhaps just protobuf:

https://github.com/stepancheg/rust-protobuf

[dswd]

I thought long about this, thus my late response.
The trusted keys in VpnCloud are meant to be manually configured. At first it sounds tempting to automate that in some regard. However, automating trusted keys, essentially takes the "trust" out of the trusted keys and offloads that to the automation process (i.e. DNS in your case).
Using beacons to configure trusted keys (especially via DNS records) does not sound like a good solution to me. DNS records are notoriously slow to propagate updates. So your new nodes would take a long time to become "trusted". Also in an emergency case where one private key is breached, you can't react immediately. Also the beacon feature in VpnCloud is designed to spread information, not to reliably revoke it.

How about storing all trusted keys in a git repo, one file per node (done by ansible, maybe pgp protected) and then regularly pull that repo and on change build the new trusted keys list with a small script and restart the VpnCloud instance?
The long-term roadmap also contains plans for a REST-API that could allow you to manage trusted keys and peers on a running instance without restarting it.

[rngadam]

• DNS records don't have to be slow to update. We use Google Domains (and we could point our DNS resolution directly to Google servers at 8.8.8.8 and 8.8.4.4) with a TTL of 1m and eventually will run our own DNS facilities on our own infrastructure. This would still compare advantageously with pulling.
• a versioned and extensible data format such as protobuf and capnproto should make it relatively easy to add revocation flags
• Yes, the trusted public keys right now are stored in Github and are sync by Ansible. However, it requires each of us in this decentralized network to execute the Ansible script. This could be automated too (based on changes and webhook callback from Github).
• a REST-API would be helpful for dynamic managing trusted keys and peers. Although I'm guessing we'd end up creating ssh tunnels to each node and executing a curl command instead of exposing an HTTP port... So why not an inotify on a directory of public keys and a directory of nodes configs and reloading from there? then we'd only need to scp files to every node (ssh with public key authentication both in the VPN network and public Internet is setup). implementing REST-API seems significantly more involved than that and harder to properly secure.