[CLI] DKG ceremony crash upon existing validator_keys folder

[crisog]

🐞 Bug Report

Description

This shouldn't have happened. Apparently, all the other nodes in the cluster received their key shares except me. It doesn't seem to be an obvious recovery point, the only ideas that come to head are:

1. Everyone deletes their validator_keys folder, which is not practical when you deal with users using dappnode and have no direct access to the OS. (This is assuming DKG can be repeated)
2. A new cluster is created

In my opinion, what should happen is: if validator_keys folder exists, then a new directory gets created with a random prefix, i.e. xyz_validator_keys and the keys are written there. This way the key shares wouldn't have been lost after a successful DKG ceremony

🔬 Minimal Reproduction

docker run -u $(id -u):$(id -g) --rm -v "$(pwd)/:/opt/charon" obolnetwork/charon:v0.18.0 dkg --definition-file="https://api.obol.tech/dv/<DV_CLUSTER>" --publish

🔥 Error

15:28:54.256 INFO dkg        Connected to peer 2 of 3                 {"peer": "nervous-painter"}
15:48:55.335 INFO dkg        Connected to peer 2 of 3                 {"peer": "strange-jewelry"}
16:28:54.630 INFO dkg        Connected to peer 2 of 3                 {"peer": "nervous-painter"}
16:31:05.277 INFO dkg        Connected to peer 3 of 3                 {"peer": "shy-computer"}
16:31:06.453 INFO dkg        All peers connected, starting DKG ceremony
16:31:11.644 ERRO cmd        Fatal error: mkdir /validator_keys: mkdir .charon/validator_keys: file exists
    dkg/disk.go:128 .writeKeysToDisk
    dkg/dkg.go:315 .Run
    cmd/dkg.go:35 .func1
    cmd/cmd.go:80 .func1
    main.go:19 .main

🌍 Your Environment

Operating System:

  
 Docker Image: obolnetwork/charon:v0.18.0
  

What version of Charon are you running? (Which release)

  
v0.18.0
  

Anything else relevant (validator index / public key)?

[gsora]

Hi @crisog!

First and foremost, could you try reproducing this issue with our latest release, v0.19.1?

Apparently, all the other nodes in the cluster received their key shares except me.

Since the DKG process is a P2P process, this is unfortunately a thing that can happen: maybe a peer closed its terminal, maybe it crashed, maybe their internet connection failed.

While your terminal showed the validator_keys-related error, other peers should've shown messages stating that the DKG process has failed and must be re-executed: one can always re-run the DKG process with the same definition file by cleaning their .charon directory as explained here.

The keys that other peers have received should be ignored, since the DKG process did not finish in an ordered fashion for all peers.

In my opinion, what should happen is: if validator_keys folder exists, then a new directory gets created with a random prefix, i.e. xyz_validator_keys and the keys are written there. This way the key shares wouldn't have been lost after a successful DKG ceremony

As you pointed out, in the context of e.g. a Dappnode deployment even if the DKG created a different validator_keys directory, one would have to dig into the Dappnode configuration files and figure out a way to point the validator stack to this new directory.

---

Thank you for pointing out this UX issue with Dappnode and similar tools, food for thought for our development team!