The witness node's blockchain ID is dependent on the binary structure of the genesis file

[neura-sx]

The process of generating blockchain ID is dependent on the binary structure of the genesis.json file. As a result, two files that are treated as equal by github produce different behavior of the witness node, i.e. different blockchain IDs are generated when the witness node is run with these commands:

witness_node --genesis-json genesis1.json
witness_node --genesis-json genesis2.json

where genesis1.json and genesis2.json are seen as equal files by github but their binary structure is different, as seen on those snapshots:

This behavior causes problems when a witness node tries to join an existing network - it's not able to do so because it expects a different blockchain ID even though the genesis.json file has been cloned from the same repository.

(One could say it's a github bug but unfortunately this is how github works - it's not guaranteed that the binary structure of a text file is the same on all OS).

[theoreticalbts]

How did you download the first file? Specifically what URL did you go to, what browser did you use, and what version? Did it load the genesis.json text in the browser window and you clicked Save As..., or did it prompt you to download the file?

[theoreticalbts]

Hmm, https://help.github.com/articles/dealing-with-line-endings/ may have information.

I wonder why our existing Windows builds aren't bitten by this? Maybe the folks who build it are downloading a tarball instead of doing a checkout.

[xeroc]

I wonder why our existing Windows builds aren't bitten by this? Maybe the folks who build it are downloading a tarball instead of doing a checkout.

I am not sure the OP is talking about the public BTS network and your windows build. I think he was talking about the testnetwork setup.

https://github.com/bitshareseurope/graphene-testnet/

What is wired is that if I compile the above version in Release mode (no explicit egenesis), and let it include the genesis.json file as cloned by git. It will not generate the same chain id, while when I start the witness with --genesis-json genesis.json it will produce the same chain id.

[neura-sx]

I am not sure the OP is talking about the public BTS network and your windows build. I think he was talking about the testnetwork setup.

Yes. I'm talking about the testnet and Windows 10. I did not encounter a similar problem on the public BTS network - I've manged to compile the witness node on Windows and use it without problems with the embedded genesis file. So it's just the testnet where we have this issue.

How did you download the first file? Specifically what URL did you go to, what browser did you use, and what version? Did it load the genesis.json text in the browser window and you clicked Save As..., or did it prompt you to download the file?

In the first case the file was cloned with the rest of the repository using the git clone command.
In the second case it was downloaded using the Download ZIP option on github web-page.

I'm using FF 44.0 on Windows 10.

[theoreticalbts]

OK, I think I now have enough information to close this issue!

When launching a long-lived chain, we use programs/genesis_util/canonical_format.py to make the genesis "canonical" JSON as output by

json.dump(obj, out, separators=(",", ":"), sort_keys=True)

which removes whitespace and sorts keys in JSON dictionaries ("canonical" in removing several formatting changes). This will remove all newline characters from your genesis file (newlines embedded in strings will be written as escape sequences and won't be actual characters in the file).

This is why the issue doesn't affect us -- we always use canonical_format.py. FYI, you can use python_format.py to "undo" the canonical formatting and get a pretty format with indentation and whitespace (but note the results will use platform-dependent newlines!)

My suggestions are:

• If you're creating a new chain, run your genesis file through canonical_format.py before publishing it. It should get rid of line translation issues and has the added benefit of making the file smaller.
• If you're creating a new chain, and you want to show a pretty genesis file with newlines and whitespace in your Github, you can create the ugly file by running canonical_format.py as part of the build process by editing CMakeLists.txt. If you do this, your build will then depend on having a working Python installation (which is why we don't do this).
• If you have an existing chain with a pretty genesis, or want to disregard the above advice and create a new chain with a pretty genesis, and you're using Git to distribute your genesis file, you should add a .gitattributes file to your repository as discussed here.

[theoreticalbts]

Copy-pasted the above points and added a reference to this ticket to the egenesis wiki article.