Adding multiple upstream support to doc generation

[Martyrshot]

This PR fixes an issue discovered by Michael (#1117 (comment)).

This will update legacy yaml files so that the new format is followed, and when a yaml file is updated the script it indicate it has done so. When this happens I recommend double checking the yaml file to make sure the ordering "makes sense" to human eyes.

I also manually patched the yaml files of schemes that the copy_from_upstream.py script doesn't manage.

[Martyrshot]

All of the src/* files will be copy from upstream using the new template format, which looks like it has an indenting issue. I've fixed that issue as part of this PR.

The docs/algorithms/sigs getting updated to the pqclean commit specified in copy_from_upstream.yml

The docs/algorithms/kems will be a combination of an updated pqclean commit and the new yaml and markdown format.

No worries about the german. Maybe more exposure to it will encourage me to start taking lessons again. :-)

[baentsch]

Thanks again for the improvements: Some issues observed in the previous review are gone.
However IMO this PR --like any PR touching these scripts and config files-- should

• contain the algorithm documentation MDs corresponding to the changed YMLs (i.e., as resultant from running scripts/update_docs_from_yaml.py)
• ensure that running scripts/copy_from_upstream/copy_from_upstream.py (followed by scripts/update_docs_from_yaml.py) does not result in any changes to the git repo's local state (such as to ensure further changes to scripts/copy_from_upstream/copy_from_upstream.yml do not lead to any changes other than those driven by changes to that file).

When running the first command on martyrshot/main, this happens:

~/git/martyrshot/liboqs$ git status
Auf Branch main
Ihr Branch ist auf demselben Stand wie 'origin/main'.

nichts zu committen, Arbeitsverzeichnis unverändert
~/git/martyrshot/liboqs$ python3 scripts/update_docs_from_yaml.py 
Updating ./docs/algorithms/kem/bike.md
Updating ./docs/algorithms/kem/classic_mceliece.md
Traceback (most recent call last):
  File "scripts/update_docs_from_yaml.py", line 45, in <module>
    out_md.write('  - **Source**: {}\n'.format(kem_yaml['primary-upstream']['source']))
KeyError: 'primary-upstream'
~/git/martyrshot/liboqs$ git status
Auf Branch main
Ihr Branch ist auf demselben Stand wie 'origin/main'.

Änderungen, die nicht zum Commit vorgemerkt sind:
  (benutzen Sie "git add <Datei>...", um die Änderungen zum Commit vorzumerken)
  (benutzen Sie "git checkout -- <Datei>...", um die Änderungen im Arbeitsverzeichnis zu verwerfen)

	geändert:       docs/algorithms/kem/bike.md
	geändert:       docs/algorithms/kem/classic_mceliece.md

keine Änderungen zum Commit vorgemerkt (benutzen Sie "git add" und/oder "git commit -a")

When doing a dry-run for the second command, this happens:

~/git/martyrshot/liboqs/scripts/copy_from_upstream$ python3 copy_from_upstream.py verify
diff: /home/mib/git/martyrshot/liboqs/src/kem/kyber/pqcrystals-kyber_kyber512_aarch64: Datei oder Verzeichnis nicht gefunden
diff: verify_from_upstream/src/kem/kyber/pqcrystals-kyber_kyber512_aarch64: Datei oder Verzeichnis nicht gefunden
diff: /home/mib/git/martyrshot/liboqs/src/kem/kyber/pqcrystals-kyber_kyber768_aarch64: Datei oder Verzeichnis nicht gefunden
diff: verify_from_upstream/src/kem/kyber/pqcrystals-kyber_kyber768_aarch64: Datei oder Verzeichnis nicht gefunden
diff: /home/mib/git/martyrshot/liboqs/src/kem/kyber/pqcrystals-kyber_kyber1024_aarch64: Datei oder Verzeichnis nicht gefunden
diff: verify_from_upstream/src/kem/kyber/pqcrystals-kyber_kyber1024_aarch64: Datei oder Verzeichnis nicht gefunden
diff: /home/mib/git/martyrshot/liboqs/src/kem/saber/pqclean_lightsaber_aarch64: Datei oder Verzeichnis nicht gefunden
diff: /home/mib/git/martyrshot/liboqs/src/kem/saber/pqclean_saber_aarch64: Datei oder Verzeichnis nicht gefunden
diff: /home/mib/git/martyrshot/liboqs/src/kem/saber/pqclean_firesaber_aarch64: Datei oder Verzeichnis nicht gefunden
-----
Total schemes: 171 - 165 match upstream up to local patches, 6 differ
-----
Patches applied:
	pqclean-sphincs.patch
	pqclean-kyber-armneon-yml.patch
	pqclean-kyber-armneon-shake.patch
	pqcrystals-kyber-yml.patch
	pqcrystals-kyber-ref-shake.patch
	pqcrystals-kyber-avx2-shake.patch
	pqcrystals-dilithium-yml.patch
	pqcrystals-dilithium-ref-shake.patch
	pqcrystals-dilithium-avx2-shake.patch
-----
Schemes that differ from upstream:
Name: pqcrystals-kyber_kyber512_aarch64, expected upstream: https://github.com/pq-crystals/kyber.git - commit: faf5c3fe33e0b61c7c8a7888dd862bf5def17ad2
Name: pqcrystals-kyber_kyber768_aarch64, expected upstream: https://github.com/pq-crystals/kyber.git - commit: faf5c3fe33e0b61c7c8a7888dd862bf5def17ad2
Name: pqcrystals-kyber_kyber1024_aarch64, expected upstream: https://github.com/pq-crystals/kyber.git - commit: faf5c3fe33e0b61c7c8a7888dd862bf5def17ad2
Name: pqclean_lightsaber_aarch64, expected upstream: https://github.com/PQClean/PQClean.git - commit: 7eb978b4a733696bd7197278aa84216095674524
Name: pqclean_saber_aarch64, expected upstream: https://github.com/PQClean/PQClean.git - commit: 7eb978b4a733696bd7197278aa84216095674524
Name: pqclean_firesaber_aarch64, expected upstream: https://github.com/PQClean/PQClean.git - commit: 7eb978b4a733696bd7197278aa84216095674524
-----

Accordingly, an actual execution yields massive changes to the git repo's state:

~/git/martyrshot/liboqs/scripts/copy_from_upstream$ python3 copy_from_upstream.py copy
Updating format of Classic-McEliece-348864. Please double check ordering of yaml file
Updating format of HQC-128. Please double check ordering of yaml file
Updating format of NTRU-HPS-2048-509. Please double check ordering of yaml file
Updating format of ntrulpr653. Please double check ordering of yaml file
Updating format of LightSaber-KEM. Please double check ordering of yaml file
~/git/martyrshot/liboqs/scripts/copy_from_upstream$ git status
Auf Branch main
Ihr Branch ist auf demselben Stand wie 'origin/main'.

Änderungen, die nicht zum Commit vorgemerkt sind:
  (benutzen Sie "git add <Datei>...", um die Änderungen zum Commit vorzumerken)
  (benutzen Sie "git checkout -- <Datei>...", um die Änderungen im Arbeitsverzeichnis zu verwerfen)

	geändert:       ../../docs/algorithms/kem/bike.md
	geändert:       ../../docs/algorithms/kem/classic_mceliece.md
	geändert:       ../../docs/algorithms/kem/classic_mceliece.yml
	geändert:       ../../docs/algorithms/kem/hqc.yml
	geändert:       ../../docs/algorithms/kem/ntru.yml
	geändert:       ../../docs/algorithms/kem/ntruprime.yml
	geändert:       ../../docs/algorithms/kem/saber.yml
	geändert:       ../../src/kem/kyber/kem_kyber_1024.c
	geändert:       ../../src/kem/kyber/kem_kyber_512.c
	geändert:       ../../src/kem/kyber/kem_kyber_768.c
	geändert:       ../../src/kem/saber/CMakeLists.txt
	geändert:       ../../src/kem/saber/kem_saber_firesaber.c
	geändert:       ../../src/kem/saber/kem_saber_lightsaber.c
	geändert:       ../../src/kem/saber/kem_saber_saber.c

Unversionierte Dateien:
  (benutzen Sie "git add <Datei>...", um die Änderungen zum Commit vorzumerken)

	../../src/kem/saber/pqclean_firesaber_aarch64/
	../../src/kem/saber/pqclean_lightsaber_aarch64/
	../../src/kem/saber/pqclean_saber_aarch64/

keine Änderungen zum Commit vorgemerkt (benutzen Sie "git add" und/oder "git commit -a")

[baentsch]

@Martyrshot Thanks for the latest commits: They resolve the errors reported above.

However, the execution of copy_from_upstream.py copy on a "clean slate" 'martyrshort/liboqs/main' still causes git status to report many changed files; also the MD files are still not part of the PR.

In turn, when I do a test-commit on your latest changes (and after running python3 scripts/update_docs_from_yaml.py to create the MD files), everything behaves as I'd expect it: Further runs of these scripts don't yield changes to the repo's files, so I wonder whether all that's missing now is a git add -A on your side to complete the PR (?).

[Martyrshot]

I'm just fixing one issue where copy_from_upstream copies in implementations that are new to upstream but haven't been integrated into liboqs yet (the aarch64 implementations in pqclean for the sabre family). Once that's done then I think it's safe to do a git add -A. Shouldn't be too much longer. Sorry about the wait...

[Martyrshot]

I haven't removed the need for the second step (python3 scripts/update_docs_from_yaml.py) yet, but I need to run to a meeting. I think once that's done then this PR is ready to go. Sorry about all the problems.

[baentsch]

Sorry about all the problems.

No worries. To the contrary, I'm glad that faster ARM code becomes available and our automation logic becomes better.

[baentsch]

This is pretty operating system specific code. But then again I don't recall whether we ever run these scripts on Windows...

[Martyrshot]

We using subprocess.run (via shell()) in several places, so I think it's ok if this script is unix only?

[Martyrshot]

Assuming the arm64 test is some random docker issue (and all other tests pass) this pr should be ready for final review.

tests pass on my arm based mac, so it should be ok.

[dstebila]

We using subprocess.run (via shell()) in several places, so I think it's ok if this script is unix only?

We only need to be able to run the upstream code on Unix; preferably it would run on both Linux and macOS.

[Martyrshot]

We using subprocess.run (via shell()) in several places, so I think it's ok if this script is unix only?

We only need to be able to run the upstream code on Unix; preferably it would run on both Linux and macOS.

I just tested that the script runs both on macOS and linux. But I found one small issue I need to fix with ignore before merging.

[Martyrshot]

But I found one small issue I need to fix with ignore before merging.

I fixed this issue, and we should be good to go now.