Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update default net to nn-c38c3d8d3920.nnue #4666

Closed

Conversation

linrock
Copy link
Contributor

@linrock linrock commented Jul 6, 2023

This was a later epoch from the same experiment that led to the previous master net.
After training, it was prepared the same way:

  1. greedy permuting L1 weights with Permute master net weights to increase sparsity #4620
  2. leb128 compression with Support for LEB128 compression of feature transformer parameters. nnue-pytorch#251
  3. greedy 2- and 3- cycle permuting with Update default data to nn-a3d1bfca1672.nnue #4640

Local elo at 25k nodes per move (vs. L1-1536 nn-fdc1d0fe6455.nnue):
nn-epoch739.nnue : 20.2 +/- 1.7

Passed STC:
https://tests.stockfishchess.org/tests/view/64a050b33ee09aa549c4e4c8
LLR: 2.95 (-2.94,2.94) <0.00,2.00>
Total: 195552 W: 49977 L: 49430 D: 96145
Ptnml(0-2): 556, 22775, 50607, 23242, 596

Passed LTC:
https://tests.stockfishchess.org/tests/view/64a127bd3ee09aa549c4f60c
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 235452 W: 60327 L: 59609 D: 115516
Ptnml(0-2): 119, 25173, 66426, 25887, 121

bench 2427629

This was a later epoch from the same experiment that led to the
previous master net. After training, it was prepared the same way:

1. greedy permuting L1 weights with official-stockfish#4620
2. leb128 compression with official-stockfish/nnue-pytorch#251
3. greedy 2- and 3- cycle permuting with official-stockfish#4640

Local elo at 25k nodes per move (vs. L1-1536 nn-fdc1d0fe6455.nnue):
nn-epoch739.nnue : 20.2 +/- 1.7

Passed STC:
https://tests.stockfishchess.org/tests/view/64a050b33ee09aa549c4e4c8
LLR: 2.95 (-2.94,2.94) <0.00,2.00>
Total: 195552 W: 49977 L: 49430 D: 96145
Ptnml(0-2): 556, 22775, 50607, 23242, 596

Passed LTC:
https://tests.stockfishchess.org/tests/view/64a127bd3ee09aa549c4f60c
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 235452 W: 60327 L: 59609 D: 115516
Ptnml(0-2): 119, 25173, 66426, 25887, 121

bench 2427629
@vondele
Copy link
Member

vondele commented Jul 6, 2023

hmm for whatever reason the CI is failing again on MacOS.

I wonder if something is non-standard on our regexp that it fails to match bench 2427629 on MacOS ?

@Sopel97
Copy link
Member

Sopel97 commented Jul 6, 2023

have you tried skipping the first step and only relying on "greedy 2- and 3- cycle permuting with #4640" ?

official-stockfish/nnue-pytorch#254

@linrock
Copy link
Contributor Author

linrock commented Jul 6, 2023

have you tried skipping the first step and only relying on "greedy 2- and 3- cycle permuting with #4640" ?

i have, it completely froze up on a few occasions and i don't know why. meanwhile, i haven't run into issues running the 3 steps sequentially other than it being resource intensive and taking about an hour per net.

i can give official-stockfish/nnue-pytorch#254 a try later.

@ppigazzini
Copy link
Contributor

hmm for whatever reason the CI is failing again on MacOS.

I wonder if something is non-standard on our regexp that it fails to match bench 2427629 on MacOS ?

Ubuntu extract the same bench

@vondele
Copy link
Member

vondele commented Jul 6, 2023

yeah, we discussing on discord.
It is the order of commits as extracted with the loop that is unexpected

@ppigazzini
Copy link
Contributor

In local test with this branch the script works fine:
bash --noprofile --norc -eo pipefail ../bench.sh
benchref=2427629
From commit: 9f4cd3b
Reference bench: 2427629

@vondele
Copy link
Member

vondele commented Jul 6, 2023

it might be related to how this branch is merged in master during the CI ?

@vondele
Copy link
Member

vondele commented Jul 6, 2023

look at the additional output in https://github.com/official-stockfish/Stockfish/actions/runs/5478194778/jobs/9978344031
(under Extract bench .... step)

@vondele vondele added the to be merged Will be merged shortly label Jul 6, 2023
@vondele vondele closed this in e699fee Jul 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
to be merged Will be merged shortly
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants