Update NNUE architecture to SFNNv9 with larger L1 size of 3072 #5149

Viren6 · 2024-04-02T04:11:01Z

Part 1: PyTorch Training, @linrock

Trained with a 10-stage sequence from scratch, starting in May 2023:
https://github.com/linrock/nnue-tools/blob/master/exp-sequences/3072-10stage-SFNNv9.yml

While the training methods were similar to the L1-2560 training sequence, the last two stages introduced min-v2 binpacks, where bestmove capture and in-check position scores were not zeroed during minimization, for compatibility with skipping SEE >= 0 positions and future research.

Training data can be found at:
https://robotmoon.com/nnue-training-data

This net was tested at epoch 679 of the 10th training stage:
https://tests.stockfishchess.org/tests/view/65f32e460ec64f0526c48dbc

Part 2: SPSA Training, @Viren6

The net was then SPSA tuned. This consisted of the output weights (32 * 8) and biases (8) as well as the L3 biases (32 * 8) and L2 biases (16 * 8), totalling 648 params in total.

The SPSA tune can be found here:
https://tests.stockfishchess.org/tests/view/65fc33ba0ec64f0526c512e3

With the help of @Disservin , the initial weights were extracted with:
https://github.com/Viren6/Stockfish/tree/new228

The net was saved with the tuned weights using:
https://github.com/Viren6/Stockfish/tree/new241

Earlier nets of the SPSA failed STC compared to the base 3072 net of part 1:
https://tests.stockfishchess.org/tests/view/65ff356e0ec64f0526c53c98
Therefore it is suspected that the SPSA at VVLTC has added extra scaling on top of the scaling of increasing the L1 size.

Passed VVLTC 1:
https://tests.stockfishchess.org/tests/view/6604a9020ec64f0526c583da
LLR: 2.94 (-2.94,2.94) <0.00,2.00>
Total: 53042 W: 13554 L: 13256 D: 26232
Ptnml(0-2): 12, 5147, 15903, 5449, 10

Passed VVLTC 2:
https://tests.stockfishchess.org/tests/view/660ad1b60ec64f0526c5dd23
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 17506 W: 4574 L: 4315 D: 8617
Ptnml(0-2): 1, 1567, 5362, 1818, 5

STC Elo estimate:
https://tests.stockfishchess.org/tests/view/660b834d01aaec5069f87cb0

bench 1823302

bench 1823302 Co-Authored-By: Linmiao Xu <lin@robotmoon.com>

vondele · 2024-04-02T06:57:04Z

merged as 0716b84

@Viren6

Created by first retraining the spsa-tuned master net `nn-ae6a388e4a1a.nnue` with: - using v6-dd data without bestmove captures removed - addition of T80 mar2024 data - increasing loss by 20% when Q is too high - torch.compile changes for marginal training speed gains And then SPSA tuning weights of epoch 899 following methods described in: official-stockfish#5149 This net was reached at 92k out of 120k steps in this 70+0.7 th 7 SPSA tuning run: https://tests.stockfishchess.org/tests/view/66413b7df9f4e8fc783c9bbb Thanks to @Viren6 for suggesting usage of: - c value 4 for the weights - c value 128 for the biases Scripts for automating applying fishtest spsa params to exporting tuned .nnue are in: https://github.com/linrock/nnue-tools/tree/master/spsa Before spsa tuning, epoch 899 was nn-f85738aefa84.nnue https://tests.stockfishchess.org/tests/view/663e5c893a2f9702074bc167 After initially training with max-epoch 800, training was resumed with max-epoch 1000. ``` experiment-name: 3072--S11--more-data-v6-dd-t80-mar2024--see-ge0-20p-more-loss-high-q-sk28-l8 nnue-pytorch-branch: linrock/nnue-pytorch/3072-r21-skip-more-wdl-see-ge0-20p-more-loss-high-q-torch-compile-more start-from-engine-test-net: False start-from-model: /data/config/apr2024-3072/nn-ae6a388e4a1a.nnue early-fen-skipping: 28 training-dataset: /data/S11-mar2024/: - leela96.v2.min.binpack - test60-2021-11-12-novdec-12tb7p.v6-dd.min.binpack - test78-2022-01-to-05-jantomay-16tb7p.v6-dd.min.binpack - test80-2022-06-jun-16tb7p.v6-dd.min.binpack - test80-2022-08-aug-16tb7p.v6-dd.min.binpack - test80-2022-09-sep-16tb7p.v6-dd.min.binpack - test80-2023-01-jan-16tb7p.v6-sk20.min.binpack - test80-2023-02-feb-16tb7p.v6-sk20.min.binpack - test80-2023-03-mar-2tb7p.v6-sk16.min.binpack - test80-2023-04-apr-2tb7p.v6-sk16.min.binpack - test80-2023-05-may-2tb7p.v6.min.binpack # official-stockfish#4782 - test80-2023-06-jun-2tb7p.binpack - test80-2023-07-jul-2tb7p.binpack # official-stockfish#4972 - test80-2023-08-aug-2tb7p.v6.min.binpack - test80-2023-09-sep-2tb7p.binpack - test80-2023-10-oct-2tb7p.binpack # S9 new data: official-stockfish#5056 - test80-2023-11-nov-2tb7p.binpack - test80-2023-12-dec-2tb7p.binpack # S10 new data: official-stockfish#5149 - test80-2024-01-jan-2tb7p.binpack - test80-2024-02-feb-2tb7p.binpack # S11 new data - test80-2024-03-mar-2tb7p.binpack /data/filt-v6-dd/: - test77-dec2021-16tb7p-filter-v6-dd.binpack - test78-juntosep2022-16tb7p-filter-v6-dd.binpack - test79-apr2022-16tb7p-filter-v6-dd.binpack - test79-may2022-16tb7p-filter-v6-dd.binpack - test80-jul2022-16tb7p-filter-v6-dd.binpack - test80-oct2022-16tb7p-filter-v6-dd.binpack - test80-nov2022-16tb7p-filter-v6-dd.binpack num-epochs: 1000 lr: 4.375e-4 gamma: 0.995 start-lambda: 0.8 end-lambda: 0.7 ``` Training data can be found at: https://robotmoon.com/nnue-training-data/ Local elo at 25k nodes per move: nn-epoch899.nnue : 4.6 +/- 1.4 Passed STC: https://tests.stockfishchess.org/tests/view/6645454893ce6da3e93b31ae LLR: 2.95 (-2.94,2.94) <0.00,2.00> Total: 95232 W: 24598 L: 24194 D: 46440 Ptnml(0-2): 294, 11215, 24180, 11647, 280 Passed LTC: https://tests.stockfishchess.org/tests/view/6645522d93ce6da3e93b31df LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 320544 W: 81432 L: 80524 D: 158588 Ptnml(0-2): 164, 35659, 87696, 36611, 142 bench 1955748

@Viren6

Created by first retraining the spsa-tuned main net `nn-ae6a388e4a1a.nnue` with: - using v6-dd data without bestmove captures removed - addition of T80 mar2024 data - increasing loss by 20% when Q is too high - torch.compile changes for marginal training speed gains And then SPSA tuning weights of epoch 899 following methods described in: official-stockfish#5149 This net was reached at 92k out of 120k steps in this 70+0.7 th 7 SPSA tuning run: https://tests.stockfishchess.org/tests/view/66413b7df9f4e8fc783c9bbb Thanks to @Viren6 for suggesting usage of: - c value 4 for the weights - c value 128 for the biases Scripts for automating applying fishtest spsa params to exporting tuned .nnue are in: https://github.com/linrock/nnue-tools/tree/master/spsa Before spsa tuning, epoch 899 was nn-f85738aefa84.nnue https://tests.stockfishchess.org/tests/view/663e5c893a2f9702074bc167 After initially training with max-epoch 800, training was resumed with max-epoch 1000. ``` experiment-name: 3072--S11--more-data-v6-dd-t80-mar2024--see-ge0-20p-more-loss-high-q-sk28-l8 nnue-pytorch-branch: linrock/nnue-pytorch/3072-r21-skip-more-wdl-see-ge0-20p-more-loss-high-q-torch-compile-more start-from-engine-test-net: False start-from-model: /data/config/apr2024-3072/nn-ae6a388e4a1a.nnue early-fen-skipping: 28 training-dataset: /data/S11-mar2024/: - leela96.v2.min.binpack - test60-2021-11-12-novdec-12tb7p.v6-dd.min.binpack - test78-2022-01-to-05-jantomay-16tb7p.v6-dd.min.binpack - test80-2022-06-jun-16tb7p.v6-dd.min.binpack - test80-2022-08-aug-16tb7p.v6-dd.min.binpack - test80-2022-09-sep-16tb7p.v6-dd.min.binpack - test80-2023-01-jan-16tb7p.v6-sk20.min.binpack - test80-2023-02-feb-16tb7p.v6-sk20.min.binpack - test80-2023-03-mar-2tb7p.v6-sk16.min.binpack - test80-2023-04-apr-2tb7p.v6-sk16.min.binpack - test80-2023-05-may-2tb7p.v6.min.binpack # official-stockfish#4782 - test80-2023-06-jun-2tb7p.binpack - test80-2023-07-jul-2tb7p.binpack # official-stockfish#4972 - test80-2023-08-aug-2tb7p.v6.min.binpack - test80-2023-09-sep-2tb7p.binpack - test80-2023-10-oct-2tb7p.binpack # S9 new data: official-stockfish#5056 - test80-2023-11-nov-2tb7p.binpack - test80-2023-12-dec-2tb7p.binpack # S10 new data: official-stockfish#5149 - test80-2024-01-jan-2tb7p.binpack - test80-2024-02-feb-2tb7p.binpack # S11 new data - test80-2024-03-mar-2tb7p.binpack /data/filt-v6-dd/: - test77-dec2021-16tb7p-filter-v6-dd.binpack - test78-juntosep2022-16tb7p-filter-v6-dd.binpack - test79-apr2022-16tb7p-filter-v6-dd.binpack - test79-may2022-16tb7p-filter-v6-dd.binpack - test80-jul2022-16tb7p-filter-v6-dd.binpack - test80-oct2022-16tb7p-filter-v6-dd.binpack - test80-nov2022-16tb7p-filter-v6-dd.binpack num-epochs: 1000 lr: 4.375e-4 gamma: 0.995 start-lambda: 0.8 end-lambda: 0.7 ``` Training data can be found at: https://robotmoon.com/nnue-training-data/ Local elo at 25k nodes per move: nn-epoch899.nnue : 4.6 +/- 1.4 Passed STC: https://tests.stockfishchess.org/tests/view/6645454893ce6da3e93b31ae LLR: 2.95 (-2.94,2.94) <0.00,2.00> Total: 95232 W: 24598 L: 24194 D: 46440 Ptnml(0-2): 294, 11215, 24180, 11647, 280 Passed LTC: https://tests.stockfishchess.org/tests/view/6645522d93ce6da3e93b31df LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 320544 W: 81432 L: 80524 D: 158588 Ptnml(0-2): 164, 35659, 87696, 36611, 142 bench 1955748

@Viren6

Created by first retraining the spsa-tuned main net `nn-ae6a388e4a1a.nnue` with: - using v6-dd data without bestmove captures removed - addition of T80 mar2024 data - increasing loss by 20% when Q is too high - torch.compile changes for marginal training speed gains And then SPSA tuning weights of epoch 899 following methods described in: official-stockfish#5149 This net was reached at 92k out of 120k steps in this 70+0.7 th 7 SPSA tuning run: https://tests.stockfishchess.org/tests/view/66413b7df9f4e8fc783c9bbb Thanks to @Viren6 for suggesting usage of: - c value 4 for the weights - c value 128 for the biases Scripts for automating applying fishtest spsa params to exporting tuned .nnue are in: https://github.com/linrock/nnue-tools/tree/master/spsa Before spsa tuning, epoch 899 was nn-f85738aefa84.nnue https://tests.stockfishchess.org/tests/view/663e5c893a2f9702074bc167 After initially training with max-epoch 800, training was resumed with max-epoch 1000. ``` experiment-name: 3072--S11--more-data-v6-dd-t80-mar2024--see-ge0-20p-more-loss-high-q-sk28-l8 nnue-pytorch-branch: linrock/nnue-pytorch/3072-r21-skip-more-wdl-see-ge0-20p-more-loss-high-q-torch-compile-more start-from-engine-test-net: False start-from-model: /data/config/apr2024-3072/nn-ae6a388e4a1a.nnue early-fen-skipping: 28 training-dataset: /data/S11-mar2024/: - leela96.v2.min.binpack - test60-2021-11-12-novdec-12tb7p.v6-dd.min.binpack - test78-2022-01-to-05-jantomay-16tb7p.v6-dd.min.binpack - test80-2022-06-jun-16tb7p.v6-dd.min.binpack - test80-2022-08-aug-16tb7p.v6-dd.min.binpack - test80-2022-09-sep-16tb7p.v6-dd.min.binpack - test80-2023-01-jan-16tb7p.v6-sk20.min.binpack - test80-2023-02-feb-16tb7p.v6-sk20.min.binpack - test80-2023-03-mar-2tb7p.v6-sk16.min.binpack - test80-2023-04-apr-2tb7p.v6-sk16.min.binpack - test80-2023-05-may-2tb7p.v6.min.binpack # official-stockfish#4782 - test80-2023-06-jun-2tb7p.binpack - test80-2023-07-jul-2tb7p.binpack # official-stockfish#4972 - test80-2023-08-aug-2tb7p.v6.min.binpack - test80-2023-09-sep-2tb7p.binpack - test80-2023-10-oct-2tb7p.binpack # S9 new data: official-stockfish#5056 - test80-2023-11-nov-2tb7p.binpack - test80-2023-12-dec-2tb7p.binpack # S10 new data: official-stockfish#5149 - test80-2024-01-jan-2tb7p.binpack - test80-2024-02-feb-2tb7p.binpack # S11 new data - test80-2024-03-mar-2tb7p.binpack /data/filt-v6-dd/: - test77-dec2021-16tb7p-filter-v6-dd.binpack - test78-juntosep2022-16tb7p-filter-v6-dd.binpack - test79-apr2022-16tb7p-filter-v6-dd.binpack - test79-may2022-16tb7p-filter-v6-dd.binpack - test80-jul2022-16tb7p-filter-v6-dd.binpack - test80-oct2022-16tb7p-filter-v6-dd.binpack - test80-nov2022-16tb7p-filter-v6-dd.binpack num-epochs: 1000 lr: 4.375e-4 gamma: 0.995 start-lambda: 0.8 end-lambda: 0.7 ``` Training data can be found at: https://robotmoon.com/nnue-training-data/ Local elo at 25k nodes per move: nn-epoch899.nnue : 4.6 +/- 1.4 Passed STC: https://tests.stockfishchess.org/tests/view/6645454893ce6da3e93b31ae LLR: 2.95 (-2.94,2.94) <0.00,2.00> Total: 95232 W: 24598 L: 24194 D: 46440 Ptnml(0-2): 294, 11215, 24180, 11647, 280 Passed LTC: https://tests.stockfishchess.org/tests/view/6645522d93ce6da3e93b31df LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 320544 W: 81432 L: 80524 D: 158588 Ptnml(0-2): 164, 35659, 87696, 36611, 142 bench 1995552

@Viren6

Created by first retraining the spsa-tuned main net `nn-ae6a388e4a1a.nnue` with: - using v6-dd data without bestmove captures removed - addition of T80 mar2024 data - increasing loss by 20% when Q is too high - torch.compile changes for marginal training speed gains And then SPSA tuning weights of epoch 899 following methods described in: official-stockfish#5149 This net was reached at 92k out of 120k steps in this 70+0.7 th 7 SPSA tuning run: https://tests.stockfishchess.org/tests/view/66413b7df9f4e8fc783c9bbb Thanks to @Viren6 for suggesting usage of: - c value 4 for the weights - c value 128 for the biases Scripts for automating applying fishtest spsa params to exporting tuned .nnue are in: https://github.com/linrock/nnue-tools/tree/master/spsa Before spsa tuning, epoch 899 was nn-f85738aefa84.nnue https://tests.stockfishchess.org/tests/view/663e5c893a2f9702074bc167 After initially training with max-epoch 800, training was resumed with max-epoch 1000. ``` experiment-name: 3072--S11--more-data-v6-dd-t80-mar2024--see-ge0-20p-more-loss-high-q-sk28-l8 nnue-pytorch-branch: linrock/nnue-pytorch/3072-r21-skip-more-wdl-see-ge0-20p-more-loss-high-q-torch-compile-more start-from-engine-test-net: False start-from-model: /data/config/apr2024-3072/nn-ae6a388e4a1a.nnue early-fen-skipping: 28 training-dataset: /data/S11-mar2024/: - leela96.v2.min.binpack - test60-2021-11-12-novdec-12tb7p.v6-dd.min.binpack - test78-2022-01-to-05-jantomay-16tb7p.v6-dd.min.binpack - test80-2022-06-jun-16tb7p.v6-dd.min.binpack - test80-2022-08-aug-16tb7p.v6-dd.min.binpack - test80-2022-09-sep-16tb7p.v6-dd.min.binpack - test80-2023-01-jan-16tb7p.v6-sk20.min.binpack - test80-2023-02-feb-16tb7p.v6-sk20.min.binpack - test80-2023-03-mar-2tb7p.v6-sk16.min.binpack - test80-2023-04-apr-2tb7p.v6-sk16.min.binpack - test80-2023-05-may-2tb7p.v6.min.binpack # official-stockfish#4782 - test80-2023-06-jun-2tb7p.binpack - test80-2023-07-jul-2tb7p.binpack # official-stockfish#4972 - test80-2023-08-aug-2tb7p.v6.min.binpack - test80-2023-09-sep-2tb7p.binpack - test80-2023-10-oct-2tb7p.binpack # S9 new data: official-stockfish#5056 - test80-2023-11-nov-2tb7p.binpack - test80-2023-12-dec-2tb7p.binpack # S10 new data: official-stockfish#5149 - test80-2024-01-jan-2tb7p.binpack - test80-2024-02-feb-2tb7p.binpack # S11 new data - test80-2024-03-mar-2tb7p.binpack /data/filt-v6-dd/: - test77-dec2021-16tb7p-filter-v6-dd.binpack - test78-juntosep2022-16tb7p-filter-v6-dd.binpack - test79-apr2022-16tb7p-filter-v6-dd.binpack - test79-may2022-16tb7p-filter-v6-dd.binpack - test80-jul2022-16tb7p-filter-v6-dd.binpack - test80-oct2022-16tb7p-filter-v6-dd.binpack - test80-nov2022-16tb7p-filter-v6-dd.binpack num-epochs: 1000 lr: 4.375e-4 gamma: 0.995 start-lambda: 0.8 end-lambda: 0.7 ``` Training data can be found at: https://robotmoon.com/nnue-training-data/ Local elo at 25k nodes per move: nn-epoch899.nnue : 4.6 +/- 1.4 Passed STC: https://tests.stockfishchess.org/tests/view/6645454893ce6da3e93b31ae LLR: 2.95 (-2.94,2.94) <0.00,2.00> Total: 95232 W: 24598 L: 24194 D: 46440 Ptnml(0-2): 294, 11215, 24180, 11647, 280 Passed LTC: https://tests.stockfishchess.org/tests/view/6645522d93ce6da3e93b31df LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 320544 W: 81432 L: 80524 D: 158588 Ptnml(0-2): 164, 35659, 87696, 36611, 142 closes official-stockfish#5254 bench 1995552

Created by setting output weights (256) and biases (8) of the previous main net nn-ddcfb9224cdb.nnue to values found around 12k / 120k spsa games at 120+1.2 This used modified fishtest dev workers to construct .nnue files from spsa params, then load them with EvalFile when running tests: https://github.com/linrock/fishtest/tree/spsa-file-modified-nnue/worker Inspired by researching loading spsa params from files: official-stockfish/fishtest#1926 Scripts for modifying nnue files and preparing params: https://github.com/linrock/nnue-pytorch/tree/no-gpu-modify-nnue spsa params: weights: [-127, 127], c_end = 6 biases: [-8192, 8192], c_end = 64 Example of reading output weights and biases from the previous main net using nnue-pytorch and printing spsa params in a format compatible with fishtest: ``` import features from serialize import NNUEReader feature_set = features.get_feature_set_from_name("HalfKAv2_hm") with open("nn-ddcfb9224cdb.nnue", "rb") as f: model = NNUEReader(f, feature_set).model c_end_weights = 6 c_end_biases = 64 for i in range(8): for j in range(32): value = round(int(model.layer_stacks.output.weight[i, j] * 600 * 16) / 127) print(f"oW[{i}][{j}],{value},-127,127,{c_end_weights},0.0020") for i in range(8): value = int(model.layer_stacks.output.bias[i] * 600 * 16) print(f"oB[{i}],{value},-8192,8192,{c_end_biases},0.0020") ``` For more info on spsa tuning params in nets: official-stockfish#5149 official-stockfish#5254 Passed STC: https://tests.stockfishchess.org/tests/view/66894d64e59d990b103f8a37 LLR: 2.94 (-2.94,2.94) <0.00,2.00> Total: 32000 W: 8443 L: 8137 D: 15420 Ptnml(0-2): 80, 3627, 8309, 3875, 109 Passed LTC: https://tests.stockfishchess.org/tests/view/6689668ce59d990b103f8b8b LLR: 2.94 (-2.94,2.94) <0.50,2.50> Total: 172176 W: 43822 L: 43225 D: 85129 Ptnml(0-2): 97, 18821, 47633, 19462, 75 bench 993416

Created by setting output weights (256) and biases (8) of the previous main net nn-ddcfb9224cdb.nnue to values found around 12k / 120k spsa games at 120+1.2 This used modified fishtest dev workers to construct .nnue files from spsa params, then load them with EvalFile when running tests: https://github.com/linrock/fishtest/tree/spsa-file-modified-nnue/worker Inspired by researching loading spsa params from files: official-stockfish/fishtest#1926 Scripts for modifying nnue files and preparing params: https://github.com/linrock/nnue-pytorch/tree/no-gpu-modify-nnue spsa params: weights: [-127, 127], c_end = 6 biases: [-8192, 8192], c_end = 64 Example of reading output weights and biases from the previous main net using nnue-pytorch and printing spsa params in a format compatible with fishtest: ``` import features from serialize import NNUEReader feature_set = features.get_feature_set_from_name("HalfKAv2_hm") with open("nn-ddcfb9224cdb.nnue", "rb") as f: model = NNUEReader(f, feature_set).model c_end_weights = 6 c_end_biases = 64 for i in range(8): for j in range(32): value = round(int(model.layer_stacks.output.weight[i, j] * 600 * 16) / 127) print(f"oW[{i}][{j}],{value},-127,127,{c_end_weights},0.0020") for i in range(8): value = int(model.layer_stacks.output.bias[i] * 600 * 16) print(f"oB[{i}],{value},-8192,8192,{c_end_biases},0.0020") ``` For more info on spsa tuning params in nets: official-stockfish#5149 official-stockfish#5254 Passed STC: https://tests.stockfishchess.org/tests/view/66894d64e59d990b103f8a37 LLR: 2.94 (-2.94,2.94) <0.00,2.00> Total: 32000 W: 8443 L: 8137 D: 15420 Ptnml(0-2): 80, 3627, 8309, 3875, 109 Passed LTC: https://tests.stockfishchess.org/tests/view/6689668ce59d990b103f8b8b LLR: 2.94 (-2.94,2.94) <0.50,2.50> Total: 172176 W: 43822 L: 43225 D: 85129 Ptnml(0-2): 97, 18821, 47633, 19462, 75 closes official-stockfish#5459 bench 1120091

L1 3072

b3860ff

bench 1823302 Co-Authored-By: Linmiao Xu <lin@robotmoon.com>

vondele closed this Apr 2, 2024

vondele added to be merged Will be merged shortly 🚀 gainer Gains elo labels Apr 2, 2024

Viren6 mentioned this pull request Apr 2, 2024

Param limit of SPSA limiting possible tunes official-stockfish/fishtest#1922

Open

linrock mentioned this pull request Apr 30, 2024

Increase L1 size to 3072 official-stockfish/nnue-pytorch#280

Merged

linrock mentioned this pull request May 17, 2024

Update default main net to nn-c721dfca8cd3.nnue #5254

Closed

linrock mentioned this pull request Jul 6, 2024

Update default main net to nn-74f1d263ae9a.nnue #5459

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update NNUE architecture to SFNNv9 with larger L1 size of 3072 #5149

Update NNUE architecture to SFNNv9 with larger L1 size of 3072 #5149

Viren6 commented Apr 2, 2024 •

edited

Loading

vondele commented Apr 2, 2024

Update NNUE architecture to SFNNv9 with larger L1 size of 3072 #5149

Update NNUE architecture to SFNNv9 with larger L1 size of 3072 #5149

Conversation

Viren6 commented Apr 2, 2024 • edited Loading

Part 1: PyTorch Training, @linrock

Part 2: SPSA Training, @Viren6

vondele commented Apr 2, 2024

Viren6 commented Apr 2, 2024 •

edited

Loading