OpenCL Major Speedup, Defaults, GTP and Other Fixes

This release should be a significant OpenCL performance improvement for users without NVIDIA tensor core GPUs - namely anything less top-end than an RTX 20xx card or similar. For NVIDIA tensor-core-supporting GPUs, the CUDA version is likely to still be faster though. Also, many other fixes and a few missing features have been added.

NOTE: The new OpenCL implementation will need to re-tune itself again the first time you start this new version, so be patient on the first new startup and/or run it in the console the first time.

If you're a new user, don't forget to check out this section for getting started and basic usage.

New Neural Nets! Yay!

• g170-b20c256x2-s1913382912-d435450331 ("g170 20 block s1.91G") - A new 20 block net that is yet another 115 Elo (+/- 30) stronger than the previous net. This should be the new strongest KataGo net!

• g170e-b15c192-s1672170752-d466197061 ("g170e 15 block s1.67G") - This 15 block net is probably the last extended-training 15 block net that KataGo will be producing. It is probably about 20-50 Elo stronger than the previous one, which might put it about on par with ELF OpenGo v2 at equal playouts, for high hundreds or low thousands of playouts. Making it a very strong net, given that it is only 15 x 192 in size, and hopefully ideal for weaker to moderate-level hardware.

These are attached below. For all other currently-released g170 nets, they are here: https://d3dndmfyhecmj0.cloudfront.net/g170/neuralnets/index.html

Notable Changes in This Release

• Much improved xgemm implementation for OpenCL version - overall OpenCL performance should be improved by 10%-50%, depending on your hardware and threads!

• All options in the GPU-related sections of the GTP config are also now optional and have better defaults. KataGo will automatically choose a batch size, and on the CUDA version it will automatically detect what flavor of GPU you have and enable or disable FP16 accordingly. Multiple GPUs will not be used automatically however - if you want to let KataGo use a larger cache to be a little faster or to have it use multiple GPUs, or run into problems with the automatic FP16 choice - you can still override the defaults.

• Benchmark's thread suggestion greatly improved (./katago benchmark -config GTP_CONFIG.cfg -model MODEL.txt.gz), based on some new test data. The old version was a too conservative particularly on very strong machines - the new one will be a bit more aggressive about recommending larger numbers of threads.

• GTP commands final_status_list and final_score will now use KataGo's neural net to guess an evaluation of the position if invoked when the game is not over or not fully cleaned up. For Japanese rules games, such as if you're running it on KGS - this should make KataGo now able to score and mark dead stones in all common cases (I think)! The heuristics here may still be a bit rough however and could possibly behave weirdly in certain sekis, or there may be more basic issues since I haven't specifically gotten set up to test on KGS, so let me know if you run into issues.

• GTP command fixed_handicap is now supported in KataGo.

• A few new options for users running selfplay training - can now terminate train.py after a fixed number of epochs, can now terminate gatekeeper once it's done passing a net, can now disable autoreject of old nets.

EDIT: Reverted the automatic use of FP16 on the CUDA version for Pascal-architecture NVIDIA GPUs, when cudaUseFP16=auto. FP16 is a mild performance boost for many of these GPUs, but on some setups there might be a chance of just a precision loss for little gain, and also maybe could cause issues for some users. If you have a recentish but non-tensor-core GPU, you can try setting cudaUseFP16=true instead of cudaUseFP16=auto in your gtp config and benchmark it.
EDIT: And fixed some additional bugs in the GTP protocol regarding races between pondering and other commands, and some long-standing issues with handicap stone handling. Additionally, KataGo will now tolerate handicap being placed by alternating black moves and white passes at the start of a game.

Some More Neural Nets

This is an upload of some stronger neural nets for KataGo... but they are obsoleted by v1.3.2
For the latest released version of the code or engine and stronger neural nets, see: https://github.com/lightvector/KataGo/releases/tag/v1.3.2

Available are (click "Assets" below):

• g170-b20c256x2-s1420141824-d350969033.txt.gz ("g170 20 block s1.42G") - A new 20 block net that is possibly another 100 Elo (+/- 30) stronger than the net released with v1.3.1. This should be the strongest released KataGo net so far!

• g170e-b15c192-s1305382144-d335919935.txt.gz ("g170e 15 block s1.31G") - A very strong extended-training 15-block net, learning from games played by later 20-block nets. This net is perhaps 350 Elo stronger than the selfplay 15-block net bundled with the v1.3 release, which might put it at almost as strong as LZ-ELF OpenGo v2 at equal visits or playouts for small numbers (e.g. around 1000), despite only being 15x192 rather than 20x256. If you're on weaker hardware and prefer a somewhat faster search speed, this might be a nice net to use. Like all v1.3+ nets, it supports all the usual features - board sizes, komi, JP rules, handicap play, etc.

This "release" may be modified directly as the run progresses and more neural nets are available. If you would like to see the full set of publicly released KataGo nets or data so far, see https://d3dndmfyhecmj0.cloudfront.net/g170/neuralnets/index.html or more broadly https://d3dndmfyhecmj0.cloudfront.net/.

Enjoy!

Better Parameters and Defaults, Bugfixes, Minor GTP stuff

For stronger neural nets and newer code, see this later release!

This is a quick followup release to the major changes in v1.3. It fixes some bugs and and improves a few things.

If you are upgrading from a much older version of KataGo, you probably want to skim over the release notes for v1.3 as well!

Changes

• You can now delete or comment out the entire bottom half of your GTP config! You can remove everything below Root move selection and biases including all the Internal params. All of these parameters will now use good defaults if not specified. Deleting or commenting them out is recommended so that you pick up any future improvements to these parameters automatically, rather than manually having to update your config. (Except for any specific values that you are deliberately adjusting or experimenting with, of course).

• Along with the above note, cpuctExplorationLog has been adjusted to 0.4. Based on a few thousand test games, the old value of 0.6 was likely a little large, the newer value should be slightly stronger at a variety of numbers of playouts. If you are using a v1.3 or older GTP config that hardcodes 0.6 or another value, this change will get picked up automatically if you delete the bottom half of the config as recommended above, or you can manually adjust this value yourself, otherwise you will NOT get this change.

• Implements lz-genmove_analyze and kata-genmove_analyze GTP extensions.

• Improves FP16 performance on Pascal-architecture NVIDIA GPUs (hopefully).

• Fixes a bug in lead estimation training that could cause KataGo to sometimes greatly underestimate the lead if the lead is more extreme than about 70 points, and sometimes cap out around there (such as still reporting 70 when actually the true lead is 100+ points). Earlier g170 neural nets may continue to exhibit this bug in some cases (possibly not consistently), but ongoing nets such as the newer one attached here should be good.

• Some other very minor fixes and improvements.

New Net

For stronger neural nets than this, see this later release!
Attached here is a newer net from the ongoing run ("g170 20 block s1.04G"). It should be about 120 Elo stronger than the strongest previous net. This net will also work with the prior v1.3 release - the changes in this release are independent changes with no effects on compatibility of KataGo nets.

See the v1.3 release for some more neural nets, or here for all nets from this run released so far.

Have fun!

New Run, JP Rules, Handicap play, Lead Estimation...

Note: newer version v1.3.2 is out, improving and fixing many things on top of this release!

For stronger neural nets, see also the same later release!

New Run!

KataGo is on a fresh new major run ("g170"). Due to some more improvements, it is training faster and better. Fairly recently it caught up to the strength of the run from June, so it is not actually much stronger yet, but it seems like catching up is a good time for a release, because there are a lot of new features. The main page notes have also been updated.

• Japanese-like rules, stone-counting rules and many other combinations of rules. These can be can be configured in the gtp config as usual, and for developers, they can also be set dynamically using a few new GTP extensions. Note regarding Japanese-like rules: KataGo will usually fill the dame, but might not do so 100% of the time, since under its ruleset this is not strictly required - but for game analysis purposes, it should not matter much. There's some chance it also occasionally gives up a point by playing an unnecessary defense, if it is winning by enough and wants to be safe. Lastly, some kinds of seki positions or double-ko may still confuse it a little - getting all the details right is quite hard! But hopefully in the common case KataGo performs well.
• Handicap game "PDA" - ("playoutDoublingAdvantage") - a new option to configure KataGo to play much more aggressively (or even overplay a little) in handicap games, for hopefully much stronger handicap play. This is the result of some special training using unbalanced playouts, so that KataGo learns how to fight weaker versions of itself (while the weaker versions simultaneously learn how to play safely and resist).
• Much better score estimation - KataGo used to report an estimate of the selfplay average score - the average score that it would achieve if it self-played from a position. So for example, if black were to get an extra stone for free then older KataGo nets might report for example +20, even though the stone should be worth only fair_komi * 2 ~= 14 points. Why? Because KataGo correctly predicted that in self-play, the side that was behind might give up 6 more points on average over the entire rest of the game by taking additional risks to try to swing the game back. These new neural nets are also now trained to estimate the lead - the number of points needed to make the game fair, in this case 14. By default now, KataGo with the latest neural nets will show the estimated lead rather than the estimated selfplay average score - which should be much less confusing.
• Log-scaling cPUCT - KataGo used to explore too little at large numbers of visits. This change should make KataGo quite a bit stronger at high visits. The differences mostly start to kick in around 3000+ playouts, or 6000+ visits. At around 20000 visits this change maybe might start to be worth close to +100 Elo, including even for older nets. Note: old GTP configs will NOT get this by default, see notes about upgrading below.

Some new things for developers:

• Analysis engine - ./katago analysis -model <NEURALNET>.txt.gz -config <ANALYSIS_CONFIG>.cfg -analysis-threads N - a new batched analysis engine that uses a JSON protocol that might be much more efficient for many use cases than GTP. Documentation here. (NOTE: if you were already using this engine pre-official-release, the API has changed a little in a non-compatible way, please check the docs for the protocol)

• Tensorflow 1.5 and Multi-GPU training - KataGo has upgraded the training scripts to use TF 1.5 and support multi-GPU now. Specify something like -multi-gpus 0,1 to train.sh or to train.py. There are also some minor changes to one or two of the training-related scripts - see the main page readme for details.

• Other Training Changes - By default, KataGo does not use batch norm any more, but you can turn it back on if you like (in python/modelconfigs.py). There are also many changes to training targets, the training data format, and other details that probably break compatibility. So if you already had a training run going, unfortunately you will NOT be able to directly continue training with this new KataGo version, due to all these changes. However, any nets you train on the older version will still work with v1.3. Therefore it should still be possible to bootstrap a new run if you want, by using the older version nets to selfplay enough games with v1.3, and then training an entirely fresh new neural net on that v1.3 selfplay data with the new targets and upgraded TF 1.5 code.

Upgrading

Some notes about upgrading to this v1.3:

• Old GTP configs should still work. However, there have been quite a few new parameter additions and improvements to the default parameters, and some features like the log-scaling cPUCT require these changes. So it's recommended that you start fresh with the new provided gtp_example.cfg (included with the executables below, or available here, and simply copy over the changes you had to numSearchThreads, cudaUseFP16, and other settings.

• The benchmark command may be helpful in testing settings: ./katago benchmark -model <NEURALNET>.txt.gz -config <GTP_CONFIG>.cfg. And if you are changing settings or seemingly running into any problems, it's highly recommended you run this command directly in a terminal/console/command-line window first, rather than running KataGo with a GUI program, to test and see what's going on.

• If you are using the OpenCL version with a new neural net, it will need to re-tune the first new time you run it - which may take a while. In a GUI like Lizzie or Sabaki it may look like a hang - if you want to see the progress of tuning, try running the benchmark command above directly (which will tune first if needed).

New Neural Nets

Two neural nets are attached!

• g170-b20c256x2-s668214784-d222255714.txt.gz ("g170 20 block s668M") - Supports all the features above, and should be mildly stronger than g104's old 20 block net (unless my tests got very unlucky with measurement noise), making this the strongest KataGo net now, by a little bit.
• g170e-b10c128-s1141046784-d204142634.txt.gz ("g170e 10 block s1.14G") - A very strong 10 block net. It should be almost equal in strength to g104's old 15 block net, despite being a much smaller size. Since it is smaller, it will run waaay faster, particularly on weaker hardware.

These two above are probably the main ones you might want, but see here for a few additional neural nets if you like. More and stronger nets may be uploaded in the future some time after this run progresses further.

Please feel free to create a Github issue, or to report problems or ask for help in the LZ Discord if you encounter any issues with the attached executables. If problems are found, I'll try to fix them and recompile and bump the tag of this release over the next few days.

Igo Hatsuyoron 120 Special Net

This is not exactly a release of KataGo itself, but rather an upload of a specially trained neural net that should understand and be able to provide strong analysis for Igo Hatsuyoron 120. This is the same neural net that was behind the analysis here: https://lifein19x19.com/viewtopic.php?f=18&t=16995

You should be able to run it yourself with existing versions of KataGo (https://github.com/lightvector/KataGo/releases/tag/v1.2), by swapping it out for the usual KataGo neural net file.
Note: if you use it, don't trust KataGo's score prediction, on this problem it will not be good. Instead, adjust komi and watch the winrate.

It should also still play normal games at a superhuman level too, although I have not directly tested its normal-game strength. Due to having focused training on this one tsumego, it is probably weaker than the standard 20-block net (here) that has been out for a while. But if somehow turns out to be stronger, that wouldn't entirely surprise me - feel free to try it!

Enjoy!

OpenCL, Windows Support, other features and fixes

As of this version, OpenCL is now working! Compiling for Windows on MSVC is now supported, with instructions in the main README, and attached to this release are pre-compiled Windows binaries.

If you have questions or run into issues, feel free to open an issue on this GitHub, or for possibly more interactive feedback, you can check out the Leela Zero Discord (which is also home to a fair amount of KataGo-related discussion as well).

Notes about OpenCL vs CUDA:

• Upon the first startup, the OpenCL version should do a bunch of tuning to optimize parameters to your GPU. This might take a while, and is completely normal.
• You should probably take a look at the notes at the top of gtp_example.cfg and play around with some of the parameters for best performance. If you have a strong GPU, it is quite possible that you could get a factor of 2x-4x performance gain by adjusting the number of threads or other parameters to the optimal values for your system. Unlike the OpenCL-specific GPU parameters, these values are not automatically tuned - future versions of KataGo will probably include better support or automation for tuning these values as well though.
• If you have more than one OpenCL device (e.g. some Intel CPU and also a proper GPU) and it's choosing the wrong one, edit openclDeviceToUse in gtp_example.cfg
• The OpenCL version has in all likelihood NOT been optimized quite as heavily as LZ has, you should not expect it to be nearly as fast since it's new and also KataGo is the first time I (lightvector) have ever written GPU code in any form and am not particularly expert yet. :)
• If you have an NVIDIA GPU, the CUDA version could be faster (or not! it varies). For the CUDA version, you will need to install CUDA 10.1 on your own, as well as CUDNN 7.6.1 (https://developer.nvidia.com/cuda-toolkit) (https://developer.nvidia.com/cudnn). When installing CUDA, you will also need to restart your computer after installing for the changes to pick up properly (CUDA does not tell you this).

Enjoy!

Where to get the neural nets

For pre-trained neural net models, see the prior release.

Changes

Aside from OpenCL and Windows support, minor changes include:

• Use smaller NN buffer for boards smaller than 19x19, performance improvement
• Fixed bug that was preventing ".gz" files from loading properly in Windows.
• analysisPVLen can now configure the maximum length of analysis variations
• dynamicScoreZeroWeight can experimentally configure KataGo's score utility, intended for handicap games.
• Resignation is enabled by default, minor changes to example GTP config params.
• KataGo now by default builds as katago or katago.exe instead of main.
• KataGo now supports loadsgf GTP command.
• GTP extension commands are now documented
• A wide variety of bugfixes and internal cleanups.

Update (2019-07-19) - a few more changes

• Added another change where unless explicitly configured otherwise (in gtp.cfg), KataGo will share OpenCL tuning across all board sizes, since tuning can be a bit slow to do on every board size.
• Added precompiled Linux binaries. These binaries are mostly dynamically linked, so you will still have to ensure you have the appropriate shared libraries (and CUDA and CUDNN for the CUDA version).

Update (2019-07-20) - out of beta now! A few more changes:

• Added some logic to select a reasonable device for OpenCL by default.
• Fixed bug where evaluation sign was flipped in log files.
• Fixed bug where trying to make a move past the end of the game would return an illegal move.
• Fixed some handling of how handicap games were treated with respect to komi updates and board clearing.

OpenCL, Windows Support, other features and fixes (beta)

See updated release at https://github.com/lightvector/KataGo/releases/tag/v1.2

This was a beta version for testing first, it has since been superceeded (and also I screwed up some editing on Github, so I lost the beta version of the release message and files here).

Strong Neural Net, LCB, and many bugfixes

KataGo has a new run ("g104") that has reached as strong as or slightly stronger than LZ-ELFv2! As a result of some improved hyperparameters and other training improvements, starting from scratch this run surpassed the old 1-week run in the previous release in only 3.5 days, and reached perhaps slightly stronger than ELF and near LZ200 after a total 19 days on an average of less than 27 Nvidia V100 GPUs.

g104 models are trained on 9x9 to 19x19 square boards. They will probably generalize somewhat to other sizes, including rectangular boards, but are not guaranteed to play well for boards outside that size range. They are trained on area scoring, positional and situational superko, and suicide on or off. They will behave erratically for other scoring rules or other ko rules.

For match games, this release also features an implementation of LCB (leela-zero/leela-zero#2282) giving a noticeable boost in match strength, as well as a variety of bugfixes and improvements. Self-play training data written by this release is not compatible with training data produced using the prior release, due to a slight format change in the .npz data files.

For the full history of models and training data (rather than only the strongest of each size), see here. If you are curious to look at SGFs of the self-play games, they are included as well.

Also attached for convenience is a precompiled Linux executable, which is the main OS I've tested on, still requiring CUDA 10 and CUDNN to be installed, for now. Compiling yourself may actually be a bit more flexible though - I've actually gotten a working compile on a system with CUDA 9 for example. Support for other OSes, including Windows, and elimination of dependence on CUDA, is being worked on and is not ready yet, but should not be too much longer.

Enjoy!

NOTE: If you're on windows, and you want to try a pre-compiled version that another user has managed to get working there, you can check out #2. Among other things though, you will need to explicitly unzip the neural net ".txt.gz" file to get the neural net to load properly in that version, as well as to ensure you have up-to-date version of your GPU drivers and the right version of cudnn64_7.dll. More-official windows support still in progress.

(edit: 2019-06-18, 2019-06-19 - bumped the tag for this release with some GTP bugfixes/improvements)

Initial release

Initial release of KataGo as of the completion of its initial main run ("g65") up to roughly the strength of LZ130 in a week on a few dozen GPUs.

Included is the strongest neural net of each size from the run, see README_models.txt for details. These neural nets should be ready to use with the code once compiled. Note that although the KataGo code and the self-play training theoretically support territory scoring (i.e. Japanese-style scoring) and simple ko rules (as opposed to just superko), these neural nets were not trained with such rules, and will likely only work properly with area scoring and superko.

For the full history of models and training data (rather than only the strongest of each size), see here.

(edit April 30: Adjusted directory structure of zipped models slightly to be a bit friendlier to latest python code)