The state of official-stockfish/books

[NightlyKing]

I'm starting this discussion here because it seems that issues in the official-stockfish/books repository don't receive attention.

Our repository of opening books is in quite a bad state. It is functional, but that's it. First, let me list the obvious flaws:

1. No documentation.
2. No naming convention (the filenames are, in fact, quite confusing.).
3. Everything is in the same directory.
4. No standardized way to accept new books.
5. No separation between 'legacy' and 'actively used' books (could be solved by solving point 3).

@snicolet mentioned the lack of documentation of /books in our discord and hinted that working on the documentation could be a good starting point for a new contributor. I have to disagree given the circumstances. In the current state, there's hardly anything to document. The only thing that's easy enough to find out is which books are actively used.

I have some suggestions. There are two relatively easy things to do (assuming that my proposed changes don't break fishtest):

• Create new directories. At the very least, we need a directory for "legacy books" and one for "actively used books". One may create new directories for specialized books (e.g. FRC/DFRC books and endgame books). Also, scripts like epd2pgn.py should be in their own directory. The same goes for things like cutechess-cli-win.zip. Why do we keep cutechess-cli in our book repository in the first place?
• Only accept new books if the documentation is updated as well.

My other suggestions are not as easy to implement and require more discussion:

• Establish a naming convention for our books. I'd suggest that we don't accept usernames in our filenames. The username of the creator should instead be part of the documentation. Additionally, I'd like to propose that we only use lower case letters. The goal should be that the filename is easy to read and easy to use in scripts (damn you, case sensitive file systems). Further, it should convey what the book is about. A name like "hybrid_book_beta.pgn.zip" says little to nothing about the intention behind its creation.
• Establish a standardized way to accept new books. This seems to be even more complicated than creating a naming convention. I'm not even sure if it's helpful to have a rigid standard for accepting new books. Perhaps agreeing on a set of guidelines would be easier while also providing some flexibility.

I think it makes sense to resolve these issues first. If done properly, it'll be much easier to write useful documentation. IMHO, what should and shouldn't be included in the documentation is another discussion we may start at a later point.

Obviously, the issues I describe aren't of high priority.

[Disservin]

I think the main purpose of the books repo is to simply act as a file server, it also requires very little changes and what not. It's not really meant to beginner friendly because no one should really touch it, unless the maintainer of Stockfish thinks we need a book to continue testing.

[NightlyKing]

I think the main purpose of the books repo is to simply act as a file server

As of now it is. I just think it would be useful if it was more than that :)

[cj5716]

I think the book name is already standardized across all engines, we just select a book from those there for testing

Edit: most of the books are from here, already named by Stefan Pohl https://www.sp-cc.de/anti-draw-openings.htm

[NightlyKing]

Even within the UHO book "series", we break the naming "convention" by having books like UHO_4060_v2.epd.zip.

I'm also not quite sure what you mean by "the book name is already standardized across all engines"? What I meant is that it'd be nice to have a standard way of naming books that get accepted into our /books repo. That'd also mean renaming the UHO books to conform to our standard.

[cj5716]

Up to maintainer to decide whether to rename books I guess.

[dav1312]

UHO 4060 is a filtered compilation of UHO books. Instead of Dragon centipawn evals it has Stockfish win rate evals.