The great Woogles refactor

[domino14]

We have a lot of backend tech debt that we want to fix this year. The major tech debt is as follows, but this is not an exhaustive list:

• Game/tournament/user in-memory caches prevent us from having more than one API server, and deploys are not zero-downtime
• Use of separate stores with separate DB connections + gorm create difficulty with transactions
• Mix of protobuf and JSON everywhere causes confusion and makes it hard to search the database
• Other smaller items, like tournament representation in DB should be split by division, not update after every "Ready" request, mod items should be saved in a better format, old games should be offloaded to S3, etc.

The following is a proposed architecture redesign.

This is a circle of services with NATS in the middle.

Each "service"/module communicates through NATS. Each module is easily testable, deployable, and independent. Each only has inputs/outputs.

Communication between services and NATS should be protobuf. Communication to user doesn't need to be protobuf. Socket comms can even be plain text with a translation layer. Twirp API can maybe be replaced with something more RESTful if desired. See OpenAPI.

Sample requests

Make a move in a game

• Player makes a move in a game; gets sent via socket
• Write GAME_EVT to NATS conn with game ID and information about the game. (Turn/evt number should be included too, as a good way to ensure the sequence is correct).
• NATS publishes to listening game service(s)
  • We can use a "Queue group" so that only one game service takes the request
• Game service sends a request (via NATS) for the game information (use request/response with a queue group)
• Store service picks up the request, sends game information back as protobuf
• Game service deserializes it and plays the game to the current turn, and then plays the new turn.
  • Deserialization should be optimized somewhat. Even right now it is very fast, HastyBot does this every turn.
• Game service publishes state back to NATS
  • e.g. GAME_STATE_MSG, GAME_ID, full pb, last turn pb
• Any listeners/subscribers can do the needful:
  • The db can save the state back, and if the game is over, it can save it to S3
    • Save game + new user ratings in a transaction
  • The socket server can publish just the last move (or maybe the full pb might simplify the front end)
  • Anti-cheating module can be listening for finished games and do whatever analysis on them
  • Tourney module can be listening for an ended game and update it
  • Automod/other module can be listening and apply penalties to users who do bad things
  • etc

Request last games in a profile

• Twirp API module picks up request, sends a message via NATS using req-response
• Store module picks up the request, makes the necessary DB requests and sends it back through protobuf
• Twirp API module receives answer, does any necessary conversions to send back data to user.

Help needed

If you are a developer and are excited to help us build this new architecture, please get in touch with us via Discord, or here. We would love your help.

[domino14]

Step 1

• Validate this approach by moving the gameplay component to its own service
• Create utility functions to communicate across different services with different patterns (pubsub and request/response)

[domino14]

Copied-and-pasted from closed issue #914:

This PR is a draft and will probably be closed. The files in it should likely be merged into #905

This aims to do a few things:

First, eliminate the asymmetry between message formats going from the socket server to the API server and from the API server to the socket server. This is eventually because API server messages, or whatever replaces it, will be pushed in a fan-out way to multiple services potentially. They should not have a special length encoding.

The goal is to include the message type in the subject name in both directions. i.e. currently socket server pushes to ipc.pb.3.userid.connid.whatever, which the API server picks up. 3 is the message type here. But API server pushes to channels that look like userid.abcdef, which do not contain the message type. Instead, there's an EventWrapper that adds the type and length of the packet to the binary encoding.

This is unideal for several reasons:

limits packet length artificially
receivers will have different ways to decode different messages. In the future the socket server should not be the only receiver.
others?
The idea is to have all servers communicate with each other with the same binary protobuf. The message type would have to be included in the topic so that they know how to decode the protobuf. The API server could push, for example, to userid.pb.4.abcdef.

Then, the socket server would append the message type and length to the binary encoding right before sending it out to websocket consumers. It can also split it up into pieces, or even turn it into plain text, etc.

The socket server should be modified to be able to handle both types of messages. If pb.x is in the topic name, it would add the length/type encoding before forwarding it on, otherwise, it just forwards the message on. Later on we can remove the legacy encoding.