Playing Hollow Knight with Reinforcement Learning

The repo uses Deep Q-Network to learn how to play boss fight in Hollow Knight.

but it should be able to run on most Hall of Gods Boss with only one boss health bar.

You need to install the Satchel and EnemyHPBar Mod to correctly run (so it can recognize boss HP), I have made some modifications (custom HP bar color) for more stable recognition, the mod files can be found in Managed folder.

Note: the "master" branch contains files and algorithms working in progress, "hornet_new" branch contains a version that was able to train an agent that can defeat hornet most of the time, that branch will no longer be modified unless important updates (e.g. bug fix) is needed

---

Platform and requirements

• Python 3 (tested with Python 3.10)
• Windows 10 or 11 (tested on 11)
• Screen with at least width/height of (1280, 720) resolution (note that this can be modified to smaller resolution in hkenv.py, but then you need to rewrite quite a bit of recognition codes)
• CUDA GPU (CPU is way too slow for real time game, and I used some GPU only features)
• The newest version of Hollow Knight Game (tested with Steam version)
• packages listed in requirements.txt (recommended to install in virtual environment)

---

Usage

run Hollow knight in window mode, make sure use the following keymap settings:

charms I used (you have to use these if you use my pretrained weight, but you can train from scratch with any charm you like):

change game "full screen" to OFF, change resolution to 1280x720

Then, manually walk to one of the Hall of Gods statues, until you see the 'Challenge' prompt, run the following from terminal in a virtual environment with all required packages installed:

python train.py

It will take over keyboards to control the game, if you want to force quit in the middle, Ctrl+C in terminal will work.
While running, do not click on other windows, stay focused on the Hollow Knight window, unless you want to stop running

---

Tentative Plan:

• Update Structure
  • Add requirements.txt, mod files
  • Add instructions to train
  • Add more docstrings (currently done for important ones, may add more in future)
  • Add a simple sanity check that runs gym cartpole
• Train an agent that can defeat Hornet
  • You can see how the agent trained with hornet_new branch perform during evaluation at https://youtu.be/yP6tUWmXCa4
• Add code for additional functionalities (like saving buffer, model...)
  • Saving model periodically
  • Tensorboard logging
  • Save/load random agent experiences
  • Add evaluate in trainer
• Improve the algorithm/model/implementation
  • Frame Stacking
  • Spectral Normalization
  • Huber Loss
  • Double DQN
  • Dueling DQN
  • Multistep return
  • improve reward function (make it denser)
  • Image Augmentation (DrQ)
  • ResNet-like model
  • Squeeze and Excitation based model
  • Noisy Network
  • NAdam Optimizer
  • Reset Training
  • Proportional Prioritized Experience Replay
  • SVEA
  • Averaged-DQN

Inspired by https://github.com/ailec0623/DQN_HollowKnight
It is a very interesting project, and the author has already defeated Hornet with a trained agent. However, that project uses CE and Windows API heavily, which I am less familiar with, and I also want to practice with extensions on dqn, so I decided to write one from scratch on my own.

---

Changes

• Use Huber Loss instead of MSE
• Add Spectral Normalization in model
• Add Double DQN
• Add Dueling DQN (No gradient rescaling)
• Add no magnitude reward (so all rewards are either 1, 0, or -1)
• Use LeakyReLU instead of ReLU
• Remove Dash, it is way too hard to use
• Add Fixed time for each step
• Rewrite reward function
• Add save/load random explorations
• Reduce learning update frequency
• Add Multistep return
• Add DrQ
• Remove Spectral Normalization
• Add more models
• Add Noisy Network based model
• Use Kornia for augmentation (significantly faster than numpy/torchvision)
• Add Intensity augmentation
• Use NAdam optimizer
• remove non-cuda support to use cuda mix-precision
• Add reset training
• Add Proportional Prioritized experience replay
• Add SVEA
• Add Averaged-DQN

---

Training:

• 12/18/2022: agent stuck in repeating a sequence of actions, and eventually stopped doing anything (no op) most of the time.
• 12/19/2022: significantly reduced learning update frequency, the agent no longer converge to a no op situation, so it appears that the problem was overfitting. My next goal would be addressing sparse reward, multistep return is my first attempt.
• 12/20/2022: The agent defeated Hornet the first time (at evaluation), the winning agent is a saved copy from the episode with the highest training reward. Unfortunately, there is a bug in the environment, leading to infinity reward, and broke the agent later on.
• 12/21/2022: The agent stays in a suboptimal policy that tries to die as fast as possible, likely due to the small negative reward for each step, so I removed it
• 12/22/2022-12/23/2022: focus on improving code, code now takes about 10% less time compared to before
• 12/24/2022: The agent again stays in suboptimal policy
• 12/25/2022: The agent now can defeat Hornet with a decent win rate (6 win out of 10 test episodes with bestmodel), this weight uses AttentionExtractor, model weight can be found here
• 12/26/2022: I modified the reward function, now the agent knows to chase the boss (at least sometimes), and got 5 win out of 10 test episodes.
• 12/27/2022: tweaked several hyperparameters, the agent won 10/10 test episodes (note that this does not mean it will always win, but it does mean it got better). This weight uses SimpleExtractor, weight can be found here
• 1/1/2023: starting to train on Hive Knight

---

References (order based on my implementation order)

• Playing Atari with Deep Reinforcement Learning

• Deep Reinforcement Learning with Double Q-learning

• Dueling Network Architectures for Deep Reinforcement Learning

• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective

• Rainbow: Combining Improvements in Deep Reinforcement Learning

• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels

• Deep Residual Learning for Image Recognition

• Squeeze-and-Excitation Networks

• A ConvNet for the 2020s

• Bag of Tricks for Image Classification with Convolutional Neural Networks

• Noisy Networks for Exploration

• The Primacy Bias in Deep Reinforcement Learning

• Prioritized Experience Replay

• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation

• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning

• https://github.com/toshikwa/fqf-iqn-qrdqn.pytorch/blob/master/fqf_iqn_qrdqn/network.py

• https://github.com/pytorch/vision/blob/main/torchvision/models/vgg.py

• https://www.youtube.com/watch?v=NP8pXZdU-5U