This repo hosts some educational scripts for doing basic training on a decoder-only transformer, using node-mlx with Node.js.
Files:
model.js
- defines the model.input.txt
- text file used for training the model.train.js
- script for traning.generate.js
- script for generating text using the trained model.
Only Macs with Apple Silicon are supported.
Download dependencies and run the training script, which generates
tokenizer.json
and weights.safetensors
:
npm install
node train.js
Then use the generate script to actually generate some text from the weights:
node generate.js
After understanding the basics of model training, you can check the train-llama3-js repo on how to train a Llama3 model with large datasets.
Public domain.