Hugging face trainer callback integration (#33)

Co-authored-by: Jae-Won Chung <jwnchung@umich.edu>

.gitignore

@@ -11,3 +11,4 @@ dist/
 *.json
 **/.DS_Store
 .cache/
+env/

docs/getting_started/index.md

@@ -106,6 +106,21 @@ for epoch in range(100):
     The [`GlobalPowerLimitOptimizer`][zeus.optimizer.power_limit.GlobalPowerLimitOptimizer] supports multiple [`OptimumSelector`][zeus.optimizer.power_limit.OptimumSelector]s that chooses one power limit among all the profiled power limits.
     Selectors that are current implemented are [`Energy`][zeus.optimizer.power_limit.Energy], [`Time`][zeus.optimizer.power_limit.Time], [`ZeusCost`][zeus.optimizer.power_limit.ZeusCost] and [`MaxSlowdownConstraint`][zeus.optimizer.power_limit.MaxSlowdownConstraint].
 
+### `HFGlobalPowerLimitOptimizer`
+For easy use with [HuggingFace 🤗 Transformers](https://huggingface.co/docs/transformers/en/index), [`HFGlobalPowerLimitOptimizer`][zeus.optimizer.power_limit.HFGlobalPowerLimitOptimizer] is a drop-in compatible [HuggingFace 🤗 Trainer Callback](https://huggingface.co/docs/transformers/en/main_classes/callback). When initializing a [HuggingFace 🤗 Trainer](https://huggingface.co/docs/transformers/main_classes/trainer), initialize and pass in [`HFGlobalPowerLimitOptimizer`][zeus.optimizer.power_limit.HFGlobalPowerLimitOptimizer] as shown below:
+
+```python
+    monitor = ZeusMonitor()
+    optimizer = HFGlobalPowerLimitOptimizer(monitor)
+
+    # Initialize HuggingFace 🤗 Trainer
+    trainer = Trainer(
+        ...,
+        callbacks=[optimizer], # Add the `HFGlobalPowerLimitOptimizer` callback
+    )
+```
+Refer to our [HuggingFace 🤗 example training code for fine-tuning using HFGlobalPowerLimitOptimizer](https://github.com/ml-energy/zeus/tree/master/examples/huggingface/) for complete running examples for single-GPU and multi-GPU training using [HuggingFace 🤗 Trainer](https://huggingface.co/docs/transformers/main_classes/trainer).
+
 ## Recurring jobs
 
 !!! Info

examples/huggingface/README.md

@@ -0,0 +1,47 @@
+# Integrating Zeus with HuggingFace 🤗
+
+This example will demonstrate how to integrate Zeus with `HuggingFace 🤗 Trainer` using `HFGlobalPowerLimitOptimizer`.
+
+[`run_clm.py`](run_clm.py) was adapted from [HuggingFace 🤗's example training code for fine-tuning language models](https://github.com/huggingface/transformers/tree/f3aa7db439a2a3942f76c115197fe953984ac334/examples/pytorch/language-modeling).
+
+## Dependencies
+
+Use the included requirements.txt file to include all extra dependencies:
+```sh
+    pip install -r requirements.txt
+```
+
+## `ZeusMonitor` and `HFGlobalPowerLimitOptimizer`
+
+- [`ZeusMonitor`](http://ml.energy/zeus/reference/monitor/#zeus.monitor.ZeusMonitor): Measures the GPU time and energy consumption of arbitrary code blocks.
+- [`HFGlobalPowerLimitOptimizer`](https://ml.energy/zeus/reference/optimizer/power_limit/#zeus.optimizer.power_limit.HFGlobalPowerLimitOptimizer): Online-profiles each power limit with `ZeusMonitor` and finds the cost-optimal power limit. Calls GlobalPowerLimitOptimizer under the hood.
+
+## Integration with HuggingFace 🤗 Trainer
+For easy use with [HuggingFace 🤗 Transformers](https://huggingface.co/docs/transformers/en/index), [`HFGlobalPowerLimitOptimizer`](https://ml.energy/zeus/reference/optimizer/power_limit/#zeus.optimizer.power_limit.HFGlobalPowerLimitOptimizer) is a drop-in compatible [HuggingFace 🤗 Trainer Callback](https://huggingface.co/docs/transformers/en/main_classes/callback). When initializing a [HuggingFace 🤗 Trainer](https://huggingface.co/docs/transformers/main_classes/trainer), initialize and pass in [`HFGlobalPowerLimitOptimizer`](https://ml.energy/zeus/reference/optimizer/power_limit/#zeus.optimizer.power_limit.HFGlobalPowerLimitOptimizer) as shown below:
+
+```python
+    monitor = ZeusMonitor()
+    optimizer = HFGlobalPowerLimitOptimizer(monitor)
+
+    # Initialize HuggingFace 🤗 Trainer
+    trainer = Trainer(
+        ...,
+        callbacks=[optimizer], # Add the `HFGlobalPowerLimitOptimizer` callback
+    )
+```
+
+## Running the Example
+
+By default, `Trainer` will make use of all available GPUs. If you would like to use only a subset of the GPUs, specify the `CUDA_VISIBLE_DEVICES` environment variable, which Zeus will also automatically respect.
+
+```bash
+python run_clm.py \
+    --model_name_or_path gpt2 \
+    --dataset_name wikitext \
+    --dataset_config_name wikitext-2-raw-v1 \
+    --per_device_train_batch_size 8 \
+    --per_device_eval_batch_size 8 \
+    --do_train \
+    --do_eval \
+    --output_dir /tmp/test-clm
+```

examples/huggingface/requirements.txt

@@ -0,0 +1,8 @@
+accelerate >= 0.12.0
+torch >= 1.3
+datasets >= 1.8.0
+sentencepiece != 0.1.92
+protobuf
+evaluate
+scikit-learn
+transformers>=4.37.2