Skip to content

Commit

Permalink
Update examples/huggingface/README.md
Browse files Browse the repository at this point in the history
Co-authored-by: Jae-Won Chung <jwnchung@umich.edu>
  • Loading branch information
parthraut and jaywonchung authored Feb 16, 2024
1 parent 02391a4 commit 4579431
Showing 1 changed file with 5 additions and 20 deletions.
25 changes: 5 additions & 20 deletions examples/huggingface/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,11 @@ For easy use with [HuggingFace 🤗 Transformers](https://huggingface.co/docs/tr
)
```

## Fine-tuning Example
## Running the Example

By default, `Trainer` will make use of all available GPUs. If you would like to use only a subset of the GPUs, specify the `CUDA_VISIBLE_DEVICES` environment variable, which Zeus will also automatically respect.

```bash
# Single-GPU example fine-tunes GPT-2 on WikiText-2. We're using the raw WikiText-2
# (no tokens were replaced before the tokenization).
# The loss here is that of causal language modeling.
python run_clm.py \
--model_name_or_path gpt2 \
--dataset_name wikitext \
Expand All @@ -43,19 +43,4 @@ python run_clm.py \
--per_device_eval_batch_size 8 \
--do_train \
--do_eval \
--output_dir /tmp/test-clm \
--gpu_indices="0" # Specify GPUs to ZeusMonitor. If left out, Zeus Monitor uses all available GPUs.

# Multi-GPU example fine-tunes GPT-2 on WikiText-2 using 4 GPUs
torchrun \
--nproc_per_node 4 run_clm.py \
--model_name_or_path gpt2 \
--dataset_name wikitext \
--dataset_config_name wikitext-2-raw-v1 \
--per_device_train_batch_size 8 \
--per_device_eval_batch_size 8 \
--do_train \
--do_eval \
--output_dir /tmp/test-clm \
--gpu_indices="0,1,2,3" # Specify GPUs to ZeusMonitor. If left out, Zeus Monitor uses all available GPUs.
```
--output_dir /tmp/test-clm

0 comments on commit 4579431

Please sign in to comment.