Skip to content

Commit

Permalink
fix image links in esm2 model card (NVIDIA#584)
Browse files Browse the repository at this point in the history
### Description
Updates broken image links in ESM-2 model card

### Type of changes
<!-- Mark the relevant option with an [x] -->

- [ ]  Bug fix (non-breaking change which fixes an issue)
- [ ]  New feature (non-breaking change which adds functionality)
- [ ]  Refactor
- [x]  Documentation update
- [ ]  Other (please describe):

### CI Pipeline Configuration
Configure CI behavior by checking relevant boxes below. This will
automatically apply labels.

- [x]
[SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci)
- Skip all continuous integration tests
- [ ]
[INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests)
- Execute notebook validation tests in pytest

> [!NOTE]
> By default, the notebooks validation tests are skipped unless
explicitly enabled.

### Usage
<!--- How does a user interact with the changed code -->
```python
TODO: Add code snippet
```

### Pre-submit Checklist
<!--- Ensure all items are completed before submitting -->

 - [ ] I have tested these changes locally
 - [ ] I have updated the documentation accordingly
 - [ ] I have added/updated tests as needed
 - [ ] All existing tests pass successfully

---------

Signed-off-by: Peter St. John <pstjohn@nvidia.com>
  • Loading branch information
pstjohn authored Jan 15, 2025
1 parent 81e0b24 commit 448f353
Show file tree
Hide file tree
Showing 8 changed files with 154 additions and 164 deletions.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
298 changes: 148 additions & 150 deletions docs/docs/assets/images/esm2/esm2_model_scaling.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
14 changes: 4 additions & 10 deletions docs/docs/models/ESM-2/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,29 +121,23 @@ checkpoints is consistent with their outputs when evaluated with the HuggingFace

#### Single-node Training Performance

<figure markdown="span">
![ESM-2 Single-Device Training Performance](../assets/images/esm2/esm2_single_node_training_perf.svg){ width="350" }
</figure>
![ESM-2 Single-Device Training Performance](../../assets/images/esm2/esm2_single_node_training_perf.png)

The pure-pytorch baseline (compiled with `torch.compile()`) raised an out-of-memory error for batch sizes larger than 16
at the ESM2-650M model size. The `bionemo2` model could handle batch sizes of 46, reaching a model flops utilization of
59.2% on an NVIDIA A100.

#### Model Scaling

<figure markdown="span">
![ESM-2 Model Scaling](../assets/images/esm2/esm2_model_scaling.svg)
</figure>
![ESM-2 Model Scaling](../../assets/images/esm2/esm2_model_scaling.png)

Training ESM-2 at the 650M, 3B, and 15B model variants show improved performance with the BioNeMo2 framework over the
pure-pytorch baseline. These experiments were conducted on 16x NVIDIA A100 or 16x NVIDIA H100 GPUs split across two
nodes.
nodes. <sup>*</sup>*Note:* 15B model variants were trained on 64 GPUs with the BioNeMo2 framework.

#### Device Scaling

<figure markdown="span">
![ESM-2 Device Scaling](../assets/images/esm2/esm2_device_scaling.svg){ width="400" }
</figure>
![ESM-2 Device Scaling](../../assets/images/esm2/esm2_device_scaling.png)

Training ESM-3B on 256 NVIDIA A100s on 32 nodes achieved 96.85% of the theoretical linear throughput expected from
extrapolating single-node (8 GPU) performance, representing a model flops utilization of 60.6% at 256 devices.
4 changes: 1 addition & 3 deletions docs/docs/models/ESM-2/pre-training.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,7 @@ and train/test splits are available.

Validation perplexity evaluated on the NVIDIA validation set.

<figure markdown="span">
![ESM-2 Pre-training Convergence](../assets/images/esm2/esm2_pretrain_convergence.svg){ width="350" }
</figure>
![ESM-2 Pre-training Convergence](../../assets/images/esm2/esm2_pretrain_convergence.png)

| Model Size | Perplexity at 500k updates |
| -------------- | ------ |
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/models/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ State-of-the-art models are continually integrated into the BioNeMo Framework. T

| **Model** | **Modality** | **Uses** |
| ------------------------------------------ | ------------------ | --------------------------------------------- |
| [ESM-2](./esm2.md) | Protein | Representation Learning |
| [ESM-2](./ESM-2/index.md) | Protein | Representation Learning |
| [Geneformer](./geneformer.md) | Single Cell | Representation Learning |

For more information about the models included in BioNeMo Framework, refer to the Model Cards linked in the table above or the original publications referenced in the respective model descriptions.

0 comments on commit 448f353

Please sign in to comment.