Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dataloaders to use Aggregation Lists #264

Merged
merged 28 commits into from
Mar 23, 2023
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
77e56ad
First commit
al-rigazzi Feb 27, 2023
7287875
Tests work, need docs
al-rigazzi Feb 27, 2023
17c3a48
Update docs
al-rigazzi Feb 27, 2023
766da9a
Update tutorial, changelog
al-rigazzi Feb 28, 2023
be114e4
Print output and error for failed dataloader tests
al-rigazzi Feb 28, 2023
2b8daa2
Address reviewers' feedback
al-rigazzi Mar 10, 2023
b674caf
Merge branch 'develop' into update_dataloaders
al-rigazzi Mar 10, 2023
7980435
Add test for DataInfo.__repr__
al-rigazzi Mar 10, 2023
67c4a9e
Make style
al-rigazzi Mar 10, 2023
6563fb5
Address reviewer's comments
al-rigazzi Mar 15, 2023
c4b53f1
Fix numpy type in sim
al-rigazzi Mar 16, 2023
1cd096c
Merge branch 'develop' into update_dataloaders
al-rigazzi Mar 16, 2023
b303f92
Merge branch 'develop' into update_dataloaders
al-rigazzi Mar 17, 2023
c90c777
Make tests local for coverage
al-rigazzi Mar 19, 2023
ea4b5f0
Add test for wrong type
al-rigazzi Mar 19, 2023
9b9482b
Remove unused files
al-rigazzi Mar 19, 2023
afc2f65
Reduce workers in pytorch dataloader test
al-rigazzi Mar 19, 2023
e9d8902
Remove train_torch
al-rigazzi Mar 19, 2023
e889219
Augment coverage
al-rigazzi Mar 19, 2023
b52fbc0
Fix crashing torch dl
al-rigazzi Mar 21, 2023
82bedb2
Remove pragma dir
al-rigazzi Mar 21, 2023
2c85470
Make pytorch dataloader less aggressive
al-rigazzi Mar 21, 2023
08781ab
Use main thread for workers
al-rigazzi Mar 21, 2023
70999e1
Remove prefetch factor
al-rigazzi Mar 21, 2023
c702914
Revert last changes
al-rigazzi Mar 22, 2023
33e8eca
Use external training process for torch
al-rigazzi Mar 22, 2023
cd92c9a
Fix coverage for unreachable code
al-rigazzi Mar 22, 2023
7959514
Merge branch 'develop' into update_dataloaders
al-rigazzi Mar 22, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion doc/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,15 @@ This section details changes made in the development branch that have not yet be

Description

- Update ML data loaders to make use of SmartRedis's aggregation lists
- Update to current version of Redis
- Fix bug in colocated database entrypoint when loading PyTorch models
- Add support for RedisAI 1.2.7, pyTorch 1.11.0, Tensorflow 2.8.0, ONNXRuntime 1.11.1
- Allow for models to be launched independently as batch jobs

Detailed Notes

- The PyTorch and TensorFlow data loaders were update to make use of aggregation lists. This breaks their API, but makes them easier to use. (PR264_)
- Update from Redis version 6.0.8 to 7.0.5. (PR258_)
- Fix bug in colocated database entrypoint stemming from uninitialized variables. This bug affects PyTorch models being loaded into the database. (PR237_)
- The release of RedisAI 1.2.7 allows us to update support for recent versions of pyTorch, Tensorflow, and ONNX (PR234_)
Expand All @@ -39,7 +41,8 @@ Detailed Notes
the object referenced at `Model.batch_settings` as the batch settings for the job. If the check
is not satisfied, the `Model` is launched in the traditional manner as a job step. (PR245_)

.. _PR255: https://github.com/CrayLabs/SmartSim/pull/258
.. _PR264: https://github.com/CrayLabs/SmartSim/pull/264
.. _PR258: https://github.com/CrayLabs/SmartSim/pull/258
.. _PR245: https://github.com/CrayLabs/SmartSim/pull/245
.. _PR237: https://github.com/CrayLabs/SmartSim/pull/237
.. _PR234: https://github.com/CrayLabs/SmartSim/pull/234
Expand Down
4 changes: 2 additions & 2 deletions smartsim/ml/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

from .data import (
DynamicDataDownloader,
StaticDataDownloader,
DataInfo,
DataDownloader,
TrainingDataUploader,
form_name,
)
Loading