ZenML version 0.71.0 delivers a new Modal step operator integration as its core feature, enabling efficient cloud execution for ML pipelines with granular hardware configuration options. The release strengthens enterprise capabilities through improved token management and dashboard features while expanding artifact handling with dynamic naming and enhanced visualization support. Additionally, it includes various infrastructure improvements and bug fixes that enhance the platform's stability and usability, particularly around Docker connectivity, Kubernetes management, and service connector operations.
New Feature: Modal Step Operator Integration
ZenML now integrates with Modal, bringing lightning-fast cloud execution capabilities to your ML pipelines. This new step operator allows you to execute individual pipeline steps on Modal's specialized compute instances, offering notable speed, particularly for Docker image building and hardware provisioning. With simple configuration options, you can precisely specify hardware requirements like GPU type, CPU count, and memory for each step, making it ideal for resource-intensive ML workloads.
New Feature: AWS Image Builder
Don't want to worry about Docker locally? Now build images remotely in AWS. Docs: https://docs.zenml.io/stack-components/image-builders/aws
Other Highlights
- Workload API Token Management: Refactored token management for improved security with a generic API token dispenser.
- Dashboard Enhancements:
- Introduced service account management capabilities.
- Added API key creation and integration features.
- Dynamic Artifact Naming: Introduced capability to dynamically name artifacts.
- Visualization Enhancements: Made dictionaries and lists visualizable, added JSON visualization type.
Additional Features and Improvements
- Improved error messages for Docker daemon connectivity
- Enhanced SageMaker URL handling
- Simplified model version artifact linkage
- Added testing for pipeline templates
- Improved Kubernetes pod and label length management
- Allowed skipping type annotations for step inputs
- Enabled using feature service instances instead of just names
Bug Fixes
- Fixed issues with getting out of an inaccessible active stack
- Fixed race conditions in the service connector type registry
- Resolved migration test complications
- Corrected documentation links
- Fixed artifact store and artifact URI handling
- Addressed various scalability and compatibility issues
Documentation Updates
- Added documentation redirects
- Updated PyTorch documentation links
- Improved service connector documentation
What's Changed
- Refactored workload API token management for better security and implemented generic API token dispenser by @stefannica in #3154
- Add 0.70.0 to the migration tests by @avishniakov in #3190
- Adjustments to the PR template by @bcdurak in #3194
- [docs] Fix links in the how-to section of docs by @wjayesh in #3196
- Fixing sagemaker urls to take the settings into consideration by @bcdurak in #3195
- Add cached run into testing of migrations by @avishniakov in #3199
- Fix service connector type registry race conditions by @stefannica in #3202
- Refactor container resource configuration in Vertex Orchestrator test by @avishniakov in #3203
- [docs] Add missing redirects by @wjayesh in #3200
- Add links to
uvnew PyTorch documentation by @strickvl in #3204 - Fix broken docs link by @strickvl in #3208
- Bugfix for getting out of an inaccessible active stack with no permissions by @bcdurak in #3198
- Simplify model version artifact linkage by @schustmi in #3175
- Reenable macos testing by @avishniakov in #3205
- Various fixes and improvements by @stefannica in #3211
- Pass config path during zenml pipeline build by @schustmi in #3212
- Add test for running templates by @schustmi in #3192
- Fix service connector docstring by @schustmi in #3216
- Improve error message when docker daemon is not reachable by @schustmi in #3214
- Don't run migration for empty updates by @schustmi in #3210
- Remove
--checkfrom format script and fix naming by @safoinme in #3218 - More scalability improvements by @schustmi in #3206
- Use correct keyword for artifact store open by @schustmi in #3220
- Fix passing of some sagemaker settings by @schustmi in #3221
- Add hint when trying to connect with api key by @schustmi in #3222
- Allow passing None values as parameter for optional complex types by @schustmi in #3215
- Limit kubernetes pod and label length by @schustmi in #3217
- Updating the quickstart example to use the new
log_metadataby @bcdurak in #3188 - Allow skipping type annotations for step inputs by @schustmi in #3223
- Modal Step Operator by @strickvl in #2948
- Add dynamic artifacts naming, documentation and tests by @avishniakov in #3201
- Run template CLI command and bugfix by @schustmi in #3225
- Make dicts/lists visualizable and add JSON as viz type by @wjayesh in #2882
- Instances of the
FeatureServices are now used instead of only the names of the FeatureServices. by @aiakide in #3209 - Quickstart fixes by @schustmi in #3227
- Add missing docs by @schustmi in #3226
- Misc cleanup by @schustmi in #3229
- Fix input resolution for steps with dynamic artifact names by @schustmi in #3228
- Follow-up on the
run_metadatachanges by @bcdurak in #3193 - Fixed broken links by @htahir1 in #3232
- Fixed wandb login problem in Quickstart by @htahir1 in #3233
- Misc bugfixes by @schustmi in #3234
- Add additional way to fetch docker repo digest by @schustmi in #3231
- AWS Image Builder implementation by @stefannica in #2904
- Disable client-side caching for some orchestrators by @schustmi in #3235
- Fix artifact uris for artifacts with name placeholders by @schustmi in #3237
- Materializer test fix on Windows by @bcdurak in #3238
- Fix GET step run endpoint to return unhydrated response if requested by @schustmi in #3240
- Pipeline run API token fixes and improvements by @stefannica in #3242
Full Changelog: 0.70.0...0.71.0
