Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cloud RBAC #1999

Merged
merged 123 commits into from
Nov 24, 2023
Merged
Show file tree
Hide file tree
Changes from 101 commits
Commits
Show all changes
123 commits
Select commit Hold shift + click to select a range
449b046
WIP RBAC
schustmi Sep 21, 2023
f88b49c
Merge branch 'develop' into feature/rbac
schustmi Oct 11, 2023
c46adea
Merge branch 'develop' into feature/rbac
schustmi Oct 13, 2023
667af06
Add dehydration POC
schustmi Oct 13, 2023
7183034
rename default stack/components
schustmi Oct 16, 2023
7354819
Merge branch 'develop' into feature/rbac
schustmi Oct 16, 2023
5baec80
Start removing sharing
schustmi Oct 16, 2023
831ed0c
Some fixes
schustmi Oct 17, 2023
f2f6843
Remove auth scopes
schustmi Oct 17, 2023
41a6f5d
Validate source class
schustmi Oct 17, 2023
738ac04
Some cleanup and docstrings
schustmi Oct 17, 2023
4e98925
More cleanup
schustmi Oct 17, 2023
d19ea40
Fix import
schustmi Oct 18, 2023
0ff6a8e
Fix some mypy issues
schustmi Oct 19, 2023
88d4494
Add helm chart option for rbac implementation source
schustmi Oct 19, 2023
a00c666
Cleanup
schustmi Oct 19, 2023
9853853
Merge branch 'develop' into feature/rbac
schustmi Oct 23, 2023
0eb831c
Fix alembic order
schustmi Oct 23, 2023
f552c1c
Add 403 response in OpenAPI
schustmi Oct 23, 2023
df77f37
Better page dehydration
schustmi Oct 23, 2023
8f25102
ZenML cloud rbac implementation
schustmi Oct 23, 2023
0b64217
Don't include recursive ids
schustmi Oct 23, 2023
90d16c8
Update docstring
schustmi Oct 23, 2023
2925d21
Migrate private components
schustmi Oct 24, 2023
c82d486
Add docstrings
schustmi Oct 24, 2023
39a9e43
More utility functions
schustmi Oct 24, 2023
4378245
Pass secretEnvironment as kubernetes secret
schustmi Oct 24, 2023
ed6bb84
Remove service connector sharing
schustmi Oct 24, 2023
bc9229d
Fix docstrings
schustmi Oct 24, 2023
f8bbb37
Fix most tests
schustmi Oct 24, 2023
5039e8e
Merge branch 'develop' into feature/rbac
schustmi Oct 25, 2023
af56cf1
Fix more tests
schustmi Oct 25, 2023
10fe410
Remove old tests
schustmi Oct 25, 2023
11985f0
Fix alembic order
schustmi Oct 25, 2023
a82d93c
Higher level helper functions for rbac endpoints
schustmi Oct 26, 2023
822fd22
More endpoints implemented
schustmi Oct 26, 2023
5d72cc6
More endpoints implemented
schustmi Oct 27, 2023
6aac957
Secrets endpoints
schustmi Oct 27, 2023
af7b0ca
Custom cache key for local artifact store
schustmi Oct 30, 2023
ccfbd3b
WIP single default stack
schustmi Oct 30, 2023
9c3749b
Merge branch 'develop' into feature/rbac
schustmi Oct 30, 2023
d519c17
Fix config
schustmi Oct 30, 2023
0ecba87
Respect ownership in list endpoints
schustmi Oct 30, 2023
dda2905
Respect ownership in service connector endpoints
schustmi Oct 31, 2023
78b82bc
Refactor internal request models
schustmi Oct 31, 2023
9e10e05
Disable workspace endpoints
schustmi Oct 31, 2023
df862d6
Merge branch 'develop' into feature/rbac
schustmi Nov 3, 2023
e57eb63
Model endpoints
schustmi Nov 3, 2023
6c440bc
Merge branch 'develop' into feature/rbac
schustmi Nov 16, 2023
1278009
Start post-merge cleanup
schustmi Nov 16, 2023
2a9c7ad
Some after merge fixes
schustmi Nov 17, 2023
ead6262
More fixes and hopeless attempts to fix type checking
schustmi Nov 17, 2023
5e23c8f
Some ugly workarounds to partially make typechecking work
schustmi Nov 17, 2023
55b5889
Remove role from log message
schustmi Nov 17, 2023
530487a
Merge branch 'develop' into feature/rbac
schustmi Nov 17, 2023
b1829e7
Some adjustments for new model endpoints
schustmi Nov 17, 2023
094c1f6
Fix migrations
schustmi Nov 17, 2023
446b76b
More mypy fixes
schustmi Nov 17, 2023
7faefed
Some docstrings
schustmi Nov 17, 2023
05d28d8
Permissions for workspaces, service accounts and tags
schustmi Nov 20, 2023
2c406ef
Merge branch 'develop' into feature/rbac
schustmi Nov 20, 2023
1e5f080
Early exit if rbac disabled
schustmi Nov 20, 2023
c0412a4
Run test only with rest
schustmi Nov 20, 2023
2b9bebc
Fix sql pagination
schustmi Nov 20, 2023
438e718
Small adjustments
schustmi Nov 20, 2023
ebda193
Fix service accounts
schustmi Nov 20, 2023
5047b91
Reorder migration table drops
schustmi Nov 20, 2023
ad4f2bf
Some more endpoints fixed
schustmi Nov 20, 2023
1c19a6e
Generalize rbac on filter models
schustmi Nov 21, 2023
00bd06f
Add rbac check to service connector resources endpoint
schustmi Nov 21, 2023
5a3b63f
run metadata
schustmi Nov 21, 2023
96ee0dd
docstrings
schustmi Nov 21, 2023
43552b9
Fix filter model edge case
schustmi Nov 21, 2023
5514a1c
Linting
schustmi Nov 21, 2023
a3416ea
Request timeouts
schustmi Nov 21, 2023
1e0501d
Add stack composition and fix CLI deleted columns
schustmi Nov 21, 2023
045f6c6
Use shorter suffix in migration
schustmi Nov 21, 2023
b2055cf
Remove jwt token permissions
schustmi Nov 22, 2023
7e10099
Move rbacImplementationSource to helm chart auth section
schustmi Nov 22, 2023
a38ebea
Update src/zenml/cli/server.py
schustmi Nov 22, 2023
c4e22d8
Merge branch 'develop' into feature/rbac
schustmi Nov 22, 2023
a7fc9d8
Move database initialization into SQLZenStore
schustmi Nov 22, 2023
4d64e5d
Remove unused security_scopes args
schustmi Nov 22, 2023
701e8d5
Make flavors user scoped
schustmi Nov 22, 2023
cdf7f05
Fix mypy issues
schustmi Nov 22, 2023
d55cb37
Use internal flavor request model
schustmi Nov 22, 2023
139f807
Re-add delete user endpoint
schustmi Nov 22, 2023
33a4105
Enable user/service account deletion tests for rest
schustmi Nov 22, 2023
0e8d890
Use IllegalOperationError instead of HTTP exception
schustmi Nov 22, 2023
34105ed
Remove unused property
schustmi Nov 22, 2023
27aaf84
Raise different errors depending on permission denied and missing body
schustmi Nov 22, 2023
241651b
Fix wrong error args
schustmi Nov 22, 2023
dca2a0b
Don't fail early for service accounts
schustmi Nov 22, 2023
dafdce3
Fix stack and flavor schemas to ignore user and workspace during update
stefannica Nov 22, 2023
ea0eb25
Catch correct error
schustmi Nov 22, 2023
21ad7d5
Merge branch 'develop' into feature/rbac
schustmi Nov 23, 2023
54f5137
Apply rbac to new endpoint
schustmi Nov 23, 2023
2c5e457
Fix alembic order
schustmi Nov 23, 2023
5c53679
Create default stack when creating workspace
schustmi Nov 23, 2023
198545c
Fix workspace statistics inconsistency issue
schustmi Nov 23, 2023
942e812
Merge branch 'develop' into feature/rbac
schustmi Nov 23, 2023
ded578d
Fix some docstrings
schustmi Nov 23, 2023
b09b7b0
Fix some tests
schustmi Nov 23, 2023
c613c58
Use action enum in more places
schustmi Nov 23, 2023
6331448
Auto-update of E2E template
actions-user Nov 23, 2023
4889df7
Add rbac on user endpoints
schustmi Nov 23, 2023
96ef7c9
Merge branch 'feature/rbac' of github.com:zenml-io/zenml into feature…
schustmi Nov 23, 2023
8a5cf5a
Add early return for dehydration and more response models to mapping
schustmi Nov 23, 2023
d1ebd44
Don't filter by user ID when finding stack components during tests
schustmi Nov 23, 2023
4396df5
Merge branch 'develop' into feature/rbac
schustmi Nov 23, 2023
9c4df61
Fix some tests
schustmi Nov 24, 2023
5329e5e
Make model version test robust to existing models
schustmi Nov 24, 2023
ce5f78a
Merge branch 'develop' into feature/rbac
schustmi Nov 24, 2023
5f5b40a
Fix tests that verify account deletion with owned resources
stefannica Nov 24, 2023
56d3203
Merge branch 'feature/rbac' of github.com:zenml-io/zenml into feature…
stefannica Nov 24, 2023
b344973
Merge branch 'feature/rbac' of github.com:zenml-io/zenml into feature…
schustmi Nov 24, 2023
d737470
Require pipeline run permissions in api token endpoint
schustmi Nov 24, 2023
0d08530
Fix zen store tests after breaking them
stefannica Nov 24, 2023
686f860
Merge branch 'feature/rbac' of github.com:zenml-io/zenml into feature…
stefannica Nov 24, 2023
ade951c
Auto-update of E2E template
actions-user Nov 24, 2023
286b4e5
Prevent random failure on secret reference test
schustmi Nov 24, 2023
bc4186f
Exponential backoff when sending requests to cloud api
schustmi Nov 24, 2023
e9ae271
Merge branch 'feature/rbac' of github.com:zenml-io/zenml into feature…
schustmi Nov 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions src/zenml/artifact_stores/base_artifact_store.py
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,18 @@ def path(self) -> str:
"""
return self.config.path

@property
def custom_cache_key(self) -> Optional[bytes]:
"""Custom cache key.

Any artifact store can override this property in case they need
additional control over the caching behavior.

Returns:
Custom cache key.
"""
return None

# --- User interface ---
@abstractmethod
def open(self, name: PathType, mode: str = "r") -> Any:
Expand Down
15 changes: 12 additions & 3 deletions src/zenml/artifact_stores/local_artifact_store.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,9 +75,6 @@ def ensure_path_local(cls, path: str) -> str:
def is_local(self) -> bool:
"""Checks if this stack component is running locally.

This designation is used to determine if the stack component can be
shared with other users or if it is only usable on the local host.

Returns:
True if this config is for a local component, False otherwise.
"""
Expand Down Expand Up @@ -136,6 +133,18 @@ def local_path(self) -> Optional[str]:
"""
return self.path

@property
def custom_cache_key(self) -> Optional[bytes]:
"""Custom cache key.

The client ID is returned here to invalidate caching when using the same
local artifact store on multiple client machines.

Returns:
Custom cache key.
"""
return GlobalConfiguration().user_id.bytes


class LocalArtifactStoreFlavor(BaseArtifactStoreFlavor):
"""Class for the `LocalArtifactStoreFlavor`."""
Expand Down
106 changes: 4 additions & 102 deletions src/zenml/cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -182,12 +182,7 @@
```

For fields marked as being of type `BOOL`, you can use the 'True' or 'False'
values to filter the output. For example, to find all orchestrators that are
currently shared, you would type:

```shell
zenml orchestrator list --is_shared="True"
```
values to filter the output.

Finally, for fields marked as being of type `DATETIME`, you can pass in datetime
values in the `%Y-%m-%d %H:%M:%S` format. These can be combined with the `gte`,
Expand Down Expand Up @@ -228,8 +223,6 @@
zenml artifact-store list
```

If you wish to update/share

If you wish to delete a particular artifact store, pass the name of the
artifact store into the CLI with the following command:

Expand Down Expand Up @@ -737,20 +730,6 @@
zenml stack register STACK_NAME ... --set
```

If you want to share the stack and all of its components with everyone using
the same ZenML deployment, simply pass along the `--share` flag.

```bash
zenml stack register STACK_NAME ... --share
```

Even if you haven't done so at creation time of the stack, you can always
decide to do so at a later stage.

```bash
zenml stack share STACK_NAME
```

To list the stacks that you have registered within your current ZenML
workspace, type:

Expand Down Expand Up @@ -1288,11 +1267,11 @@ def my_pipeline(...):
ssl_verify_server_cert: false
```

Managing users, teams, workspaces and roles
Managing users and workspaces
-------------------------------------------

When using the ZenML service, you can manage permissions by managing users,
teams, workspaces and roles using the CLI.
When using the ZenML service, you can manage permissions by managing users and
workspaces and using the CLI.
If you want to create a new user or delete an existing one, run either

```bash
Expand All @@ -1303,87 +1282,11 @@ def my_pipeline(...):
zenml user delete USER_NAME
```

A freshly created user will by default be assigned the admin role. This
behavior can be overwritten:
```bash
zenml user create USER_NAME --role guest
```

To see a list of all users, run:
```bash
zenml user list
```

A team is a grouping of many users that allows you to quickly assign and
revoke roles. If you want to create a new team, run:

```bash
zenml team create TEAM_NAME
```
To add one or more users to a team, run:
```bash
zenml team add TEAM_NAME --user USER_NAME [--user USER_NAME ...]
```
Similarly, to remove users from a team run:
```bash
zenml team remove TEAM_NAME --user USER_NAME [--user USER_NAME ...]
```
To delete a team (keep in mind this will revoke any roles assigned to this
team from the team members), run:
```bash
zenml team delete TEAM_NAME
```

To see a list of all teams, run:
```bash
zenml team list
```

A role groups permissions to resources. Currently, there are the following
globally scoped roles to choose from: 'write', 'read' and 'me'. To create
a role, run one of the following commands:
```bash
zenml role create ROLE_NAME -p write -p read -p me
zenml role create ROLE_NAME -p read
```

To delete a role run:
```bash
zenml role delete ROLE_NAME
```

To see a list of all roles, run:
```bash
zenml role list
```

You can also update the role name and the attached permissions of a role:
```bash
zenml role update [-n <NEW_NAME>| -r <PERMISSION_TO_REMOVE>| -a <PERMISSION_TO_ADD>]
```

If you want to assign or revoke a role from users or teams, you can run

```bash
zenml role assign ROLE_NAME --user USER_NAME [--user USER_NAME ...]
zenml role assign ROLE_NAME --team TEAM_NAME [--team TEAM_NAME ...]
```
or
```bash
zenml role revoke ROLE_NAME --user USER_NAME [--user USER_NAME ...]
zenml role revoke ROLE_NAME --team TEAM_NAME [--team TEAM_NAME ...]
```

You can see a list of all current role assignments by running:

```bash
zenml role assignment list
```

At any point you may inspect all available permissions:
```bash
zenml permission list
```

Managing service accounts
-------------------------
Expand Down Expand Up @@ -1567,7 +1470,6 @@ def my_pipeline(...):
from zenml.cli.model import * # noqa
from zenml.cli.model_registry import * # noqa
from zenml.cli.pipeline import * # noqa
from zenml.cli.role import * # noqa
from zenml.cli.secret import * # noqa
from zenml.cli.served_model import * # noqa
from zenml.cli.server import * # noqa
Expand Down
Loading
Loading