Storage Rest API, Refine Storage Data and CLI (#3754)

* Add CRUD for Storage, Set storage configs for group, Get user storage config in user info Add Storage as Rest API to implement storage CRUD Add Rest API to set storage configs for group Modify User Rest API to get user available storage configs in user info Update storage_plugin cli tool for new schema Change webportal to call Storage Rest API to get storage info Create namespace and secret when deploy rest-server
microsoft · Oct 22, 2019 · 9676024 · 9676024
1 parent be28456
commit 9676024
Show file tree

Hide file tree

Showing 20 changed files with 1,782 additions and 125 deletions.
diff --git a/contrib/storage_plugin/README.MD b/contrib/storage_plugin/README.MD
@@ -8,7 +8,7 @@ A tool to manage external storage in PAI.
     - [ Setup server ](#Usages_setup_server)	
     - [ Create storage server in PAI ](#Usages_server)
     - [ Create storage config in PAI ](#Usages_config)
-    - [ Create storage user in PAI(optional) ](#Usages_user)
+    - [ Set storage config access for group ](#Usages_groupsc)
     - [ Use Storage in PAI ](#Usages_job)
     - [ Example ](#Usages_example)
 - [ Storage data structure ](#Data_structure)
@@ -19,7 +19,7 @@ A tool to manage external storage in PAI.
         - [ Azureblob Server data structure ](#Azureblob_data)
         - [ Hdfs Server data structure ](#Hdfs_data)
     - [ Config data structure ](#Config_data)
-    - [ User data structure ](#User_data)
+    - [ Config in group data ](#Config_in_group_data)
 
 ## What is team wise storage <a name="Team_storage"></a>
 Team wise storage is a solution that helps admin to manage NAS(network attached storage) by team/group. After admin configured team wise storage settings, users can easily use NAS in their jobs.<br/>
@@ -95,14 +95,13 @@ Create config using command:
 python storagectl.py config set CONFIG_NAME GROUP_NAME [-s SERVER_NAME_1 SERVER_NAME_2 ...] [-m MOUNT_POINT SERVER PATH]... [-d]
 ```
 
-### Create storage user in PAI (optional) <a name="Usages_user"></a>
+### Set storage config access for group <a name="Usages_groupsc"></a>
 In PAI dev-box, swith to folder pai/contrib/storage-plugin
 
-Create user config using command:
+Set storage config access for group using command:
 ```
-python storagectl.py user set USER_NAME SERVER_NAME_1 [SERVER_NAME_2 ...]
+python storagectl.py groupsc add GROUP_NAME CONFIG_NAME
 ```
-User config defines user's permissions on servers.
 
 ### Use Storage info in job container <a name="Usages_job"></a>
 User can use team wise storage through job submit page. Please refer to related page for details.
@@ -121,7 +120,8 @@ The structure of samba server is as follows:
 Now we want all members of "paigroup" mount server's data folder to /data, and user's data (e.g user1) to /user by default. The admin should setup storage config in PAI using:
 ```bash
 python storagectl.py server set smbserver samba 10.0.0.1 root paismb paipwd local
-python storagectl.py config set configsmb paigroup -s smbserver -m /data smbserver data -m /user smbserver 'users/${PAI_USER_NAME}' -d
+python storagectl.py config set configsmb -s smbserver -m /data smbserver data -m /user smbserver 'users/${PAI_USER_NAME}' -d
+python storagectl.py groupsc add paigroup configsmb
 ```
 Then when "paiuser" from "paigroup" uses job submit page, the configsmb will be shown and user can choose whether to use it <br/>
 
@@ -220,11 +220,19 @@ Then when "paiuser" from "paigroup" uses job submit page, the configsmb will be
 }
 ```
 
-### User data structure <a name="User_data"></a>
+### Config in group data<a name="Config_in_group_data"></a>
+- Which storage configs that a group can access is stored in group data's extension field. For example, a group that can access STORAGE_CONFIG is like following:
 ```json
 {
-	"servers": [
-		"servername",
-	]
+	"groupname": "groupname",
+	"externalName": "externalName",
+	"description": "description",
+	"extension": {
+		"acls": {
+			"admin": false,
+			"virtualClusters": [],
+			"storageConfigs": ["STORAGE_CONFIG"]
+		}
+	}
 }
 ```
diff --git a/contrib/storage_plugin/storagectl.md b/contrib/storage_plugin/storagectl.md
@@ -18,10 +18,10 @@ A tool to manage your storage config.
     - [ List config ](#Config_list) 
     - [ Delete config ](#Config_delete) 
 
-- [ Manage user ](#User_config)
-    - [ Set user ](#User_set)
-    - [ List user ](#User_list) 
-    - [ Delete user ](#User_delete) 
+- [ Manage group storage access ](#Groupsc_config)
+    - [ Add group storage config ](#Groupsc_add)
+    - [ List group storage configs ](#Groupsc_list) 
+    - [ Delete group storage config ](#Groupsc_delete) 
 
 
 ## Manage Server <a name="Server_config"></a> 
@@ -95,20 +95,19 @@ python storagectl.py config delete CONFIG_NAME
 ```
 
 
-## Manage User Config <a name="User_config"></a> 
-Manage PAI user's specific servers.
-### Set user config <a name="User_set"></a> 
+## Manage User Config <a name="Groupsc_config"></a> 
+Manage PAI group's storage config access.
+### Add group storage config <a name="Groupsc_set"></a> 
 ```
-python storagectl.py user set USER_NAME SERVER_NAME_1 [SERVER_NAME_2 ...]
+python storagectl.py groupsc add GROUP_NAME CONFIG_NAME
 ```
 
-### List user config <a name="User_list"></a> 
+### List group storage config <a name="Groupsc_list"></a> 
 ```
-python storagectl.py user list [-n USER_NAME_1, USER_NAME_2 ...]
+python storagectl.py groupsc list GROUP_NAME
 ```
-- If -n specified, list certain users. Otherwise list all users.
 
-### Delete user config <a name="User_delete"></a> 
+### Delete group storage config <a name="Groupsc_delete"></a> 
 ```
-python storagectl.py user delete USER_NAME
+python storagectl.py groupsc delete GROUP_NAME CONFIG_NAME
 ```
diff --git a/contrib/storage_plugin/storagectl.py b/contrib/storage_plugin/storagectl.py
@@ -36,6 +36,8 @@
 
 from utils.storage_util import *
 
+import binascii
+
 logger = logging.getLogger(__name__)
 
 # Save server config to k8s secret
@@ -44,11 +46,10 @@ def save_secret(secret_name, name, content_dict):
     secret_dict[name] = base64.b64encode(json.dumps(content_dict))
     patch_secret(secret_name, secret_dict, "pai-storage")
 
-
 def show_secret(args):
     secret_data = get_secret(args.secret_name, "pai-storage")
     if secret_data is None:
-        logger.info("No secret found.")
+        logger.error("No secret found.")
     else:
         for key, value in secret_data.iteritems():
             if args.name is None or key in args.name:
@@ -97,7 +98,6 @@ def config_set(args):
     try:
         content_dict = dict()
         content_dict["name"] = args.name
-        content_dict["gpn"] = args.gpn
         content_dict["servers"] = args.servers
         content_dict["default"] = args.default
         if args.mount_info is not None:
@@ -117,13 +117,42 @@ def config_set(args):
     else:
         save_secret("storage-config", args.name, content_dict)
 
+def get_group_extension(group_name):
+    group_hex = binascii.hexlify(group_name)
+    secret_data = get_secret(group_hex, "pai-group")
+    if secret_data is None:
+        logger.error("No group found.")
+        return None
+    else:
+        extension = json.loads(base64.b64decode(secret_data["extension"]))
+        return extension
+
+def groupsc_add(args):
+    extension = get_group_extension(args.group_name)
+    if extension is not None:
+        storageConfigs = extension["acls"]["storageConfigs"]
+        if args.config_name not in storageConfigs:
+            storageConfigs.append(args.config_name)
+            secret_dict = dict()
+            secret_dict["extension"] = base64.b64encode(json.dumps(extension))
+            patch_secret(binascii.hexlify(args.group_name), secret_dict, "pai-group")
+            logger.info("Successfully added storage config to group!")
 
-def user_set(args):
-    content_dict = dict()
-    content_dict["upn"] = args.name
-    content_dict["servers"] = args.servers
-    save_secret("storage-user", args.name, content_dict)
+def groupsc_delete(args):
+    extension = get_group_extension(args.group_name)
+    if extension is not None:
+        storageConfigs = extension["acls"]["storageConfigs"]
+        if args.config_name in storageConfigs:
+            storageConfigs.remove(args.config_name)
+            secret_dict = dict()
+            secret_dict["extension"] = base64.b64encode(json.dumps(extension))
+            patch_secret(binascii.hexlify(args.group_name), secret_dict, "pai-group")
+            logger.info("Successfully deleted storage config from group!")
 
+def groupsc_list(args):
+    extension = get_group_extension(args.group_name)
+    if extension is not None:
+        print(extension["acls"]["storageConfigs"])
 
 def setup_logger_config(logger):
     """
@@ -201,9 +230,8 @@ def main():
     # ./storagectl.py config set CONFIG_NAME GROUP_NAME [-s SERVER_NAME_1 SERVER_NAME_2 ...] [-m MOUNT_POINT SERVER PATH]... [-d]
     config_set_parser = config_subparsers.add_parser("set")
     config_set_parser.add_argument("name", help="Config name")
-    config_set_parser.add_argument("gpn", help="Config group name")
     config_set_parser.add_argument("-s", "--server", dest="servers", nargs="+", help="-s SERVER_NAME_1 SERVER_NAME_2 ...")
-    config_set_parser.add_argument("-m", "--mountinfo", dest="mount_info", nargs=3, action="append", help="-m MOUNT_POINT SERVER PATH")
+    config_set_parser.add_argument("-m", "--mountinfo", dest="mount_info", nargs=3, action="append", help="-m MOUNT_POINT SERVER SUB_PATH")
     config_set_parser.add_argument("-d", "--default", action="store_true", help="Mount by default")
     config_set_parser.set_defaults(func=config_set)
     # ./storagectl.py config list [-n CONFIG_NAME_1, CONFIG_NAME_2 ...] [-g GROUP_NAME_1, GROUP_NAME_2 ...]
@@ -216,22 +244,23 @@ def main():
     config_del_parser.add_argument("name")
     config_del_parser.set_defaults(func=delete_secret, secret_name="storage-config")
 
-    # ./storagectl.py user ...
-    user_parser = subparsers.add_parser("user", description="Manage user", formatter_class=argparse.RawDescriptionHelpFormatter)
-    user_subparsers = user_parser.add_subparsers(help="Manage user")
-    # ./storagectl.py user set USER_NAME SERVER_NAME_1 [SERVER_NAME_2 ...]
-    user_set_default_parser = user_subparsers.add_parser("set")
-    user_set_default_parser.add_argument("name")
-    user_set_default_parser.add_argument("servers", nargs="+", help="")
-    user_set_default_parser.set_defaults(func=user_set)
-    # ./storagectl.py user list [-n USER_NAME_1, USER_NAME_2 ...]
-    user_list_parser = user_subparsers.add_parser("list")
-    user_list_parser.add_argument("-n", "--name", dest="name", nargs="+", help="filter result by names")
-    user_list_parser.set_defaults(func=show_secret, secret_name="storage-user")
-    # ./storagectl.py user delete USER_NAME
-    user_del_parser = user_subparsers.add_parser("delete")
-    user_del_parser.add_argument("name")
-    user_del_parser.set_defaults(func=delete_secret, secret_name="storage-user")
+    # ./storagectl.py groupsc add|delete|list
+    groupsc_parser = subparsers.add_parser("groupsc", description="Manage group storage config", formatter_class=argparse.RawDescriptionHelpFormatter)
+    groupsc_subparsers = groupsc_parser.add_subparsers(help="Manage group storage config")
+    # ./storagectl.py groupsc add GROUP_NAME STORAGE_CONFIG_NAME
+    groupsc_add_parser = groupsc_subparsers.add_parser("add")
+    groupsc_add_parser.add_argument("group_name")
+    groupsc_add_parser.add_argument("config_name")
+    groupsc_add_parser.set_defaults(func=groupsc_add)
+    # ./storagectl.py groupsc delete GROUP_NAME STORAGE_CONFIG_NAME
+    groupsc_delete_parser = groupsc_subparsers.add_parser("delete")
+    groupsc_delete_parser.add_argument("group_name")
+    groupsc_delete_parser.add_argument("config_name")
+    groupsc_delete_parser.set_defaults(func=groupsc_delete)
+    # ./storagectl.py groupsc list GROUP_NAME
+    groupsc_list_parser = groupsc_subparsers.add_parser("list")
+    groupsc_list_parser.add_argument("group_name")
+    groupsc_list_parser.set_defaults(func=groupsc_list)
 
     args = parser.parse_args()
     args.func(args)