diff --git a/docs/_includes/menu.md b/docs/_includes/menu.md index 30b6a48c4c..c894159ad9 100644 --- a/docs/_includes/menu.md +++ b/docs/_includes/menu.md @@ -27,7 +27,6 @@ Installing - [Configuration Guide](configuration.html) - [Single Sign On Authentication](simpleSAMLphp.html) - [LDAP Authentication](simpleSAMLphp-ldap.html) - - [Storage (Beta) Guide](storage.html) - [Database Guide](databases.html) - [Upgrade Guide](upgrade.html) @@ -44,6 +43,7 @@ Using - [Frequently Asked Questions](faq.html) - [Troubleshooting](troubleshooting.html) - [Cloud Metrics (Beta)](cloud.html) +- [Storage Metrics (Beta)](storage.html) Resource Manager Notes ---------------------- diff --git a/docs/storage.md b/docs/storage.md index a7b7f7047b..c6e175dfeb 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -1,12 +1,72 @@ --- -title: Storage (Beta) +title: Storage Metrics (Beta) --- +The Storage Realm provides metrics relating to storage subsystems including +local disk and network attached storage. Each individual storage subsystem is +treated as a separate storage resource (e.g., GPFS, Isilon, NFS, Lustre, etc.) +This provides a mechanism for tracking utilization for a single storage +resource as well as an aggregate across resources and allows for viewing this +data by mount point, department, PI/project, and user. In addition, storage +metrics can be plotted alongside data from other realms such as Job Accounting +and Job Performance. + +The data currently required to provide these metrics is described below in +Input Format and are typically collected from the quota system on these storage +resource. Detailed information such as access and modification times of +individual files is not currently supported as the collection of this +information is meta-data intensive and can adversely affect the performance of +the filesystem. + **NOTE: Storage metrics are currently considered beta quality.** These instructions use the file paths from the RPM installation. If you've installed from source they will need to be adjusted accordingly. +## Input Format + +Storage data must be formatted in JSON files. These files will be validated +against the JSON Schema +`/etc/xdmod/etl/etl_schemas.d/storage/usage.schema.json`. + +**NOTE**: The thresholds and usage numbers are all measured in bytes. +Mountpoint names are currently limited to 255 characters. + +### Input Fields + +- `resource` - Storage resource name. +- `mountpoint` - File system mountpoint. +- `user` - User system username. +- `pi` - PI system username. +- `dt` - Date and time data was collected. Must be in RFC 3339 format + (e.g. `2017-01-01T00:00:00Z`). Must be UTC. +- `soft_threshold` - Quota soft threshold measured in bytes. +- `hard_threshold` - Quota hard threshold measured in bytes. +- `file_count` - Number of files. +- `logical_usage` - Logical usage measured in bytes. +- `physical_usage` - Physical usage measured in bytes. + +### Example + +```json +[ + { + "resource": "nfs", + "mountpoint": "/home", + "user": "jdoe", + "pi": "pi_username", + "dt": "2017-01-01T00:00:00Z", + "soft_threshold": 1000000, + "hard_threshold": 1200000, + "file_count": 10000, + "logical_usage": 100000, + "physical_usage": 100000 + }, + + ... +] +``` + ## Setup ### Add Storage Resource @@ -17,7 +77,7 @@ select a storage resource type or you will need to manually add the roles configuration as described in the following section. The resource name (also referred to as the resource code; not the formal name) -must then be used in the JSON storage input files described below. +must then be used in the JSON storage input files described above. ### Update Roles Configuration @@ -128,30 +188,6 @@ After adding the file you must update the ACL database tables: $ acl-config && acl-import ``` -## Input Format - -**NOTE**: The thresholds and usage numbers are all measured in bytes. -Mountpoint names are currently limited to 255 characters. - -```json -[ - { - "resource": "nfs", - "mountpoint": "/home", - "user": "jdoe", - "pi": "pi_username", - "dt": "2017-01-01T00:00:00", - "soft_threshold": 1000000, - "hard_threshold": 1200000, - "file_count": 10000, - "logical_usage": 100000, - "physical_usage": 100000 - }, - - ... -] -``` - ## Data Ingestion All of the following commands must be executed in the order specified below to