diff --git a/docs/bespoke/eddash/index.md b/docs/bespoke/eddash/index.md deleted file mode 100644 index 89c00f52d..000000000 --- a/docs/bespoke/eddash/index.md +++ /dev/null @@ -1,7 +0,0 @@ -# EIDFWorkshops - -[Ed-DaSH Notebook Service](./workshops/) - -[Ed-DaSH Virtual Machines](./safe-registration/) - -[JupyterHub Notebook Service Access](./jhub-git/) diff --git a/docs/bespoke/eddash/jhub-git.md b/docs/bespoke/eddash/jhub-git.md deleted file mode 100644 index c62e9c6a1..000000000 --- a/docs/bespoke/eddash/jhub-git.md +++ /dev/null @@ -1,45 +0,0 @@ -# EIDF JupyterHub Notebook Service Access - -Using the EIDF JupyterHub, users can access a range of services including standard interactive Python notebooks as well -as RStudio Server. - -## Ed-DaSH Workshops - -### Accessing - -In order to access the EIDF JupyterHub, authentication is through GitHub, so you must have an account on -[https://github.com](https://github.com) and that account must be a member of the appropriate organization in GitHub. -Please ask your project admin or workshop instructor for the workshop GitHub organization details. Please follow the -relevant steps listed below to prepare. - -1. If you do not have a GitHub account associated with the email you registered for the workshop with, follow the steps - described in [Step 1: Creating a GitHub Account](#step-1-creating-a-github-account) -1. If you do already have a GitHub account associated with the email address you registered for the workshop with, - follow the steps described in - [Step 2: Registering with the Workshop GitHub Organization](#step-2-registering-with-the-workshop-github-organization) - -#### Step 1: Creating a GitHub Account - -1. Visit [https://github.com/signup](https://github.com/signup) in your browser -1. Enter the email address that you used to register for the workshop -1. Complete the remaining steps of the GitHub registration process -1. Send an email to [ed-dash-support@mlist.is.ed.ac.uk](mailto:ed-dash-support@mlist.is.ed.ac.uk) from your GitHub - registered email address, including your GitHub username, and ask for an invitation to the workshop GitHub organization -1. Wait for an email from GitHub inviting you to join the organization, then follow the steps in - [Step 2: Registering with the Workshop GitHub Organization](#step-2-registering-with-the-workshop-github-organization) - -#### Step 2: Registering With the Workshop GitHub Organization - -1. If you already have a GitHub account associated with the email address that you registered for the workshop with, you - should have received an email inviting you to join the relevant GitHub organization. If you have not, email - [ed-dash-support@mlist.is.ed.ac.uk](mailto:ed-dash-support@mlist.is.ed.ac.uk) from your GitHub registered email - address, including your GitHub username, and ask for an invitation to the workshop GitHub organization -1. Once you have been invited to the GitHub organization, you will receive an email with the invitation; click on the - ![GitHub-Org-Button](/eidf-docs/images/access/github-btn.png){: class="inline-img"} button as shown - ![GitHub-Invitation](/eidf-docs/images/access/github-invitation.png){: class="border-img center"} - *Invitation to join the workshop GitHub organization* -1. Clicking on the button in the email will open a new web page with another form as shown below - ![GitHub-Invitation-2](/eidf-docs/images/access/github-invitation-2.png){: class="border-img center"} - *Form to accept the invitation to join the GitHub organization* -1. Again, click on the ![Join-GitHub-Org-Button](/eidf-docs/images/access/join-btn.png){: class="inline-img"} button to - confirm, then the Ed-DaSH-Training GitHub organization page will open diff --git a/docs/bespoke/eddash/safe-registration.md b/docs/bespoke/eddash/safe-registration.md deleted file mode 100644 index cbba5e75a..000000000 --- a/docs/bespoke/eddash/safe-registration.md +++ /dev/null @@ -1,42 +0,0 @@ -# Accessing - -In order to access the EIDF VDI and connect to EIDF data science cloud VMs, you need to have an active -[SAFE](https://safe.epcc.ed.ac.uk) account. If you already have a SAFE account, you can skip ahead to the -[Request Project Membership](#step-2-request-project-membership) instructions. Otherwise, follow the -[Register Account in EPCC SAFE](#step-1-register-account-in-epcc-safe) instructions immediately below to create the -account. - -!!! info - Please also see [Register](https://epcced.github.io/safe-docs/safe-for-users/#register) and - [Join a project](https://epcced.github.io/safe-docs/safe-for-users/#how-to-request-to-join-a-project) in the SAFE - documentation for more information. - -## Step 1: Register Account in EPCC SAFE - -1. Go to [SAFE signup](https://safe.epcc.ed.ac.uk/signup.jsp) and complete the registration form - 1. Mandatory fields are: Email, Nationality, First name, Last name, Institution for reporting, Department, and Gender - 1. Your Email should be the one you used to register for the EIDF service (or Ed-DaSH workshop) - 1. If you are unsure, enter 'University of Edinburgh' for Institution for reporting and 'EIDF' for Department - ![SAFE-signup-screenshot](/eidf-docs/images/SAFE_website_signup.png){: class="border-img"} - *SAFE registration form* -1. Submit the form, then accept the - [SAFE Acceptable Use policy](https://www.archer2.ac.uk/about/policies/safe_acceptable_use_policy.html) on the next - page - ![SAFE-policy-screenshot](/eidf-docs/images/SAFE_acceptable_use.png){: class="border-img"} - *SAFE User Access Agreement* -1. After you have completed the registration form and accepted the policy, you will receive an email from - [support@archer2.ac.uk](mailto:support@archer2.ac.uk) with a password reset URL -1. Visit the link in the email and generate a new password, then submit the form -1. You will now be logged into your new account in SAFE - -## Step 2: Request Project Membership - -1. While logged into SAFE, select the ‘Request Access’ menu item from the 'Projects' menu in the top menu bar -1. This will open the 'Apply for project membership' page -1. Enter the appropriate project ID into the ‘Project’ field and click the ‘Next’ button - ![SAFE-apply01-screenshot](/eidf-docs/images/SAFE_Apply_Project_01.png){: class="border-img"} - *Apply for project membership in SAFE* -1. In the 'Access route' drop down field that appears, select 'Request membership' (not 'Request machine account') - ![SAFE-apply01-screenshot](/eidf-docs/images/SAFE_Apply_Project_02.png){: class="border-img"} - *Request project membership in SAFE* -1. The project owner will then receive notification of the application and accept your request diff --git a/docs/bespoke/eddash/workshops.md b/docs/bespoke/eddash/workshops.md deleted file mode 100644 index 1f6c3ff78..000000000 --- a/docs/bespoke/eddash/workshops.md +++ /dev/null @@ -1,80 +0,0 @@ -# Workshop Setup - -**Please follow the instructions in [JupyterHub Notebook Service Access](../jhub-git/#ed-dash-workshops) -to arrange access to the EIDF Notebook service before continuing. The table below provides the login URL and the relevant -GitHub organization to register with.** - -| Workshop | Login URL | GitHub Organization | -|----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------| -| [Ed-DaSH Introduction to Statistics](https://edcarp.github.io/2022-05-03_ed-dash_intro-statistics) | [https://secure.epcc.ed.ac.uk/ed-dash-hub](https://secure.epcc.ed.ac.uk/ed-dash-hub) | [Ed-DaSH-Training](https://github.com/Ed-DaSH-Training) | -| [Ed-DaSH High-Dimensional Statistics](https://edcarp.github.io/2022-05-17_ed-dash_high-dim-stats) | [https://secure.epcc.ed.ac.uk/ed-dash-hub](https://secure.epcc.ed.ac.uk/ed-dash-hub) | [Ed-DaSH-Training](https://github.com/Ed-DaSH-Training) | -| [Ed-DaSH Introduction to Machine Learning with Python](https://edcarp.github.io/2022-08-23_ed-dash_machine_learning/) | [https://secure.epcc.ed.ac.uk/ed-dash-hub](https://secure.epcc.ed.ac.uk/ed-dash-hub) | [Ed-DaSH-Training](https://github.com/Ed-DaSH-Training) | -| [N8 CIR Introduction to Artificial Neural Networks in Python](https://n8cir.org.uk/events/introduction-machine-learning-python/) | [https://secure.epcc.ed.ac.uk/ed-dash-hub](https://secure.epcc.ed.ac.uk/ed-dash-hub) | [Ed-DaSH-Training](https://github.com/Ed-DaSH-Training) | - -Please follow the sequence of instructions described in the sections below to get ready for the workshop: - -1. [Step 1: Accessing the EIDF Notebook Service for the First Time](#step-1-accessing-the-eidf-notebook-service-for-the-first-time) -1. [Step 2: Login to EIDF JupyterHub](#step-2-login-to-the-eidf-notebook-service) -1. [Step 3: Creating a New R Script](#step-3-creating-a-new-r-script) - -## Step 1: Accessing the EIDF Notebook Service for the First Time - -We will be using the Notebook service provided by the [Edinburgh International Data Facility -(EIDF)](https://www.ed.ac.uk/edinburgh-international-data-facility). Follow the steps listed below to gain -access. - -* Visit [https://secure.epcc.ed.ac.uk/ed-dash-hub](https://secure.epcc.ed.ac.uk/ed-dash-hub) in your browser - -!!! warning - If you are receiving an error response such as '403: Forbidden' when you try to access - [https://secure.epcc.ed.ac.uk/ed-dash-hub](https://secure.epcc.ed.ac.uk/ed-dash-hub), please send an email to - [ed-dash-support@mlist.is.ed.ac.uk](mailto:ed-dash-support@mlist.is.ed.ac.uk) to request access and also include - your IP address which you can find by visiting [https://whatismyipaddress.com/](https://whatismyipaddress.com/) in - your browser. Please be aware that if you are accessing the service from outside of the UK, your access might be - blocked until you have emailed us with your IP address. - -1. Click on the ![GitHub-Signin-Button](/eidf-docs/images/access/github-signin-btn.png){: class="inline-img"} button -1. You will be asked to sign in to GitHub, as shown in the form below - ![GitHub-Signin](/eidf-docs/images/access/github-signin.png){: class="border-img center"} - *GitHub sign in form for access to EIDF Notebook Service* -1. Enter your GitHub credentials, or click on the ‘Create an account’ link if you do not already have one, and follow - the prerequisite instructions to register with GitHub and join the workshop organization -1. Click on the ‘Sign in’ button -1. On the next page, you will be asked whether to authorize the workshop organization to access your GitHub account as - shown below - ![GitHub-Authorize](/eidf-docs/images/access/github-authorize.png){: class="border-img center"} - *GitHub form requesting authorization for the workshop organization* -1. Click on the ![GitHub-Authorize-Button](/eidf-docs/images/access/authorize-btn.png){: class="inline-img"} button -1. At this point, you will receive an email to the email address that you registered with in GitHub, stating that - “A third-party OAuth application has been added to your account” for the workshop -1. If you receive a ‘403 : Forbidden’ error message on the next screen (if you did not already do so as in step 4 of the - prerequisites section) send an email to [ed-dash-support@mlist.is.ed.ac.uk](mailto:ed-dash-support@mlist.is.ed.ac.uk) - from your GitHub registered email address, including your GitHub username, and ask for an invitation to the workshop - organization. Otherwise, skip to the next step. **N.B. If you are accessing the service from outside of the UK, you - may see this error; if so, please contact - [ed-dash-support@mlist.is.ed.ac.uk](mailto:ed-dash-support@mlist.is.ed.ac.uk) to enable access** -1. If you receive a ‘400 : Bad Request’ error message, you need to accept the invitation that has been emailed to you - to join the workshop organization as in the prerequisite instructions - -## Step 2: Login to the EIDF Notebook Service - -Now that you have completed registration with the workshop GitHub organization, you can access the workshop RStudio -Server in EIDF. - -1. Return to the [https://secure.epcc.ed.ac.uk/ed-dash-hub](https://secure.epcc.ed.ac.uk/ed-dash-hub) -1. You will be presented with a choice of server as a list of radio buttons. Select the appropriate one as labelled for - your workshop and press the orange 'Start' button -1. You will now be redirected to the hub spawn pending page for your individual server instance -1. You will see a message stating that your server is launching. If the page has not updated after 10 seconds, simply - refresh the page with the <CTRL> + R or <F5> keys in Windows, or <CMD> + R in macOS -1. Finally, you will be redirected to either the RStudio Server if it's a statistics workshop, or the Jupyter Lab - dashboard otherwise, as shown in the screenshots below - ![RStudio-Server-Screen](/eidf-docs/images/access/rstudio-server-screen.png){: class="border-img center"} - *The RStudio Server UI* - ![Jupyter-Lab-Dashboard](/eidf-docs/images/access/jupyterlab-dashboard.png){: class="border-img center"} - *The Jupyter Lab Dashboard* - -## Step 3: Creating a New R Script - -Follow these [quickstart instructions](/eidf-docs/services/rstudioserver/quickstart#creating-a-new-r-script) to create your first -R script in RStudio Server! diff --git a/docs/bespoke/index.md b/docs/bespoke/index.md deleted file mode 100644 index d3249a78d..000000000 --- a/docs/bespoke/index.md +++ /dev/null @@ -1,3 +0,0 @@ -# Bespoke Services - -[Ed-DaSH](./eddash/) diff --git a/docs/images/access/authorize-btn.png b/docs/images/access/authorize-btn.png deleted file mode 100644 index ed878823a..000000000 Binary files a/docs/images/access/authorize-btn.png and /dev/null differ diff --git a/docs/images/access/github-authorize.png b/docs/images/access/github-authorize.png deleted file mode 100644 index ea7e1c84f..000000000 Binary files a/docs/images/access/github-authorize.png and /dev/null differ diff --git a/docs/images/access/github-btn.png b/docs/images/access/github-btn.png deleted file mode 100644 index ccc32939d..000000000 Binary files a/docs/images/access/github-btn.png and /dev/null differ diff --git a/docs/images/access/github-invitation-2.png b/docs/images/access/github-invitation-2.png deleted file mode 100644 index 5b1ae923b..000000000 Binary files a/docs/images/access/github-invitation-2.png and /dev/null differ diff --git a/docs/images/access/github-invitation.png b/docs/images/access/github-invitation.png deleted file mode 100644 index 52e3f7bb4..000000000 Binary files a/docs/images/access/github-invitation.png and /dev/null differ diff --git a/docs/images/access/github-signin-btn.png b/docs/images/access/github-signin-btn.png deleted file mode 100644 index 9d3afdf42..000000000 Binary files a/docs/images/access/github-signin-btn.png and /dev/null differ diff --git a/docs/images/access/github-signin.png b/docs/images/access/github-signin.png deleted file mode 100644 index b0eb1178b..000000000 Binary files a/docs/images/access/github-signin.png and /dev/null differ diff --git a/docs/images/access/join-btn.png b/docs/images/access/join-btn.png deleted file mode 100644 index b82b140d8..000000000 Binary files a/docs/images/access/join-btn.png and /dev/null differ diff --git a/docs/images/access/jupyterhub-dashboard.png b/docs/images/access/jupyterhub-dashboard.png new file mode 100644 index 000000000..a4a7b8561 Binary files /dev/null and b/docs/images/access/jupyterhub-dashboard.png differ diff --git a/docs/images/access/jupyterhub-login.png b/docs/images/access/jupyterhub-login.png new file mode 100644 index 000000000..b28ef13af Binary files /dev/null and b/docs/images/access/jupyterhub-login.png differ diff --git a/docs/images/access/jupyterhub-select-server.png b/docs/images/access/jupyterhub-select-server.png new file mode 100644 index 000000000..9ff551222 Binary files /dev/null and b/docs/images/access/jupyterhub-select-server.png differ diff --git a/docs/images/access/jupyterhub-startup.png b/docs/images/access/jupyterhub-startup.png new file mode 100644 index 000000000..e0022d842 Binary files /dev/null and b/docs/images/access/jupyterhub-startup.png differ diff --git a/docs/images/access/jupyterlab-dashboard.png b/docs/images/access/jupyterlab-dashboard.png deleted file mode 100644 index a83adf08c..000000000 Binary files a/docs/images/access/jupyterlab-dashboard.png and /dev/null differ diff --git a/docs/images/access/portal-s3-accounts.png b/docs/images/access/portal-s3-accounts.png new file mode 100644 index 000000000..10fef7b93 Binary files /dev/null and b/docs/images/access/portal-s3-accounts.png differ diff --git a/docs/images/access/portal-s3-keys.png b/docs/images/access/portal-s3-keys.png new file mode 100644 index 000000000..5b58ca07b Binary files /dev/null and b/docs/images/access/portal-s3-keys.png differ diff --git a/docs/images/access/rstudio-server-screen-2.png b/docs/images/access/rstudio-server-screen-2.png deleted file mode 100644 index 440ae52f7..000000000 Binary files a/docs/images/access/rstudio-server-screen-2.png and /dev/null differ diff --git a/docs/images/access/rstudio-server-screen.png b/docs/images/access/rstudio-server-screen.png deleted file mode 100644 index 8ace0946b..000000000 Binary files a/docs/images/access/rstudio-server-screen.png and /dev/null differ diff --git a/docs/services/cs2/index.md b/docs/services/cs2/index.md index 9697583a0..8dbcad7b9 100644 --- a/docs/services/cs2/index.md +++ b/docs/services/cs2/index.md @@ -1,5 +1,5 @@ # Cerebras CS-2 -[Get Access](./access/) +[Get Access](./access.md) -[Running codes](./run/) +[Running codes](./run.md) diff --git a/docs/services/cs2/run.md b/docs/services/cs2/run.md index cf7f280ee..26885ecf5 100644 --- a/docs/services/cs2/run.md +++ b/docs/services/cs2/run.md @@ -6,7 +6,7 @@ The Cerebras CS-2 Wafer-scale cluster (WSC) uses the Ultra2 system as a host sys ## Connecting to the cluster -To gain access to the CS-2 WSC you need to login to the host system, Ultra2. See the [documentation for Ultra2](../ultra2/run.md#login). +To gain access to the CS-2 WSC you need to login to the host system, Ultra2. See the [documentation for Ultra2](../ultra2/connect.md#login). ## Running Jobs diff --git a/docs/services/datacatalogue/docs.md b/docs/services/datacatalogue/docs.md deleted file mode 100644 index 9c3cd3fbf..000000000 --- a/docs/services/datacatalogue/docs.md +++ /dev/null @@ -1,7 +0,0 @@ -# Service Documentation - -## Metadata - -For more information on metadata, please read the following: [Metadata](metadata.md) - -## Online support diff --git a/docs/services/datacatalogue/index.md b/docs/services/datacatalogue/index.md index a51cc9e97..be1db4494 100644 --- a/docs/services/datacatalogue/index.md +++ b/docs/services/datacatalogue/index.md @@ -1,9 +1,3 @@ # EIDF Data Catalogue Information -[QuickStart](./quickstart/) - -[Tutorial](./tutorial/) - -[Documentation](./docs/) - -[Metadata information](./metadata/) +[Metadata information](./metadata.md) diff --git a/docs/services/datacatalogue/quickstart.md b/docs/services/datacatalogue/quickstart.md deleted file mode 100644 index 05eb5293e..000000000 --- a/docs/services/datacatalogue/quickstart.md +++ /dev/null @@ -1,7 +0,0 @@ -# Quickstart - -## Accessing - -## First Task - -## Further information diff --git a/docs/services/datacatalogue/tutorial.md b/docs/services/datacatalogue/tutorial.md deleted file mode 100644 index 32fb20b5e..000000000 --- a/docs/services/datacatalogue/tutorial.md +++ /dev/null @@ -1,3 +0,0 @@ -# Tutorial - -## First Query diff --git a/docs/services/index.md b/docs/services/index.md index 462342a48..34145041f 100644 --- a/docs/services/index.md +++ b/docs/services/index.md @@ -2,18 +2,20 @@ ## Computing Services -[Data Science Virtual Desktops](./virtualmachines/) +[Data Science Virtual Desktops](./virtualmachines/index.md) -[Managed File Transfer](./mft/) +[Managed File Transfer](./mft/index.md) -[Managed JupyterHub](./jhub/) +[Notebooks](./jhub/index.md) -[Cerebras CS-2](./cs2/) +[Cerebras CS-2](./cs2/index.md) -[Ultra2](./ultra2/) +[Ultra2](./ultra2/index.md) -[Graphcore Bow Pod64](./graphcore/) +[Graphcore Bow Pod64](./graphcore/index.md) -## Data Management Services +## Data Services -[Data Catalogue](./datacatalogue/) +[S3](./s3/index.md) + +[Data Catalogue](./datacatalogue/index.md) diff --git a/docs/services/jhub/docs.md b/docs/services/jhub/docs.md deleted file mode 100644 index 7380953e9..000000000 --- a/docs/services/jhub/docs.md +++ /dev/null @@ -1,3 +0,0 @@ -# Service Documentation - -## Online support diff --git a/docs/services/jhub/index.md b/docs/services/jhub/index.md index 2afda1fe1..b8ba01aaa 100644 --- a/docs/services/jhub/index.md +++ b/docs/services/jhub/index.md @@ -1,7 +1,8 @@ -# EIDF Jupyterhub +# EIDF Notebook Service -[QuickStart](./quickstart/) +The EIDF Notebook Service is a scalable Jupyterhub deployment in the EIDF Data Science Cloud. -[Tutorial](./tutorial/) +The Notebook Service is open to all EIDF users and offers a selection of data science environments +and user interfaces, including Jupyter notebooks, Jupyter Lab and RStudio. -[Documentation](./docs/) +Follow [Quickstart](./quickstart.md) to start using the EIDF Notebook Service. diff --git a/docs/services/jhub/quickstart.md b/docs/services/jhub/quickstart.md index 05eb5293e..481cd0b18 100644 --- a/docs/services/jhub/quickstart.md +++ b/docs/services/jhub/quickstart.md @@ -2,6 +2,58 @@ ## Accessing -## First Task +Access the EIDF Notebooks in your browser by opening [https://notebooks.eidf.ac.uk/](https://notebooks.eidf.ac.uk/). You must be a member of an active EIDF project and have a user account to use the EIDF Notebook Service. -## Further information +![JupyterHub Login Page](../../images/access/jupyterhub-login.png) + +Click on "Sign In with SAFE". You will be redirected to the SAFE login page. + +Log into the SAFE if you're not logged in already. +If you have more than one account you will be presented with the form "Approve Token" and a choice of user accounts for the Notebook Service. +This account is the user in your notebooks and you can share data with your DSC VMs within the same project. + +Select the account you would like to use from the dropdown "User Account" at the end of the form. Then press "Accept" to return to the EIDF Notebook Service where you can select a server environment. + +![JupyterHub Login Page](../../images/access/jupyterhub-select-server.png) + +Select the environment that you would like to use for your notebooks and press "Start". Now your notebook container will be launched. This may take a little while. + +![JupyterHub Login Page](../../images/access/jupyterhub-startup.png) + +## First Notebook + +You will be presented with the JupyterLab dashboard view when the container has started. + +![JupyterHub Login Page](../../images/access/jupyterhub-dashboard.png) + +The availability of launchers depends on the environment that you selected. + +For example launch a Python 3 notebook or an R notebook from the dashboard. You can also launch a terminal session. + +## Python packages + +Note that Python packages are installed into the system space of your container by default. +However this means that they are not available after a restart of your notebook container which may happen when your session was idle for a while. +We recommend specifying `--user` to install packages into your user directory to preserve installations across sessions. + +To install python packages in a notebook use the command: + +```jupyter +!pip install --user +``` + +or run the command in a terminal: + +```bash +pip install --user +``` + +## Data + +There is a project space mounted in `/project_data`. Only project accounts have permissions to view and write to their project folder in this space. Here you can also share data with DSC VMs in your project. Please contact the helpdesk if you would like to mount this project space to one of your VMs. Data placed in `/project_data/shared` is shared with other notebook users outside your project. + +## Limits + +Note that there are limited amounts of memory and cores available per user. Users do not have sudo permissions in the containers so you cannot install any system packages. + +Currently there is no access to GPUs. You can submit jobs to the EIDF GPU Service but you cannot run your notebooks on a GPU. diff --git a/docs/services/jhub/tutorial.md b/docs/services/jhub/tutorial.md deleted file mode 100644 index a386958c1..000000000 --- a/docs/services/jhub/tutorial.md +++ /dev/null @@ -1,3 +0,0 @@ -# Tutorial - -## First notebook diff --git a/docs/services/rstudioserver/docs.md b/docs/services/rstudioserver/docs.md deleted file mode 100644 index 7380953e9..000000000 --- a/docs/services/rstudioserver/docs.md +++ /dev/null @@ -1,3 +0,0 @@ -# Service Documentation - -## Online support diff --git a/docs/services/rstudioserver/index.md b/docs/services/rstudioserver/index.md deleted file mode 100644 index 5e3f3a74f..000000000 --- a/docs/services/rstudioserver/index.md +++ /dev/null @@ -1,7 +0,0 @@ -# EIDF R Studio Server - -[QuickStart](./quickstart/) - -[Tutorial](./tutorial/) - -[Documentation](./docs/) diff --git a/docs/services/rstudioserver/quickstart.md b/docs/services/rstudioserver/quickstart.md deleted file mode 100644 index e188f5534..000000000 --- a/docs/services/rstudioserver/quickstart.md +++ /dev/null @@ -1,19 +0,0 @@ -# Quickstart - -## Accessing - -## First Task - -### Creating a New R Script - -Your RStudio Server session has been initialised now. If you are participating in a workshop, then all the packages and -data required for the workshop have been loaded into the workspace. All that remains is to create a new R script to contain -your code! - -1. In the RStudio Server UI, open the File menu item at the far left of the main menu bar at the top of the page -1. Hover over the ‘New File’ sub-menu item, then select ‘R Script’ from the expanded menu -1. A new window pane will appear in the UI as shown below, and you are now ready to start adding the R code to your script! - ![RStudio-Server-Screen-2](/eidf-docs/images/access/rstudio-server-screen-2.png){: class="border-img center"} - *RStudio Server UI screen with new script* - -## Further information diff --git a/docs/services/rstudioserver/tutorial.md b/docs/services/rstudioserver/tutorial.md deleted file mode 100644 index a386958c1..000000000 --- a/docs/services/rstudioserver/tutorial.md +++ /dev/null @@ -1,3 +0,0 @@ -# Tutorial - -## First notebook diff --git a/docs/services/s3/index.md b/docs/services/s3/index.md new file mode 100644 index 000000000..d59ee3f72 --- /dev/null +++ b/docs/services/s3/index.md @@ -0,0 +1,26 @@ +# Overview + +The EIDF S3 Service is an object store with an interface that is compatible with a subset of the Amazon S3 RESTful API. + +## Service Access + +Users should have an EIDF account as described in [EIDF Accounts](../../access/project.md). + +Project leads can request an object store allocation through a request to the EIDF helpdesk. + +## Access keys + +Select your project at [https://portal.eidf.ac.uk/project/](https://portal.eidf.ac.uk/project/). +Your access keys are displayed in the table at the top of the page. + +![Portal-S3-Access-Keys](../../images/access/portal-s3-keys.png){: class="border-img"} + +For each account, the quota and the number of buckets that it is permitted to create is shown, as well as the access keys. +Click on "Secret" to view the access secret. +You will need the access key, the corresponding access secret and the endpoint `https://s3.eidf.ac.uk` to connect to the EIDF S3 Service with an S3 client. + +## Further information + +[Access management](./manage.md): Project management guide to managing accounts and access permissions for your S3 allocation. + +[Tutorial](./tutorial.md): Examples using EIDF S3 diff --git a/docs/services/s3/manage.md b/docs/services/s3/manage.md new file mode 100644 index 000000000..7078e7f3b --- /dev/null +++ b/docs/services/s3/manage.md @@ -0,0 +1,71 @@ +# Manage EIDF S3 access + +Access keys and accounts for the object store are managed by project managers via the [EIDF Portal](https://portal.eidf.ac.uk/). + +## Request an allocation + +An object store allocation for a project may be requested by contacting the EIDF helpdesk. + +## Object store accounts + +Select your project at [https://portal.eidf.ac.uk/project/](https://portal.eidf.ac.uk/project/) +and jump to "S3 Allocation" on the project page to manage access keys and accounts. + +![Portal-S3-Access-Keys](../../images/access/portal-s3-accounts.png){: class="border-img"} + +S3 buckets and objects are owned by an account. +Each account has a quota for storage and the number of buckets that it can create. +The sum of all account quotas is limited by the total storage quota of the project object store allocation shown at the top. + +An account with the minimum storage quota (1B) and zero buckets is effectively read only +as it may not create new buckets and so cannot upload files. + +To create an account: + +1. On the project page scroll to "S3 Allocation" +1. Click "Add Account" +1. Enter: + * an account name (letters, numbers and underscore `_` only) + * a description + * a quota: a number with a unit B, kB, MiB (MB), GiB (GB), or TiB (TB), the minimum is 1B (1 Byte). + * the number of buckets that the account may create +1. Click "Create Account" + +You will not be allowed to create an account with the quota greater than the available storage quota of the project. + +It may take a little while for the account to become available. +Refresh the project page to update the list of accounts. + +## Access keys + +To use S3 (listing or creating buckets, listing objects or uploading and downloading files) you need an access key and a secret. +An account can own any number of access keys. +These keys share the account's quota and have access to the same buckets. + +To create an access key: + +1. Select an account and click "Add Key" +1. Click "Create Access Key" + +It can take a little while for the access keys to become available. +Refresh the project page to update the list of keys. + +## Access key permissions + +You can control which project members are allowed to view an access key and secret in the EIDF Portal or the SAFE. +Project managers and the PI have access to all S3 accounts and can view associated access keys and secrets in the project management view. + +To grant view permissions for an access key to a project member: + +1. Click on the "Edit" icon next to the key. +1. Select the project members that will have view permissions for this access key. +1. Press "Update Permissions" + +It can take a little while for the permissions update to complete. + +!!! note + Anyone who knows an access key and secret will be able to perform the associated activities via the S3 API regardless of the view permissions. + +## Delete an access key + +Click on the "Bin" icon next to a key and press "Delete" on the form. diff --git a/docs/services/s3/tutorial.md b/docs/services/s3/tutorial.md new file mode 100644 index 000000000..1cbf9acf7 --- /dev/null +++ b/docs/services/s3/tutorial.md @@ -0,0 +1,242 @@ +# Tutorial + +Buckets owned by an EIDF project are placed in a tenancy in the EIDF S3 Service. +The project code is a prefix on the bucket name, separated by a colon (`:`), for example `eidfXX1:somebucket`. +Note that some S3 client libraries do not accept bucket names in this format. + +## AWS CLI + +The following examples use the AWS Command Line Interface (AWS CLI) to connect to EIDF S3. + +### Setup + +Install with pip + +```bash +python -m pip install awscli +``` + +Installers are available for various platforms if you are not using Python: see [https://aws.amazon.com/cli/](https://aws.amazon.com/cli/) + +### Configure + +Set your access key and secret as environment variables or configure a credentials file at `~/.aws/credentials` on Linux or `%USERPROFILE%\.aws\credentials` on Windows. + +Credentials file: + +```ini +[default] +aws_access_key_id= +aws_secret_access_key= +``` + +Environment variables: + +```bash +export AWS_ACCESS_KEY_ID= +export AWS_SECRET_ACCESS_KEY= +``` + +The parameter `--endpoint-url https://s3.eidf.ac.uk` must always be set when calling a command. + +### Commands + +List the buckets in your account: + +```bash +aws s3 ls --endpoint-url https://s3.eidf.ac.uk +``` + +Create a bucket: + +```bash +aws s3api create-bucket --bucket --endpoint-url https://s3.eidf.ac.uk +``` + +Upload a file: + +```bash +aws s3 cp s3:// --endpoint-url https://s3.eidf.ac.uk +``` + +Check that the file above was uploaded successfully by listing objects in the bucket: + +```bash +aws s3 ls s3:// --endpoint-url https://s3.eidf.ac.uk +``` + +To read from a public bucket without providing credentials, add the option `--no-sign-request` to the call: + +```bash +aws s3 ls s3:// --no-sign-request --endpoint-url https://s3.eidf.ac.uk +``` + +## Python using `boto3` + +The following examples use the Python library `boto3`. + +### Install + +Installation: + +```bash +python -m pip install boto3 +``` + +### Usage + +By default, the `boto3` Python library raises an error that bucket names with a colon `:` (as used by the EIDF S3 Service) are invalid, so we have to switch off the bucket name validation: + +```python +import boto3 +from botocore.handlers import validate_bucket_name + +s3 = boto3.resource('s3', endpoint_url='https://s3.eidf.ac.uk') +s3.meta.client.meta.events.unregister('before-parameter-build.s3', validate_bucket_name) +``` + +List buckets: + +```python +for bucket in s3.buckets.all(): + print(f'{bucket.name}') +``` + +List objects in a bucket: + +```python +project_code = 'eidfXXX' +bucketname = 'somebucket' +bucket = s3.Bucket(f'{project_code}:{bucket_name}') +for obj in bucket.objects.all(): + print(f'{obj.key}') +``` + +Upload a file to a bucket: + +```python +bucket = s3.Bucket(f'{project_code}:{bucket_name}') +bucket.upload_file('./somedata.csv', 'somedata.csv') +``` + +## Access policies + +Bucket permissions use IAM policies. You can grant other accounts (within the same project or from other projects) read or write access to your buckets. +For example to grant permissions to put, get, delete and list objects in bucket `eidfXX1:somebucket` to the account `account2` in project `eidfXX2`: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "AllowAccessToBucket", + "Principal": { + "AWS": [ + "arn:aws:iam::eidfXX2:user/account2", + ] + }, + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:ListBucket", + "s3:DeleteObject", + ], + "Resource": [ + "arn:aws:s3:::/*", + "arn:aws:s3::eidfXX1:somebucket" + ] + } + ] +} +``` + +You can chain multiple policies in the statement array: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Principal": { ... } + "Effect": "Allow", + "Action": [ ... ], + "Resource": [ ... ] + }, + { + "Principal": { ... } + "Effect": "Allow", + "Action": [ ... ], + "Resource": [ ... ] + } + ] +} +``` + +Give public read access to a bucket (listing and downloading files): + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": "*", + "Action": ["s3:ListBucket"], + "Resource": [ + f"arn:aws:s3::eidfXX1:somebucket" + ] + }, + { + "Effect": "Allow", + "Principal": "*", + "Action": ["s3:GetObject"], + "Resource": [ + f"arn:aws:s3::eidfXX1:somebucket/*" + ] + } + ] +} +``` + +### Set policy using AWS CLI + +Grant permissions stored in an IAM policy file: + +```bash +aws put-bucket-policy --bucket --policy "$(cat bucket-policy.json)" +``` + +### Set policy using Python `boto3` + +Grant permissions to another account: In this example we grant `ListBucket` and `GetObject` permissions to account `account1` in project `eidfXX1` and `account2` in project `eidfXX2`. + +```python +import json + +bucket_policy = { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "AWS": [ + "arn:aws:iam::eidfXX1:user/account1", + "arn:aws:iam::eidfXX2:user/account2", + ] + }, + "Action": [ + "s3:ListBucket", + "s3:GetObject" + ], + "Resource": [ + f"arn:aws:s3::eidfXX1:{bucket_name}" + f"arn:aws:s3::eidfXX1:{bucket_name}/*" + ] + } + ] +} + +policy = bucket.Policy() +policy.put(Policy=json.dumps(bucket_policy)) +``` diff --git a/docs/services/ultra2/index.md b/docs/services/ultra2/index.md index 716250eca..6c1e848d6 100644 --- a/docs/services/ultra2/index.md +++ b/docs/services/ultra2/index.md @@ -1,5 +1,7 @@ # Ultra2 Large Memory System -[Overview](./access/) -[Connect](./connect/) -[Running jobs](./run/) +[Overview](./access.md) + +[Connect](./connect.md) + +[Running jobs](./run.md) diff --git a/docs/services/virtualmachines/quickstart.md b/docs/services/virtualmachines/quickstart.md index cbaf78b3d..1d815f80d 100644 --- a/docs/services/virtualmachines/quickstart.md +++ b/docs/services/virtualmachines/quickstart.md @@ -35,7 +35,7 @@ Now you have to wait for your PI or project manager to accept your request to jo with this account, and to change the password for this account. 1. Before you log in for the first time with a new user account, you must change your password as described - [below](../../services/virtualmachines/quickstart.md#set-or-change-the-password-for-your-user-account). + [below](../../services/virtualmachines/quickstart.md#set-or-change-the-password-for-a-user-account). 1. Follow the link to the Guacamole login or log in directly at [https://eidf-vdi.epcc.ed.ac.uk/vdi/](https://eidf-vdi.epcc.ed.ac.uk/vdi/). diff --git a/mkdocs.yml b/mkdocs.yml index d4788b3a8..cd1f30dec 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -50,10 +50,6 @@ nav: # - "Using the MFT": services/mft/using-the-mft.md # - "SFTP": services/mft/sftp.md - "Policies": services/virtualmachines/policies.md - # - "Managed JupyterHub": - # - "QuickStart": services/jhub/quickstart.md - # - "Tutorial": services/jhub/tutorial.md - # - "Documentation": services/jhub/docs.md - "Cerebras CS-2": - "Get Access": services/cs2/access.md - "Running codes": services/cs2/run.md @@ -79,7 +75,14 @@ nav: - "Profiling": services/graphcore/training/L3_profiling.md - "Other Frameworks": services/graphcore/training/L4_other_frameworks.md - "Graphcore FAQ": services/graphcore/faq.md - - "Data Management Services": + # - "Notebooks": + # - "Overview": services/jhub/index.md + # - "Quickstart": services/jhub/quickstart.md + - "Data Services": + - "S3": + - "Overview": services/s3/index.md + - "Manage": services/s3/manage.md + - "Tutorial": services/s3/tutorial.md - "Data Catalogue": - "Metadata information": services/datacatalogue/metadata.md - "Safe Haven Services": @@ -94,11 +97,6 @@ nav: - "Submitting Scripts to Slurm": safe-haven-services/superdome-flex-tutorial/L3_submitting_scripts_to_slurm.md - "Parallelised Python Analysis": safe-haven-services/superdome-flex-tutorial/L4_parallelised_python_analysis.md - "Parallelised R Analysis": safe-haven-services/superdome-flex-tutorial/L5_parallelised_r_analysis.md - - "Bespoke Services": - - "Ed-DaSH": - - "Workshop Notebook Service": bespoke/eddash/workshops.md - - "Workshop Virtual Machines": bespoke/eddash/safe-registration.md - - "JupyterHub Notebook Service Access": bespoke/eddash/jhub-git.md - "EIDF Known Issues": known-issues/index.md - "EIDF Frequently Asked Questions": faq/index.md