Add docs framework for how to clone a new DANDI instance #104
Conversation
Build out docs for Linc clone
Include more specific deployment token
Trivial env change
|
@aaronkanzer Why do the instructions say to create an account on PyPI? That should only be done if you're planning to release packages on PyPI, which has nothing to do with interacting with DANDI. |
Hi @jwodder, These instructions are meant for the developers of the data archive and associated tools, especially for developing a new DANDI-like ecosystem which we are doing for the LINC project. Since the DANDI CLI and Python API are a method of interacting with the archive, Aaron added instructions here for releasing the Python package to PyPI. Hope this helps to answer your question. |
Hi @jwodder, we are releasing a clone of the |
|
@satra @waxlamp @jwodder @jjnesbitt @mvandenburgh @asmacdo -- just wanted to bump this, any chance a quick read-through, feedback could occur? The outcomes of this handbook will help inform what we automate/abstract into infra-as-code vs. what remains manual, thus any feedback is greatly appreciated |
docs/60_initialize_vendors.md
Outdated
| • **Heroku** | ||
|
|
||
| • **AWS** | ||
|
|
||
| • **GitHub** | ||
|
|
||
| • **Terraform Cloud** | ||
|
|
||
| • **Netlify** | ||
|
|
||
| • **Sentry** | ||
|
|
||
| • **PyPI** |
There was a problem hiding this comment.
i think it would be good to add what each of these services provide to the infrastructure.
There was a problem hiding this comment.
Thanks Satra -- included a brief blurb for what each service is responsible for
| style="width: 60%; height: auto; display: block; margin-left: auto; margin-right: auto;"/> | ||
| <br/><br/> | ||
|
|
||
| Keep this value for further steps. |
There was a problem hiding this comment.
It would be valuable to know what sizes of instances we have for DANDI and LINC to guide installations of other instances.
There was a problem hiding this comment.
Sure -- I can add, they are defined in the DANDI Infra api.tf Girder extension here: https://github.com/dandi/dandi-infrastructure/blob/master/terraform/api.tf#L14-L18
On this note, has any stress-testing ever been done to evaluate if these worker sizes are appropriate or not for DANDI?
| style="width: 60%; height: auto; display: block; margin-left: auto; margin-right: auto;"/> | ||
| <br/><br/> | ||
|
|
||
| Your frontend should be able to deploy to an auto-generated URL via Netlify now! Steps for domain management and configuration are described further in the [Frontend Deployment](../64_dandi_archive/#frontend-deployment) section of these within the DANDI Archive setup. |
There was a problem hiding this comment.
Question came up on knowing how many minutes is needed by netlify for DANDI instance.
There was a problem hiding this comment.
When you say minutes in this context for Netlify, do you mean "build minutes"? (e.g. how long Netlify runners are required to run to deploy?) Or something else?
| First go to your `Account Settings`: | ||
|
|
||
| <br/><br/> | ||
| <img |
There was a problem hiding this comment.
This image was a bit mis-leading when following the instructions. The image led us to believe we needed to go to our Heroku App -> Settings, however what we ended up needing to do was the following:
- Click the Profile icon in the top right
- Select "Account Settings" in the dropdown
- Scroll to find the "API Key" section
There was a problem hiding this comment.
Let's add docs on where to update the following components for the web app:
- Logos
- Favicon
- Central text
- Footer text/links
- Top navigation links
There was a problem hiding this comment.
Let's add a section on how to handle database migrations.
| For data management (predominately `upload`, `download` and `validation` of data | ||
| to/from DANDI), a local CLI (command line interface) is used. |
There was a problem hiding this comment.
| For data management (predominately `upload`, `download` and `validation` of data | |
| to/from DANDI), a local CLI (command line interface) is used. | |
| Data management including upload, download, and validation is handled through a local DANDI Client. The DANDI Client provides both a command line interface (CLI) and Python API. |
| f"http://{instancehost}:8085", | ||
| f"http://{instancehost}:8000/api", | ||
| ), | ||
| "<your-dandi-clone>": DandiInstance( # Your own "dandi"" |
There was a problem hiding this comment.
| "<your-dandi-clone>": DandiInstance( # Your own "dandi"" | |
| "<your-dandi-clone>": DandiInstance( # Your own "dandi" |
|
|
||
| Resources | ||
|
|
||
| • [Source code and instructions]( https://github.com/dandi/dandi-hub) |
There was a problem hiding this comment.
| • [Source code and instructions]( https://github.com/dandi/dandi-hub) | |
| • [Source code and instructions](https://github.com/dandi/dandi-hub) |
| - Getting Started: "59_getting_started_replicating_dandi.md" | ||
| - Initialize Vendor Accounts for DANDI: "60_initialize_vendors.md" | ||
| - DANDI Authentication: "61_dandi_authentication.md" | ||
| - DANDI CLI: "62_dandi_cli.md" |
There was a problem hiding this comment.
| - DANDI CLI: "62_dandi_cli.md" | |
| - DANDI Client: "62_dandi_cli.md" |
| @@ -40,6 +40,14 @@ nav: | |||
| - Notes: "40_development.md" | |||
| - REST API Swagger: https://api.dandiarchive.org/swagger | |||
| - REST API Redoc: https://api.dandiarchive.org/redoc | |||
| - Create DANDI Instance: | |||
| - Getting Started: "59_getting_started_replicating_dandi.md" | |||
There was a problem hiding this comment.
With this change, when a user selects Create DANDI Instance they will be directed to the "59_getting_started_replicating_dandi.md" page.
| - Getting Started: "59_getting_started_replicating_dandi.md" | |
| - "59_getting_started_replicating_dandi.md" |
| The series of docs in this directory define how to create your own DANDI ecosystem. | ||
|
|
There was a problem hiding this comment.
| The series of docs in this directory define how to create your own DANDI ecosystem. |
| For data management (predominately `upload`, `download` and `validation` of data | ||
| to/from DANDI), a local CLI (command line interface) is used. | ||
|
|
||
| ## Referencing your API |
There was a problem hiding this comment.
| ## Referencing your API | |
| ## Update the DANDI Client to reference your API |
| ## Handling Versioning | ||
|
|
||
| DANDI CLI leverages a tool called [versioneer](https://pypi.org/project/versioneer/) for semantic versioning in PyPI. | ||
|
|
||
| Upon merging of a PR into `main`, if a given GitHub label is attached to the PR (`major`, `minor` or `patch` specifically) | ||
| `versioneer` will generate a human-readable CHANGELOG entry, and then push to PyPI the proper new semantic version. | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
There was a problem hiding this comment.
I would suggest removing the documentation on versioneer and just linking to the developer instructions, which includes these details.
| ## Handling Versioning | |
| DANDI CLI leverages a tool called [versioneer](https://pypi.org/project/versioneer/) for semantic versioning in PyPI. | |
| Upon merging of a PR into `main`, if a given GitHub label is attached to the PR (`major`, `minor` or `patch` specifically) | |
| `versioneer` will generate a human-readable CHANGELOG entry, and then push to PyPI the proper new semantic version. | |
| ## Resources | |
| - [PyPI package](https://pypi.org/project/dandi/) | |
| - [Source code](https://github.com/dandi/dandi-cli) | |
| - [Developer instructions](https://github.com/dandi/dandi-cli/blob/master/DEVELOPMENT.md) |
| @@ -0,0 +1,57 @@ | |||
| For data management (predominately `upload`, `download` and `validation` of data | |||
| to/from DANDI), a local CLI (command line interface) is used. | |||
|
|
|||
There was a problem hiding this comment.
| For your DANDI Archive clone, you do not need to create a clone of the DANDI Client. By following the instructions below, your users will be able to utilize the DANDI Client to interface with your DANDI Archive instance. | |
| **Note**: Users will be prompted for a `DANDI_API_KEY` | ||
| env. var [see here for code reference](https://github.com/dandi/dandi-cli/blob/6aa414c4db47394970f586cc4fb9758a634aef87/dandi/dandiapi.py#L492-L499) | ||
|
|
||
| You do not need to create a unique value here -- a user can just reference the `DANDI_API_KEY` their API issues in their respective clone. |
There was a problem hiding this comment.
| **Note**: Users will be prompted for a `DANDI_API_KEY` | |
| env. var [see here for code reference](https://github.com/dandi/dandi-cli/blob/6aa414c4db47394970f586cc4fb9758a634aef87/dandi/dandiapi.py#L492-L499) | |
| You do not need to create a unique value here -- a user can just reference the `DANDI_API_KEY` their API issues in their respective clone. | |
| ## Access credentials | |
| Users will be prompted for a `DANDI_API_KEY` | |
| environment variable. This variable does not need to be unique to your DANDI clone. A user can just set their `DANDI_API_KEY` to the value that your DANDI API clone issues. See docs on [storing access credentials](https://www.dandiarchive.org/handbook/13_upload/#storing-access-credentials). |
| } | ||
| ``` | ||
|
|
||
| Once your DANDI clone is added to list of available `DandiInstance` objects, you should be able to perform operations via the `-i` flag -- for example: |
There was a problem hiding this comment.
| Once your DANDI clone is added to list of available `DandiInstance` objects, you should be able to perform operations via the `-i` flag -- for example: | |
| Once your DANDI clone is added to the list of available `DandiInstance` objects, you should be able to perform operations via the `-i` flag. For example: |
|
|
||
| • **AWS** | ||
|
|
||
| Provides storage buckets, as well as domain management, for resources across the DANDI ecosystem |
There was a problem hiding this comment.
| Provides storage buckets, as well as domain management, for resources across the DANDI ecosystem | |
| Provides storage buckets, as well as domain management, for resources across the DANDI ecosystem. As well as the services (Kubernetes, etc.) for deploying the JupyterHub. |
Looking for review for now, no need to merge
These documents provide a step-by-step process if another user would like to launch their own Dandi-like ecosystem
please see here if you'd like to observe a live link: https://aquamarine-profiterole-e20e84.netlify.app/
or specifically:
https://lincbrain.github.io/handbook/40_initialization/