From 218c18d1fcc10985fa76ba781b6bae742626dc4f Mon Sep 17 00:00:00 2001 From: PGijsbers
You're looking to contribute to the AI-on-Demand metadata catalogue, that's great! +AI-on-Demand is an open project and we welcome contributions by all, no matter their experience or background.
+There are many types of contributions to be made to the metadata catalogue, and this section +covers a non-exhaustive list. If you find something that's not outlined here, just reach out to us!
+Good examples for contributions that do not require any coding include:
+You can help improve our documentation by correcting mistakes, such as typos, adding clarifications, +or even adding new sections. For small changes, documentation can be edited directly on GitHub by +finding the related markdown file and clicking the pencil icon in the top-right corner (✐).
+If you intend to make bigger changes, please first open a new issue the documents the suggested change. +That way, we can give you feedback before you write, and verify that it is actually a change we would be interested in adopting. +For big changes, we also recommend you to follow the instructions on "Setting up a development environment" +so that you can render the changed documentation locally.
+For code changes, please first co-ordinate with the developers on what you will work on. +You can do this by either leaving a comment on an existing issue that you would like to help with, +or by opening a new issue proposing your change. By first communicating with the developers, they +can let you know ahead of time whether or not the change is wanted, make sure they have time to +support you, and provide any feedback. We really want to avoid a scenario where you may work hard on a contribution +only to find out that it is not in line with the vision of the project and thus will not be accepted. +When starting your first code contribution, visit the "Setting up a development environment" +section for more information on how to get started.
+Helping people on the GitHub issue tracker just requires a GitHub account. +You can help by people by answering their questions, or weighing in on discussions. +Even if you do not have an answer, you can verify that you can reproduce the behavior they report or +ask clarifying questions to make their bug report better (see also "Reporting Bugs"). +This helps the core contributors resolve the issue with more ease and is hugely appreciated. +As always, please be kind and patient with others.
+When you find a bug and want to report it, the first step is to check that it has not already been reported. +Use the search functionality of the GitHub issue tracker to find identical or related issues. +If your issue or bug has already been reported, please do not open a new issue. +Instead, either "upvote" the original report by leaving a 👍 reaction or, +if you have additional information which may be relevant to the discussion, leave a comment on that issue.
+If your bug isn't reported yet, create a new issue. +Provide a clear explanation of the expected behavior and the observed behavior, and explain why you think this is a bug. +Add instructions on how to reproduce the expected behavior through a Minimal Reproducible Example. +Without it, it may be very hard for contributors to solve the issue (or may not even understand it).
+First, make sure you can get the local metadata catalogue up and running by following the "Hosting" instructions.
+During the installation step, use git
to clone the repository.
+If you have write access to this repository, you can follow the instruction as-is.
+If you do not have write access to this repository, you must fork it.
+After forking the repository, your clone command will be (substituting USERNAME
for your GitHub username):
git clone https://github.com/USERNAME/AIOD-rest-api.git
+
Always make sure to install your dependencies in a local environment, for example with the built in venv
module:
python -m pip venv venv
+source venv/bin/activate
+
and then install the Python dependencies
+python -m pip install -e ".[dev, docs]"
+
we install the optional dev
(developer) and docs
(documentation) dependencies so that we can
+build documentation and run tests locally.
It is also generally useful to set add an override.env
file to the project's root directory with
+the line USE_LOCAL_DEV=true
added. This will allow utility scripts ./scripts/up.sh
and ./scripts/down.sh
+to start docker containers in a way that reflects local changes.
See the "Developer Documentation" for the technical documentation of this project. +More to be added.
+ + + + + + + + + + + + + +This page has information on how to host your own metadata catalogue. +If you plan to locally develop the REST API, please follow the installation procedure in "Contributing" +after following the instructions on this page.
+The platform is tested on Linux, but should also work on Windows and MacOS. +Additionally, it needs Docker and +Docker Compose (version 2.21.0 or higher).
+Starting the metadata catalogue is as simple as spinning up the docker containers through docker compose. +This means that other than the prerequisites, no installation steps are necessary. +However, we do need to fetch files the latest release of the repository:
+git clone https://github.com/aiondemand/AIOD-rest-api.git
+
It is also possible to clone using SSH. +If you plan to develop the metadata catalogue, check the "Contributing" page +for more information on this step.
+<> Code
button and download the ZIP
file. From the root of the project directory (i.e., the directory with the docker-compose.yaml
file), run:
We provide the following script as a convenience. +This is especially useful when running with a non-default or development configuration, +more on that later. +
./scripts/up.sh
+
docker compose up -d
+
This will start a number of services running within one docker network:
+These services are described in more detail in their dedicated pages. +After the previous command was executed successfully, you can navigate to localhost +and see the REST API documentation. This should look similar to the api.aiod.eu page, +but is connected to your local database and services.
+To start connector services that automatically index data from external platforms into the metadata catalogue,
+you must specify their docker-compose profiles (as defined in the docker-compose.yaml
file).
+For example, you can use the following commands when starting the connectors for OpenML and Zenodo.
./scripts/up.sh openml zenodo-datasets
+
docker compose --profile openml --profile zenodo-datasets up -d
+
The full list of connector profiles are:
+There are two main places to configure the metadata catalogue services:
+environment variables configured in .env
files, and REST API configuration in a .toml
file.
+The default files are ./.env
and ./src/config.default.toml
shown below.
If you want to use non-default values, we strongly encourage you not to overwrite the contents of these files.
+Instead, you can create ./override.env
and ./config.override.toml
files to override those files.
+When using the ./scripts/up.sh
script to launch your services, these overrides are automatically taken into account.
# Configures the REST API
+# TODO: refactor configuration (https://github.com/aiondemand/AIOD-rest-api/issues/82)
+
+# Information on which database to connect to
+[database]
+host = "sqlserver"
+port = 3306
+database = "aiod"
+username = "root"
+password = "ok"
+
+# Additional options for development
+[dev]
+reload = true
+request_timeout = 10 # seconds
+
+# Authentication and authorization
+[keycloak]
+server_url = "http://keycloak:8080/aiod-auth/"
+realm = "aiod"
+client_id = "aiod-api" # a private client, used by the backend
+client_id_swagger = "aiod-api-swagger" # a public client, used by the Swagger Frontend
+openid_connect_url = "http://localhost/aiod-auth/realms/aiod/.well-known/openid-configuration"
+scopes = "openid profile roles"
+role = "edit_aiod_resources"
+
# DEV
+# Override USE_LOCAL_DEV to "true" to automatically mount your
+# source code to the containers when using `scripts/up.sh`.
+USE_LOCAL_DEV=
+
+# REST API
+AIOD_REST_PORT=8000
+
+#MYSQL
+MYSQL_ROOT_PASSWORD=ok
+
+#KEYCLOAK
+HOSTNAME=localhost
+KEYCLOAK_ADMIN=admin
+KEYCLOAK_ADMIN_PASSWORD=password
+KEYCLOAK_CLIENT_SECRET="QJiOGn09eCEfnqAmcPP2l4vMU8grlmVQ"
+REDIRECT_URIS=http://${HOSTNAME}/docs/oauth2-redirect
+POST_LOGOUT_REDIRECT_URIS=http://${HOSTNAME}/aiod-auth/realms/aiod/protocol/openid-connect/logout
+AIOD_KEYCLOAK_PORT=8080
+
+EGICHECKINALIAS=
+
+#ELASTICSEARCH
+ES_USER=elastic
+ES_PASSWORD=changeme
+ES_DISCOVERY_TYPE=single-node
+ES_ROLE="edit_aiod_resources"
+ES_JAVA_OPTS="-Xmx256m -Xms256m"
+AIOD_ES_HTTP_PORT=9200
+AIOD_ES_TRANSPORT_PORT=9300
+
+#LOGSTASH
+LS_JAVA_OPTS="-Xmx256m -Xms256m"
+AIOD_LOGSTASH_BEATS_PORT=5044
+AIOD_LOGSTASH_PORT=5000
+AIOD_LOGSTASH_API_PORT=9600
+
+#NGINX
+AIOD_NGINX_PORT=80
+
+#DATA STORAGE
+DATA_PATH=./data
+BACKUP_PATH=./data/backups
+
Overwriting these files directly will likely complicate updating to newer releases due to merge conflicts.
+First, stop running services: +
./scripts/down.sh
+
git fetch origin
+git checkout vX.Y.Z
+
Then run the startup commands again (either up.sh
or docker compose
).
We use Alembic to automate database schema migrations +(e.g., adding a table, altering a column, and so on). +Please refer to the Alembic documentation for more information. +Commands below assume that the root directory of the project is your current working directory.
+Warning
+Database migrations may be irreversible. Always make sure there is a backup of the old database.
+Build the database schema migration docker image with: +
docker build -f alembic/Dockerfile . -t aiod-migration
+
With the sqlserver container running, you can migrate to the latest schema with
+docker run -v $(pwd)/alembic:/alembic:ro -v $(pwd)/src:/app -it --network aiod-rest-api_default aiod-migration
+
since the default entrypoint of the container specifies to upgrade the database to the latest schema.
+Make sure that the specified --network
is the docker network that has the sqlserver
container.
+The alembic directory is mounted to ensure the latest migrations are available,
+the src directory is mounted so the migration scripts can use defined classes and variable from the project.
The REST API allows you to retrieve, update, or remove asset metadata in the metadata catalogue. +The assets are indexed from many different platforms, such as educational resources from AIDA, +datasets from HuggingFace, models from OpenML, and many more.
+The REST API is available at https://api.aiod.eu
and documentation on endpoints
+is available on complementary Swagger and ReDoc pages.
To use the REST API, simply make HTTP requests to the different endpoints.
+Generally, these are GET
requests when retrieving data, PUT
requests when modifying data, POST
requests when adding data, and DEL
requests when deleting data.
+Here are some examples on how to list datasets in different environments:
This example uses the requests
library to list datasets.
import requests
+response = requests.get("https://api.aiod.eu/datasets/v1?schema=aiod&offset=0&limit=10")
+print(response.json())
+
This example uses curl to retrieve data from the command line.
+curl -X 'GET' \
+'https://api.aiod.eu/datasets/v1?schema=aiod&offset=0&limit=10' \
+-H 'accept: application/json'
+
Additionally, we also provide an aiondemand
package for Python
+to simplify access to the REST API. You can see an example of that below, and we refer to their dedicated
+documentation pages for full installation and usage instructions.
import aiod
+aiod.datasets.get_list()
+
By navigating to the Swagger documentation, you can find information and examples on how to access the different endpoints.
+For example, if we navigate to the GET /datasets/v1
+endpoint and expand the documentation by clicking on the down chevron (v
), we can see the different query parameters
+and can execute a call directly on the API:
Click the Try it out
button to be able to modify the parameter values and then click the execute
button to make the request directly from the documentation page.
+Under response
you will also see an example on how to make the request through the command line using curl
, e.g.:
curl -X 'GET' \
+ 'https://api.aiod.eu/datasets/v1?schema=aiod&offset=0&limit=10' \
+ -H 'accept: application/json'
+
Below the example, you will find a section Server Response
which displays the actual response from the service (if you clicked execute
).
+Normally, this should look similar to the image below; a HTTP Status Code,
+and data (in JSON).
Below the actual server response is a response
section which lists information about the possible responses, including
+for example different error codes.
Tip
+When exploring these endpoints, prefer to connect to the test server instead to avoid editing production data. +You can find the test API at https://aiod-dev.i3a.es.
+The POST
and PUT
endpoints allow the addition or modification of assets on the platform.
+You can explore them in a similar way as with the GET
endpoints, with two important differences.
The first is that they require authentication.
+To authenticate within the Swagger pages, navigate to the top and click Authorize
and log in.
+Scroll up to OpenIdConnect (OAuth2, authorization_code with PKCE)
and click Authorize
to be taken to
+the keycloak login page. Log in with your preferred identity provider through EGI Check-in
.
The second important distinction as that you will provide data through a JSON body instead of individual parameters.
+The documentation page will prepopulate example data to help you know what information to provide under
+the Example Value
tab of the Request Body
section. To know what values are accepted, you can click the
+Schema
tab instead.
The ReDoc documentation provides pretty similar functionality to the Swagger documentation. +The main difference is that the Swagger page allows you to run queries against the REST API, whereas the ReDoc documentation does not. +However, some people prefer the organisation of ReDoc, +especially with respect to documentation of the expected responses and the schema documentation.
+The Swagger documentation gives examples on how to use CURL for the various endpoints.
+To see examples, simply expand the endpoint's documentation and click Try it out
, fill in any parameters, and click Execute
.
+The query will be executed, but it will also generate a curl
command which matches the query.
For example, listing the first 10 datasets:
+curl -X 'GET' \
+ 'http://api.aiod.eu/datasets/v1?schema=aiod&offset=0&limit=10' \
+ -H 'accept: application/json'
+