Skip to content

Commit

Permalink
Add contributing guide, continue reorganisation
Browse files Browse the repository at this point in the history
  • Loading branch information
PGijsbers committed Dec 16, 2024
1 parent 188a55b commit 40ca190
Show file tree
Hide file tree
Showing 6 changed files with 193 additions and 78 deletions.
96 changes: 96 additions & 0 deletions docs/Contributing.md
Original file line number Diff line number Diff line change
@@ -1 +1,97 @@
# Contributing

You're looking to contribute to the AI-on-Demand metadata catalogue, that's great!
AI-on-Demand is an open project and we welcome contributions by all, no matter their experience or background.

## Types of Contributions
There are many types of contributions to be made to the metadata catalogue, and this section
covers a non-exhaustive list. If you find something that's not outlined here, just reach out to us!

### No Code
Good examples for contributions that do not require any coding include:

- Giving [our GitHub repository](https://github.com/aiondemand/AIOD-rest-api) a star ⭐️
- [Helping people on the GitHub issue tracker](#helping-people-on-github) 🤝
- [Reporting bugs](#reporting-bugs) 🐛

### Documentation
You can help improve our documentation by correcting mistakes, such as typos, adding clarifications,
or even adding new sections. For small changes, documentation can be edited directly on GitHub by
finding the related markdown file and clicking the pencil icon in the top-right corner (✐).

If you intend to make bigger changes, please first open a new issue the documents the suggested change.
That way, we can give you feedback before you write, and verify that it is actually a change we would be interested in adopting.
For big changes, we also recommend you to follow the instructions on ["Setting up a development environment"](#setting-up-a-development-environment)
so that you can render the changed documentation locally.

### Code
For code changes, please first co-ordinate with the developers on what you will work on.
You can do this by either leaving a comment on an existing issue that you would like to help with,
or by opening a new issue proposing your change. By first communicating with the developers, they
can let you know ahead of time whether or not the change is wanted, make sure they have time to
support you, and provide any feedback. We really want to avoid a scenario where you may work hard on a contribution
only to find out that it is not in line with the vision of the project and thus will not be accepted.
When starting your first code contribution, visit the ["Setting up a development environment"](#setting-up-a-development-environment)
section for more information on how to get started.

## Helping people on GitHub
Helping people on the [GitHub issue tracker](https://github.com/aiondemand/AIOD-rest-api/issues) just requires a GitHub account.
You can help by people by answering their questions, or weighing in on discussions.
Even if you do not have an answer, you can verify that you can reproduce the behavior they report or
ask clarifying questions to make their bug report better (see also ["Reporting Bugs"](#reporting-bugs)).
This helps the core contributors resolve the issue with more ease and is hugely appreciated.
As always, please be kind and patient with others.

## Reporting Bugs
When you find a bug and want to report it, the first step is to check that it has not already been reported.
Use the search functionality of the GitHub issue tracker to find identical or related issues.
If your issue or bug has already been reported, please do not open a new issue.
Instead, either "upvote" the original report by leaving a 👍 reaction or,
if you have additional information which may be relevant to the discussion, leave a comment on that issue.

If your bug isn't reported yet, create a new issue.
Provide a clear explanation of the expected behavior and the observed behavior, and explain why you think this is a bug.
Add instructions on how to reproduce the expected behavior through a [Minimal Reproducible Example](https://stackoverflow.com/help/minimal-reproducible-example).
Without it, it may be very hard for contributors to solve the issue (or may not even understand it).

## Setting up a development environment

### Cloning
First, make sure you can get the local metadata catalogue up and running by following the ["Hosting" instructions](Hosting.md).
During the installation step, use `git` to clone the repository.
If you have write access to this repository, you can follow the instruction as-is.
If you do not have write access to this repository, you must [fork it](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo).
After forking the repository, your clone command will be (substituting `USERNAME` for your GitHub username):

```commandline
git clone https://github.com/USERNAME/AIOD-rest-api.git
```

### Installing Dependencies
Always make sure to install your dependencies in a local environment, for example with the built in `venv` module:

```commandline
python -m pip venv venv
source venv/bin/activate
```

and then install the Python dependencies

```commandline
python -m pip install -e ".[dev, docs]"
```

we install the optional `dev` (developer) and `docs` (documentation) dependencies so that we can
build documentation and run tests locally.

### Configuration

It is also generally useful to set add an `override.env` file to the project's root directory with
the line `USE_LOCAL_DEV=true` added. This will allow utility scripts `./scripts/up.sh` and `./scripts/down.sh`
to start docker containers in a way that reflects local changes.

## Making a Code Changes
See the ["Developer Documentation"](developer/index.md) for the technical documentation of this project.
More to be added.

[//]: # (## Setting up a pull request)
9 changes: 7 additions & 2 deletions docs/Hosting.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Hosting the Metadata Catalogue
This page has information on how to host your own metadata catalogue.
If you plan to locally develop the REST API, please follow the installation procedure in ["Contributing"](../contributing) instead.
If you plan to locally develop the REST API, please follow the installation procedure in ["Contributing"](../contributing)
after following the instructions on this page.

## Prerequisites
The platform is tested on Linux, but should also work on Windows and MacOS.
Expand All @@ -17,6 +18,10 @@ However, we do need to fetch files the latest release of the repository:
git clone https://github.com/aiondemand/AIOD-rest-api.git
```

It is also possible to clone using [SSH](https://docs.github.com/en/authentication/connecting-to-github-with-ssh).
If you plan to develop the metadata catalogue, check the ["Contributing"](Contributing.md#cloning) page
for more information on this step.

=== "UI (browser)"

* Navigate to the project page [aiondemand/AIOD-rest-api](https://github.com/aiondemand/AIOD-rest-api).
Expand Down Expand Up @@ -146,4 +151,4 @@ Make sure that the specified `--network` is the docker network that has the `sql
The alembic directory is mounted to ensure the latest migrations are available,
the src directory is mounted so the migration scripts can use defined classes and variable from the project.

[//]: # (TODO: Write documentation for when some of the migrations are not applicable. E.g., when a database was created in a new release.)
[//]: # (TODO: Write documentation for when some of the migrations are not applicable. E.g., when a table was created in a new release.)
78 changes: 7 additions & 71 deletions docs/developer/index.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# AIoD API
# Metadata Catalogue API

!!! note

This page was the old readme. Re-organizing and updating it into our structured documentation
pages is work in progress. This page will serve as an overview page that serves as a
"getting started" page and quick reference, with references to pages with in-depth information.

This repository contains the AI on Demand (AIoD) REST API. It is built with
[FastAPI](https://fastapi.tiangolo.com/)
Expand Down Expand Up @@ -34,19 +40,6 @@ Note that this synchronization process between the platform and the database, is
the synchronization between database instances. The latter is under discussion in the AIoD
Synchronization Meetings.

### AIoD Metadata

The models are found in `src/database/model`. The AIoD Metadata team is responsible for
determining the fields of the metadata, whereafter the classes are implemented in this metadata
catalogue. To check the existing fields, the easiest way it to start this application (see
"Using Docker Compose") and check the automatically generated swagger documentation
(http://localhost:8000/docs).

We use inheritance to make sure that generic fields, such as name and description, are present
and consistent over all resources. A partial overview of the metadata model can be found in the
following figure:

![AIoD Metadata model](../media/AIoD_Metadata_Model.drawio.png)

## Prerequisites
- Linux/MacOS/Windows (should all work)
Expand Down Expand Up @@ -258,60 +251,3 @@ We provide several scripts to facilitate the scheduling of backups and the manua

## Releases

### Breaking changes
Breaking changes of a resource include deleting a field, changing the name of an existing field,
or changing the datatype of a field. Adding new fields is not a breaking change.

On a breaking change for a resource (e.g. for Dataset), a new router with a new version should
be created. The existing router should be deprecated, and rewritten so that it can handle the
new metadata of the database. This deprecation of a router will be visible in the Swagger
documentation. Calls to a deprecated router will still work, but a response header "Deprecated"
will be added with the deprecation date. The deprecated router will then be deleted on the next
release.

On non-breaking changes of a resource, a new version is not needed for the corresponding router.

Example:
- Start www.aiod.eu/api/datasets/v0
- Release 1: www.aiod.eu/api/datasets/v0 (no breaking changes)
- Release 2:
- www.aiod.eu/api/datasets/v0 (deprecated)
- www.aiod.eu/api/datasets/v1
- Release 3: www.aiod.eu/api/datasets/v1

### Database migration

The database should always be up-to-date with the latest version of the metadata. As database
migration tool, [Alembic](https://alembic.sqlalchemy.org/en/latest/) is the default choice for
SQLAlchemy. The setup of this db migration for AIOD remains a TODO for now.

### Changelog

As changelog we use the Github tags. For each release, a release branch should be created with a
bumped version in the pyproject.toml, and merged with the master. The tag should contain a
message detailing all the breaking and non-breaking changes. This message should adhere to the
guiding principles as described in https://keepachangelog.com/.

- Show all tags: https://github.com/aiondemand/AIOD-rest-api/tags
- Show a specific tag: https://github.com/aiondemand/AIOD-rest-api/releases/tag/0.3.20220501

This information can also be extracted using the Github REST API.


### Create a release
To create a new release,
1. Make sure all requested functionality is merged with the `develop` branch.
2. From develop: `git checkout -b release/[VERSION]`. Example of version: `1.1.20231129`
3. Update the version in `pyproject.toml`.
4. Test all (most of) the functionality. Checkout the project in a new directory and remove all
your local images, and make sure it works out-of-the box.
5. Go to https://github.com/aiondemand/AIOD-rest-api/releases and draft a new release from the
release branch. Look at all closed PRs and create a changelog
6. Create a PR from release branch to master
7. After that's merged, create a PR from master to develop
8. Deploy on the server(s):
- Check which services currently work (before the update). It's a sanity check for if a service _doesn't_ work later.
- Update the code on the server by checking out the release
- Merge configurations as necessary
- Make sure the latest database migrations are applied: see ["Schema Migrations"](developer/migration.md#update-the-database)
9. Notify everyone (e.g., in the API channel in Slack).
63 changes: 63 additions & 0 deletions docs/developer/releases.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Releasing and Versioning

The project loosely uses [Semantic Versions](https://semver.org) where the patch/micro number matches the release date.

## Breaking changes

!!! note "Work in Progress"

Guidelines in "Breaking Changes" are the desired workflow, but in practice we are not always following them
as 1) the metadata model wasn't yet matured and 2) the infrastructure for this needs to be
developed. For now, we make sure all URLs are at least under a version suffix, which makes
support in the future possible.

Breaking changes of a resource include deleting a field, changing the name of an existing field,
or changing the datatype of a field. Adding new fields is not a breaking change.

On a breaking change for a resource (e.g. for Dataset), a new router with a new version should
be created. The existing router should be deprecated, and rewritten so that it can handle the
new metadata of the database. This deprecation of a router will be visible in the Swagger
documentation. Calls to a deprecated router will still work, but a response header "Deprecated"
will be added with the deprecation date. The deprecated router will then be deleted on the next
release.

On non-breaking changes of a resource, a new version is not needed for the corresponding router.

Example:
- Start www.aiod.eu/api/datasets/v0
- Release 1: www.aiod.eu/api/datasets/v0 (no breaking changes)
- Release 2:
- www.aiod.eu/api/datasets/v0 (deprecated)
- www.aiod.eu/api/datasets/v1
- Release 3: www.aiod.eu/api/datasets/v1

## Changelog

As changelog we use the Github tags. For each release, a release branch should be created with a
bumped version in the pyproject.toml, and merged with the master. The tag should contain a
message detailing all the breaking and non-breaking changes. This message should adhere to the
guiding principles as described in https://keepachangelog.com/.

- Show all tags: https://github.com/aiondemand/AIOD-rest-api/tags
- Show a specific tag: https://github.com/aiondemand/AIOD-rest-api/releases/tag/0.3.20220501

This information can also be extracted using the Github REST API.


## Creating a release
To create a new release,
1. Make sure all requested functionality is merged with the `develop` branch.
2. From develop: `git checkout -b release/[VERSION]`. Example of version: `1.1.20231129`
3. Update the version in `pyproject.toml`.
4. Test all (most of) the functionality. Checkout the project in a new directory and remove all
your local images, and make sure it works out-of-the box.
5. Go to https://github.com/aiondemand/AIOD-rest-api/releases and draft a new release from the
release branch. Look at all closed PRs and create a changelog
6. Create a PR from release branch to master
7. After that's merged, create a PR from master to develop
8. Deploy on the server(s):
- Check which services currently work (before the update). It's a sanity check for if a service _doesn't_ work later.
- Update the code on the server by checking out the release
- Merge configurations as necessary
- Make sure the latest database migrations are applied: see ["Schema Migrations"](developer/migration.md#update-the-database)
9. Notify everyone (e.g., in the API channel in Slack).
24 changes: 19 additions & 5 deletions docs/developer/schema/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,32 @@ created by the developer of FASTApi, the framework we use for routing.

SQLModel makes it possible to define only a single model instead of defining the database-layer
(SQLAlchemy) and the logic-layer (Pydantic) separately.
Our implementation relies on inheritance to follow the same class hierarchy as defined in the [metadata schema](https://github.com/aiondemand/metadata-schema),
this makes sure that generic fields, such as name and description, are present and consistent over all resources,
and changes to the conceptual model and the model implementation should be similar.

A partial overview of the metadata model can be found in the
following figure:

![AIoD Metadata model](../../media/AIoD_Metadata_Model.drawio.png)


## Reading the Conceptual Metadata Schema
Documentation and/or a reference to how to interpret the metadata schema hosted at the dedicated repository are to be added (after meeting Thursday 12 Dec 2024).
Tools and documentation on how to read the conceptual metadata model are currently being written.
This section will be updated at a later date (as of 16-12-2024).

## Reading the Metadata Schema Implementation
Documentation to be added.
This section will be updated at a later date (as of 16-12-2024) and will describe:
- The use various class variants, such as `XBase`, `XORM`, `XCreate`, with a link to the ["objects"](objects.md) page.
- A brief discussion on how to read an attribute definition, with a link to the ["attributes"](attributes.md) page.
- A brief discussion on how to relationships an attribute definition, with a link to the ["relationships"](relationships.md) page.

## Changing the Metadata Schema Implementation
On a high level, changes to the metadata schema implementation consist of three steps:
updating the schema implementation in [`src/database/model`](https://github.com/aiondemand/AIOD-rest-api/tree/develop/src/database/model),
updating or adding tests which test those changes,
and adding a [database migration script]() which updates the database accordingly.

* updating the schema implementation in [`src/database/model`](https://github.com/aiondemand/AIOD-rest-api/tree/develop/src/database/model),
* updating or adding tests which test those changes, and
* adding a [database migration script]() which updates the database accordingly.

This last step isn't needed during development, where you may recreate a database anytime to model changes.
However, to deploy the changed schema in production we need to be able to change the database,
Expand Down
1 change: 1 addition & 0 deletions mkdocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ nav:
- 'Attributes': developer/schema/attributes.md
- 'Relationships': developer/schema/relationships.md
- 'Objects': developer/schema/objects.md
- 'Contributing': Contributing.md
- 'Unorganized Docs':
- 'Code Advice': developer/code.md
- 'Keycloak': developer/auth.md
Expand Down

0 comments on commit 40ca190

Please sign in to comment.