This repository contains the core management system for Max Planck Institute Data Systems Group's Satellite Data Augmentation Project. It is being developed by DAD (Dream, Aim, Deliver) as part of the collaboration with the Max Planck Institute.
Please fork this repository to your own GitHub account and clone it to your local machine. Please avoid pushing to this repository directory directly, either to the main branch or to any other branch.
gh repo clone <your-username>/kernel-planckster
git remote add upstream https://github.com/dream-aim-deliver/kernel-planckster.gitThen head over to
https://github.com/<your_username>/kernel-planckster/settings/actions
and enable GitHub Actions for your fork by selecting the Allow all actions and reusable workflows option.
# AT THE ROOT OF THE PROJECT
python3 -m venv .venv
source .venv/bin/activate
# Install poetry
pip install poetryOn ARM architectures ( like Apple Silicone), you might need to install psycopg2 with the following command:
brew install libpq --build-from-source
brew install openssl
export LDFLAGS="-L/opt/homebrew/opt/openssl@1.1/lib -L/opt/homebrew/opt/libpq/lib"
export CPPFLAGS="-I/opt/homebrew/opt/openssl@1.1/include -I/opt/homebrew/opt/libpq/include"
pip3 install psycopg2
Continue with the setup as follows:
# Install dependencies
poetry install
# Setup pre-commit
pre-commit install
pre-commit run --all-files
# Set up environment variables for pytest in pyproject.toml as needed, but the defaults should workKernel Planckster is configured via environment variables directly. These environment variables are NOT loaded from any .env file. You can set these environment variables in your shell or in your IDE.
| Name | Default Value |
|---|---|
| KP_ROOT_DIRECTORY | ./tests/mocks |
| KP_SOURCE_DATA_DIR | source_data |
| KP_RDBMS_HOST | localhost |
| KP_RDBMS_PORT | 5435 |
| KP_RDBMS_DBNAME | kp-db |
| KP_RDBMS_USERNAME | postgres |
| KP_RDBMS_PASSWORD | postgres |
| KP_FASTAPI_PORT | 8005 |
| KP_OBJECT_STORE_HOST | localhost |
| KP_OBJECT_STORE_PORT | 9002 |
| KP_OBJECT_STORE_ACCESS_KEY | minio |
| KP_OBJECT_STORE_SECRET_KEY | minio123 |
| KP_OBJECT_STORE_BUCKET | default |
| KP_OBJECT_STORE_SIGNED_URL_EXPIRY | 60 |
Using docker containers to spin up an SQL database, you can autogenerate migrations with Alembic. From the root of the project, do:
docker compose --profile dev up -d
alembic upgrade head
alembic revision --autogenerate -m "migration message"
alembic upgrade head
alembic downgrade base
alembic upgrade head
docker compose --profile dev downMake sure to fix any errors given by the alembic commands above before executing the next one and commiting the changes. In particular, you might need to fix the alembic version files.
For development, the environment variable KP_MODE has to be set to development. This is the default value used by the docker-compose.yml and launch.json files.
In development mode, postgres, fastapi and adminer are all running in docker containers. These containers will be started automatically when you start FastAPI launch configuration in VSCode or when you run the dev script in the root of the project, as shown below:
poetry run devThe containers will be removed when you stop the dev script or stop the FastAPI launch configuration in VSCode.
| Service | Host/Port | Mode |
|---|---|---|
| FastAPI | http://localhost:8000 | development |
| Postgres | 0.0.0.0:5432 | development |
| Adminer UI | http://localhost:8080 | development |
FastAPI provides the Swagger UI at http://localhost:8000/docs to manually test the API endpoints, and the ReDoc UI at http://localhost:8000/redoc for a documentation view of the endpoints.
In development mode, you can access Adminer interface at http://localhost:8080 to check the database. The credentials are:
System: PostgreSQL
Server: db
Username: postgres
Password: postgres
Database: kp-db
To test the object store in dev mode, you can run:
poetry run dev:storageThis will do the same as poetry run dev but also start a minio container:
| Service | Host/Port | Mode |
|---|---|---|
| MinIO | http://localhost:9001 | development |
You can access MinIO at http://localhost:9001 to check the object storage. The credentials are:
Access Key: minio
Secret Key: minio123
You run tests on the command line with
poetry run pytest -sOr in the VSCode test explorer UI.
In test mode, postgres, fastapi, adminer, and minio are all running docker containers and will be automatically removed once the tests are finished.
DANGLING CONTAINER WARNING: If you are debugging tests in VSCode and notice that your test is failing, which triggers a breakpoint, then DO NOT STOP the debugger. Just press F5 and let the tests finish. The containers will be removed automatically once the tests are finished. Otherwise, you will end up with dangling test containers that will have to be removed manually. To remove these containers manually:
cd tests
docker compose downIn testing mode, you can access the services as follows:
| Service | Host/Port | Mode |
|---|---|---|
| FastAPI | http://localhost:8005 | test |
| Postgres | 0.0.0.0:5435 | test |
| Adminer UI | http://localhost:8085 | test |
| MinIO | http://localhost:9002 | test |
In test mode, you can access Adminer interface at http://localhost:8085 to check the database. The credentials are:
System: PostgreSQL
Server: db
Username: postgres
Password: postgres
Database: kp-db
You can also access MinIO at http://localhost:9002 to check the object storage. The credentials are defined in the pyproject.toml file, under the pytest ini_options section.
In production mode, you must configure the dependencies like MinIO, Postgres, Kafka, etc. via environment variables.
See the configuration section above for the list of environment variables.
Below is an example of environment variables that you can set in your shell if you are starting the provided docker-compose.yml file
export KP_MODE=production
export KP_FASTAPI_HOST=0.0.0.0
export KP_FASTAPI_PORT=80
export KP_FASTAPI_RELOAD=false
export KP_RDBMS_HOST=localhost
export KP_RDBMS_PORT=5435
export KP_RDBMS_DBNAME=kp-db
export KP_RDBMS_USERNAME=postgres
export KP_RDBMS_PASSWORD=postgres
export KP_OBJECT_STORE_HOST=localhost
export KP_OBJECT_STORE_PORT=9002
export KP_OBJECT_STORE_ACCESS_KEY=minio
export KP_OBJECT_STORE_SECRET_KEY=minio123
export KP_OBJECT_STORE_BUCKET=default
export KP_OBJECT_STORE_SIGNED_URL_EXPIRY=60
Then you can run the server with:
poetry run startIn production mode, the Uvicorn server is not started with the --reload flag, so you will have to restart the server manually if you make any changes to the code.
Additionally, the --proxy-headers flag is set to True by default in production mode. This is to ensure that the correct client IP address is logged in the server logs. This will tell Uvicorn to trust the headers sent by that proxy telling it that the application is running behind HTTPS, etc.
We use VSCode as our IDE. If you use VSCode, please install the recommended extensions.
Please use Issues to report any bugs or feature requests.
Once you have been assigned an issue, please create a branch with the following naming convention:
feature-<issue number>-<short description>
We recommend using the provided create-feature-branch utility to create a branch with the correct name.
This script will also pull the latest changes from the remote repository and create the new branch for you to work in.
./tools/create-feature-branch <issue number> <short description>Commit your change. The commit command must include a specific message format:
git commit -m "<component>: <change_message> #<issue number>"
Valid component names are listed in the label list and are usually specified on the issue of the change.
Add additional explanations to the body of the commit, such as motivation for certain decisions and background information. Here are some general rules: https://cbea.ms/git-commit/.
Using multiple commits is allowed as long as they achieve an independent, well-defined, change and are well-described. Otherwise multiple commits should be squashed.
Before submitting a pull request, please:
- Run pytest, at the root of the project, and fix all the errors:
poetry run pytest -s- Run mypy, at the root of the project, and fix all type errors:
poetry run mypy .- Run black, at the root of the project
poetry run black .Push the commit to your forked repository and create the pull request. Try to keep the Pull Request simple, it should achieve the single objective described in the issue. Multiple enhancements/fixes should be split into multiple Pull Requests.
Watch the pull request for comments and reviews. For any pull requests update, please try to squash/amend your commits to avoid “in-between” commits.
If you add a github-recognised keyword in the pull request name or in the pull request description, then the associated issue can be closed automatically once the pull request is merged, e.g.:
<component>: <change_message> Fix #<issue number>