Skip to content

Commit

Permalink
Merge kraken-common, kraken-core, kraken-std and kraken-wrapper into …
Browse files Browse the repository at this point in the history
…kraken-build (#125)

* Merge kraken-common, kraken-core, kraken-std and kraken-wrapper into kraken-build

* kraken-build/: improvement: Merge `kraken-common`, `kraken-core`, `kraken-std` and `kraken-wrapper` packages into a single `kraken-build` package.

* Updated PR references in 1 changelogs.

skip-checks: true

* update CI, disable Selftest and Kraken-std integration tests, fix kraken-build/pyproject.toml

* format imports with isort

* upgrade Mypy python version

* one more isort

* adjustments

* install poetry/pdm into CI environment

* Move Python projects for integration tests into examples/ folder

* also move Cargo examples for integration tests into examples/ folder

* Improve Python integration tests by unsetting the locally activate virtual environment

* fix install pdm in ci

* comment out 3.11 and 3.12 for now because of Dill

* update Python constraint due to Dill

* add mention to readme that we cant use 3.11+ until Dill 0.3.8 is released (uqfoundation/dill#595)

* remove unused import

* run rustup update to make sure cargo is available

* use example_dir

* fix

* disable pyupgrade for now, we will do that in another MR for kraken-common and kraken-wrapper which is now elevated to 3.10+

* re-enable selftest

* need poetry/rustup in selftest as well

* improve integration test for Python when running locally where Poetry can cache a copy of the same wheel with a different hash from a previous test run

* also fix PDM for repeated tests locally

* mark deactivate_venv fixture as autouse

* doctest fix

---------

Co-authored-by: GitHub Action <github-action@users.noreply.github.com>
  • Loading branch information
NiklasRosenstein and GitHub Action authored Nov 8, 2023
1 parent c75e413 commit b39de16
Show file tree
Hide file tree
Showing 655 changed files with 2,065 additions and 2,139 deletions.
86 changes: 41 additions & 45 deletions .github/workflows/python.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,61 +13,57 @@ jobs:
strategy:
matrix:
include:
- python-version: "3.7"
only: --only=kraken-common,kraken-wrapper
- python-version: "3.8"
only: --only=kraken-common,kraken-wrapper
- python-version: "3.9"
only: --only=kraken-common,kraken-wrapper
- python-version: "3.10"
only: ''
- python-version: "3.11"
only: --only=kraken-common,kraken-wrapper
# Can't test with 3.11/3.12 until Dill relased 0.3.8, see https://github.com/uqfoundation/dill/issues/595
# - python-version: "3.11"
# - python-version: "3.12"
steps:
- uses: actions/checkout@v3
- uses: NiklasRosenstein/slap@gha/install/v1
- uses: actions/setup-python@v4
with: { python-version: "${{ matrix.python-version }}" }
- run: pip install pipx && pipx install poetry && pipx install pdm
- run: rustup update
- run: slap install --link --no-venv-check ${{ matrix.only }}
- run: slap test ${{ matrix.only }}

kraken-std-integration-test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- python-version: "3.10"
test-type: integrationTest
# kraken-std-integration-test:
# runs-on: ubuntu-latest
# strategy:
# fail-fast: false
# matrix:
# include:
# - python-version: "3.10"
# test-type: integrationTest

steps:
- uses: actions/checkout@v3
with: { fetch-depth: 0 }
- uses: NiklasRosenstein/slap@gha/install/v1
- uses: actions/setup-python@v4
with: { python-version: "${{ matrix.python-version }}" }
- run: pip install pipx && pipx install kraken-wrapper
# steps:
# - uses: actions/checkout@v3
# with: { fetch-depth: 0 }
# - uses: NiklasRosenstein/slap@gha/install/v1
# - uses: actions/setup-python@v4
# with: { python-version: "${{ matrix.python-version }}" }
# - run: pip install pipx && pipx install kraken-wrapper

# Need Poetry and PDM for integration tests.
- run: pipx install poetry
- run: curl -sSL https://raw.githubusercontent.com/pdm-project/pdm/main/install-pdm.py | python3 -
# # Need Poetry and PDM for integration tests.
# - run: pipx install poetry
# - run: curl -sSL https://raw.githubusercontent.com/pdm-project/pdm/main/install-pdm.py | python3 -

# Make sure we have Docker available for the ete-tests.
- uses: docker/setup-buildx-action@v2
# # Make sure we have Docker available for the ete-tests.
# - uses: docker/setup-buildx-action@v2

- run: cd kraken-std && krakenw -v run ${{ matrix.test-type }} -v
env:
ARTIFACTORY_CARGO_REPOSITORY: ${{ vars.ARTIFACTORY_CARGO_REPOSITORY }}
CLOUDSMITH_CARGO_REPOSITORY: ${{ vars.CLOUDSMITH_CARGO_REPOSITORY }}
CLOUDSMITH_INTEGRATION_TEST_CREDENTIALS: |
{
"api_key": "${{ secrets.CLOUDSMITH_API_KEY }}",
"owner": "${{ vars.CLOUDSMITH_OWNER }}",
"user": "${{ vars.CLOUDSMITH_USER }}"
}
ARTIFACTORY_INTEGRATION_TEST_CREDENTIALS: |
{
"token": "${{ secrets.ARTIFACTORY_TOKEN }}",
"url": "${{ vars.ARTIFACTORY_URL }}",
"user": "${{ vars.ARTIFACTORY_USER }}"
}
# - run: cd kraken-std && krakenw -v run ${{ matrix.test-type }} -v
# env:
# ARTIFACTORY_CARGO_REPOSITORY: ${{ vars.ARTIFACTORY_CARGO_REPOSITORY }}
# CLOUDSMITH_CARGO_REPOSITORY: ${{ vars.CLOUDSMITH_CARGO_REPOSITORY }}
# CLOUDSMITH_INTEGRATION_TEST_CREDENTIALS: |
# {
# "api_key": "${{ secrets.CLOUDSMITH_API_KEY }}",
# "owner": "${{ vars.CLOUDSMITH_OWNER }}",
# "user": "${{ vars.CLOUDSMITH_USER }}"
# }
# ARTIFACTORY_INTEGRATION_TEST_CREDENTIALS: |
# {
# "token": "${{ secrets.ARTIFACTORY_TOKEN }}",
# "url": "${{ vars.ARTIFACTORY_URL }}",
# "user": "${{ vars.ARTIFACTORY_USER }}"
# }
14 changes: 8 additions & 6 deletions .github/workflows/selftest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,21 @@ env:
COLUMNS: 200

jobs:
kraken-std-selftest:
selftest:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: NiklasRosenstein/slap@gha/install/v1
- uses: actions/setup-python@v4
with: { python-version: "3.10" }
- run: pip install pipx && pipx install poetry && pipx install pdm
- run: rustup update
- run: slap install --link --no-venv-check ${{ matrix.only }}
- run: cd kraken-std && kraken run fmt lint test -vv
- run: cd kraken-std && kraken q ls
- run: cd kraken-std && kraken q tree
- run: cd kraken-std && kraken q viz
- run: cd kraken-std && kraken q d python.mypy
- run: cd kraken-build && kraken run fmt lint test -vv
- run: cd kraken-build && kraken q ls
- run: cd kraken-build && kraken q tree
- run: cd kraken-build && kraken q viz
- run: cd kraken-build && kraken q d python.mypy

examples-docker-manual:
runs-on: ubuntu-latest
Expand Down
File renamed without changes.
File renamed without changes.
6 changes: 6 additions & 0 deletions examples/cargo-hello-world-lib/.cargo/config.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# This file is partially managed by Kraken. Comments and manually added repositories are not preserved.
[registries.crates-io]
protocol = "sparse"

[net]
git-fetch-with-cli = true
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ id = "c48a0427-6967-42b5-9e2c-e91ecf3b0862"
type = "fix"
description = "reduce networkx requirement for Python 3.7 compatibility, fix lints"
author = "@NiklasRosenstein"
component = "core"
Original file line number Diff line number Diff line change
@@ -1,13 +1,22 @@
release-date = "2022-08-12"
release-date = "2022-07-04"

[[entries]]
id = "973b5316-22e5-4835-abf4-7aae925b2dc1"
type = "improvement"
description = "update `docker_build()` to accept `dockerfile` parameter as a string and use `Project.to_path()`"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "9ab7842b-3b90-44c6-89ba-a59f5db9cca7"
type = "improvement"
description = "use Pip `--use-feature=in-tree-build` flag, this greatly improves the installation time for packages from local requirements"
author = "@NiklasRosenstein"
component = "wrapper"

[[entries]]
id = "1a3531f1-0f68-4993-bddb-c78a1715190b"
type = "improvement"
description = "no longer remove environment before install (allows VENV to upgrade an existing environment)"
author = "@NiklasRosenstein"
component = "wrapper"
Original file line number Diff line number Diff line change
@@ -1,13 +1,22 @@
release-date = "2022-08-12"
release-date = "2022-07-19"

[[entries]]
id = "b8054627-4bdb-4d47-b228-8406cf492844"
type = "breaking change"
description = "upgrade to be compatible with `kraken-core ^0.3.0`, add Novella docs and publish to Github actions"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "884e2e86-4e3f-4041-913b-f11a2688dfba"
type = "fix"
description = "`VENV` environment type now ensures Pip is up to date in the environment before installing requirements; removed `--use-feature=in-tree-build` flag is it was removed from `20.0.4` to `21.0` and is now the default"
author = "@NiklasRosenstein"
component = "wrapper"

[[entries]]
id = "58a83374-41df-4b1e-aeed-4351e4761bf8"
type = "improvement"
description = "print time it took to install the build environmet"
author = "@NiklasRosenstein"
component = "wrapper"
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,11 @@ id = "f0ce41b6-cafb-4d3d-8806-f16a703e554f"
type = "feature"
description = "add `kraken.std.python` package which provides tasks and factories for black, flake8, isort, mypy and pytest"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "03134444-c09f-4223-9797-0e38c075f85e"
type = "fix"
description = "fix mention of default build enviroment type in `--use` option"
author = "@NiklasRosenstein"
component = "wrapper"
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,32 @@ id = "d63edec0-7c4a-4516-9d57-a1f260564fda"
type = "improvement"
description = "RenderFileTask no longer tries to call the content supplier in `finalize()`, fix info line on the file being written"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "c3af61ab-bf19-4232-8e81-91a5926fc31a"
type = "improvement"
description = "add INFO log for subcommand executed by python `EnvironmentAwareDispatchTask`"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "e7dc557e-1699-48be-89f9-1ab0f8e4dde6"
type = "fix"
description = "Python `MypyTask` no longer adds the test directory if it is a subdirectory of the source directory, which used to cause an error in Mypy"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "e89c2e20-8d8d-4b92-a8ff-9e63d1a73f96"
type = "improvement"
description = "add `MypyTask.python_version` property"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "c80dcb8a-50ca-483c-9392-c43954ce85eb"
type = "improvement"
description = "no longer warn about lock file being outdated when `--upgrade` option is passed"
author = "@NiklasRosenstein"
component = "wrapper"
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,18 @@ id = "c53e0963-e23a-4723-8617-6ef690116c26"
type = "improvement"
description = "`KanikoBuildTask` now errors in `finalize()` if the `cache_repo` contains a `:` (colon)"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "b54e7e07-740f-449d-8a90-a68b795ff622"
type = "improvement"
description = "Add `SlapEnvironmentHandler` and automatic detection of an appropriate environment handler; add `InstallTask` and `install()` function to `kraken.std.python`; if the instal task exists then all toolchain tasks in Python will depend on it"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "7dddf643-9a7a-40d0-b044-6802e81d3ead"
type = "improvement"
description = "In the `VENV` environment type, do not upgrade Pip if the build env is reused (with `KRAKENW_INCREMENTAL=1`)"
author = "@NiklasRosenstein"
component = "wrapper"
Original file line number Diff line number Diff line change
@@ -1,4 +1,11 @@
release-date = "2022-08-16"
release-date = "2022-07-19"

[[entries]]
id = "e232926e-05d1-4166-a2b1-c35550a2b3a0"
type = "fix"
description = "fix interpretation of `KanikoBuildTask.load` property (before it would always load)"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "33e416a0-5c98-420e-867d-c7be861248d6"
Expand All @@ -8,3 +15,4 @@ author = "@NiklasRosenstein"
issues = [
"https://github.com/kraken-build/kraken-wrapper/issues/1",
]
component = "wrapper"
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,18 @@ id = "403d8b1b-c6ec-4c84-8aed-11e3cf5b2f92"
type = "improvement"
description = "Refactor Python task, add tasks for build and publish and the ability to register package indices that will be used for publishing (and should also be used for installing but we dont do that yet)"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "6974c2f9-c439-4ce1-9571-22a49ab69e0a"
type = "feature"
description = "add `kraken.std.git` module and `kraken.std.python.utils.git_version_to_python()` function"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "f5001ad6-560d-42d0-b5ce-eeb3d8e89ad0"
type = "fix"
description = "the `krakenw auth` command can now be used globally and does not require the presence of a `.kraken.py` file"
author = "@NiklasRosenstein"
component = "wrapper"
15 changes: 15 additions & 0 deletions kraken-build/.changelog/0.1.16.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
release-date = "2022-07-20"

[[entries]]
id = "f9618f8f-ac6f-4354-bc0e-d9a81446886e"
type = "fix"
description = "fix publishing"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "29fb048d-2839-4646-b3e1-ece78b7f3ff3"
type = "fix"
description = "add missing requirement `keyring`"
author = "@NiklasRosenstein"
component = "wrapper"
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,18 @@ id = "74a06257-41dd-4809-a259-5b493d2cfcc0"
type = "feature"
description = "add `BlackTask.additional_files` and `IsortTask.additional_files`"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "00ec24e0-6fb2-4179-858d-98a2eb2b79a0"
type = "fix"
description = "fix Python build system now properly moving the distributions instead of just renaming, which fails across file system boundaries"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "5bc09df8-ab0e-4cb4-8282-bab9fa172b1c"
type = "fix"
description = "Update handling of `--password-stdin` flag, if this and `--password` is not specified, we will use `getpass()`"
author = "@NiklasRosenstein"
component = "wrapper"
15 changes: 15 additions & 0 deletions kraken-build/.changelog/0.1.18.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
release-date = "2022-07-20"

[[entries]]
id = "6623360f-6899-4242-9fe5-a050791815a5"
type = "feature"
description = "add Python `PublishTask.skip_existing`"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "865cae3d-5021-4705-b981-6a23c2477341"
type = "fix"
description = "create parent directories when saving config"
author = "@NiklasRosenstein"
component = "wrapper"
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,11 @@ id = "659c9bc6-075f-4c2c-9fa7-ce0707ff1e2e"
type = "fix"
description = "fix `:blackFormat` actually formatting instead of checking and `:blackCheck` actually to just check instead of formatting"
author = "@NiklasRosenstein"
component = "std"

[[entries]]
id = "f21658ab-5a43-4c0a-8030-aa50bc56acae"
type = "improvement"
description = "fall back to writing password into file as plaintext if no keyring backend is available"
author = "@NiklasRosenstein"
component = "wrapper"
Original file line number Diff line number Diff line change
@@ -1,7 +1,15 @@
release-date = "2022-08-12"
release-date = "2022-07-04"

[[entries]]
id = "b6a6a191-06a9-4a14-a5af-d1dbf0a960c8"
type = "improvement"
description = "add `Project.to_path()`"
author = "@NiklasRosenstein"
component = "core"

[[entries]]
id = "76d24e29-1989-4062-9b82-387fe553e91d"
type = "improvement"
description = "set `_KRAKENW_ENV_TYPE` and `_KRAKENW_ENV_PATH` in environment when dispatching to build environment"
author = "@NiklasRosenstein"
component = "wrapper"
Loading

0 comments on commit b39de16

Please sign in to comment.