Skip to content

Commit

Permalink
Improving builds
Browse files Browse the repository at this point in the history
  • Loading branch information
erikbosch committed Mar 7, 2024
1 parent b9d5cc9 commit d869593
Show file tree
Hide file tree
Showing 28 changed files with 63 additions and 75 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,6 @@ In general the naming shall follow: **<type>(_<sub-type>)_<componenent-name>.tar

## How to create a new release

See [Release Process](https://github.com/eclipse/kuksa.val.services/wiki/Release-Process)
See [Release Process](https://github.com/eclipse-kuksa/kuksa-incubation/wiki/Release-Process)


2 changes: 1 addition & 1 deletion .github/workflows/hvac_integration_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ jobs:
# files: ./results/IntegrationTest/junit.xml

- name: Upload Integration Test Logs
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
if: always()
with:
name: integration_test.log
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/hvac_service.yml
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ jobs:
push: true
tags: |
${{ steps.meta.outputs.tags }}
ttl.sh/kuksa.val.services/hvac_service-${{github.sha}}
ttl.sh/eclipse-kuksa/kuksa-incubation/hvac_service-${{github.sha}}
labels: ${{ steps.meta.outputs.labels }}

- name: Build container and push to ttl.sh
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ jobs:
# Upload MegaLinter artifacts
- name: Archive production artifacts
if: ${{ success() }} || ${{ failure() }}
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: MegaLinter reports
path: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ jobs:
uses: actions/checkout@v4

- name: Download all artifacts
uses: actions/download-artifact@v3
uses: actions/download-artifact@v4
with:
path: bin/

Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/scripts/hvac-install-ci-tooling.sh
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ sudo apt-get -qqy install --no-install-recommends \
gcovr \
clang-format \
cppcheck \
valgrind
valgrind \
python3-setuptools

# Install PIP
[ -z "$(which pip3)" ] && sudo apt-get -qqy install --no-install-recommends --fix-missing python3-pip
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/scripts/seat-install-ci-tooling.sh
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ sudo apt-get -qqy install --no-install-recommends \
gcovr \
clang-format \
cppcheck \
valgrind
valgrind \
python3-setuptools

# Install PIP
[ -z "$(which pip3)" ] && sudo apt-get -qqy install --no-install-recommends --fix-missing python3-pip
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/seat_integration_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ jobs:
# files: ./results/IntegrationTest/junit.xml

- name: Upload Integration Test Logs
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
if: always()
with:
name: integration_test.log
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/seat_service.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ jobs:
uses: actions/checkout@v4

- name: Retrieve build binaries
uses: actions/download-artifact@v3
uses: actions/download-artifact@v4
with:
path: ${{github.workspace}}/seat_service

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/seat_service_debug_build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ jobs:
run: ./build-debug.sh ${{ matrix.target-arch }}

- name: Upload binaries
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: bin_vservice-seat_${{ matrix.target-arch }}_debug
path: |
Expand All @@ -67,7 +67,7 @@ jobs:

- name: Upload Test results (x86_64)
if: ${{ matrix.target-arch == 'x86_64' }}
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: report_test_vservice-seat
path: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/seat_service_docu_build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ jobs:
docs/doxygen/out/html/
- name: Upload docu
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: docu_vservice-seat
path: ${{github.workspace}}/seat_service/docu_vservice-seat.tar.gz
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/seat_service_release_build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ jobs:
target/${{ matrix.target-arch }}/release/licenses/ \
proto/
- name: Upload artifacts
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: bin_vservice-seat_${{ matrix.target-arch }}_release.tar.gz
path: ${{github.workspace}}/seat_service/bin_vservice-seat_${{ matrix.target-arch }}_release.tar.gz
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/seat_service_seatctrl_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,15 +66,15 @@ jobs:
minimum_coverage: 70

- name: Upload Code coverage (html)
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: report_codecov_vservice-seat-ctrl
path: |
${{github.workspace}}/seat_service/build_seat_controller/x86_64/report_codecov_*/**
if-no-files-found: warn

- name: Upload Test Results
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: report_test_vservice-seat-ctrl
path: |
Expand Down
2 changes: 1 addition & 1 deletion .vscode/scripts/run-feedercan.sh
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ export DAPR_GRPC_PORT="$FEEDERCAN_GRPC_PORT"
FEEDERCAN_VERSION=$(jq -r '.feedercan.version // empty' "$CONFIG_JSON")

### Default feedercan settings (repo, tag are optional in json)
FEEDERCAN_REPO="https://github.com/eclipse/kuksa.val.feeders.git"
FEEDERCAN_REPO="https://github.com/eclipse-kuksa/kuksa-can-provider.git"
if [ -z "$FEEDERCAN_VERSION" ]; then
echo "Coudln't find feedercan version from $CONFIG_JSON"
exit 1
Expand Down
2 changes: 1 addition & 1 deletion hvac_service/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ FROM scratch
#FROM alpine:latest
# FROM arm64v8/python:3.8.12-alpine as Runtime

LABEL org.opencontainers.image.source="https://github.com/eclipse/kuksa.val.services"
LABEL org.opencontainers.image.source="https://github.com/eclipse-kuksa/kuksa-incubation"

# needed as /dist/binary unpacks and runs from /tmp
WORKDIR /tmp
Expand Down
6 changes: 3 additions & 3 deletions hvac_service/docker-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ if [ "$DOCKER_ARCH" = "multiarch" ]; then
else
if [ $LOCAL -eq 1 ]; then
DOCKER_ARGS="--load -t $DOCKER_ARCH/$DOCKER_IMAGE"
DOCKER_EXPORT="($DOCKER_ARCH/$DOCKER_IMAGE ghcr.io/eclipse/kuksa.val.services/$DOCKER_IMAGE:prerelease)"
DOCKER_EXPORT="($DOCKER_ARCH/$DOCKER_IMAGE ghcr.io/eclipse-kuksa/kuksa-incubation/$DOCKER_IMAGE:prerelease)"
else
DOCKER_ARGS="--platform linux/$DOCKER_ARCH -t $DOCKER_ARCH/$DOCKER_IMAGE --output type=oci,dest=$DOCKER_EXPORT"
fi
Expand All @@ -101,8 +101,8 @@ echo "# docker buildx build $DOCKER_ARGS -f Dockerfile $CONTEXT_DIR"
DOCKER_BUILDKIT=1 docker buildx build $DOCKER_ARGS -f Dockerfile "$CONTEXT_DIR" $DOCKER_EXT

if [ $? -eq 0 ]; then
echo "docker image tag $DOCKER_ARCH/$DOCKER_IMAGE ghcr.io/eclipse/kuksa.val.services/$DOCKER_IMAGE:prerelease"
docker image tag $DOCKER_ARCH/$DOCKER_IMAGE ghcr.io/eclipse/kuksa.val.services/$DOCKER_IMAGE:prerelease
echo "docker image tag $DOCKER_ARCH/$DOCKER_IMAGE ghcr.io/eclipse-kuksa/kuksa-incubation/$DOCKER_IMAGE:prerelease"
docker image tag $DOCKER_ARCH/$DOCKER_IMAGE ghcr.io/eclipse-kuksa/kuksa-incubation/$DOCKER_IMAGE:prerelease
echo "# Exported $DOCKER_ARCH/$DOCKER_IMAGE in $DOCKER_EXPORT"
docker image ls | grep "/$DOCKER_IMAGE"
fi
10 changes: 2 additions & 8 deletions hvac_service/integration_test/it-config
Original file line number Diff line number Diff line change
Expand Up @@ -24,27 +24,22 @@ KDB_TAG=$(jq -r '.databroker.imageVersion // empty' "$CONFIG_JSON")
[ -z "$FEEDER_TAG" ] && export FEEDER_TAG="v0.1.0"

# allow custom tags from CI
[ -z "$SEAT_TAG" ] && export SEAT_TAG="v0.1.0"
[ -z "$HVAC_TAG" ] && export HVAC_TAG="v0.1.0"

### Local docker image config (may force rebuild + tag local images)
# export HVAC_TAG="prerelease"
# export SEAT_TAG="prerelease"

### ghcr.io image config
export DOCKER_REPO="ghcr.io"
export KDB_IMAGE="${DOCKER_REPO}/eclipse/kuksa.val/databroker:${KDB_TAG}"
export FEEDER_IMAGE="${DOCKER_REPO}/eclipse/kuksa.val.feeders/dbc2val:${FEEDER_TAG}"
export SEAT_IMAGE="${DOCKER_REPO}/eclipse/kuksa.val.services/seat_service:${SEAT_TAG}"
export HVAC_IMAGE="${DOCKER_REPO}/eclipse/kuksa.val.services/hvac_service:${HVAC_TAG}"
export FEEDER_IMAGE="${DOCKER_REPO}/eclipse-kuksa/kuksa-can-provider/can-provider:${FEEDER_TAG}"
export HVAC_IMAGE="${DOCKER_REPO}/eclipse-kuksa/kuksa-incubation/hvac_service:${HVAC_TAG}"

export KDB_CONTAINER="it-databroker"
export SEAT_CONTAINER="it-seat_service"
export HVAC_CONTAINER="it-hvac_service"
export FEEDER_CONTAINER="it-feedercan"

KDB_HOST_PORT="35555"
SEAT_HOST_PORT="35051"
HVAC_HOST_PORT="35052"
DOCKER_KDB_PORT="55555"
DOCKER_KDB_ADDRESS="${KDB_CONTAINER}:$DOCKER_KDB_PORT"
Expand All @@ -59,7 +54,6 @@ FEEDER_ENV="-e DBC_FILE=$FEEDER_MOUNT_DIR/it-can.dbc -e MAPPING_FILE=$FEEDER_MOU
FEEDER_LOG="info" #,dbcfeeder=debug,kuksa_client.grpc=debug,dbcfeederlib.canplayer=debug,dbcfeederlib.canreader=debug,dbcfeederlib.dbc2vssmapper=debug"

export KDB_DOCKER_OPT="-p ${KDB_HOST_PORT}:55555/tcp --expose ${KDB_HOST_PORT} -e RUST_LOG=info,databroker=debug,vehicle_data_broker=debug -e VEHICLEDATABROKER_DAPR_APP_ID=kdb --name ${KDB_CONTAINER}"
export SEAT_DOCKER_OPT="-p ${SEAT_HOST_PORT}:50051/tcp --expose ${SEAT_HOST_PORT} -e BROKER_ADDR=${DOCKER_KDB_ADDRESS} -e SIM=cansim -e SC_RPM=100 -e VEHICLEDATABROKER_DAPR_APP_ID=kdb --name ${SEAT_CONTAINER}"
export HVAC_DOCKER_OPT="-p ${HVAC_HOST_PORT}:50052/tcp --expose ${HVAC_HOST_PORT} -e VDB_ADDRESS=${DOCKER_KDB_ADDRESS} -e VEHICLEDATABROKER_DAPR_APP_ID=kdb -e LOG_LEVEL=debug --name ${HVAC_CONTAINER}"
export FEEDER_DOCKER_OPT="-e KUKSA_ADDRESS=${KDB_CONTAINER} -e KUKSA_PORT=$DOCKER_KDB_PORT ${FEEDER_ENV} ${FEEDER_MOUNT_OPT} -e LOG_LEVEL=${FEEDER_LOG} --name ${FEEDER_CONTAINER}"

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Overview

For more details on mapping format, check [dbc2val/mapping.md](https://github.com/eclipse/kuksa.val.feeders/blob/main/dbc2val/mapping/mapping.md)
For more details on mapping format, check [kuksa-can-provider mapping](https://github.com/eclipse-kuksa/kuksa-can-provider/blob/main/mapping/README.md).

## Generating json config for Integration Tests

Expand Down
6 changes: 3 additions & 3 deletions hvac_service/prerequisite_settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
},
"feedercan": {
"version": "main",
"imageVersion": "0.4.1"
"imageVersion": "0.4"
},
"kuksa_client": {
"version": "0.4.1"
"kuksa_clientz": {
"version": "0.4"
}
}
19 changes: 19 additions & 0 deletions seat_service/.devcontainer/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,22 @@ RUN apt-get -qqy install --no-install-recommends \
jq


# Install PIP
RUN apt-get -qqy install --fix-missing \
python3 \
python3-pip \
python3-setuptools

# Default pip and setuptools are too old...
RUN pip3 install --upgrade pip setuptools

# conan: dependency management
# conan needed > 1.43 for gtest and < 2.0
# cantools: code generation from .dbc file
RUN pip3 install \
'conan==1.56.0' \
'cantools==37.0.1'

RUN apt-get clean

# Install basic utils needed inside devcontainer
Expand Down Expand Up @@ -99,5 +115,8 @@ RUN echo $TZ > /etc/timezone && \
rm /etc/localtime && \
dpkg-reconfigure -f noninteractive tzdata


WORKDIR /workspace

#TODO: fix and uncomment: This is needed for local development under linux
#USER dockeruser
4 changes: 2 additions & 2 deletions seat_service/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,8 +215,8 @@ The service examples and related tests in this repository use VSS signals. VSS s
and backward incompatible changes may be introduced as part of major releases.
Some of the tests in this repository relies on using latest version
of [KUKSA.val Databroker](https://github.com/eclipse/kuksa.val/pkgs/container/kuksa.val%2Fdatabroker) and
[KUKSA.val DBC Feeder](https://github.com/eclipse/kuksa.val.feeders/pkgs/container/kuksa.val.feeders%2Fdbc2val).
Some code in the repository (like [Proto](https://github.com/eclipse/kuksa.val.services/tree/main/proto) definitions)
[KUKSA CAN Provider](https://github.com/eclipse-kuksa/kuksa-can-provider).
Some code in the repository (like [Proto](proto) definitions)
have been copied from [KUKSA.val](https://github.com/eclipse/kuksa.val).
This means that regressions may occur when KUKSA.val or KUKSA.val Feeders are updated. The intention for other KUKSA.val
Expand Down
1 change: 0 additions & 1 deletion seat_service/build-debug.sh
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,6 @@ if [ "$PACK" = "1" ]; then
mkdir -p "$ARCHIVE_PATH"

echo "# Copy all proto files in ./proto/"
cp -ra "$SCRIPT_DIR/../proto" .
cp -ra "$SCRIPT_DIR/proto" .
cp -ra "$BUILD_DIR/." "$ARCHIVE_PATH"

Expand Down
1 change: 0 additions & 1 deletion seat_service/build-release.sh
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,6 @@ if [ "$PACK" = "1" ]; then
mkdir -p "$ARCHIVE_PATH"

echo "# Copy all proto files in ./proto/"
cp -ra "$SCRIPT_DIR/../proto" .
cp -ra "$SCRIPT_DIR/proto" .
cp -ra "$BUILD_DIR/." "$ARCHIVE_PATH"

Expand Down
6 changes: 3 additions & 3 deletions seat_service/docker-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ if [ "$DOCKER_ARCH" = "multiarch" ]; then
else
if [ $LOCAL -eq 1 ]; then
DOCKER_ARGS="--load -t $DOCKER_ARCH/$DOCKER_IMAGE"
DOCKER_EXPORT="($DOCKER_ARCH/$DOCKER_IMAGE ghcr.io/eclipse/kuksa.val.services/$DOCKER_IMAGE:prerelease)"
DOCKER_EXPORT="($DOCKER_ARCH/$DOCKER_IMAGE ghcr.io/eclipse-kuksa/kuksa-incubation/$DOCKER_IMAGE:prerelease)"
else
DOCKER_ARGS="--platform linux/$DOCKER_ARCH -t $DOCKER_ARCH/$DOCKER_IMAGE --output type=oci,dest=$DOCKER_EXPORT"
fi
Expand All @@ -132,8 +132,8 @@ echo "# docker buildx build $DOCKER_ARGS --f seat_service/Dockerfile $CONTEXT_DI
DOCKER_BUILDKIT=1 docker buildx build $DOCKER_ARGS -f seat_service/Dockerfile "$CONTEXT_DIR" $DOCKER_EXT

if [ "$DOCKER_ARCH" != "multiarch" ]; then
echo "docker image tag $DOCKER_ARCH/$DOCKER_IMAGE ghcr.io/eclipse/kuksa.val.services/$DOCKER_IMAGE:prerelease"
docker image tag $DOCKER_ARCH/$DOCKER_IMAGE ghcr.io/eclipse/kuksa.val.services/$DOCKER_IMAGE:prerelease
echo "docker image tag $DOCKER_ARCH/$DOCKER_IMAGE ghcr.io/eclipse-kuksa/kuksa-incubation/$DOCKER_IMAGE:prerelease"
docker image tag $DOCKER_ARCH/$DOCKER_IMAGE ghcr.io/eclipse-kuksa/kuksa-incubation/$DOCKER_IMAGE:prerelease
fi

echo "# Exported $DOCKER_ARCH/$DOCKER_IMAGE in $DOCKER_EXPORT"
Expand Down
29 changes: 5 additions & 24 deletions seat_service/integration_test/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
Integration tests can be run in develop (local) mode for Dapr validation or with VAL docker containers (also running on CI).

- Integration tests in Local mode are using VS Code tasks defined in `.vscode/tasks.json`, see [README there](../.vscode/README.md).
- Container mode uses released containers for databroker and dbc2val and can also build seat and hvac service containers to test with unreleased changes.
- Container mode uses released containers for databroker and dbc2val and can also build seat service containers to test with unreleased changes.

##

## Integration test overview

Expand All @@ -16,25 +18,6 @@ Integration tests `./integration_test/test_*.py` are based on pytest and use ext
SeatService is tested in simulated CAN mode `CAN="cansim"`. It uses an external script for asking the seat to move to desired positions (0, 1000, invalid),
using `seat_svc_client` as grpc client.

### HVAC Service integration test

`integration_test/test_hvac.py` - This test checks if Hvac Service datapoints (`Vehicle.Cabin.DesiredAmbientAirTemperature`,
`Vehicle.Cabin.IsAirConditioningActive`) are registered in Kuksa Data Broker and and after invoking external script to call
HVAC client, checks if expected values are fed to Data Broker.

### CAN Feeder (dbc2val) integration test

`integration_test/test_feeder.py` - dbc2val is also included for integration testing as we don't have a master project to test all components.

This test covers several dbc2val datapoints (`Vehicle.OBD.Speed`, `Vehicle.OBD.EngineLoad`).

It checks if they are registered and also that some datapoints have changing values.

**NOTE:** dbc2val default config must be changed as it feeds to `kuksa.val` server by default.\
Custom configuration files are mapped to container as volume from: `./integration_test/volumes/dbc2val` and
environment variables (`FEEDERCAN_*`) are set in `./it-config` script and also in vscode task: `run-feedercan.sh`.\
Make sure `test_feeder.py` datapoints are available (and changing each second) in `./volumes/dbc2val/it-candump0.log`.

### Data Broker subscription helper

`.integration_test/broker_subscriber.py` - Helper for Data Broker grpc subscriptions. Not a testcase, but useful for command line scripting and checking events without Data Broker client.
Expand Down Expand Up @@ -81,7 +64,6 @@ Python Integration tests depend on the following VS Code tasks:
- `ensure-dapr`
- `run-databroker`
- `run-seatservice`
- `run-hvacservice`
- `run-feedercan`

It is possible to use VS Code `Testing` panel for debugging failed test cases and also directly running the python file in debug mode.
Expand All @@ -95,10 +77,9 @@ To force using this mode (e.g. in CI) export `USE_DAPR=0` environment variable f
Relevant scripts:

- `integration_test/it-config` : This config defines the used images, tags, and docker specific options per val container.\
**NOTE**: It gets Data Broker and dbc2val tags from `./prerequisite_settings.json`, and if `SEAT_TAG`, `HVAC_TAG` environment variables are set to `prerelease`,
seat_service and hvac_service are build from local sources and tagged as ghcr.io images.
**NOTE**: It gets Data Broker and dbc2val tags from `./prerequisite_settings.json`, and if `SEAT_TAG` environment variables are set to `prerelease`,
seat_service are build from local sources and tagged as ghcr.io images.
- `integration_test/it-seat-move.sh`: This script is used to execute `seat_svc_client` from seat_service container to initiate seat movement for integration tests.
- `integration_test/it-hvac-cli.sh`: This script is used to execute `./hvac_service/testclient.py` to request hvac temperature and status changes.
- `integration_test/it-setup.sh`: This is the main script handling val containers lifecycle:

```text
Expand Down
Loading

0 comments on commit d869593

Please sign in to comment.