Skip to content

Commit

Permalink
Status of quarantined tests is stored in Github Issue (#10119)
Browse files Browse the repository at this point in the history 
(cherry picked from commit 86d8e34)
  • Loading branch information
@potiuk
potiuk committed Nov 16, 2020
1 parent 3465802 commit 9292c5e
Show file tree
Hide file tree
Showing 7 changed files with 375 additions and 7 deletions.
67 changes: 61 additions & 6 deletions CI.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,8 @@ The following components are part of the CI infrastructure
CI run types
============

The following CI Job runs are currently run for Apache Airflow, and each of the runs have different
purpose and context.
The following CI Job run types are currently run for Apache Airflow (run by ci.yaml workflow and
quarantined.yaml workflows) and each of the run types have different purpose and context.

Pull request run
----------------
Expand Down Expand Up @@ -126,7 +126,17 @@ DockerHub when pushing ``v1-10-stable`` manually.
All runs consist of the same jobs, but the jobs behave slightly differently or they are skipped in different
run categories. Here is a summary of the run categories with regards of the jobs they are running.
Those jobs often have matrix run strategy which runs several different variations of the jobs
(with different Backend type / Python version, type of the tests to run for example)
(with different Backend type / Python version, type of the tests to run for example). The following chapter
describes the workflows that execute for each run.

Workflows
=========

CI Build Workflow
-----------------

This workflow is a regular workflow that performs the regular checks - none of the jobs should fail.
The tests to run do not contain quarantined tests.

+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
| Job | Description | Pull Request Run | Direct Push/Merge Run | Scheduled Run |
Expand All @@ -148,17 +158,62 @@ Those jobs often have matrix run strategy which runs several different variation
+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
| Tests Kubernetes | Run Kubernetes test | Yes (if tests-triggered) | Yes | Yes * |
+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
| Quarantined tests | Those are tests that are flaky and we need to fix them | Yes (if tests-triggered) | Yes | Yes * |
| Test OpenAPI client gen | Tests if OpenAPIClient continues to generate | Yes | Yes | Yes * |
+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
| Helm tests | Runs tests for the Helm chart | Yes | Yes | Yes * |
+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
| Constraints | Upgrade constraints to latest eagerly pushed ones (only if tests successful) | - | Yes | Yes * |
| Constraints | Upgrade constraints to latest eagerly pushed ones (only if tests successful) | - | Yes | Yes * |
+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
| Constraints push | Pushes updated constraints (only if tests successful) | - | Yes | - |
| Constraints push | Pushes updated constraints (only if tests successful) | - | Yes | Yes * |
+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
| Push Prod images | Pushes production images to GitHub Private Image Registry to cache the build images for following runs | - | Yes | - |
+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
| Push CI images | Pushes CI images to GitHub Private Image Registry to cache the build images for following runs | - | Yes | - |
+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
| Tag Repo nightly | Tags the repository with nightly tagIt is a lightweight tag that moves nightly | - | - | Yes. Triggers DockerHub build for public registry |
+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+

Quarantined build workflow
--------------------------

This workflow runs only quarantined tests. Those tests do not fail the build even if some tests fail (only if
the whole pytest execution fails). Instead this workflow updates one of the issues where we keep status
of quarantined tests. Once the test succeeds in NUM_RUNS subsequent runs, it is marked as stable and
can be removed from quarantine. You can read more about quarantine in `<TESTING.rst>`_

The issues are only updated if the test is run as direct push or scheduled run and only in the
``apache/airflow`` repository - so that the issues are not updated in forks.

The issues that gets updated are different for different branches:

* master: `Quarantine tests master <https://github.com/apache/airflow/issues/10118>`_
* v1-10-stable: `Quarantine tests v1-10-stable <https://github.com/apache/airflow/issues/10127>`_
* v1-10-test: `Quarantine tests v1-10-test <hhttps://github.com/apache/airflow/issues/10128>`_

+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
| Job | Description | Pull Request Run | Direct Push/Merge Run | Scheduled Run |
+===========================+================================================================================================================+====================================+=================================+======================================================================+
| Cancel previous workflow | Cancels the previously running workflow run if there is one running | Yes | Yes | Yes * |
+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
| Trigger tests | Checks if tests should be triggered | Yes | Yes | Yes * |
+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
| Quarantined tests | Those are tests that are flaky and we need to fix them | Yes (if tests-triggered) | Yes (Updates quarantine issue) | Yes * (updates quarantine issue) |
+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+

Cancel other workflow runs workflow
-----------------------------------

This workflow is run only on schedule (every 5 minutes) it's only purpose is to cancel other running
``CI Build`` workflows if important jobs failed in those runs. This is to save runners for other runs
in case we know that the build will not succeed anyway without some basic fixes to static checks or
documentation - effectively implementing missing "fail-fast" (on a job level) in Github Actions
similar to fail-fast in matrix strategy.

The jobs that are considered as "fail-fast" are:

* Static checks
* Docs
* Prepare Backport packages
* Helm tests
* Build Prod Image
* TTest OpenAPI client gen
4 changes: 4 additions & 0 deletions scripts/ci/docker-compose/base.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,10 @@ services:
- RUN_INTEGRATION_TESTS
- ONLY_RUN_LONG_RUNNING_TESTS
- ONLY_RUN_QUARANTINED_TESTS
- GITHUB_TOKEN
- GITHUB_REPOSITORY
- ISSUE_ID
- NUM_RUNS
- BREEZE
- INSTALL_AIRFLOW_VERSION
- DB_RESET
Expand Down
32 changes: 32 additions & 0 deletions scripts/ci/in_container/quarantine_issue_header.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Quarantined issues

Please do not update status or list of the issues manually. It is automatically updated during
Quarantine workflow, when the workflow executes in the context of Apache Airflow repository.
This happens on schedule (4 times a day) or when a change has been merged or pushed
to the relevant branch.

You can update "Comment" column in the issue list - the update process will read and preserve this column.

# Status update
Last status update (UTC): {{ DATE_UTC_NOW }}

# List of Quarantined issues
25 changes: 25 additions & 0 deletions scripts/ci/in_container/run_ci_tests.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,31 @@ if [[ "${RES}" == "0" && ${CI:="false"} == "true" ]]; then
bash <(curl -s https://codecov.io/bash)
fi

MAIN_GITHUB_REPOSITORY="apache/airflow"

if [[ ${ONLY_RUN_QUARANTINED_TESTS:=} = "true" ]]; then
if [[ ${GITHUB_REPOSITORY} == "${MAIN_GITHUB_REPOSITORY}" ]]; then
if [[ ${RES} == "1" || ${RES} == "0" ]]; then
echo
echo "Pytest exited with ${RES} result. Updating Quarantine Issue!"
echo
"${IN_CONTAINER_DIR}/update_quarantined_test_status.py" "${RESULT_LOG_FILE}"
else
echo
echo "Pytest exited with ${RES} result. NOT Updating Quarantine Issue!"
echo
fi
else
echo
echo "Github repository '${GITHUB_REPOSITORY}'. NOT Updating Quarantine Issue!"
echo
fi
else
echo
echo "Regular tests. NOT Updating Quarantine Issue!"
echo
fi

if [[ ${CI:=} == "true" ]]; then
dump_airflow_logs
fi
Expand Down
Loading

0 comments on commit 9292c5e

Please sign in to comment.