build: try to speed up Github workflows #12090

ktmud · 2020-12-17T04:19:02Z

SUMMARY

CI has been slow recently. This PR is the latest effort in trying to mitigate:

Update the Cancel Previous Run job added in build: cancel previous github action runs #11940 to cancel future jobs as well.
Skip dependabot branches when possible (e.g. don't run Python jobs for npm updates).
Merge lighter actions (Prefer Typescript and License Check) into one workflow (there are overheads in starting a job).

Add a script to allow committers to manually cancel CI jobs in queue. This is useful on a busy day when the CI pipeline got clogged by continuous new commits to open PRs. Users have to configure GITHUB_TOKEN in their env variables in order to use this.

(superset) ➜  incubator-superset git:(manage-github-workflow) ✗ ./scripts/cancel_github_workflows.py --help 
Usage: cancel_github_workflows.py [OPTIONS] BRANCH_OR_PULL

  Cancel running or queued GitHub workflows by branch or pull request ID.

  By default jobs that are already running or are the latest of the branch
  or PR will not be cancelled.

Options:
  --repo TEXT                     Default is apache/incubator-superset
  --event [pull_request|push|issue]
  --include-last / --no-include-last
                                  Whether to also cancel the lastest run.
  --include-running / --no-include-running
                                  Whether to also cancel running workflows
  --help                          Show this message and exit.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A

TEST PLAN

Test locally and CI

ADDITIONAL INFORMATION

Has associated issue: this is a rerun of build: try to speed up Github workflows #12089 because it was blocked by CI
Changes UI
Requires DB Migration.
Confirm DB Migration upgrade and downgrade tested.
Introduces new feature or API
Removes existing feature or API

codecov-io · 2020-12-17T04:33:17Z

Codecov Report

Merging #12090 (85993fd) into master (eb3c2b2) will decrease coverage by 7.92%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #12090      +/-   ##
==========================================
- Coverage   67.14%   59.22%   -7.93%     
==========================================
  Files        1002      951      -51     
  Lines       49300    46658    -2642     
  Branches     5010     4312     -698     
==========================================
- Hits        33103    27633    -5470     
- Misses      16072    19025    +2953     
+ Partials      125        0     -125

Flag	Coverage Δ
cypress	`51.44% <ø> (+<0.01%)`	⬆️
javascript	`?`
python	`63.59% <ø> (-0.57%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...uperset-frontend/src/dashboard/util/dnd-reorder.js	`0.00% <0.00%> (-100.00%)`	⬇️
...rset-frontend/src/dashboard/util/getEmptyLayout.js	`0.00% <0.00%> (-100.00%)`	⬇️
...dashboard/components/resizable/ResizableHandle.jsx	`0.00% <0.00%> (-100.00%)`	⬇️
.../src/dashboard/util/getFilterScopeFromNodesTree.js	`0.00% <0.00%> (-93.48%)`	⬇️
...set-frontend/src/views/CRUD/alert/ExecutionLog.tsx	`11.76% <0.00%> (-88.24%)`	⬇️
...src/dashboard/components/gridComponents/Header.jsx	`10.52% <0.00%> (-86.85%)`	⬇️
superset-frontend/src/components/IconTooltip.tsx	`13.33% <0.00%> (-86.67%)`	⬇️
...rc/dashboard/components/gridComponents/Divider.jsx	`13.33% <0.00%> (-86.67%)`	⬇️
superset/db_engines/hive.py	`0.00% <0.00%> (-85.72%)`	⬇️
...end/src/SqlLab/components/ExploreResultsButton.jsx	`8.00% <0.00%> (-84.00%)`	⬇️
... and 378 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb3c2b2...85993fd. Read the comment docs.

Add a script to cancel previous GitHub workflows and optimize the workflows in hope to speed up CI. - Skip dependabot for python jobs - Merge Prefer TypeScript and License Check - Cancel all tests in one workflow

nytai · 2021-01-06T21:07:31Z

.github/workflows/misc.yml

+        continue-on-error: true
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:


This check is essentially useless since the comments will never work for PRs coming from forks (due to the GITHUB_TOKEN access). This would be a good candidate to port over to the workflow_run approach mentioned in https://securitylab.github.com/research/github-actions-preventing-pwn-requests. Obviously, out of scope for this PR

villebro

A few non-blocking comments, LGTM.

villebro · 2021-01-07T08:05:28Z

scripts/cancel_github_workflows.py

+from typing import Iterable, List, Optional, Union
+
+import click
+import requests


Whenever possible, we should probably try to use urllib3 over requests to avoid licensing issues.

Isn't it using Apache 2.0 License as well?

Oh wow, they've apparently relicensed. There was a refactor effort some time ago to remove explicit dependency on requests as it wasn't Apache compatible back then (see #7643). Good to know.

Hmm, I can't find any reference of it ever having been anything but Apache 2.0. Apparently I'm misremembering something or this has been a major misunderstanding.

I couldn't let this one go.. 😄 I read old mailing list archives and turned up this: https://lists.apache.org/thread.html/49c4d03d3aadab5c2d1c20ccf1cb7ca02edbc94961e6429dbf6a082b%40%3Cdev.superset.apache.org%3E Turns out requests was pulling in chardet which is LGPL, which was the problem. Turns out chardet was dropped as of 09/2020: encode/httpx#1269 🎉

So it means it's OK to use requests again, I guess?

Uhh, I went down a link in this issue without noticing that it was pointing to another project: psf/requests#4848 Apparently requests still uses chardet, and there are no plans to remove it. As long as this is the case we're not allowed to reference requests as a required dependency, but it's ok to be had as an optional dependency. As long as we restrict the usage to development scripts we're fine, but we shouldn't use it in the core code.

Got it. Thanks for the clarification and investigation!

villebro · 2021-01-07T08:11:00Z

.github/workflows/superset-python-unittest.yml

@@ -0,0 +1,141 @@
+# Python MySQL unit tests


I'm mostly indifferent to having these split up versus all in one file, but I'd like to avoid shuffling these around unnecessarily to avoid losing the git history.

There was a hypothesis that there are overheads involved in scheduling multiple workflows vs running them in one. Not sure how much it helps but I still think it makes sense to combine these three considering they are very similar in nature and not particularly expensive to run.

pull-request-size bot added the size/XL label Dec 17, 2020

ktmud force-pushed the manage-github-workflow branch from 16a2754 to c6ddbbf Compare December 21, 2020 07:47

ktmud force-pushed the manage-github-workflow branch 2 times, most recently from 03494b0 to 1c4aba0 Compare January 4, 2021 18:36

pull-request-size bot added size/L and removed size/XL labels Jan 4, 2021

ktmud force-pushed the manage-github-workflow branch from 1c4aba0 to 3ca6ea6 Compare January 5, 2021 16:51

pull-request-size bot added size/XL and removed size/L labels Jan 5, 2021

ktmud force-pushed the manage-github-workflow branch from e8faa48 to 98e3922 Compare January 5, 2021 22:35

ktmud closed this Jan 5, 2021

ktmud reopened this Jan 5, 2021

ktmud force-pushed the manage-github-workflow branch 3 times, most recently from 35c8d68 to 29df9f4 Compare January 6, 2021 00:06

ktmud added 6 commits January 6, 2021 12:54

build: another approach to cancel Github workflows

556b52e

Add a script to cancel previous GitHub workflows and optimize the workflows in hope to speed up CI. - Skip dependabot for python jobs - Merge Prefer TypeScript and License Check - Cancel all tests in one workflow

Display number of jobs to cancel

0622739

build: ignore github actions from dependabot

be3098b

Merge mysql postgres and sqlite tests

abf1d21

Don't use pull_request_target

6434014

Dont ignore pull request checks

85993fd

ktmud force-pushed the manage-github-workflow branch from 29df9f4 to 85993fd Compare January 6, 2021 20:54

nytai approved these changes Jan 6, 2021

View reviewed changes

nytai reviewed Jan 6, 2021

View reviewed changes

villebro approved these changes Jan 7, 2021

View reviewed changes

ktmud merged commit f482849 into apache:master Jan 7, 2021

ktmud deleted the manage-github-workflow branch January 7, 2021 08:47

ktmud mentioned this pull request Jan 10, 2021

build: add cancel-workflow-runs action locally #12394

Merged

6 tasks

ktmud mentioned this pull request Jan 20, 2021

fix(CI): properly configure cancel duplicates #12625

Merged

6 tasks

ktmud mentioned this pull request Jan 21, 2021

fix(CI): remove duplicate hive and presto tests #12638

Merged

6 tasks

sebastianliebscher mentioned this pull request Sep 9, 2023

build: use GitHub Actions concurrency groups to cancel workflows #25243

Closed

9 tasks

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.0.0 labels Mar 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: try to speed up Github workflows #12090

build: try to speed up Github workflows #12090

ktmud commented Dec 17, 2020 •

edited

Loading

codecov-io commented Dec 17, 2020 •

edited

Loading

nytai Jan 6, 2021

villebro left a comment

villebro Jan 7, 2021

ktmud Jan 7, 2021

villebro Jan 7, 2021

villebro Jan 7, 2021

villebro Jan 9, 2021

ktmud Jan 9, 2021

villebro Jan 10, 2021

ktmud Jan 10, 2021

villebro Jan 7, 2021

ktmud Jan 7, 2021

build: try to speed up Github workflows #12090

build: try to speed up Github workflows #12090

Conversation

ktmud commented Dec 17, 2020 • edited Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

codecov-io commented Dec 17, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ktmud commented Dec 17, 2020 •

edited

Loading

codecov-io commented Dec 17, 2020 •

edited

Loading