Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed up CI #2635

Closed
emilk opened this issue Jul 7, 2023 · 4 comments
Closed

Speed up CI #2635

emilk opened this issue Jul 7, 2023 · 4 comments
Assignees
@jprochazk
Labels
😤 annoying Something in the UI / SDK is annoying to use 🧑‍💻 dev experience developer experience (excluding CI)

Comments

@emilk
Copy link
Member

emilk commented Jul 7, 2023 •
edited
Loading

The CI is so slow these days. I can't take it.

  • Build Web Wasm 4m20s
  • Build wheels: 11m
    • Building: 5m30s
    • e2e roundtrip tests: 3m30s
  • Build Web Demo 4m40s
  • Upload Web Demo: 40s

Ideas:

  • Is the web-viewer built twice (again in "Build wheels")?
  • Can we speed up e2e roundtrip tests? And/or just run on main - they seldom catch anything anyway
  • What is slow when building web demo? Can we put the required datasets on the Docker image instead of downloading them?
  • We can use a smaller/faster set of examples for test.rerun.io
@emilk emilk added 🧑‍💻 dev experience developer experience (excluding CI) 😤 annoying Something in the UI / SDK is annoying to use labels Jul 7, 2023
@abey79
Copy link
Member

abey79 commented Jul 11, 2023 •
edited
Loading

I just noticed that, is it known?

image

Here: https://github.com/rerun-io/rerun/actions/runs/5517330792/jobs/10059800524

@jprochazk jprochazk mentioned this issue Jul 18, 2023
Merged
3 tasks
jprochazk added a commit that referenced this issue Jul 18, 2023
@jprochazk
Use large runners (#2740)
4c1cc8a 
<!--
Open the PR up as a draft until you feel it is ready for a proper
review.

Do not make PR:s from your own `main` branch, as that makes it difficult
for reviewers to add their own fixes.

Add any improvements to the branch as new commits to make it easier for
reviewers to follow the progress. All commits will be squashed to a
single commit once the PR is merged into `main`.

Make sure you mention any issues that this PR closes in the description,
as well as any other related issues.

To get an auto-generated PR description you can put "copilot:summary" or
"copilot:walkthrough" anywhere.
-->

### What

Part of #2635

We weren't actually using the larger runners for certain very expensive
jobs, specifically the wheels build was using the basic 2-core runners.

Note: Review without the first commit, it's just formatting

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/2740) (if
applicable)

- [PR Build Summary](https://build.rerun.io/pr/2740)
- [Docs preview](https://rerun.io/preview/pr%3Ajan%2Fbig-machines/docs)
- [Examples
preview](https://rerun.io/preview/pr%3Ajan%2Fbig-machines/examples)
@jprochazk
Copy link
Member

jprochazk commented Jul 21, 2023 •
edited
Loading

After a few PRs:

I'm happy to report that we have shaved off anywhere from 8 to 12 minutes from the "main" path of the PR CI runs.

What this means is build web + build wheel -> build demo app -> upload web demo takes about 8-12 minutes, where previously it took about 17-20 minutes. (Side note: rendering done by GHA here is not great, jobs overlap in strange ways, you have to hover a job to see the actual dependency path between some subset of the jobs)

There's still a lot of room for improvement.

  • Cache more things in GCS CI: Cache everything in cloud storage #2762
  • Better job sequencing (figure out how to get cargo to not compile everything 4 times in the lint job)
  • (ab)use Docker more (to store datasets and/or maybe even support incremental compilation to some extent...?)

@Wumpf
Copy link
Member

Wumpf commented Jul 21, 2023

Lot of work done this cycle. Let's create a new tracking issue for remaining items @jprochazk

@Wumpf Wumpf closed this as completed Jul 21, 2023
@jprochazk
Copy link
Member

#2776

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jprochazk jprochazk

Labels
😤 annoying Something in the UI / SDK is annoying to use 🧑‍💻 dev experience developer experience (excluding CI)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@emilk @Wumpf @jprochazk @abey79