Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: speed up win64 test #8728

Merged
merged 41 commits into from
Jan 6, 2024
Merged

ci: speed up win64 test #8728

merged 41 commits into from
Jan 6, 2024

Conversation

Jefffrey
Copy link
Contributor

@Jefffrey Jefffrey commented Jan 3, 2024

Which issue does this PR close?

Relates to #8696

Rationale for this change

Apply changes per analysis from rust-lang/rust#119560

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@Jefffrey Jefffrey changed the title Change back to rust 1.74.1 for testing win CI Debugging win CI slowness Jan 3, 2024
@Jefffrey
Copy link
Contributor Author

Jefffrey commented Jan 3, 2024

Limit to running only tpcds_physical_q44 with backtrace feature:

good (2023-10-29 nightly) = 70 seconds

bad (2023-10-30 nightly) = 692 seconds

Without backtrace:

good (2023-10-29 nightly) = 1.25 seconds

bad (2023-10-30 nightly) = 1.26 seconds


Trying more recent toolchains, with backtrace:

2023-12-21 nightly = 682 seconds

2024-01-01 nightly = 682 seconds

@github-actions github-actions bot added the sql SQL Planner label Jan 6, 2024
# use higher optimization level to overcome Windows rust slowness for tpc-ds
# and speed builds: https://github.com/apache/arrow-datafusion/issues/8696
# Cargo profile docs https://doc.rust-lang.org/cargo/reference/profiles.html?profile-settings#profile-settings
RUSTFLAGS: "-C debuginfo=0 -C opt-level=1 -C target-feature=+crt-static -C incremental=false -C codegen-units=256"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the other optimization flags as it caused compilation to take 27 min

Without them, it takes 8 min

@Jefffrey Jefffrey changed the title Debugging win CI slowness ci: speed up win check Jan 6, 2024
@Jefffrey Jefffrey changed the title ci: speed up win check ci: speed up win64 test Jan 6, 2024
@Jefffrey Jefffrey marked this pull request as ready for review January 6, 2024 10:10
@Jefffrey
Copy link
Contributor Author

Jefffrey commented Jan 6, 2024

win64 test ran in 33min

Still slightly slower than before (around 27min I believe)

cc @comphead @alamb

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much @Jefffrey This looks really good to me. Also watching you debug this issue I think is a master class in careful engineering. Very impressive 💪

Screenshot 2024-01-06 at 9 44 33 AM

# Cargo profile docs https://doc.rust-lang.org/cargo/reference/profiles.html?profile-settings#profile-settings
RUSTFLAGS: "-C debuginfo=0 -C opt-level=1 -C target-feature=+crt-static -C incremental=false -C codegen-units=256"
# Minimize producing debug symbols to keep memory usage down
# Set debuginfo=line-tables-only as debuginfo=0 causes immensely slow build
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 thank you

Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, wow, so debuginfo another level fixed the problem, nicely done @Jefffrey

@comphead comphead merged commit 4289737 into apache:main Jan 6, 2024
22 checks passed
@Jefffrey Jefffrey deleted the test_win_ci branch January 7, 2024 00:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants