High CPU usage when cargo updates its index

[gsurrel]

Problem

I used the command cargo install cross, and while downloading the crates index (few minutes), my CPU fan started making noise. After checking, a CPU code was at 100% usage.

Steps

1. Run cargo install cross (any anything that acually updated the index)
2. While waiting for the download to complete, check the usage

Possible Solution(s)

Maybe the display is updated each time a chunk of data is received, which triggers a lot of updates for a long time, in case the network connection is slow.

Notes

Output of cargo version, running on macOS 10.14.6:

$ cargo version
cargo 1.49.0 (d00d64df9 2020-12-05)

[Eh2406]

Sorry for not replying. Has this resolved itself, or do we have a way to reproduce it?

[gsurrel]

Yes, still the same. Actually, I checked more carefully:

• 2-10% load until 72% progress (Step "Updating crates.io index")
• Then, 90-100% load until completion of the index update

Tested now with cargo 1.52.0 (6976741 2021-04-21), on macOS 10.14.6

EDIT: it's 100% but on a single thread (there are others software that run well using the other cores at the same time, so it's not crushing the computer like the parallel compilation that comes afterward :))

[Eh2406]

Can you run a profiler on it when it is using a full cpu, and get some info on what it is doing and where it is spending its time?

[gsurrel]

How shall I do that? I've now real knowledge in performance profiling Rust apps, however I can easily execute commands :)

I installed Instruments and did the following:

• Create a new empty project
• Add a dependency to the project
• Remove the ~/.cargo/registry directory
• Run cargo update

The trace contains the following tracks:

• Time profiler
• Allocations
• Counters

Once I saved and compressed the track to share it, I have a 103 MB file. Should I send it somehow? Do you think you have the relevant data for investigating the issue?

[Eh2406]

I do not use Mac myself. @ehuss Would that file be worth looking at, or do you have advice for a next research step?

[ehuss]

Here's the instructions on how to run instruments:

1. Create a new Cargo project "foo".
2. cd foo
3. Add a dependency (I use cargo add bitflags, though any will do).
4. mkdir chome to create a new cargo home directory.
5. Open instruments.
6. Choose the blank template.
7. Choose the target...
   1. Find the cargo process. rustup which cargo will tell you the path. For me it is /Users/eric/.rustup/toolchains/nightly-x86_64-apple-darwin/bin/cargo. You may need to check "Show Hidden Files".
   2. Add environment variable CARGO_HOME set to /path/to/foo/chome (with the correct path)
   3. Add argument fetch
   4. Add working directory /path/to/foo
8. Add "Time Profiler"
9. Click record
10. Wait for it to finish (if it is a slow network, may take a few minutes)
11. Drag across the region in the timeline with high cpu

12. Click "Call Tree" > "Invert Call Tree" at the bottom of the window
13. You should now see functions that are expensive. You can expand the little arrows to see who is calling each function, and generally explore the profile.

Here's a picture of what mine looks like:

On mine, almost all of the high CPU time (about 9.5 seconds) is spent in sha1 and inflate.

[ehuss]

Out of curiousity, I tried a build using OpenSSL SHA1 instead of the custom collision-detection SHA1, and it had a pretty dramatic performance improvement (went from about 8s to 5s). I also noticed that zlib-ng was fair bit faster, too, though not as dramatic.

[gsurrel]

I did the manipulation, but I guess the information I got is quite useless:

Anyway, the CPU load is very similar to what you have so if there is any unexpected behavior in here, you can find it already. If everything is normal, this issue can be closed 😃