refactor: pypi mapping into cached client #3318
Conversation
|
#3322 This should fix the |
|
I asked @mariusvniekerk on Zulip to try this. Maybe @tobiasfischer or @matthewfeickert could also give it a go to see if that helps with slow PyPI mapping issues that you observed. |
@wolfv I was too slow in testing the original PR @wolfv linked me to on Discord, and the artifacts had expired. You'd like testing of the Updating lock file takes forever Discord questions with the build artifacts from this PR though? |
|
yes, @matthewfeickert - this one is the better implementation. If you could test again with artifacts from this PR here that would be great. |
| } | ||
|
|
||
| /// Given a set of `RepoDataRecord`s, amend the purls for each record. | ||
| pub async fn ament_purls( |
There was a problem hiding this comment.
| pub async fn ament_purls( | |
| pub async fn amend_purls( |
| } else if custom_mappings.is_mapping_for_record(record) { | ||
| custom_mappings.derive_purls(record).await | ||
| } else { | ||
| self.derive_purls_from_clients(record).await |
There was a problem hiding this comment.
could we implement DerivePurls for MappingClinet itself? and inside it will call both hash_mapping or compressed
|
|
||
| let derived_purls = if matches!(mapping_source, MappingSource::Disabled) { | ||
| Ok(None) | ||
| } else if custom_mappings.is_mapping_for_record(record) { |
There was a problem hiding this comment.
just a nitpick ( not sure if it was written previously by me 😓 )
we check if custom_mappings = mapping_source, otherwise use default to create it ( meaning that we don't have custom location ) .
here we always will ask is_mapping_for_record, even if the custom mapping is explicitly not present.
can we make it a match on mapping_source, and just act directly based on what type of the mapping we have?
|
I've tested it on a slow mapping example, as reported by @mariusvniekerk . after profiling with cargo flamegraph I can't find any samples for |
|
@baszalmstra can we close this PR then? #3079 |
|
Yep! |
Refactors the pypi mapping in a more client based approach. The
MappingClienthas a function that can be called to amend purls. It also serves as the entry point for coalescing of requests and in memory caching.Also fixes #2615