Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a set for TargetPython.get_tags for performance #12204

Merged
merged 10 commits into from
Aug 6, 2023

Commits on Aug 4, 2023

  1. Use a set for TargetPython.get_tags

    This speeds up a representative pip-compile by about 40%
    
    ```
    λ hyperfine -w 1 -M 3 -p 'rm req.txt || true' 'python -m piptools compile req.in --output-file req.txt --strip-extras --resolver=backtracking'
    Benchmark 1: python -m piptools compile req.in --output-file req.txt --strip-extras --resolver=backtracking
      Time (mean ± σ):     28.552 s ±  2.552 s    [User: 20.477 s, System: 1.547 s]
      Range (min … max):   26.446 s … 31.390 s    3 runs
    
    λ hyperfine -w 1 -M 3 -p 'rm req.txt || true' 'python -m piptools compile req.in --output-file req.txt --strip-extras --resolver=backtracking'
    Benchmark 1: python -m piptools compile req.in --output-file req.txt --strip-extras --resolver=backtracking
      Time (mean ± σ):     17.570 s ±  0.450 s    [User: 11.537 s, System: 1.345 s]
      Range (min … max):   17.095 s … 17.989 s    3 runs
    ```
    
    This makes the set.isdisjoint operation done over here much cheaper:
    https://github.com/pypa/pip/blob/593b85f4a/src/pip/_internal/models/wheel.py#L92
    
    The reason for this is because a Python usually has a lot of supported
    tags. The Python I used above has 2000 supported tags! Whereas a wheel
    usually only has one or two file tags.
    
    The CPython code will unfortunately iterate over all 2000 tags to check
    if there's a match. Only if the other collection is a set will CPython
    think to swap the operands to iterate over the shorter collection:
    https://github.com/python/cpython/blob/35963da40f/Objects/setobject.c#L1352
    hauntsaninja committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    91c4fc4 View commit details
    Browse the repository at this point in the history
  2. news

    hauntsaninja committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    bcde032 View commit details
    Browse the repository at this point in the history
  3. fix test

    hauntsaninja committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    47f21a7 View commit details
    Browse the repository at this point in the history
  4. get_sorted_tags

    hauntsaninja committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    c17ff03 View commit details
    Browse the repository at this point in the history
  5. get_unsorted_tags

    hauntsaninja committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    53f9ce7 View commit details
    Browse the repository at this point in the history
  6. test failure

    hauntsaninja committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    a0c804f View commit details
    Browse the repository at this point in the history
  7. remove caching test

    hauntsaninja committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    ba7863d View commit details
    Browse the repository at this point in the history
  8. fix ordering

    hauntsaninja committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    f8fbbb1 View commit details
    Browse the repository at this point in the history
  9. delint

    hauntsaninja committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    18926cb View commit details
    Browse the repository at this point in the history
  10. fix nits

    hauntsaninja committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    dfdf4ec View commit details
    Browse the repository at this point in the history