Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure Pip cache operations are atomic. #1778

Merged
merged 1 commit into from
May 25, 2022
Merged

Conversation

jsirois
Copy link
Member

@jsirois jsirois commented May 25, 2022

By forcing the temporary directories Pip creates to be on the same
filesystem as the Pip cache, we indirectly ensure the caching Pip does
is atomic and safe in the presence of parallel Pex runs.

Fixes #1776

By forcing the temporary directories Pip creates to be on the same
filesystem as the Pip cache, we indirectly ensure the caching Pip does
is atomic and safe in the presence of parallel Pex runs.

Fixes pex-tool#1776
@jsirois
Copy link
Member Author

jsirois commented May 25, 2022

I hand-tested this with the same repro.sh mentioned in the slack thread on this issue:

#!/usr/bin/env bash

set -euo pipefail

rm -rf ~/.pex/built_wheels/ ~/.pex/installed_wheels/ ~/.pex/wheels/ psutil-*.pex

pids=()
for i in $(seq 1 5); do
  python -m pex --no-wheel psutil -o psutil-$i.pex "$@" &
  pids+=($!)
done
for pid in ${pids[@]}; do
  wait ${pid}
done

That runs without failure (I waited for >10 attempts) via:

 count=0; while ./repro.sh; do echo "Attempt $((count+=1))"; done
Attempt 1
Attempt 2
Attempt 3
Attempt 4
Attempt 5
Attempt 6
Attempt 7
Attempt 8
Attempt 9
Attempt 10
Attempt 11
Attempt 12
Attempt 13
Attempt 14
Attempt 15
Attempt 16
Attempt 17
Attempt 18
Attempt 19
Attempt 20
Attempt 21
Attempt 22
Attempt 23
Attempt 24
Attempt 25
Attempt 26
Attempt 27
^C

@jsirois jsirois merged commit 2bc67ef into pex-tool:main May 25, 2022
@jsirois jsirois deleted the issues/1776 branch May 25, 2022 22:37
@stuhood
Copy link

stuhood commented May 31, 2022

Do we think that this might have been related to pantsbuild/pants#15003?

@jsirois
Copy link
Member Author

jsirois commented May 31, 2022

Yes, that Pants issue spawned this fix. I've updated / closed that ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pex use of pip wheel to build sdists into wheels can be non-atomic.
3 participants