Make cache management more dynamic #3578
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
PBENCH-1301 This could be refined, but I want to get it up for review. I'd love to get it on the staging server for testing with more limited disk space, although I can ask for 95% or 880Gib on my laptop and see it free cache. (I'm out all next week, but I may spend a bit of time trying to clean this up: e.g., trying to add some unit tests.) First, this adds code to track the unpacked size of tarballs to help with managing cache goals. This is based on a `du -s -B1` of the unpacked directory tree, and stored as metadata. When asked to unpack, it will check whether there's sufficient space on the cache device, and if not it will reclaim cache with a goal sufficient to accommodate the target dataset. The `pbench-tree-manage` command can now target either % free or bytes free, and the background timer job will attempt to free 20% of the drive every 4 hours instead of targeting "old" tarballs. (The `last_ref` timestamp is now used only to sort the list of datasets with live cache on input to reclaim so that we free the oldest first.)
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
That's old news, now. I resubmitted your build and it failed unit tests. |
|
Actually, it did both. I'm not sure why the cleanup is generating a blank argument to |
|
Ah. It's more of that "asynchronous pytest run with implicit dependencies". Dang. So it is a real problem. 😦 |
Apparently the filtered output of
😞 |
|
While you're thinking about this stuff...did you see my comment from the other week? |
Yeah, I saw that. I was tempted to just pull it into this PR although it deserves its own and I'm not particularly inclined to deal with that now. Then again, maybe I should just throw it into the pot ... |
OK, good.
No, no...you are right to resist that temptation. (I just didn't know how closely related or how big a change it would require.) |
It's a trivial one-character change, I think, since I believe that the plural form they want us to use takes |
`Tarball.__init__` now tries to look up a `Dataset`, and therefore (if the constructor isn't mocked out) we need a DB. In parallel execution, we may hit a test requiring DB without a fixture to initialize it. The failures are therefore somewhat random. I've added fixtures to fix the ones I can reproduce locally with `jenkins/run tox -- server python -n16`. TBD whether the CI finds more!
PBENCH-1301
This could be refined, but I want to get it up for review. I'd love to get it on the staging server for testing with more limited disk space, although I can ask for 95% or 880Gib on my laptop and see it free cache. (I'm out all next week, but I may spend a bit of time trying to clean this up: e.g., trying to add some unit tests.)
First, this adds code to track the unpacked size of tarballs to help with managing cache goals. This is based on a
du -s -B1of the unpacked directory tree, and stored as metadata.When asked to unpack, it will check whether there's sufficient space on the cache device, and if not it will reclaim cache with a goal sufficient to accommodate the target dataset.
The
pbench-tree-managecommand can now target either % free or bytes free, and the background timer job will attempt to free 20% of the drive every 4 hours instead of targeting "old" tarballs. (Thelast_reftimestamp is now used only to sort the list of datasets with live cache on input to reclaim so that we free the oldest first.)