Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request: bulk: give priority to release requests over stage requests #7632

Open
onnozweers opened this issue Aug 6, 2024 · 1 comment
Open

Comments

@onnozweers
Copy link
Contributor

onnozweers commented Aug 6, 2024

Dear dCache devs,

My impression is, that currently, the bulk service processes requests in the order they come in. The number of threads (concurrently active requests) is limited, so it can happen that a few stage requests are keeping all threads busy, and release (=unpin) requests are not getting through because they remain queued.

So, you might have a situation where, for instance, Atlas is staging a lot, and the stage pools fills up, and the unpin requests are waiting for the stage operations to complete, so they are not getting through, although they would be the solution for the space congestion.

I've had to do some tuning to make this work better:

# Number of concurrent active requests.
# Default: 100
# If stage pools fill up, increasing this may help 
# to let "release" requests through. 
bulk.limits.container-processing-threads=1000

With 1000 threads, I saw that unpin requests could get through so they could free up space so that staging could continue.

So I'd like to suggest that release requests have priority over stage requests. Release requests take very little time and they free up the necessary space for staging to succeed. If that is too difficult or undesired, it might help a lot to increase the default number of threads. And perhaps while we're at it, bulk.limits.max-requests-per-user could have a bigger default too (like 50000 instead of 5000). Atlas would be very happy.

An alternative idea might be to reserve some threads for only release requests, so they never stall when stage requests are stuck.

Cheers,
Onno

@onnozweers onnozweers changed the title Request: bulk: give priority to release requests over stage requests Request: bulk: give priority to release requests over stage requests Aug 6, 2024
@onnozweers
Copy link
Contributor Author

I see now that other schedulers have already been anticipated: https://github.com/dCache/dcache/blob/b772cd54a1ae45453da86dd87649add89f6e4745/skel/share/defaults/bulk.properties#L50C1-L58C92

#  ---- Algorithm for promoting requests to active.
#
#       Implementations:
#
#           -   org.dcache.services.bulk.manager.scheduler.LeastRecentFirstScheduler:
#               selects the least recent requests to run in order of arrival
#               (first-come first-served).
#
bulk.request-scheduler=org.dcache.services.bulk.manager.scheduler.LeastRecentFirstScheduler

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant