Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle highly concurrent conversion tasks #46

Closed
imeoer opened this issue Jul 11, 2022 · 2 comments · Fixed by #143
Closed

Handle highly concurrent conversion tasks #46

imeoer opened this issue Jul 11, 2022 · 2 comments · Fixed by #143
Labels
bug Something isn't working

Comments

@imeoer
Copy link
Collaborator

imeoer commented Jul 11, 2022

These errors are likely to be thrown when acceld is handling highly concurrent conversion tasks:

convert in worker: convert image: failed commit on ref "converter-write-json-sha256:xxx": commit failed: rename /var/root/io.containerd.content.v1.content/ingest/yyy/data /var/root/io.containerd.content.v1.content/blobs/sha256/xxx: no such file or directory: unknown

convert in worker: push image: content digest sha256:zzz: not found

And containerd get locks slows down each task, containerd logs:

ERRO[2022-07-11T10:28:16.564583960Z] (*service).Write failed                       error="rpc error: code = Unavailable desc = ref default/1/convert-estargz-from-sha256:xxx locked for 5.41890936s (since 2022-07-11 10:28:10.633991309 +0000 UTC m=+98.894109053): unavailable" ref="convert-estargz-from-sha256:xxx"

Reproduciable steps:

  1. acceld --config ./misc/config.yaml.estargz.tmpl
  2. create a batch of conversion tasks:
for i in (seq 1 50)                                                                                                                                                                                                             
   accelctl task create localhost/busybox:latest
end
@imeoer imeoer added the bug Something isn't working label Jul 11, 2022
@imeoer
Copy link
Collaborator Author

imeoer commented Jul 12, 2022

Try to fix in containerd/containerd#7155

@Desiki-high
Copy link
Collaborator

Desiki-high commented May 27, 2023

@imeoer Can we use the singleflight in LocalAdapter.Dispatch to resolve the same conversion tasks?

Desiki-high added a commit to Desiki-high/acceleration-service that referenced this issue May 30, 2023
Fixes: goharbor#46

Signed-off-by: Yadong Ding <ding_yadong@foxmail.com>
Desiki-high added a commit to Desiki-high/acceleration-service that referenced this issue May 30, 2023
Fixes: goharbor#46

Signed-off-by: Yadong Ding <ding_yadong@foxmail.com>
Desiki-high added a commit to Desiki-high/acceleration-service that referenced this issue May 31, 2023
Fixes: goharbor#46

Signed-off-by: Yadong Ding <ding_yadong@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants