-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
questions about optimizing image downloads - currently rate stuck at 1 image per 5 seconds #69
Comments
related notes involving bisque internal performance tests: |
@jhpoelen I suggest transferring large data directly through the CyVerse Data Store, such as using the CLI aka iCommands, rather than transferring through bisque directly: https://learning.cyverse.org/ds/move_data/ |
@edwins thanks for your pointer and glad to hear there's another method to access CyVerse resources. I tried to install the icommands cli on target system: Linux larus 5.15.0-58-generic #64-Ubuntu , and found:
and
Do you happen to know who to contact about this installation issue? |
Hi @jhpoelen If you have dependency issues while installing iCommands, Gocommands is another option. It is highly portable. |
@iychoi - thanks! I was able to download Gocommands, and run it -
Now, I'll try and figure out how to configure the tool to get the images associated with endpoints like -
see also bio-guoda/preston-brit-2022#4 and associated repo https://github.com/bio-guoda/preston-brit--2022 . If you have any tips or tricks, please do share! |
Does cyverse support anonymous read access via gocmd ? |
You would be able to find the physical path of the image when you click the image in Bisque in the description/attributes. Yes, cyverse allows anonymous read access to most of datasets in |
Yay for anonymous access! Following your tip and https://learning.cyverse.org/ds/gocommands/#anonymous-access-to-the-cyverse-data-store, I tried:
Apologies for asking these silly questions, this is my first time trying to use the gocmd tool. |
The bug was fixed in gocommands v0.4.5. Please check the version. |
So, I have an older version. I'll upgrade in a minute. By the way - I was also able to connect via sftp -
Thanks again for taking the time to reply. |
Yes, SFTP is another convenient way to access your data and it also supports anonymous access. |
Huge thanks to the Cyverse folks at UofA for helping with this issue! @jhpoelen Hopefully we answered your iRODS latency issue questions. For now, I will close this issue, but feel free to reopen or reach out if you have further questions. |
@amilworks thanks for the follow-up on this issue. Unfortunately, I was unable to understand how to retrieve the bisque cyverse images via sftp or irods. I am sure it is possible, but I after spending some time on it, I gave up. However, I was able to use some URL endpoint to retrieve the (original) images via the BisQue web stack. See e.g., bio-guoda/preston-brit-2022#3 (comment) . Again, I appreciate your replies and hope that some day I'll have more time / reason to better understand your sophisticated BisQue tool to manage and analyze images and image volumes. |
Hi @amilworks -
Thanks for your work in developing BisQue .
I am working with an instance of BisQue hosted by Cyverse, and I'd like to understand more efficient ways in accessing large sets of images through Bisque.
Currently, I am using URLs like:
https://bisque.cyverse.org/image_service/image/00-D9CTqQSfzuC553PxBwbt4d/resize:4000/format:jpeg
and
https://bisque.cyverse.org/image_service/image/00-D9CTqQSfzuC553PxBwbt4d
and, it appears that, regardless of the size of the image, the transfer rate is limited to 1 image per 5 seconds.
Is this expected?
Do you have suggestions on how to make accessing hundreds of thousands of images through your service a little faster?
Thanks for your time in reviewing my questions.
For context, see bio-guoda/preston-brit-2022#3 .
fyi @jbest
The text was updated successfully, but these errors were encountered: