Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error using GDAL Virtual File Systems #2227

Closed
wenjieji86 opened this issue Feb 8, 2020 · 11 comments
Closed

Error using GDAL Virtual File Systems #2227

wenjieji86 opened this issue Feb 8, 2020 · 11 comments
Labels
awaiting_feedback Awaiting feedback from reporter

Comments

@wenjieji86
Copy link

Hi,

I am trying to open and read tif images I stored on Google Cloud storage using the virtual file systems. My GDAL version is 2.2.3 (somehow failed to upgrade on Google Colab).
Here's what I have tried:
!gdalinfo --config GS_ACCESS_KEY_ID 'myID' --config GS_SECRET_ACCESS_KEY 'myKEY' /vsigs/bucket_name/subfolder/image.tif
and I would get:
ERROR 11: HTTP response code: 403 - Failed writing header gdalinfo failed - unable to open '/vsigs/bucket_name/subfolder/image.tif'.
Then I tried:
!gdalinfo /vsicurl/https://storage.cloud.google.com/bucket_name/subfolder/image.tif
which I would get 1 random error out of 2:
ERROR 1: Line 1255: </head> doesn't have matching <head>. gdalinfo failed - unable to open '/vsicurl/https://storage.cloud.google.com/bucket_name/subfolder/image.tif'
or:
ERROR 3: Cannot read 74247 bytes gdalinfo failed - unable to open '/vsicurl/https://storage.cloud.google.com/bucket_name/subfolder/image.tif'

I have also tried:
!gdalinfo --config CPL_VSIL_CURL_USE_HEAD FALSE /vsicurl/https://storage.cloud.google.com/bucket_name/subfolder/image.tif
and I would get:
ERROR 4: '/vsicurl/https://storage.cloud.google.com/bucket_name/subfolder/image.tif' not recognized as a supported file format. gdalinfo failed - unable to open '/vsicurl/https://storage.cloud.google.com/bucket_name/subfolder/image.tif'.

I am new to gdal and python, I need help. Thanks.
@rouault
Copy link
Member

rouault commented Feb 8, 2020

@rouault rouault added the awaiting_feedback Awaiting feedback from reporter label Feb 9, 2020
@wenjieji86
Copy link
Author

@rouault no, just letters and '_' in folder and file names.
storage.googleapis.com did not work either. This is interesting because the public url generated by google cloud storage api starts with 'storage.googleapis.com', but that doensn't work for google tensorflow keras api (tf.keras.utils.get_file) it only works if I use 'storage.cloud.google.com'

'CPL_GS_ENDPOINT' is not mentioned in gdal documentation. What is it for? I tried set the variable to both 'storage.googleapis.com' and 'storage.cloud.google.com' and did not work...

@rouault
Copy link
Member

rouault commented Feb 10, 2020

Do your settings work with google cloud client "gsutil ls gs://your_bucket" ?

@wenjieji86
Copy link
Author

@rouault
Yes, it works and I was able to download all files using
google.cloud.storage.Blob(file.name,gcs_bucket).download_to_file(file_obj)

@rouault
Copy link
Member

rouault commented Feb 10, 2020

Define CPL_CURL_VERBOSE=YES as environment variable, re-run and post output

@rouault
Copy link
Member

rouault commented Feb 10, 2020

seems like authentication problem...?

yes. Check your GS_ACCESS_KEY_ID and GS_SECRET_ACCESS_KEY values. Maybe you've switched them ?

@rouault
Copy link
Member

rouault commented Feb 10, 2020

Ah so your bucket name is "test_bl,_data" with a comma ?

@wenjieji86
Copy link
Author

@rouault
My bad. I tried again with correct bucket name and still got the 403 error...
I believe the GS_ACCESS_KEY_ID and GS_SECRET_ACCESS_KEY values are correct - I got them from Google Cloud console credentials page.

Does it matter than I am running my code in colab and I have authenticated using the following code (to use the gcs client apis):
from google.colab import auth auth.authenticate_user()

@rouault
Copy link
Member

rouault commented Feb 10, 2020

I've no knowledge of Colab, but I fail to see how it would interact with GDAL. GDAL doesn't its own thing. I'm a bit at loss if it is just a user issue on your side or some real bug. You're not running the latest GDAL version either, so maybe there would be some later fixed bug, but digging a bit in history doesn't show anything particularly relevant.
Perhaps try outside of Colab in a normal Linux or Windows environment to see if you can reproduce

@wenjieji86
Copy link
Author

@rouault
Now I am pretty sure it's an authentication issue.
I tired on my local PC (python 3.6 + GDAL 3.0) and ran into the same issue...
Looking into GCS authentical...
Thanks!

@wenjieji86
Copy link
Author

@rouault
OK, I figured it out...
I should have gotten the id and the secret for a service account of my project rather than creating an OAuth Client ID and secret...

Thank you for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting_feedback Awaiting feedback from reporter
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rouault @wenjieji86