Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Created VSIError to report VSI errors. #98

Closed
wants to merge 17 commits into from
Closed

Created VSIError to report VSI errors. #98

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
9164eb1
Created VSIError to report VSI errors.
lossyrob Feb 26, 2016
213ec02
Added autotest to vagrant sync folders
lossyrob Mar 20, 2016
73d14b3
Changed vsil_crypt Open to to take additional bool parameter.
lossyrob Mar 20, 2016
9357020
Use int instead of bool for C API
lossyrob Mar 20, 2016
bb64a68
Merged in trunk.
lossyrob Mar 20, 2016
a2c914f
Account for a NULL XML message in AWS error handling.
lossyrob Mar 20, 2016
6650156
Changed VSIErrorV to private static; don't return bool from C API fun…
lossyrob Mar 20, 2016
7981479
Include cpl_vsi_error.obj in makefile.vc
lossyrob Mar 20, 2016
f5bb1ed
Fixed comment for VSItoCPLError
lossyrob Mar 20, 2016
16f6230
Added VSIError calls to swig bindings.
lossyrob Mar 20, 2016
8e17293
Fixed test that should now be expected VSIError message.
lossyrob Mar 20, 2016
6ee58bd
Set VSI errors in S3 streaming during read call.
lossyrob Mar 20, 2016
38a8687
Fixes tests to work with VSI errors.
lossyrob Mar 20, 2016
208a530
Generate python bindings during Travis CI
lossyrob Mar 20, 2016
7a1887e
Don't use CPL_DLL on static private functions; use TRUE and FALSE ins…
lossyrob Mar 20, 2016
88d28b6
Merge remote-tracking branch 'upstream/trunk' into vsierror
lossyrob Mar 21, 2016
2d8556a
Modified basic_test.py to work with both windows and mac/linux.
lossyrob Mar 21, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions autotest/gcore/basic_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,13 @@
# Nothing exciting here. Just trying to open non existing files,
# or empty names, or files that are not valid datasets...

non_existing_error_msg = 'No such file or directory'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid this will not pass on Windows

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. I got the impression that the autotest was only running on Linux (it still fails my mac in some ways, even trunk, whereas Travis passes). Are the autotests targeted to work across all platforms? If so, do you have suggestions on how to test on Windows?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The autotest also run on Windows and Mac.
For Windows, there's an AppVeyor config in this branch : https://github.com/rouault/gdal_coverage/blob/trunk_vc12_full/appveyor.yml . It could/should probably be merged into main repo. For now, master is merged regularly in this branch.
For Mac, this is https://github.com/rouault/gdal_coverage/blob/trunk_travis_macosx/.travis.yml . Could probably be unified into the main travis.yml with some matrix logic.
With the current setup, I guess you could probably fork that repo and merge your branch into the 2 above branches, and create PR to trigger the builds ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'll give that a shot once I get the travis build to pass in this repo.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, the way you have the branches set up, I don't believe that will work. I'll try to find another way to test on Windows.


def basic_test_1():
gdal.PushErrorHandler( 'CPLQuietErrorHandler' )
ds = gdal.Open('non_existing_ds', gdal.GA_ReadOnly)
gdal.PopErrorHandler()
if ds is None and gdal.GetLastErrorMsg() == '`non_existing_ds\' does not exist in the file system,\nand is not recognized as a supported dataset name.\n':
if ds is None and gdal.GetLastErrorMsg() == non_existing_error_msg:
return 'success'
else:
return 'fail'
Expand All @@ -53,7 +55,7 @@ def basic_test_2():
gdal.PushErrorHandler( 'CPLQuietErrorHandler' )
ds = gdal.Open('non_existing_ds', gdal.GA_Update)
gdal.PopErrorHandler()
if ds is None and gdal.GetLastErrorMsg() == '`non_existing_ds\' does not exist in the file system,\nand is not recognized as a supported dataset name.\n':
if ds is None and gdal.GetLastErrorMsg() == non_existing_error_msg:
return 'success'
else:
return 'fail'
Expand All @@ -62,7 +64,7 @@ def basic_test_3():
gdal.PushErrorHandler( 'CPLQuietErrorHandler' )
ds = gdal.Open('', gdal.GA_ReadOnly)
gdal.PopErrorHandler()
if ds is None and gdal.GetLastErrorMsg() == '`\' does not exist in the file system,\nand is not recognized as a supported dataset name.\n':
if ds is None and gdal.GetLastErrorMsg() == non_existing_error_msg:
return 'success'
else:
return 'fail'
Expand All @@ -71,7 +73,7 @@ def basic_test_4():
gdal.PushErrorHandler( 'CPLQuietErrorHandler' )
ds = gdal.Open('', gdal.GA_Update)
gdal.PopErrorHandler()
if ds is None and gdal.GetLastErrorMsg() == '`\' does not exist in the file system,\nand is not recognized as a supported dataset name.\n':
if ds is None and gdal.GetLastErrorMsg() == non_existing_error_msg:
return 'success'
else:
return 'fail'
Expand Down Expand Up @@ -106,7 +108,7 @@ def basic_test_7_internal():
except:
# Special case: we should still be able to get the error message
# until we call a new GDAL function
if gdal.GetLastErrorMsg() != '`non_existing_ds\' does not exist in the file system,\nand is not recognized as a supported dataset name.\n':
if gdal.GetLastErrorMsg() != non_existing_error_msg:
gdaltest.post_reason('did not get expected error message')
return 'fail'

Expand Down
94 changes: 51 additions & 43 deletions autotest/gcore/vsis3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,12 @@
import gdaltest
import webserver

def open_for_read(uri):
"""
Opens a test file for reading.
"""
return gdal.VSIFOpenExL(uri, 'rb', 1)

###############################################################################
def vsis3_init():

Expand All @@ -59,55 +65,57 @@ def vsis3_1():
if drv is None:
return 'skip'

# RETODO: Bind to swig, change test

# Missing AWS_SECRET_ACCESS_KEY
gdal.ErrorReset()
with gdaltest.error_handler():
f = gdal.VSIFOpenL('/vsis3/foo/bar', 'rb')
if f is not None or gdal.GetLastErrorMsg().find('AWS_SECRET_ACCESS_KEY') < 0:
f = open_for_read('/vsis3/foo/bar')
if f is not None or gdal.VSIGetLastErrorMsg().find('AWS_SECRET_ACCESS_KEY') < 0:
gdaltest.post_reason('fail')
print(gdal.GetLastErrorMsg())
print(gdal.VSIGetLastErrorMsg())
return 'fail'

gdal.ErrorReset()
with gdaltest.error_handler():
f = gdal.VSIFOpenL('/vsis3_streaming/foo/bar', 'rb')
if f is not None or gdal.GetLastErrorMsg().find('AWS_SECRET_ACCESS_KEY') < 0:
f = open_for_read('/vsis3_streaming/foo/bar')
if f is not None or gdal.VSIGetLastErrorMsg().find('AWS_SECRET_ACCESS_KEY') < 0:
gdaltest.post_reason('fail')
print(gdal.GetLastErrorMsg())
print(gdal.VSIGetLastErrorMsg())
return 'fail'

gdal.SetConfigOption('AWS_SECRET_ACCESS_KEY', 'AWS_SECRET_ACCESS_KEY')

# Missing AWS_ACCESS_KEY_ID
gdal.ErrorReset()
with gdaltest.error_handler():
f = gdal.VSIFOpenL('/vsis3/foo/bar', 'rb')
if f is not None or gdal.GetLastErrorMsg().find('AWS_ACCESS_KEY_ID') < 0:
f = open_for_read('/vsis3/foo/bar')
if f is not None or gdal.VSIGetLastErrorMsg().find('AWS_ACCESS_KEY_ID') < 0:
gdaltest.post_reason('fail')
print(gdal.GetLastErrorMsg())
print(gdal.VSIGetLastErrorMsg())
return 'fail'

gdal.SetConfigOption('AWS_ACCESS_KEY_ID', 'AWS_ACCESS_KEY_ID')

# ERROR 1: The AWS Access Key Id you provided does not exist in our records.
gdal.ErrorReset()
with gdaltest.error_handler():
f = gdal.VSIFOpenL('/vsis3/foo/bar.baz', 'rb')
if f is not None or gdal.GetLastErrorMsg() == '':
f = open_for_read('/vsis3/foo/bar.baz')
if f is not None or gdal.VSIGetLastErrorMsg() == '':
if f is not None:
gdal.VSIFCloseL(f)
if gdal.GetConfigOption('APPVEYOR') is not None:
return 'success'
gdaltest.post_reason('fail')
print(gdal.GetLastErrorMsg())
print(gdal.VSIGetLastErrorMsg())
return 'fail'

gdal.ErrorReset()
with gdaltest.error_handler():
f = gdal.VSIFOpenL('/vsis3_streaming/foo/bar.baz', 'rb')
if f is not None or gdal.GetLastErrorMsg() == '':
f = open_for_read('/vsis3_streaming/foo/bar.baz')
if f is not None or gdal.VSIGetLastErrorMsg() == '':
gdaltest.post_reason('fail')
print(gdal.GetLastErrorMsg())
print(gdal.VSIGetLastErrorMsg())
return 'fail'

return 'success'
Expand Down Expand Up @@ -147,7 +155,7 @@ def vsis3_2():
gdal.SetConfigOption('AWS_VIRTUAL_HOSTING', 'NO')
gdal.SetConfigOption('AWS_S3_ENDPOINT', '127.0.0.1:%d' % gdaltest.webserver_port)

f = gdal.VSIFOpenL('/vsis3/s3_fake_bucket/resource', 'rb')
f = open_for_read('/vsis3/s3_fake_bucket/resource')
if f is None:
gdaltest.post_reason('fail')
return 'fail'
Expand All @@ -159,7 +167,7 @@ def vsis3_2():
print(data)
return 'fail'

f = gdal.VSIFOpenL('/vsis3_streaming/s3_fake_bucket/resource', 'rb')
f = open_for_read('/vsis3_streaming/s3_fake_bucket/resource')
if f is None:
gdaltest.post_reason('fail')
return 'fail'
Expand All @@ -173,7 +181,7 @@ def vsis3_2():

# Test with temporary credentials
gdal.SetConfigOption('AWS_SESSION_TOKEN', 'AWS_SESSION_TOKEN')
f = gdal.VSIFOpenL('/vsis3/s3_fake_bucket_with_session_token/resource', 'rb')
f = open_for_read('/vsis3/s3_fake_bucket_with_session_token/resource')
if f is None:
gdaltest.post_reason('fail')
return 'fail'
Expand Down Expand Up @@ -203,7 +211,7 @@ def vsis3_2():
return 'fail'

# Test region and endpoint 'redirects'
f = gdal.VSIFOpenL('/vsis3/s3_fake_bucket/redirect', 'rb')
f = open_for_read('/vsis3/s3_fake_bucket/redirect')
if f is None:
gdaltest.post_reason('fail')
return 'fail'
Expand All @@ -216,7 +224,7 @@ def vsis3_2():
return 'fail'

# Test region and endpoint 'redirects'
f = gdal.VSIFOpenL('/vsis3_streaming/s3_fake_bucket/redirect', 'rb')
f = open_for_read('/vsis3_streaming/s3_fake_bucket/redirect')
if f is None:
gdaltest.post_reason('fail')
return 'fail'
Expand All @@ -230,50 +238,50 @@ def vsis3_2():

gdal.ErrorReset()
with gdaltest.error_handler():
f = gdal.VSIFOpenL('/vsis3_streaming/s3_fake_bucket/non_xml_error', 'rb')
if f is not None or gdal.GetLastErrorMsg().find('bla') < 0:
f = open_for_read('/vsis3_streaming/s3_fake_bucket/non_xml_error')
if f is not None or gdal.VSIGetLastErrorMsg().find('bla') < 0:
gdaltest.post_reason('fail')
print(gdal.GetLastErrorMsg())
print(gdal.VSIGetLastErrorMsg())
return 'fail'

gdal.ErrorReset()
with gdaltest.error_handler():
f = gdal.VSIFOpenL('/vsis3_streaming/s3_fake_bucket/invalid_xml_error', 'rb')
if f is not None or gdal.GetLastErrorMsg().find('<oops>') < 0:
f = open_for_read('/vsis3_streaming/s3_fake_bucket/invalid_xml_error')
if f is not None or gdal.VSIGetLastErrorMsg().find('<oops>') < 0:
gdaltest.post_reason('fail')
print(gdal.GetLastErrorMsg())
print(gdal.VSIGetLastErrorMsg())
return 'fail'

gdal.ErrorReset()
with gdaltest.error_handler():
f = gdal.VSIFOpenL('/vsis3_streaming/s3_fake_bucket/no_code_in_error', 'rb')
if f is not None or gdal.GetLastErrorMsg().find('<Error/>') < 0:
f = open_for_read('/vsis3_streaming/s3_fake_bucket/no_code_in_error')
if f is not None or gdal.VSIGetLastErrorMsg().find('<Error/>') < 0:
gdaltest.post_reason('fail')
print(gdal.GetLastErrorMsg())
print(gdal.VSIGetLastErrorMsg())
return 'fail'

gdal.ErrorReset()
with gdaltest.error_handler():
f = gdal.VSIFOpenL('/vsis3_streaming/s3_fake_bucket/no_region_in_AuthorizationHeaderMalformed_error', 'rb')
if f is not None or gdal.GetLastErrorMsg().find('<Error>') < 0:
f = open_for_read('/vsis3_streaming/s3_fake_bucket/no_region_in_AuthorizationHeaderMalformed_error')
if f is not None or gdal.VSIGetLastErrorMsg().find('<Error>') < 0:
gdaltest.post_reason('fail')
print(gdal.GetLastErrorMsg())
print(gdal.VSIGetLastErrorMsg())
return 'fail'

gdal.ErrorReset()
with gdaltest.error_handler():
f = gdal.VSIFOpenL('/vsis3_streaming/s3_fake_bucket/no_endpoint_in_PermanentRedirect_error', 'rb')
if f is not None or gdal.GetLastErrorMsg().find('<Error>') < 0:
f = open_for_read('/vsis3_streaming/s3_fake_bucket/no_endpoint_in_PermanentRedirect_error')
if f is not None or gdal.VSIGetLastErrorMsg().find('<Error>') < 0:
gdaltest.post_reason('fail')
print(gdal.GetLastErrorMsg())
print(gdal.VSIGetLastErrorMsg())
return 'fail'

gdal.ErrorReset()
with gdaltest.error_handler():
f = gdal.VSIFOpenL('/vsis3_streaming/s3_fake_bucket/no_message_in_error', 'rb')
if f is not None or gdal.GetLastErrorMsg().find('<Error>') < 0:
f = open_for_read('/vsis3_streaming/s3_fake_bucket/no_message_in_error')
if f is not None or gdal.VSIGetLastErrorMsg().find('<Error>') < 0:
gdaltest.post_reason('fail')
print(gdal.GetLastErrorMsg())
print(gdal.VSIGetLastErrorMsg())
return 'fail'

return 'success'
Expand All @@ -285,7 +293,7 @@ def vsis3_3():

if gdaltest.webserver_port == 0:
return 'skip'
f = gdal.VSIFOpenL('/vsis3/s3_fake_bucket2/a_dir/resource3.bin', 'rb')
f = open_for_read('/vsis3/s3_fake_bucket2/a_dir/resource3.bin')
if f is None:
gdaltest.post_reason('fail')
return 'fail'
Expand Down Expand Up @@ -562,7 +570,7 @@ def vsis3_extra_1():
print('Missing S3_RESOURCE for running gdaltest_list_extra')
return 'skip'

f = gdal.VSIFOpenL('/vsis3/' + gdal.GetConfigOption('S3_RESOURCE'), 'rb')
f = open_for_read('/vsis3/' + gdal.GetConfigOption('S3_RESOURCE'))
if f is None:
gdaltest.post_reason('fail')
return 'fail'
Expand All @@ -575,7 +583,7 @@ def vsis3_extra_1():
return 'fail'

# Same with /vsis3_streaming/
f = gdal.VSIFOpenL('/vsis3_streaming/' + gdal.GetConfigOption('S3_RESOURCE'), 'rb')
f = open_for_read('/vsis3_streaming/' + gdal.GetConfigOption('S3_RESOURCE'))
if f is None:
gdaltest.post_reason('fail')
return 'fail'
Expand All @@ -589,7 +597,7 @@ def vsis3_extra_1():

# Invalid bucket : "The specified bucket does not exist"
gdal.ErrorReset()
f = gdal.VSIFOpenL('/vsis3/not_existing_bucket/foo', 'rb')
f = open_for_read('/vsis3/not_existing_bucket/foo')
with gdaltest.error_handler():
gdal.VSIFReadL(1, 1, f)
gdal.VSIFCloseL(f)
Expand All @@ -600,7 +608,7 @@ def vsis3_extra_1():

# Invalid resource
gdal.ErrorReset()
f = gdal.VSIFOpenL('/vsis3_streaming/' + gdal.GetConfigOption('S3_RESOURCE') + '/invalid_resource.baz', 'rb')
f = open_for_read('/vsis3_streaming/' + gdal.GetConfigOption('S3_RESOURCE') + '/invalid_resource.baz')
if f is not None:
gdaltest.post_reason('fail')
print(gdal.GetLastErrorMsg())
Expand Down