Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gdalwarp with multithreading return corrupted data (netcdf + geos projection) #1989

Closed
vincentsarago opened this issue Nov 6, 2019 · 3 comments · Fixed by OSGeo/PROJ#1726
Closed

gdalwarp with multithreading return corrupted data (netcdf + geos projection) #1989

vincentsarago opened this issue Nov 6, 2019 · 3 comments · Fixed by OSGeo/PROJ#1726
Assignees
@rouault
Milestone
2.4.4

Comments

@vincentsarago
Copy link
Contributor

Hi, following some experiment mentioned in #1985, I encountered another strange behaviour when doing warping with multithreading enabled.

Steps to reproduce the problem.

$ aws s3 cp s3://noaa-goes16/ABI-L1b-RadF/2019/295/14/OR_ABI-L1b-RadF-M6C01_G16_s20192951400357_e20192951410065_c20192951410137.nc .

$ gdalwarp netcdf:OR_ABI-L1b-RadF-M6C01_G16_s20192951400357_e20192951410065_c20192951410137.nc:Rad rad_multi.tif -wo NUM_THREADS=8 -t_srs EPSG:4326

See the image above 👇 . Using multithreading creates lines with nodata values where it should be valid values.

result

When reducing the size of the chunk (reducing memory) fixes the lines (but print some error message, which from my understanding, are expected because some points are in space.)

$ gdalwarp netcdf:OR_ABI-L1b-RadF-M6C01_G16_s20192951400357_e20192951410065_c20192951410137.nc:Rad rad_multi_oneMb.tif -wm 1 -wo NUM_THREADS=8 -t_srs EPSG:4326
....
ERROR 1: Too many points (529 out of 529) failed to transform, unable to compute output bounds.
Warning 1: Unable to compute source region for output window 10474,9855,699,658, skipping.
....

Note: Not sure if this a bug report or more an issue to serve as documentation for people doing such work.

GDAL version and provenance

gdal 3.0.1 (source)
rouault added a commit to rouault/PROJ that referenced this issue Nov 12, 2019
@rouault
Fix proj_assign_context()/pj_set_ctx() with pipelines and alternative…
9a23a23 
… coord operations

Fixes OSGeo/gdal#1989

pj_set_ctx() only changes the context to the main object. It should also
recurse down to the steps of the pipeline and the alternative coordinate
operations hold in alternativeCoordinateOperations

In the GDAL use case with multithreaded reprojection, and objects being transferred
between thread, this would cause a failed coordinate transformation to affect
an unrelated transformation of another thread...
@rouault rouault mentioned this issue Nov 12, 2019
Merged
@rouault rouault self-assigned this Nov 12, 2019
backporting bot pushed a commit to OSGeo/PROJ that referenced this issue Nov 12, 2019
@rouault
Fix proj_assign_context()/pj_set_ctx() with pipelines and alternative…
bf8dc3b 
… coord operations

Fixes OSGeo/gdal#1989

pj_set_ctx() only changes the context to the main object. It should also
recurse down to the steps of the pipeline and the alternative coordinate
operations hold in alternativeCoordinateOperations

In the GDAL use case with multithreaded reprojection, and objects being transferred
between thread, this would cause a failed coordinate transformation to affect
an unrelated transformation of another thread...
@vincentsarago
Copy link
Contributor Author

@rouault thanks for looking at this ❤️
Does this mean that if I pull Proj6.2 now I'll get the fix or should I wait for next PROJ release ?

@rouault
Copy link
Member

rouault commented Nov 12, 2019

Several options:

rouault added a commit to OSGeo/PROJ that referenced this issue Nov 12, 2019
@rouault
Fix proj_assign_context()/pj_set_ctx() with pipelines and alternative…
f970636 
… coord operations

Fixes OSGeo/gdal#1989

pj_set_ctx() only changes the context to the main object. It should also
recurse down to the steps of the pipeline and the alternative coordinate
operations hold in alternativeCoordinateOperations

In the GDAL use case with multithreaded reprojection, and objects being transferred
between thread, this would cause a failed coordinate transformation to affect
an unrelated transformation of another thread...
rouault added a commit to rouault/gdal that referenced this issue Nov 12, 2019
@rouault
Multithreaded warper: make sure a transformer object is used by the t…
83c2506 
…hread which created it (fixes OSGeo#1989). This workarounds a PROJ bug also fixed per OSGeo/PROJ#1726
rouault added a commit that referenced this issue Nov 12, 2019
@rouault
Merge pull request #2009 from rouault/fix_1989
aeca89a 
Multithreaded warper: make sure a transformer object is used by the the thread which created it (fixes #1989)
rouault added a commit that referenced this issue Nov 12, 2019
@rouault
Multithreaded warper: make sure a transformer object is used by the t…
08482d9 
…hread which created it (fixes #1989). This workarounds a PROJ bug also fixed per OSGeo/PROJ#1726
rouault added a commit that referenced this issue Nov 12, 2019
@rouault
Multithreaded warper: make sure a transformer object is used by the t…
4ee512d 
…hread which created it (fixes #1989). This workarounds a PROJ bug also fixed per OSGeo/PROJ#1726
@rouault
Copy link
Member

rouault commented Nov 12, 2019

Backported to 3.0 and 2.4 branches

@rouault rouault added this to the 2.4.4 milestone Nov 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rouault rouault

Labels
None yet
Projects
None yet
Milestone
2.4.4
Development

Successfully merging a pull request may close this issue.

Fix proj_assign_context()/pj_set_ctx() with pipelines and alternative coord operations rouault/PROJ
2 participants
@rouault @vincentsarago