Performance regression of GDALWarp due to PROJ>6 #3470
Milestone
Comments
|
It seems that you have noticed that GDAL project prefers to discuss on the mailing list but you still decided to try to discuss in the bug tracker. But the GDAL community is really there, not here. Can you find anything usable from other Proj 6 migrations, for example https://mapserver.gis.umn.edu/id/development/rfc/ms-rfc-126.html |
|
Hi @jratike80 as mentioned in the issue template, we can still use this place to propose feature or report bug! To me this issue feels like a real issue (coordinates transformation pipeline being recalculated over and over... if I'm right 🤷♂️). I'll have a look at your links 🙏. Feel free to closes if you do feel that this is not the the right place to discuss ☝️ |
|
Let's keep this open but also ask on gdal-dev too. I think there are enough PROJ developers there to help. |
|
Have you already tried to use the pipeline instead of -t_srs?
|
|
Timestamps can be got with CPL_TIMESTAMP=ON:
There are likely a few savings that could be potentially done by caching / re-using CRS A to CRS B transformation path computations. Should probably be done on the GDAL side (the philosophy of use of PROJ should be to minimize the number of calls to such costly operations) |
rouault
added a commit
to rouault/gdal
that referenced
this issue
Mar 16, 2021
Fixes a GDAL 3.2 performance regression. Since GDAL 3.2, we expose the JPEG quality in metadata and read it at dataset opening, and for each overview. Previously we only read it when updating an existing dataset. For a quality of 85, guessing it for each overview level takes approximatively 14 ms on a Core i7-10750H @2.6 GHz, which sums up to ~ 100 ms when opening a dataset with 6 overview levels, such as in the test case of OSGeo#3470 We fix that by precomputing the MD5Sum of the quantization tables. Before: ``` $ time gdalwarp -q wrk/m_3209542_se_15_060_20181115_20190222.tif wrk/out.tif -t_srs EPSG:3857 -ts 1024 1024 -overwrite real 0m0,282s user 0m0,249s sys 0m0,033s ``` After: ``` $ time gdalwarp -q wrk/m_3209542_se_15_060_20181115_20190222.tif wrk/out.tif -t_srs EPSG:3857 -ts 1024 1024 -overwrite real 0m0,179s user 0m0,143s sys 0m0,037s ```
rouault
added a commit
that referenced
this issue
Mar 16, 2021
Fixes a GDAL 3.2 performance regression. Since GDAL 3.2, we expose the JPEG quality in metadata and read it at dataset opening, and for each overview. Previously we only read it when updating an existing dataset. For a quality of 85, guessing it for each overview level takes approximatively 14 ms on a Core i7-10750H @2.6 GHz, which sums up to ~ 100 ms when opening a dataset with 6 overview levels, such as in the test case of #3470 We fix that by precomputing the MD5Sum of the quantization tables. Before: ``` $ time gdalwarp -q wrk/m_3209542_se_15_060_20181115_20190222.tif wrk/out.tif -t_srs EPSG:3857 -ts 1024 1024 -overwrite real 0m0,282s user 0m0,249s sys 0m0,033s ``` After: ``` $ time gdalwarp -q wrk/m_3209542_se_15_060_20181115_20190222.tif wrk/out.tif -t_srs EPSG:3857 -ts 1024 1024 -overwrite real 0m0,179s user 0m0,143s sys 0m0,037s ```
rouault
added a commit
to rouault/PROJ
that referenced
this issue
Mar 18, 2021
createFromCRSCodesWithIntermediates() runs a rather costly self-join. Only run it if the source and target CRS are the source/target of a coordinate operation. This helps for the performance of proj_create_crs_to_crs() when run on projected CRS for example that are extremely unlikely to be the source/target of an operation (except currently the Finish ones). For the EPSG:26915 to EPSG:3857 case of OSGeo/gdal#3470, this helps decreasing the time of proj_create_crs_to_crs() from 18 ms to 10 ms.
rouault
added a commit
to rouault/gdal
that referenced
this issue
Mar 18, 2021
use it in GDALCreateReprojectionTransformerEx() Together with the improvement in PROJ >= 8.0.1 of PROJ OSGeo#2582, this helps reducing the time of the gdalwarp invokation of OSGeo#3470 from 180 ms to 135 ms (compared to 80 ms with GDAL 2.4 + PROJ 5.2)
rouault
added a commit
to rouault/gdal
that referenced
this issue
Mar 18, 2021
Helps decreasing the gdalwarp time of OSGeo#3470 from 135 ms to 123 ms.
rouault
added a commit
to rouault/gdal
that referenced
this issue
Mar 18, 2021
Helps decreasing the gdalwarp time of OSGeo#3470 from 123 ms to 113 ms. And with OSGeo/PROJ#2583, we go down to 105 ms. Compared to the 80 ms of GDAL 2.4/PROJ 5.2, and the ~ 300ms of GDAL 3.2/PROJ 8
rouault
added a commit
to OSGeo/PROJ
that referenced
this issue
Mar 18, 2021
createFromCRSCodesWithIntermediates() runs a rather costly self-join. Only run it if the source and target CRS are the source/target of a coordinate operation. This helps for the performance of proj_create_crs_to_crs() when run on projected CRS for example that are extremely unlikely to be the source/target of an operation (except currently the Finish ones). For the EPSG:26915 to EPSG:3857 case of OSGeo/gdal#3470, this helps decreasing the time of proj_create_crs_to_crs() from 18 ms to 10 ms.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected behavior and actual behavior.
👋 Dear GDAL contributors and community. First feel free to close this if you feel this isn't the place to report/discuss this. I'll happily move this to the mailing list or maybe to the PROJ project 🤷♂️ .
First reported in rasterio/rasterio#2121 and linked to #1662 . Basically we are experiencing a
hugeperformance downgrade when using GDAL>=3 with PROJ >6. We understand the changes needed for PROJ >6 and are really happy to have more precise coordinate transform 😄 . I think the main performance issue with Gdal Warp or warpedVRT is that it doesn'tcachethe coordinate transformation. I'm not sure such a thing is possible 🤷♂️ but looking at the log it seems that GDAL is calculating the same pipeline multiple times. I'm not even sure why it does it multiple time...Use Case
I'm mostly using GDAL for project involving Dynamic Tiling from COG (non aligned with Grid) and thus we are looking for high (or relatively) performance for small parallel warping tasks that may or may not involve data caching.
Steps to reproduce the problem.
GDAL 3.2 PROJ 7.1 (centos)
File:
https://rio-tiler-dev.s3.amazonaws.com/data/m_3209542_se_15_060_20181115_20190222.tifGDAL 2.4 PROJ 5
Operating system
centos, alpine, ubuntu, MacOS
GDAL version and provenance
Rasterio wheel and/or sources
Thanks for your time and feedback 🙏
The text was updated successfully, but these errors were encountered: