Pixelfunctions integration #141

jmichel-otb · 2016-07-06T14:03:25Z

This PR contains integration of Antonio pixel functions within frmts/vrt. I did my best to preserve git history.

Though I am not familiar with the autotest suite, I merged the test from Antonio as well. This probably needs an in-depth review.

…g bands

…ctions

rouault · 2016-07-06T14:20:26Z

autotest/gdrivers/pixfun.py

+
+# @TODO: remove in non-plugin configuration
+#gdal.SetConfigOption('CPL_DEBUG', 'PIXFUN')
+gdal.SetConfigOption('GDAL_DRIVER_PATH', '../..')


Must be removed :
+gdal.SetConfigOption('GDAL_DRIVER_PATH', '../..')
+gdal.AllRegister()

rouault · 2016-07-06T14:28:12Z

Some documentation of the pixel functions in frmts/vrt/vrt_tutorial.dox would be needed as well.

…eview)

…from PR review)

rouault · 2016-07-06T15:46:11Z

From the Travis log:

clang -I/home/travis/build/OSGeo/gdal/gdal/port -I/home/travis/build/OSGeo/gdal/gdal/gcore -I/home/travis/build/OSGeo/gdal/gdal/alg -I/home/travis/build/OSGeo/gdal/gdal/ogr -I/home/travis/build/OSGeo/gdal/gdal/ogr/ogrsf_frmts -I/home/travis/build/OSGeo/gdal/gdal/gnm -I/home/travis/build/OSGeo/gdal/gdal/apps -fPIC  -g -DDEBUG -ftrapv  -Wall -Wdeclaration-after-statement -Wextra -Winit-self -Wunused-parameter -Wmissing-prototypes -Wmissing-declarations -Wformat -Werror=format-security -Wno-format-nonliteral -Wshorten-64-to-32 -Wshadow -Werror=vla -Wdeclaration-after-statement -Wnull-dereference -Wextra -Werror -I../raw -DOGR_ENABLED -D_REENTRANT  -I/home/travis/build/OSGeo/gdal/gdal/port -I/usr/local/include -I/usr/include  -DGDAL_COMPILATION -c -o ../o/pixelfunctions.o pixelfunctions.c

pixelfunctions.c:919:20: error: no previous prototype for function

      'GDALRegisterDefaultPixelFunc' [-Werror,-Wmissing-prototypes]

CPLErr CPL_STDCALL GDALRegisterDefaultPixelFunc()

                   ^

1 error generated.

jmichel-otb · 2016-07-07T11:28:03Z

travis-ci build is ok, and I think I adressed all the comments.

…nj, etc... for complex data types (initial contribution by Antonio Valentino, integration done by Julien Michel, OSGeo/gdal#141) git-svn-id: https://svn.osgeo.org/gdal/trunk/gdal@34584 f0d54148-0727-0410-94bb-9a71ac55c965

…ypes, that happened to be a real bug (ref OSGeo/gdal#141) git-svn-id: https://svn.osgeo.org/gdal/trunk/gdal@34585 f0d54148-0727-0410-94bb-9a71ac55c965

rouault · 2016-07-07T14:36:01Z

@avalentino @jmichel-otb After integration in GDAL trunk, I get a failure with gcc 5.2 -faddress=sanitize in https://s3.amazonaws.com/archive.travis-ci.org/jobs/143036944/log.txt :

�[1m�[31m==28355==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x615000077ae0 at pc 0x7fcb48db55d5 bp 0x7ffd81337ab0 sp 0x7ffd81337260
�[1m�[0m�[1m�[34mREAD of size 16 at 0x615000077ae0 thread T0�[1m�[0m
    #0 0x7fcb48db55d4 in __asan_memcpy (/home/travis/build/rouault/gdal_coverage/install-gcc-5.2.0/lib64/libasan.so.2.0.0+0x8b5d4)
    #1 0x7fcb34687656 in GDALCopyWords /home/travis/build/rouault/gdal_coverage/gdal/gcore/rasterio.cpp:2447
    #2 0x7fcb33ca1a45 in ImagPixelFunc /home/travis/build/rouault/gdal_coverage/gdal/frmts/vrt/pixelfunctions.c:166
    #3 0x7fcb3421d94f in VRTDerivedRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) /home/travis/build/rouault/gdal_coverage/gdal/frmts/vrt/vrtderivedrasterband.cpp:391
    #4 0x7fcb345cbc40 in GDALRasterBand::RasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) /home/travis/build/rouault/gdal_coverage/gdal/gcore/gdalrasterband.cpp:286
    #5 0x7fcb345cc109 in GDALRasterIOEx /home/travis/build/rouault/gdal_coverage/gdal/gcore/gdalrasterband.cpp:349
    #6 0x7fcb1f499e5e in BandRasterIONumPy(void*, int, double, double, double, double, PyArrayObject*, int, GDALRIOResampleAlg, int (*)(double, char const*, void*), void*) extensions/gdal_array_wrap.cpp:4028
    #7 0x7fcb1f49ae0e in _wrap_BandRasterIONumPy extensions/gdal_array_wrap.cpp:4853
    #8 0x4f95a4 in PyEval_EvalFrameEx (/usr/bin/python2.7+0x4f95a4)
    #9 0x5008c1 in PyEval_EvalCodeEx (/usr/bin/python2.7+0x5008c1)
    #10 0x4f9ab7 in PyEval_EvalFrameEx (/usr/bin/python2.7+0x4f9ab7)
    #11 0x5008c1 in PyEval_EvalCodeEx (/usr/bin/python2.7+0x5008c1)
    #12 0x4f9ab7 in PyEval_EvalFrameEx (/usr/bin/python2.7+0x4f9ab7)
    #13 0x4f9d01 in PyEval_EvalFrameEx (/usr/bin/python2.7+0x4f9d01)
    #14 0x4f9d01 in PyEval_EvalFrameEx (/usr/bin/python2.7+0x4f9d01)
    #15 0x4f9d01 in PyEval_EvalFrameEx (/usr/bin/python2.7+0x4f9d01)
    #16 0x5008c1 in PyEval_EvalCodeEx (/usr/bin/python2.7+0x5008c1)
    #17 0x4f9255 in PyRun_StringFlags (/usr/bin/python2.7+0x4f9255)
    #18 0x501558 in PyRun_SimpleStringFlags (/usr/bin/python2.7+0x501558)
    #19 0x49a4c9 in Py_Main (/usr/bin/python2.7+0x49a4c9)
    #20 0x7fcb46b0e7ec in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x217ec)
    #21 0x41bb30  (/usr/bin/python2.7+0x41bb30)

�[1m�[32m0x615000077ae0 is located 0 bytes to the right of 480-byte region [0x615000077900,0x615000077ae0)
�[1m�[0m�[1m�[35mallocated by thread T0 here:�[1m�[0m
    #0 0x7fcb48dc102a in malloc (/home/travis/build/rouault/gdal_coverage/install-gcc-5.2.0/lib64/libasan.so.2.0.0+0x9702a)
    #1 0x7fcb348a3d47 in VSIMalloc /home/travis/build/rouault/gdal_coverage/gdal/port/cpl_vsisimple.cpp:524
    #2 0x7fcb348a44e0 in VSIMallocVerbose /home/travis/build/rouault/gdal_coverage/gdal/port/cpl_vsisimple.cpp:932
    #3 0x7fcb3421c9d9 in VRTDerivedRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) /home/travis/build/rouault/gdal_coverage/gdal/frmts/vrt/vrtderivedrasterband.cpp:344
    #4 0x7fcb345cbc40 in GDALRasterBand::RasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) /home/travis/build/rouault/gdal_coverage/gdal/gcore/gdalrasterband.cpp:286
    #5 0x7fcb345cc109 in GDALRasterIOEx /home/travis/build/rouault/gdal_coverage/gdal/gcore/gdalrasterband.cpp:349
    #6 0x7fcb1f499e5e in BandRasterIONumPy(void*, int, double, double, double, double, PyArrayObject*, int, GDALRIOResampleAlg, int (*)(double, char const*, void*), void*) extensions/gdal_array_wrap.cpp:4028

I cannot reproduce with Valgrind though, but there's something fishy in ImagPixelFunc() around lines 164-170 (in the complex case)

        /* ---- Set pixels ---- */
        for( iLine = 0; iLine < nYSize; ++iLine ) {
            GDALCopyWords(((GByte *)pImag) + nLineSpaceSrc * iLine,
                          eSrcType, nPixelSpaceSrc,
                          ((GByte *)pData) + nLineSpace * iLine,
                          eBufType, nPixelSpace, nXSize);
        }

The 2nd argument of GDALCopyWords (eSrcType) is probably wrong, since for example in the case of a GDT_CFloat64 we want to copy only the GDAL_Float64, hence the out-of-bounds read on the last copied pixel. The issue might also be found in other pixel functions that work similarly. We would probably a new function in gcore/gdal_misc.cpp GDALDataType GDALGetNonComplexDataType(GDALDataType eDT)

avalentino · 2016-07-07T16:29:11Z

@rouault that part of the code is quite tricky.
IMO the problem is not the eSrcType, rather the fact that the pointer to the source buffer pImag is set to point to the first imaginary part and not to the beginning of the buffer

        void* pImag = ((GByte *)papoSources[0])
                    + GDALGetDataTypeSize( eSrcType ) / 8 / 2;

There is an offset of half complex
The GDALCopyWords iterates on the source buffer correctly but in the last iteration the pointer refers to a complex that is partially outside the buffer.
The second element of the complex number (the one outside the buffer) should be never used but probably this is the reason of the issue you are observing.

rouault · 2016-07-07T16:34:04Z

IMO the problem is not the eSrcType

I'm pretty sure it is the issue. Since in GDALCopyWords(srcbuffer, srctype, srcspacing) srctype is used to determine the "width" of each word. So if you pass CFloat64 but you point to a single double element this is wrong. srctype should be set to Float64 here, and srcspacing left to nPixelSpaceSrc=GDALGetDataTypeSize( eSrcType ) / 8.

avalentino · 2016-07-07T16:44:49Z

srctype should be set to Float64 here, and srcspacing left to nPixelSpaceSrc=GDALGetDataTypeSize( eSrcType ) / 8.

understood.
In this case, as you already said, we need a

GDALDataType GDALGetNonComplexDataType(GDALDataType eDT)

to select the correct eSrcType

rouault · 2016-07-07T16:53:51Z

Looking at the coverage results at http://rawgit.com/rouault/gdalautotest-coverage-results/master/coverage_html/frmts/vrt/pixelfunctions.c.gcov.html , it seems that the "complex" isn't covered, and "log10" when feed with complex datatype. Regarding log10, I see in the complex case log10(r_r+i_i) and in the real case log10(fabs(r)). That doesn't seem very consistent. I guess a sqrt() is needed in the complex case ?

rouault · 2016-07-07T16:57:24Z

In the documentation of "phase", we shoul probably mention that the returned value will be in radians in [-PI,PI] range

schwehr · 2016-07-07T18:19:08Z

Discussion of the sanitizer issue is in GDAL's Trac issues here:

https://trac.osgeo.org/gdal/ticket/6587

rouault · 2016-07-08T14:35:53Z

Is anyone on fixing the buffer overflow issue ?

jmichel-otb · 2016-07-08T15:00:52Z

Nop sorry I procrastinated a lot of work to move on those 2 pull requests and now I need to get back to it ;)
As far as I can tell, imag is the only pixel function where we copy the second half of the complex pixel.

avalentino · 2016-07-08T16:05:41Z

Is anyone on fixing the buffer overflow issue ?

@rouault probably I can work on it during the weekend

avalentino · 2016-07-09T16:57:52Z

@jmichel-otb , @rouault the fix for ImagPixelFunc is in my branch

https://github.com/avalentino/gdal/tree/pixelfunctions-integration

I also addressed other changes requested by Even and added a new pixfun for dB.

rouault · 2016-07-09T17:08:32Z

@avalentino Looks good to me. I let you commit that back to SVN ?

avalentino · 2016-07-09T18:10:37Z

Hi @rouault, I'm merging my branch into svn.
Just 2 questions:

i noticed that the complex pixel function has been removed, it is wanted?
it is not clear to me the reason of 4f428d1. I assume that the phase of a negative number is PI, is it correct?

rouault · 2016-07-09T18:15:31Z

i noticed that the complex pixel function has been removed, it is wanted?

Where do you see it removed ? It is still there : https://github.com/OSGeo/gdal/blob/trunk/gdal/frmts/vrt/pixelfunctions.cpp#L188

it is not clear to me the reason of 4f428d1. I assume that the phase of a negative number is PI, is it correct?

Yes pi for the phase of a negative number is what is intended, and what will you get.
dfPixVal is a double value, and the 3rd argument of GDALCopyWords() must be the stride of the source, so that makes no sense. Here the source is a constant value, so the stride must be 0.

avalentino · 2016-07-09T18:27:25Z

Where do you see it removed ? It is still there : https://github.com/OSGeo/gdal/blob/trunk/gdal/frmts/vrt/pixelfunctions.cpp#L188

sorry, the test has been removed

Yes pi for the phase of a negative number is what is intended, and what will you get. ...

yes, sorry, my mistake

rouault · 2016-07-09T23:07:52Z

OK we are good now. Closing

This pull request creates a new metadata domain, called DERIVED_SUBDATASETS, for all datasets supported by gdal. This new metadata domain reports derived subdatasets that actually are derived vrt raster band using some of the pixel functions from Valentino. Derived datasets are only exposed if there is at least one raster band, and only derived datasets that make sense are exposed (complex related ones are exposed only if there is at least one complex raster band). The used pixel functions where integrated in another pull request, that has been merged in this one (see OSGeo#141). All metadata, geotransform and projection reference are imported from the orginal dataset. For the sake of numerical precision, derived bands will have Float64 or CFloat64 precision. Implementation details: - Reading of derived subdatasets are handled by a new dataset class (frtms/derived/deriveddataset.cpp) - Reporting the new metadata domain and its entries is done at gdaldataset class level, within GetMetaDataDomainList() and GetMetaData() methods, - Mapping between pixel functions and available derived dataset is done in a static array in frmts/dervied/derivedlist.c. Adding new derived datasets should be easy. Here is an example on Sentinel1 dataset : $ gdalinfo -nogcp -mdd DERIVED_SUBDATASETS s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff Driver: GTiff/GeoTIFF Files: s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff Size is 17663, 31106 Coordinate System is `' Metadata: AREA_OR_POINT=Area TIFFTAG_DATETIME=2015:06:20 01:52:30 TIFFTAG_IMAGEDESCRIPTION=Sentinel-1A SM SLC L1 TIFFTAG_SOFTWARE=Sentinel-1 IPF 002.45 Metadata (DERIVED_SUBDATASETS): DERIVED_SUBDATASET_0_NAME=DERIVED_SUBDATASET:AMPLITUDE:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_0_DESC=Amplitude of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_1_NAME=DERIVED_SUBDATASET:PHASE:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_1_DESC=Phase of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_2_NAME=DERIVED_SUBDATASET:REAL:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_2_DESC=Real part of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_3_NAME=DERIVED_SUBDATASET:IMAG:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_3_DESC=Imaginary part of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_4_NAME=DERIVED_SUBDATASET:CONJ:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_4_DESC=Conjugate of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_5_NAME=DERIVED_SUBDATASET:INTENSITY:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_5_DESC=Intensity (squared amplitude) of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_6_NAME=DERIVED_SUBDATASET:LOGAMPLITUDE:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_6_DESC=log10 of amplitude of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff Image Structure Metadata: INTERLEAVE=BAND Corner Coordinates: Upper Left ( 0.0, 0.0) Lower Left ( 0.0,31106.0) Upper Right (17663.0, 0.0) Lower Right (17663.0,31106.0) Center ( 8831.5,15553.0) Band 1 Block=17663x1 Type=CInt16, ColorInterp=Gray And : $ gdalinfo -nogcp DERIVED_SUBDATASET:AMPLITUDE:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff Driver: COMPLEXDERIVED/Complex derived bands Files: s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff Size is 17663, 31106 Coordinate System is `' Origin = (0.000000000000000,0.000000000000000) Pixel Size = (1.000000000000000,1.000000000000000) Metadata: AREA_OR_POINT=Area TIFFTAG_DATETIME=2015:06:20 01:52:30 TIFFTAG_IMAGEDESCRIPTION=Sentinel-1A SM SLC L1 TIFFTAG_SOFTWARE=Sentinel-1 IPF 002.45 Corner Coordinates: Upper Left ( 0.0000000, 0.0000000) Lower Left ( 0.000, 31106.000) Upper Right ( 17663.000, 0.000) Lower Right ( 17663.000, 31106.000) Center ( 8831.500, 15553.000) Band 1 Block=128x128 Type=Float64, ColorInterp=Undefined

This pull request creates a new metadata domain, called DERIVED_SUBDATASETS, for all datasets supported by gdal. This new metadata domain reports derived subdatasets that actually are derived vrt raster band using some of the pixel functions from Valentino. Derived datasets are only exposed if there is at least one raster band, and only derived datasets that make sense are exposed (complex related ones are exposed only if there is at least one complex raster band). The used pixel functions where integrated in another pull request, that has been merged in this one (see OSGeo#141). All metadata, geotransform and projection reference are imported from the orginal dataset. For the sake of numerical precision, derived bands will have Float64 or CFloat64 precision. Implementation details: - Reading of derived subdatasets are handled by a new dataset class (frtms/derived/deriveddataset.cpp) - Reporting the new metadata domain and its entries is done at gdaldataset class level, within GetMetaDataDomainList() and GetMetaData() methods, - Mapping between pixel functions and available derived dataset is done in a static array in frmts/dervied/derivedlist.c. Adding new derived datasets should be easy. Here is an example on Sentinel1 dataset : $ gdalinfo -nogcp -mdd DERIVED_SUBDATASETS s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff Driver: GTiff/GeoTIFF Files: s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff Size is 17663, 31106 Coordinate System is `' Metadata: AREA_OR_POINT=Area TIFFTAG_DATETIME=2015:06:20 01:52:30 TIFFTAG_IMAGEDESCRIPTION=Sentinel-1A SM SLC L1 TIFFTAG_SOFTWARE=Sentinel-1 IPF 002.45 Metadata (DERIVED_SUBDATASETS): DERIVED_SUBDATASET_0_NAME=DERIVED_SUBDATASET:AMPLITUDE:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_0_DESC=Amplitude of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_1_NAME=DERIVED_SUBDATASET:PHASE:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_1_DESC=Phase of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_2_NAME=DERIVED_SUBDATASET:REAL:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_2_DESC=Real part of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_3_NAME=DERIVED_SUBDATASET:IMAG:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_3_DESC=Imaginary part of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_4_NAME=DERIVED_SUBDATASET:CONJ:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_4_DESC=Conjugate of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_5_NAME=DERIVED_SUBDATASET:INTENSITY:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_5_DESC=Intensity (squared amplitude) of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_6_NAME=DERIVED_SUBDATASET:LOGAMPLITUDE:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff DERIVED_SUBDATASET_6_DESC=log10 of amplitude of input bands from s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff Image Structure Metadata: INTERLEAVE=BAND Corner Coordinates: Upper Left ( 0.0, 0.0) Lower Left ( 0.0,31106.0) Upper Right (17663.0, 0.0) Lower Right (17663.0,31106.0) Center ( 8831.5,15553.0) Band 1 Block=17663x1 Type=CInt16, ColorInterp=Gray And : $ gdalinfo -nogcp DERIVED_SUBDATASET:AMPLITUDE:s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff Driver: COMPLEXDERIVED/Complex derived bands Files: s1a-s6-slc-vv-20150619t195043-20150619t195101-006447-00887d-001.tiff Size is 17663, 31106 Coordinate System is `' Origin = (0.000000000000000,0.000000000000000) Pixel Size = (1.000000000000000,1.000000000000000) Metadata: AREA_OR_POINT=Area TIFFTAG_DATETIME=2015:06:20 01:52:30 TIFFTAG_IMAGEDESCRIPTION=Sentinel-1A SM SLC L1 TIFFTAG_SOFTWARE=Sentinel-1 IPF 002.45 Corner Coordinates: Upper Left ( 0.0000000, 0.0000000) Lower Left ( 0.000, 31106.000) Upper Right ( 17663.000, 0.000) Lower Right ( 17663.000, 31106.000) Center ( 8831.500, 15553.000) Band 1 Block=128x128 Type=Float64, ColorInterp=Undefined Squashed commit of the following: commit 6371674 Merge: acd8343 20b0f9d Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 13 09:08:37 2016 +0200 Merge branch 'trunk' into enhance-complex-datasets-with-pf commit acd8343 Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 13 09:04:15 2016 +0200 DOC: Moving DERIVED dataset entry at the correct location in supported formats table commit bf03eed Author: Julien Michel <julien.michel@cnes.fr> Date: Tue Jul 12 17:31:13 2016 +0200 DOC: Adding documentation derived datasets commit 1a9a5fa Author: Julien Michel <julien.michel@cnes.fr> Date: Tue Jul 12 16:57:56 2016 +0200 ENH: Remove last magic number commit 3a8ff49 Author: Julien Michel <julien.michel@cnes.fr> Date: Tue Jul 12 15:44:21 2016 +0200 TEST: Check expected checksum following modification of intensity function commit 72f37d6 Author: Julien Michel <julien.michel@cnes.fr> Date: Tue Jul 12 15:44:02 2016 +0200 TEST: Remove duplicate pixfun.py commit eba1603 Author: Julien Michel <julien.michel@cnes.fr> Date: Tue Jul 12 15:09:53 2016 +0200 COMP: Remove pixel function C file, as it has been renamed to .cpp commit 5b785c4 Author: Julien Michel <julien.michel@cnes.fr> Date: Tue Jul 12 14:20:40 2016 +0200 TEST: Fix remaining jp2openjpeg failing tests commit 3fcc552 Author: Julien Michel <julien.michel@cnes.fr> Date: Tue Jul 12 14:20:24 2016 +0200 ENH: Add the DERIVED_SUBDATASETS domain to the filtered domain in gdaljp2metadata commit 5d922d9 Merge: 84afb22 d9ba836 Author: Julien Michel <julien.michel@cnes.fr> Date: Tue Jul 12 13:33:13 2016 +0200 Merge branch 'trunk' into enhance-complex-datasets-with-pf commit 84afb22 Author: Julien Michel <julien.michel@cnes.fr> Date: Tue Jul 12 12:17:44 2016 +0200 TEST: Fixing tests failing because of new metadata domain DERIVED_SUBDATASETS commit a897355 Merge: 8b7793e bf7b478 Author: Julien Michel <julien.michel@cnes.fr> Date: Tue Jul 12 10:45:45 2016 +0200 Merge branch 'trunk' into enhance-complex-datasets-with-pf commit 8b7793e Author: Julien Michel <julien.michel@cnes.fr> Date: Fri Jul 8 15:43:59 2016 +0200 BUG: Only report DERIVED metadata domain if there is at least one raster band commit f6aaf6d Author: Julien Michel <julien.michel@cnes.fr> Date: Fri Jul 8 15:05:56 2016 +0200 BUG: Ensure that DERIVED mdd is not reported twice commit d96cf59 Author: Julien Michel <julien.michel@cnes.fr> Date: Fri Jul 8 14:44:42 2016 +0200 TEST: More complete tests, including geoTransform, Proj and Checksum tests (from PR review) commit e1c072a Author: Julien Michel <julien.michel@cnes.fr> Date: Fri Jul 8 14:44:14 2016 +0200 BUG: We must also now the return type of the pixel function (for instance conj is CFloat64, but mod is Float64) commit b0abc50 Author: Julien Michel <julien.michel@cnes.fr> Date: Fri Jul 8 14:23:11 2016 +0200 BUG: Fix geotransform setting commit 153e215 Author: Julien Michel <julien.michel@cnes.fr> Date: Fri Jul 8 13:48:45 2016 +0200 TEST: Fix tests failing due to incorrect number of metadata domains commit 17516f6 Author: Julien Michel <julien.michel@cnes.fr> Date: Fri Jul 8 13:42:14 2016 +0200 TEST: Fix autotest for derived datasets commit 2490c3d Author: Julien Michel <julien.michel@cnes.fr> Date: Fri Jul 8 13:40:59 2016 +0200 ENH: Better implementation of access to available subdatasets description commit e7965e4 Merge: 57e607d 5c295ab Author: Julien Michel <julien.michel@cnes.fr> Date: Fri Jul 8 12:13:01 2016 +0200 MRG: Merge trunk into branch commit 57e607d Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 17:15:44 2016 +0200 ENH: Report correct block size (from PR review) commit a65e6ef Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 17:12:40 2016 +0200 ENH: Avoid exposing complex relevant derived datasets if there is not at least one complex raster band commit 03ca1aa Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 16:58:35 2016 +0200 COMP: Adding C wrapping stuff to make it compile commit c32128a Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 16:25:33 2016 +0200 ENH: Renamed cderived -> derived (as it is not limited to complex datasets) commit a42afc2 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 16:20:05 2016 +0200 ENH: Hide static const DerivedDatasetDescription asDDSDesc [] in a compiled file to avoid duplication in binaries (from PR review) commit 44f6f4d Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 15:43:50 2016 +0200 STY: papo -> o (from PR review) commit 4713785 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 15:39:15 2016 +0200 ENH: Replace magic number by appropriate definition (from PR review) commit 842bff5 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 15:36:43 2016 +0200 DOC: Fix copyright commit 5742267 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 15:36:01 2016 +0200 ENH: Remove unused code (from PR review) commit 07a3c7e Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 15:35:06 2016 +0200 STY: Fix if indentation (from PR review) commit 27ab9f5 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 15:33:23 2016 +0200 ENH: Clear debug traces (from PR review) commit 96ff920 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 15:32:10 2016 +0200 ENH: CPLStringList * -> CPLStringList (from PR review) commit f014748 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 13:47:15 2016 +0200 TEST: travis-ci failing due to python package imported but not used commit afed253 Merge: db55680 e8cb1ae Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 12:08:30 2016 +0200 Merge branch 'pixelfunctions-integration' into enhance-complex-datasets-with-pf commit e8cb1ae Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 12:07:56 2016 +0200 TEST: Fixing test failing due to bad indent commit 0345b9e Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 11:45:29 2016 +0200 COMP: Trying to fix travis-ci compilation error commit db55680 Merge: 2162049 de301c3 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 11:40:23 2016 +0200 Merge branch 'pixelfunctions-integration' into enhance-complex-datasets-with-pf commit de301c3 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 11:39:52 2016 +0200 TEST: Make numpy dependency optional (from PR review) commit 2162049 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 11:29:19 2016 +0200 TEST: Adding simple opening tests for cderived datasets commit f1d0ab1 Merge: 7ff31aa 8783995 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 11:25:51 2016 +0200 Merge branch 'pixelfunctions-integration' into enhance-complex-datasets-with-pf commit 8783995 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 11:25:12 2016 +0200 COMP: Fix include of gdal_vrt.h commit 7ff31aa Merge: 318ed3d 66e3388 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 11:07:38 2016 +0200 Merge branch 'pixelfunctions-integration' into enhance-complex-datasets-with-pf commit 66e3388 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 11:06:30 2016 +0200 COMP: Add gdal_vrt.h include to get declaration of prototype for GDALRegisterDefaultPixelFunc commit 318ed3d Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 11:02:41 2016 +0200 COMP: Adding make file for windows commit f7fbcc2 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 10:59:32 2016 +0200 BUG: Use the correct pixel function commit 97124fc Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 10:53:46 2016 +0200 DOC: Add copyright notices commit ec88c34 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 10:50:12 2016 +0200 ENH: Support several types of derived datasets, all declared in one place commit f264169 Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 09:31:52 2016 +0200 ENH: Remove hard-coded pixel function now that we have them within gdal commit 07eae9f Author: Julien Michel <julien.michel@cnes.fr> Date: Thu Jul 7 09:14:39 2016 +0200 ENH: Add derived dataset to GDALmake.opt.in commit c2e7853 Merge: dbd8187 0e14ea1 Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 17:06:49 2016 +0200 Merge branch 'pixelfunctions-integration' into enhance-complex-datasets-with-pf commit 0e14ea1 Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 17:03:40 2016 +0200 COMP: Make pixelfunctions static, and solve link error commit d9fd29b Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 16:44:23 2016 +0200 DOC: Adding a section about default pixel functions in vrt tutorial (from PR review) commit 5e40669 Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 16:34:33 2016 +0200 ENH: Update makefile.vc as well (from PR review) commit 6f95e7d Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 16:32:51 2016 +0200 ENH: Avoid ignoring c files (from PR review) commit bf13853 Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 16:28:49 2016 +0200 ENH: Remove code that only makes sense as part of a plugin (from PR review) commit f0f79ef Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 16:27:30 2016 +0200 ENH: Removing useless files (from PR review) commit dbd8187 Merge: a75faf5 a53f540 Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 16:04:45 2016 +0200 Merge branch 'pixelfunctions-integration' into enhance-complex-datasets-with-pf commit a53f540 Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 15:55:51 2016 +0200 ENH: Register default pixel functions commit 22c92a8 Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 15:55:27 2016 +0200 COMP: Add prototypes for all functions commit 9d18110 Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 15:55:00 2016 +0200 COMP: Adding pixelfunction file for compilation commit 8ffbb53 Merge: 22919bf 67373d6 Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 15:43:56 2016 +0200 Merge branch 'pixelfunctions-import' into pixelfunctions-integration commit 67373d6 Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 15:42:54 2016 +0200 TEST: Removing those tests since they seem unrelated to the pixel functions commit a75faf5 Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 15:18:44 2016 +0200 ENH: Remove useless trace commit a5a8adb Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 15:12:09 2016 +0200 ENH: Ensure that geotiff dataset calls base GetMetadata() and GetMetadataDomainList() commit a7b6a89 Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 15:11:30 2016 +0200 ENH: Implement virtual GetMetadata() and GetMetadataDomainList() in GDALDataset commit 94d8ecc Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 15:10:55 2016 +0200 ENH: Minor fixes in derived dataset commit de11bcd Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 11:53:56 2016 +0200 Prepare for integration in Gdal commit d384b9d Author: Julien Michel <julien.michel@cnes.fr> Date: Wed Jul 6 10:19:20 2016 +0200 ENH: Report projectionRef, geoTransform and GCPs for derived subdatasets commit 665d879 Author: Julien Michel <julien.michel@cnes.fr> Date: Tue Jul 5 13:00:25 2016 +0200 ENH: Expose derived subdatasets in SetBand() commit e277ccf Author: Julien Michel <julien.michel@cnes.fr> Date: Mon Jul 4 16:16:33 2016 +0200 ENH: Comments from review commit a984a78 Author: Julien Michel <julien.michel@cnes.fr> Date: Mon Jul 4 14:25:11 2016 +0200 ENH: Adding a complex derived dataset to allow for on-the-fly computation of amplitude while reading commit ac7f70a Author: Antonio Valentino <antonio.valentino@tiscali.it> Date: Mon Feb 1 23:48:59 2016 +0100 Fix floating point trunation in mod function (closes OSGeogh-3) commit 4668088 Author: Antonio Valentino <antonio.valentino@tiscali.it> Date: Tue Jun 23 22:34:00 2015 +0200 Rename makecomplex into complex commit 6648310 Merge: 4038d40 d7eaf5f Author: Antonio Valentino <antonio.valentino@tiscali.it> Date: Tue Jun 23 22:25:01 2015 +0200 Merge remote-tracking branch 'bopen/master' commit d7eaf5f Author: Alessandro Amici <alexamici@gmail.com> Date: Tue Jun 23 17:02:43 2015 +0200 Add the MakeCompexPixelFunc that make a complex out of a real and imag bands commit 4038d40 Author: Antonio Valentino <antonio.valentino@tiscali.it> Date: Sun Apr 13 12:39:30 2014 +0200 Minor improvements commit 01e325c Author: Antonio Valentino <antonio.valentino@tiscali.it> Date: Sun Apr 13 12:39:07 2014 +0200 Silence compiler warnings commit 29f6bd3 Author: Antonio Valentino <antonio.valentino@tiscali.it> Date: Sun Apr 13 12:37:57 2014 +0200 Remove unnecessary includes commit 2cfd449 Author: Antonio Valentino <antonio.valentino@tiscali.it> Date: Sun Apr 13 12:36:38 2014 +0200 Improved the README.txt file commit cceed0b Author: Antonio Valentino <antonio.valentino@tiscali.it> Date: Sun Apr 13 11:47:01 2014 +0200 Initial commit

This pull request creates a new metadata domain, called DERIVED_SUBDATASETS, for all datasets supported by gdal. This new metadata domain reports derived subdatasets that actually are derived vrt raster band using some of the pixel functions from Valentino. Derived datasets are only exposed if there is at least one raster band, and only derived datasets that make sense are exposed (complex related ones are exposed only if there is at least one complex raster band). The used pixel functions where integrated in another pull request, that has been merged in this one (see #141). All metadata, geotransform and projection reference are imported from the orginal dataset. For the sake of numerical precision, derived bands will have Float64 or CFloat64 precision. Implementation details: - Reading of derived subdatasets are handled by a new dataset class (frmts/derived/deriveddataset.cpp) - Reporting the new metadata domain and its entries is done at gdaldataset class level, within GetMetaDataDomainList() and GetMetaData() methods, - Mapping between pixel functions and available derived dataset is done in a static array in frmts/dervied/derivedlist.c. Adding new derived datasets should be easy. git-svn-id: https://svn.osgeo.org/gdal/trunk@34650 f0d54148-0727-0410-94bb-9a71ac55c965

avalentino and others added 15 commits April 13, 2014 11:47

Initial commit

cceed0b

Improved the README.txt file

2cfd449

Remove unnecessary includes

29f6bd3

Silence compiler warnings

01e325c

Minor improvements

4038d40

Add the MakeCompexPixelFunc that make a complex out of a real and ima…

d7eaf5f

…g bands

Merge remote-tracking branch 'bopen/master'

6648310

Rename makecomplex into complex

4668088

Fix floating point trunation in mod function (closes OSGeogh-3)

ac7f70a

Prepare for integration in Gdal

de11bcd

TEST: Removing those tests since they seem unrelated to the pixel fun…

67373d6

…ctions

Merge branch 'pixelfunctions-import' into pixelfunctions-integration

8ffbb53

COMP: Adding pixelfunction file for compilation

9d18110

COMP: Add prototypes for all functions

22c92a8

ENH: Register default pixel functions

a53f540

rouault reviewed Jul 6, 2016
View reviewed changes

ENH: Removing useless files (from PR review)

f0f79ef

jmichel-otb added 5 commits July 6, 2016 16:28

ENH: Remove code that only makes sense as part of a plugin (from PR r…

bf13853

…eview)

ENH: Avoid ignoring c files (from PR review)

6f95e7d

ENH: Update makefile.vc as well (from PR review)

5e40669

DOC: Adding a section about default pixel functions in vrt tutorial (…

d9fd29b

…from PR review)

COMP: Make pixelfunctions static, and solve link error

0e14ea1

jmichel-otb added 4 commits July 7, 2016 11:25

COMP: Fix include of gdal_vrt.h

8783995

TEST: Make numpy dependency optional (from PR review)

de301c3

COMP: Trying to fix travis-ci compilation error

0345b9e

TEST: Fixing test failing due to bad indent

e8cb1ae

jmichel-otb mentioned this pull request Jul 7, 2016

Enhance gdal capabilities for complex datasets #142

Closed

rouault closed this Jul 9, 2016

jmichel-otb mentioned this pull request Jul 13, 2016

PR: Enhance gdal capabilities for complex datasets (squashed) #143

Closed

jmichel-otb mentioned this pull request Jul 13, 2016

Enhance gdal capabilities for complex datasets (squashed) #144

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pixelfunctions integration #141

Pixelfunctions integration #141

jmichel-otb commented Jul 6, 2016

rouault Jul 6, 2016

rouault commented Jul 6, 2016

rouault commented Jul 6, 2016

jmichel-otb commented Jul 7, 2016

rouault commented Jul 7, 2016

avalentino commented Jul 7, 2016

rouault commented Jul 7, 2016 •

edited

Loading

avalentino commented Jul 7, 2016

rouault commented Jul 7, 2016

rouault commented Jul 7, 2016

schwehr commented Jul 7, 2016

rouault commented Jul 8, 2016

jmichel-otb commented Jul 8, 2016

avalentino commented Jul 8, 2016 •

edited

Loading

avalentino commented Jul 9, 2016

rouault commented Jul 9, 2016

avalentino commented Jul 9, 2016

rouault commented Jul 9, 2016

avalentino commented Jul 9, 2016

rouault commented Jul 9, 2016

Pixelfunctions integration #141

Pixelfunctions integration #141

Conversation

jmichel-otb commented Jul 6, 2016

rouault Jul 6, 2016

Choose a reason for hiding this comment

rouault commented Jul 6, 2016

rouault commented Jul 6, 2016

jmichel-otb commented Jul 7, 2016

rouault commented Jul 7, 2016

avalentino commented Jul 7, 2016

rouault commented Jul 7, 2016 • edited Loading

avalentino commented Jul 7, 2016

rouault commented Jul 7, 2016

rouault commented Jul 7, 2016

schwehr commented Jul 7, 2016

rouault commented Jul 8, 2016

jmichel-otb commented Jul 8, 2016

avalentino commented Jul 8, 2016 • edited Loading

avalentino commented Jul 9, 2016

rouault commented Jul 9, 2016

avalentino commented Jul 9, 2016

rouault commented Jul 9, 2016

avalentino commented Jul 9, 2016

rouault commented Jul 9, 2016

rouault commented Jul 7, 2016 •

edited

Loading

avalentino commented Jul 8, 2016 •

edited

Loading