OpenJpegProcessor: Invalid magic value for BMP file.

[RussellMcOrmond]

We are trying to configure OpenJpegProcessor, which makes use of opj_dump and opj_decompress from http://www.openjpeg.org/ which we have separately installed.

We are using an unmodified Cantaloupe-3.3.zip distributed from Github, with our properties file indicating "processor.jp2 = OpenJpegProcessor".

If curious, our Docker configuration we have so far is at https://github.com/c7a/cihm-cantaloupe

When we try to access one of our JPEG2000 images we get an error about the BMP file:

cihm-cantaloupe_1  | 14:45:15.996 [qtp1029991479-7265] INFO  e.i.l.c.p.OpenJpegProcessor - Invoking opj_dump -i /repository/cihmz2/aip/oocihm/350/oocihm.99400/data/sip/data/files/oocihm.99400.1.jp2
cihm-cantaloupe_1  | 14:45:16.002 [qtp1029991479-7265] INFO  e.i.l.c.p.OpenJpegProcessor - Invoking opj_decompress -i /repository/cihmz2/aip/oocihm/350/oocihm.99400/data/sip/data/files/oocihm.99400.1.jp2 -o /tmp/cantaloupe-e215daea-4e7c-441d-b21d-f6732bbe9c2a.bmp
cihm-cantaloupe_1  | 14:45:16.007 [qtp1029991479-7265] ERROR o.r.C.Server - An exception occurred writing the response entity
cihm-cantaloupe_1  | java.io.IOException: java.lang.IllegalArgumentException: Invalid magic value for BMP file.
cihm-cantaloupe_1  |    at edu.illinois.library.cantaloupe.resource.ImageRepresentation.write(ImageRepresentation.java:120)

If I run the command (in our Docker container, then copy the bmp image out) I am able to run ImageMagick's identify on it:

russell@russell-desktop2:~/Downloads/a$ identify cantaloupe-e215daea-4e7c-441d-b21d-f6732bbe9c2a.bmp
cantaloupe-e215daea-4e7c-441d-b21d-f6732bbe9c2a.bmp BMP3 2687x3412 2687x3412+0+0 8-bit sRGB 256c 9.173MB 0.030u 0:00.010

This is the latest version of compress, as pre-packaged for Alpine Linux.
https://pkgs.alpinelinux.org/package/v3.5/main/x86_64/openjpeg-tools

/tmp $ opj_decompress -h

This is the opj_decompress utility from the OpenJPEG project.
It decompresses JPEG 2000 codestreams to various image formats.
It has been compiled against openjp2 library v2.1.2.

When we go to the info.json URL for that identifier we get the image information with the same width and height as we get from identify, so the output from opj_dump is being handled without a problem.

This may turn out to be a configuration question and not a bug, but I thought I'd file this issue as I'm not sure what to try next. Whatever the problem turns out to be, other users may find the answer helpful as well.

[adolski]

It looks like the BMP reader isn't liking the image that opj_decompress is writing. It's complaining about the magic value, which should be the very first thing opj_decompress writes, so maybe nothing is getting to it.

Note that /tmp/cantaloupe-[uuid].bmp is a symlink to /dev/stdout; this is so that the ImageIO BMP reader can consume the output from opj_decompress directly. I guess the Cantaloupe user does have permission to write to this, because it did create it, but, just throwing it out there anyway.

Would you be able to post the source JP2 and the BMP somewhere where I can look at them?

[RussellMcOrmond]

http://cdn.canadiana.ca/cantaloupe/oocihm.99400.1.jp2 is the source image, and http://cdn.canadiana.ca/cantaloupe/oocihm.99400.1.bmp was generated inside the Cantaloupe container with opj_decompress -i /repository/cihmz2/aip/oocihm/350/oocihm.99400/data/sip/data/files/oocihm.99400.1.jp2 -o oocihm.99400.1.bmp

I refreshed the Docker container to enable more debugging and I now get different error messages.

cihm-cantaloupe_1  | 21:17:16.198 [qtp1029991479-22] DEBUG e.i.l.c.r.AbstractResource - readInfo(): read from oocihm.99400/data/sip/data/files/oocihm.99400.1.jp2 in 1 msec
cihm-cantaloupe_1  | 21:17:16.221 [qtp1029991479-22] INFO  e.i.l.c.p.OpenJpegProcessor - Invoking opj_decompress -i /repository/cihmz2/aip/oocihm/350/oocihm.99400/data/sip/data/files/oocihm.99400.1.jp2 -o /tmp/cantaloupe-30db6c07-4261-44c2-a2f8-e660a18eef18.bmp
cihm-cantaloupe_1  | 21:17:16.229 [qtp1029991479-22] DEBUG e.i.l.c.p.i.AbstractImageReader - createReader(): ignoring metadata? true
cihm-cantaloupe_1  | 21:17:16.231 [qtp1029991479-22] DEBUG e.i.l.c.p.i.AbstractImageReader - createReader(): using com.sun.imageio.plugins.bmp.BMPImageReader
cihm-cantaloupe_1  | 21:17:16.245 [qtp1029991479-22] ERROR o.r.C.Server - An exception occurred writing the response entity
cihm-cantaloupe_1  | java.io.IOException: edu.illinois.library.cantaloupe.processor.ProcessorException: Unable to read the image header. (command output:
cihm-cantaloupe_1  | )
cihm-cantaloupe_1  |    at edu.illinois.library.cantaloupe.resource.ImageRepresentation.write(ImageRepresentation.java:120)
cihm-cantaloupe_1  |    at org.restlet.engine.adapter.ServerCall.writeResponseBody(ServerCall.java:513)
cihm-cantaloupe_1  |    at org.restlet.engine.adapter.ServerCall.sendResponse(ServerCall.java:457)

I see that symbolic link being created, so it's not missing and doesn't have permission problems.

/tmp $ ls -l
total 8
lrwxrwxrwx    1 cantalou nogroup         11 May 10 21:17 cantaloupe-30db6c07-4261-44c2-a2f8-e660a18eef18.bmp -> /dev/stdout
drwxr-xr-x    2 cantalou nogroup       4096 May 10 21:17 hsperfdata_cantaloupe
drwxr-xr-x    3 cantalou nogroup       4096 May 10 21:17 jetty-0.0.0.0-8182-Cantaloupe-3.3.war-_-any-8417398787360820156.dir
/tmp $ 

[adolski]

Both images look valid to me. On my end, Cantaloupe 3.3 + OpenJpegProcessor + OpenJPEG 2.1.2 has no problem with the jp2.

I'm afraid I don't have any ideas. I do know that OpenJpegProcessor uses opj_decompress the same way KakaduProcessor uses kdu_expand, so seeing if KakaduProcessor does the same thing might uncover a little more information.

I haven't tried your Docker setup as I'm not familiar with using Docker, but if you could point me to a quick howto (for Mac) I might be able to do it, though probably not right away.

[RussellMcOrmond]

Will try to get more details next week. At conference today and tomorrow (We gave brief demo of what we've done so far with IIIF). Will see about putting wrapper around opj_decompress to log the environment might be helpful (library path issues that didn't apply to command line? Will find out).

I'll also try outside the container in case there is something with containers that disables /dev/stdout

We have some Mac people who work here, so I'll try to get one of them to send some details on how that would work.

[RussellMcOrmond]

Seems that opj_decompress isn't quiet when it is running. I renamed opj_decompress and created a bash wrapper around it to see what the output was.

bash-4.3# cat /usr/bin/opj_decompress
#!/bin/bash

opj_decompress.orig $@ | tee /tmp/output.txt 
bash-4.3# head -1 /tmp/output.txt 
[INFO] Start to read j2k main header (3147).
bash-4.3# 

I filtered that first line:

#!/bin/bash
opj_decompress.orig $@ | tail -n+2 

And now it will work.

It appears that a quiet mode is already in the works uclouvain/openjpeg#130 , although I can't tell the status of that.

The info_callback() in https://github.com/uclouvain/openjpeg/blob/master/src/bin/jp2/opj_decompress.c unfortunately goes to stddout, so Cantaloupe's use of /dev/stdout as the destination is in conflict.

OpenJpeg apparently has a native Java interface, although I don't know enough Java to comment.

[RussellMcOrmond]

I should have included the following for amusement -- it is interesting that the file is able to be read with the logging information embedded within the output once the header information is the first thing read:

russell@huntsman:~/cihm-cantaloupe$ strings /repository/tmp/out.bmp | grep '\[INFO\]'
[INFO] Main header has been correctly decoded.
[INFO] No decoded area parameters, set the decoded area to the whole image
[INFO] Header of tile 1 / 1 has been read.
[INFO] Tile 1/1 has been decoded.
[INFO] Image data has been updated with tile 1.
[INFO] Stream reached its end !
[INFO] Generated Outfile /tmp/cantaloupe-57c47080-4d68-4498-9f3a-2e893a5dd2a7.bmp
russell@huntsman:~/cihm-cantaloupe$ 

[adolski]

Thanks for looking into this. It's a pity that there's no way to quiet opj_decompress.

There is a related issue for adding the explicit ability to write to stdout, instead of having to resort to a symlink hack: uclouvain/openjpeg#224

OpenJPEG does have a JNI (mechanism for invoking native code from Java) wrapper, but the last time I looked into it, it was broken: uclouvain/openjpeg#656

[RussellMcOrmond]

I've submitted uclouvain/openjpeg#928 as a possible fix. Hopefully this will at least spark the right people to take a look at this issue and solve in a better way if they don't like this option.

[adolski]

Cool, thanks for doing that!

[mateusz-matela]

uclouvain/openjpeg#928 has been merged a few weeks ago. We still need to wait for a fix in cantaloupe, right? Or is it enough to rebuild openjpeg from the newest source?

[adolski]

All you have to do is rebuild openjpeg.

I should probably document this.

[RussellMcOrmond]

I haven't had a chance to confirm from the latest sources, but with the quiet mode Cantaloupe worked.

Might be useful to leave this ticket open as documentation of the issue, until we have a new release of OpenJPEG to reference in the documentation.

[RussellMcOrmond]

I had a chance to build a new Docker image with Cantaloupe and the new OppenJPEG.

Seems that there isn't a way to add a custom command-line option to the call to opj_decompress via a config option (somehow I assumed there was), so I can't add the --quiet option to the command-line.

opj_decompress -i ...

needs to become

opj_decompress -quiet -i ...

Apologies, but I don't know Java well enough to create a configuration option in the property files to facilitate this, or even how to build such that a modification to OpenJpegProcessor.java could be tested.

[adolski]

No problem, I will add this to the release/3.3 branch for inclusion in 3.3.2.

[adolski]

OpenJpegProcessor now checks the opj_decompress version and adds the -quiet flag if supported.

[RussellMcOrmond]

Sorry for the delay, I was on vacation.

I built our Docker image using 3.3.2 and 'master' branch from OpenJpeg, and everything now works fine: http://universalviewer.io/uv.html?manifest=http://presentation.demo.canadiana.ca/iiif/oocihm.99400/manifest

Thank you.