Cannot convert to images to jp2

[dannylamb]

Although jp2 support in imagemagick is advertised, it always appears to end in a core dump, resulting in a file of size 0.

root@claw:/home/ubuntu/islandora# convert --version
Version: ImageMagick 6.9.7-4 Q16 x86_64 20170114 http://www.imagemagick.org
Copyright: © 1999-2017 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC Modules OpenMP 
Delegates (built-in): bzlib djvu fftw fontconfig freetype jbig jng jp2 jpeg lcms lqr ltdl lzma openexr pangocairo png tiff wmf x xml zlib

root@claw:/home/ubuntu/islandora# convert CCITT_1.TIF CCITT_1.jp2
Aborted (core dumped)

root@claw:/home/ubuntu/islandora# ls -alh CCITT_1.jp2
-rw-r--r-- 1 vagrant vagrant 0 Sep 27 12:47 CCITT_1.jp2

This is just for Ubuntu 18.04, btw. Centos 7 may very well be fine, I haven't tried it yet.

[seth-shaw-unlv]

On CentOS 7:

[vagrant@claw ~]$ convert --version
Version: ImageMagick 6.7.8-9 2019-08-08 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
Features: OpenMP   

[vagrant@claw ~]$ sudo convert test.tif test.jp2
convert: Unknown field with tag 306 (0x132) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/820.
convert: Wrong data type 3 for "PixelXDimension"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/820.
convert: Wrong data type 3 for "PixelYDimension"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/820.
convert: Incompatible type for "FileSource"; tag ignored. `TIFFFetchNormalTag' @ warning/tiff.c/TIFFWarnings/820.
convert: Incompatible type for "SceneType"; tag ignored. `TIFFFetchNormalTag' @ warning/tiff.c/TIFFWarnings/820.

[vagrant@claw ~]$ ls -l
total 8868
drwxr-xr-x. 1 vagrant vagrant     832 Sep 27 14:28 islandora
-rw-r-----. 1 vagrant vagrant 6973528 Sep 27 14:26 test
-rw-r--r--. 1 vagrant vagrant 2104197 Sep 27 14:31 test-0.jp2

It threw a few warnings (it was probably quirks with that particular Tiff) but I was able to open and view the Jp2 file just fine.

[dannylamb]

I can't get imagemagick with jp2 support going on Ubuntu using the lyrasis:imagemagick-jp2 PPA. In desperation, I tried using GraphicsMagick instead of ImageMagick, and it works right out of the box with no extra compilation steps. Supposedly GraphicsMagick has less in terms of features than ImageMagick, so I'm wondering if anyone out there knows of a must-have feature we'd be missing if we made the switch for Ubuntu installs?

@Islandora/8-x-committers?

[seth-shaw-unlv]

Thinking of the few use cases I can think of (size control, auto-orientation, profile stripping, and watermarking) all appear available with GraphicsMagick. 👍

[jonathangreen]

I know a little bit about this:

GraphicsMagick uses the jasper library to do its jp2 conversions. Imagemagick used to use jasper, but now uses OpenJpeg to do so. A side effect of this change of library means jp2 support is no longer included in ubuntu. This was done because jasper was a dead project (although it appears to be alive again now?).

Anyway after all that long winded background. My experience has been that jasper doesn't handle weird or malformed jp2 images as well as openjpeg. That said, since graphicsmagick supports JP2 out of the box, maybe it makes sense to switch to it as the default, as long as it doesn't exclude the possibilty of someone configuring to use imagemagick if they want to compile it with jp2 support and it works better for their use case.

[jonathangreen]

@dannylamb so I just gave this a test since I was going to build a new package to see if it would fix things, but it seems like its just working for me. Heres what I did to test. It would be nice if someone else could give this a try and make sure it works for them.

$ vagrant up --no-provision

$ vagrant ssh

$ sudo add-apt-repository ppa:lyrasis/imagemagick-jp2

$ sudo apt-get update

$ sudo apt-get install imagemagick 

$ convert --version
Version: ImageMagick 6.9.7-4 Q16 x86_64 20170114 http://www.imagemagick.org
Copyright: © 1999-2017 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC Modules OpenMP
Delegates (built-in): bzlib djvu fftw fontconfig freetype jbig jng jp2 jpeg lcms lqr ltdl lzma openexr pangocairo png tiff wmf x xml zlib

$ wget https://www.fileformat.info/format/tiff/sample/3794038f08df403bb446a97f897c578d/download -O CCITT_1.TIF
--2020-02-05 19:49:07--  https://www.fileformat.info/format/tiff/sample/3794038f08df403bb446a97f897c578d/download
Resolving www.fileformat.info (www.fileformat.info)... 104.27.185.199, 104.27.184.199, 2606:4700:3032::681b:b9c7, ...
Connecting to www.fileformat.info (www.fileformat.info)|104.27.185.199|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18382 (18K) [application/binary]
Saving to: ‘CCITT_1.TIF’

CCITT_1.TIF                            100%[============================================================================>]  17.95K  --.-KB/s    in 0.06s

2020-02-05 19:49:08 (320 KB/s) - ‘CCITT_1.TIF’ saved [18382/18382]

$ convert CCITT_1.TIF test.jp2

$ ls -lah test.jp2
-rw-rw-r-- 1 vagrant vagrant 95K Feb  5 19:49 test.jp2

$ file test.jp2
test.jp2: JPEG 2000 Part 1 (JP2)

$ convert test.jp2 test.tif

$ file test.tif
test.tif: TIFF image data, little-endian, direntries=13, height=2376, bps=2, compression=none, PhotometricIntepretation=BlackIsZero, orientation=upper-left, width=1728

Since this isn't working on a fully provisioned box and it does seem to be working here, maybe we are doing something that breaks imagemagick somewhere else in the playbook?

[dannylamb]

I can replicate what you've done on a clean ubuntu box. I'm diving in again to see what we're doing wrong 👨‍🔧

[dannylamb]

I can't see how what we do is messing with this, other than at this point, anotehr package must be pulling in a dependency that messes with this? I'll try with a fresh box and start installing suspects and see if I can find one.

[dannylamb]

So if i run it up to the crayfish role, and then install from the lyrasis:ppa, it's broken. This means we're installing something before we even get to Crayfish that conflicts. It is not any of the Crayfish packages or after in the installer that's the problem. I'm going to build it play by play until I find where it breaks and hopefully I can whittle it down to particular package.

[jonathangreen]

Was just thinking about this. If I had to bet its probably the grok role breaking this. Since grok is based on OpenJpeg maybe its messing with the shared libraries somehow? I didn't get a chance to test though.

If that is the case maybe it makes sense to just drop grok and used normal openjpeg.

[dannylamb]

Looks like there's been a lot of progress in the grok project, including linux artifacts. Looks like we won't even have to build anymore, which may fix this.

[dannylamb]

I've just confirmed that building v2.3.0 is the culprit. I up'd a new box with --no-provision and installed imagemagick from the ppa. Convert worked fine. The I build grok and tried convert again... no 🎲

Not entirely sure what's up with the artifacts they supply from their github releases. Is it possible to just use those @jonathangreen or do we really have to build from source?

I'll try building 3.1.0 and see if that also gives us the same results.

[dannylamb]

3.1.0 works! I'll make a PR.

[dannylamb]

I made two. One for the playbook and another on the role to update the default to 3.1.0.

[dannylamb]

Practically speaking, this is resolved via Islandora-Devops/islandora-playbook@5a0e2b1, although there still is a nagging issue of tests failing on islandora-deprecated/ansible-role-grok#8, which i'm going to split into a separate issue.