Out-of-bound left shift in opj_j2k_setup_encoder (src/lib/openjp2/j2k.c)

[ProbeFuzzer]

(This problem is discovered with UBSAN enabled)
On latest version (2.3) and master branch of openjpeg:
there is an integer overflow caused by out-of-bound left shift in opj_j2k_setup_encoder function (src/lib/openjp2/j2k.c), which could cause denial of service via a crafted bmp file.

src/lib/openjp2/j2k.c:7304:48: runtime error: shift exponent 4294967295 is too large for 32-bit type 'int'

To reproduce this issue, run: bin/opj_compress -n 1 -i $POC -o OUTPUT

The POC could be downloaded at: https://github.com/ProbeFuzzer/poc/blob/master/openjpeg/openjpeg_2-3_opj_compress_integer-overflow_opj_j2k_setup_encoder.bmp

[carnil]

This was assigned CVE-2018-5785

[hlef]

The underlying issue is actually in src/bin/jp2/convertbmp.c: BMP files declaring compression = 3 (BI_BITFIELDS) with header size <= 56 keep their color mask values to zero. This is not an issue for BMP files with bit count 16 because of a check added in 16240e2 which sets default values to the color masks if they are all 0. We should maybe have something like that for other bit counts as well. Also, there might be some similar issues in the case where header size >= 56 + some masks are intentionally set to zero.

I'll investigate further and PR a patch.

[hlef]

Just submitted a PR addressing this issue.

[tmdahr1245]

Hello.

I would like to tell you about CVE-2018-5785 addressed in this issue.

I found this vulnerability in version 2.1.2, and I also confirmed that integer overflow occurs in that version.
When reproducing the exploit, the PoC above was used as it is and the reproduction process is the same.

We would appreciate it if you confirm the information and reflect it.
Thanks.