Heap Buffer Overflow in smooth2() in cmsgamma.c

[Xuhxx]

Description

A heap buffer overflow vulnerability has been identified in thesmooth2() in cmsgamma.c in lcms2-2.16. The vulnerability occurs due to numerical instability in the calculation of array elements, which can lead to out-of-bounds memory access.

Result of ASAN:

==882101==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000038 at pc 0x556d323d0b73 bp 0x7fff522736c0 sp 0x7fff522736b8
READ of size 4 at 0x602000000038 thread T0
    #0 0x556d323d0b72 in smooth2 /home/CVE-lcms/project/lcms2.16/src/cmsgamma.c:1173:12
    #1 0x556d323cf72a in cmsSmoothToneCurve /home/CVE-lcms/project/lcms2.16/src/cmsgamma.c:1251:25
    #2 0x556d323c30a0 in main /home/CVE-lcms/harness/harness-1.cc:46:29
    #3 0x7fcfea7b0d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
    #4 0x7fcfea7b0e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
    #5 0x556d322e86d4 in _start (/home/CVE-lcms/harness/harness-1+0x776d4) (BuildId: f7451d7038c1b1f2)

0x602000000038 is located 0 bytes after 8-byte region [0x602000000030,0x602000000038)
allocated by thread T0 here:
    #0 0x556d323824fe in malloc (/home/CVE-lcms/harness/harness-1+0x1114fe) (BuildId: f7451d7038c1b1f2)
    #1 0x556d323d5a4d in _cmsMallocDefaultFn /home/CVE-lcms/project/lcms2.16/src/cmserr.c:98:20
    #2 0x556d323d590d in _cmsMalloc /home/CVE-lcms/project/lcms2.16/src/cmserr.c:268:12
    #3 0x556d323d5a7a in _cmsMallocZeroDefaultFn /home/CVE-lcms/project/lcms2.16/src/cmserr.c:107:16
    #4 0x556d323d5f51 in _cmsMallocZero /home/CVE-lcms/project/lcms2.16/src/cmserr.c:275:12
    #5 0x556d323d5d8b in _cmsCallocDefaultFn /home/CVE-lcms/project/lcms2.16/src/cmserr.c:161:12
    #6 0x556d323d6147 in _cmsCalloc /home/CVE-lcms/project/lcms2.16/src/cmserr.c:282:12
    #7 0x556d323cf2f4 in cmsSmoothToneCurve /home/CVE-lcms/project/lcms2.16/src/cmsgamma.c:1229:41
    #8 0x556d323c30a0 in main /home/CVE-lcms/harness/harness-1.cc:46:29
    #9 0x7fcfea7b0d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
    #10 0x7fcfea7b0e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
    #11 0x556d322e86d4 in _start (/home/CVE-lcms/harness/harness-1+0x776d4) (BuildId: f7451d7038c1b1f2)

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/CVE-lcms/project/lcms2.16/src/cmsgamma.c:1173:12 in smooth2
Shadow bytes around the buggy address:
  0x601ffffffd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x601ffffffe00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x601ffffffe80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x601fffffff00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x601fffffff80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x602000000000: fa fa 02 fa fa fa 00[fa]fa fa 00 fa fa fa 00 fa
  0x602000000080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x602000000100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x602000000180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x602000000200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x602000000280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==882101==ABORTING

Affected Version

lcms2-2.16

POC

Poc-smooth2.zip
You can reproduce the vulnerability with:
./harness input

Analysis

The vulnerability is triggered in the following code sequence:

Little-CMS/src/cmsgamma.c

Lines 1173 to 1174 in 453bafe

	d[2] = w[2] + 5 * lambda - d[1] * c[1] * c[1];
	c[2] = (-4 * lambda - d[1] * c[1] * e[1]) / d[2];

The issue arises when:

1. The calculation of d[2] results in a value very close to zero or zero itself
2. This near-zero value is then used as a divisor in calculating c[2]
3. The division operation with a near-zero divisor produces an extremely large value for c[2]
4. This large value leads to buffer overflow in subsequent array accesses

[mm2]

Please submit source code of the caller program.

[Xuhxx]

Thank you for comment.
And sorry to report the unconfirmed issue as a vulnerability, it is not appropriate.
Here is my harness code:

int main(int argc, char *argv[]) {
    if (argc < 2) {
        fprintf(stderr, "Usage: %s <input_file>\n", argv[0]);
        return 1;
    }

    FILE *fp = fopen(argv[1], "rb");
    if (!fp) return 2;
    
    fseek(fp, 0, SEEK_END);
    long size = ftell(fp);
    fseek(fp, 0, SEEK_SET);
    
    uint8_t *data = (uint8_t*)malloc(size);
    fread(data, 1, size, fp);
    fclose(fp);

    const size_t min_size = sizeof(float) + sizeof(double) + sizeof(uint32_t);
    if (size < min_size) {
        free(data);
        return 0;
    }

    float input_value;
    double lambda;
    uint32_t table_entries;
    
    memcpy(&input_value, data, sizeof(float));
    memcpy(&lambda, data + sizeof(float), sizeof(double));
    memcpy(&table_entries, data + sizeof(float) + sizeof(double), sizeof(uint32_t));

    cmsToneCurve* curve = cmsBuildTabulatedToneCurve16(NULL, table_entries, NULL);
    if (!curve) {
        free(data);
        return 0;
    }

    cmsBool smooth_result = cmsSmoothToneCurve(curve, lambda);
    cmsFreeToneCurve(curve);
    free(data);
    
    return !smooth_result;

Thanks for any follow-up advice!

[mm2]

Thanks, this is not exploitable as this function is never called on normal color management, is there only as a helper for low-level programming and investigation.

However, I am adding a code guard for the very unlikely mistake of somebody trying to smooth a table of only one element.
Thanks for discovering the glitch.