Potential segmentation fault due to incorrect realloc in CABAC_encoder_bitstream::check_size_and_resize

[litios]

Summary

There is a segmentation fault in CABAC_encoder_bitstream::append_byte and CABAC_encoder_bitstream::write_startcode if the call to realloc fails in check_size_and_resize

Tested with:

• Commit: 6fc550f (master)
• PoC: any YUV input file.
• cmd: ./enc265/enc265 -i <yuv file>

Analysis

When executing CABAC_encoder_bitstream::check_size_and_resize, there is no check to ensure the call to realloc is successful.

In case it is not, it will return NULL that will be set as the value of data_mem. Later on, the variable is accessed in CABAC_encoder_bitstream::append_byte:

  if (byte<=3) {
    /**/ if (state< 2 && byte==0) { state++; }
    else if (state==2 && byte<=3) {
      data_mem[ data_size++ ] = 3;

      if (byte==0) state=1;
      else         state=0;
    }
    else { state=0; }
  }
  else { state=0; }


  // write actual data byte

  data_mem[ data_size++ ] = byte;

and CABAC_encoder_bitstream::write_startcode:

  data_mem[ data_size+0 ] = 0;
  data_mem[ data_size+1 ] = 0;
  data_mem[ data_size+2 ] = 1;
  data_size+=3;

resulting in a segmentation fault due to trying to access a NULL pointer.

Impact

• Crash

Patch

In order to prevent a crash, a check for data_mem not being NULL should be added that can handle the case appropriately.

[farindk]

Thanks. I have fixed the potential crash.
Note that this code is used only in the encoder part and that is not used anywhere.