fileinfo crashes in fileinfo::CoffDetector::detectFileClass() due to infinite recursion

[bansan85]

fileinfo runs in infinity loop

Input

fileinfo FILE
infinity_loop.zip

Output

Backtrace:

#0  0x0000555555bfac64 in retdec::fileformat::FileFormat::getWordLength (this=0x555557047fb0) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/file_format.cpp:688
#1  0x0000555555baa6a6 in retdec::fileformat::CoffFormat::is32BitArchitecture (this=<optimized out>) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/coff/coff_format.cpp:750
#2  retdec::fileformat::CoffFormat::getBytesPerWord (this=<optimized out>) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/coff/coff_format.cpp:515
#3  0x0000555555bfacea in retdec::fileformat::FileFormat::getWordLength (this=0x555557047fb0) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/file_format.cpp:688
#4  0x0000555555baa6a6 in retdec::fileformat::CoffFormat::is32BitArchitecture (this=<optimized out>) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/coff/coff_format.cpp:750
#5  retdec::fileformat::CoffFormat::getBytesPerWord (this=<optimized out>) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/coff/coff_format.cpp:515
#6  0x0000555555bfacea in retdec::fileformat::FileFormat::getWordLength (this=0x555557047fb0) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/file_format.cpp:688
#7  0x0000555555baa6a6 in retdec::fileformat::CoffFormat::is32BitArchitecture (this=<optimized out>) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/coff/coff_format.cpp:750
#8  retdec::fileformat::CoffFormat::getBytesPerWord (this=<optimized out>) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/coff/coff_format.cpp:515
#9  0x0000555555bfacea in retdec::fileformat::FileFormat::getWordLength (this=0x555557047fb0) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/file_format.cpp:688
#10 0x0000555555baa6a6 in retdec::fileformat::CoffFormat::is32BitArchitecture (this=<optimized out>) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/coff/coff_format.cpp:750
#11 retdec::fileformat::CoffFormat::getBytesPerWord (this=<optimized out>) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/coff/coff_format.cpp:515
#12 0x0000555555bfacea in retdec::fileformat::FileFormat::getWordLength (this=0x555557047fb0) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/file_format.cpp:688
#13 0x0000555555baa6a6 in retdec::fileformat::CoffFormat::is32BitArchitecture (this=<optimized out>) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/coff/coff_format.cpp:750
#14 retdec::fileformat::CoffFormat::getBytesPerWord (this=<optimized out>) at /home/legarrec/info/programmation/retdec/src/fileformat/file_format/coff/coff_format.cpp:515
............

From master (63f1a3de)

[s3rvac]

Thank you for the report. I can confirm that fileinfo crashes due to infinite recursion between the following functions:

retdec::fileformat::CoffFormat::getBytesPerWord [..] fileformat/file_format/coff/coff_format.cpp:515
retdec::fileformat::FileFormat::getWordLength [..] fileformat/file_format/file_format.cpp:688
retdec::fileformat::CoffFormat::is32BitArchitecture [..] fileformat/file_format/coff/coff_format.cpp:750

They are called from

fileinfo::CoffDetector::detectFileClass [..] fileinfo/file_detector/coff_detector.cpp:366

The problem is that getBytesPerWord() calls is32BitArchitecture(), which calls getWordLength(), which again calls getBytesPerWord().

[mbandzi]

This happens only with (extremely rare) PowerPC and MIPS R4000 or as in this case broken binaries. I decided to break infinity recursion in function getBytesPerWord() by removing is32BitArchitecture() call for PowerPC and MIPS machines so is32BitArchitecture() can still relay on getBytesPerWord() function for all other cases. Applied in eb22bc3.

[bansan85]

Thanks for having solved my crashes.