Tribler crashes with "Windows fatal exception: stack overflow" on a torrent with deep a directory structure

[kozlovsky]

In Tribler GUI tests, we have a test_add_deeptorrent test. It was added in PR #7015 as a check that Tribler does not crash with RecursionError if a torrent has deep nesting of directories.

Currently, in the main branch, the test is failing on Windows with another error:

Windows fatal exception: stack overflow

Current thread 0x00005e4c (most recent call first):
  File "C:\dev\tribler\src\tribler\gui\widgets\torrentfiletreewidget.py", line 253 in <lambda>
  File "C:\dev\tribler\src\tribler\gui\widgets\torrentfiletreewidget.py", line 72 in subtree
  File "C:\dev\tribler\src\tribler\gui\widgets\torrentfiletreewidget.py", line 252 in get_selected_items
  File "C:\dev\tribler\src\tribler\gui\widgets\torrentfiletreewidget.py", line 225 in fill_entries
  File "C:\dev\tribler\src\tribler\gui\dialogs\startdownloaddialog.py", line 198 in on_received_metainfo
  File "C:\dev\tribler\src\tribler\gui\utilities.py", line 411 in trackback_wrapper
  File "C:\dev\tribler\src\tribler\gui\tribler_request_manager.py", line 228 in on_finished
  File "C:\dev\tribler\src\tribler\gui\tribler_request_manager.py", line 122 in <lambda>
  File "C:\dev\tribler\src\tribler\gui\utilities.py", line 411 in trackback_wrapper
  File "C:\dev\tribler\src\tribler\gui\tests\test_gui.py", line 157 in wait_for_list_populated
  File "C:\dev\tribler\src\tribler\gui\tests\test_gui.py", line 442 in test_add_deeptorrent

The reason for the error is the following:

The test adds a torrent that contains a file nested into 10000 subdirectories. Tribler displays the torrent content using TorrentFileTreeWidget class inherited from QTreeWidget. A tree item is represented as an instance of a DownloadFileTreeWidgetItem class inherited from QTreeWidgetItem.

As it turns out, QTreeWidgetItem works very inefficiently if its items contain deeply nested sub-items. Some simple operations can take seconds. The actual error happens here (in the last line):

    def get_selected_items(self):
        selected_items = []
        for ind in range(self.topLevelItemCount()):
            item = self.topLevelItem(ind)
            for subitem in item.subtree(
                    filter_by=lambda x: x.checkState(CHECKBOX_COL) in (Qt.PartiallyChecked, Qt.Checked)

This x.checkState(CHECKBOX_COL) call for a top-level item is where the stack overflow happens. The code executed inside checkState is a C++ code that we cannot directly affect.

The previous fix in #7015 skips the recursive traversing of sub-items in our Python code to avoid provoking Python's RecursionError, but the actual number of created tree items in the test is still 10000, and that causes the crash in the C++ code of Qt. I did a quick test to determine the maximum level of directory nesting in a torrent file that does not provoke the stack overflow crash; on my Windows 64 machine, it is 6095 nested tree items. It is theoretically possible to increase that number by increasing the stack size. But in Python, it is impossible to specify the main thread's stack size without changing the compilation options, and the Qt GUI must run in the main thread. The default stack size on Windows 64 is 1 megabyte.

So, to fix the error, we should limit the actual number of nested items in TorrentFileTreeWidget. To do it, we can stop creating subdirectories at some level of nesting and add subdirectory/filename as a single item with file size. In that case, it will no longer be necessary to skip over nested items as #7015 does because there will be no more items in a tree with too deep nesting.