Many errors Cannot set the number of threads, already set

[benoit74]

While running khan academy, we have tons of errors like below.

Most videos are encoded probably but some fails with the error message Cannot set the number of threads, already set.

Looks like Python is reusing the same preset object under some conditions (don't know why it is not systematic, kind of a mystery for now).

Might be a candidate for a patch release.

[kolibri2zim::2024-02-21 09:19:05,307] DEBUG:Uploading cbfdbcf3bf9d0dcdbc5fcd85dfccfa1d.webm to S3::6/9/69f2e527a5ae4b0ba3fee3ba5c47696d/videowebmlow with {'checksum': 'cbfdbcf3bf9d0dcdbc5fcd85dfccfa1d', 'encoder_version': '2'}
[kolibri2zim::2024-02-21 09:21:13,797] DEBUG:Re-encoded 7523aed4b349183f87076647f13a574c.mp4 successfuly
[kolibri2zim::2024-02-21 09:21:13,799] DEBUG:Added 7523aed4b349183f87076647f13a574c.webm from re-encoded file
[kolibri2zim::2024-02-21 09:21:14,013] DEBUG:Uploading 5a5746fc83dc6158519740bca1ded6a5.webm to S3::3/8/389174634a2a4cddabec43c4ac10868e/videowebmlow with {'checksum': '5a5746fc83dc6158519740bca1ded6a5', 'encoder_version': '2'}
[kolibri2zim::2024-02-21 09:24:26,863] DEBUG:Re-encoded 6b6108557e4d4df717021e25e22da5f0.mp4 successfuly
[kolibri2zim::2024-02-21 09:24:26,864] DEBUG:Added 6b6108557e4d4df717021e25e22da5f0.webm from re-encoded file
[kolibri2zim::2024-02-21 09:24:26,932] DEBUG:Uploading 7523aed4b349183f87076647f13a574c.webm to S3::2/1/21f478c28cd44fd6b2916cde571ad8c5/videowebmlow with {'checksum': '7523aed4b349183f87076647f13a574c', 'encoder_version': '2'}
[kolibri2zim::2024-02-21 09:24:26,970] ERROR:Error re-encoding 548ff569e5a5c612a984b275cba193ac.mp4: Cannot set the number of threads, already set
[kolibri2zim::2024-02-21 09:24:26,971] ERROR:Cannot set the number of threads, already set
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/concurrent/futures/process.py", line 263, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/retrying.py", line 56, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/retrying.py", line 266, in call
    raise attempt.get()
          ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/retrying.py", line 301, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/local/lib/python3.12/site-packages/six.py", line 719, in reraise
    raise value
  File "/usr/local/lib/python3.12/site-packages/retrying.py", line 251, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
                      ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kolibri2zim/debug.py", line 55, in safer_reencode
    return reencode(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/zimscraperlib/video/encoding.py", line 61, in reencode
    args = _build_ffmpeg_args(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/zimscraperlib/video/encoding.py", line 23, in _build_ffmpeg_args
    raise AttributeError("Cannot set the number of threads, already set")
AttributeError: Cannot set the number of threads, already set
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/kolibri2zim/scraper.py", line 474, in video_conversion_completed
    future.result()
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
AttributeError: Cannot set the number of threads, already set

[benoit74]

Problem is that we use safer_reencode which retries the re-encoding. And this fails due to openzim/python-scraperlib#144