Can we disable multiprocessing in CodeQL for AWS Lambda compatibility?

[AmarBhosale]

Our internal security scanner orchestrator requires that each tool runs within its own AWS Lambda function. However, we're facing issues with running CodeQL, particularly for Python, because CodeQL's Python extractor relies on multiprocessing, which apparently isn't supported in AWS Lambda. Is there a way to disable multiprocessing in CodeQL to allow it to function properly in the AWS Lambda environment?

Additional information on the error we are getting:

Initializing database at /tmp/codeql-db.
Running build command: []
Running command in /tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182: [/opt/codeql/python/tools/autobuild.sh]
[2024-09-13 02:00:06] [build-stderr] /bin/sh: line 1: python2: command not found
[2024-09-13 02:00:06] [build-stdout] No directories containing root identifiers were found. Returning working directory as root.
[2024-09-13 02:00:06] [build-stdout] Will try to guess Python version, as it was not specified in `lgtm.yml`
[2024-09-13 02:00:06] [build-stdout] Trying to guess Python version based on Trove classifiers in setup.py
[2024-09-13 02:00:06] [build-stdout] Did not find setup.py (expected it to be at /tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182/setup.py)
[2024-09-13 02:00:06] [build-stdout] Trying to guess Python version based on travis file
[2024-09-13 02:00:06] [build-stdout] Did not find any travis files (expected them at either ['/tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182/.travis.yml', '/tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182/travis.yml'])
[2024-09-13 02:00:06] [build-stdout] Trying to guess Python version based on installed versions
[2024-09-13 02:00:06] [build-stdout] Wanted to run Python 2, but it is not available. Using Python 3 instead
[2024-09-13 02:00:06] [build-stdout] This script is running Python 3, but Python 2 is also available (as 'python3')
[2024-09-13 02:00:06] [build-stdout] Could not guess Python version, will use default: Python 3
[2024-09-13 02:00:06] [build-stdout] Calling python3 -S /opt/codeql/python/tools/python_tracer.py --verbosity 3 -z all -c /tmp/codeql-db/working/trap_cache -R /tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182
[2024-09-13 02:00:08] [build-stderr] Traceback (most recent call last):
[2024-09-13 02:00:08] [build-stderr]   File "/opt/codeql/python/tools/python_tracer.py", line 53, in <module>
[2024-09-13 02:00:08] [build-stderr]     semmle.populator.main(original_path)
[2024-09-13 02:00:08] [build-stderr]   File "/opt/codeql/python/tools/python3src.zip/semmle/populator.py", line 60, in main
[2024-09-13 02:00:08] [build-stderr]   File "/opt/codeql/python/tools/python3src.zip/semmle/logging.py", line 85, in __init__
[2024-09-13 02:00:08] [build-stderr]   File "/usr/lib64/python3.9/multiprocessing/context.py", line 103, in Queue
[2024-09-13 02:00:08] [build-stderr]     return Queue(maxsize, ctx=self.get_context())
[2024-09-13 02:00:08] [build-stderr]   File "/usr/lib64/python3.9/multiprocessing/queues.py", line 43, in __init__
[2024-09-13 02:00:08] [build-stderr]     self._rlock = ctx.Lock()
[2024-09-13 02:00:08] [build-stderr]   File "/usr/lib64/python3.9/multiprocessing/context.py", line 68, in Lock
[2024-09-13 02:00:08] [build-stderr]     return Lock(ctx=self.get_context())
[2024-09-13 02:00:08] [build-stderr]   File "/usr/lib64/python3.9/multiprocessing/synchronize.py", line 162, in __init__
[2024-09-13 02:00:08] [build-stderr]     SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
[2024-09-13 02:00:08] [build-stderr]   File "/usr/lib64/python3.9/multiprocessing/synchronize.py", line 57, in __init__
[2024-09-13 02:00:08] [build-stderr]     sl = self._semlock = _multiprocessing.SemLock(
[2024-09-13 02:00:08] [build-stderr] FileNotFoundError: [Errno 2] No such file or directory
[2024-09-13 02:00:08] [build-stderr] Traceback (most recent call last):
[2024-09-13 02:00:08] [build-stderr]   File "/opt/codeql/python/tools/index.py", line 28, in <module>
[2024-09-13 02:00:08] [build-stderr]     buildtools.index.main()
[2024-09-13 02:00:08] [build-stderr]   File "/opt/codeql/python/tools/python3src.zip/buildtools/index.py", line 409, in main
[2024-09-13 02:00:08] [build-stderr]   File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
[2024-09-13 02:00:08] [build-stderr]     raise CalledProcessError(retcode, cmd)
[2024-09-13 02:00:08] [build-stderr] subprocess.CalledProcessError: Command '['python3', '-S', '/opt/codeql/python/tools/python_tracer.py', '--verbosity', '3', '-z', 'all', '-c', '/tmp/codeql-db/working/trap_cache', '-R', '/tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182']' returned non-zero exit status 1.
[2024-09-13 02:00:08] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/opt/codeql/python/tools/autobuild.sh])
A fatal error occurred: Exit status 1 from command: [/opt/codeql/python/tools/autobuild.sh]

[jketema]

Hi @AmarBhosale,

Unfortunately it is not possible to disable multiprocessing.