Snakemake 8.25.2/8.27 bug running scripts (possible bug from .snakemake/scripts script generation/usage)

[cachedpotato]

Snakemake version

Vesion: Tested on 8.25.2 and 8.27.1

Describe the bug

I have a rule that runs a python script (so I use script: ... instead of shell: ...) The script uses tensorflow, and I got an error related to importing one of the models. at first I thought it was a TF error, but I realised that if I explicitly run what snakemake tried to do (the command shown in the log file), I don't get any errors.

I think the error has something to do with snakemake generating temporary scripts and running it from there. I'd like to know why this happens or at the very least an option to keep that temporary script that gets generated inside the .snakemake/scripts directory so that I can see for myself.

Logs

Activating conda environment: .snakemake/conda/b583972d43d752c0abf5c4821488ac74_
Activating conda environment: .snakemake/conda/b583972d43d752c0abf5c4821488ac74_
3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0]
Traceback (most recent call last):
File "/pipeline/debug/.snakemake/scripts/tmp_j84s87z.script.py", line 7, in
from tensorflow.keras.models import load_model
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74_/lib/python3.7/site-packages/tensorflow/init.py", line 99, in
from tensorflow_core import *
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74_/lib/python3.7/site-packages/tensorflow_core/init.py", line 34, in
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File "", line 1019, in handle_fromlist
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74/lib/python3.7/site-packages/tensorflow/init.py", line 50, in getattr
module = self.load()
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74/lib/python3.7/site-packages/tensorflow/init.py", line 44, in load
module = importlib.import_module(self.name)
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74/lib/python3.7/importlib/init.py", line 127, in import_module
return bootstrap.gcd_import(name[level:], package, level)
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74/lib/python3.7/site-packages/tensorflow_core/python/init.py", line 83, in
from tensorflow.python import keras
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74/lib/python3.7/site-packages/tensorflow_core/python/keras/init.py", line 26, in
from tensorflow.python.keras import activations
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74/lib/python3.7/site-packages/tensorflow_core/python/keras/init.py", line 26, in
from tensorflow.python.keras import activations
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74_/lib/python3.7/site-packages/tensorflow_core/python/keras/activations.py", line 23, in
from tensorflow.python.keras.utils.generic_utils import deserialize_keras_object
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74_/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/init.py", line 38, in
from tensorflow.python.keras.utils.multi_gpu_utils import multi_gpu_model
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74_/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/multi_gpu_utils.py", line 22, in
from tensorflow.python.keras.engine.training import Model
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74_/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/init.py", line 24, in
from tensorflow.python.keras.engine.input_layer import Input
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74_/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/input_layer.py", line 25, in
from tensorflow.python.keras.distribute import distributed_training_utils
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74_/lib/python3.7/site-packages/tensorflow_core/python/keras/distribute/distributed_training_utils.py", line 38, in
from tensorflow.python.keras import callbacks
File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74_/lib/python3.7/site-packages/tensorflow_core/python/keras/callbacks.py", line 55, in
import requests
File "/miniconda/lib/python3.11/site-packages/requests/init.py", line 43, in
import urllib3
File "/miniconda/lib/python3.11/site-packages/urllib3/init.py", line 14, in
from . import exceptions
File "/miniconda/lib/python3.11/site-packages/urllib3/exceptions.py", line 26, in
TYPE_REDUCE_RESULT = tuple[typing.Callable[..., object], tuple[object, ...]]
TypeError: 'type' object is not subscriptable
RuleException:
CalledProcessError in file /pipeline/debug/snaketest.smk, line 25:
Command 'source /miniconda/bin/activate '/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74'; set -euo pipefail; python /pipeline/debug/.snakemake/scripts/tmp_j84s87z.script.py' returned non-zero exit status 1.
[Fri Feb 14 05:12:04 2025]

Minimal example

The rule I used looks likethe following:
rule test:
log:
"/pipeline/test/log.log",
input:
in = ...
output:
out = ...
conda:
"/pipeline/workflow/envs/conda_env.yaml",
script:
"/pipeline/workflow/scripts/script.py"

Inside the python file I have this line:
from tensorflow.keras.models import load_model

I ran the rule using:
snakemake -c 1 --use-conda -s test.smk

Additional context

Like I said, I tried to run the command which is in the log:
'source /miniconda/bin/activate '/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74_'; set -euo pipefail; python /pipeline/debug/.snakemake/scripts/tmp_j84s87z.script.py'
The only change I made was change the tmp script with the original and substitute any wildcards that was used to actual file names, and it ran perfectly fine.

[cademirch]

This looks related to the Python version being used in your conda env for the rule. Can you try using a newer Python that 3.7? >3.10 would be a good place to start.

[cachedpotato]

The conda env that I'm using for the rule requires that I use Python < 3.8 (TF 1.15). It'd be great if I could somehow solve this without upgrading my python version, as for that I'd have to upgrade TF as well and I'm not really in a position to do that atm. Currently using a workaround by using shell instead of script (and parsing the arguments manually) but any other solution/recommendation would be greatly appreciated.

Thanks!

[cademirch]

Looking at your traceback again:

File "/pipeline/debug/.snakemake/conda/b583972d43d752c0abf5c4821488ac74_/lib/python3.7/site-packages/tensorflow_core/python/keras/callbacks.py", line 55, in
import requests
File "/miniconda/lib/python3.11/site-packages/requests/init.py", line 43, in
import urllib3
File "/miniconda/lib/python3.11/site-packages/urllib3/init.py", line 14, in
from . import exceptions

I'm not sure this is entirely Snakemake's fault, but rather a conda/python issue. The type error is the result of TF importing requests, which for some reason is importing urllib3 from your base env, which is Python 3.11. What does your tensorflow env yaml look like? Also are you running Snakemake from a conda install? If so is it installed in its own env or in the base env?