[BugFix] Fix unwanted lazy stacks (#2102) #1618

Job	Run time
unittests-meltingpot / linux-job	10m 2s
unittests-pettingzoo / linux-job	9m 15s
unittests-atari-dqn (3.9, 12.1) / linux-job	13m 23s
unittests-brax (3.11, 12.1) / linux-job	8m 53s
unittests-d4rl (3.9, 12.1) / linux-job	8m 41s
unittests-envpool (3.9, 12.1) / linux-job	8m 37s
unittests-gendgrl (3.9, 12.1) / linux-job	6m 56s
unittests-gym (3.9, 12.1) / linux-job	19m 27s
unittests-jumanji (3.9, 12.1) / linux-job	8m 39s
unittests-minari (3.9, 12.1) / linux-job	5m 27s
unittests-openx (3.9, 12.1) / linux-job	26m 38s
unittests-robohive (3.9, 12.1) / linux-job	48m 32s
unittests-roboset (3.9, 12.1) / linux-job	9m 40s
unittests-sklearn (3.9, 12.1) / linux-job	5m 44s
unittests-smacv2 (3.9, 12.1) / linux-job	12m 33s
unittests-vd4rl (3.9, 12.1) / linux-job	9m 40s
unittests-vmas (3.9, 12.1) / linux-job	11m 40s
	3h 43m 47s

Provide feedback