cudaPackages.backendStdenv: mismatch with stdenv causes libstdc++ errors #220341

ConnorBaker · 2023-03-09T14:23:03Z

Describe the bug

Per #218265 (comment), when using CUDA-enabled scipy it is possible to run into libstdc++.so.6: version 'GLIBCXX_3.4.30' not found errors.

Historically the fix for this problem has been to ensure the derivations of CUDA-enabled packages use a compiler compatible with the version of NVCC being used, instead of the default compiler provided by stdenv.

Additional context

This is a followup from #218265.

Notify maintainers

@NixOS/cuda-maintainers

The text was updated successfully, but these errors were encountered:

blurgyy · 2023-03-10T16:26:30Z

I've encountered similar issues while trying to use jax and optax at 1e383aa. With cudaSupport enabled, jax fails 6 tests at pytestCheckPhase and can be worked around via setting doCheck = false; in overridePythonAttrs, while optax fails at pythonImportsCheckPhase, overriding with pythonImportsCheck = [] makes the build succeed but cannot import in the built environment (expected since it did not pass the pythonImportsCheckPhase).

samuela · 2023-03-10T18:09:24Z

Hi @blurgyy can you post the error logs from these failures?

bcdarwin · 2023-03-10T19:44:44Z

This scipy issue seemingly also breaks python310Packages.grad-cam with CUDA enabled during the pythonImportsCheck phase (full log):

Check whether the following modules can be imported: pytorch_grad_cam pytorch_grad_cam.metrics pytorch_grad_cam.metrics.cam_mult_image pytorch_grad_cam.metrics.road pytorch_grad_cam.utils pytorch_grad_cam.utils.image pytorch_grad_cam.utils.model_targets
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <lambda>
  File "/nix/store/iw1vmh509hcbby8dbpsaanbri4zsq7dj-python3-3.10.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/nix/store/ys8jbngkrfvrfkfyzqqpn7zldbhd0107-python3.10-grad-cam-1.4.6/lib/python3.10/site-packages/pytorch_grad_cam/__init__.py", line 16, in <module>
    from pytorch_grad_cam.feature_factorization.deep_feature_factorization import DeepFeatureFactorization, run_dff_on_image
  File "/nix/store/ys8jbngkrfvrfkfyzqqpn7zldbhd0107-python3.10-grad-cam-1.4.6/lib/python3.10/site-packages/pytorch_grad_cam/feature_factorization/deep_feature_factorization.py", line 5, in <module>
    from sklearn.decomposition import NMF
  File "/nix/store/vgyqalfrz6lx0967l651ms0hvsglhky2-python3.10-scikit-learn-1.2.1/lib/python3.10/site-packages/sklearn/__init__.py", line 82, in <module>
    from .base import clone
  File "/nix/store/vgyqalfrz6lx0967l651ms0hvsglhky2-python3.10-scikit-learn-1.2.1/lib/python3.10/site-packages/sklearn/base.py", line 17, in <module>
    from .utils import _IS_32BIT
  File "/nix/store/vgyqalfrz6lx0967l651ms0hvsglhky2-python3.10-scikit-learn-1.2.1/lib/python3.10/site-packages/sklearn/utils/__init__.py", line 25, in <module>
    from .fixes import parse_version, threadpool_info
  File "/nix/store/vgyqalfrz6lx0967l651ms0hvsglhky2-python3.10-scikit-learn-1.2.1/lib/python3.10/site-packages/sklearn/utils/fixes.py", line 19, in <module>
    import scipy.stats
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/stats/__init__.py", line 467, in <module>
    from ._stats_py import *
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/stats/_stats_py.py", line 46, in <module>
    from . import distributions
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/stats/distributions.py", line 8, in <module>
    from ._distn_infrastructure import (rv_discrete, rv_continuous, rv_frozen)
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/stats/_distn_infrastructure.py", line 24, in <module>
    from scipy import optimize
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/__init__.py", line 211, in __getattr__
    return _importlib.import_module(f'scipy.{name}')
  File "/nix/store/iw1vmh509hcbby8dbpsaanbri4zsq7dj-python3-3.10.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/__init__.py", line 413, in <module>
    from ._linprog import linprog, linprog_verbose_callback
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_linprog.py", line 21, in <module>
    from ._linprog_highs import _linprog_highs
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_linprog_highs.py", line 20, in <module>
    from ._highs._highs_wrapper import _highs_wrapper
ImportError: /nix/store/205vsmbfhq1q2vhgskpqyymqvba4mscp-gcc-11.3.0-lib/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_highs/_highs_wrapper.cpython-310-x86_64-linux-gnu.so)
error: builder for '/nix/store/2klb2r6j9lz9vx7wzlmgmzm9060020bq-python3.10-grad-cam-1.4.6.drv' failed with exit code 1

blurgyy · 2023-03-11T03:00:51Z

Hi @blurgyy can you post the error logs from these failures?

Sure @samuela, here are the logs:

jax

flake.nix:

{
  description = "Python environment with jax";

  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";

  outputs = { self, nixpkgs, flake-utils, ... }: let
    system = "x86_64-linux";
    pkgs = import nixpkgs {
      inherit system;
      config = {
        allowUnfree = true;
        cudaSupport = true;
      };
    };
  in {
    devShells.${system} = {
      default = pkgs.mkShell {  # impure
        buildInputs = with pkgs; [
          (python3.withPackages (pp: with pp; [
              jax
            ]))
        ];
      };
    };
  };
}

log (last 100 lines):

/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/fft/_helper.py:3: in <module>
    from ._pocketfft import helper as _helper
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/fft/_pocketfft/__init__.py:3: in <module>
    from .basic import *
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/fft/_pocketfft/basic.py:6: in <module>
    from . import pypocketfft as pfft
E   ImportError: /nix/store/205vsmbfhq1q2vhgskpqyymqvba4mscp-gcc-11.3.0-lib/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-310-x86_64-linux-gnu.so)
________________ ERROR collecting tests/scipy_optimize_test.py _________________
ImportError while importing test module '/build/source/tests/scipy_optimize_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/nix/store/iw1vmh509hcbby8dbpsaanbri4zsq7dj-python3-3.10.10/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/scipy_optimize_test.py:18: in <module>
    import scipy.optimize
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/__init__.py:413: in <module>
    from ._linprog import linprog, linprog_verbose_callback
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_linprog.py:21: in <module>
    from ._linprog_highs import _linprog_highs
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_linprog_highs.py:20: in <module>
    from ._highs._highs_wrapper import _highs_wrapper
E   ImportError: /nix/store/205vsmbfhq1q2vhgskpqyymqvba4mscp-gcc-11.3.0-lib/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_highs/_highs_wrapper.cpython-310-x86_64-linux-gnu.so)
_________________ ERROR collecting tests/scipy_signal_test.py __________________
ImportError while importing test module '/build/source/tests/scipy_signal_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/nix/store/iw1vmh509hcbby8dbpsaanbri4zsq7dj-python3-3.10.10/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/scipy_signal_test.py:22: in <module>
    import scipy.signal as osp_signal
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/signal/__init__.py:309: in <module>
    from . import _sigtools, windows
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/signal/windows/__init__.py:41: in <module>
    from ._windows import *
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/signal/windows/_windows.py:7: in <module>
    from scipy import linalg, special, fft as sp_fft
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/__init__.py:211: in __getattr__
    return _importlib.import_module(f'scipy.{name}')
/nix/store/iw1vmh509hcbby8dbpsaanbri4zsq7dj-python3-3.10.10/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/fft/__init__.py:92: in <module>
    from ._helper import next_fast_len
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/fft/_helper.py:3: in <module>
    from ._pocketfft import helper as _helper
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/fft/_pocketfft/__init__.py:3: in <module>
    from .basic import *
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/fft/_pocketfft/basic.py:6: in <module>
    from . import pypocketfft as pfft
E   ImportError: /nix/store/205vsmbfhq1q2vhgskpqyymqvba4mscp-gcc-11.3.0-lib/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-310-x86_64-linux-gnu.so)
__________________ ERROR collecting tests/scipy_stats_test.py __________________
ImportError while importing test module '/build/source/tests/scipy_stats_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/nix/store/iw1vmh509hcbby8dbpsaanbri4zsq7dj-python3-3.10.10/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/scipy_stats_test.py:22: in <module>
    import scipy.stats as osp_stats
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/stats/__init__.py:467: in <module>
    from ._stats_py import *
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/stats/_stats_py.py:46: in <module>
    from . import distributions
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/stats/distributions.py:8: in <module>
    from ._distn_infrastructure import (rv_discrete, rv_continuous, rv_frozen)
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/stats/_distn_infrastructure.py:24: in <module>
    from scipy import optimize
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/__init__.py:211: in __getattr__
    return _importlib.import_module(f'scipy.{name}')
/nix/store/iw1vmh509hcbby8dbpsaanbri4zsq7dj-python3-3.10.10/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/__init__.py:413: in <module>
    from ._linprog import linprog, linprog_verbose_callback
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_linprog.py:21: in <module>
    from ._linprog_highs import _linprog_highs
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_linprog_highs.py:20: in <module>
    from ._highs._highs_wrapper import _highs_wrapper
E   ImportError: /nix/store/205vsmbfhq1q2vhgskpqyymqvba4mscp-gcc-11.3.0-lib/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_highs/_highs_wrapper.cpython-310-x86_64-linux-gnu.so)
_________ ERROR collecting tests/third_party/scipy/line_search_test.py _________
ImportError while importing test module '/build/source/tests/third_party/scipy/line_search_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/nix/store/iw1vmh509hcbby8dbpsaanbri4zsq7dj-python3-3.10.10/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/third_party/scipy/line_search_test.py:2: in <module>
    import scipy.optimize
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/__init__.py:413: in <module>
    from ._linprog import linprog, linprog_verbose_callback
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_linprog.py:21: in <module>
    from ._linprog_highs import _linprog_highs
/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_linprog_highs.py:20: in <module>
    from ._highs._highs_wrapper import _highs_wrapper
E   ImportError: /nix/store/205vsmbfhq1q2vhgskpqyymqvba4mscp-gcc-11.3.0-lib/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_highs/_highs_wrapper.cpython-310-x86_64-linux-gnu.so)
=========================== short test summary info ============================
ERROR tests/ode_test.py
ERROR tests/scipy_fft_test.py
ERROR tests/scipy_optimize_test.py
ERROR tests/scipy_signal_test.py
ERROR tests/scipy_stats_test.py
ERROR tests/third_party/scipy/line_search_test.py
========== 10644 passed, 1085 skipped, 6 errors in 1193.43s (0:19:53) ==========
/nix/store/c3f4jdwzn8fm9lp72m91ffw524bakp6v-stdenv-linux/setup: line 1593: pop_var_context: head of shell_variables not a function context

Full log: jax.log

optax

flake.nix:

{
  description = "Python environment with jax";

  inputs.nixpkgs.url = "github:nixos/nixpkgs/1e383aada51b416c6c27d4884d2e258df201bc11";

  outputs = { self, nixpkgs, flake-utils, ... }: let
    system = "x86_64-linux";
    pkgs = import nixpkgs {
      inherit system;
      config = {
        allowUnfree = true;
        cudaSupport = true;
      };
    };
  in {
    devShells.${system} = {
      default = pkgs.mkShell {  # impure
        buildInputs = with pkgs; [
          (python3.withPackages (pp: with pp; let
            jax-nocheck = jax.overridePythonAttrs (o: {
                doCheck = false;
              });
            chex-fixed = chex.override {
              jax = jax-nocheck;
            };
          in [
              (optax.override { chex = chex-fixed; })
            ]))
        ];
      };
    };
  };
}

log (last 50 lines):

@nix { "action": "setPhase", "phase": "pythonRemoveBinBytecodePhase" }
pythonRemoveBinBytecodePhase
@nix { "action": "setPhase", "phase": "pythonImportsCheckPhase" }
pythonImportsCheckPhase
Executing pythonImportsCheckPhase
Check whether the following modules can be imported: optax
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <lambda>
  File "/nix/store/iw1vmh509hcbby8dbpsaanbri4zsq7dj-python3-3.10.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/nix/store/qbwaq1sq7ihlyz2syw2gamvfpx1xjk2w-python3.10-optax-0.1.4/lib/python3.10/site-packages/optax/__init__.py", line 18, in <module>
    from optax._src.alias import adabelief
  File "/nix/store/qbwaq1sq7ihlyz2syw2gamvfpx1xjk2w-python3.10-optax-0.1.4/lib/python3.10/site-packages/optax/_src/alias.py", line 24, in <module>
    from optax._src import factorized
  File "/nix/store/qbwaq1sq7ihlyz2syw2gamvfpx1xjk2w-python3.10-optax-0.1.4/lib/python3.10/site-packages/optax/_src/factorized.py", line 27, in <module>
    from optax._src import utils
  File "/nix/store/qbwaq1sq7ihlyz2syw2gamvfpx1xjk2w-python3.10-optax-0.1.4/lib/python3.10/site-packages/optax/_src/utils.py", line 22, in <module>
    import jax.scipy.stats.norm as multivariate_normal
  File "/nix/store/0vbh981q4bzf58bhqhc5kjr9k8hlv4kn-python3.10-jax-0.4.1/lib/python3.10/site-packages/jax/scipy/stats/__init__.py", line 15, in <module>
    from jax.scipy.stats import bernoulli as bernoulli
  File "/nix/store/0vbh981q4bzf58bhqhc5kjr9k8hlv4kn-python3.10-jax-0.4.1/lib/python3.10/site-packages/jax/scipy/stats/bernoulli.py", line 15, in <module>
    from jax._src.scipy.stats.bernoulli import (
  File "/nix/store/0vbh981q4bzf58bhqhc5kjr9k8hlv4kn-python3.10-jax-0.4.1/lib/python3.10/site-packages/jax/_src/scipy/stats/bernoulli.py", line 16, in <module>
    import scipy.stats as osp_stats
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/stats/__init__.py", line 467, in <module>
    from ._stats_py import *
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/stats/_stats_py.py", line 46, in <module>
    from . import distributions
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/stats/distributions.py", line 8, in <module>
    from ._distn_infrastructure import (rv_discrete, rv_continuous, rv_frozen)
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/stats/_distn_infrastructure.py", line 24, in <module>
    from scipy import optimize
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/__init__.py", line 211, in __getattr__
    return _importlib.import_module(f'scipy.{name}')
  File "/nix/store/iw1vmh509hcbby8dbpsaanbri4zsq7dj-python3-3.10.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/__init__.py", line 413, in <module>
    from ._linprog import linprog, linprog_verbose_callback
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_linprog.py", line 21, in <module>
    from ._linprog_highs import _linprog_highs
  File "/nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_linprog_highs.py", line 20, in <module>
    from ._highs._highs_wrapper import _highs_wrapper
ImportError: /nix/store/205vsmbfhq1q2vhgskpqyymqvba4mscp-gcc-11.3.0-lib/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/yi7jc5p2mlwb3j37j7gwj15bk45j6xqs-python3.10-scipy-1.9.3/lib/python3.10/site-packages/scipy/optimize/_highs/_highs_wrapper.cpython-310-x86_64-linux-gnu.so)

Full log: optax.log

edit:
I just noticed in jax's test error logs that 5 of 6 errors are related to SciPy, and optax's import error also traces to SciPy (didn't notice these when I got the errors or when I posted the logs here).

nixos-discourse · 2023-03-16T12:43:40Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-45/26397/1

SomeoneSerge · 2023-03-16T14:18:10Z

Ok, so I've no idea what's happening, but I'm looking at:

E   ImportError: /nix/store/yiflcg7zmirny3654g8l8f85sz958gqk-gcc-11.3.0-lib/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/hvs05k932v31cfckv49nx1pin0qjd20v-python3.10-scipy-1.10.1/lib/python3.10/site-packages/scipy/optimize/_highs/_highs_wrapper.cpython-310-x86_64-linux-gnu.so)

This appears to mean that _highs_wrapper.whatever wants libstdc++ 3.4.30, and tried to look it up in gcc11's libstdc++ which hasn't got it:

❯ objdump -T /nix/store/yiflcg7zmirny3654g8l8f85sz958gqk-gcc-11.3.0-lib/lib/libstdc++.so.6 | rg 'GLIBCXX_3.4.30
❯ objdump -T '/nix/store/hvs05k932v31cfckv49nx1pin0qjd20v-python3.10-scipy-1.10.1/lib/python3.10/site-packages/scipy/optimize/_highs/_highs_wrapper.cpython-310-x86_64-linux-gnu.so' | rg GLIBCXX_3.4.3
0000000000000000      DF *UND*  0000000000000000 (GLIBCXX_3.4.30) _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE

It would seem that libstdc++ is DT_NEEDED for scipy-whatever-wrapper:

❯ patchelf --print-needed '/nix/store/hvs05k932v31cfckv49nx1pin0qjd20v-python3.10-scipy-1.10.1/lib/python3.10/site-packages/scipy/optimize/_highs/_highs_wrapper.cpython-310-x86_64-linux-gnu.so'
libstdc++.so.6
libm.so.6
libgcc_s.so.1
libc.so.6
ld-linux-x86-64.so.2

And it expects to get it from gfortran, idk why:

❯ patchelf --print-rpath '/nix/store/hvs05k932v31cfckv49nx1pin0qjd20v-python3.10-scipy-1.10.1/lib/python3.10/site-packages/scipy/optimize/_highs/_highs_wrapper.cpython-310-x86_64-linux-gnu.so'
/nix/store/8bmp6r3a0xfha3wj36phlc47clh9w81l-glibc-2.35-224/lib:/nix/store/zmln2p5q1swj2dxqbzq7rcnm74wa584v-gfortran-12.2.0-lib/lib

We can see that gfortran's libstdc++ has the looked for symbols:

❯ objdump -T /nix/store/zmln2p5q1swj2dxqbzq7rcnm74wa584v-gfortran-12.2.0-lib/lib/libstdc++.so | rg GLIBCXX_3.4.30
0000000000000000 g    DO *ABS*  0000000000000000  GLIBCXX_3.4.30 GLIBCXX_3.4.30
000000000015d930  w   DF .text  000000000000186b  GLIBCXX_3.4.30 _ZNKSt8time_getIwSt19istreambuf_iteratorIwSt11char_traitsIwEEE21_M_extract_via_formatES3_S3_RSt8ios_baseRSt12_Ios_IostateP2tmPKwRSt16__time_get_state
00000000000d9f00 g    DF .text  000000000000000c  GLIBCXX_3.4.30 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE
0000000000110bc0  w   DF .text  0000000000001e83  GLIBCXX_3.4.30 _ZNKSt7__cxx118time_getIwSt19istreambuf_iteratorIwSt11char_traitsIwEEE21_M_extract_via_formatES4_S4_RSt8ios_baseRSt12_Ios_IostateP2tmPKwRSt16__time_get_state
0000000000136b60  w   DF .text  00000000000017b2  GLIBCXX_3.4.30 _ZNKSt8time_getIcSt19istreambuf_iteratorIcSt11char_traitsIcEEE21_M_extract_via_formatES3_S3_RSt8ios_baseRSt12_Ios_IostateP2tmPKcRSt16__time_get_state
000000000012abb0  w   DF .text  0000000000000010  GLIBCXX_3.4.30 _ZNKSt11__timepunctIcE15_M_am_pm_formatEPPKc
00000000000dca50 g    DF .text  000000000000005e  GLIBCXX_3.4.30 _ZSt21__glibcxx_assert_failPKciS0_S0_
0000000000105c80  w   DF .text  0000000000002285  GLIBCXX_3.4.30 _ZNKSt7__cxx118time_getIcSt19istreambuf_iteratorIcSt11char_traitsIcEEE21_M_extract_via_formatES4_S4_RSt8ios_baseRSt12_Ios_IostateP2tmPKcRSt16__time_get_state
00000000000c8180 g    DF .text  000000000000043e  GLIBCXX_3.4.30 _ZNSt16__time_get_state17_M_finalize_stateEP2tm
0000000000151550  w   DF .text  0000000000000010  GLIBCXX_3.4.30 _ZNKSt11__timepunctIwE15_M_am_pm_formatEPPKw

Could it be that when python the interpreter dlopen's jaxlib the symbols from libstdc++ found through jaxlib's runpath end up in some uhghh process-global namespace, and that by the time python dlopen's scipy it already ignores all the look ups for libstdc++? Frankly, I can't say I understand the mechanics too well

SomeoneSerge · 2023-03-16T14:38:58Z

After a brief ~~disillusionment session~~ chat with nixpgks' gurus, I think we should rebuild scipy with the same nvcc-compatible stdenv. This would probably require #173411 again

SomeoneSerge · 2023-03-16T14:44:01Z

Ideally, we'd also have means to automatically check (assert/warn) whether we're trying to use runtime dependencies that want an incompatible libstdc++. In the first approximation we could add passthru.cc in scipy and assert cc == scipy.cc in jax. The priority for now is just to un-break jax though

SomeoneSerge · 2023-03-16T15:29:54Z

The broader issue is there are (many?) more packages than just jax that break in this way. I had already mentioned gpflow, I have just observed the same with geoopt. How do other people figure out that they need to rebuild scipy?

SomeoneSerge · 2023-03-28T14:14:54Z

Another failure: easyocr.
I'm beginning to think that package-level overrides are less sustainable than rebuilding the entire nixpkgs with a compatible stdenv

SomeoneSerge · 2023-03-28T14:23:32Z

Cf.

# pkgs/top-level/all-packages.nix
  inherit (let
      num =
        if (with stdenv.targetPlatform; isVc4 || libc == "relibc") then 6
        else 12;
      numS = toString num;
    in {
      gcc = pkgs.${"gcc${numS}"};
      gccFun = callPackage (../development/compilers/gcc + "/${numS}");
    }) gcc gccFun;
  gcc-unwrapped = gcc.cc;

I don't want to, but we could add a conditional there, checking cudaSupport...

samuela · 2023-03-28T19:31:03Z

Another failure: easyocr.
I'm beginning to think that package-level overrides are less sustainable than rebuilding the entire nixpkgs with a compatible stdenv

IIRC stdenv's GCC is gcc12 and cudaPackages_12 is also gcc12... could we fix these errors by upgrading cudaPackages to cudaPackages_12?

SomeoneSerge · 2023-03-28T20:23:33Z

Yes, that'd solve it for as long as until the next gcc major release. We've been waiting for pytorch: #222778

Maybe we can already start running tests

SomeoneSerge · 2023-03-28T20:29:00Z

@ConnorBaker I think we can rename the issue into something more discoverable: "cudaPackages.backendStdenv causes libstdc++ mismatch errors when lagging behind stdenv", or "cuda: libstdc++.so.6: version 'GLIBCXX_3.4.30' not found"

nviets · 2023-04-01T01:16:42Z

Is anyone able to confirm if #224150 stems from the same issue? Xgboost with R+CUDA support fails near the end on:

...
[ 99%] Linking CXX executable /build/source/xgboost
/nix/store/8qm6sjqa09a03glzmafprpp69k74l4lm-binutils-2.40/bin/ld: /nix/store/vl893j5kphwcnqyf3qrxcmmjc8zrfa5q-icu4c-72.1/lib/libicuuc.so.72: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'

The GLIBCXX_3.4.30 version appears to match up with what's discussed here.

SomeoneSerge · 2023-04-09T19:44:07Z

Of the "big" packages, only jax is still failing: https://hercules-ci.com/github/SomeoneSerge/nixpkgs-unfree/jobs/3864

samuela · 2023-04-09T20:13:57Z

Hmm, IIUC #223664 was intended to fix this issue, and it seems it's been largely successful. Any idea why it's working on all packages except JAX?

SomeoneSerge · 2023-04-09T20:48:52Z

@samuela it would appear that our jedi tricks aren't sufficient to force bazel into using the correct libstdc++ version, and I'm pretty sure this isn't unrelated to us using cudatoolkit.cc

I have a suspicion that we're observing an instance of #225493, which is why I filed it right now. In fact, I'm pretty certain: if all wrapCCWith does is it adds propagatedBuildInputs, then there's no way bazel would take heed of that.

Maybe we could verify this by... adding stdenv.cc.cc.lib to the symlinkJoin (choosing the correct stdenv right in the expression, ad hoc, just for the test)

SomeoneSerge · 2023-05-09T07:43:25Z

Looking at https://hercules-ci.com/github/SomeoneSerge/nixpkgs-cuda-ci/jobs/4343, I don't think there are any libstdc++ related build errors left. Any future problems should probably be tracked in #226165

nixos-discourse · 2023-08-30T18:03:03Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/cuda-team-roadmap-update-2023-08-29/32379/1

ConnorBaker added 0.kind: bug Something is broken 6.topic: cuda Parallel computing platform and API labels Mar 9, 2023

ConnorBaker added this to CUDA Team Mar 9, 2023

github-project-automation bot moved this to 🆕 New in CUDA Team Mar 9, 2023

ConnorBaker mentioned this issue Mar 9, 2023

cudaPackages: point nvcc at a compatible -ccbin #218265

Merged

14 tasks

bcdarwin mentioned this issue Mar 15, 2023

Update to work with nixpkgs unstable LKS-CHART/medical-imaging-nix#1

Merged

SomeoneSerge mentioned this issue Mar 28, 2023

python39Packages.jax build failure on x86_64-linux as of b7a30876 #221739

Closed

SomeoneSerge mentioned this issue Mar 28, 2023

cudaPackages: use the same libstdc++ as the rest of nixpkgs #223664

Merged

13 tasks

ConnorBaker changed the title ~~scipy: cuda-enabled builds can encounter libstdc++ version errors~~ cudaPackages.backendStdenv: mismatch with stdenv causes libstdc++ errors Mar 29, 2023

ConnorBaker changed the title ~~cudaPackages.backendStdenv: mismatch with stdenv causes libstdc++ errors~~ cudaPackages.backendStdenv: mismatch with stdenv causes libstdc++ errors Mar 29, 2023

SomeoneSerge moved this from 🆕 New to 👀 In review in CUDA Team Mar 31, 2023

bcdarwin mentioned this issue Apr 1, 2023

xgboost: 1.7.4 -> 1.7.5 #224150

Merged

4 tasks

SomeoneSerge mentioned this issue Apr 4, 2023

🦦 NixOS 23.05 — Feature Freeze & Release Blockers #224457

Closed

49 tasks

nviets mentioned this issue Apr 5, 2023

cudaPackages: bump cudaPackages_11 -> cudaPackages_12 #222778

Closed

2 tasks

SomeoneSerge moved this from 👀 Awaits reviews to 🏗 In progress in CUDA Team Apr 13, 2023

SomeoneSerge mentioned this issue Apr 14, 2023

cudaPackages: extend cc-wrapper and rewrite cudaStdenv to properly solve libstdc++ issues #226165

Closed

SomeoneSerge closed this as completed May 9, 2023

github-project-automation bot moved this from 🏗 In progress to ✅ Done in CUDA Team May 9, 2023

bcdarwin mentioned this issue Aug 31, 2023

Build failure: python3Packages.qudida (with config.cudaSupport) #252616

Open

hacker1024 mentioned this issue Sep 9, 2023

OpenCV with CUDA fails to build with linker errors anduril/jetpack-nixos#109

Open

ruro mentioned this issue Oct 1, 2023

opencv4 and scipy cause glibc version conflicts with cudaSupport #258327

Closed

SomeoneSerge mentioned this issue Jan 20, 2024

cc-wrapper: relocate proprietary-compiler-specific material #282220

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cudaPackages.backendStdenv: mismatch with stdenv causes libstdc++ errors #220341

cudaPackages.backendStdenv: mismatch with stdenv causes libstdc++ errors #220341

ConnorBaker commented Mar 9, 2023

blurgyy commented Mar 10, 2023

samuela commented Mar 10, 2023

bcdarwin commented Mar 10, 2023

blurgyy commented Mar 11, 2023 •

edited

Loading

nixos-discourse commented Mar 16, 2023

SomeoneSerge commented Mar 16, 2023

SomeoneSerge commented Mar 16, 2023

SomeoneSerge commented Mar 16, 2023

SomeoneSerge commented Mar 16, 2023

SomeoneSerge commented Mar 28, 2023

SomeoneSerge commented Mar 28, 2023

samuela commented Mar 28, 2023

SomeoneSerge commented Mar 28, 2023

SomeoneSerge commented Mar 28, 2023 •

edited

Loading

nviets commented Apr 1, 2023 •

edited

Loading

SomeoneSerge commented Apr 9, 2023

samuela commented Apr 9, 2023

SomeoneSerge commented Apr 9, 2023 •

edited

Loading

SomeoneSerge commented May 9, 2023

nixos-discourse commented Aug 30, 2023

cudaPackages.backendStdenv: mismatch with stdenv causes libstdc++ errors #220341

cudaPackages.backendStdenv: mismatch with stdenv causes libstdc++ errors #220341

Comments

ConnorBaker commented Mar 9, 2023

Describe the bug

Additional context

Notify maintainers

blurgyy commented Mar 10, 2023

samuela commented Mar 10, 2023

bcdarwin commented Mar 10, 2023

blurgyy commented Mar 11, 2023 • edited Loading

nixos-discourse commented Mar 16, 2023

SomeoneSerge commented Mar 16, 2023

SomeoneSerge commented Mar 16, 2023

SomeoneSerge commented Mar 16, 2023

SomeoneSerge commented Mar 16, 2023

SomeoneSerge commented Mar 28, 2023

SomeoneSerge commented Mar 28, 2023

samuela commented Mar 28, 2023

SomeoneSerge commented Mar 28, 2023

SomeoneSerge commented Mar 28, 2023 • edited Loading

nviets commented Apr 1, 2023 • edited Loading

SomeoneSerge commented Apr 9, 2023

samuela commented Apr 9, 2023

SomeoneSerge commented Apr 9, 2023 • edited Loading

SomeoneSerge commented May 9, 2023

nixos-discourse commented Aug 30, 2023

blurgyy commented Mar 11, 2023 •

edited

Loading

SomeoneSerge commented Mar 28, 2023 •

edited

Loading

nviets commented Apr 1, 2023 •

edited

Loading

SomeoneSerge commented Apr 9, 2023 •

edited

Loading