Fatal Python Error when running docker compose #1440
Comments
|
@EshamAaqib You got a similar error recently: I think for vLLM. What was your resolution? |
|
@mohammedouhibi Please check your drivers are cuda 12.1+ compatible for this image. E.g. cuda driver 535+ is ok. |
I have upgraded my drivers to 'Driver Version: 545.29.06 CUDA Version: 12.3', and of course rebooted. The error indicating a probelm with cuda is gone, But i still have the same error popping up where python crashes. Here's the new output: |
|
We are just using llama.cpp (llama_cpp_python) and that's what's failing. Seen before. Maybe you don't have AVX or something like that. You would need to rebuild the docker image after changing https://github.com/h2oai/h2ogpt/blob/main/docs/linux_install.sh#L149-L157 |
Hi, i stumbled upon this issue after following instructions from the README_DOCKER.md file. how do i change this file and how do i then rebuild the docker image? @pseudotensor |
|
@tomerjr That instruction is old. Instead you would set the envs before installing, and for docker that means set those envs before building. h2ogpt/docker_build_script_ubuntu.sh Lines 41 to 44 in ab89711 Choose llama_cpp_python ARGS for your system according to llama_cpp_python backend documentation |
I'm a bit stuck here trying to run it on my server.
container successfully built, but running 'docker compose up' returns :
h2ogpt-main# docker compose up [+] Running 1/0 ✔ Container h2ogpt-main-h2ogpt-1 Created 0.0s Attaching to h2ogpt-1 h2ogpt-1 | /h2ogpt_conda/lib/python3.10/site-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.) h2ogpt-1 | return torch._C._cuda_getDeviceCount() > 0 h2ogpt-1 | No GPUs detected h2ogpt-1 | Using Model llama h2ogpt-1 | fatal: not a git repository (or any of the parent directories): .git h2ogpt-1 | load INSTRUCTOR_Transformer h2ogpt-1 | max_seq_length 512 h2ogpt-1 | Starting get_model: llama h2ogpt-1 | Fatal Python error: Illegal instruction h2ogpt-1 | h2ogpt-1 | Current thread 0x00007f8781ade280 (most recent call first): h2ogpt-1 | File "/h2ogpt_conda/lib/python3.10/ctypes/__init__.py", line 374 in __init__ h2ogpt-1 | File "/h2ogpt_conda/lib/python3.10/site-packages/llama_cpp/llama_cpp.py", line 74 in _load_shared_library h2ogpt-1 | File "/h2ogpt_conda/lib/python3.10/site-packages/llama_cpp/llama_cpp.py", line 87 in <module> h2ogpt-1 | File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed h2ogpt-1 | File "<frozen importlib._bootstrap_external>", line 883 in exec_module h2ogpt-1 | File "<frozen importlib._bootstrap>", line 688 in _load_unlocked h2ogpt-1 | File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked h2ogpt-1 | File "<frozen importlib._bootstrap>", line 1027 in _find_and_load h2ogpt-1 | File "/h2ogpt_conda/lib/python3.10/site-packages/llama_cpp/__init__.py", line 1 in <module> h2ogpt-1 | File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed h2ogpt-1 | File "<frozen importlib._bootstrap_external>", line 883 in exec_module h2ogpt-1 | File "<frozen importlib._bootstrap>", line 688 in _load_unlocked h2ogpt-1 | File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked h2ogpt-1 | File "<frozen importlib._bootstrap>", line 1027 in _find_and_load h2ogpt-1 | File "/workspace/src/gpt4all_llm.py", line 366 in validate_environment h2ogpt-1 | File "/h2ogpt_conda/lib/python3.10/site-packages/pydantic/v1/main.py", line 1100 in validate_model h2ogpt-1 | File "/h2ogpt_conda/lib/python3.10/site-packages/pydantic/v1/main.py", line 339 in __init__ h2ogpt-1 | File "/h2ogpt_conda/lib/python3.10/site-packages/langchain_core/load/serializable.py", line 107 in __init__ h2ogpt-1 | File "/workspace/src/gpt4all_llm.py", line 187 in get_llm_gpt4all h2ogpt-1 | File "/workspace/src/gpt4all_llm.py", line 30 in get_model_tokenizer_gpt4all h2ogpt-1 | File "/workspace/src/gen.py", line 2858 in get_model h2ogpt-1 | File "/workspace/src/gen.py", line 2443 in get_model_retry h2ogpt-1 | File "/workspace/src/gen.py", line 2110 in main h2ogpt-1 | File "/h2ogpt_conda/lib/python3.10/site-packages/fire/core.py", line 691 in _CallAndUpdateTrace h2ogpt-1 | File "/h2ogpt_conda/lib/python3.10/site-packages/fire/core.py", line 475 in _Fire h2ogpt-1 | File "/h2ogpt_conda/lib/python3.10/site-packages/fire/core.py", line 141 in Fire h2ogpt-1 | File "/workspace/src/utils.py", line 65 in H2O_Fire h2ogpt-1 | File "/workspace/generate.py", line 16 in entrypoint_main h2ogpt-1 | File "/workspace/generate.py", line 20 in <module> h2ogpt-1 | h2ogpt-1 | Extension modules: _cffi_backend, simplejson._speedups, charset_normalizer.md, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, numpy.linalg.lapack_lite, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._isolve._iterative, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.linalg._flinalg, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize.__nnls, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.special.cython_special, scipy.stats._stats, scipy.stats.beta_ufunc, scipy.stats._boost.beta_ufunc, scipy.stats.binom_ufunc, scipy.stats._boost.binom_ufunc, scipy.stats.nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, scipy.stats.hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats.ncf_ufunc, scipy.stats._boost.ncf_ufunc, scipy.stats.ncx2_ufunc, scipy.stats._boost.ncx2_ufunc, scipy.stats.nct_ufunc, scipy.stats._boost.nct_ufunc, scipy.stats.skewnorm_ufunc, scipy.stats._boost.skewnorm_ufunc, scipy.stats.invgauss_ufunc, scipy.stats._boost.invgauss_ufunc, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._statlib, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.tslib, pandas._libs.lib, pandas._libs.hashing, pyarrow.lib, pyarrow._hdfsio, pandas._libs.ops, numexpr.interpreter, pyarrow._compute, pandas._libs.arrays, pandas._libs.index, pandas._libs.join, pandas._libs.sparse, pandas._libs.reduction, pandas._libs.indexing, pandas._libs.internals, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.tslibs.strptime, pandas._libs.groupby, pandas._libs.testing, pandas._libs.parsers, pandas._libs.json, lz4._version, lz4.frame._frame, psutil._psutil_linux, psutil._psutil_posix, fontTools.misc.bezierTools, PIL._imaging, lxml._elementpath, lxml.etree, fontTools.varLib.iup, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, matplotlib._c_internal_utils, matplotlib._path, kiwisolver._cext, matplotlib._image, PIL._imagingft, av._core, av.logging, av.bytesource, av.buffer, av.audio.format, av.enum, av.error, av.utils, av.option, av.descriptor, av.container.pyio, av.dictionary, av.format, av.stream, av.container.streams, av.sidedata.motionvectors, av.sidedata.sidedata, av.packet, av.container.input, av.container.output, av.container.core, av.codec.context, av.video.format, av.video.reformatter, av.plane, av.video.plane, av.video.frame, av.video.stream, av.codec.codec, av.frame, av.audio.layout, av.audio.plane, av.audio.frame, av.audio.stream, av.audio.fifo, av.filter.pad, av.filter.link, av.filter.context, av.filter.graph, av.filter.filter, av.audio.resampler, yaml._yaml, google._upb._message, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5o, h5py.h5l, h5py._selector, pyclipper._pyclipper, shapely.speedups._speedups, rapidfuzz._feature_detector_cpp, rapidfuzz.distance._initialize_cpp, rapidfuzz.distance.metrics_cpp, rapidfuzz.fuzz_cpp, rapidfuzz.process_cpp_impl, rapidfuzz.utils_cpp, scipy.cluster._vq, scipy.cluster._hierarchy, scipy.cluster._optimal_leaf_ordering, sentencepiece._sentencepiece, cython.cimports.libc.math, multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, grpc._cython.cygrpc, greenlet._greenlet, sqlalchemy.cyextension.collections, sqlalchemy.cyextension.immutabledict, sqlalchemy.cyextension.processors, sqlalchemy.cyextension.resultproxy, sqlalchemy.cyextension.util, PIL._webp, regex._regex, sklearn.__check_build._check_build, sklearn.utils._isfinite, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.utils.sparsefuncs_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.utils._vector_sentinel, sklearn.feature_extraction._hashing_fast, pycrfsuite._pycrfsuite, sklearn.utils._random, sklearn.utils._seq_dataset, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.linear_model._cd_fast, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, sklearn.datasets._svmlight_format_fast (total: 318) h2ogpt-1 exited with code 0Additional context:
OS: Ubuntu 22.04.3 LTS (Jammy Jellyfish)
CPU: Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz (12 cores)
GPU:NVIDIA GeForce GTX 1660
GPU Driver: NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.0
Docker version: Docker version 24.0.7, build afdd53b
Feel free to ask for additional context
Thank you.
The text was updated successfully, but these errors were encountered: