dyld: Symbol not found: _PyExc_SystemError in tests if there is #[pymethods] in project

[deaz]

Tests fail with error when there are methods in #[pymethods] impl for struct:

Compiling pyo3-test v0.1.0 (/Users/ivan/pyo3-test)
    Finished dev [unoptimized + debuginfo] target(s) in 1.10s
     Running target/debug/deps/pyo3_test-34e869060858761f
dyld: Symbol not found: _PyExc_SystemError
  Referenced from: /Users/ivan/pyo3-test/target/debug/deps/pyo3_test-34e869060858761f
  Expected in: flat namespace
 in /Users/ivan/pyo3-test/target/debug/deps/pyo3_test-34e869060858761f
error: process didn't exit successfully: `/Users/ivan/pyo3-test/target/debug/deps/pyo3_test-34e869060858761f` (signal: 6, SIGABRT: process abort signal)

🌍 Environment

• Your operating system and version: macOS 10.14.2 (18C54)
• Your python version: 3.7.2
• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: i don't remeber, probably brew install python3. I used virtualenv via pipenv
• Your rust version (rustc --version): 1.34.0-nightly (8ae730a44 2019-02-04)
• Are you using the latest pyo3 version? Have you tried using latest master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")? I use 0.6.0-alpha.2. With latest master error is the same.

💥 Reproducing

Run cargo test for the following code:

use pyo3::prelude::*;

#[pyclass]
struct Class {}

#[pymethods]
impl Class {
    #[new]
    fn __new__(obj: &PyRawObject) -> PyResult<()> {
        obj.init(|| Class {})
    }
}

#[pymodule]
fn pyo3_test(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_class::<Class>()
}

#[cfg(test)]
mod tests {
    #[test]
    fn test() {}
}

Also i created minimal project for reproducing the issue: https://github.com/deaz/pyo3-test

[kngwyu]

Thanks for reporting!
Basically we need

[dependencies.pyo3]
features = []

to test project.
For extension projects(i.e. projects with features = ["extension-module"]), you can use setuptools-rust to run test.
@konstin
Do you know any workaround other than that?

[deaz]

As workaround I added #[cfg(not(test))] to my struct and impl and it is working fine

[konstin]

First of all, thank you for the test repository!

As @kngwyu said, we need to deactivate features for the tests. A proper solution for this requires rust-lang/cargo#2911, which unfortunately is unlikely to be implemented soon.

For now, you can use the following as workaround:

[package]
name = "pyo3-test"
version = "0.1.0"
authors = ["Ivan Vologdin <vologdin.nsk@gmail.com>"]
edition = "2018"

[lib]
name = "pyo3_test"
crate-type = ["cdylib"]

[dependencies.pyo3]
version = "0.6.0-alpha.2"

[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]

Tests can then be run with cargo test --no-default-features, while cargo build still works as expected.

With a bit context this workaround would be very useful in the guide.

[deaz]

Thanks for help!

[boredstiff]

Just ran into this issue - thank you both for posting really great instructions on how to work around this.

[konstin]

BTW this is also in the guide: https://pyo3.rs/v0.7.0-alpha.1/advanced.html#testing

[liquidscorpio]

The work-around mentioned in #340 (comment) does not work if the test is run on a workspace where one of member crates uses PyO3.

[mvaled]

I'm getting this error as well, however I'm not using #[pymethods] yet. My entire Rust code now is just:

use pyo3::prelude::*;

#[pymodule]
fn travertine(_py: Python, _m: &PyModule) -> PyResult<()> {
    Ok(())
}

#[cfg(test)]
mod test {
    use super::*;
    use pyo3::types::{IntoPyDict, PyDateTime};

    #[test]
    fn pydelta_conversion() {
        let gil = Python::acquire_gil();
        let py = gil.python();
        let datetime = py.import("datetime").unwrap();
        let locals = [("datetime", datetime)].into_py_dict(py);
        let now: &PyDateTime = py
            .eval("datetime.datetime.utcnow()", None, Some(&locals))
            .unwrap()
            .downcast()
            .unwrap();
        println!("{:?}", now);
    }
}

That suffices to trigger the error. The entire repository is in https://github.com/mvaled/pyo3bug340

The issue is reproducible with stable version of pyo3.

[davidhewitt]

Yes. TBH this issue is (I believe) really a dupe of #771

[gilescope]

Maybe we're going around this the wrong way, maybe when maturin builds it should add in --features=pyo3/extension-module because it knows it's required. That way the cargo test would just work.

We can already do:

maturin develop --cargo-extra-args="--features pyo3/extension-module"

Why not add that flag in automatically if it's clear that it's not there? Having pyo3/extension-module turned on as a default feature seems like a footgun as people will just run cargo test and expect stuff to work on cli, IDEs etc.

I've posted this in the maturin issues list here:
PyO3/maturin#325

[attack68]

I just ran into this problem also. I had quite an extensive project which was working fine with cargo test --lib. A few commits later and the cc linker issue came up on my both my different Macs. The changes made did not seem to be that significant, and certainly not as obvious as the original post assumes some attachment of [pymethods]: I had these already working well so this was not the cause.

Rolling back removed the issue and I tried to remake the changes to identify the cause but couldn't unfortunately.

This was particularly confusing after appearing on a reasonably developed project and apparently after rather innocuous changes.

[alex]

Please see https://pyo3.rs/v0.21.0/faq#i-cant-run-cargo-test-or-i-cant-build-in-a-cargo-workspace-im-having-linker-issues-like-symbol-not-found-or-undefined-reference-to-_pyexc_systemerror

…

On Tue, Mar 26, 2024 at 2:22 AM JHM Darbyshire ***@***.***> wrote: I just ran into this problem also. I had quite an extensive project which was working fine with cargo test --lib. A few commits later and the cc linker issue came up on my both my different Macs. The changes made did not seem to be that significant, and certainly not as obvious as the original post assumes some attachment of [pymethods]: I had these already working well so this was not the cause. Rolling back removed the issue and I tried to remake the changes to identify the cause but couldn't unfortunately. This was particularly confusing after appearing on a reasonably developed project and apparently after rather innocuous changes. — Reply to this email directly, view it on GitHub <#340 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBBI6JT4FQ77L5OKO7DY2EA2JAVCNFSM4GUTIVYKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBRHE2DQNZYGEZA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

[glevco]

@davidhewitt @alex I've been reading about this problem in the FAQ and related issues, and couldn't find any mentions or examples of running PyO3 code from a Rust test in a PyO3 lib.

For reference, I created a new hello world project with the exact steps from the Getting started guide, then added this to Cargo.toml (as suggested in the FAQ):

[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]

And then added this simple test to the lib.rs file:

#[cfg(test)]
mod tests {
    #[test]
    fn test() {
        // pyo3::Python::with_gil(|py| {}); // uncommenting this breaks tests
        assert_eq!(1, 1);
    }
}

Running cargo tests works¹. If we uncomment the first line of the test, it fails with cc linking issues. I tried every combination suggested in the FAQ but couldn't get this to work. Do you know if it's possible?

---

¹ This is also weird by the way, because according to the FAQ, for the configuration I set above, I should call cargo test --no-default-features instead of cargo test for it to work. But in fact, the opposite happens. I wonder if the docs are outdated?

[anuradhawick]

@glevco good catch! There may be some inconsistency in docs I guess. Also the flag seems to be required for cargo build and cargo build --release not for testing. May be testing takes place in the default feature, while we need the pyo3/extension-module for the actual build.

[rnag]

@davidhewitt @alex I've been reading about this problem in the FAQ and related issues, and couldn't find any mentions or examples of running PyO3 code from a Rust test in a PyO3 lib.

For reference, I created a new hello world project with the exact steps from the Getting started guide, then added this to Cargo.toml (as suggested in the FAQ):

[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]

And then added this simple test to the lib.rs file:

#[cfg(test)]
mod tests {
    #[test]
    fn test() {
        // pyo3::Python::with_gil(|py| {}); // uncommenting this breaks tests
        assert_eq!(1, 1);
    }
}

Running cargo tests works1. If we uncomment the first line of the test, it fails with cc linking issues. I tried every combination suggested in the FAQ but couldn't get this to work. Do you know if it's possible?

1 This is also weird by the way, because according to the FAQ, for the configuration I set above, I should call cargo test --no-default-features instead of cargo test for it to work. But in fact, the opposite happens. I wonder if the docs are outdated?

I can't reproduce. The same code works for me. Except I had to add pyo3::prepare_freethreaded_python() in the new versions of pyo3.

My full code:

#[cfg(test)]
mod tests {
    #[test]
    fn test() {
        pyo3::prepare_freethreaded_python();
        pyo3::Python::with_gil(|py| {}); // uncommenting this breaks tests
        assert_eq!(1, 1);
    }
}

VS Code runs it like:

cargo test --package <mod> --lib --no-default-features -- path::tests::test --exact --show-output 

By the way, to get that to work I added this in my .vscode/settings.json:

{
    "rust-analyzer.runnables.extraArgs": [
        "--no-default-features"
    ]
}

My cargo.toml is looking like:

# See https://github.com/PyO3/pyo3/issues/340#issuecomment-461514532
[dependencies.pyo3]
version = "0.22.5"

[features]
extension-module = ["pyo3/abi3-py39", "pyo3/extension-module"]
default = ["extension-module"]

I will add that I spent at least an hour trying to debug it, so I agree the docs could use a section on this - maybe something like "Calling Python from Rust in Cargo Unit Tests" or similar.

[mrexodia]

I ran into this issue, but solved it in a slightly different way. Instead of using the rust test feature I created a separate tests crate in my repository. From there I refer to the parent create, but disable the default features (thanks @rnag for the suggestion).

tests/Cargo.toml:

[package]
name = "tests"
edition = "2021"

[[bin]]
name = "tests"
path = "tests.rs"

[dependencies]
icicle-python = { path = "..", default-features = false }
pyo3 = { version = "*" }

The additional issue is that you need to set [lib].create-type = ["cdylib", "lib"] in the parent Cargo.toml to get rid of all the linker errors. Relevant commit: icicle-emu/icicle-python@f8d047e

To run the tests I do cd tests && cargo run 🤷🏻‍♂️

Update: Also added the tests create to a [workspace], which makes things a bit cleaner and allows you to run stuff with just: cargo run -p tests.