Linking to a locally installed secp256k1 (instead of system)

[AdamISZ]

Hi,
Thought I'd ask this as I'm struggling with this and it's possible you'll have a good idea about it @dgpv .

In joinmarket we currently use the binding coincurve which uses cffi and then either (a) links to an existing system libsecp256k1 if it's available (can be disabled) or (b) uses a "local" build by sourcing a tar archive from the secp256k1 repo; this generates during setup a *.so file in the coincurve directory (so e.g. in my virtualenv i have the file /path/to/venv/lib/python3.6/site-packages/coincurve/_libsecp256k1.cpython-36m-x86_64-linux-gnu.so) and then cffi uses that.

I want to do the following: have python-bitcointx use a locally built libsecp256k1 instance so as to not overwrite any existing system installation and not to require sudo usage in installation. Building libsecp as part of the Joinmarket installation is fine, generally, as we require the same for libsodium anyway, so it's an easy add-on (I've already done it), just adding a --prefix to the ./configure arguments. But what I don't know is how I can get python-bitcointx to recognize a local installation (or, anywhere apart from system, i.e. /usr/local/lib, /usr/lib etc.); all I understand is that there is a load_secp256k1_library which calls ctypes.util.find_library, but from examining that code I can't see how there would be any configuration that could tell it to recognize a shared library in a different location. (LD_LIBRARY_PATH can do it, but that's hacky and not portable, I believe).

Do you have either (a) a thought on how I could achieve this as-is or (b) think it's possible to add a patch to make this feasible in python-bitcointx? Thanks.

[dgpv]

I expected that OS-dependent way would be used, that ctypes.util.find_library() implements, so this scenario was not considered.

I agree that an option to give the path to load_secp256k1_library() is needed.

[dgpv]

will add this in a moment

[dgpv]

Please check e34a847

[dgpv]

Although this is probably not what you need (still useful option though), as the library is imported in secp256k1.py with default path, and will give ImportError. Need some thought how this can be handled.

[dgpv]

What I see as an option is adding some function like set_custom_secp256k1_path() to bitcointx/__init__.py so that you can do

import bitcointx

bitcointx.set_custom_secp256k1_path('/your/path')

before importing anything else from bitcointx.

that will set bitcointx.util._secp256k1_library_path global var, that bitcointx.core.secp256k1 will be able to access

How is the same task is handled in coincurve ?

[AdamISZ]

What I see as an option

Yes, that was along the lines of what seemed the most reasonable solution to me.

How is the same task is handled in coincurve ?

I have not yet figured out. It uses cffi, and the setup.py uses distutils to build the shared object afaict. I guess this is somehow built into how the cffi package works, see e.g this module and how the member variable library_dirs there works; is it perhaps that cffi builds its own library object using something like rpath, which has a dependency on the base library? That would kinda make sense. But sorry, just spitballing here, haven't got to the bottom of it.

[dgpv]

please check set_custom_secp256k1_path() added in 7933310

[AdamISZ]

Confirmed 7933310 working by test: installed libsecp256k1.so in a custom location on a clean VM (Ubuntu1804),using --prefix argument to configure, then did a pip install of this package (pip3 install . in main dir of pbtx, inside a virtualenv), then ran python command prompt in the venv and did import bitcointx followed by bitcointx.set_custom_secp256k1_path('/path/to/.so') followed by import bitcointx.core, confirmed it loads OK, and confirmed it doesn't load OK with out the path setting function.
Thanks a lot!

[dgpv]

I also added path option to load_bitcoinconsensus_library() in 84a4c11