libmem_filesys.so: cannot open shared object file

[vwxyzjn]

Running into the issue of libmem_filesys.so: cannot open shared object file. I tried googling but could not find any info on this file

Additionally, any chance the Preview 3 and prior versions can be downloaded somewhere? Being able to download them might help with reproducibility :)

python joint_monkey.py 
Importing module 'gym_37' (/home/costa/Documents/go/src/github.com/cleanrl/cleanrl/ppo_continuous_action_isaacgym/isaacgym/isaacgym/_bindings/linux-x86_64/gym_37.so)
[Error] [carb] [Plugin: libcarb.gym.plugin.so] Could not load the dynamic library from /home/costa/Documents/go/src/github.com/cleanrl/cleanrl/ppo_continuous_action_isaacgym/isaacgym/isaacgym/_bindings/linux-x86_64/libcarb.gym.plugin.so. Error: libmem_filesys.so: cannot open shared object file: No such file or directory
[Warning] [carb] Potential plugin preload failed: /home/costa/Documents/go/src/github.com/cleanrl/cleanrl/ppo_continuous_action_isaacgym/isaacgym/isaacgym/_bindings/linux-x86_64/libcarb.gym.plugin.so
Setting GYM_USD_PLUG_INFO_PATH to /home/costa/Documents/go/src/github.com/cleanrl/cleanrl/ppo_continuous_action_isaacgym/isaacgym/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
[Error] [carb] Failed to acquire interface: [carb::gym::Gym v0.1], by client: carb.gym.python.gym_37 (plugin name: (null))
Traceback (most recent call last):
  File "joint_monkey.py", line 59, in <module>
    gym = gymapi.acquire_gym()
RuntimeError: Failed to acquire interface: carb::gym::Gym (pluginName: nullptr)

[vwxyzjn]

Think I found it to be shipped with the isaacgym package. Gonna debug a bit more.

[vwxyzjn]

Looks like this was an issue with my particular setup. Sorry for the ping.

[gavrielstate]

@vwxyzjn - might be useful for others to say what your issue was in case others come across it.

[vwxyzjn]

Hey, sorry @gavrielstate, this was specific to my poetry setup. Regular users shouldn't encounter this issue. I have ironed out all the details of running IsaacGym w/ CleanRL. Would you be interested in giving it a try?

The PR is at vwxyzjn/cleanrl#233, and the installation instruction is at https://cleanrl-git-isaacgym-vwxyzjn.vercel.app/rl-algorithms/ppo/#usage_8

Should be as straightforward as

git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl
git checkout isaacgym

cd cleanrl/ppo_continuous_action_isaacgym
# put `IsaacGym_Preview_4_Package.tar.gz` here 
stat IsaacGym_Preview_4_Package.tar.gz
mkdir temp_isaacgym
tar -xf IsaacGym_Preview_4_Package.tar.gz -C temp_isaacgym
mv temp_isaacgym/isaacgym/python/* isaacgym
rm -rf temp_isaacgym

poetry install -E isaacgym
# poetry run pip3 install torch --upgrade --extra-index-url https://download.pytorch.org/whl/cu113 # required only for 30xx GPUs
poetry run python ppo_continuous_action_isaacgym.py --help
poetry run python ppo_continuous_action_isaacgym.py --env-id Ant

[vwxyzjn]

Leave a note here. The cause of the issue seems to be that the isaacgym package is not setup as a usual python package, so it is apparently really important to do an editable install like pip install -e path-to-isaacgym, otherwise this error would appear.

Normal install like pip install path-to-isaacgym would somehow make isaacgym not able to find libmem_filesys.so, which is clearly present in site-packages/isaacgym/_bindings/linux-x86_64/. This issue means that any 3rd party RL libraries need to do pip install -e path-to-isaacgym to make things work.

(cleanrl-py3.7) ➜  ppo_continuous_action_isaacgym git:(poetry1.2) ✗ python
Python 3.7.8 (default, Mar 30 2022, 09:38:46) 
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import isaacgym
*** Warning: failed to preload USD libs
Importing module 'gym_37' (/home/costa/.cache/pypoetry/virtualenvs/cleanrl-0hpcRfYV-py3.7/lib/python3.7/site-packages/isaacgym/_bindings/linux-x86_64/gym_37.so)
[Error] [carb] [Plugin: libcarb.gym.plugin.so] Could not load the dynamic library from /home/costa/.cache/pypoetry/virtualenvs/cleanrl-0hpcRfYV-py3.7/lib/python3.7/site-packages/isaacgym/_bindings/linux-x86_64/libcarb.gym.plugin.so. Error: libmem_filesys.so: cannot open shared object file: No such file or directory
[Warning] [carb] Potential plugin preload failed: /home/costa/.cache/pypoetry/virtualenvs/cleanrl-0hpcRfYV-py3.7/lib/python3.7/site-packages/isaacgym/_bindings/linux-x86_64/libcarb.gym.plugin.so
Setting GYM_USD_PLUG_INFO_PATH to /home/costa/.cache/pypoetry/virtualenvs/cleanrl-0hpcRfYV-py3.7/lib/python3.7/site-packages/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
>>> isaacgym.__path__
['/home/costa/.cache/pypoetry/virtualenvs/cleanrl-0hpcRfYV-py3.7/lib/python3.7/site-packages/isaacgym']
(cleanrl-py3.7) ➜  ppo_continuous_action_isaacgym git:(poetry1.2) ✗ ls /home/costa/.cache/pypoetry/virtualenvs/cleanrl-0hpcRfYV-py3.7/lib/python3.7/site-packages/isaacgym/_bindings/linux-x86_64/ | grep libmem_filesys
libmem_filesys.so

The current distribution mode of Isaac Gym feels quite tricky, and it could make things challenging to reproduce. @gavrielstate, would it be possible to revisit #23 and consider an API-KEY-based approach, where the user would need to agree to the license agreement to get a key to run the local Isaac gym?

With the current distribution mode, it also seems more difficult to do CI as a third-party user since I would need to store the isaacgym code in an encrypted location...

[scharalambous3]

Hi @vwxyzjn,
Thanks for sharing this info. Have you been able to understand the root of the problem?
In my case the same erros

*** Warning: failed to preload USD libs

and

Error: libmem_filesys.so: cannot open shared object file: No such file or directory

arise when I try to load a pybind11 wrapper of a C++ package
This error persists even when installing them with the -e flag
Any help will be greatly appreciated!

[vwxyzjn]

Could you give this a try: https://docs.cleanrl.dev/rl-algorithms/ppo/#usage_10.

I just gave it a try and it still works.

[scharalambous3]

I'm not using cleanrl, but I've encountered the same error from a different setup (trying to import a pybind11 wrapped C++ package).Have you been able to identify what environment variable was the root of the problem? For example, sys.path doesn't seem to be it

[DanHrmti]

@scharalambous3 have you been able to find a solution for this?
I have encountered the same warning+Error when trying to run the joint_monkey.py example on a clean installation of isaacgym.

[scharalambous3]

The source of the error in my case was a conflict in my environment. The libtf.so of isaacgym was conflicting with the respective one of ROS. Hope that helps

[DanHrmti]

Not exactly my case, but thank you for the quick reply :)

[RayYoh]

Not exactly my case, but thank you for the quick reply :)

How did you tickle it?

[DanHrmti]

Adding the path to the library's location (...../isaacgym/python/isaacgym/_bindings/linux-x86_64) to the LD_LIBRARY_PATH environment variable solved the "Error: libmem_filesys.so: cannot open shared object file: No such file or directory" error for me.

"*** Warning: failed to preload USD libs" error message persists but does not prevent IsaacGym from working properly in my case.

[LouieTsai71]

Adding the path to the library's location (...../isaacgym/python/isaacgym/_bindings/linux-x86_64) to the LD_LIBRARY_PATH environment variable solved the "Error: libmem_filesys.so: cannot open shared object file: No such file or directory" error for me.

"*** Warning: failed to preload USD libs" error message persists but does not prevent IsaacGym from working properly in my case.

May I ask how you did this exactly?

[DanHrmti]

I added the following line to the end of the .bashrc file: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<PATH>,
where <PATH> is the path to the relevant library's location.

[HariP19]

The source of the error in my case was a conflict in my environment. The libtf.so of isaacgym was conflicting with the respective one of ROS. Hope that helps

@scharalambous3 Do you remember how you managed to solve the libtf.so issue? I am trying to use a ros package with isaacgym using pybind11 and I am facing the same issue. Importing the ROS lib first causes undefined symbol errors in importing Isaac Gym and vice versa

[scharalambous3]

There is a name conflict with one of isaac's libraries. You must rebuild the ROS tf library from source and rename it to something else