drop() not called automatically on FreeBSD/ HardenedBSD

[dmilith]

Making a long story short:

Library::open(Some(DEFAULT_LIBKVMPRO_SHARED), RTLD_NOW)
            .and_then(|lib| {
                let function_from_symbol: Symbol<extern "C" fn(uid_t) -> kvmpro_t> = unsafe { lib.get(b"get_process_usage_t\0") }?;
                let object: kvmpro_t = function_from_symbol(uid);
                mem::forget(lib);
                Ok(
                   String::from_utf8(object.bytes[0..object.length].to_vec()).unwrap_or("[]".to_string())
                )
            })
            .map_err(|err| {
                error!("FAILURE: processes_of_uid_short(): Unable to load shared library! Error: {:?}", err);
            })
            .unwrap_or("[]".to_string())

This code without that additional "mem::forget(lib);" call, will cause lib to not call drop, and cause a memleak.

For future generations :)

[dmilith]

If you want to see the whole thing, it's open source: https://github.com/VerKnowSys/sysapi/blob/master/src/lib.rs#L165

[nagisa]

I have fetched your project into a FreeBSD 11.1 VM, removed the mem::forget calls and I am seeing Drop being called just fine.

---

I have verified this in two distinct ways:

1. I changed a dependency on libloading to

   libloading = {path = "rust_libloading"} # local checkout
   

2. I changed libloading to print a message from <Library as Drop>::drop;

3. Ran a simple snippet that simply calls sysapi::soload::processes_of_uid() and observed the drop being actually called;

4. Changed the dependency on libloading back to libloading = "0.5";

5. Ran the same simple snippet under truss and observed that on library load the dynamic library was mapped and unmapped properly:

   openat(..., "libkvmpro.so", ...);
   ...
   mmap(0xADDRESS, somesize, PROT_READ | PROT_EXEC, ...) = 0x0
   

   and later on, once Library::drop is called:

   munmap(0xADDRESS, ...) = 0x0
   

---

I have verified the same behaviour in both debug and release modes. Is there more information on the issue that would help one to uncover the issue? One easy way to verify whether Library::drop is being called is to run the program with a breakpoints on dlopen and dlclose set. I would’ve done so myself, but gdb on my system is outdated enough that it is unable to handle the version 4 dwarf debug info.

[dmilith]

Very interesting. One thing I'm sure of is - without that manual mem::forget(), each request calling my API was leaking ~100-200KiB of RSS in RAM. My report with not calling drop() was a guess - that's true. Now I wonder why I have to make that explicit "forget call".

Thanks for checking this out!

[dmilith]

Second possibility is that issue is present only on HardenedBSD (is also FreeBSD using same 11-stable branch but has some additional security mitigations applied).

[nagisa]

Shared libraries are reference counted at the system level either way – a single .so will not have memory for it allocated multiple times, regardless of whether you unload it or not. You should not be able to observe continuous memory leak as a result of shared library not getting unloaded.

[dmilith]

Maybe it's not about .so loading/ unloading, but about data read (from function in my .so library) is still keeping like some strong pointer to data which is not auto freed?

[nagisa]

It could be the case if your shared library uses thread-locals, however libloading is oblivious to that functionality.

[dmilith]

My library talks to BSD kernel directly over kernel-kvm interface + procstat.
It's not thread safe for sure, but I control mutex over Libloading :) With that additional mem::forget() it works flawlessly :)

Thanks for insight and your time. Much appreciated :)

[dmilith]

Um.. I forgot to ask - after seeing that drop() being called - have you watched Ressident Set Size of your process? Maybe drop() is called, but here I notice +100KiB RSS usage each function call (of course talking about case without mem:forget(lib)). I wonder if process leaks on FreeBSD 11.1 as well (even if drop() was called)? It was hardly noticable normally.. but after I did ab/siege request flood.. I noticed mem leakage.

[nagisa]

I did not monitor RSS at all. And besides mem::forget leaks the library intentionally, that is, it is guaranteed that the library will not be dropped. I would recommend to use a memory profiling tool and make sure of the exact cause. Valgrind, massif, etc are good options for a tool here.

[nagisa]

I’ll close this for now since I don’t see anything actionable on the libloading side.