improve ccall symbol lookup

[carlobaldassi]

I've been trying to have two different versions of a library available, in order to check which one to use, but I can't find out how to make this work. I even tried to explicitly separate the handles to the two versions of the library, but then ccall always calls the one which was loaded first (i.e. it ignores the library name, and just uses the function name).

Here's the example code:

julia> const pkgd_dl = dlopen("/home/carlo/.julia/GLPK/deps/usr/lib/libglpk")
Ptr{Void} @0x0000000003455da0

julia> const system_dl = dlopen("libglpk")
Ptr{Void} @0x000000000326d740

julia> ccall(:add_library_mapping, Cint, (Ptr{Cchar}, Ptr{Void}), "pkgd_libglpk", pkgd_dl)
0

julia> ccall(:add_library_mapping, Cint, (Ptr{Cchar}, Ptr{Void}), "system_libglpk", system_dl)
0

julia> bytestring(ccall(("glp_version", "pkgd_libglpk"), Ptr{Uint8}, ()))
"4.48"

julia> bytestring(ccall(("glp_version", "system_libglpk"), Ptr{Uint8}, ()))
Warning: Possible conflict in library symbol glp_version
"4.48"

The last line should read "4.45" (and indeed it does if I don't load the other version first).
I checked that the add_library_sym in src/ccall.cpp does find the correct handles and returns different pointers in the two cases.

Is it possible at all to have two different functions with the same name from different libraries? Or am I doing something wrong?

[carlobaldassi]

Also, as a side note, if I try to close the first handle before trying with the second, I get a segfault:

julia> const pkgd_dl = dlopen("/home/carlo/.julia/GLPK/deps/usr/lib/libglpk")
Ptr{Void} @0x0000000002a78660

julia> ccall(:add_library_mapping, Cint, (Ptr{Cchar}, Ptr{Void}), "pkgd_libglpk", pkgd_dl)
0

julia> bytestring(ccall(("glp_version", "pkgd_libglpk"), Ptr{Uint8}, ()))
"4.48"

julia> dlclose(pkgd_dl)

julia> const system_dl = dlopen("libglpk")
Ptr{Void} @0x0000000002fc8ba0

julia> ccall(:add_library_mapping, Cint, (Ptr{Cchar}, Ptr{Void}), "system_libglpk", system_dl)
0

julia> bytestring(ccall(("glp_version", "system_libglpk"), Ptr{Uint8}, ()))
Warning: Possible conflict in library symbol glp_version
Segmentation fault (core dumped)

[JeffBezanson]

I have been wondering whether to allow this, since it doesn't work in native code unless you call dlopen/dlsym explicitly. But clearly, it would be a nice feature. A related use case is when you want to determine which library to load at startup time. So there should be effectively two kinds of uses of native symbols:

1. The static kind, using the native linker in the usual way.
2. Associate a global variable with each call site, and use dlsym to initialize this global the first time the call happens. This allows later binding but caches the result for performance.

The question is what the interface should be. My current idea is to use the difference between ccall(:sym, ...) and ccall((:sym, :lib), ...). Clearly in the first case you are not controlling what library the symbol comes from, so it's understood that you are at the mercy of the native linker. To go with that, we'd need a function to add a library reference (ala -l). Something like library("soname"). In the second case we'd use the global variable trick.

With this I believe we could get rid of add_library_mapping, and would only need dlopen/dlsym in very exotic cases.

cc @StefanKarpinski @ViralBShah @vtjnash @loladiro

[JeffBezanson]

Forgot one thing: the difference between a late-bound lookup that wants to be cached, versus one that actually wants to recompute the address on every call.

We could use ccall((:sym, libvar), ...) where libvar is not a constant to request the cached, late-bound behavior. You would assign libvar as soon as you know what library to use. Other forms like ccall(dlsym(), ...) or ccall(var, ...) would be completely dynamic.

[JeffBezanson]

See #1713

[ViralBShah]

Late bound library names would be certainly extremely useful. On mac, this would help ensure that we are picking the right library, when the symbol is present in multiple libraries.

Cc: @staticfloat

[staticfloat]

Yeap, that was my first thought as well. This would basically remove any problems of external libraries polluting our namespace with duplicate symbols, right?

[vtjnash]

can you see if this is still a problem? should be fixed in 0.3

[Keno]

I'll take silence as confirmation that this is fixed. If not, please reopen.