Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error Loading #1

Closed
ilyakorsunsky opened this issue Jul 1, 2018 · 33 comments
Closed

Error Loading #1

ilyakorsunsky opened this issue Jul 1, 2018 · 33 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@ilyakorsunsky
Copy link

ilyakorsunsky commented Jul 1, 2018

Thanks for this implementation, really looking forward to having a native R/Rcpp implementation to use on my big datasets!

The package seems to install fine but then there is a problem loading the shared object. I am running this on R3.5. Do you know what this could be?

Error: package or namespace load failed for ‘uwot’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so':
dlopen(/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so, 6): Symbol not found: __ZN13umap_gradient8clip_maxE
Referenced from: /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so
Expected in: flat namespace
in /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so
Error: loading failed
Execution halted
ERROR: loading failed

@jlmelville
Copy link
Owner

I failed to correctly configure RcppParallel and was getting away with it on Windows when building the package via devtools.

I think I've fixed the underlying issue. Could you please try again with the latest build and let me know? I have added Appveyor and Travis CI to catch Linux (and Windows) installation issues.

What OS are you installing this on, by the way? Looks likes Mac?

@ilyakorsunsky
Copy link
Author

Tried it again on Mac and Linux (redhat?) and got the same error again

@jlmelville
Copy link
Owner

jlmelville commented Jul 1, 2018

Which versions of R are you running? Sorry you, already said 3.5.

And do you know which version of Red Hat?

@jlmelville
Copy link
Owner

It might also be helpful to know if the Mac version of R is the homebrew version or one installed from https://cran.r-project.org/bin/macosx/ -- and do any other packages that use C++ get loaded correctly?

@jlmelville
Copy link
Owner

The Rcpp FAQ PDF http://dirk.eddelbuettel.com/code/rcpp/Rcpp-FAQ.pdf mentions "symbols not found" errors when registering dynamically loaded symbols and ensuring the RcppExports files are correctly regenerated.

I've updated the NAMESPACE entry to use useDynLib(uwot, .registration=TRUE) and regenerated the RcppExports. Maybe that will help? I am clutching at straws a bit due to being unable to reproduce this currently.

@ilyakorsunsky
Copy link
Author

Just realized that my redhat install is actually getting stuck on RcppParallel, so I'll need to fix that first

@bob-rietveld
Copy link

Hi, I am having the same issue. I am on Mac (v 10.13.1) and R 3.3. My C++ packages appear to be loading correctly. Here is the error message

Thanks for your help.

==

Error in dyn.load(file, DLLpath = DLLpath, ...) :
unable to load shared object '/Library/Frameworks/R.framework/Versions/3.3/Resources/library/uwot/libs/uwot.so':
dlopen(/Library/Frameworks/R.framework/Versions/3.3/Resources/library/uwot/libs/uwot.so, 6): Symbol not found: __ZN13umap_gradient8clip_maxE
Referenced from: /Library/Frameworks/R.framework/Versions/3.3/Resources/library/uwot/libs/uwot.so
Expected in: flat namespace
in /Library/Frameworks/R.framework/Versions/3.3/Resources/library/uwot/libs/uwot.so
Error: loading failed
Execution halted

@jlmelville
Copy link
Owner

I'm sorry it's still not working for people on Macs. I've added OS X to the Travis CI configuration, but it's building correctly on there. I don't know if the output there is helpful.

Are you using the CRAN R or one from Homebrew? Is there any chance that other C++ libraries were built with clang but uwot is being forced to use gcc or something? I don't have access to a Mac so I can't do a lot more than google for similar error messages.

@jlmelville jlmelville added bug Something isn't working help wanted Extra attention is needed labels Aug 2, 2018
@bob-rietveld
Copy link

Thanks for the prompt answer. I am using CRAN R, and as far as I can see clang is used. I tried the suggestions from Rccp FAQ pdf and removed R/RcppExports.R and src/RcppExports.cpp and then rebuild the package. Unfortunately, the error persists. Not sure what to do, since I know nothing of Rccp.

I'll update my R version do a clean install and see how that goes. I'll report the results.

@jasonserviss
Copy link

I noticed that this thread has not had any recent activity. Despite the fact that I have previously been able to install uwot, I have also experienced the problem described above and first noticed it ~2 weeks ago. Trying to install again today, I noted that it is still an issue. Since I was previously able to install the package I gradually stepped backwards in the commit history to find a "version" I can currently install. It turns out that although I can install 347f45d (Link to RcppParallel and RcppAnnoy., Jun 23), I cannot install 45fa4bd (Multi-threaded NN search., Jun 23). It would seem that either a) something in the later commit broke installation on some OSX or b) something has subsequently changed on some OSX since that change which has broken the installation.

I am using R version 3.5.1 and will include the full output at installation below. Hopefully this "clue" might help identify the issue so that it can be resolved.

Installing uwot
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file --no-environ
--no-save --no-restore --quiet CMD INSTALL
'/Users/jasonserviss/Github/uwot'
--library='/Library/Frameworks/R.framework/Versions/3.5/Resources/library'
--with-keep.source --install-tests

  • installing source package ‘uwot’ ...
    ** libs
    make: Nothing to be done for `all'.
    installing to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs
    ** R
    ** tests
    ** byte-compile and prepare package for lazy loading
    ** help
    *** installing help indices
    ** building package indices
    ** testing if installed package can be loaded
    Error: package or namespace load failed for ‘uwot’ in dyn.load(file, DLLpath = DLLpath, ...):
    unable to load shared object '/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so':
    dlopen(/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so, 6): Symbol not found: __ZN3tbb4task13note_affinityEt
    Referenced from: /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so
    Expected in: flat namespace
    in /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so
    Error: loading failed
    Execution halted
    ERROR: loading failed
  • removing ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot’
  • restoring previous ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot’
    Error: Command failed (1)

@jlmelville
Copy link
Owner

Thank you for the detective work, @jasonserviss. Is this the error message from the latest build or from 45fa4bd? The only thing that jumps out at me from the diff between the two commits is that I was mixing std::size_t and size_t in the added function. I have fixed this inconsistency across the code base in the latest commit. Would be nice if that was the problem (probably not though)!

Previous reported errors are from symbols from functions in uwot, but __ZN3tbb4task13note_affinityEt is from the TBB library, which RcppParallel uses internally. This usually means that the NAMESPACE file hasn't correctly exported functions from RcppParallel, but we are doing that.

Otherwise it might mean that there is a problem with TBB. Seems hard to believe that RcppParallel could be correctly built and installed if TBB wasn't around, so I would expect to see that error if for some reason we aren't including TBB at link time. With g++ there should be a -ltbb argument during the link step. For example, the Appveyor CI Windows build has the following in 00install.out:

C:/Rtools/mingw_64/bin/g++ -shared -s -static-libgcc -o uwot.dll tmp.def RcppExports.o gradient.o nn_parallel.o optimize.o perplexity.o smooth_knn.o supervised.o transform.o -Lc:/RLibrary/RcppParallel/lib/x64 -ltbb -ltbbmalloc -Lc:/R/bin/x64 -lR

Unfortunately, I don't have this information in the travis Mac CI. The contents of 00install.out might be useful here, but I don't know how easy it is to get hold of that during a failed build. If anyone is using RStudio on Mac, they could try importing uwot as a project from the github repo and then the initial build output from devtools::load_all(".") would contain a lot more diagnostic info.

@jasonserviss
Copy link

It looks as though the error message was potentially from an earlier build (my fault, sorry for the confusion). The installation from both the most recent commit (sorry still not working) and the commit that worked (347f45d) are shown below. Below that I show devtools::load_all('.') for the freshly cloned repo (I print the DESCRIPTION file to show the version). Not sure any of this is going to help a lot... Let me know if there is anything else I can do that comes to mind.

devtools::install_github('jlmelville/uwot', ref = 'def7898e7738f63f5be5ce9aa811baad4f6909b7')
#> Downloading GitHub repo jlmelville/uwot@def7898e7738f63f5be5ce9aa811baad4f6909b7
#> from URL https://api.github.com/repos/jlmelville/uwot/zipball/def7898e7738f63f5be5ce9aa811baad4f6909b7
#> Installing uwot
#> '/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file  \
#>   --no-environ --no-save --no-restore --quiet CMD INSTALL  \
#>   '/private/var/folders/6w/_8bn209n7_dbqbk7hdst6hfm0000gp/T/RtmpFkwl6D/devtoolsd9c14db71ce9/jlmelville-uwot-def7898'  \
#>   --library='/Library/Frameworks/R.framework/Versions/3.5/Resources/library'  \
#>   --install-tests
#> 
#> Installation failed: Command failed (1)
  devtools::install_github('jlmelville/uwot', ref = '347f45d4ed1fd0c9f4377bb10b0acf36c9ac2c7a')
#> Downloading GitHub repo jlmelville/uwot@347f45d4ed1fd0c9f4377bb10b0acf36c9ac2c7a
#> from URL https://api.github.com/repos/jlmelville/uwot/zipball/347f45d4ed1fd0c9f4377bb10b0acf36c9ac2c7a
#> Installing uwot
#> '/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file  \
#>   --no-environ --no-save --no-restore --quiet CMD INSTALL  \
#>   '/private/var/folders/6w/_8bn209n7_dbqbk7hdst6hfm0000gp/T/RtmpFkwl6D/devtoolsd9c172a7b431/jlmelville-uwot-347f45d'  \
#>   --library='/Library/Frameworks/R.framework/Versions/3.5/Resources/library'  \
#>   --install-tests
#> 

standard output and standard error

* installing *source* package ‘uwot’ ...
** libs
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c RcppExports.cpp -o RcppExports.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c gradient.cpp -o gradient.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c nn_parallel.cpp -o nn_parallel.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c optimize.cpp -o optimize.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c perplexity.cpp -o perplexity.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c smooth_knn.cpp -o smooth_knn.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c supervised.cpp -o supervised.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c transform.cpp -o transform.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/clang4/lib -o uwot.so RcppExports.o gradient.o nn_parallel.o optimize.o perplexity.o smooth_knn.o supervised.o transform.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation are out of sync. Falling back to library file for linking.
installing to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs
** R
** tests
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
Error: package or namespace load failed for ‘uwot’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so':
  dlopen(/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so, 6): Symbol not found: __ZN13umap_gradient8clip_maxE
  Referenced from: /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so
  Expected in: flat namespace
 in /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot’

* installing *source* package ‘uwot’ ...
** libs
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c RcppExports.cpp -o RcppExports.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c gradient.cpp -o gradient.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c optimize.cpp -o optimize.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c perplexity.cpp -o perplexity.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c smooth_knn.cpp -o smooth_knn.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/clang4/lib -o uwot.so RcppExports.o gradient.o optimize.o perplexity.o smooth_knn.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation are out of sync. Falling back to library file for linking.
installing to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs
** R
** tests
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (uwot)

Created on 2018-10-12 by the reprex package (v0.2.0).

Session info
devtools::session_info()
#> Session info -------------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.5.1 (2018-07-02)
#>  system   x86_64, darwin15.6.0        
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  tz       Europe/Stockholm            
#>  date     2018-10-12
#> Packages -----------------------------------------------------------------
#>  package   * version date       source        
#>  backports   1.1.2   2017-12-13 CRAN (R 3.5.0)
#>  base      * 3.5.1   2018-07-05 local         
#>  compiler    3.5.1   2018-07-05 local         
#>  curl        3.2     2018-03-28 CRAN (R 3.5.0)
#>  datasets  * 3.5.1   2018-07-05 local         
#>  devtools    1.13.6  2018-06-27 CRAN (R 3.5.0)
#>  digest      0.6.17  2018-09-12 CRAN (R 3.5.1)
#>  evaluate    0.11    2018-07-17 CRAN (R 3.5.0)
#>  git2r       0.23.0  2018-07-17 CRAN (R 3.5.0)
#>  graphics  * 3.5.1   2018-07-05 local         
#>  grDevices * 3.5.1   2018-07-05 local         
#>  htmltools   0.3.6   2017-04-28 CRAN (R 3.5.0)
#>  httr        1.3.1   2017-08-20 CRAN (R 3.5.0)
#>  knitr       1.20    2018-02-20 CRAN (R 3.5.0)
#>  magrittr    1.5     2014-11-22 CRAN (R 3.5.0)
#>  memoise     1.1.0   2017-04-21 CRAN (R 3.5.0)
#>  methods   * 3.5.1   2018-07-05 local         
#>  R6          2.2.2   2017-06-17 CRAN (R 3.5.0)
#>  Rcpp        0.12.19 2018-10-01 CRAN (R 3.5.0)
#>  rmarkdown   1.10    2018-06-11 cran (@1.10)  
#>  rprojroot   1.3-2   2018-01-03 CRAN (R 3.5.0)
#>  stats     * 3.5.1   2018-07-05 local         
#>  stringi     1.2.4   2018-07-20 CRAN (R 3.5.0)
#>  stringr     1.3.1   2018-05-10 CRAN (R 3.5.0)
#>  tools       3.5.1   2018-07-05 local         
#>  utils     * 3.5.1   2018-07-05 local         
#>  withr       2.1.2   2018-03-15 CRAN (R 3.5.0)
#>  yaml        2.2.0   2018-07-25 CRAN (R 3.5.0)
setwd('~/Desktop/uwot-master/')
readLines('DESCRIPTION')
#>  [1] "Package: uwot"                                                                                               
#>  [2] "Title: The Uniform Manifold Approximation and Projection (UMAP) Method for Dimensionality Reduction"         
#>  [3] "Version: 0.0.0.9004"                                                                                         
#>  [4] "Authors@R: person(\"James\", \"Melville\", email = \"[email protected]\", role = c(\"aut\", \"cre\"))"
#>  [5] "Author: James Melville <[email protected]>"                                                           
#>  [6] "Maintainer: James Melville <[email protected]>"                                                       
#>  [7] "Description: The dimensionality reduction method UMAP "                                                      
#>  [8] "  (McInnes and Healy, 2018, <arxiv:1802.03426>). It is similar "                                             
#>  [9] "  to, but faster than, the t-Distributed Stochastic Neighbor Embedding (t-SNE) "                             
#> [10] "  method."                                                                                                   
#> [11] "License: GPL-3"                                                                                              
#> [12] "URL: https://github.com/jlmelville/uwot"                                                                     
#> [13] "BugReports: https://github.com/jlmelville/uwot/issues"                                                       
#> [14] "Encoding: UTF-8"                                                                                             
#> [15] "LazyData: true"                                                                                              
#> [16] "Suggests: testthat,"                                                                                         
#> [17] "    covr"                                                                                                    
#> [18] "RoxygenNote: 6.1.0"                                                                                          
#> [19] "Depends: Matrix"                                                                                             
#> [20] "LinkingTo: Rcpp, RcppArmadillo, RcppProgress, RcppParallel, RcppAnnoy"                                       
#> [21] "Imports: Rcpp,"                                                                                              
#> [22] "    methods,"                                                                                                
#> [23] "    FNN,"                                                                                                    
#> [24] "    RSpectra,"                                                                                               
#> [25] "    RcppAnnoy,"                                                                                              
#> [26] "    RcppParallel,"                                                                                           
#> [27] "    irlba"                                                                                                   
#> [28] "SystemRequirements: GNU make"
devtools::load_all('.')
#> Loading uwot
#> Loading required package: Matrix
#> Error in dyn.load(dllfile): unable to load shared object '/Users/jasonserviss/Desktop/uwot-master/src/uwot.so':
#>   dlopen(/Users/jasonserviss/Desktop/uwot-master/src/uwot.so, 6): Symbol not found: __ZN13umap_gradient8clip_maxE
#>   Referenced from: /Users/jasonserviss/Desktop/uwot-master/src/uwot.so
#>   Expected in: flat namespace
#>  in /Users/jasonserviss/Desktop/uwot-master/src/uwot.so

Created on 2018-10-12 by the reprex package (v0.2.0).

@jlmelville
Copy link
Owner

Thanks for the detailed output. The good news is that at least we are back to having a consistent error message.

The symbol that can't be found is clip_max, which is a static constexpr in the gradient classes, but which is accessed via a member instance. I can't see why that isn't ok, but to simplify matters I have converted it to a member variable which is initialized in the constructor initializer list. I also discovered that I had neglected to add any header guards in the header files, which I have also remedied.

At this point, I doubt if this will fix the problem, but it might change the error message to something more useful.

Some other things to check on: are you using a custom ~/.R/Makevars file? There's a possibility this is some g++ vs clang issue, but I'm not sure how the Mac package binaries are built or whether it matters.

One other thing you could try is installing the package and tell it not to delete uwot.so when it fails:

devtools::install_github("jlmelville/uwot", args="--no-clean-on-error")

Then try running otool -L /Users/jasonserviss/Desktop/uwot-master/src/uwot.so and at least make sure everything is linking to where it ought to be.

@jasonserviss
Copy link

jasonserviss commented Oct 13, 2018

Yes, I have a custom Makevars. It contains:

LDFLAGS= -L/usr/local/clang4/lib
CC=/usr/local/clang4/bin/clang
CXX=/usr/local/clang4/bin/clang++
CXX1X=/usr/local/clang4/bin/clang++
CXX98=/usr/local/clang4/bin/clang++
CXX11=/usr/local/clang4/bin/clang++
CXX14=/usr/local/clang4/bin/clang++
CXX17=/usr/local/clang4/bin/clang++

I ended up with this custom Makevars by enabling OpenMP use in R via: OpenMP in R on OS X. If I temporarily remove the custom Makevars, I can install version 0.0.0.9004 (e34ae33) without any issues. So I guess that might isolate the "problem". I am not so familiar with different compilers to really be able to understand the details of what is going on (you indicate in your comment "some g++ vs clang issue"). Maybe it is possible to setup your Travis build to use clang to help you further diagnose the problem...? Or, alternatively, if I can do anything else on my end let me know.

Output below:

Installing uwot
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file --no-environ  \
  --no-save --no-restore --quiet CMD INSTALL  \
  '/Users/jasonserviss/Desktop/uwotCleanNew/uwot-master'  \
  --library='/Library/Frameworks/R.framework/Versions/3.5/Resources/library'  \
  --with-keep.source --install-tests 

* installing *source* package ‘uwot’ ...
** libs
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c RcppExports.cpp -o RcppExports.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c gradient.cpp -o gradient.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c nn_parallel.cpp -o nn_parallel.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c optimize.cpp -o optimize.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c perplexity.cpp -o perplexity.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c smooth_knn.cpp -o smooth_knn.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c supervised.cpp -o supervised.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c transform.cpp -o transform.o
clang++ -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o uwot.so RcppExports.o gradient.o nn_parallel.o optimize.o perplexity.o smooth_knn.o supervised.o transform.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation are out of sync. Falling back to library file for linking.
installing to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs
** R
** tests
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (uwot)

@jlmelville
Copy link
Owner

The problem might not be clang in general, but the version of clang that you're using. I currently know nothing about the various details of what compiler toolchain is used to build the R package binaries on Mac, what versions of which compilers are compatible with each other, what linker flags are needed and so on. I'm sure a definitive answer lies in various FAQs and the combined wisdom of googling for the "symbol not found" error with respect to other Rcpp-using projects, but not being a Mac user, I may not be the right person for this particular job.

For now I am suggesting not using a custom Makevars during building. I would be keen to hear if this works (or doesn't) for others.

@robsalasco
Copy link

I have the same issue on my mac!

@bob-rietveld
Copy link

I did a clean install of R 3.5.1. and the package now installs and I can run the examples. Hope this helps. My OS is 10.13.1

@jlmelville
Copy link
Owner

@robsalasco, are you using a custom Makevars?

@good-marketing, glad you got it fixed, were you also using a custom Makevars? With the clean install of R, was the library also created fresh? Perhaps installing the newest versions of Rcpp and RcppParallel can help.

@robsalasco
Copy link

@robsalasco, are you using a custom Makevars?

Hi @jlmelville! I tried compiling with and without custom Makevars.

@jlmelville
Copy link
Owner

The good news is that at least I was able to reproduce this error by attempting to build on r-hub with the "macOS 10.9 Mavericks, R-oldrel" configuration.

The bad news is I don't know what's causing the problem.

@LTLA
Copy link
Contributor

LTLA commented Dec 14, 2018

I would guess that it's because optimize.cpp refers to gradient.clip_max but it should really be Gradient::clip_max now that clip_max is a static data member.

jlmelville added a commit that referenced this issue Dec 14, 2018
Correctly refer to static data member, fixes #1.
@jlmelville jlmelville reopened this Dec 14, 2018
@jlmelville
Copy link
Owner

I don't know why gradient.clip_max causes problems: has there been a change to the C++ standard that disallows static member access through an instance? Thanks to @LTLA anyway.

The r-hub "macOS 10.9 Mavericks, R-oldrel" configuration that was failing is now passing again. This gives me hope that we are back to the situation where only custom Makevars is causing persistent problems.@robsalasco, if you are able, would you attempting to re-install from the current master?

For anyone else with problems, here's an example of the successful r-hub compilation output. Perhaps someone can compare with the flags and paths they are using and spot any potential differences (this is for R 3.3.3):

2723#> clang++ -std=c++11 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -DARMA_64BIT_WORD=1 -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -I"/Users/usergUphdZt8/R/Rcpp/include" -I"/Users/usergUphdZt8/R/RcppArmadillo/include" -I"/Users/usergUphdZt8/R/RcppProgress/include" -I"/Users/usergUphdZt8/R/RcppParallel/include" -I"/Users/usergUphdZt8/R/RcppAnnoy/include" -fPIC -Wall -mtune=core2 -g -O2 -c transform.cpp -o transform.o

2724#> clang++ -std=c++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o uwot.so RcppExports.o connected_components.o gradient.o nn_parallel.o optimize.o perplexity.o smooth_knn.o supervised.o transform.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation

@LTLA
Copy link
Contributor

LTLA commented Dec 14, 2018

Possibly something esoteric to do with constexpr? TBH I've never tried to access static members via an instance, mostly for semantic reasons.

Edit: see the end of https://en.cppreference.com/w/cpp/language/static, which suggests that we needed a definition in gradient.cpp along the lines of:

constexpr int umap_gradient::clip_max;
constexpr int apumap_gradient::clip_max;
constexpr int tumap_gradient::clip_max;
constexpr int largevis_gradient::clip_max;

@robsalasco
Copy link

@jlmelville 🎉 it works! the current master compiles without problem :)

@jlmelville
Copy link
Owner

Thanks for the feedback @robsalasco. Hopefully we are back to normal.

@robsalasco
Copy link

@jlmelville thanks for the fix! hope the package can be submitted to CRAN soon.

@jlmelville
Copy link
Owner

uwot is now on CRAN. Hopefully the Mac binaries can be used without problems. I will eventually close this issue if there is no re-occurrence.

@jasonserviss
Copy link

CRAN version works for me (although I haven't been using a custom Makevars for sometime).

yuhanH pushed a commit to yuhanH/uwot that referenced this issue Jul 20, 2020
yuhanH pushed a commit to yuhanH/uwot that referenced this issue Jul 20, 2020
Correctly refer to static data member, fixes jlmelville#1.
@laresbernardo
Copy link

Hi! I have just installed the CRAN version and most recent Github version and both return the same error when trying to use the umap function:

Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so':
  dlopen(/Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so, 6): Symbol not found: _EXTPTR_PTR
  Referenced from: /Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so
  Expected in: /Library/Frameworks/R.framework/Resources/lib/libR.dylib
 in /Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so

Here is my sessionInfo():

R version 4.0.0 (2020-04-24)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] uwot_0.1.8.9001 Matrix_1.2-18   word2vec_0.3.2  udpipe_0.8.4-1  ggplot2_3.3.2   lares_4.9.5     dplyr_1.0.2    
[8] openxlsx_4.2.2  h2o_3.30.1.3   

loaded via a namespace (and not attached):
 [1] tinytex_0.25      tidyselect_1.1.0  xfun_0.16         purrr_0.3.4       lattice_0.20-41   colorspace_1.4-1 
 [7] vctrs_0.3.4       generics_0.0.2    htmltools_0.5.0   yaml_2.2.1        rlang_0.4.8       pillar_1.4.6     
[13] glue_1.4.2        withr_2.3.0       lifecycle_0.2.0   plyr_1.8.6        stringr_1.4.0     munsell_0.5.0    
[19] gtable_0.3.0      rvest_0.3.6       zip_2.1.1         evaluate_0.14     knitr_1.29        Rcpp_1.0.5       
[25] scales_1.1.1      jsonlite_1.7.1    config_0.3        packrat_0.5.0     digest_0.6.25     stringi_1.5.3    
[31] rlist_0.4.6.1     grid_4.0.0        tools_4.0.0       bitops_1.0-6      magrittr_1.5      RCurl_1.98-1.2   
[37] patchwork_1.0.1   tibble_3.0.3      crayon_1.3.4      tidyr_1.1.2       pkgconfig_2.0.3   ellipsis_0.3.1   
[43] data.table_1.13.0 xml2_1.3.2        pROC_1.16.2       lubridate_1.7.9   rmarkdown_2.3     httr_1.4.2       
[49] rstudioapi_0.11   R6_2.4.1          compiler_4.0.0   

@jlmelville
Copy link
Owner

hi @laresbernardo is your install of RcppAnnoy ok? Does library(RcppAnnoy) work, for instance?

@laresbernardo
Copy link

hi @laresbernardo is your install of RcppAnnoy ok? Does library(RcppAnnoy) work, for instance?

Hi @jlmelville Thanks for reaching back!
No, I get a similar error when calling the library:

> library(RcppAnnoy)
Error: package or namespace load failed for ‘RcppAnnoy’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so':
  dlopen(/Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so, 6): Symbol not found: _EXTPTR_PTR
  Referenced from: /Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so
  Expected in: /Library/Frameworks/R.framework/Resources/lib/libR.dylib
 in /Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so
In addition: Warning message:
package ‘RcppAnnoy’ was built under R version 4.0.2 

Any idea as of why is this happening and/our how to fix it?

@jlmelville
Copy link
Owner

@laresbernardo this seems to be a problem with installing RcppAnnoy, not a uwot problem so I can only give typical advice about package problems: try removing RcppAnnoy and then reinstalling it. If you are able to, see if upgrading to R 4.0.2 helps. See if CRAN vs installing from the RcppAnnoy github repo makes a difference. Sorry I can't be more help.

@jlmelville
Copy link
Owner

It seems appropriate to close this after a long period of inactivity (thus guaranteeing it will start causing problems again, but I'll take the risk).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

7 participants