expose_imports and load_all: namespace found within global environments

[mdneuzerling]

Prework

• Read and abide by drake's code of conduct.
• Search for duplicates among the existing issues, both open and closed.
• If you think your issue has a quick and definite solution, consider posting to Stack Overflow under the drake-r-package tag. (If you anticipate extended follow-up and discussion, you are already in the right place!)

Description

Thank you for such a great package!

I'm experimenting with combining a package workflow and drake. I've creating a template in a repo. I want to expose changes in my custom functions so I'm using expose_imports, as per the drake manual. But this leads to a "namespace found within global environments" error. I'm executing my plan as below (The actual plan is much larger but I've truncated it to the first error):

library(drake)
devtools::load_all()
expose_imports("DrakeModelling")
plan <- drake_plan(
  reviews = download_and_read_data(),
  vocabulary = create_vocabulary(reviews$review)
)
make(plan)
# namespace found within global environments

The plan and everything else works without expose_imports, but this doesn't track changes in the custom functions. With expose_imports the vis_drake_graph of the above plan shows my functions.

I've been at this for a while. I've tried, without success:

• Installing the package and loading it as normal, without load_all.
• Running make with prework = devtools::load_all()
• Replacing the vocabulary function with one that imports a pre-defined vocabulary.rds (in which case, the error moves onto the next part of the complete plan)
• Creating a new environment as below (the plan executes, but the vis_drake_graph doesn't show the custom functions)

library(drake)
devtools::load_all()
my_env <- rlang::env()
expose_imports("DrakeModelling", envir = my_env)
plan <- drake_plan(
  reviews = download_and_read_data(),
  vocabulary = create_vocabulary(reviews$review,
                                 doc_proportion_min = 25 / nrow(reviews)),
  envir = my_env
)
make(plan, envir = my_env)

Errors

> traceback()
13: stop(msg, call. = FALSE)
12: handle_build_exceptions(target = target, meta = meta, config = config)
11: conclude_build_impl.default(value, target, meta, config)
10: conclude_build_impl(value, target, meta, config)
9: conclude_build(build, config)
8: local_build(target = targets[1], config = config, downstream = targets[-1])
7: loop_check(config)
6: drake_backend.loop(config)
5: drake_backend(config)
4: run_native_backend(config)
3: process_targets(config)
2: make_impl(config)
1: make(plan, prework = "devtools::load_all()")

> drake::diagnose(vocabulary)
$error
<simpleError in frameTypes(env): namespace found within global environments>

Reproducible example

I've created a branch in which I've trimmed everything out except what's necessary to run the first plan above: minimal branch

Desired result

• Expose all imports from my custom "DrakeModelling" package, visualised through vis_drake_graph, and
• Execute the plan.

Session info

R version 3.6.2 (2019-12-12)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_AU.UTF-8       LC_NUMERIC=C               LC_TIME=en_AU.UTF-8       
 [4] LC_COLLATE=en_AU.UTF-8     LC_MONETARY=en_AU.UTF-8    LC_MESSAGES=en_AU.UTF-8   
 [7] LC_PAPER=en_AU.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] DrakeModelling_0.1.0 tidyselect_1.0.0     drake_7.10.0         testthat_2.3.1       devtools_2.2.2      
[6] usethis_1.5.1.9000  

loaded via a namespace (and not attached):
 [1] pkgload_1.0.2        jsonlite_1.6         foreach_1.4.8        here_0.1             RcppParallel_4.4.4  
 [6] assertthat_0.2.1     base64url_1.4        yaml_2.2.0           remotes_2.1.1        slam_0.1-47         
[11] sessioninfo_1.1.1    pillar_1.4.3         backports_1.1.5      lattice_0.20-38      glue_1.3.1          
[16] digest_0.6.23        randomForest_4.6-14  htmltools_0.4.0      Matrix_1.2-18        tm_0.7-7            
[21] pkgconfig_2.0.3      purrr_0.3.3          processx_3.4.1       tibble_2.1.3         txtq_0.2.0          
[26] generics_0.0.2       ellipsis_0.3.0       withr_2.1.2          NLP_0.2-0            cli_2.0.1           
[31] magrittr_1.5         crayon_1.3.4         memoise_1.1.0        ps_1.3.0             tokenizers_0.2.1    
[36] storr_1.2.1          janeaustenr_0.1.5    fs_1.3.1             fansi_0.4.1          SnowballC_0.6.0     
[41] xml2_1.2.2           pkgbuild_1.0.6       tools_3.6.2          data.table_1.12.8    prettyunits_1.1.1   
[46] hms_0.5.3            formatR_1.7          stringr_1.4.0        lambda.r_1.2.4       callr_3.4.1         
[51] compiler_3.6.2       mlapi_0.1.0          rlang_0.4.4          futile.logger_1.4.3  grid_3.6.2          
[56] iterators_1.0.12     rstudioapi_0.11      htmlwidgets_1.5.1    visNetwork_2.0.9     igraph_1.2.4.2      
[61] codetools_0.2-16     R6_2.4.1             lubridate_1.7.4      dplyr_0.8.4          filelock_1.0.2      
[66] rprojroot_1.3-2      tidytext_0.2.2       futile.options_1.0.1 text2vec_0.5.1       readr_1.3.1         
[71] desc_1.2.0           stringi_1.4.5        parallel_3.6.2       Rcpp_1.0.3           vctrs_0.2.2 

[mdneuzerling]

I was able to resolve the issue by running update.packages() (I'm not sure why I didn't think to try this sooner!). The first code chunk above now executes the plan. However, it's hard to pin down exactly what package was causing this problem. I'll close the issue, but here's my new session info:

R version 3.6.2 (2019-12-12)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_AU.UTF-8       LC_NUMERIC=C               LC_TIME=en_AU.UTF-8       
 [4] LC_COLLATE=en_AU.UTF-8     LC_MONETARY=en_AU.UTF-8    LC_MESSAGES=en_AU.UTF-8   
 [7] LC_PAPER=en_AU.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] DrakeModelling_0.1.0 tidyselect_1.0.0     drake_7.10.0         testthat_2.3.1       devtools_2.2.2      
[6] usethis_1.5.1.9000  

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.3          lubridate_1.7.4     here_0.1            txtq_0.2.0          lattice_0.20-40    
 [6] visNetwork_2.0.9    prettyunits_1.1.1   ps_1.3.2            assertthat_0.2.1    rprojroot_1.3-2    
[11] digest_0.6.24       RhpcBLASctl_0.20-17 slam_0.1-47         R6_2.4.1            backports_1.1.5    
[16] pillar_1.4.3        rlang_0.4.4         rstudioapi_0.11     data.table_1.12.8   callr_3.4.2        
[21] Matrix_1.2-18       desc_1.2.0          tidytext_0.2.2      stringr_1.4.0       readr_1.3.1        
[26] htmlwidgets_1.5.1   igraph_1.2.4.2      compiler_3.6.2      janeaustenr_0.1.5   pkgconfig_2.0.3    
[31] pkgbuild_1.0.6      htmltools_0.4.0     tibble_2.1.3        lgr_0.3.3           randomForest_4.6-14
[36] fansi_0.4.1         crayon_1.3.4        dplyr_0.8.4         withr_2.1.2         SnowballC_0.6.0    
[41] grid_3.6.2          jsonlite_1.6.1      magrittr_1.5        storr_1.2.1         tokenizers_0.2.1   
[46] cli_2.0.1           stringi_1.4.6       fs_1.3.1            remotes_2.1.1       NLP_0.2-0          
[51] xml2_1.2.2          ellipsis_0.3.0      filelock_1.0.2      vctrs_0.2.2         generics_0.0.2     
[56] tools_3.6.2         float_0.2-3         glue_1.3.1          mlapi_0.1.0         purrr_0.3.3        
[61] hms_0.5.3           text2vec_0.6        yaml_2.2.1          processx_3.4.2      pkgload_1.0.2      
[66] parallel_3.6.2      tm_0.7-7            sessioninfo_1.1.1   base64url_1.4       rsparse_0.3.3.4    
[71] memoise_1.1.0  

[wlandau]

Glad you figured it out! Let me know if the error comes back. I tried, but could not reproduce it (with my own versions of packages). Possibly related: https://github.com/klmr/modules/issues/80?

To make it easier to troubleshoot next time, might I suggest shipping an renv lockfile too? That will take the hassle of making sure my package versions agree with yours, and it will make the error easier to reproduce.

[wlandau]

I just added new package-tracking functionality in targets tar_option_set(imports = your_packages) (ropensci/targets#239, ropensci/targets#241). See https://wlandau.github.io/targets-manual/practice.html#packages for details.