JupyterLab 3 compatibility #457

twavv · 2021-06-09T18:38:16Z

Closes #442.

twavv · 2021-06-10T21:59:21Z

The issue I have currently is that @webio/jupyter-lab-provider depends on @webio/webio which in turn depends on a (rather old) version of SystemJS.

SystemJS has a conditional import on fs (it only actually tries to require fs if it's running under node). Webpack does not like this. For the other providers (e.g., Jupyter notebook), I am able to alias fs to an empty module. I tried to do that with webpack.config.js, but it seems that's not being respected when I try to actually install the extension.

Console output

➜  jupyter-lab-provider git:(td/jupyterlab-v3) ✗ jupyter labextension install .
/usr/local/lib/python3.9/site-packages/jupyter_server/transutils.py:13: FutureWarning: The alias `_()` will be deprecated. Use `_i18n()` instead.
  warnings.warn(warn_msg, FutureWarning)
Building jupyterlab assets (development)
An error occured.
RuntimeError: JupyterLab failed to build
See the log file for details:  /var/folders/xs/ffgdzwh5083d95nkrt93hfk40000gn/T/jupyterlab-debug-1cecxvs1.log

➜  jupyter-lab-provider git:(td/jupyterlab-v3) ✗ cat /var/folders/xs/ffgdzwh5083d95nkrt93hfk40000gn/T/jupyterlab-debug-1cecxvs1.log
Node v14.16.1

Yarn configuration loaded.
> /Users/travis/.asdf/shims/npm pack /Users/travis/.julia/dev/WebIO/packages/jupyter-lab-provider
npm notice 
npm notice 📦  @webio/[email protected]
npm notice === Tarball Contents === 
npm notice 438B   README.md           
npm notice 16.8kB dist/labextension.js
npm notice 1.4kB  package.json        
npm notice === Tarball Details === 
npm notice name:          @webio/jupyter-lab-provider             
npm notice version:       0.8.15                                  
npm notice filename:      @webio/jupyter-lab-provider-0.8.15.tgz  
npm notice package size:  5.5 kB                                  
npm notice unpacked size: 18.6 kB                                 
npm notice shasum:        c064c504e8aca8c2d56c1035df35aa47a2fcdc32
npm notice integrity:     sha512-lQdYJJmsSL0Ah[...]2xkacY/FZSTgA==
npm notice total files:   3                                       
npm notice 
webio-jupyter-lab-provider-0.8.15.tgz

Node v14.16.1

Yarn configuration loaded.
Building jupyterlab assets (development)
> /Users/travis/.asdf/shims/npm pack /Users/travis/.julia/dev/WebIO/packages/jupyter-lab-provider
npm notice 
npm notice 📦  @webio/[email protected]
npm notice === Tarball Contents === 
npm notice 438B   README.md           
npm notice 16.8kB dist/labextension.js
npm notice 1.4kB  package.json        
npm notice === Tarball Details === 
npm notice name:          @webio/jupyter-lab-provider             
npm notice version:       0.8.15                                  
npm notice filename:      @webio/jupyter-lab-provider-0.8.15.tgz  
npm notice package size:  5.5 kB                                  
npm notice unpacked size: 18.6 kB                                 
npm notice shasum:        c064c504e8aca8c2d56c1035df35aa47a2fcdc32
npm notice integrity:     sha512-lQdYJJmsSL0Ah[...]2xkacY/FZSTgA==
npm notice total files:   3                                       
npm notice 
webio-jupyter-lab-provider-0.8.15.tgz

> node /usr/local/lib/python3.9/site-packages/jupyterlab/staging/yarn.js install --non-interactive
yarn install v1.21.1
[1/5] Validating package.json...
[2/5] Resolving packages...
success Already up-to-date.
Done in 0.51s.

> node /usr/local/lib/python3.9/site-packages/jupyterlab/staging/yarn.js yarn-deduplicate -s fewer --fail
yarn run v1.21.1
$ /usr/local/share/jupyter/lab/staging/node_modules/.bin/yarn-deduplicate -s fewer --fail
Done in 0.42s.

> node /usr/local/lib/python3.9/site-packages/jupyterlab/staging/yarn.js run build:dev
yarn run v1.21.1
$ jlpm run build
/usr/local/lib/python3.9/site-packages/jupyter_server/transutils.py:13: FutureWarning: The alias `_()` will be deprecated. Use `_i18n()` instead.
  warnings.warn(warn_msg, FutureWarning)
$ jlpm run clean && webpack
/usr/local/lib/python3.9/site-packages/jupyter_server/transutils.py:13: FutureWarning: The alias `_()` will be deprecated. Use `_i18n()` instead.
  warnings.warn(warn_msg, FutureWarning)
$ rimraf build
[webpack-cli] ModuleNotFoundError: Module not found: Error: Can't resolve 'fs' in '/usr/local/share/jupyter/lab/staging/node_modules/systemjs/dist'
    at /usr/local/share/jupyter/lab/staging/node_modules/webpack/lib/Compilation.js:1668:28
    at /usr/local/share/jupyter/lab/staging/node_modules/webpack/lib/NormalModuleFactory.js:712:13
    at eval (eval at create (/usr/local/share/jupyter/lab/staging/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:10:1)
    at /usr/local/share/jupyter/lab/staging/node_modules/webpack/lib/NormalModuleFactory.js:273:22
    at eval (eval at create (/usr/local/share/jupyter/lab/staging/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:9:1)
    at /usr/local/share/jupyter/lab/staging/node_modules/webpack/lib/NormalModuleFactory.js:402:22
    at /usr/local/share/jupyter/lab/staging/node_modules/webpack/lib/NormalModuleFactory.js:117:11
    at /usr/local/share/jupyter/lab/staging/node_modules/webpack/lib/NormalModuleFactory.js:628:24
    at /usr/local/share/jupyter/lab/staging/node_modules/webpack/lib/NormalModuleFactory.js:782:8
    at /usr/local/share/jupyter/lab/staging/node_modules/webpack/lib/NormalModuleFactory.js:902:5
resolve 'fs' in '/usr/local/share/jupyter/lab/staging/node_modules/systemjs/dist'
  Parsed request is a module
  using description file: /usr/local/share/jupyter/lab/staging/node_modules/systemjs/package.json (relative path: ./dist)
    Field 'browser' doesn't contain a valid alias configuration
    resolve as module
      /usr/local/share/jupyter/lab/staging/node_modules/systemjs/dist/node_modules doesn't exist or is not a directory
      /usr/local/share/jupyter/lab/staging/node_modules/systemjs/node_modules doesn't exist or is not a directory
      /usr/local/share/jupyter/lab/staging/node_modules/node_modules doesn't exist or is not a directory
      looking for modules in /usr/local/share/jupyter/lab/staging/node_modules
        single file module
          using description file: /usr/local/share/jupyter/lab/staging/package.json (relative path: ./node_modules/fs)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
              /usr/local/share/jupyter/lab/staging/node_modules/fs doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              /usr/local/share/jupyter/lab/staging/node_modules/fs.js doesn't exist
            .json
              Field 'browser' doesn't contain a valid alias configuration
              /usr/local/share/jupyter/lab/staging/node_modules/fs.json doesn't exist
            .wasm
              Field 'browser' doesn't contain a valid alias configuration
              /usr/local/share/jupyter/lab/staging/node_modules/fs.wasm doesn't exist
        /usr/local/share/jupyter/lab/staging/node_modules/fs doesn't exist
      /usr/local/share/jupyter/lab/node_modules doesn't exist or is not a directory
      /usr/local/share/jupyter/node_modules doesn't exist or is not a directory
      /usr/local/share/node_modules doesn't exist or is not a directory
      /usr/local/node_modules doesn't exist or is not a directory
      /usr/node_modules doesn't exist or is not a directory
      /node_modules doesn't exist or is not a directory
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

JupyterLab failed to build
Traceback (most recent call last):

  File "/usr/local/lib/python3.9/site-packages/jupyterlab/debuglog.py", line 47, in debug_logging
    yield

  File "/usr/local/lib/python3.9/site-packages/jupyterlab/labextensions.py", line 133, in start
    build(clean_staging=self.should_clean,

  File "/usr/local/lib/python3.9/site-packages/jupyterlab/commands.py", line 469, in build
    return handler.build(name=name, version=version, static_url=static_url,

  File "/usr/local/lib/python3.9/site-packages/jupyterlab/commands.py", line 678, in build
    raise RuntimeError(msg)

RuntimeError: JupyterLab failed to build

Exiting application: lab

jasongrout · 2021-06-10T22:21:01Z

To double-check things, can you look at the build_log.json that is generated when you build with --development True and make sure that your fallback for fs is being used? That build_log.json file is the webpack config that is actually being used to build your extension.

olynch · 2021-07-08T22:55:32Z

I am very much looking forwards for this to be merged, and am happy to try and help in any way.

jasongrout · 2021-07-09T04:43:18Z

Can you verify that the fs setting is being used, as I mentioned above?

thomasbrandon · 2021-07-18T05:09:22Z

I took a look (though I have very little node/webpack etc experience) and seems this an issue with how jupyter-labextension install works. A jlpm run build of the provider seems to work OK.

The resolve.fallback.fs setting is included in the packages/jupyter-lab-provider/labextension/build_log.json.
However, it seems these settings aren't used by jupyter-labextension install. It packs the extension then unpacks into it's staging directory and builds there. But it's build_log.json (~/.local/share/jupyter/lab/staging/build/build_log.json for me) doesn't include resolve.fallback.fs and just uses fixed settings from @jupyterlab/builders webpack.config.base.ts, called by staging/webpack.config.js. Indeed, if I locally edit my webpack.config.base.js (in staging/node_modules) to add resolve.fallback.fs then a jupyter lab build will succeed in building the staged @webio modules.

Looking at the docs it seems that only prebuilt modules can use custom webpack settings as noted here. I did try the jupyter labextension develop . they suggest for developing prebuilt extensions but it assumes the extension is packaged as a python project so errors trying to find setup.py. So I just symlinked the jlpm run build generated labextension content into my jupyter data directory's labextension folder and it seems to work (somewhat). I didn't install @webio/webio at all as it gets bundled into the jupyter-lab-provider output along with systemjs.

With this the extension does seem to be loaded into JupyterLab properly, WebIO content gets displayed, scripts are loaded in the browser (both extension and assets), and the WebIORenderer gets created when you add WebIO content. However observables (and other things depending on comms) don't seem to be working. Will try further to debug this but not clearly related to the build issue.

codecov · 2021-09-09T19:07:53Z

Codecov Report

Merging #457 (dfc943c) into master (b2efd5e) will decrease coverage by 1.78%.
The diff coverage is 38.46%.

@@            Coverage Diff             @@
##           master     #457      +/-   ##
==========================================
- Coverage   60.46%   58.67%   -1.79%     
==========================================
  Files          16       16              
  Lines         736      738       +2     
==========================================
- Hits          445      433      -12     
- Misses        291      305      +14

Impacted Files	Coverage Δ
src/providers/ijulia.jl	`0.00% <0.00%> (-25.00%)`	⬇️
src/render.jl	`31.81% <ø> (-2.28%)`	⬇️
src/scope.jl	`61.73% <62.50%> (-3.18%)`	⬇️
src/providers/mux.jl	`3.57% <0.00%> (-7.15%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b2efd5e...dfc943c. Read the comment docs.

jasongrout · 2021-09-10T23:17:14Z

Congratulations! 🎉 👏 That was a lot of work!

epatters · 2021-09-10T23:22:31Z

Yes, thanks! As a downstream user, I appreciate this.

twavv · 2021-09-11T08:40:51Z

Just a casual two full workdays 🥲

@epatters Would you be willing to test it out? Install WebIO#master and see if you can get it working with JupyterLab? Trying to use it within Jupyter should then show a link to the docs with instructions for how to enable the Jupyter extension. I'll add it to the readme, but I'd like to just make sure it's a reasonable process to go through.

Would appreciate if anything is unclear or you run into issues if you'd let me know!

twavv · 2021-09-13T22:30:55Z

This is merged and released now. Would appreciate feedback from anyone who wants to use WebIO with JupyterLab.

davibarreira · 2022-05-13T16:53:51Z

Hey @travigd , it seems that the error is still going on. Check #491.
I'm using the most current version of jupyter-lab. I figured that for some reason, the observables work when I create a fresh new notebook. Yet, once I restart the kernel, it stops working... Restarting jupyterlab does not solve the issue. For some odd reason, only starting a fresh new notebook file.
I've tried reinstalling jupyterlab, but it did not solve the issue. :(

twavv added 3 commits September 8, 2021 14:57

WIP: JupyterLab 3 compatibility

b6d9352

Allow Observables 0.3 and 0.4

b4622cd

Use Jupyter labextension cookiecutter

80b4473

twavv force-pushed the td/jupyterlab-v3 branch from eac891b to 80b4473 Compare September 9, 2021 19:01

twavv added 6 commits September 9, 2021 12:47

Update labextension LICENSE, relax compat requirements

4a4526e

Combine Jupyter labextension and serverextension

a2b4dde

Add Jupyter serverextension installation stuff

c04510a

Add nbextension to single Python package

2c5775c

Update Jupyter installation process

76a41c5

Fix diagnostic warning messages and whatnot

90e3498

twavv mentioned this pull request Sep 10, 2021

Update ijulia.md #461

Closed

Remove ijulia-tests reference

21023ce

twavv mentioned this pull request Sep 10, 2021

Update not-detected.md #463

Closed

Attempt to remove old Jupyter integration during build

dfc943c

twavv changed the title ~~WIP: JupyterLab 3 compatibility~~ JupyterLab 3 compatibility Sep 10, 2021

twavv merged commit 28a990e into master Sep 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JupyterLab 3 compatibility #457

JupyterLab 3 compatibility #457

twavv commented Jun 9, 2021

twavv commented Jun 10, 2021

jasongrout commented Jun 10, 2021

olynch commented Jul 8, 2021

jasongrout commented Jul 9, 2021

thomasbrandon commented Jul 18, 2021

codecov bot commented Sep 9, 2021 •

edited

Loading

jasongrout commented Sep 10, 2021

epatters commented Sep 10, 2021

twavv commented Sep 11, 2021

twavv commented Sep 13, 2021

davibarreira commented May 13, 2022

JupyterLab 3 compatibility #457

JupyterLab 3 compatibility #457

Conversation

twavv commented Jun 9, 2021

twavv commented Jun 10, 2021

jasongrout commented Jun 10, 2021

olynch commented Jul 8, 2021

jasongrout commented Jul 9, 2021

thomasbrandon commented Jul 18, 2021

codecov bot commented Sep 9, 2021 • edited Loading

Codecov Report

jasongrout commented Sep 10, 2021

epatters commented Sep 10, 2021

twavv commented Sep 11, 2021

twavv commented Sep 13, 2021

davibarreira commented May 13, 2022

codecov bot commented Sep 9, 2021 •

edited

Loading