Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to 1.39.0 [ Jenkins ] #275

Merged
merged 4 commits into from
Jul 5, 2023
Merged

Update to 1.39.0 [ Jenkins ] #275

merged 4 commits into from
Jul 5, 2023

Conversation

jfaltermeier
Copy link
Contributor

@jfaltermeier jfaltermeier commented Jul 3, 2023

What it does

  • update gitignore
  • update path to electron-main script
  • upgrade dependencies compatible with node 16
  • inversify update changes
  • add secondary-window package
  • enable backend bundling

Contributed on behalf of STMicroelectronics

How to test

Build and check that everything is still working

Review checklist

Reminder for reviewers

* update gitignore
* update path to electron-main script
* upgrade dependencies compatible with node 16
* inversify update changes

Contributed on behalf of STMicroelectronics

Signed-off-by: Johannes Faltermeier <[email protected]>
Contributed on behalf of STMicroelectronics

Signed-off-by: Johannes Faltermeier <[email protected]>
@jfaltermeier
Copy link
Contributor Author

Please note that this does not yet use a bundled backend (eclipse-theia/theia#12412)

When I try to create a bundled backend locally, I get the following error during webpack. This is on an ubuntu machine, so I don't know why it tries to package windows certs.

theia-blueprint: ERROR in ../../node_modules/@vscode/proxy-agent/out/index.js 380:70-105
theia-blueprint: Module not found: Error: Can't resolve '@vscode/windows-ca-certs/build/Release/crypt32.node' in '/home/johannes/Git/theia-example/node_modules/@vscode/proxy-agent/out'
theia-blueprint: resolve '@vscode/windows-ca-certs/build/Release/crypt32.node' in '/home/johannes/Git/theia-example/node_modules/@vscode/proxy-agent/out'
theia-blueprint:   Parsed request is a module
theia-blueprint:   using description file: /home/johannes/Git/theia-example/node_modules/@vscode/proxy-agent/package.json (relative path: ./out)
theia-blueprint:     resolve as module
theia-blueprint:       /home/johannes/Git/theia-example/node_modules/@vscode/proxy-agent/out/node_modules doesn't exist or is not a directory
theia-blueprint:       looking for modules in /home/johannes/Git/theia-example/node_modules/@vscode/proxy-agent/node_modules
theia-blueprint:         /home/johannes/Git/theia-example/node_modules/@vscode/proxy-agent/node_modules/@vscode/windows-ca-certs doesn't exist
theia-blueprint:       /home/johannes/Git/theia-example/node_modules/@vscode/node_modules doesn't exist or is not a directory
theia-blueprint:       /home/johannes/Git/theia-example/node_modules/node_modules doesn't exist or is not a directory
theia-blueprint:       looking for modules in /home/johannes/Git/theia-example/node_modules
theia-blueprint:         /home/johannes/Git/theia-example/node_modules/@vscode/windows-ca-certs doesn't exist
theia-blueprint:       looking for modules in /home/johannes/Git/node_modules
theia-blueprint:         /home/johannes/Git/node_modules/@vscode/windows-ca-certs doesn't exist
theia-blueprint:       /home/johannes/node_modules doesn't exist or is not a directory
theia-blueprint:       /home/node_modules doesn't exist or is not a directory
theia-blueprint:       /node_modules doesn't exist or is not a directory

@jfaltermeier jfaltermeier changed the title [WIP] Update to 1.39.0 Update to 1.39.0 Jul 3, 2023
@jfaltermeier jfaltermeier marked this pull request as ready for review July 3, 2023 13:57
@jfaltermeier
Copy link
Contributor Author

Please note that this does not yet use a bundled backend (eclipse-theia/theia#12412)

When I try to create a bundled backend locally, I get the following error during webpack. This is on an ubuntu machine, so I don't know why it tries to package windows certs.

theia-blueprint: ERROR in ../../node_modules/@vscode/proxy-agent/out/index.js 380:70-105
theia-blueprint: Module not found: Error: Can't resolve '@vscode/windows-ca-certs/build/Release/crypt32.node' in '/home/johannes/Git/theia-example/node_modules/@vscode/proxy-agent/out'
theia-blueprint: resolve '@vscode/windows-ca-certs/build/Release/crypt32.node' in '/home/johannes/Git/theia-example/node_modules/@vscode/proxy-agent/out'
theia-blueprint:   Parsed request is a module
theia-blueprint:   using description file: /home/johannes/Git/theia-example/node_modules/@vscode/proxy-agent/package.json (relative path: ./out)
theia-blueprint:     resolve as module
theia-blueprint:       /home/johannes/Git/theia-example/node_modules/@vscode/proxy-agent/out/node_modules doesn't exist or is not a directory
theia-blueprint:       looking for modules in /home/johannes/Git/theia-example/node_modules/@vscode/proxy-agent/node_modules
theia-blueprint:         /home/johannes/Git/theia-example/node_modules/@vscode/proxy-agent/node_modules/@vscode/windows-ca-certs doesn't exist
theia-blueprint:       /home/johannes/Git/theia-example/node_modules/@vscode/node_modules doesn't exist or is not a directory
theia-blueprint:       /home/johannes/Git/theia-example/node_modules/node_modules doesn't exist or is not a directory
theia-blueprint:       looking for modules in /home/johannes/Git/theia-example/node_modules
theia-blueprint:         /home/johannes/Git/theia-example/node_modules/@vscode/windows-ca-certs doesn't exist
theia-blueprint:       looking for modules in /home/johannes/Git/node_modules
theia-blueprint:         /home/johannes/Git/node_modules/@vscode/windows-ca-certs doesn't exist
theia-blueprint:       /home/johannes/node_modules doesn't exist or is not a directory
theia-blueprint:       /home/node_modules doesn't exist or is not a directory
theia-blueprint:       /node_modules doesn't exist or is not a directory

I looks like I can reproduce this behaviour with the electron example from the main Theia repository as well by replacing
"bundle": "yarn rebuild && theia build --mode development", with "bundle": "yarn rebuild && theia build",

So either this is a general issue with Ubuntu/Linux or with my local environment.

@msujew
Copy link
Member

msujew commented Jul 4, 2023

@jfaltermeier There is some weird issue with the bundler going on, and I'm not sure why it occurs. I've provided a commit over at 12f3d2b that fixes the backend build and runtime issues.

@jfaltermeier
Copy link
Contributor Author

@jfaltermeier There is some weird issue with the bundler going on, and I'm not sure why it occurs. I've provided a commit over at 12f3d2b that fixes the backend build and runtime issues.

Thanks! I will update my WIP commit accordingly tomorrow and will look into using the minimized build with electron-builder.

@marcdumais-work marcdumais-work self-requested a review July 4, 2023 15:11
@marcdumais-work
Copy link
Contributor

@jfaltermeier looks good generally. I will review more deeply after you make the update tomorrow.

* adjust electron-builder config and Dockerfile

Co-authored-by: Johannes Faltermeier <[email protected]>
Signed-off-by: Johannes Faltermeier <[email protected]>
@jfaltermeier jfaltermeier changed the title Update to 1.39.0 Update to 1.39.0 [ Jenkins ] Jul 5, 2023
@jfaltermeier
Copy link
Contributor Author

@marcdumais-work I've updated the PR.
The Github workflows are green and on the Eclipse Jenkins, Windows and MacOS are green. For Linux the Eclipse Jenkins does not spawn Build nodes atm. I will open a ticket at the helpdesk.

@JonasHelming
Copy link
Contributor

@marcdumais-work I would like to post the release announcement tomorrow if possible, so if you could apporve today in case you do not find issue, that would be great!

@marcdumais-work
Copy link
Contributor

I've pulled the latest version of the PR branch and am building - will test soon

@marcdumais-work
Copy link
Contributor

There seems to be an issue with ripgrep dependency on the Jenkins build - I have not seen this locally yet:

theia-blueprint: Downloaded ffmpeg shared library { version: "23.3.9", dist: "/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/electron/dist" }.
theia-blueprint: Successfully replaced "/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/electron/dist/libffmpeg.so".
theia-blueprint: "/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/electron/dist/libffmpeg.so" does not contain proprietary codecs (16 found).
theia-blueprint: node:internal/modules/cjs/loader:933
theia-blueprint:   const err = new Error(message);
theia-blueprint:               ^
theia-blueprint: Error: Cannot find module '@vscode/ripgrep/bin/rg'
theia-blueprint: Require stack:
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/native-webpack-plugin/lib/native-webpack-plugin.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/native-webpack-plugin/lib/index.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/applications/electron/gen-webpack.node.config.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/applications/electron/webpack.config.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack-cli/lib/webpack-cli.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack-cli/lib/bootstrap.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack-cli/bin/cli.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack/bin/webpack.js
theia-blueprint:     at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
theia-blueprint:     at Function.resolve (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/v8-compile-cache/v8-compile-cache.js:164:23)
theia-blueprint:     at NativeWebpackPlugin.copyRipgrep (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/native-webpack-plugin/lib/native-webpack-plugin.js:91:36)
theia-blueprint:     at /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/native-webpack-plugin/lib/native-webpack-plugin.js:78:28
theia-blueprint:     at Hook.eval [as callAsync] (eval at create (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:7:1)
theia-blueprint:     at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/tapable/lib/Hook.js:18:14)
theia-blueprint:     at /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack/lib/Compiler.js:882:27
theia-blueprint:     at /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/neo-async/async.js:2818:7
theia-blueprint:     at done (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/neo-async/async.js:3522:9)
theia-blueprint:     at alreadyWritten (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack/lib/Compiler.js:714:8) {
theia-blueprint:   code: 'MODULE_NOT_FOUND',
theia-blueprint:   requireStack: [
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/native-webpack-plugin/lib/native-webpack-plugin.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/native-webpack-plugin/lib/index.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/applications/electron/gen-webpack.node.config.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/applications/electron/webpack.config.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack-cli/lib/webpack-cli.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack-cli/lib/bootstrap.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack-cli/bin/cli.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack/bin/webpack.js'
theia-blueprint:   ]
theia-blueprint: }
theia-blueprint: Error: webpack exited with an unexpected code: 1.
theia-blueprint:     at ChildProcess.<anonymous> (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/application-manager/lib/application-process.js:66:28)
theia-blueprint:     at ChildProcess.emit (node:events:526:28)
theia-blueprint:     at maybeClose (node:internal/child_process:1092:16)
theia-blueprint:     at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
theia-blueprint: �[2K�[1G�[31merror�[39m Command failed with exit code 1.
theia-blueprint: �[2K�[1G�[34minfo�[39m Visit �[1mhttps://yarnpkg.com/en/docs/cli/run�[22m for documentation about this command.
theia-blueprint: �[2K�[1G�[31merror�[39m Command failed with exit code 1.
theia-blueprint: �[2K�[1G�[34minfo�[39m Visit �[1mhttps://yarnpkg.com/en/docs/cli/run�[22m for documentation about this command.

@marcdumais-work
Copy link
Contributor

In JenkinsFile I see that we apparently only use a GITHUB TOKEN for one case, the initial build of the various installers. But not in the following cases. A missing GH TOken can explain the ripgrep platform-specific executable not being downloaded, because of GITHUB API restrictions:

image

image

@jfaltermeier
Copy link
Contributor Author

There seems to be an issue with ripgrep dependency on the Jenkins build - I have not seen this locally yet:

theia-blueprint: Downloaded ffmpeg shared library { version: "23.3.9", dist: "/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/electron/dist" }.
theia-blueprint: Successfully replaced "/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/electron/dist/libffmpeg.so".
theia-blueprint: "/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/electron/dist/libffmpeg.so" does not contain proprietary codecs (16 found).
theia-blueprint: node:internal/modules/cjs/loader:933
theia-blueprint:   const err = new Error(message);
theia-blueprint:               ^
theia-blueprint: Error: Cannot find module '@vscode/ripgrep/bin/rg'
theia-blueprint: Require stack:
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/native-webpack-plugin/lib/native-webpack-plugin.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/native-webpack-plugin/lib/index.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/applications/electron/gen-webpack.node.config.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/applications/electron/webpack.config.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack-cli/lib/webpack-cli.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack-cli/lib/bootstrap.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack-cli/bin/cli.js
theia-blueprint: - /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack/bin/webpack.js
theia-blueprint:     at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
theia-blueprint:     at Function.resolve (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/v8-compile-cache/v8-compile-cache.js:164:23)
theia-blueprint:     at NativeWebpackPlugin.copyRipgrep (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/native-webpack-plugin/lib/native-webpack-plugin.js:91:36)
theia-blueprint:     at /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/native-webpack-plugin/lib/native-webpack-plugin.js:78:28
theia-blueprint:     at Hook.eval [as callAsync] (eval at create (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:7:1)
theia-blueprint:     at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/tapable/lib/Hook.js:18:14)
theia-blueprint:     at /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack/lib/Compiler.js:882:27
theia-blueprint:     at /home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/neo-async/async.js:2818:7
theia-blueprint:     at done (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/neo-async/async.js:3522:9)
theia-blueprint:     at alreadyWritten (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack/lib/Compiler.js:714:8) {
theia-blueprint:   code: 'MODULE_NOT_FOUND',
theia-blueprint:   requireStack: [
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/native-webpack-plugin/lib/native-webpack-plugin.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/native-webpack-plugin/lib/index.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/applications/electron/gen-webpack.node.config.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/applications/electron/webpack.config.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack-cli/lib/webpack-cli.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack-cli/lib/bootstrap.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack-cli/bin/cli.js',
theia-blueprint:     '/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/webpack/bin/webpack.js'
theia-blueprint:   ]
theia-blueprint: }
theia-blueprint: Error: webpack exited with an unexpected code: 1.
theia-blueprint:     at ChildProcess.<anonymous> (/home/jenkins/agent/workspace/Theia_PRs_PR-275/node_modules/@theia/application-manager/lib/application-process.js:66:28)
theia-blueprint:     at ChildProcess.emit (node:events:526:28)
theia-blueprint:     at maybeClose (node:internal/child_process:1092:16)
theia-blueprint:     at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
theia-blueprint: �[2K�[1G�[31merror�[39m Command failed with exit code 1.
theia-blueprint: �[2K�[1G�[34minfo�[39m Visit �[1mhttps://yarnpkg.com/en/docs/cli/run�[22m for documentation about this command.
theia-blueprint: �[2K�[1G�[31merror�[39m Command failed with exit code 1.
theia-blueprint: �[2K�[1G�[34minfo�[39m Visit �[1mhttps://yarnpkg.com/en/docs/cli/run�[22m for documentation about this command.

I will look into it. Note that this is happening for the build step that tries to adjust some metadata produced by electron-builder after the application was already built. So this only affects the metadata for the updater and a fix should not affect the application itself.

@marcdumais-work
Copy link
Contributor

Note that this is happening for the build step that tries to adjust some metadata produced by electron-builder after the application was already built.

I see, thanks. Our messages crossed above - have you seen mine? I think you are missing the GitHub token in that final step, which can lead to random failures to download the ripgrep executable at npm package install time.

@jfaltermeier
Copy link
Contributor Author

Note that this is happening for the build step that tries to adjust some metadata produced by electron-builder after the application was already built.

I see, thanks. Our messages crossed above - have you seen mine? I think you are missing the GitHub token in that final step, which can lead to random failures to download the ripgrep executable at npm package install time.

Thanks, I've just pushed a commit that tries that

@marcdumais-work
Copy link
Contributor

@jfaltermeier I wonder, do you know why it's necessary to run yarn install before updating the metadata? The way our repo is setup, this will rebuild both apps and all plugins from scratch, though not the packages (which is good). If we can remove that, it would speed-up the Jenkins build and avoid issues.

image

@marcdumais-work
Copy link
Contributor

[...] The way our repo is setup, this (yarn install) will rebuild both apps and all plugins from scratch

I try to improve the repo's build scripts in my WIP PR, that I will rebase on this one here after it's merged.

Copy link
Contributor

@marcdumais-work marcdumais-work left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @jfaltermeier

We need to have Jenkins CI pass. i have built and tested locally - seems to work fine. Now we have detachable windows on the Electron app! I do not have access to a Windows machine to test, so there might be some uncertainty there (I tested on linux Ubuntu).

@marcdumais-work
Copy link
Contributor

Jenkins CI has passed!

@jfaltermeier jfaltermeier merged commit 022878d into master Jul 5, 2023
@marcdumais-work
Copy link
Contributor

Thanks @jfaltermeier and @msujew!

@jfaltermeier
Copy link
Contributor Author

@jfaltermeier I wonder, do you know why it's necessary to run yarn install before updating the metadata? The way our repo is setup, this will rebuild both apps and all plugins from scratch, though not the packages (which is good). If we can remove that, it would speed-up the Jenkins build and avoid issues.

I think it might be required because this is a stash that was initially built on windows, but the signing/upload steps are performed on ubuntu again.
I think since we only invoke the script via ts-node, we might try to use it like this https://github.com/eclipse-theia/theia-blueprint/blob/5fe2d301210a4e4fdde8e5eb91a25988214cacbf/Jenkinsfile#L252 https://github.com/eclipse-theia/theia-blueprint/blob/5fe2d301210a4e4fdde8e5eb91a25988214cacbf/Jenkinsfile#L253 instead of via yarn electron ...

@marcdumais-work
Copy link
Contributor

I think it might be required because this is a stash that was initially built on windows, but the signing/upload steps are performed on ubuntu again.

Makes sense, thanks @jfaltermeier . In my WIP PR, running "yarn install" will be a valid alternative, since it will do much less vs now (similarly to the main Theia repo)

@marcdumais-work marcdumais-work deleted the jf/1.39 branch August 14, 2023 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants