Builder in 25.0.0 finds files outside project and throws error

[blakebyrnes]

25.0.0

20.11.0

31.2.1 (current)

All, but testing on mac

Building in 24.x didn't have this problem. I tried to upgrade and have an issue where the filter is throwing an error about files it finds even though they're outside the directory where the build is happening (and also have all the dependencies set to no-hoist and that also didn't work)

      "**/electron-builder",
      "**/app-builder-lib",
      "**/builder-util",
      "**/builder-util-runtime",
      "**/dmg-builder",

Here's the error stack for one iteration (I've tried a number of ways to do this. This stack is before I did nohoist. The question is: How can I set a relative directory for the file filter not to go above?

<PATH>/platform/build-dist/cloud/main/.env.cloud.defaults must be under <PATH>/platform/build-dist/desktop/main/app/
at getRelativePath (<PATH>/platform/node_modules/app-builder-lib/src/util/filter.ts:32:13)
    at AsarPackager.unpackPattern (<PATH>/platform/node_modules/app-builder-lib/src/util/filter.ts:57:22)
    at AsarPackager.createPackageFromFiles (<PATH>/platform/node_modules/app-builder-lib/src/asar/asarUtil.ts:130:82)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at AsarPackager.pack (<PATH>/platform/node_modules/app-builder-lib/src/asar/asarUtil.ts:50:41)
    at <PATH>/platform/node_modules/app-builder-lib/src/platformPackager.ts:439:11
    at async Promise.all (index 0)
    at AsyncTaskManager.awaitTasks (<PATH>/platform/node_modules/builder-util/src/asyncTaskManager.ts:65:25)
    at MacPackager.doPack (<PATH>/platform/node_modules/app-builder-lib/src/platformPackager.ts:293:5)
    at MacPackager.pack (<PATH>/platform/node_modules/app-builder-lib/src/macPackager.ts:201:9)
    at Packager.doBuild (<PATH>/platform/node_modules/app-builder-lib/src/packager.ts:458:9)
    at executeFinally (<PATH>/platform/node_modules/builder-util/src/promise.ts:12:14)
    at Packager.build (<PATH>/platform/node_modules/app-builder-lib/src/packager.ts:392:31)
    at executeFinally (<PATH>/platform/node_modules/builder-util/src/promise.ts:12:14)```

[beyondkmp]

Can you provide a minimal reproducible demo?

[blakebyrnes]

I can't get it to happen in a small example. You can see the full project here: https://github.com/ulixee/platform/tree/main/desktop

The monorepo has a weird build process, which is probably why something is happening here that is out of the ordinary. We compile typescript into an entirely different directory and then builds node modules in that directory.

The desktop app is a submodule with a two app structure. So we have to actually copy over the project files so that the sub dependencies are found and packaged into the app.

I was trying out the new version because somehow only half of the native modules are added to asar unpacked. The latest version has the same problem - even though electron rebuilder shows them being rebuilt.

[jhlee8804]

same too...

[beyondkmp]

@blakebyrnes @jhlee8804 Could you enable DEBUG mode to get more info?

export DEBUG=electron-builder

[mmaietta]

Might be worth testing and reporting logs with next electron-builder 25.1.6 instead ust to make triaging/debugging easier

[blakebyrnes]

@beyondkmp Sorry, I totally restructured my project to workaround this bug and no longer running into it after the restructure. However, I think it was possibly caused by workspaces I had linked to that were in other directories. Maybe something with how electron builder is following symlinks and determining when they're considered "out of the workspace"?

[rathboma]

So I'm seeing this with Beekeeper Studio. I also think it has something to do with workspaces.

I'm seeing electron builder resolve a package from the wrong location:

Output from electron-builder:

{
    "name": "ssh2",
    "version": "1.8.0",
    "dir": "/home/rathboma/Projects/beekeeper-studio/ssh2"
  },

Where it SHOULD get it from:

/home/rathboma/Projects/beekeeper-studio/studio/node_modules/ssh2

Replication Steps with open source projects

1. mkdir Projects directory
2. cd Projects
3. git clone [email protected]:mscdex/ssh2.git
4. git clone [email protected]:beekeeper-studio/beekeeper-studio.git
5. cd beekeeper-studio
6. yarn
7. yarn electron:build --linux AppImage

@beyondkmp let me know if this helps enough

[beyondkmp]

@rathboma I've tested v25.1.8, and it's been fixed. You can update your electron-builder version.

[jhlee8804]

@blakebyrnes Thanks for your work. I tested with v25.1.8, but the issue still persists.

desktop-app:build:win:   ? not packed into asar archive  file=addons\Remote\Remote64.dll reason=contains executable code
desktop-app:build:win:   ? async task error  error=D:\clients\natives\Remote\build\Debug\NativeRemote.exp must be under D:\clients\apps\desktop\
desktop-app:build:win:   ? D:\clients\natives\Remote\build\Debug\NativeRemote.exp must be under D:\clients\apps\desktop\  failedTask=build stackTrace=Error: D:\clients\natives\Remote\build\Debug\NativeRemote.exp must be under D:\clients\apps\desktop\
desktop-app:build:win:     at getRelativePath (D:\clients\node_modules\app-builder-lib\src\util\filter.ts:32:13)
desktop-app:build:win:     at AsarPackager.unpackPattern (D:\clients\node_modules\app-builder-lib\src\util\filter.ts:57:22)
desktop-app:build:win:     at AsarPackager.createPackageFromFiles (D:\clients\node_modules\app-builder-lib\src\asar\asarUtil.ts:129:82)
desktop-app:build:win:     at processTicksAndRejections (node:internal/process/task_queues:95:5)
desktop-app:build:win:     at AsarPackager.pack (D:\clients\node_modules\app-builder-lib\src\asar\asarUtil.ts:49:41)
desktop-app:build:win:     at D:\clients\node_modules\app-builder-lib\src\platformPackager.ts:439:11
desktop-app:build:win:     at async Promise.all (index 0)
desktop-app:build:win:     at AsyncTaskManager.awaitTasks (D:\clients\node_modules\builder-util\src\asyncTaskManager.ts:65:25)
desktop-app:build:win:     at WinPackager.doPack (D:\clients\node_modules\app-builder-lib\src\platformPackager.ts:293:5)
desktop-app:build:win:     at WinPackager.pack (D:\clients\node_modules\app-builder-lib\src\platformPackager.ts:138:5)
desktop-app:build:win:     at Packager.doBuild (D:\clients\node_modules\app-builder-lib\src\packager.ts:459:9)
desktop-app:build:win:     at executeFinally (D:\clients\node_modules\builder-util\src\promise.ts:12:14)
desktop-app:build:win:     at Packager.build (D:\clients\node_modules\app-builder-lib\src\packager.ts:393:31)
desktop-app:build:win:     at executeFinally (D:\clients\node_modules\builder-util\src\promise.ts:12:14)
desktop-app:build:win: ?ELIFECYCLE? Command failed with exit code 1.
desktop-app:build:win: ERROR: command finished with error: command (D:\clients\apps\desktop) pnpm run build:win exited (1)
command (D:\clients\apps\desktop) pnpm run build:win exited (1)

 Tasks:    0 successful, 1 total
Cached:    0 cached, 1 total
  Time:    2m49.467s
Failed:    desktop-app#build:win

 ERROR  run failed: command  exited (1)
?ELIFECYCLE? Command failed with exit code 1.

[beyondkmp]

@jhlee8804 Is client/natives npm node module? Could you set debug mode to get more info or could you provide a minimal reproducible demo code?

https://www.electron.build/index.html#debug

[jhlee8804]

@beyondkmp Please see the attached file. build_win.log

[beyondkmp]

@jhlee8804 Thanks for the detailed debug info. Your issue is the same as this one(#8576) . Could you test the v26.0.0-alpha.4 version again?

[jhlee8804]

@beyondkmp Another error occurs in v26.0.0-alpha.4 versions. build_win2.log

[beyondkmp]

@mmaietta It appears to be an issue caused by electron/asar integration.

copy file config

  - from: src/renderer/libs/viewer
    to: out/renderer/libs/viewer
    filter:
      - '!**/*.{md,map,pdf}'
      - '!**/LICENSE'
      - '!**/*.{md,map,pdf}'
      - '!**/LICENSE'

error info

  ⨯ unable to copy, file is symlinked outside the package  source=src\renderer\libs\viewer\pdfjs\3.10.111\build\pdf.sandbox.js realPathFile=D:\Messenger\clients\packages\viewer\pdfjs\3.10.111\build\pdf.sandbox.js
  • async task error  error=Cannot copy file (pdf.sandbox.js) symlinked to file (pdf.sandbox.js) outside the package as that violates asar security integrity
  ⨯ unable to copy, file is symlinked outside the package  source=src\renderer\libs\viewer\pdfjs\3.10.111\build\pdf.worker.js realPathFile=D:\Messenger\clients\packages\viewer\pdfjs\3.10.111\build\pdf.worker.js
  • async task error  error=Cannot copy file (pdf.worker.js) symlinked to file (pdf.worker.js) outside the package as that violates asar security integrity
  ⨯ unable to copy, file is symlinked outside the package  source=src\renderer\libs\viewer\pdfjs\3.10.111\build\pdf.js realPathFile=D:\Messenger\clients\packages\viewer\pdfjs\3.10.111\build\pdf.js
  • async task error  error=Cannot copy file (pdf.js) symlinked to file (pdf.js) outside the package as that violates asar security integrity
  ⨯ unable to copy, file is symlinked outside the package  source=src\renderer\libs\viewer\pdfjs\3.10.111\web\cmaps\78-EUC-V.bcmap realPathFile=D:\Messenger\clients\packages\viewer\pdfjs\3.10.111\web\cmaps\78-EUC-V.bcmap
  • async task error  error=Cannot copy file (78-EUC-V.bcmap) symlinked to file (78-EUC-V.bcmap) outside the package as that violates asar security integrity
  ⨯ unable to copy, file is symlinked outside the package  source=src\renderer\libs\viewer\pdfjs\3.10.111\web\cmaps\78-EUC-H.bcmap realPathFile=D:\Messenger\clients\packages\viewer\pdfjs\3.10.111\web\cmaps\78-EUC-H.bcmap
  • async task error  error=Cannot copy file (78-EUC-H.bcmap) symlinked to file (78-EUC-H.bcmap) outside the package as that violates asar security integrity
  ⨯ unable to copy, file is symlinked outside the package  source=src\renderer\libs\viewer\pdfjs\3.10.111\web\cmaps\78-H.bcmap realPathFile=D:\Messenger\clients\packages\viewer\pdfjs\3.10.111\web\cmaps\78-H.bcmap
  • async task error  error=Cannot copy file (78-H.bcmap) symlinked to file (78-H.bcmap) outside the package as that violates asar security integrity
  ⨯ unable to copy, file is symlinked outside the package  source=src\renderer\libs\viewer\pdfjs\3.10.111\web\cmaps\78-RKSJ-H.bcmap realPathFile=D:\Messenger\clients\packages\viewer\pdfjs\3.10.111\web\cmaps\78-RKSJ-H.bcmap
  • async task error  error=Cannot copy file (78-RKSJ-H.bcmap) symlinked to file (78-RKSJ-H.bcmap) outside the package as that violates asar security integrity
  ⨯ unable to copy, file is symlinked outside the package  source=src\renderer\libs\viewer\pdfjs\3.10.111\web\cmaps\78-V.bcmap realPathFile=D:\Messenger\clients\packages\viewer\pdfjs\3.10.111\web\cmaps\78-V.bcmap
  • async task error  error=Cannot copy file (78-V.bcmap) symlinked to file (78-V.bcmap) outside the package as that violates asar security integrity
  ⨯ unable to copy, file is symlinked outside the package  source=src\renderer\libs\viewer\pdfjs\3.10.111\web\cmaps\78-RKSJ-V.bcmap realPathFile=D:\Messenger\clients\packages\viewer\pdfjs\3.10.111\web\cmaps\78-RKSJ-V.bcmap
  • async task error  error=Cannot copy file (78-RKSJ-V.bcmap) symlinked to file (78-RKSJ-V.bcmap) outside the package as that violates asar security integrity
  • async task error  error=Cannot cleanup: 

electron-builder/packages/app-builder-lib/src/asar/asarUtil.ts

Lines 103 to 111 in f84a083

	const realPathRelative = path.relative(this.config.appDir, realPathFile)
	const symlinkTarget = path.resolve(this.rootForAppFilesWithoutAsar, realPathRelative)
	const isOutsidePackage = realPathRelative.startsWith("..")
	if (isOutsidePackage) {
	log.error({ source: log.filePath(source), realPathFile: log.filePath(realPathFile) }, `unable to copy, file is symlinked outside the package`)
	throw new Error(
	`Cannot copy file (${path.basename(source)}) symlinked to file (${path.basename(realPathFile)}) outside the package as that violates asar security integrity`
	)
	}

It shouldn't consider everything outside of appDir as problematic. For example, in the issue above, it's copying a pdfjs folder from outside the appDir, and within pdfjs there are some symlink files that point to files within the pdfjs folder itself.

patch

diff --git a/packages/app-builder-lib/src/asar/asarUtil.ts b/packages/app-builder-lib/src/asar/asarUtil.ts
index 386dc993..4a2afc49 100644
--- a/packages/app-builder-lib/src/asar/asarUtil.ts
+++ b/packages/app-builder-lib/src/asar/asarUtil.ts
@@ -86,7 +86,7 @@ export class AsarPackager {
         return
       }
     }
-    const writeFileOrSymlink = async (transformedData: string | Buffer | undefined, source: string, destination: string, stat: fs.Stats) => {
+    const writeFileOrSymlink = async (transformedData: string | Buffer | undefined, source: string, destination: string, stat: fs.Stats,sourceDir:string) => {
       copiedFiles.add(destination)

       // If transformed data, skip symlink logic
@@ -100,9 +100,9 @@ export class AsarPackager {
         return this.copyFileOrData(undefined, source, destination, stat)
       }

-      const realPathRelative = path.relative(this.config.appDir, realPathFile)
-      const symlinkTarget = path.resolve(this.rootForAppFilesWithoutAsar, realPathRelative)
+      const realPathRelative = path.relative(sourceDir, realPathFile)
       const isOutsidePackage = realPathRelative.startsWith("..")
+      const symlinkTarget = path.resolve(this.rootForAppFilesWithoutAsar, realPathRelative)
       if (isOutsidePackage) {
         log.error({ source: log.filePath(source), realPathFile: log.filePath(realPathFile) }, `unable to copy, file is symlinked outside the package`)
         throw new Error(
@@ -130,7 +130,7 @@ export class AsarPackager {
         const dest = path.resolve(this.rootForAppFilesWithoutAsar, relative)

         matchUnpacker(file, dest, metadata)
-        taskManager.addTask(writeFileOrSymlink(transformedData, file, dest, metadata))
+        taskManager.addTask(writeFileOrSymlink(transformedData, file, dest, metadata, fileSet.src))

         if (taskManager.tasks.length >