package the Echogarden project into a standalone executable

[Tendaliu]

I can't package the Echogarden project into a standalone executable using tools like pkg. When I use pkg, I get the following warnings: Warning Babel parse has failed: 'await' is only allowed within async functions and at the top levels of modules. (2:12)

Warning Babel parse has failed: 'await' is only allowed within async functions and at the top levels of modules. (8:0)
Warning Entry 'main' not found in %1
%1: C:\Users\Sunda\AppData\Roaming\npm\node_modules\echogarden\node_modules\tinyld\package.json
%2: C:\Users\Sunda\AppData\Roaming\npm\node_modules\echogarden\dist\text-language-detection\TinyLDLanguageDetection.js
Warning Babel parse has failed: import.meta may appear only with 'sourceType: "module"' (58:56)
Warning Babel parse has failed: import.meta may appear only with 'sourceType: "module"' (323:51)
Warning Babel parse has failed: Unexpected character ''. (1:0)

[rotemdan]

Hi, thanks for your interest in trying to make a standalone executable! That is something I'd eventually want to have.

I tried running pkg and got the same error messages. I can't really know what they really mean, since there are so many factors that could go wrong with a tool like this, given such a large project with so many complex dependencies.

Actually, this isn't something I've tried to do, yet, since I know that given all the complexities of the dependencies, like static resource files I load, path resolutions I perform, workarounds to dependency package bugs, a few native executables I use, internal package system, optional native bindings that can be installed (for example, SAPI support on Windows etc.) it would probably require several days of work, a week, or possibly be impossible using a tool like pkg.

The choices I've made, to use a global package, and to avoid native bindings as much as possible, means that all it takes is to install Node.js (no extra compilation tools needed), and then npm install echogarden -g. It's really not that hard. Updating using npm update echogarden -g, also means that it is super-easy and efficient to get the most recent updates quickly.

I did think about a standalone deployment approach, that may make it easier for some users to run, also since I have server capabilities in development, and I want to allow users to easily download and install it locally, and then interface with the server using a web UI or browser extension.

The direction I was heading towards was to bundle a portable Node executable (and related files) with a directory containing the base package, plus a pre-installed node_modules directory, and ship it as a zip file or a self-extracting archive.

That may work actually, since all the path resolutions done internally should be relative to the base package directory. Even if there's a problem it would most likely be easy to fix.

There are downloadable portable distributions of the Node.js runtime (including npm) on the Node.js download page, like:

https://nodejs.org/dist/v18.17.0/node-v18.17.0-win-x64.zip

The internal structure looks like this:

So all it may take is to ensure that the CLI Launcher script (./dist/cli/CLILauncher.ts - a wrapper script that starts the CLI process with custom options to the node executable), is prioritized to resolve the local Node.js executable, rather then looking for the global one. That's relatively to do, say, to first look for a local executable called node/node.exe (depending on platform), and then fall back to try a global one.

This is something I was planning to try at some future point, but since you're showing interest, I might work on it sooner.

In any case, does a portable node bundled with a local install of Echogarden suit your needs? If so, I might try to get it to work.

Edit: If the portable installation supports npm install locally and you could try to run ./npm install echogarden locally, so it would just install the package into the portable directory's own npm_modules? And then try npx echogarden? If this works it's possible that no changes in the code are needed? (even if doesn't work, that's worth a try).

[Tendaliu]

Thank you for your comprehensive response and for considering my request. I appreciate your detailed explanation about the challenges of creating a standalone executable, and I understand the complexities involved.

I want to confirm that a portable Node.js bundled with a local installation of Echogarden is exactly what I need. This solution would perfectly suit my requirements, and I believe it would greatly simplify the deployment process for other users as well.

Please proceed with this if it is feasible and doesn't impose too much of a burden on your current workload. Your work on Echogarden is greatly appreciated, and I'm looking forward to the future developments of this project.

Best Regards

[rotemdan]

I tested this and it seems to work (assuming you're using Windows, but it should be similar with other platforms):

1. Download https://nodejs.org/dist/v18.17.0/node-v18.17.0-win-x64.zip (or whatever is right for the platform)
2. In the extracted directory, run npm install echogarden (for whatever reason the local npm.cmd gave me an error, so I used the global one. This needs to be investigated).
3. Then run node.exe node_modules/echogarden/dist/cli/CLILauncher.js speak "Hello World!" --engine=espeak

(npx gave me an error, so I used node.exe directly)

I uninstalled my global install of node.js to ensure it uses the local Node executable.

Now what we need is any tool that bundles the 316MB (10,000 files) directory content to a single executable, and when run calls some bundled batch file (or shell script on Linux) that runs node.exe node_modules/echogarden/dist/cli/CLILauncher.js [arguments]. I haven't really investigated which are the tools for that.

(CLILauncher.js uses the same node executable that started it - the possible modification I described earlier may not be needed - anyway, if there are issues, please let me know)

Edit: it could be that the portable distribution of Node doesn't actually include npm itself? Only the batch files that start it? Maybe npm needs to be installed locally? But trying npm install npm on that directory didn't work for me.. I don't know..

[Tendaliu]

Yes, it works for me too when I add a path before npm.cmd. It executes as expected. Regarding the tool to bundle the directory content into a single executable, perhaps we could consult with ChatGPT for potential solutions

[rotemdan]

Turns out there's a simpler method to start the CLI, using the auto-generated batch files that are created in the node_modules/.bin directory when npm install echogarden is run:

.\node_modules\.bin\echogarden.cmd speak "Hello World!" --engine=espeak

If this works, I'd prefer to use it since it would adapt to whatever future changes are done in the package.json bin property.

The auto-generated script echogarden.cmd detects the node executable in this way

:find_dp0
SET dp0=%~dp0

...


IF EXIST "%dp0%\node.exe" (
  SET "_prog=%dp0%\node.exe"
) ELSE (
  SET "_prog=node"
  SET PATHEXT=%PATHEXT:;.JS;=;%
)

I don't exactly know what it means. I'm not an expert on batch files.

Anyway, you can try to use this file as the target executable to run.

Based on my searches, and answer from Claude 2 chatbot:

• On Linux there's a program called makeself that may be able to help.
• On Windows 7Zip and WinRAR can make a self extracting archive from a directory, and they may have the ability to run an executable afterwards, but I don't know if they are designed to invisibly extract to a temporary directory and immediately run an executable. I don't know about Windows tools that can do this.

In general it's probably better to set the compression to "none", as otherwise it would be very slow to start.

I'll keep looking for better solutions.

(also: in the latest version I fixed an issue with module path detection for a particular library (kuromoji, used only for Japanese segmentation), that failed in the portable version. )

[rotemdan]

After asking Claude 2 about the batch file .\node_modules\.bin\echogarden.cmd:

@ECHO off
GOTO start
:find_dp0
SET dp0=%~dp0
EXIT /b
:start
SETLOCAL
CALL :find_dp0

IF EXIST "%dp0%\node.exe" (
  SET "_prog=%dp0%\node.exe"
) ELSE (
  SET "_prog=node"
  SET PATHEXT=%PATHEXT:;.JS;=;%
)

endLocal & goto #_undefined_# 2>NUL || title %COMSPEC% & "%_prog%"  "%dp0%\..\echogarden\dist\cli\CLILauncher.js" %*

It explained that %~dp0 resolves to the batch file directory, which is node-v18.17.0-win-x64\node_modules\.bin\ (I confirmed this by adding ECHO %~dp0 to the batch file).

This means that it either looks for the node executable in the batch file directory itself, or uses the default node command, that will try to run the node.exe file in the current directory, or fall back to the global one. So, it should work if you ensure that the current working directory is the base directory of the portable distribution.

Also, I was able to install npm in the portable directory, and use npx, by renaming the npm.cmd batch file it had to some other name and then running npm install npm to install npm locally (and then renaming back the npm.cmd batch file). Apparently, they don't bundle the npm tool with it, so it needs to be installed separately. It's not the most convenient.

Anyway, the scripts in node_modules/.bin: echogarden, echogarden.cmd and echogarden.ps1 can be used as target executables (.ps1 is for PowerShell I believe)

Edit: I retried the entire process. The npm package is definitely bundled in the zip archive. This time, running local npm and npx worked immediately, without needing to install npm in some other way. I don't know why I kept getting the error Error: Cannot find module 'node-v18.17.0-win-x64\node_modules\npm\bin\npx-cli.js' earlier. This is odd.

[rotemdan]

I managed to get a working single executable in Linux using makeself. (I used WSL2):

1. I downloaded and extracted https://nodejs.org/dist/v18.17.0/node-v18.17.0-linux-x64.tar.xz and renamed the directory to echogarden-portable.
2. I npm installed using ./bin/node ./lib/node_modules/npm/bin/npm-cli.js install echogarden (not the most elegant, but it uses the bundled npm. Maybe there's an easier way)
3. In the base directory, I created a shell script called echogarden.sh to start the Echogarden CLI:

#!/bin/sh

./bin/node ./node_modules/.bin/echogarden "$@"

4. I downloaded makeself and extracted it.
5. I created a self extracting and running executable using:

./makeself.sh --nocomp --nomd5 --nocrc ../echogarden-portable ./echogarden Echogarden ./echogarden.sh

(--nocomp --nomd5 --nocrc disable compression and checksums to get the fastest possible startup time).

The resulting executable is 407MB.

The time to extract is less than 5 seconds (which is great for more than 10,000 files), and it correctly passes any command-line arguments. When the executable is started, there is a status showing the extraction process:

Verifying archive integrity... All good.
Uncompressing Echogarden  100%

I don't think it's possible to silent it. I found an issue on the makeself tracker requesting this feature, but it was closed by the opener.

I also tried making a self-extracting 7-zip archive on Windows. It shows a prompt window (which may be possible to hide), and in general is much, much slower than makeself, even with no compression. I'm not sure if I'd consider it usable at that speed.

makeself uses tar, and does work in wsl/cygwin/git bash on Windows, but these require additional software..

Edit: It seems that the extraction time is slow partially due to Windows antivirus. With real-time antivirus protection disabled, 7-zip self-extracted in about 15 seconds, which is much faster than than the 1 minute it took with the Antivirus enabled, but still slower than tar on Linux. I also tried WinRAR, and its extraction time was about the same, maybe slower. Just note that my PC is very very old, and I have a SATA SSD. You can try for yourself to see what speeds you're getting (makeself may be much faster for those with a modern PC and an NVME SSD).

[Tendaliu]

You are amazing. Thanks a lot

[rotemdan]

Thanks, but it turns out this solution doesn't really work properly. makeself seemed to work at the beginning, but then coming back to test it again I realized it doesn't allow reading and writing to the local directory (I tried, it doesn't even get the correct CWD), or even any directory other than its own (its temporary one).

This is an intentional decision by the developer. I don't think there's a technical barrier of some sort (looks like the protections he added may have taken more work than not having them). He says:

Makeself is not meant to use as an application package like a regular application binary. There are serious security issues surrounding this kind of usage and I suggest you find a better way to do this...

I made some conversations with Claude (chatbot) about how self-extracting archives work, so I have a better idea now.

It's actually not that difficult to make a basic solution that would perform the extraction, and then run some command in the extracted directory. With a minimalist approach, omitting things like user-interface, compression, signatures, and verification, the core of what it does is not very big.

I am surprised there are no simple solutions for this (that I know of, or GPT-4, or Claude 2, for the matter..). It's just some compiled extraction code, with some binary data appended to it.

When I say "not that difficult".. I mean it would probably take at least a week of work though.. (from start, to a working debugged results, I mean) I can't see the benefit of doing something like this myself right now, compared to the benefit of this project. At least at this time. If at some point I feel it is strongly important, I might, but I still suspect it wouldn't be a great use of my time (may be seen as a distraction in retrospect).

[Tendaliu]

According to the latest information I have, the reason for the PKG packaging failure is that it does not support ES6.

[rotemdan]

Yes, I noticed that was mentioned in the repository.

But also, the way it performs the packaging would also require adding many special files, like binary files and modules (such as .dll files), JSON and configuration files, WASM resource files etc. as external resources that pkg would be needed to be extract and loaded from a temporary directory. It would require a large effort to figure out what modules load these resources, and what to do about each case. I don't think pkg is currently suited to deal with such a complex project.