Why are "install script" times 20 seconds?

[timothymiller]

Thank you for your hard work creating a wonderful library for sqlite.

Background

I am running better-sqlite3 in a turborepo with Cloudflare D1. My pnpm install times are 22 seconds, even with turborepo cache enabled.

Most of the total time is spent on this step:

node_modules/better-sqlite3: Running install script, done in 20.6s

How to reproduce

My reproduction repo is here.

Call to action

What is happening in this install script? Can it be cached? Any recommendations are much appreciated!

[coniferconcepts]

Seeing this as well. Thanks!

[ItsWendell]

../../node_modules/better-sqlite3: Running install script, done in 2m 23.1s
../../node_modules/es5-ext: Running postinstall script, done in 195ms
../../node_modules/esbuild: Running postinstall script, done in 176ms
../../node_modules/protobufjs: Running postinstall script, done in 161ms
../../node_modules/turbo: Running postinstall script, done in 331ms
../../node_modules/workerd: Running postinstall script, done in 203ms
../../node_modules/yarn: Running preinstall script, done in 158ms
## Custom post install output here removed (4 seconds)
Done in 2m 59.6s

I'm seeing super slow times locally too, luckily we don't have this bad of times on our CI.

[neoxpert]

I don't have slow install times for better-sqlite3 in my projects. Which NodeJS / Electron version are you using / targetting, have you checked if there is a recompile happening?

[timothymiller]

I don't have slow install times for better-sqlite3 in my projects. Which NodeJS / Electron version are you using / targetting, have you checked if there is a recompile happening?

I'm targeting Node.js v18 (latest LTS) in local dev. The lib is being used in a cloudflare workers environment for communicating with D1.

How could I check if there is a recompile happening?

[timothymiller]

I'm seeing high single CPU usage during the better-sqlite3 install script stage. Seems like it is recompiling?

Certainly this lib supports mac-arm64?

[neoxpert]

It is supported, yes. Otherwise the compilation would fail anyway. But right now there is no prebuilt binary for the use in NodeJS on arm64, only for Electron. There is still an open pull request #1002 that would add those prebuilt binaries.

A pragmatic way to see if the module has been recompiled, without fiddling with logs and flasgs etc. would be to have a look at node_modules/better-sqlite3/build/Release after the module has been installed. If there is any other file than better_sqlite3.node, the module has been recompiled.

[timothymiller]

The issue seems to have been re-introduced in 8.5.1 & 8.5.2.
I'm seeing recompiles happen on 8.5.2 on macOS arm64

[neoxpert]

As you are using pnpm, do you have node-gyp installed for it (pnpm install -g node-gyp)? Might be the same issue as in #1027.

[timothymiller]

I do have node-gyp installed

[enfipy]

Also suffering from this issue (persists in 8.6.0). The only package that takes 30+ seconds to recompile every pnpm i on my M1 mac.

I'm using Nx (same with Turbo), with Cloudflare D1 and DrizzleORM. I will be more than glad to provide more information if needed to fix this issue.

P.S: [email protected] is installed globally.

UPDATE: Issue persists in [email protected].

[neoxpert]

When better-sqlite3 get's recompiled everytime you are either using a runtime where no prebuilt binary is available or something in the toolchain (pnpm, prebuild-install .. ) resolves the wrong architecture / url of the prebuilt binary to load. I don't think that the overall problem is to be solved by better-sqlite3 itself but within the tools used to provide, dsitribute and resolve the prebuilt binaries.

Does the issue also occur using npm instead of pnpm? Does it also happen when installing this minimal project?

{
  "name": "testing",
  "version": "1.0.0",
  "engines" : { 
    "node": ">=18.0.0"
  },
  "dependencies": {
    "better-sqlite3": "9.1.1"
  }
}

[mceachen]

Closing due to inactivity