[Feature Request] default to yarn install --ignore-optional or allow user to configure using an env variable

[paprikati]

Lots of modern JS projects have devDependencies that you need in order to build the app (webpack etc.). But they also use big tools like storybook and jest which are not required at buildTime or runTime.

I would like to avoid installing these during my build process (because its slow). One way would be to put them as optionalDependencies in the package.json, and then run yarn install --ignore-optional.

Is this something the buildpack could support? I can see a few options:

• custom install script (like heroku-install)
• env variable YARN_IGNORE_OPTIONAL
• default to include --ignore-optional flag

[danielleadams]

@paprikati Is it too much to exclude installation of all the devDependencies? Or do you just need some of the devDependencies in the build? If you set YARN_PRODUCTION to true, then the buildpack will only install dependencies. (Docs: https://devcenter.heroku.com/articles/nodejs-support#only-installing-dependencies)

optionalDependencies, as I understand them, are really for packages that may have an install error or may be inaccessible to the install environment (like a conflicting operating system), but the install does not exit with a failure. Docs here: https://classic.yarnpkg.com/en/docs/dependency-types#toc-optionaldependencies. I'm not sure if this is the use case it's meant for.

On the other hand, I could see this as a useful flag, but not one that should be implemented at the Heroku/platform level since it's a flag for the package manager. Have you thought about opening an issue with Yarn to support this feature?

[paprikati]

So the problem I've got is that I need devDependencies like webpack, but not stuff like storybook or jest.

Arguably a nicer solution would be a yarn buildDependencies section or similar - so I'll open something with them, although not expecting a quick response.

It would be really useful to have some way in heroku of avoiding pulling storybook, jest etc. - would love any other suggestions

[danielleadams]

I have been thinking more about this, and I think it's tricky because the idea is that the devDependencies vs. dependencies should solve this - but now we have build dependencies and test dependencies, which fall under dev dependencies, but create bloat on Heroku builds. Therefore, this does make me think it may be a Heroku thing to figure out.

@paprikati can you provide me more details here? How much more time are you seeing on install times? If you add build dependencies (such as webpack) to dependencies and skip install of devDependencies, how much does your slug size increase? Thanks!

[paprikati]

If I use devDependencies (rather than putting webpack etc. into dependencies), then the overall build time is about 16s slower, which is about 10% of the total time (167s -> 152s) . It could be worse if I had a more complex testing infrastructure (e.g. an integration test harness like cypress).

Confusingly, if I put webpack etc. into dependencies, and set YARN_PRODUCTION=true then the slug size goes from 109Mb -> 95Mb, which I do not understand (it suggests that yarn prune is not working quite right, perhaps?).

To be totally honest, the way the JS ecosystem works right now, we would probably want to only be pushing a build artifact into the slug, rather than all the code (as you're essentially double storing everything), but I realise that's a bit scope-creep-ish from the original issue!

[danielleadams]

I agree with your final point 😄 especially when we get into compiled languages like TypeScript. There's a .slugignore file that I think fits what you need. (It may be worth it for me to add something to Node.js Heroku docs too.) Here are the Heroku docs: https://devcenter.heroku.com/articles/slug-compiler

For the slug size, what is the NODE_ENV set to? We use yarn install to run an additional install for dependencies because yarn doesn't have the "prune" functionality. (see here) If NODE_ENV is not set to production for the environment the above install is running, it will install all the dependencies (see docs) which could explain the larger slug size.

UPDATE: I wrote this and then realized that there's an earlier else that would catch NODE_ENV != "production". (https://github.com/heroku/heroku-buildpack-nodejs/blob/master/lib/dependencies.sh#L114) My guess is that it either has to do with the ignoring scripts in the pruning step or how yarn cache works. This is worth investigating though.

[masad-frost]

The "prune" step for yarn does absolutely nothing to dev deps from what I can tell