RFC: process.env.RAZZLE_RUNTIME_XXXX

[jaredpalmer]

Current Behavior

People struggle with how Razzle deals with .env variables (i.e. stringifying them at build time by webpack) just like create-react-app.

Expected Behavior

Razzle should have a way to honor runtime variables so users can more easily deploy their apps to Now/Heroku/Azure etc.

Suggested Solution

Make PORT, HOST, and any other env variables prefixed with RAZZLE_RUNTIME_XXXXXXX available during runtime.

Before

Available at compile time (i.e. will be stringified by webpack)

• RAZZLE_XXXXXX
• PORT
• HOST

After

Available at runtime

• PORT
• HOST
• RAZZLE_RUNTIME_XXXXXXX

Available at compile time (i.e. will be stringified by webpack)

• RAZZLE_XXXXXX

Discussion

Another alternative is to only stringify variables prefixed with RAZZLE_XXXX like the razzle-heroku plugin does. This would also be backwards compatible too. On the one hand, this would make it easier to work with how Heroku names it's config environment variables (e.g. MONGO_URI). On the other hand, this would be too easy to mess up by accident (i.e. reference a runtime variable within shared isomorphic code that is undefined on the client...exploding the application).

Related

#527 #526 #514 #477 #356 #285

[gregmartyn]

I personally wouldn't worry too much about process.env being unavailable on the client. We already have to know that 'window' is unavailable on the server; it makes sense to me that process.env would have the opposite behavior. (there's no node process to have an env for, and browsers don't expose their env vars that I'm aware of) I think of process.env as a place where server secrets can end up, so I'd rather it was unavailable on the client-side by default.

If it were up to me, process.env.RAZZLE_INLINED_XXXXXX variables would get compiled in during the build, (and available on the client as inline DefinePlugin'd strings) and anything else would only be available on the server-side.

NODE_ENV could be inlined too, as it's mostly used as a description of the build and not something variable like the environment that the code is running in. There are also some performance reasons to do so.

I do like the idea of PORT and HOST being variable at runtime. I may be running the same build artifacts in different environments.

[ferdinandsalis]

I can confirm the struggle. The current behaviour was not what I expected and especially tripped me up when deploying with up on aws. I have now customized the razzle.config.js so that all razzle specific vars are prefixed with RAZZLE_XXX and those are stringified at compile time. The rest ist available on process.env.XXX. I also agree with @gregmartyn about the unavailability of runtime vars through process.env.

[bartlangelaan]

While trying to set Razzle up in an existing React project we faced this issue. The port isn't overridable at run-time, and other process.env variables aren't available in the server.

Current behaviour at build-time:

process.env.PORT
process.env.NODE_ENV;
process.env.ASSETS_MANIFEST;
process.env.HOSTING_SET_VARIABLE;

becomes on both client and server:

3000;
'development';
'/Users/[...]/build/assets.json';
undefined;

While this assures that client and server can use exactly the same process envs, this makes it impossible to override at runtime or use other environment variables.

If you want to fully be backwards compatible, it should be transpiled to this in build-time (on the server, client can stay the same):

process.env.PORT || 3000;
process.env.NODE_ENV || 'development';
process.env.ASSETS_MANIFEST || '/Users/[...]/build/assets.json';
process.env.HOSTING_SET_VARIABLE;

This way, you can use process.env.PORT in both the client and the server. It will default to 3000.

If razzle is built with PORT=80, the client have process.env.PORT transpiled to 80 and the server will have it transpiled to process.env.PORT || 80. This will result in the same behaviour as how it is now.

If razzle is run with PORT=81 (while built with 80), the clients environment variable will stay 80 while the server variable will result in 81.

This behaviour can of course lead to unexpected behaviour, but it does provide the most flexible usage of process.env while maintaining full backwards compatibility. The port can still be overwritten on the server at run-time, and other environment variables set by hosting platforms will work on the server as-is.

[gregmartyn]

I think the underlying issue here is that Razzle is currently trying to pack multiple disparate functions into one object, and fundamentally change existing functionality.

It's trying to:

1. Turn process.env variables into constants for performance reasons (see minification example on https://webpack.js.org/plugins/define-plugin/#usage and nodejs slowness at Cache accesses to process.env? nodejs/node#3104)
2. Make those constants available to both the server and client so that isometric code has less to worry about.

Problems:

• process.env is for environment variables. Turning it into a set of constants changes its expected behavior.
• process.env often contains sensitive info including passwords, so it's not a great source for data to share with the client. If it's partially shared, and partially not shared, the dev must know that this one thing -- process.env -- not only doesn't behave like it does natively, but that it also has different behavior depending on the keys' prefixes.
• It's being turned into constants at build time, which is different from how environment variables behave in other contexts. Docker, for example, makes a distinction between build-time variables (useful when the build itself requires variable info depending on where it's running) and environment variables (useful when one pre-built image is deployed to different environments). Elastic Beanstalk, Heroku, et al. do the same.
• The suggested "RAZZLE_RUNTIME_" prefix isn't self-explanatory. It should convey that it's available client-side and that it's not a variable. I think "INLINED_" covers that, as inlining is commonly used to refer to something that happens at build time. It's not perfect, ("ISOMORPHIC_INLINED_"?) but it's also trying to be short.

My preference would be to split this functionality into different parts, and to not tamper with process.env at all.
The DefinePlugin could call out to a file like razzle.config.js to get the build constants
Recommend using a pattern similar to what Redux does with configureStore(window.PRELOADED_STATE) to pass data from server to client.

For backward compatibility, upgrade notes could provide a sample razzle.config.js that does the defines the old fashioned way.

[gregmartyn]

One addendum: I said "to not tamper with process.env at all." but I could see there being an exception for process.env.NODE_ENV for the performance reasons from my comment above and that "NODE_ENV" itself has generally taken on the meaning of "NODE_ENV = production means that this is an optimized build"

[jaredpalmer]

@gregmartyn we have to set NODE_ENV for perf optimizations and to use babel preset as it works now. My initial reasoning for messing with env was to make moving from CRA much easier as SSR is often added after a project has already started. We also use CRA on other projects so it simplifies our build tooling (slightly).

[gregmartyn]

Yeah; agreed that NODE_ENV is a useful exception. It's its own thing and closely tied to the build, so it's not surprising. I skipped right over CRA from a custom SSR solution, so I'm not really familiar with how they do things. It does look like this is an issue over there too: facebook/create-react-app#2353

I think this is a bigger issue for Razzle than CRA because the runtime code in a CRA app doesn't run on the server at all. CRA can do whatever it wants with process.env because as far as its client-side code is concerned, it'd be empty otherwise. Razzle on the other hand starts express for its SSR, and that code would reasonably expect process.env to have it's usual semantics with access to the full set of node runtime environment variables. Process.env has actual meaning on the server, so it's unfortunate that CRA coopted it for a different use-case. They could've used some other name instead of "process.env" like "cra.inlines". Instead, isomorphic code gets hit by a decision that was made when only considering the client-side.

[krazyjakee]

It should be noted in red everywhere that RAZZLE_XXX environment variables are ALL made available on the client.

How do I use sensitive environment variables without it being sent to the client?

[jaredpalmer]

They are not sent to the client u less you reference them in isomorphic code

[krazyjakee]

@jaredpalmer perhaps this issue is specific to afterjs then? I am only referencing them in server code.

[ajsharp]

I'd like to add a vote for the ability to define environment variables without the RAZZLE prefix. At the very least, process.env should not be wiped out on the server-side, which disables you from using something like dotenv to load server-side env variables. This seems like way too intrusive of an assumption to make about the application environment.

I'm not quite clear on how razzle currently injects environment variables into the client and server, but certainly you wouldn't want server-specific stuff on the client. Unfortunately this is sort of a deal-breaker for me right now.

[ikappas]

I am reposting my proposed solution for an isomorphic react app from #477 (comment)

The main concept is to use a placeholder at compile time that is injected at runtime just before server execution in order properly set runtime environment variables. This solution is for running the server in a docker container but could probably be adapted for this RFC.

Note that in this solution the RAZZLE_XXXX environment variables are matched and injected along with HOST, PORT and REDIS_URL.

---

I've personally struggled with this issue and spend several hours figuring out a solution to this issue.

This is inherent to webpack compilation and not related to razzle it self.

After looking into how create-react-app handles this, and porting some javascript and ruby code, across two projects, I've successfully deployed a razzle typescript react app in a docker container on heroku with the following solution:

env.ts

This script is used as a module to handle runtime env.

export interface EnvironmentStore {
  NODE_ENV?: string;
  [key: string]: string | undefined;
}

// Capture environment as module variable to allow testing.
let compileTimeEnv: EnvironmentStore;
try {
  compileTimeEnv = process.env as EnvironmentStore;
} catch (error) {
  compileTimeEnv = {};
  // tslint:disable-next-line no-console
  console.log(
    '`process.env` is not defined. ' +
    'Compile-time environment will be empty.'
  );
}

// This template tag should be rendered/replaced with the environment in production.
// Padded to 4KB so that the data can be inserted without offsetting character
// indexes of the bundle (avoids breaking source maps).
/* tslint:disable:max-line-length */
const runtimeEnv = '{{RAZZLE_VARS_AS_BASE64_JSON__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________}}';
/* tslint:enable:max-line-length */

// A function returning the runtime environment, so that
// JSON parsing & errors occur at runtime instead of load time.
export const loadRuntimeEnv = (): EnvironmentStore => {
  let env;
  if (typeof env === 'undefined') {
    if (compileTimeEnv.NODE_ENV === 'production') {
      try {
        env = JSON.parse((Buffer.from(runtimeEnv.trim(), 'base64').toString()));
      } catch (error) {
        env = {};
        const overflowsMessage = runtimeEnv.slice(32, 33) !== null;
        // tslint:disable-next-line no-console
        console.error(
          'Runtime env vars cannot be parsed. Content is `%s`',
          runtimeEnv.slice(0, 31) + (overflowsMessage ? '…' : '')
        );
      }

    } else {
      env = compileTimeEnv;
    }
  }
  return env;
};

export default loadRuntimeEnv;

usage:

import { loadRuntimeEnv, EnvironmentStore } from './env';
const env: EnvironmentStore = loadRuntimeEnv();

const serverHost: string =env.RAZZLE_SERVER_HOST || 'localhost';

docker-start.js

This script is used as the entrypoint point instead of server.js and is used to inject {{RAZZLE_VARS_AS_BASE64_JSON___... }} placehoder with the actual runtime environment variables.

require('newrelic');
const logger = require('heroku-logger');
const path = require('path');
const fs = require('fs');

const PLACEHOLDER = /\{\{RAZZLE_VARS_AS_BASE64_JSON_*?\}\}/;
const MATCHER = /^RAZZLE_/i;

const InjectableEnv = {

    inject: function(file, ...args) {

        const buffer = fs.readFileSync(file, { encoding: 'utf-8' });
        let injectee = buffer.toString();

        const matches = injectee.match(PLACEHOLDER);
        if (!matches) {
            return;
        }

        const placeholderSize = matches[0].length;

        let env = InjectableEnv.create(args);
        const envSize = env.length;
        const newPadding = placeholderSize - envSize;
        if (newPadding < 0) {
            console.log('You need to increase your placeholder size');
            process.exit();
        }
        const padding = Array(newPadding).join(' ');
        env = InjectableEnv.pad(padding, env);

        const injected = injectee.replace(PLACEHOLDER, env);

        fs.writeFileSync(file, injected, { encoding: 'utf-8' });
    },

    create: function() {

        const vars = Object.keys(process.env)
            .filter(key => MATCHER.test(key))
            .reduce((env, key) => {
                env[key] = process.env[key];
                return env;
            }, {});

        vars.NODE_ENV = process.env.NODE_ENV;

        if (typeof process.env.HOST !== 'undefined' && typeof vars.RAZZLE_SERVER_HOST === 'undefined') {
          vars.RAZZLE_SERVER_HOST = process.env.HOST;
        }

        if (typeof process.env.PORT !== 'undefined' && typeof vars.RAZZLE_SERVER_PORT === 'undefined') {
          vars.RAZZLE_SERVER_PORT = process.env.PORT;
        }

        if (typeof process.env.REDIS_URL !== 'undefined' && typeof vars.RAZZLE_REDIS_URL === 'undefined') {
          vars.RAZZLE_REDIS_URL = process.env.REDIS_URL;
        }

        return Buffer.from(JSON.stringify(vars)).toString('base64');
    },

    pad: function(pad, str, padLeft) {
        if (typeof str === 'undefined')
            return pad;
        if (padLeft) {
            return (pad + str).slice(-pad.length);
        } else {
            return (str + pad).substring(0, pad.length);
        }
    }
}

const root = process.cwd();
const serverBundle = path.resolve(path.join(root, '/build/server.js'));

if (fs.existsSync(serverBundle)) {
    logger.info('Injecting runtime env');
    InjectableEnv.inject(serverBundle);
    logger.info('Launching server instance');
    require(serverBundle);
}

Dockerfile

# You should always specify a full version here to ensure all of your developers
# are running the same version of Node.
FROM node:8.9.4

ENV NODE_ENV=production \
    REACT_BUNDLE_PATH=/static/js/vendor.js \
    PATH=/app/node_modules/.bin:$PATH \
    NPM_CONFIG_LOGLEVEL=warn

RUN curl -o- -L https://yarnpkg.com/install.sh | bash

# use changes to package.json to force Docker not to use the cache
# when we change our application's nodejs dependencies:
COPY package.json yarn.lock /tmp/
RUN cd /tmp \
  && yarn install --production=false --pure-lockfile \
  && mkdir -p /app \
  && cp -a /tmp/node_modules /app \
  && yarn cache clean \
  && rm -rf *.*

# From here we load our application's code in, therefore the previous docker
# "layer" thats been cached will be used if possible
WORKDIR /app
ADD . /app

RUN yarn build

EXPOSE 3000

CMD ["node", "docker-start.js"]

Please note that the overflow message processing is not finished, but I hope this helps

References:

Heroku Buildpack for create-react-app
Inner layer of Heroku Buildpack for create-react-app

[gregmartyn]

This is inherent to webpack compilation and not related to razzle it self.

Razzle is the one setting up DefinePlugin. This is solvable in Razzle.

I think I follow what you're saying. Tell me if I got this wrong: At build time, put placeholders into process.env that get string replaced on instance startup in the server build. It's meant to handle server secrets. (I don't see why it couldn't run on the client build too though) Problems: It won't work with HMR. It's a hack -- it introduces an arbitrary 4k boundary. In its current form, it doesn't address env vars that have to be shared with the client -- those remain build-time constants. It's an extra startup step for containers.

To rehash a lot of what I said in #528 (comment)

I think the solution is to recognize that Razzle and CRA are trying to pack more functionality into process.env than it should have. To get it to work with Docker, we're trying to have one object with fields having one of 4 possible states: static (build time) and dynamic (here, container start time), secrets and non-secrets. We could come up with prefixes for all 4 of those states (process.env.STATIC_PRIVATE_X, process.env.DYNAMIC_PUBLIC_Y, ...) but I think we'd be much better off with a cleaner solution.

If process.env were to behave the way it does natively -- as a store of server secrets -- then things are a lot easier to understand. There's one exception: NODE_ENV as a build-time inline, but that's fine because it's a property of the build. It wouldn't make sense to set NODE_ENV at runtime.

All that's left is a way to get data to the client. I don't see why this is using process.env at all. Why not use e.g. razzle.build.X for static stuff, and pass dynamic stuff to the client the same way redux does?

There's another issue where process.env is slow on Node, but that's best addressed with a cache layer that reads process.env once.

[ikappas]

@gregmartyn I agree that this is a hack ... and It does introduce an arbitrary 4K boundary. This idea is based on what is currently done with CRA (see the posted references) and is intended for server side runtime env variables.

[tgriesser]

Opened a PR that I believe should help with this root of this issue - env vars not being available on the server at runtime, interested to hear if this solves some of the issues here.

I also agree that PORT & HOST would also ideally be left alone at server compile time.

[gregmartyn]

@tgriesser nice! That is a big improvement.
In addition to PORT and HOST, I'd add PUBLIC_PATH to the list of vars that shouldn't be compiled in.

[krazyjakee]

I am still finding sensitive custom environment variables are all being compiled into the client. I am referencing them only in server.js. Does razzle consider that isomorphic because of the hotloader? How can I prevent these from reaching the client.

[jaredpalmer]

Hey everyone, I am tackling all of this at work this week. Stay tuned. #611 is likely to get merged.

[krazyjakee]

Following the new guide in the readme with config.js worked for me with regards to removing sensitive env variables from the bundle. Awesome :D

[jaredpalmer]

See v2 notes

[benhamlin]

@jaredpalmer I might be missing something, but this is still an issue for things like PORT, despite what the readme docs imply. process.env.MY_THING works fine, but process.env.PORT is still replaced at build time and are not read at runtime. So the Heroku example doesn't actually work.

I don't see any special handling of PORT, HOST, etc as discussed in this thread.

[gregmartyn]

Note that making PORT a real variable is also blocked by #581. I have to patch that and use a razzle.config.js that creates a DefinePlugin array that removes PORT in order to get it to work. (but it does work!)

[HamidTanhaei]

if anyone want to use .env variables in runtime, use this little package.
https://www.npmjs.com/package/razzle-plugin-runtimeenv

[vbutani]

Can someone please advise how to deploy Razzle app on Azure? I am really struggling with it.

[iberodev]

if anyone want to use .env variables in runtime, use this little package.
https://www.npmjs.com/package/razzle-plugin-runtimeenv

How does it work? Could you show an example?

I think environment variables should be injected at runtime indeed. If we conteinerize a razzle app, we would like to create an image independently of the environment it runs, and read the environment variables on starting the server and serve them to the client app then.

Any other approach is not really using environment variables since it's happening only during build time.

[HamidTanhaei]

As i mentioned here:
HamidTanhaei/razzle-plugin-runtime#1 (comment)

you can use your .env and .env.development files in the runtime by razzle-plugin-runtime. it adds ability to use your env variables in your app in runtime.

for example i'm using it to config axios:
axios.defaults.baseURL = ${process.env.RAZZLE_APP_API_BASE_PATH}${process.env.RAZZLE_APP_API_VERSION};
and you can provide production ENV variables for production like this:
https://github.com/jaredpalmer/razzle#adding-temporary-environment-variables-in-your-shell

[vbutani]

Can someone please advise how to deploy Razzle app on Azure? I am really struggling with it.

I resolved the Azure Port issue using the solution shared by @fabianishere at #906 (comment)