Version 4.0.0

This release of DADI Web focuses on performance improvements and simplifying of the codebase. Part of this development was to take inventory of the dependencies being loaded and we've removed some that added little benefit to the application.

The result of the changes in this release is an easier to maintain codebase and a faster experience for the end user.

Note: there is a breaking change in this release. See the Application launch section for details.

Installing the new release

See Installing for details regarding installing or upgrading your Web application.

Compression and Caching Changes

Brotli

Version 4.0 introduces Brotli as an alternative compression engine. Brotli provides, in real world testing, payloads around 20% smaller than GZip. If Brotli isn't supported by the client browser then DADI Web will fallback to GZip.

Useful reading

• https://engineering.linkedin.com/blog/2017/05/boosting-site-speed-using-brotli-compression
• https://hacks.mozilla.org/2015/11/better-than-gzip-compression-with-brotli/

Static assets

Static assets now obey configured compression settings; previously public folder assets were not subject to compression. Files will only be compressed if doing so will save space.

Compression configuration

To support the introduction of the new compression engine, the configuration setting for compression has changed. To enable compression in Version 4.0, use config.headers.useCompression rather than config.headers.useGzipCompression. The config.headers.useGzipCompression property is deprecated and will be removed in a future release.

Caching

Compressed responses and assets are now cached rather than being compressed (a relatively expensive operation) on each request. When using the local filesystem cache, files are given an extension that indicates both the type of file and the compression engine used.

For example

0869138a321114c3cfb23f10153e69f1cf810c0f.html.gzip
f869438f53115421c6624ff0568e79c1ff870601.css.br
959b1c900937674835fd97efd2f3bbf30d581ad6.svg.gzip

Cache configuration

Page caching is now on by default if caching is specified in the configuration. Page specification files no longer require cache: true for caching to be enabled.

Route processing

Version 4.0 performs route determination faster. In previous versions a request was tested against all loaded page components at the beginning of the request, and an array of matching routes was added to the middleware stack. In this version matching app-specific routes are loaded only if processing the middleware stack yields no matching handlers.

Request logging

Requests for static files are now passed through the request logger, giving more detailed access logs for the full request cycle.

Example

04:09:48.920Z  INFO dadi-web: GET / 200 10ms (module=router)
04:09:48.935Z  INFO dadi-web: GET /styles.css 200 5ms (module=router)
04:09:48.938Z  INFO dadi-web: GET /logo.png 200 8ms (module=router)
04:09:59.077Z  INFO dadi-web: GET /test.txt 200 3ms (module=router)
04:09:59.123Z  INFO dadi-web: GET /favicon.ico 200 2ms (module=router)

Security: CSRF tokens

Cross-Site Request Forgery (CSRF) is an attack that forces an end user to execute unwanted actions on a web application in which they're currently authenticated. CSRF attacks specifically target state-changing requests, not theft of data, since the attacker has no way to see the response to the forged request.

DADI Web 4.0 adds CSRF security, giving developers the ability to add a per-request CSRF token into the view context, and ensures that all POST requests supply a correct CSRF token. Without a correct token, and with CSRF enabled, users will be greeted with an HTTP 403 response.

To enable CSRF, set the security.csrf configuration option:

"security": {
  "csrf": true
}

Once enabled, the property csrfToken will be added to the view context. You will need to add this to any forms which perform a POST using the field name _csrf, like so:

<form action="/" method="post">
  <input type="text" name="test_input_safe">
  <input type="hidden" name="_csrf" value="{csrfToken}">
  <input type="submit" value="Submit form">
</form>

Application launch

BREAKING CHANGE!

Launching the application now returns a Promise which, when resolved, returns an object containing the application instance and the loaded route/page components.

// start the application
require('@dadi/web')({
  "engines":[
    require("@dadi/web-dustjs")
  ]
}).then(loaded => {
  console.log(loaded.App)
  console.log(loaded.Components)
})

This change replaces the exported modules in previous versions. To obtain a reference to these modules when the application has already started (for example when loading template helpers), require @dadi/web without passing an engine argument:

require('@dadi/web')().then(loaded => {
  console.log(loaded.App)
  console.log(loaded.Components)
})

Other Changes

• Removed support for event-logging system "Sentry". This feature was untested and unused
• Added new middleware to serve content from the public folder, removing dependency on Express.js modules serve-static and serve-favicon.
• Moved helper methods sendBackJSON, sendBackHTML into view/send.js
• Removed unused helper sendBackJSONP
• Removed outdated/unused media path.
• Refactor of cache flush under api/flush. Added corresponding error page when method is not POST.
• Added npm run format to run for standard & prettier
• Hide the err.stack from default error pages when the NODE_ENV environment variable is production (NODE_ENV=production)
• An improved developer experience: changes to event files & template partials/includes reinitialises the application without requiring a restart.

Resolved issues

• #51: cache flush command fails when no matching page is found
• #59: add CSRF token
• #158: compress response before caching
• #168: process routes after middleware
• #173: listener should trigger a 302 redirect
• #174: introduce Brotli compression
• #175: remove 'server' response header
• #193: reload templates and event files when changed on disk (without restarting app)
• #212: fix default workspace config error

Installing

In conjunction with this release of DADI Web we're also pleased to announce a new CLI tool for installing and interacting with DADI applications. Although CLI will evolve over time, it currently only supports installing new API, CDN and Web applications. If you already have an installed API, CDN or Web application and only wish to upgrade the product, see the next section "Installing/Upgrading from NPM".

Installing Web via CLI:

1. Install DADI CLI

$ npm install @dadi/cli -g

2. Change the directory to the location where you'd like your Web installation to be:

$ cd /Users/JohnDoe/Sites

3. Create a new Web installation. The following will automatically create a my-web directory and install Web within that.

$ dadi web new my-web

Note: This command will launch an interactive prompt asking you which template engines you'd like to install. If you know that beforehand, you can skip that step by providing the names of the template engine modules as parameters.

Installing Web with Dust and Pug template engines

$ dadi web new my-web --engine=@dadi/web-dustjs --engine=@dadi/web-pugjs

4. Enter the my-web directory and start the new application:

$ cd my-web
$ npm start

Installing/Upgrading from NPM

1. Navigate to an existing directory

$ cd /Users/JaneDoe/Sites/my-web

2. Install or update @dadi/web directly.

$ npm install/update @dadi/web

Version 3.1.0

Added

• #59: add support for CSRF token security. Usage information here

Changed

• #204: remove Sentry support in favour of a future error handing implementation
• #209: wait for components and routes to be loaed before exporting

Version 3.0.2

Changed

• throw error at startup when the pages directory contains templates without an engine that can handle them
• fix an issue where information about the loaded engines on the app startup screen was sometimes incorrect

Version 3.0.1

Changed

• add https://github.com/dadi/web-dustjs as dependency
• modify post install script to include the above plugin when creating the server.js file

Version 3.0.0

Breaking changes

Some of the features/changes listed below are from the release of Version 2.0. They are included here for those that are upgrading directly from Version 1.x

Multiple Template Engine Support

The biggest change in this version is support for multiple template engines. As a consequence, Dust.js is now decoupled from Web core and must be included as a dependency on projects that want to use it.

@eduardoboucas has written a handy migration guide for Version 3.0.

Additional template engine plugins will be released at a later date, along with a guide to developing one for your favourite engine, whether it's Pug, Handlebars or Liquid.

Datasource type dadiapi replaces remote

Previous to Version 2.0 the datasource source type for connecting to a DADI API was called remote. This has changed to dadiapi to ensure clarity with the updated and repurposed Remote provider.

A typical datasource specification file would now contain the following:

"source": {
  "type": "dadiapi",
  "endpoint": "1.0/articles"
}

N.B. The default is dadiapi, so there is no requirement to specify this property when connecting to a DADI API.

New features

Custom configuration environments

#184: Environments (and configuration files) are no longer limited to production, development, test and qa.

Inject request parameters into datasource endpoints

You can now pass requestParams to a datasource endpoint's URL by specifying a new setting "target": "endpoint". The param value will be injected into a placeholder in the URL identified by the field value. For example:

{
  "datasource": {
    "key": "instagram",
    "source": {
      "type": "remote",
      "protocol": "http",
      "host": "instagram.com",
      "endpoint": "{userPlaceholder}/?__a=1"
    },
    "auth": false,
    "requestParams": [
      {
        "param": "user", // from the request parameters, e.g. http://www.example.com/profile/:user
        "field": "userPlaceholder",
        "target": "endpoint"
      }
    ]
  }
}

Data Providers

Remote Provider

Connect to a miscellaneous API via HTTP or HTTPS. See the following file as an example: workspace/datasources/instagram.json

Markdown Provider

Serve content from a local folder containing text files. You can specify also specify the extension to grab. Web will process any Markdown formatting (with Marked) it finds automatically as well as any Jekyll-style front matter found. Any dates/times found will be processed through JavasScript’s Date() function.

{
  "datasource": {
    "source": {
      "type": "markdown",
      "path": "./workspace/posts",
      "extension": "md"
    }
  }
}

workspace/posts/somefolder/myslug.md

---
date: 2016-02-17
title: Your title here
---
Some *markdown*

When loaded becomes the following data:

{
  "attributes": {
    "date": "2016-02-17T00:00:00.000Z",
    "title": "Your title here",
    "_id": "myslug",
    "_ext": ".md",
    "_loc": "workspace/posts/somefolder/myslug.md",
    "_path": [
      "somefolder"
    ]
  },
  "original": "---\ndate: 2016-02-17\ntitle: Your title here\n---\nSome *markdown*",
  "contentText": "Some *markdown*",
  "contentHtml": "<p>Some <em>markdown</em></p>\n"
}

NB. _path will exclude the datasource source.path.

Virtual Hosts

DADI Web can now serve content to multiple hostnames from a single installation. Only certain configuration options can be modified for each hostname; these are currently restricted to paths (for pages/partials/datasources/etc) and global.

"virtualHosts": {
  "one": {
    "hostnames": ["domain1.dev"],
    "configFile": "one.qa.json",
    "default": true
  },
  "two": {
    "hostnames": ["domain2.dev"],
    "configFile": "two.qa.json"
  }
}

Changed

• #98: log more informative error when an event fails

Version 1.11.2

Changed

• modify the controller to make it initialise each datasource's data provider just before it is used, then destroy it after the data is returned

Version 1.11.0

Under heavy load, the singleton Web datasource cache instance could intermittently write data to the cache under a different key than anticipated. This version reverts an earlier change to make this a singleton, and now once again gives each data provider it's own cache instance.

New features

• datasource endpoints can now include querystring parameters, additional API parameters will be added to the end rather than replacing what is there

Changed

• revert back to individual cache instances for pages & datasources
• fix: allow datasource cache options to override main cache options, so datasources can specify their own TTL value

Installation notes

Installing from NPM using the command npm install @dadi/web will by default install the version with the "latest" tag. Currently, the "latest" version is 2.0.1 To install this version (1.11.0) use the command npm install @dadi/web@one, where "one" is the tag for the Version 1.x branch.

Version 2.0.0

Breaking changes

DADI API connection disabled by default

You now have to explicitly set the API config block to be enabled, previous versions assumed this value was true if not present in the configuration file. If connecting Web to a DADI API, the configuration file must now contain api settings similar to the following:

"api": {
  "enabled": true,
  "host": "127.0.0.1",
  "port": 8080
}

Datasource type dadiapi replaces remote

Previous to 2.0 the datasource source type for connecting to a DADI API was called remote. This has changed to dadiapi to ensure clarity with the updated and repurposed Remote provider.

A typical datasource specification file would now contain the following:

"source": {
  "type": "dadiapi",
  "endpoint": "1.0/articles"
}

The default is dadiapi, so there is no requirement to specify this property when connecting to a DADI API.

New Features

Default workspace

#133: Web will boot straight out of the box with a default ‘blog’ style configuration and some suggestions of next steps. The site is mobile responsive and uses the new Markdown provider datasource to serve content from the workspace/posts folder.

Better errors

#150, #152: We now have more human readable error messages out the box so your users don’t ever just receive plain JSON back. You can still override them with your own in the normal way.

Inject request parameters into datasource endpoints

You can now pass requestParams to a datasource endpoint's URL by specifying a new setting "target": "endpoint". The param value will be injected into a placeholder in the URL identified by the field value. For example:

{
  "datasource": {
    "key": "instagram",
    "source": {
      "type": "remote",
      "protocol": "http",
      "host": "instagram.com",
      "endpoint": "{userPlaceholder}/?__a=1"
    },
    "auth": false,
    "requestParams": [
      {
        "param": "user", // from the request parameters, e.g. http://www.example.com/profile/:user
        "field": "userPlaceholder",
        "target": "endpoint"
      }
    ]
  }
}

Post install

After running npm install Web will now create default workspace and config folders in your project directory. A server.js file will also be created which will let you boot the app immediately by running npm start.

The script will not overwrite existing workspace or config folders, or an existing server.js file.

Data Providers

Remote Provider

Connect to a miscellaneous API via HTTP or HTTPS. See the following file as an example: workspace/datasources/instagram.json

Markdown Provider

Serve content from a local folder containing text files. You can specify also specify the extension to grab. Web will process any Markdown formatting (with Marked) it finds automatically as well as any Jekyll-style front matter found. Any dates/times found will be processed through JavasScript’s Date() function.

{
  "datasource": {
    "source": {
      "type": "markdown",
      "path": "./workspace/posts",
      "extension": "md"
    }
  }
}

workspace/posts/somefolder/myslug.md

---
date: 2016-02-17
title: Your title here
---
Some *markdown*

When loaded becomes the following data:

{
  "attributes": {
    "date": "2016-02-17T00:00:00.000Z",
    "title": "Your title here",
    "_id": "myslug",
    "_ext": ".md",
    "_loc": "workspace/posts/somefolder/myslug.md",
    "_path": [
      "somefolder"
    ]
  },
  "original": "---\ndate: 2016-02-17\ntitle: Your title here\n---\nSome *markdown*",
  "contentText": "Some *markdown*",
  "contentHtml": "<p>Some <em>markdown</em></p>\n"
}

NB. _path will exclude the datasource source.path.

Virtual Hosts

DADI Web can now serve content to multiple hostnames from a single installation. Only certain configuration options can be modified for each hostname; these are currently restricted to paths (for pages/partials/datasources/etc) and global.

"virtualHosts": {
  "one": {
    "hostnames": ["domain1.dev"],
    "configFile": "one.qa.json",
    "default": true
  },
  "two": {
    "hostnames": ["domain2.dev"],
    "configFile": "two.qa.json"
  }
}

Changed

• #128: attach compression middleware prior to static middleware
• #130: leave url params unmodified when lowercasing urls
• #139 remove datasource path from attributes
• #144: check host header against specified hosts before serving static files
• don’t modify original schema endpoint (4901ecc)
• rename sample-workspace to workspace (c307b8e)

v1.10.0

<a name"1.10.0">

1.10.0 (2017-02-25)

Fixes

• rebuild chained datasource endpoints after applying results to the filter from the datasource it is chained to (9a1aef8)

v1.9.0

<a name"1.9.0">

1.9.0 (2017-02-13)

Features

• start in cluster mode by default (5987dd2d)
• allow array of partial paths (bbb226d9)
• add debug statements instead of log statements (e4250401)