Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🚀 Performance enhancements #196

Merged
merged 127 commits into from
Aug 31, 2017
Merged
Show file tree
Hide file tree
Changes from 106 commits
Commits
Show all changes
127 commits
Select commit Hold shift + click to select a range
124a5ef
feat: http2 #155
May 7, 2017
79ff549
chore: spelling
May 7, 2017
781b938
feat: simplify http2 definition
May 8, 2017
ea6f6bc
chore: update config doc for globalPushManifest
May 9, 2017
13b0542
feat: switch http2 for spdy
May 9, 2017
75bee8d
feat: add support for pushManifest #155
May 9, 2017
e2ba87d
style: code readability
May 9, 2017
6aab91b
feat: wait for readStream before pushing
May 9, 2017
b6a510a
chore: remove debug package
May 9, 2017
266f313
chore: add default image caching
May 9, 2017
439b7b8
chore: remove media folder
May 9, 2017
d9807ec
fix: debug message update
May 10, 2017
c829e41
fix: print server name when http2
May 10, 2017
7120446
feat: http2 add priority support & cleanup
May 10, 2017
a9f3709
feat: check stream exists
May 10, 2017
068c3ba
fix: remove push tidy
May 10, 2017
ac8b993
feat: test for compressible
May 10, 2017
a68de6d
fix: use http2 & disable gzip
May 10, 2017
582d03e
chore: disable gzip
May 10, 2017
ba04050
style: single quotes
May 11, 2017
0a21c46
feat: new servePublic middleware
May 11, 2017
60b4cc2
fix: remove unused code
May 11, 2017
98b8933
chore: organise
May 11, 2017
a211d16
feat: http2 asset compression & push
May 11, 2017
d9b479e
chore: remove titlecase on mobile name
May 11, 2017
53efa6f
Merge branch 'feature/process-routes-after-middleware' of https://git…
May 22, 2017
5fd8710
feat: combine asset serving function #167
May 22, 2017
c60faeb
feat: only allow GET and HEAD requests on assets
May 22, 2017
d914b57
fix: remove GET and HEAD restrictions
May 23, 2017
a71c2dd
feat: use http2 for https support
May 23, 2017
5ba78aa
feat: public folder tests
May 23, 2017
a67cfe7
fix: edge cases
May 23, 2017
a2bcd43
feat: explicity when public is run as middleware
May 23, 2017
e4b372f
feat: test for http2 push
May 23, 2017
aa2e210
feat: add public folder to test helper
May 23, 2017
ae02a2d
chore: tidy unused public functions
May 23, 2017
b369359
feat: tests for asset serving
May 23, 2017
72cea7b
fix: catch test errors
May 23, 2017
5af2889
fix: h2 test
May 23, 2017
86d1e4d
fix: h2 test
May 23, 2017
ebc8326
fix: test timeout
May 24, 2017
0f2d645
fix: skip test
May 24, 2017
ea87d5a
fix: servePublic middleware
May 25, 2017
890285c
fix: servePublic middleware
May 25, 2017
d6bafe2
fix: servePublic middleware
May 25, 2017
8583706
fix: set headers once and sterilize file path
May 25, 2017
3065a92
fix: set etag correctly
May 25, 2017
b684a8f
fix: add missing destroy package
May 26, 2017
7013ada
feat: spdy
May 29, 2017
ce7f37e
fix: header order
May 29, 2017
55c4b44
feat: simplify error trap
May 30, 2017
6b879b6
fix: let to var
May 30, 2017
46929be
feat: response compression #174
May 30, 2017
2812a5f
fix: tests (supertest has no br compatability)
May 30, 2017
73723a4
fix: node-uuid warning on npm install
May 30, 2017
2db16e3
feat: cache gzip/br page response
May 31, 2017
ed57291
feat: add canCompress helper
May 31, 2017
655e3bd
chore: tidy
May 31, 2017
587aac7
feat: cache file extensions
May 31, 2017
37f88ac
chore: tidy unused vars
May 31, 2017
4c7b263
feat: init cache before serving assets
May 31, 2017
f44b8a0
feat: add etag to cache files
May 31, 2017
7fee407
chore: tidy unused css
May 31, 2017
579bb06
chore: grammar
May 31, 2017
cdd63e3
fix: test
May 31, 2017
7c1e89d
fix: check cache is a function
May 31, 2017
9f1320c
fix: simplfy trail slash removal
May 31, 2017
3e154a7
feat: change redirect status code #173
Jun 1, 2017
94ab1c4
feat: cache file extensions
Jun 1, 2017
32ef5b2
fix: pass cache layer to clearCache helper
Jun 1, 2017
e971621
fix: catch cache buffer errors
Jun 1, 2017
fa32d6f
chore: remove console.log
Jun 1, 2017
0d1f52f
fix: pass current cache instance to helper
Jun 2, 2017
e519642
fix: cache flush test
Jun 2, 2017
e1a8abc
fix: update redirect test #173
Jun 2, 2017
e21d0eb
fix: update redirect test #173
Jun 2, 2017
1c0aa91
feat: primative br test
Jun 2, 2017
485cb30
feat: test for accept-encoding
Jun 2, 2017
c19a127
chore: coverage
Jun 2, 2017
0c40dcf
chore: fix double var
Jun 2, 2017
da37380
feat: make http2 optional
Jun 6, 2017
9500578
fix: remove noisy console.log
Jun 6, 2017
1502eab
feat: add content-length headers to cache file
Jun 6, 2017
c21d04f
feat: add content-length to all responses
Jun 6, 2017
219d1f2
Merge branch 'feature/http2' of https://github.com/dadi/web into feat…
Jun 6, 2017
0f2f40e
style: spacing
Jun 7, 2017
eec93f7
feat: add default long cache for favicon
Jun 9, 2017
1c37dc8
feat: cache support
Jun 9, 2017
93f9039
chore: remove console.log
Jun 9, 2017
03fa5d5
fix: blocking file processing
Jun 9, 2017
4553011
Merge branch 'master' of https://github.com/dadi/web into feature/per…
Aug 8, 2017
23fdb18
feat: remove http2 support
Aug 8, 2017
cbfb340
feat: serve compressed version when cache miss
Aug 8, 2017
f4f6de8
fix: snon-test not compatible with sinon latest
Aug 8, 2017
55e197a
chore: tests
Aug 8, 2017
9b554ac
fix: cache file generated too soon
Aug 9, 2017
07f0edd
feat: cache enabled by default
Aug 9, 2017
8a3d586
fix: check datasource.provider exists
Aug 9, 2017
e202a8c
fix: allow compression when caching is disabled
Aug 9, 2017
e721c2a
fix: add missing dust dependency
Aug 9, 2017
5dcf1f7
feat: update instanbul
Aug 9, 2017
610db4c
feat: tests
Aug 9, 2017
9bb1d85
fix: reapply #168
Aug 10, 2017
b4db30b
fix: reapply #168
Aug 10, 2017
e5f1e77
fix: reapply #168
Aug 10, 2017
8c7bc12
fix: reapply #168
Aug 10, 2017
e533020
fix: cache load order
Aug 10, 2017
b5196e7
feat: add ‘format’ script for standard & prettier
Aug 10, 2017
79f898b
style: commas
Aug 10, 2017
858ce97
fix: remove unused pushManifest
Aug 10, 2017
a5aef7c
fix: robot speak 🤖
Aug 10, 2017
6923fee
chore: tidy unused debug code
Aug 10, 2017
6e74b8b
style: comma
Aug 10, 2017
c904ce5
chore: pass arguments object
Aug 11, 2017
70ebe60
fix: expect and use object not array
jimlambie Aug 14, 2017
4e73396
Merge pull request #207 from dadi/feature/monitor
Aug 24, 2017
385262b
fix: virtualDirectories
Aug 29, 2017
2057862
Merge branch 'feature/performance' of https://github.com/dadi/web int…
Aug 29, 2017
0682fe2
Merge branch 'master' into feature/performance
Aug 29, 2017
8f0856c
chore: update iltorb
Aug 30, 2017
2fbbe50
Merge branch 'feature/performance' of https://github.com/dadi/web int…
Aug 30, 2017
192ffac
fix: virtualDirectories folder index
Aug 31, 2017
6015587
Merge branch 'master' of https://github.com/dadi/web into feature/per…
Aug 31, 2017
e9a42f0
chore: const
Aug 31, 2017
a65ac40
test: virtualDirectories
Aug 31, 2017
1a0a016
fix: apply router before servePublic middleware
Aug 31, 2017
e9cdecb
fix: add missing events path #212
Aug 31, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 49 additions & 23 deletions config.js
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,8 @@ var conf = convict({
env: 'SSL_INTERMEDIATE_CERTIFICATE_PATH'
},
sslIntermediateCertificatePaths: {
doc: 'The filenames of SSL intermediate certificates, overrides sslIntermediateCertificate (singular)',
doc:
'The filenames of SSL intermediate certificates, overrides sslIntermediateCertificate (singular)',
format: Array,
default: [],
env: 'SSL_INTERMEDIATE_CERTIFICATE_PATHS'
Expand Down Expand Up @@ -201,7 +202,8 @@ var conf = convict({
},
redis: {
enabled: {
doc: 'If enabled, cache files will be saved to the specified Redis server',
doc:
'If enabled, cache files will be saved to the specified Redis server',
format: Boolean,
default: false
},
Expand Down Expand Up @@ -232,17 +234,27 @@ var conf = convict({
},
headers: {
useGzipCompression: {
doc: "If true, uses gzip compression and adds a 'Content-Encoding:gzip' header to the response.",
doc: 'Depricated: use `useCompression` instead.',
format: Boolean,
default: true
},
useCompression: {
doc:
"If true, uses br or gzip compression where available and adds a 'Content-Encoding: [br|gzip]' header to the response.",
format: Boolean,
default: true
},
cacheControl: {
doc: "A set of custom cache control headers for different content types. For example 'cacheControl': { 'text/css': 'public, max-age=1000' }",
doc:
"A set of custom cache control headers for different content types. For example 'cacheControl': { 'text/css': 'public, max-age=1000' }",
format: Object,
default: {
'image/png': 'public, max-age=86400',
'image/jpeg': 'public, max-age=86400',
'text/css': 'public, max-age=86400',
'text/javascript': 'public, max-age=86400',
'application/javascript': 'public, max-age=86400'
'application/javascript': 'public, max-age=86400',
'image/x-icon': 'public, max-age=31536000000'
}
}
},
Expand Down Expand Up @@ -274,7 +286,8 @@ var conf = convict({
},
accessLog: {
enabled: {
doc: "If true, HTTP access logging is enabled. The log file name is similar to the setting used for normal logging, with the addition of 'access'. For example `web.access.log`.",
doc:
"If true, HTTP access logging is enabled. The log file name is similar to the setting used for normal logging, with the addition of 'access'. For example `web.access.log`.",
format: Boolean,
default: true
},
Expand All @@ -286,7 +299,8 @@ var conf = convict({
},
sentry: {
dsn: {
doc: "The 'DSN' to use for logging errors and events to a Sentry server. It should be similar to 'https://693ef18da3184cffa82144fde2979cbc:[email protected]/59524'.",
doc:
"The 'DSN' to use for logging errors and events to a Sentry server. It should be similar to 'https://693ef18da3184cffa82144fde2979cbc:[email protected]/59524'.",
format: String,
default: ''
}
Expand All @@ -307,8 +321,6 @@ var conf = convict({
format: Object,
default: {
datasources: path.join(__dirname, '/workspace/datasources'),
events: path.join(__dirname, '/workspace/events'),
media: path.join(__dirname, '/workspace/media'),
middleware: path.join(__dirname, '/workspace/middleware'),
pages: path.join(__dirname, '/workspace/pages'),
public: path.join(__dirname, '/workspace/public'),
Expand All @@ -328,29 +340,34 @@ var conf = convict({
default: 'dadiweb.sid'
},
secret: {
doc: 'This is the secret used to sign the session ID cookie. This can be either a string for a single secret, or an array of multiple secrets. If an array of secrets is provided, only the first element will be used to sign the session ID cookie, while all the elements will be considered when verifying the signature in requests.',
doc:
'This is the secret used to sign the session ID cookie. This can be either a string for a single secret, or an array of multiple secrets. If an array of secrets is provided, only the first element will be used to sign the session ID cookie, while all the elements will be considered when verifying the signature in requests.',
format: String,
default: 'dadiwebsecretsquirrel',
env: 'SESSION_SECRET'
},
resave: {
doc: 'Forces the session to be saved back to the session store, even if the session was never modified during the request.',
doc:
'Forces the session to be saved back to the session store, even if the session was never modified during the request.',
format: Boolean,
default: false
},
saveUninitialized: {
doc: "Forces a session that is 'uninitialized' to be saved to the store. A session is uninitialized when it is new but not modified.",
doc:
"Forces a session that is 'uninitialized' to be saved to the store. A session is uninitialized when it is new but not modified.",
format: Boolean,
default: false
},
store: {
doc: 'The session store instance, defaults to a new MemoryStore instance.',
doc:
'The session store instance, defaults to a new MemoryStore instance.',
format: String,
default: ''
},
cookie: {
maxAge: {
doc: "Set the cookie’s expiration as an interval of seconds in the future, relative to the time the browser received the cookie. Null means no 'expires' parameter is set so the cookie becomes a browser-session cookie. When the user closes the browser the cookie (and session) will be removed.",
doc:
"Set the cookie’s expiration as an interval of seconds in the future, relative to the time the browser received the cookie. Null means no 'expires' parameter is set so the cookie becomes a browser-session cookie. When the user closes the browser the cookie (and session) will be removed.",
format: '*',
default: 60000
},
Expand Down Expand Up @@ -393,7 +410,8 @@ var conf = convict({
default: false
},
stripIndexPages: {
doc: "A set of filenames to remove from URLs. For example ['index.php', 'default.aspx']",
doc:
"A set of filenames to remove from URLs. For example ['index.php', 'default.aspx']",
format: Array,
default: []
},
Expand All @@ -405,12 +423,14 @@ var conf = convict({
},
security: {
trustProxy: {
doc: 'If true, trusts the values specified in X-Forwarded-* headers, such as protocol and client IP address',
doc:
'If true, trusts the values specified in X-Forwarded-* headers, such as protocol and client IP address',
format: '*',
default: true
},
transportSecurity: {
doc: 'If true, requires requests to be secure. Overridden if server.protocol is set to https.',
doc:
'If true, requires requests to be secure. Overridden if server.protocol is set to https.',
format: '*',
default: false
}
Expand All @@ -428,23 +448,27 @@ var conf = convict({
default: {}
},
virtualDirectories: {
doc: 'Allows specifying folders where additional static content may reside. An array entry should like look { "path": "data/legacy_features", "index": "default.html", "forceTrailingSlash": false }',
doc:
'Allows specifying folders where additional static content may reside. An array entry should like look { "path": "data/legacy_features", "index": "default.html", "forceTrailingSlash": false }',
format: Array,
default: []
},
allowJsonView: {
doc: 'If true, allows appending ?json=true to the querystring to view the raw JSON output for each page.',
doc:
'If true, allows appending ?json=true to the querystring to view the raw JSON output for each page.',
format: Boolean,
default: false
},
toobusy: {
enabled: {
doc: 'If true, server will respond with HTTP 503 if the server is deemed too busy.',
doc:
'If true, server will respond with HTTP 503 if the server is deemed too busy.',
format: Boolean,
default: false
},
maxLag: {
doc: "The maximum amount of time in milliseconds that the event queue is behind before we consider the process 'too busy'.",
doc:
"The maximum amount of time in milliseconds that the event queue is behind before we consider the process 'too busy'.",
format: Number,
default: 70
},
Expand All @@ -455,12 +479,14 @@ var conf = convict({
}
},
cluster: {
doc: 'If true, Web runs in cluster mode, starting a worker for each CPU core',
doc:
'If true, Web runs in cluster mode, starting a worker for each CPU core',
format: Boolean,
default: true
},
debug: {
doc: 'If true, debug mode is enabled and a panel containing the JSON loaded for each page is displayed alongside the normal content.',
doc:
'If true, debug mode is enabled and a panel containing the JSON loaded for each page is displayed alongside the normal content.',
format: Boolean,
default: false
},
Expand Down
11 changes: 5 additions & 6 deletions dadi/lib/api/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,7 @@ var Api = function () {
this.protocol = config.get('server.protocol') || 'http'
this.redirectPort = config.get('server.redirectPort')

if (this.protocol === 'http') {
this.httpInstance = http.createServer(this.listener)
} else if (this.protocol === 'https') {
if (this.protocol === 'https') {
// Redirect http to https
if (this.redirectPort > 0) {
this.redirectInstance = http.createServer(this.redirectListener)
Expand Down Expand Up @@ -92,6 +90,8 @@ var Api = function () {
throw new Error(exPrefix + ex.message)
}
}
} else {
this.httpInstance = http.createServer(this.listener)
}
}

Expand Down Expand Up @@ -227,7 +227,7 @@ Api.prototype.listener = function (req, res) {
// if end of the stack, no middleware could handle the current
// request, so get matching routes from the loaded page components and
// add them to the stack just before the 404 handler, then continue the loop
// console.log(pathsLoaded, this.stack[stackIdx].name, this.stack[stackIdx+1] ? this.stack[stackIdx+1].name : 'nothing left!')

if (
this.stack[stackIdx + 1] &&
this.stack[stackIdx + 1].name === 'notFound' &&
Expand All @@ -250,7 +250,6 @@ Api.prototype.listener = function (req, res) {
_.each(matches, match => {
this.stack.splice(-1, 0, match)
})
} else {
}

pathsLoaded = true
Expand Down Expand Up @@ -289,7 +288,7 @@ Api.prototype.redirectListener = function (req, res) {
var location = 'https://' + hostname + ':' + port + req.url

res.setHeader('Location', location)
res.statusCode = 301
res.statusCode = 302
res.end()
}

Expand Down
Loading