Asynchronous Custom Plugins Startup

[asnowfix]

TL;DR

The EMG custom plugin API does not allow safe initialization of async (remote or not) resources.

Problem

Custom EMG plugins provide an init() entry point, which synchronously returns a predefined list of other entry points dealing with HTTP traffic. As soon as init() returns, the plugin is expected to be able to deal with inbound HTTP traffic. However, initialization of some resources might require some time. This is the case for example to get a remote public key (to check for JWT token validity), or to open a local database.

Actually, most of the Node.js API being asynchronous, the EMG init() synchronous call forces us to be able to deal with early race condition, while this could be easily avoided.

module.exports.init = function(config, logger, stats) {
  return {
    ondata_response: function(req, res, data, next) {
      debug('***** plugin ondata_response');
      next(null, null);
    },
    onend_response: function(req, res, data, next) {
      debug('***** plugin onend_response');
      next(null, "Hello, World!\n\n");
    }
  };
}

Our Recommendation

Add an optional 4th parameter "next" to the init() call, being a Node.js callback:

1. If absent (function arity === 3), then init() synchronously returns the expected list of HTTP handler. This is the current API & the current behavior: it maintains backward compatibility.
2. If present (function arity === 4), init() returns nothing and calls next() with error or data payload, depending on the initialization success, according to the Node.js callback model.

Example custom plugin source code using async init():

 module.exports.init = function(config, logger, stats, next) {
  db.open('./mydb.sql', function(err, mydb) {
    if (err) {
      next(err); 
    } else {
      next(null, {
        ondata_response: function(req, res, data, next) {
          debug('***** plugin ondata_response');
          next(null, null);
        },
        onend_response: function(req, res, data, next) {
          debug('***** plugin onend_response');
         next(null, "Hello, World!\n\n");
        }
      }
   });
};

[f1erro]

Nice idea, let's get this into the next release @mdobson

[mdobson]

We'll keep this in mind for upcoming backlog discussions.

[dschniepp]

Any update on this topic, I guess it seems still to be an issue?