A redis-backed manager of recurrent jobs, for node.js.
(immature, with minimal testing and probably nasty bugs)
- multiple queues, backed up by redis
- tasks have a permanent unique id and mutable associated data
- failed job runs are retried until they succeed or give up, with exponential back-off
- job retries and repetitions don't pile up
- the worker's job implementation determines at the end of each run when (and whether) to repeat
- as many worker processes as you want spread over multiple cores and machines - but you have to start them
- single point of failure in queue manager (as well as redis itself) - you need to make sure it stays up / restarts quickly and there's exactly 1 instance
npm install recurrent
Make sure redis is running and then:
var recurrent = require('recurrent'); // will manage a queue q var m = new recurrent.Manager('q').connect(); # pass arguments for redis.createClient() to connect()
recurrent
offers a (currently very incomplete) web UI:
http.createServer(m.webUI).listen(7654);
And then browse to localhost:7654
When the manager is not needed anymore:
m.stop()
var recurrent = require('recurrent'); // starts a job which will run for the first time in about 30s var c = new recurrent.Client('q').connect(); # pass arguments for redis.createClient() to connect() c.add('t1', new Date().getTime() + 30000, {my: 'data'}, function(err) { ... });
Adding again the same taskId
will reset execution time.
When the client is not needed any more:
c.stop();
var recurrent = require('recurrent'); function doWork(task, cb) { // do nothing for 600s console.log('got', task.id, 'with this data: ', task.data); setTimeout(function () { console.error('completed', task.id); // do again in about 5s cb(null, new Date().getTime()+ 5000); }, 600); } var w = new recurrent.Worker('q', doWork).connect(); # pass arguments for redis.createClient to connect()
When the job worker is not needed any more:
w.stop();
Workers must not throw.
To signal failure a worker calls back with error. It can specify a retry time:
cb('something went wrong', new Date().getTime()+ 1000); // retry in 1s
Or let recurrent
do exponential back-off
cb('something went wrong');
Exponential back-off can be configured per worker:
var w = new recurrent.Worker('q', doWork, { minBackOff: 500, // start backing off at 500 ms maxBackOff:5000, // max back-off of 5 s backOffMultiplier:1.5 // back-off 50% longer every time }).connect();
When a task is being retried after failure task.retries
contains how many retries have been attempted (including the current one).
It will undefined
for the first execution, 1
for the first retry and so on.