Cached job code?

[jeffdupont]

I'm getting a strange issue within my production code dealing with my Kue job. I've written a job that is set to delay and while testing I had manually set a value within the data. Now I'm attempting to clear out the manually set variable, but it doesn't seem to clear out and reset to the correct value. I was watching the value in the job.log() and everything seems to work fine. The job is delayed and run as I expect it, but the code for the job running seems to run old code. I've flushed out everything in redis and restarted the job server without success. I can even completely change the code in the job and it still shows the log from the old code. I can confirm the log inside the new redis key/values that it is logging the old code.

Sorry if this is a little confusing, it's fairly late and I've been going through this issue for a few hours now. I'm hoping that it's something really simple.
Thanks!

[behrad]

but the code for the job running seems to run old code.

I'm not sure I'm getting it right. if your process code is an old one running, it is nothing to do with Redis, but also your node.js process not being killed and restarted, ...

If you Kue job data on redis is not being updated, then I should have a sample code that is reproducing the problem.

[jeffdupont]

The node process running the Kue job server and the job workers are restarted. I have other node processes that must remain up in the environment, but they have no need or reference for the job server. If I add a brand new job worker process to the code in the same files as the other worker processes, it loads up and runs as expected.

[behrad]

I still have not understood what your problem is @jeffdupont

[jeffdupont]

The code on my job server is not running the latest code on my worker process. I've cleared out and restarted all node processes on the server and it still hasn't helped.

[behrad]

We have nothing as job server. the only roles are redis server + node.js worker processes

[jeffdupont]

I had to completely rename the job.create('job_name') in order to get my job process to function. Not what I wanted to do.

I'm not sure how to get this to replicate since I haven't found anyone else with the issue, and I can't replicate it myself in my local develop. The Kue is running on AWS using the ElasticCache as the redis store. I've flushed out all keys to make sure that there was nothing left to reference. I'm running a basic EC2 instance in CentOS.

[jeffdupont]

My job-server is what connects and sets up the redis and worker processes:

var Jobs = Kue.createQueue({
  redis: {
    createClientFactory: function() {
      return Redis.createClient(Config.redis_port, Config.redis_host);
    }
  }
});
// Job Handlers
var SystemHandler = require('./jobs/system').SystemHandler;

// --------------------------
//   System Jobs
// --------------------------

Jobs.process('end_of_month', SystemHandler.EndOfMonth);
Jobs.process('end_of_day', SystemHandler.EndOfDay);

Kue.Job.rangeByType('end_of_month','delayed', 0, 100, '', function (err, jobs) {
    if (err) { return handleErr(err); }
    if (!jobs.length) {
        if ( ! Config.EndOfMonthDelay ) {
          var presentDate = new Date();
          var presentMomentDate = Moment(presentDate);

          var nextDate = new Date();
          nextDate.setMonth( nextDate.getMonth() + 1 );
          nextDate.setHours(0);
          nextDate.setSeconds(0);
          nextDate.setMinutes(0);
          nextDate.setDate(1);

          var nextMomentDate = Moment(nextDate);
          Config.EndOfMonthDelay = nextMomentDate.diff(presentMomentDate);
        }

        // get the month for the delay
        var delayDate = Moment().add(Config.EndOfMonthDelay, 'ms');

        Jobs.create('end_of_month', { title: delayDate.format('MMMM YYYY') }).delay(Config.EndOfMonthDelay).priority('high').save();
    }
});

The SystemHandler job code doesn't do anything special. It's your basic connect to MySQL, pull data, loop, repeat, done. I've cut out that code as it's unnecessary for the issue since even truncating the function down to what you see below, it STILL runs whatever code it was prior.

var SystemHandler = {};

SystemHandler = {
    EndOfMonth: function(job, done) {
                      job.progress(100);
                      job.save();

                      job.log('Job Done!');

                      Jobs.create('end_of_month', { title: 'End of the month report' }).delay(20000).priority('high').save();
                      done();
    },

    EndOfDay: function(job, done) {
    }
};

exports.SystemHandler = SystemHandler;

The only way I was finally able to get it to run the proper code base was to rename all the instances of end_of_month to process_end_of_month. I did NOT need to rename the method in the SystemHandler. So my job-server code ends up like this.

var Jobs = Kue.createQueue({
  redis: {
    createClientFactory: function() {
      return Redis.createClient(Config.redis_port, Config.redis_host);
    }
  }
});
// Job Handlers
var SystemHandler = require('./jobs/system').SystemHandler;

// --------------------------
//   System Jobs
// --------------------------

Jobs.process('process_end_of_month', SystemHandler.EndOfMonth);
Jobs.process('end_of_day', SystemHandler.EndOfDay);

Kue.Job.rangeByType('process_end_of_month','delayed', 0, 100, '', function (err, jobs) {
    if (err) { return handleErr(err); }
    if (!jobs.length) {
        if ( ! Config.EndOfMonthDelay ) {
          var presentDate = new Date();
          var presentMomentDate = Moment(presentDate);

          var nextDate = new Date();
          nextDate.setMonth( nextDate.getMonth() + 1 );
          nextDate.setHours(0);
          nextDate.setSeconds(0);
          nextDate.setMinutes(0);
          nextDate.setDate(1);

          var nextMomentDate = Moment(nextDate);
          Config.EndOfMonthDelay = nextMomentDate.diff(presentMomentDate);
        }

        // get the month for the delay
        var delayDate = Moment().add(Config.EndOfMonthDelay, 'ms');

        Jobs.create('process_end_of_month', { title: delayDate.format('MMMM YYYY') }).delay(Config.EndOfMonthDelay).priority('high').save();
    }
});

So something, somewhere is holding on to the code that would process the end_of_month job. Yet rebooting the ElasticCache instance and the EC2 instance resolved nothing.

[behrad]

I think you have not resolved the exact issue yet for yourself, and if something bad is happening (which I still don't know what exactly is happening for you) after restarts, then it is related to the data on redis, and it is something inside you application logic.

I can't help you more until you bring a clear statement of your problem.

[jeffdupont]

I’ve clearly stated my problem. Kue is running code that does not exist on the server. I’ve cleared and restarted the redis instance, and killed node and restarted the EC2 instance. This is not an application logic issue here, I’ve cleared out all logic code to run just basics and anything named end_of_month for a job ends up running old code.

—
Sent from Mailbox

On Sat, Jan 3, 2015 at 10:38 AM, Behrad [email protected] wrote:

I think you have not resolved the exact issue yet for yourself, and if something bad is happening (which I still don't know what exactly is happening for you) after restarts, then it is related to the data on redis, and it is something inside you application logic.

I can't help you more until you bring a clear statement of your problem.

Reply to this email directly or view it on GitHub:
#494 (comment)

[RyanWarner]

@jeffdupont I spent nearly half a day on this issue today. Did you ever find a solve?

[behrad]

theres nothing as Cached job code? in Kue @RyanWarner
What exactly is your problem? and how it relates to Kue?

[jeffdupont]

No I've never found a solution to this yet
On Mar 5, 2015 11:03 PM, "Ryan Warner" [email protected] wrote:

@jeffdupont https://github.com/jeffdupont I spent nearly half a day on
this issue today. Did you ever find a solve?

—
Reply to this email directly or view it on GitHub
#494 (comment).

[RyanWarner]

I was running some Mocha tests and one of my tests threw an error (typo). From this point on, one of my jobs seemed to be 'frozen'. I was able to completely delete the job from my code, but it would still run. I saw console logs and database changes from within the job, even though the job no longer existed in my source code. No changes made on the job had any effect.

I was able to fix this for a short period of time by restarting my machine, but it soon happened again. This time, computer restarts aren't doing anything.

[behrad]

this is not caching something... it seems the your process hook problem which is not calling done and detaching from that job. The application code is responsible to correctly guarantee done is called for each job.
and please remember that removing a job, only deletes it's indexes within redis. nothing will happen to the currently running processor of that job :)

[RyanWarner]

How do I make sure done gets called if my server crashes in the middle of a job?

[behrad]

please read these carefully:
#391
#503 (comment)
#403

change your code accordingly, don't hesitate to ask any questions after that ;)

[behrad]

closing this for now... You are welcome to post if any further questions you have :)

[ryanvanderpol]

I know this is an old issue, but this exact same thing is happening to me, too. I have a job that I've created and it's run several times just fine. Then I changed some portion of the code within the job, but every time I run it, it's absolutely positively running the old code. I've gone so far as to put a throw new Error() in the opening line of the job's execution method and it never gets hit -- it just continues to run the old version.

I know it sounds crazy, but this is what's happening. I've completely killed all the node processes and have even rebooted the machine and it's still running the old code.

@jeffdupont or @RyanWarner did you guys ever resolve this problem?