Karma doesn't exit properly when using public api with the finish callback

[pkozlowski-opensource]

Using Karma's server public API with a done callback makes the whole node process "stuck" - there are probably some event listeners that prevent node process from exiting properly. A simple reproduce scenario:

var karma = require('karma').server;

karma.start({
  //transports: ['websocket', 'xhr-polling', 'jsonp-polling'],
  browsers: ['Chrome'],
  frameworks: ['jasmine'],
  files: [
    'src/**/*.js',
    'test/**/*.spec.js'
  ],
  singleRun: true
}, function(karmaExitCode) {
  console.log('Karma exited with ', karmaExitCode);
  console.log('but the process will never exit...');
});

Interestingly, removing the flash from the list of transports make the Karma exit after a while (~20s after test are finished on my machine).

There is a small repo with the reproduce scenario here:
https://github.com/karma-runner/gulp-karma/blob/exit_pb/exitTest.js
I will be digging into the topic but any help from someone more knowledgeable about Karma internals would be highly appreciated.

[pkozlowski-opensource]

For now it looks for me like it is https://github.com/LearnBoost/socket.io/blob/0.9/lib/stores/memory.js#L137 that blocks node.js from exiting, digging deeper.

[lazd]

This issue stops Karma's public API from working with gulp and has resulted in workarounds where child processes are spawned to start Karma servers because there's simply no non-fatal way to get the Karma server to go away.

Related to #734

[lazd]

process.exit() is fatal, so that won't work.

[lazd]

I pushed a branch illustrating the gulp issue:

https://github.com/lazd/gulp-karma-test/tree/purekarma

The issue is much better with v0.12.9 vs v0.10.9 -- it just takes two Control + C to get it to die, whereas in v0.10.9 it didn't matter how many times you hit Control + C, you had to kill -9 the process.

[pkozlowski-opensource]

@cironunes, as pointed out by @lazd process.exit() is something we want to avoid here, so it is not the way to go.

I can confirm now that most of the delays on exit are coming from socket.io:

• 15 seconds delay from the client store expiration timeout
• using the flash transport prevents socket.io from exiting altogether...

Other than this there are smaller "timeout leaks" in Karma that I'm going to address over the weekend, but yes, socket.io is the main culprit here. We either need to configure it or shutdown properly.

[pkozlowski-opensource]

OK, so finally I've found 4 different setTimout instructions that were preventing Karma from exiting properly. 2 of them are linked to the socket.io shutdown and 2 others are karma's own timeouts.

I've opened PRs (linked from here) that "fix" most of the issues, the only remaining timeout is here:
https://github.com/karma-runner/karma/blob/master/lib/launchers/process.js#L25

[vojtajina]

@pkozlowski-opensource great work! Thanks for fixing this.

I merged your first PR. The timeouts you mentioned (launchers/process.js#25), that's a bug, see #946 If you can fix it, that would be great. It has to be cleared in processExit. Let me know if you need any info.

[pkozlowski-opensource]

@vojtajina thnx for merging the things. I didn't see #946 before but yes, it describes the only remaining timeout issue. I played a bit with the code and I'm pretty positive about a fix but would like to add a test for it as well. Anyway, will take care of #946.

[bcherny]

@pkozlowski-opensource This happens for me once per 20-30 builds - the test suite finishes successfully, but hangs at gulp's Finished 'unit' after 3.08 min. How did you find the hanging setTimeouts? I'd like like to similarly instrument our code to find the culprit.

For reference, this is our gulp code:

const gulp = require('gulp')
const {Server} = require('karma')

gulp.task('unit', function (done) {
  new Server({
    configFile: './karma.conf.coffee'
  }, exitCode => {
    if (exitCode == 0) {
      done()
    } else {
      process.exit(exitCode)
    }
  }).start()
})

[Fernker]

@bcherny Were you able to find the solution to your issue? I'm seeing the same thing happen (20-30 builds) and I'd really like to solve it.

[jnwu]

As a workaround, I am forcing the karma server to be executed in a separate process via 'gulp-multi-process'

gulp.task('test', () => {
    const config = parseConfig(path.resolve('test/karma.conf.js'));
    const server = new karma.Server(config);
    server.start();
});

gulp.task('multi', (cb) => {
    return multiProcess(['test'], cb);
});

gulp.task('watch', () => {
    gulp.watch(files, ['multi']);
});

In addition my karma config has autoWatch off, and singleRun on. Hence, I am using gulp to watch my files and delegating all the test tasks (including report generation) to karma.

[bcherny]

@Fernker Missed your comment - we still see this issue every 20-30 builds. When it happens, we restart the build, and it usually passes. I've noticed that it tends to happen more often when we are running multiple builds in parallel on a single environment.

[shuyudev]

The problem remains for Karma 1.4.0 on Windows. Is there workaround for that?

[npetruzzelli]

• Windows 7 Pro
• node v6.9.4
• npm 3.10.10

I've been experiencing the same issues and have noticed the following:

• Affected:
  • Chrome
  • Firefox
• Not Affected
  • PhantomJS
  • Internet Explorer

I have a couple work arounds, granted niether is perfect:

// gulpfile.js
const path         = require('path');
const gulp         = require('gulp');
const karma        = require('karma');
const minimist     = require('minimist');
const ARGV         = minimist(process.argv.slice(2));

/**
 * Option 1
 * Use a stopper in the `run_complete` event listener.
 * Must have access to the port the server is running on.
 */
gulp.task('karma_1', function taskKarma_1(taskDone){
  /**
   * if `--single` was a command line option, for example:
   * `gulp karma_1 --single`
   */
  var someConditionIsTrue = (ARGV.single === true);
  var karmaConfig = {
    autoWatch: true,
    configFile: path.resolve('./karma.conf.js'),
    port:9876, // Stopper will need to know the port.
    singleRun: false
  };
  if(someConditionIsTrue){
    karmaConfig.singleRun = true;
    karmaConfig.autoWatch = false;
  }

  var karmaServer = new karma.Server(
    karmaConfig,
    function handleKarmaServerExit(processExitCode){
      if(processExitCode === 0 || processExitCode === null || typeof processExitCode === 'undefined'){
        taskDone();
      } else {
        var err = new Error('ERROR: Karma Server exited with code "' + processExitCode + '"');
        taskDone(err);
      }
    }
  );

  if(karmaConfig.singleRun){
    //Stop the server after the first run.
    karmaServer.on('run_complete', function(){
      karma.stopper.stop({
        port: karmaConfig.port
      });
    });
  }

  karmaServer.start();
});

/**
 * Option 2
 * Use `process.exit` in the Karma onExit handler.
 */
gulp.task('karma_2', function taskKarma_2(taskDone){
  /**
   * if `--single` was a command line option, for example:
   * `gulp karma_2 --single`
   */
  var someConditionIsTrue = (ARGV.single === true);
  var karmaConfig = {
    autoWatch: true,
    configFile: path.resolve('./karma.conf.js'),
    singleRun: false
  };
  if(someConditionIsTrue){
    karmaConfig.singleRun = true;
    karmaConfig.autoWatch = false;
  }

  var karmaServer = new karma.Server(
    karmaConfig,
    function handleKarmaServerExit(processExitCode){
      if(processExitCode === 0 || processExitCode === null || typeof processExitCode === 'undefined'){
        taskDone();
      } else {
        var err = new Error('ERROR: Karma Server exited with code "' + processExitCode + '"');
        taskDone(err);
      }
      process.exit(processExitCode); //Exit the node process
    }
  );

  karmaServer.start();
});

I hope this helps!

[npetruzzelli]

I should add clarification. The server exit handler is being called for me, but the process never exits if the affected browsers are launched. In hindsight, this may be different from the issue described above.

My "option 2" is similar to @bcherny 's code.

[Ivaylo-Lafchiev]

Windows 7
Node 8.11.2
Npm 5.6.0

I've run into the same issue, tested on [email protected] and karma@latest in my project's gulp chain. Seems there are socket.io handlers (detected using why-is-node-running) that aren't resolved for a good 30-40 seconds after karma has finished executing the tests. The above solutions provided by @npetruzzelli are of no help as they terminate the node process and the following tasks are not executed.

I was hoping #3149 would help but it seems that despite Karma exiting gracefully after the test run it seems socket.io is still holding the process hostage. I'm going to try what was suggested by @jnwu but I'm not sure this is the best solution to the problem.