Retrieve global variable after running jest programatically

[nicojs]

🚀 Feature Proposal

I would love a way to retrieve global variables from jest. For example:

test.js

global.foo = 'bar';

Running jest using the API:

import { runCLI } from 'jest';
runCLI({
      config: config,
      runInBand: true,
      silent: true
  }, [process.cwd()]).then(result => {
  result.globals.foo //=> somehow retrieve globals?
});

Motivation

I'm one of the core developers of Stryker, the mutation testing framework for JavaScript and friends. A lot of users are using Stryker with Jest. Functionally, it works fine. However, the performance is not great at the moment.

Using coverage analysis can greatly speedup the mutation testing process. For more information about coverage analysis, see https://stryker-mutator.io/blog/2018-01-10/typescript-coverage-analysis-support. Stryker has support for coverage analysis, as long as the test runner can communicate the coverage report back to the main Stryker process. It can measure both total test coverage as well as code coverage per test. Both coverage reports are different.

Example

We will use global variables like this:

1. A user runs Stryker on a Jest project using coverage analysis perTest
2. Stryker will instrument the production code for code coverage. It uses a global coverage object (using istanbul)
3. Stryker will run jest via the programmatic interface with the stryker-jest-runner. It will use this configuration options:
   • runInBand: true
   • collectCoverage: false
   • setupFiles: ['path/to/custom/stryker-jasmine/hooks/file']
4. The stryker-jest-runner communicates the global coverage object back to Stryker
5. Stryker runs mutation testing using the perTest coverage analysis mode. It will filter the exact tests to run using the setupFiles option to configure a jasmine spec filter.

Pitch

In the Jest as a Platform talk it is stated that jest is a platform because it allows you to build on top of. I think building a mutation testing framework this way fits that bill perfectly.

[thymikee]

Um, can't you use this? https://jestjs.io/docs/en/configuration#globals-object

[nicojs]

No, that only works to pass global variables to the testing environment. No way to get changed variables back out.

[rickhanlonii]

@nicojs would globalSetup and globalTeardown work? I.e. can you send the coverage results back to stryker in globalTeardown (which is like a global afterEach for each file that is tested)

[rickhanlonii]

Another option would be a custom reporter (which gets the results of each test file and all results after all tests run). This is how our built-in coverage works, here's our coverage reporter

[SimenB]

I don't think tests should "return results". However, I'm not sure what the best API for this would be. Adding it to TestResult somehow would be nice, but I'm unsure how. Maybe some sort of jest.report({myCustomData: {bla: 42}})?

A workaround you can maybe use is add an afterAll hook in setupFilesAfterEnv (setupTestFrameworkScriptFile pre Jest 24) and write something from global to disk or whatever. The data has to be serializable anyways as it'll be passed between a worker and the main process. Or you can create your own test environment and do stuff in its teardown

would globalSetup and globalTeardown work?

No, they run outside the sandbox and do not have access to its global object.

Another option would be a custom reporter

That works since we put the coverage data into TestResult explicitly, it's not a generic thing. (via https://github.com/facebook/jest/blob/b502c07e4b5f24f491f6bf840bdf298a979ec0e7/packages/jest-runner/src/run_test.js#L205)

[nicojs]

Thanks for all the responses! It is appreciated 👍

Adding it to TestResult somehow would be nice, but I'm unsure how. Maybe some sort of jest.report({myCustomData: {bla: 42}})?

Something like that would work for us. We would add it it in a custom jasmine reporter:

jasmine.getEnv().addReporter({
    jasmineDone(result) {
        jest.report({strykerCoverage: {bla: 42}})
    }
});

The data has to be serializable anyways as it'll be passed between a worker and the main process.

That isn't the case for us right? As we use runInBand: true? However, the coverage data is serializable, no worries there.

Or you can create your own test environment and do stuff in its teardown

I understand, but the problem would be that we would have to keep maintaining 'shadow' test environments for all supported test environments. Code would be identical except for the teardown part. Feels like a lot of maintenance work and error-prone for end users.

A workaround you can maybe use is add an afterAll hook in setupFilesAfterEnv (setupTestFrameworkScriptFile pre Jest 24) and write something from global to disk or whatever

Wow. Didn't event think of that. It feels dirty, but it works. If I write the coverage data to disk, I can pick it up later. Just to be clear: this is the only way to share global data in Jest today? I still would like a better way of doing it, but we can proceed to implement this.

The setupTestFrameworkScriptFile file will look like this:

if (global.jasmine) {
    jasmine.getEnv().addReporter({
        jasmineDone() {
            require('fs').writeFileSync('coveragedata', global.strykerCoverage, 'utf8');
        }
    });
}

Unfortunately, we need to use writeFileSync, as it the jasmineDone hook cannot run asyncronously. Jasmine does support async hooks since jasmine 3, but apparently Jest still uses jasmine 2, is that correct?

[rickhanlonii]

@nicojs does a Jest custom reporter work for you then?

[palmerj3]

It is also possible that you can write a custom test environment in order to have more control over the global environment. Example: https://github.com/facebook/jest/blob/master/packages/jest-environment-jsdom/src/index.js#L45

[nicojs]

@rickhanlonii

does a Jest custom reporter work for you then?

Not really, the global object seems to have already been cleared. global.foo is undefined there. Too bad, would be a clean solution to use a jest reporter in order to add custom coverage data to the test results.

@palmerj3

It is also possible that you can write a custom test environment in order to have more control over the global environment

I understand. Can we think of a way to still use the test environment that the user configured (jsdom, or other), but also load our custom environment? As specified before, I'm not interested in maintaining a "shadow" environment for every jest environment out there.

[rickhanlonii]

Is global.foo the only way to solve this problem?

[nicojs]

This is the way code coverage in JavaScript generally works, right? By passing coverage data along the global scope?

We could also use a shared node module to store the data statically:

const coverageStore = require('./coverageStore');
coverageStore.foo = 'bar';

But it suffers from the same issue. Jest will make sure the test environment doesn't interfere with the runtime environment. I.e. require('./coverageStore').foo will be undefined after the runCLI is done. This is generally what you want. There are really no use cases where you want the tests to pollute the global scope.... except code coverage 🙈

Maybe there is another way though. If we use the variable name __coverage__, jest will report that as coverage data here: https://github.com/facebook/jest/blob/2c18a53e8ff2437bba5fcb8076b754ac5f79f9f8/packages/jest-runtime/src/index.js#L471. However, I don't think it's smart, as we will essentially be passing arbitrary data this way, we might incidentally be breaking something.

[SimenB]

We need to find a solution that does not rely on Jasmine APIs, as they will be deprecated and removed at some point.

That isn't the case for us right? As we use runInBand: true? However, the coverage data is serializable, no worries there.

In your case yes, but we're not adding a feature that only works when running in band.

I understand, but the problem would be that we would have to keep maintaining 'shadow' test environments for all supported test environments. Code would be identical except for the teardown part. Feels like a lot of maintenance work and error-prone for end users.

const NodeEnvironment = require('jest-environment-node');

class StrykerEnvNode extends NodeEnvironment {
  teardown() {
    doStrukerStuff();

    return super.teardown();
  }
}

module.exports = StrykerEnvNode;

but apparently Jest still uses jasmine 2, is that correct?

It uses a heavily modified fork. We will be making jest-circus the default at some point rather than upgrading to jasmine 3, however.

Maybe there is another way though. If we use the variable name __coverage__, jest will report that as coverage data here:

Yeah, don't mess with that 😉

[palmerj3]

@nicojs it's definitely possible for you to inherit from an existing test environment so you reduce the duplication. They are just ES6 classes.

[nicojs]

@SimenB thanks for the code sample. It seems to work fine. We can read the globals from this.context.foo. The idea is that when a user configures 'node' as environment, we'll override it with StrykerNodeTestEnvironment, and as well for environment 'jsdom' with StrykerJSDomEnvironment. Just to be clear, this would mean that we don't support the @jest-environment docblock at the top of the file, right?

Another benefit of using jest's environment teardown is that it can run asynchronously. So we can use writeFile instead of writeFileSync 👍 . Just to be clear, that would still be the only way to communicate the global value back to the host process?

I still think a cleaner way of reading global variables would be preferred. But I'm willing to close this issue, if you disagree.

[SimenB]

I still think a cleaner way of reading global variables would be preferred. But I'm willing to close this issue, if you disagree.

I think it should stay open - a custom env is just a workaround, we should make it possible to add arbitrary things to the TestResult object so you can use it in Jest reporters. I don't have any suggestions on how to achieve that, however

[sandorfr]

we should make it possible to add arbitrary things to the TestResult

I was looking for a way to do just that and I'd love to see a simple api for it.
I'm trying to build a tooling that monitors calls to certain apis that are not mocked (and should be) in a large code base. I'm trying to get that data attached to the test results so I could then build a report using a custom reporter.

[sandorfr]

@SimenB Is there a way we could discuss what would be a good API to achieve this. I might be in a position where I can get the time to work on contribution on this, if we can agree on a solution.

[PFadel]

Hey guys, any news about this issue ?

[imolorhe]

I'm also interested in this API. Any news about this?

[narayanpai]

It would be great to have this feature so that we could expose any arbitrary data to the custom test reporter and output the same using jest-junit via system-out xml element

[palmerj3]

@narayanpai I added a feature to jest-junit recently which may fit your use-case. It allows you to add custom test suite properties (which is a supported part of the junit spec).

https://github.com/jest-community/jest-junit#adding-custom-testsuite-properties

[ingun37]

@palmerj3 Doesn’t the feature only let you process the result data in which only way of including any custom data from the test environment is console property which is just a plain text? Therefore, not a really good solution?

[palmerj3]

You get the same data reporters themselves get. So if your complaint is about passing custom data to reporters then you're in the right place.

[pckhoi]

I don't understand why adding custom data to test result is such a hard thing to agree on. Just allow something like testResultSet(mySymbol, {...custom data})? What would be the down side of this approach for example?

[dpchamps]

What's the status on this issue?

[nicojs]

I'm pretty sure the core team didn't make progress on this yet. I ended up implementing this feature in StrykerJS using the suggestion by @SimenB .

You can use https://github.com/stryker-mutator/stryker-js/tree/master/packages/jest-runner as inspiration

[dpchamps]

Thanks @nicojs

@SimenB would the Jest team be open to accepting a pr that introduced this functionality? I see this request from time to time, and would personally like to be able to decorate test results with various meta-datas to report on later.

Perhaps I'm looking for something that already exists: My current use case calls for integrating with various test-reporting frameworks, like testrails and xray et al... I'd like to be able to represent a specification with a well-defined property, run all of the tests and then report results to a third party with the associated id.

Something like (pseudocode incoming)

// where this value is persisted somewhere, testRailsId : ABCD
it.testRails("ABCD", "does something cool", () => { ... });

// reporter.js
class TestRailsReporter extends Reporter {
  async onRunComplete(_, results) {
    // individual results includes Maybe<{testRailsId : "ABCD"}>
  }
}

A custom node environment, or writing to the filesystem feels heavy for something that seems "straightforward" to me.

[DavidPeicho]

Any news on that? What would be needed for this change to get in?

[sammy-da]

I'm interested in this too. any updates?

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.