[jest-circus] missing fail() method

[Darep]

This might be a known issue, but I could not find an existing issue so creating one here 😊 Also, I guess fail() was a bit of an undocumented feature, but we rely on it in our app for some nice developer experience improvements.

💥 Regression Report

After upgrading to Jest v27 (with jest-circus as default) the fail() method is no longer defined.

ReferenceError: fail is not defined

Last working version

Worked up to version: 26.6.3

Stopped working in version: 27.0.0

Can circumvent in 27.x with testRunner: "jest-jasmine2" in jest.config.js

To Reproduce

Steps to reproduce the behavior:

1. Open a JS project with jest >= 27.0.0
2. Write a test that includes a fail() method call
3. Notice that any tests with a call to fail() might pass (depending on the structure), and you will see a "fail is not defined" error message in Jest v27 with jest-circus (works correctly with jest-jasmine2)

Expected behavior

Expected fail() to work by default, as before, without any changes to jest.config.js.

Link to repl or repo (highly encouraged)

See this repo for example of the regression: https://github.com/Darep/jest-circus-fail-method

Check the branch jasmine where the testRunner is changed and the tests run correctly 🙂

The repo also hilights the way we use fail(), just to give some background info & motivation from our use-case 😄

Run npx envinfo --preset jest

Paste the results here:

$ npx envinfo --preset jest
npx: installed 1 in 0.976s

  System:
    OS: macOS 11.4
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  Binaries:
    Node: 14.16.1 - ~/n/bin/node
    Yarn: 1.21.1 - ~/n/bin/yarn
    npm: 6.14.12 - ~/n/bin/npm
  npmPackages:
    jest: ^27.0.6 => 27.0.6 

[m-gagnon]

Any update on this? Many of my integration tests are missing the correct messaging now that this is undefined, and its causing a lot of confusion.

[ChrisKatsaras]

Same here! Would love to have this issue alleviated sooner than later :)

[srmagura]

As a result of this issue, there is currently a discrepancy between @types/jest, which does define fail, and jest-circus, which does not define fail. Discussion in DefinitelyTyped

As a temporary workaround, you can define your own fail function:

function fail(reason = "fail was called in a test.") {
  throw new Error(reason);
}

global.fail = fail;

[eranelbaz]

Same here

[bitsmakerde]

Same here, too. No change after fix

[vala84no]

As a result of this issue, there is currently a discrepancy between @types/jest, which does define fail, and jest-circus, which does not define fail. Discussion in DefinitelyTyped

As a temporary workaround, you can define your own fail function:

function fail(reason = "fail was called in a test.") {
  throw new Error(reason);
}

global.fail = fail;

Unfortunately that's not equivalent. The problem I'm having is that I need to fail a test from a location where any throw will be caught. Is there any more equivalent option available?

[dereekb]

I just ran into a test where I was getting "fail() is undefined" and had assumed all this time that fail worked like it used to since it exists in @types/jest.

Just to clarify why this functionality is important:

it('should fail if the document does not exist.', async () => {
      await c.accessor.delete();

      const exists = await c.accessor.exists();
      expect(exists).toBe(false);

      try {
        await c.accessor.update(c.dataForUpdate());  // should not succeed
        fail();   // previously this would signal to Jest to properly end the test as a failure, but not get caught in the below.
      } catch (e) {
        expect(e).toBeDefined(); // expect an error occured
      }
});

The above code with Jest 28 will now incorrectly always succeed, as fail() throw an exception that gets caught by the catch.

We don't want to catch any error either though, as unexpected errors should result in a test failure rather than success. We just want the tests to succeed when failures are expect.

I went ahead and created some test utility functions so I can continue using this pattern. I did end up finding and resolving a few more bugs too. Now the example test looks like:

    import { itShouldFail, expectFail } from '@dereekb/util/test';
    
    ...

    itShouldFail('if the document does not exist.', async () => {
      await c.accessor.delete();

      const exists = await c.accessor.exists();
      expect(exists).toBe(false);

      await expectFail(() => c.accessor.update(c.dataForUpdate()));
    });

It will be published on npm with @dereekb/util@^8.1.0.

I'm not too familiar with the inner workings of Jest or why it dropped the previous functionality of fail(), but I imagine it could be brought back for the cases where we are looking for a specific error. Jest's it functionality could be extended with a function that looks for failures, (I.E. it.fail, or something) and then watch for a specific exception to be thrown that is thrown by failSuccessfully() or something to that manner. Someone more familiar with building Jest extensions may see a better way to implement it as an extension as well.

[bpossolo]

also running into this while trying to upgrade from jest 26 to jest 27..

ReferenceError: fail is not defined

[abdulbasit1149]

Any update on this?

[albertodiazdorado]

Any update on this?

Or at least some information as to:

• Why this feature has been removed
• How can we achieve what we used to achieve with fail

[niklass08]

Same here, still getting the fail is not defined error, any update or insight would be great

[joeskeen]

Seeing as this thread isn't moving towards an upcoming resolution in the jest-circus runner, I figured out how to restore the missing fail() functionality without re-implementing it. Essentially, if you install jest-jasmine2 and modify your Jest config to set "test-runner": "jest-jasmine2", you can now use fail() in your tests.

See https://stackoverflow.com/a/73922010/1396477

[atz3n]

Hi, just wanted to share the workaround I'm using.
I have created a fail function using expect and a failing comparison. It also displays messages in an okayish way. It's not the cleanest solution, but it solves the problem mentioned here.
Maybe it is helpful for someone.

function fail(message = "") {
    let failMessage = "";
    failMessage += "\n";
    failMessage += "FAIL FUNCTION TRIGGERED\n";
    failMessage += "The fail function has been triggered";
    failMessage += message ? " with message:" : "";

    expect(message).toEqual(failMessage);
}

[rayniel95]

Using the answer proposed here I tested if the same behavior could be applied to Jest. My theory was correct. It is possible to fail a Jest test if you call the done callback function with some param. Here an example:

    test('this test must fail', (done) => {
        done('hey Rainyel this is a failed test')
    });

Output:

● this test must fail

Failed: "hey Rainyel this is a failed test"

   6 |
   7 | test('this test must fail', (done) => {
>  8 |   done('hey Rainyel this is a failed test')
     |   ^
   9 |
  10 | });
  11 |

  at Object.<anonymous> (src/__tests__/test.super.ts:8:3)

[chadjaros]

This is still happening in 29.5.0

[vtgn]

Hi! How this incredibly stupid bug could still be present after a so long time?! I've never seen a test framework without a fail() method in my whole life. This not fixed regression after 2 years is a huge embarrassment for jest and a huge lack of serious.

[dandv]

@vtgn: maybe because there are bigger and older issues with Jest, like #6695.

[vala84no]

I just saw @atz3n's workaround and decided to simplify it...

Write a custom fail function

function fail(message: string = '') {
  expect(`[FAIL] ${message}`.trim()).toBeFalsy();
}

Full example
Note This workaround is still inferior to the original fail

1. The original fail was automatically globally available.
2. In failure messages, code snippets showed the invocation of the original fail. This workaround shows the expect call in its implementation.

There are already simple work-arounds like this in the thread above. There's also another way you've missed that it's inferior to the original fail: it simply doesn't work if it's executed within a callback where a parent performs a catch. E.g.:

function underTest(callback) {
  try {
    return callback();
  } catch (error) {
    return false;
  }
}

In cases like this any fail() call in callback will be silently ignored and the test still passes because all it does it throw an exception anyone can catch rather than actually mark the test as failed. That's a pretty big drawback.

[thehale]

After spending a few more hours on the problem this morning, I found a satisfactory solution using a custom matcher which builds off of the fail() matcher from jest-extended.

Use a Custom Matcher (works in try/catch blocks)

function toFail(_, message) {
  this.dontThrow();  // <-- Enables recording failures in try/catch blocks
  return {
    pass: false,
    message: () => (message ? message : 'fails by .toFail() assertion'),
  };
}

expect.extend({ toFail });

Example

// FUNCTION UNDER TEST
function invokeWithTry(callback) {
  try {
    return callback();
  } catch (err) {
    return err;
  }
}

// SETUP THE CUSTOM MATCHER
function toFail(_, message) {
  this.dontThrow();  // Just record the error when a test fails, no error throwing
  return {
    pass: false,
    message: () => (message ? message : 'fails by .toFail() assertion'),
  };
}

expect.extend({ toFail });

// TESTS
test("fail works in a try/catch", () => {
  expect().not.toFail("unexpected failure"); // Correctly passes
  const result = invokeWithTry(() => expect().toFail("expected failure")); // Correctly fails
  expect(result).toBeUndefined(); // Correctly passes
});

[danielo515]

Any chance this will be fixed in any new release?

[nealeu]

Hell... I've just arrived here to find a regression from >2.5 years ago and it's full of workarounds.

Will someone either close this as won't fix or fix it!

[joeskeen]

Or just move to another testing framework. I moved one project to vitest (Jest api compatible) this week, and noticed there was an expect.fail function.

[vtgn]

Hell... I've just arrived here to find a regression from >2.5 years ago and it's full of workarounds.

Will someone either close this as won't fix or fix it!

What a shame! This would be the worst answer to do not fix it!
And yes you're right, it's pathetic that this regression on such a basic feature is more than 2,5 years old!! And being required to use workarounds is as pathetic too!

[Aniss-nahim]

A small trick i'm using to make my test fail, but without any console message.

it('Should alwyas fail', () => {
    expect(true).toBeFalsy();
})

[hieucd04]

+1
I'm having the same problem!

[aficiomaquinas]

Still having this issue, too.

[lhoosconciso]

Another way is to use the done callback, which fails the test when called with an error message:

  it('should demonstrate how to use the done callback', (done) => {
    if (somethingIsWrong) {
      done("this shouldn't happen");
    }
    done();
  });

The downside is that you'll also need to call done() to make your test succeed. Otherwise, jest assumes that this is an asynchronous test which is still running.