Introduce support for declarative test suites #744

smoyer64 · 2017-03-22T21:43:34Z

Overview

As requested in issue #681 (Introduce junit-platform-suite-api module), the annotations used for test discovery and selection in the JUnitPlatform runner have been refactored into the junit-platform-suite-api. This allows a purely JUnit 5 test suite to be defined declaratively, though it currently requires an alternate engine (as opposed to an extension).

This issue describes a generalized JUnit platform suite that's suitable for the Jupiter test engine as well as any others that might be chosen. It's my opinion that unit tests rarely require test suites, but it becomes far more common when integration, system and acceptance tests are being executed due to the very common integration with external systems that should only be initialized before the entire test run. Examples of resources that can be safely shared among tests are database connection pools (or really any connection pool), test frameworks that require significant set-up time (e.g. GWT test suites eliminate 20s of HtmlUnit set-up time for each test - it's only incurred once at the beginning of each suite).

The examples shown below indicate how the @Suite annotation can be used in conjunction with those already in the junit-platform-suite-api module. Originally I'd also defined the @BeforeSuite and @AfterSuite annotations but after further consideration decided that since suites are really just an arbitrary collection of indeterminately nested test containers, it would be better if the @BeforeAll and @AfterAll annotations were contextual (nested in the same way the hierarchy would be presented in the test tree). I've also assumed that these annotations no longer require the static modifier as discussed elsewhere.

Imagine if you will a set of acceptance tests that have a custom reporting mechanism that should be configured for the entire test run. Included within this outer suite is a set of tests that require a database connection pool and a different set of tests that require an LDAP connection pool. The top-level suite definition for the acceptance tests might look something like:

@Suite
@IncludeEngines("jupiter-engine") // not required if there's only one engine on the classpath
@IncludeClassNamePatterns(".*AcceptanceTestSuite$")
public class AcceptanceTestSuite {

    @BeforeAll
    void setUp() {
        ... // code to set-up custom test reporting
    }

    @AfterAll() {
    void tearDown() {
        ... // code to tear-down custom test reporting
    }

}

There are two notable characteristics of this test suite:

It specifies which engine will be used for the entire container/test hierarchy (assuming there is more than one).
It includes potentially numerous other test suites as sub-containers which can each have their own contextual set-up and tear-down.

A child suite for tests that require a database connection pool might look something like this:

@Suite
@IncludeTags({"integration", "database"})
@IncludeClassNamePatterns(".*IT$")
public class DatabaseAcceptanceTestSuite {

    @BeforeAll
    void setUp() {
        ... // code to set-up the database connection pool
    }

    @AfterAll() {
    void tearDown() {
        ... // code to tear-down the database connection pool
    }

}

If run under Maven, test classes that end in "IT" would be run using the maven-failsafe-plugin. If there are other classes of integration tests not run during acceptance testing it would also be necessary to make sure the acceptance test classes were not run (as they'd fail without the suite's set-up methods).

Feature Request

Provide the means for arbitrarily nested, declarative tests suites in JUnit 5 including the execution of set-up and tear-down methods

Related Issues

Deliverables

TBD

The text was updated successfully, but these errors were encountered:

mlevison · 2017-03-22T23:50:19Z

Please consider a new name other than suites - so that people with JUnit 3 history don't accidentally mis-use these for Unit Testing. Cheers Mark

sbrannen · 2017-03-24T13:44:42Z

Tentatively slated for M5.

sbrannen · 2017-03-24T13:56:34Z

@smoyer64, thanks for creating the ticket!

I think there are two main topics we need to sort out here:

suites within the JUnit Jupiter engine
declarative suite configuration at the Platform level

If I understand correctly, you are currently focusing on # 1; whereas, I would first like to focus on # 2.

Status Quo

We have generic suite API annotations.
We have a JUnit 4 based runner that supports them.

The Future

Ideally we should have Platform-level support for them as well.
If it makes sense to have additional support in JUnit Jupiter for "suites", we should consider that as well.

My Vision

I think it would be most beneficial to the entire community if the JUnit Platform provides support for declarative test suites, akin to what the JUnitPlatform runner already does, but without any ties to any testing framework.

In other words, the declarative test suite support should be analogous to using the ConsoleLauncher or JUnit Platform Gradle plugin.

This can be achieved by moving the internals for processing such suite annotations and building the corresponding LauncherDiscoveryRequest from the JUnitPlatform runner to a new home in the launcher module. This new "prediscovery" code could then be used to transparently add selectors and filters to the request before any engines are asked to discover. For example, this could potentially be performed in the DefaultLauncher.

In this manner, all test engines could benefit from being able to be configured by a generic suite (which is already supported in build scripts). From the perspective of a TestEngine, the presence of selectors and filters added from the suite declaration would be no different than those added via a build script.

sbrannen · 2017-03-24T13:58:56Z

This allows a purely JUnit 5 test suite to be defined declaratively, though it currently requires an alternate engine (as opposed to an extension).

My above proposal avoids exactly such a need.

If we support suites at the platform level, there is no need for any additional test engines.

sbrannen · 2017-03-24T14:02:01Z

By the way, my proposal does not in any way preclude the introduction of suite support specific to a particular test engine.

The idea is that the Platform (at the launcher level) would only honor the suite API annotations if a special, platform-specific annotation is present (e.g., @JUnitPlatformSuite); otherwise, the Platform would simply ignore such annotations and leave the LauncherDiscoveryRequest unchanged.

sbrannen · 2017-03-24T14:03:39Z

@sbrannen - Hopefully this is enough to start the conversation.

@smoyer64 - yes, indeed it is! 👍

sbrannen · 2017-03-24T14:08:00Z

Another topic to sort out (which I didn't explain so well above) is whether suites should be handled specially within the test plan.

With the generic approach (at the Launcher level), this would not be the case since test engines would not see the presence of a suite as a first-class citizen in the test plan. As I mentioned above, suites would be invisible to test engines.

So, if we want to include first-class support for suites as nodes within the test plan, that opens up a whole new can of worms! ... And I'm not so sure we want to go that route, at least not before GA.

smoyer64 · 2017-03-24T14:10:43Z

@sbrannen I actually think we're exactly on the same page - #687 along with #681 were the two issues that I put in to allow an extension to implement suites. The engine I've already created works fine but with a couple more extension points it would be trivial to simply make it an extension.

Let's ignore the word "suite" for a moment - I think there's a great argument that these are simply hierarchical test containers that participate in the engine's life-cycle at their appropriate level of the hierarchy. And the junit-platform-suite-api annotations are simply a declarative specification describing what tests are contained within the container.

I wrote this issue as asked in #681 because I thought you were promoting a built-in suite (which may be worthwhile) but the original two issues were simply to make it easier to write extensions that implement arbitrary containers. Breaking the tie between junit-platform-suite-api and JUnit >=4.12 was definitely the right thing to do - perhaps this issue will be another meta-issue (like #14) that spawns a series of small steps towards the final goal.

EDIT - It appears we're trying to have one of those conversations where we're both talking at the same time ;) I also should have noted that the code blocks I put in the original description are indicative of what I'd like to write when creating suites. The only really important feature included that's lacking in JUnit 4 is that @BeforeSuite/@BeforeAll and @AfterSuite/@AfterAll can be nested based on the discovery hierarchy.

mlevison · 2017-03-24T14:33:47Z

I will make another attempt to explain why I think there are risks/issues with the name. I like the concept, but the linkage to the old names causes risk.

All of my work is consulting work in organizations. Frequently when I sit down with teams I get a chance to see their code. In too many cases I see people who have JUnit 3 style tests and suites in JUnit 4. When prompted they're not aware that this is no longer the preferred way.

What I have loved about JUnit 5 is that by and large the right choices will be made by default. Re-using the suites word by default will make easier for people to accidentally mis-use them in world where people use their "intelli-sense" type technology, instead of reading the documentation and reflecting.

Cheers
Mark aka @mlevison

sormuras · 2017-03-24T14:37:15Z

Do you have a better name in mind, Mark?

sbrannen · 2017-03-24T14:57:40Z

@mlevison, I have to admit that I don't really understand your concerns.

I personally cannot think of a better name than "suite" for something that allows you to group together a collection of tests that should be run together.

In fact, the Suite support in JUnit 4 achieves exactly what we are describing here:

Using Suite as a runner allows you to manually build a suite containing tests from many classes.

We also use the term suite in the documentation for the JUnitPlatform runner:

When used on a class that serves as a test suite...

So I would argue that the familiarity with suites from JUnit 3, JUnit 4, and other testing frameworks will make the intended usage on the JUnit Platform quite clear.

sbrannen · 2017-03-24T14:58:20Z

@sbrannen I actually think we're exactly on the same page...

@smoyer64, glad to hear it!

I'll have to look into the details at a later date though.

smoyer64 · 2017-03-24T15:27:24Z

@sbrannen When I said to "ignore the word suite", I wasn't implying that I didn't think that name was appropriate ... it definitely feels familiar and I think it would actually be confusing to migrating users to suddenly use a different term (my intent in that paragraph was to point out that suites are a "kind of" test container).

I'll see about cleaning up my suite-engine and getting it moved into junit-pioneer - perhaps we can use it as a safe way to beta-test before committing to a course of action in platform/jupiter.

mlevison · 2017-03-24T17:09:05Z

On Fri, Mar 24, 2017 at 10:37 AM, Christian Stein ***@***.***> wrote: Do you have a better name in mind, Mark?

No you expect intelligent thought. I didn't yet. Ideas (that I don't like): collection, series, set, map or if we're attached to Suite something more expressive like IntegrationSuite that makes it clear that these are not intended for Unit tests. Cheers Mark

…

On Fri, Mar 24, 2017 at 10:37 AM, Christian Stein ***@***.***> wrote: Do you have a better name in mind, Mark? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#744 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAImZsXAUEHKkIlABffel3vXKWH562d9ks5ro9UcgaJpZM4Ml3IL> .

mlevison · 2017-03-24T17:14:04Z

On Fri, Mar 24, 2017 at 10:57 AM, Sam Brannen ***@***.***> wrote: @mlevison <https://github.com/mlevison>, I have to admit that I don't really understand your concerns. I personally cannot think of a better name than "suite" for something that allows you to group together a collection of tests that should be run together. In fact, the Suite support in JUnit 4 achieves exactly what we are describing here: Using Suite as a runner allows you to manually build a suite containing tests from many classes.

My point is that some years after the appearance I JUnit 4 - I still find people writing their tests as if they were using JUnit 3. Not because this was desirable, but because they didn't know that they didn't need suites anymore. Often when I point out that they're not longer needed they tidy up. My original concern, was that by reusing the name you risk the same cargo cult habits with JUnit 5 as I've seen before. Its cool if you disagree, am I at least making sense this time? Cheers Mark

@suite

Implements a test engine that allows declarative execution of test suites using the `@Suite` annotation. Internally the Suite Engine uses the JUnit Platform Launcher. The engine works by mapping the `TestIdentifier` used by the launcher to `TestDescriptor` used by the engine during discovery and execution. ``` package org.junit.platform.suite; import org.junit.platform.suite.api.SelectPackages; @suite @SelectPackages("org.junit.suite.testcases") class SelectPackageSuite { } ``` Is equivalent to: ``` import org.junit.platform.engine.discovery.DiscoverySelectors; import org.junit.platform.launcher.Launcher; import org.junit.platform.launcher.LauncherDiscoveryRequest; import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder; import org.junit.platform.launcher.core.LauncherFactory; public class Main { public static void main(String[] args) { Launcher launcher = LauncherFactory.create(); LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() .selectors(DiscoverySelectors.selectPackage("org.junit.suite.testcases")) .build(); launcher.execute(request); } } ``` The suite engine converts an annotated class into a discovery request. This request is executed and the resulting test plan is mapped to a tree of test descriptors. In essence the tree of test descriptors are a view on the test plan. So suppose the discovery requests produces this test plan: ``` JUnit Jupiter |- TestA ||- method1 ||- method2 |- TestB ||- method1 ||- method2 ``` Then that test plan is mapped to a tree of test descriptors like so: ``` SuiteEngine |- ExampleSuite ||-JUnit Jupiter |||- TestA ||||- method1 ||||- method2 |||- TestB ||||- method1 ||||- method2 ```` The unique identifiers are remapped by pre-pending the unique identifier of the suite. So: ``` junit-jupiter/TestA/method1() -> junit-suite/ExampleSuite/junit-jupiter/TestA/method1() ``` Issue: junit-team#744

@suite

Implements a test engine that allows declarative execution of test suites using the `@Suite` annotation. Internally the Suite Engine uses the JUnit Platform Launcher. The engine works by mapping the `TestIdentifier` used by the launcher to `TestDescriptor` used by the engine during discovery and execution. ``` package org.junit.platform.suite; import org.junit.platform.suite.api.SelectPackages; @suite @SelectPackages("org.junit.suite.testcases") class SelectPackageSuite { } ``` Is equivalent to: ``` import org.junit.platform.engine.discovery.DiscoverySelectors; import org.junit.platform.launcher.Launcher; import org.junit.platform.launcher.LauncherDiscoveryRequest; import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder; import org.junit.platform.launcher.core.LauncherFactory; public class Main { public static void main(String[] args) { Launcher launcher = LauncherFactory.create(); LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() .selectors(DiscoverySelectors.selectPackage("org.junit.suite.testcases")) .build(); launcher.execute(request); } } ``` The suite engine converts an annotated class into a discovery request. This request is executed and the resulting test plan is mapped to a tree of test descriptors. In essence the tree of test descriptors are a view on the test plan. So suppose the discovery requests produces this test plan: ``` JUnit Jupiter |- TestA ||- method1 ||- method2 |- TestB ||- method1 ||- method2 ``` Then that test plan is mapped to a tree of test descriptors like so: ``` SuiteEngine |- ExampleSuite ||-JUnit Jupiter |||- TestA ||||- method1 ||||- method2 |||- TestB ||||- method1 ||||- method2 ```` The unique identifiers are remapped by pre-pending the unique identifier of the suite. So: ``` junit-jupiter/TestA/method1() -> junit-suite/ExampleSuite/junit-jupiter/TestA/method1() ``` Issue: junit-team#744

@suite

Implements a test engine that allows declarative execution of test suites using the `@Suite` annotation. Internally the Suite Engine uses the JUnit Platform Launcher. The engine works by mapping the `TestIdentifier` used by the launcher to `TestDescriptor` used by the engine during discovery and execution. ``` package org.junit.platform.suite; import org.junit.platform.suite.api.SelectPackages; @suite @SelectPackages("org.junit.suite.testcases") class SelectPackageSuite { } ``` Is equivalent to: ``` import org.junit.platform.engine.discovery.DiscoverySelectors; import org.junit.platform.launcher.Launcher; import org.junit.platform.launcher.LauncherDiscoveryRequest; import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder; import org.junit.platform.launcher.core.LauncherFactory; public class Main { public static void main(String[] args) { Launcher launcher = LauncherFactory.create(); LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() .selectors(DiscoverySelectors.selectPackage("org.junit.suite.testcases")) .build(); launcher.execute(request); } } ``` The suite engine converts an annotated class into a discovery request. This request is executed and the resulting test plan is mapped to a tree of test descriptors. In essence the tree of test descriptors are a view on the test plan. So suppose the discovery requests produces this test plan: ``` JUnit Jupiter |- TestA ||- method1 ||- method2 |- TestB ||- method1 ||- method2 ``` Then that test plan is mapped to a tree of test descriptors like so: ``` SuiteEngine |- ExampleSuite ||-JUnit Jupiter |||- TestA ||||- method1 ||||- method2 |||- TestB ||||- method1 ||||- method2 ```` The unique identifiers are remapped by pre-pending the unique identifier of the suite. So: ``` junit-jupiter/TestA/method1() -> junit-suite/ExampleSuite/junit-jupiter/TestA/method1() ``` Issue: junit-team#744

@suite

Implements a test engine that allows declarative execution of test suites using the `@Suite` annotation. Internally the Suite Engine uses the JUnit Platform Launcher. The engine works by mapping the `TestIdentifier` used by the launcher to `TestDescriptor` used by the engine during discovery and execution. ``` package org.junit.platform.suite; import org.junit.platform.suite.api.SelectPackages; @suite @SelectPackages("org.junit.suite.testcases") class SelectPackageSuite { } ``` Is equivalent to: ``` import org.junit.platform.engine.discovery.DiscoverySelectors; import org.junit.platform.launcher.Launcher; import org.junit.platform.launcher.LauncherDiscoveryRequest; import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder; import org.junit.platform.launcher.core.LauncherFactory; public class Main { public static void main(String[] args) { Launcher launcher = LauncherFactory.create(); LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() .selectors(DiscoverySelectors.selectPackage("org.junit.suite.testcases")) .build(); launcher.execute(request); } } ``` The suite engine converts an annotated class into a discovery request. This request is executed and the resulting test plan is mapped to a tree of test descriptors. In essence the tree of test descriptors are a view on the test plan. So suppose the discovery requests produces this test plan: ``` JUnit Jupiter |- TestA ||- method1 ||- method2 |- TestB ||- method1 ||- method2 ``` Then that test plan is mapped to a tree of test descriptors like so: ``` SuiteEngine |- ExampleSuite ||-JUnit Jupiter |||- TestA ||||- method1 ||||- method2 |||- TestB ||||- method1 ||||- method2 ```` The unique identifiers are remapped by pre-pending the unique identifier of the suite. So: ``` junit-jupiter/TestA/method1() -> junit-suite/ExampleSuite/junit-jupiter/TestA/method1() ``` Issue: junit-team#744

@suite

Implements a test engine that allows declarative execution of test suites using the `@Suite` annotation. Internally the Suite Engine uses the JUnit Platform Launcher. The engine works by mapping the `TestIdentifier` used by the launcher to `TestDescriptor` used by the engine during discovery and execution. ``` package org.junit.platform.suite; import org.junit.platform.suite.api.SelectPackages; @suite @SelectPackages("org.junit.suite.testcases") class SelectPackageSuite { } ``` Is equivalent to: ``` import org.junit.platform.engine.discovery.DiscoverySelectors; import org.junit.platform.launcher.Launcher; import org.junit.platform.launcher.LauncherDiscoveryRequest; import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder; import org.junit.platform.launcher.core.LauncherFactory; public class Main { public static void main(String[] args) { Launcher launcher = LauncherFactory.create(); LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() .selectors(DiscoverySelectors.selectPackage("org.junit.suite.testcases")) .build(); launcher.execute(request); } } ``` The suite engine converts an annotated class into a discovery request. This request is executed and the resulting test plan is mapped to a tree of test descriptors. In essence the tree of test descriptors are a view on the test plan. So suppose the discovery requests produces this test plan: ``` JUnit Jupiter |- TestA ||- method1 ||- method2 |- TestB ||- method1 ||- method2 ``` Then that test plan is mapped to a tree of test descriptors like so: ``` SuiteEngine |- ExampleSuite ||-JUnit Jupiter |||- TestA ||||- method1 ||||- method2 |||- TestB ||||- method1 ||||- method2 ```` The unique identifiers are remapped by pre-pending the unique identifier of the suite. So: ``` junit-jupiter/TestA/method1() -> junit-suite/ExampleSuite/junit-jupiter/TestA/method1() ``` Issue: junit-team#744

@suite

Implements a test engine that allows declarative execution of test suites using the `@Suite` annotation. Internally the Suite Engine uses the JUnit Platform Launcher. The engine works by mapping the `TestIdentifier` used by the launcher to `TestDescriptor` used by the engine during discovery and execution. ``` package org.junit.platform.suite; import org.junit.platform.suite.api.SelectPackages; @suite @SelectPackages("org.junit.suite.testcases") class SelectPackageSuite { } ``` Is equivalent to: ``` import org.junit.platform.engine.discovery.DiscoverySelectors; import org.junit.platform.launcher.Launcher; import org.junit.platform.launcher.LauncherDiscoveryRequest; import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder; import org.junit.platform.launcher.core.LauncherFactory; public class Main { public static void main(String[] args) { Launcher launcher = LauncherFactory.create(); LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() .selectors(DiscoverySelectors.selectPackage("org.junit.suite.testcases")) .build(); launcher.execute(request); } } ``` The suite engine converts an annotated class into a discovery request. This request is executed and the resulting test plan is mapped to a tree of test descriptors. In essence the tree of test descriptors are a view on the test plan. So suppose the discovery requests produces this test plan: ``` JUnit Jupiter |- TestA ||- method1 ||- method2 |- TestB ||- method1 ||- method2 ``` Then that test plan is mapped to a tree of test descriptors like so: ``` SuiteEngine |- ExampleSuite ||-JUnit Jupiter |||- TestA ||||- method1 ||||- method2 |||- TestB ||||- method1 ||||- method2 ```` The unique identifiers are remapped by pre-pending the unique identifier of the suite. So: ``` junit-jupiter/TestA/method1() -> junit-suite/ExampleSuite/junit-jupiter/TestA/method1() ``` Issue: junit-team#744

@suite

Implements a test engine that allows declarative execution of test suites using the `@Suite` annotation. Internally the Suite Engine uses the JUnit Platform Launcher. The engine works by mapping the `TestIdentifier` used by the launcher to `TestDescriptor` used by the engine during discovery and execution. ``` package org.junit.platform.suite; import org.junit.platform.suite.api.SelectPackages; @suite @SelectPackages("org.junit.suite.testcases") class SelectPackageSuite { } ``` Is equivalent to: ``` import org.junit.platform.engine.discovery.DiscoverySelectors; import org.junit.platform.launcher.Launcher; import org.junit.platform.launcher.LauncherDiscoveryRequest; import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder; import org.junit.platform.launcher.core.LauncherFactory; public class Main { public static void main(String[] args) { Launcher launcher = LauncherFactory.create(); LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() .selectors(DiscoverySelectors.selectPackage("org.junit.suite.testcases")) .build(); launcher.execute(request); } } ``` The suite engine converts an annotated class into a discovery request. This request is executed and the resulting test plan is mapped to a tree of test descriptors. In essence the tree of test descriptors are a view on the test plan. So suppose the discovery requests produces this test plan: ``` JUnit Jupiter |- TestA ||- method1 ||- method2 |- TestB ||- method1 ||- method2 ``` Then that test plan is mapped to a tree of test descriptors like so: ``` SuiteEngine |- ExampleSuite ||-JUnit Jupiter |||- TestA ||||- method1 ||||- method2 |||- TestB ||||- method1 ||||- method2 ```` The unique identifiers are remapped by pre-pending the unique identifier of the suite. So: ``` junit-jupiter/TestA/method1() -> junit-suite/ExampleSuite/junit-jupiter/TestA/method1() ``` Issue: junit-team#744

@suite

Implements a test engine that allows declarative execution of test suites using the `@Suite` annotation. Internally the Suite Engine uses the JUnit Platform Launcher. The engine works by mapping the `TestIdentifier` used by the launcher to `TestDescriptor` used by the engine during discovery and execution. ``` package org.junit.platform.suite; import org.junit.platform.suite.api.SelectPackages; @suite @SelectPackages("org.junit.suite.testcases") class SelectPackageSuite { } ``` Is equivalent to: ``` import org.junit.platform.engine.discovery.DiscoverySelectors; import org.junit.platform.launcher.Launcher; import org.junit.platform.launcher.LauncherDiscoveryRequest; import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder; import org.junit.platform.launcher.core.LauncherFactory; public class Main { public static void main(String[] args) { Launcher launcher = LauncherFactory.create(); LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() .selectors(DiscoverySelectors.selectPackage("org.junit.suite.testcases")) .build(); launcher.execute(request); } } ``` The suite engine converts an annotated class into a discovery request. This request is executed and the resulting test plan is mapped to a tree of test descriptors. In essence the tree of test descriptors are a view on the test plan. So suppose the discovery requests produces this test plan: ``` JUnit Jupiter |- TestA ||- method1 ||- method2 |- TestB ||- method1 ||- method2 ``` Then that test plan is mapped to a tree of test descriptors like so: ``` SuiteEngine |- ExampleSuite ||-JUnit Jupiter |||- TestA ||||- method1 ||||- method2 |||- TestB ||||- method1 ||||- method2 ```` The unique identifiers are remapped by pre-pending the unique identifier of the suite. So: ``` junit-jupiter/TestA/method1() -> junit-suite/ExampleSuite/junit-jupiter/TestA/method1() ``` Issue: junit-team#744

@suite

Implements a test engine that allows declarative execution of test suites using the `@Suite` annotation. Internally the Suite Engine uses the JUnit Platform Launcher. The engine works adding the test descriptors of the discovered engines and tests as children to its own test descriptor. ```java package org.junit.platform.suite; import org.junit.platform.suite.api.SelectPackages; @suite @SelectPackages("org.junit.suite.testcases") class SelectPackageSuite { } ``` Is equivalent to: ```java import org.junit.platform.engine.discovery.DiscoverySelectors; import org.junit.platform.launcher.Launcher; import org.junit.platform.launcher.LauncherDiscoveryRequest; import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder; import org.junit.platform.launcher.core.LauncherFactory; public class Main { public static void main(String[] args) { Launcher launcher = LauncherFactory.create(); LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() .selectors(DiscoverySelectors.selectPackage("org.junit.suite.testcases")) .build(); launcher.execute(request); } } ``` Resolves #744.

koretzki · 2021-05-04T11:09:07Z

Hi All,
I am trying out this feature, replacing old Junit 4 test suites with this suite.
have a couple of questions :

Is there a working example/sample of using the test suite, that I can check out? preferably with SelectClasses
will it work in mixed Junit 4 / Junit 5 classes inside the same suite or all should be on the same engine?
thanks in advance.

marcphilipp · 2021-05-13T12:33:36Z

Is there a working example/sample of using the test suite, that I can check out? preferably with SelectClasses

There's a sample class in the user guide:
https://junit.org/junit5/docs/5.8.0-M1/user-guide/#test-suite

If that's not clear enough, we should improve it. Please let us know!

will it work in mixed Junit 4 / Junit 5 classes inside the same suite or all should be on the same engine?

Yes, you can freely mix tests from different engines in one suite.

@suite

Implements a test engine that allows declarative execution of test suites using the `@Suite` annotation. Internally the Suite Engine uses the JUnit Platform Launcher. The engine works adding the test descriptors of the discovered engines and tests as children to its own test descriptor. ```java package org.junit.platform.suite; import org.junit.platform.suite.api.SelectPackages; @suite @SelectPackages("org.junit.suite.testcases") class SelectPackageSuite { } ``` Is equivalent to: ```java import org.junit.platform.engine.discovery.DiscoverySelectors; import org.junit.platform.launcher.Launcher; import org.junit.platform.launcher.LauncherDiscoveryRequest; import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder; import org.junit.platform.launcher.core.LauncherFactory; public class Main { public static void main(String[] args) { Launcher launcher = LauncherFactory.create(); LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request() .selectors(DiscoverySelectors.selectPackage("org.junit.suite.testcases")) .build(); launcher.execute(request); } } ``` Resolves junit-team#744.

smoyer64 mentioned this issue Mar 22, 2017

Improve JavaDoc for junit-platform-suite-api annotations #746

Closed

smoyer64 changed the title ~~Introduce declarative test suites in JUnit 5~~ Introduce declarative test suites Mar 23, 2017

sbrannen changed the title ~~Introduce declarative test suites~~ Introduce support for declarative test suites Mar 24, 2017

sbrannen added theme: discovery type: enhancement theme: execution component: Jupiter component: Platform theme: programming model component: Vintage labels Mar 24, 2017

sbrannen added this to the 5.0 M5 milestone Mar 24, 2017

marcphilipp modified the milestones: 5.1+ Backlog, 5.0 M5 Apr 19, 2017

sbrannen mentioned this issue Jul 19, 2017

Introduce support for parallel test execution in declarative test suites #964

Closed

sbrannen added the theme: suites label Jul 19, 2017

carlspring mentioned this issue Dec 21, 2020

Investigate if and how we can extract the integration tests into suites carlspring/s3fs-nio#140

Closed

3 tasks

steve-todorov mentioned this issue Jan 24, 2021

Use tags instead of suites to separate tests carlspring/s3fs-nio#183

Closed

8 tasks

marcphilipp added status: in progress and removed status: team discussion labels Feb 7, 2021

marcphilipp assigned mpkorstanje and unassigned marcphilipp Feb 7, 2021

marcphilipp modified the milestones: 5.8 Backlog, 5.8 M1 Feb 7, 2021

marcphilipp linked a pull request Feb 7, 2021 that will close this issue

Introduce junit-platform-suite-engine #2416

Merged

7 tasks

marcphilipp removed component: Jupiter component: Vintage theme: discovery theme: execution theme: programming model labels Feb 7, 2021

marcphilipp closed this as completed in #2416 Feb 9, 2021

marcphilipp removed the status: in progress label Feb 9, 2021

besza mentioned this issue Mar 14, 2021

Update JUnit 4 to JUnit5 jgrapht/jgrapht#697

Closed

juliette-derancourt mentioned this issue Aug 13, 2021

Junit5 testSuite with SelectClasses not working in android unit test #2686

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce support for declarative test suites #744

Introduce support for declarative test suites #744

smoyer64 commented Mar 22, 2017 •

edited by sbrannen

Loading

mlevison commented Mar 22, 2017 via email

sbrannen commented Mar 24, 2017

sbrannen commented Mar 24, 2017

sbrannen commented Mar 24, 2017

sbrannen commented Mar 24, 2017

sbrannen commented Mar 24, 2017

sbrannen commented Mar 24, 2017

smoyer64 commented Mar 24, 2017 •

edited

Loading

mlevison commented Mar 24, 2017

sormuras commented Mar 24, 2017

sbrannen commented Mar 24, 2017

sbrannen commented Mar 24, 2017

smoyer64 commented Mar 24, 2017

mlevison commented Mar 24, 2017 via email

mlevison commented Mar 24, 2017 via email

koretzki commented May 4, 2021 •

edited

Loading

marcphilipp commented May 13, 2021

Introduce support for declarative test suites #744

Introduce support for declarative test suites #744

Comments

smoyer64 commented Mar 22, 2017 • edited by sbrannen Loading

Overview

Feature Request

Related Issues

Deliverables

mlevison commented Mar 22, 2017 via email

sbrannen commented Mar 24, 2017

sbrannen commented Mar 24, 2017

Status Quo

The Future

My Vision

sbrannen commented Mar 24, 2017

sbrannen commented Mar 24, 2017

sbrannen commented Mar 24, 2017

sbrannen commented Mar 24, 2017

smoyer64 commented Mar 24, 2017 • edited Loading

mlevison commented Mar 24, 2017

sormuras commented Mar 24, 2017

sbrannen commented Mar 24, 2017

sbrannen commented Mar 24, 2017

smoyer64 commented Mar 24, 2017

mlevison commented Mar 24, 2017 via email

mlevison commented Mar 24, 2017 via email

koretzki commented May 4, 2021 • edited Loading

marcphilipp commented May 13, 2021

smoyer64 commented Mar 22, 2017 •

edited by sbrannen

Loading

smoyer64 commented Mar 24, 2017 •

edited

Loading

koretzki commented May 4, 2021 •

edited

Loading