DynamicTest should be able to provide tags and source information to TestDescriptor #1178

jhorstmann · 2017-11-22T10:50:56Z

Overview

Currently the source for dynamic tests is a always a MethodSource pointing to the @TestFactory annotated method. In case the dynamic tests are based on external files, like in a cucumber implementation, it should be possible to specify the source.

Example

Given a file "helloword.feature" containing
  """
  Feature: Hello World Feature

    @hello @world
    Scenario: Hello World Scenario
        Given the user name is 'World'
         When the user clicks the hello button
         Then the output is 'Hello World'
  """
  When dynamic tests are created for this feature
  Then the test description should have a file source of "helloworld.feature" line 4
   And the test description should have the tags ["hello", "world"]

Related Issues

Deliverables

DynamicTest / DynamicContainer should have additional factory methods allowing to specify TestSource
❌ DynamicTest / DynamicContainer should have additional factory methods allowing to specify Set<TestTag>
The above information should be used when creating the DynamicTestTestDescriptor and DynamicContainerTestDescriptor

(The actual creation of the dynamic tests for a cucumber scenario is out of scope)

The text was updated successfully, but these errors were encountered:

sbrannen · 2017-11-22T14:47:37Z

Interesting idea.

However, there are some problems with the proposal.

First and foremost, both TestTag and TestSource reside in the junit-platform-engine artifact and are therefore not visible from the JUnit Jupiter programming model.

With regard to tags, it's less of an issue since tags could be supplied as Set<String> instead of Set<TestTag>; whereas, providing support for something like TestSource would require a new API that can be converted into TestSource.

Another thing to keep in mind is that tags are processed during the overall discovery phase; whereas, dynamic tests and containers are registered during the execution phase. So additional filtering by tags would have to be performed for dynamically registered tests and containers as well.

@junit-team/junit-lambda, thoughts?

sbrannen · 2018-02-12T16:32:20Z

Related to:

robeden · 2018-03-14T20:03:42Z

I would also like to be able to have DynamicTests deal with tags. Perhaps the two portions of this request should be broken apart so they could be dealt with separately?

Another thing to keep in mind is that tags are processed during the overall discovery phase; whereas, dynamic tests and containers are registered during the execution phase. So additional filtering by tags would have to be performed for dynamically registered tests and containers as well.

I don't like it as well as the framework dealing with it, but I suppose the TestFactory could do the filtering if it had access to information about what's included/excluded for the run.

marcphilipp · 2018-03-15T16:58:33Z

Tags would certainly be far less problematic to implement, so splitting would make sense.

@robeden What's your use case for tags on dynamic tests?

robeden · 2018-03-15T17:44:20Z

We're using JUnit as an engine for our internal system/integration test framework. Most of those tests are being written as shell scripts, which are discovered and handled by a @TestFactory. I based the framework on JUnit, so our shell script-based tests also have tags associated with them. It would be great to allow them to be included/excluded in all the same way normal tests are. It caught me by surprise to see DynamicTest didn't have support for them (as well as display name).

marcphilipp · 2018-03-16T16:03:52Z

Have you considered writing a TestEngine that supports discovering and executing the shell-script based tests?

robeden · 2018-03-16T17:35:46Z

No, I haven’t. I found @TestFactory and stopped there. I’ll take a look.

marcphilipp · 2018-03-16T17:39:40Z

There's a short section in the User Guide about it:
https://junit.org/junit5/docs/current/user-guide/#launcher-api-engines-custom

robeden · 2018-03-16T18:17:30Z

Definitely looks like I could do anything there... though the complexity considerably higher.

jhorstmann · 2018-04-23T11:52:32Z

Regarding the TestFactory vs. Engine discussion, I also tried the implementation as an engine, which works and allows reporting correct source locations. The downside is that there is then no easy way to integrate with existing jupiter extensions, at least none that I could find. Especially the spring extension for dependency injection is something I could use and would not want to duplicate for a different engine.

So for me, adding the possibility for source locations to TestFactory and maybe tag filtering of dynamic tests still sounds like the better solution.

Addresses #1178

This commit introduces the support to add an instance of `TestSource` to a dynamic container or test by adding static factory methods that take a test source as a parameter. Prior to this commit the source for dynamic tests was always a `MethodSource` pointing to the `@TestFactory` annotated method. Now, for example, it is possible to refer to external files, that are the underlying sources of generated containers and tests. Addresses part of: #1178

This commit introduces the support to add an instance of `java.net.URI` to a dynamic container or test by adding static factory methods that take a test source uri as a parameter. Prior to this commit the source for dynamic tests was always a `MethodSource` pointing to the `@TestFactory` annotated method. Now, for example, it is possible to refer to external files, that are the underlying sources of generated containers and tests. Addresses part of: #1178

Issue: #1178

This commit introduces the support to add an instance of `java.net.URI` to a dynamic container or test by adding static factory methods that take a test source uri as a parameter. Prior to this commit the source for dynamic tests was always a `MethodSource` pointing to the `@TestFactory` annotated method. Now, for example, it is possible to refer to external files, that are the underlying sources of generated containers and tests. Addresses part of: #1178

Issue: #1178

sormuras · 2018-06-14T07:14:57Z

Addressed via 54f11a2

sbrannen · 2018-06-14T08:48:39Z

👍

sbrannen · 2018-06-14T08:49:03Z

I updated the Deliverables to reflect what has been done.

Issue: #1178

sbrannen · 2018-06-15T08:19:15Z

See #1467 for potential support for classpath resources as custom test sources.

This commit introduces the support to add an instance of `java.net.URI` to a dynamic container or test by adding static factory methods that take a test source uri as a parameter. Prior to this commit the source for dynamic tests was always a `MethodSource` pointing to the `@TestFactory` annotated method. Now, for example, it is possible to refer to external files, that are the underlying sources of generated containers and tests. Addresses part of: junit-team#1178

Issue: junit-team#1178

Issue #1178 introduced support for creating a UriSource (specifically a FileSource, DirectorySource, or DefaultUriSource) from a URI supplied for a dynamic container or dynamic test. This commit further extends that feature by introducing support for creating a ClasspathResourceSource from a URI supplied for a dynamic container or dynamic test if the URI contains the "classpath" scheme. Otherwise, the behavior introduced in #1178 is used. Issue: #1467

Issue: #1178

@SInCE

…arate package with basic event timing Refactored to -tck format with package name changes + added concept of TestExecution + TerminationInfo SpotlessApply for checkstyle fixes Corrected README Refactored to junit-platform-test and pulled in testArtifact classes from junit-platform-engine Starting javadocs Corrected imports and styling Consolidated some of the API calls for execution recorder + javadocs / API docs ./gradlew spotlessApply Switched to using Instant for execution timings and minor refactoring Fix javadoc NPE by removing {@link} references to generics. Relevant ticket here: https://bugs.openjdk.java.net/browse/JDK-8188248 Bumped API version to tagged PR version of platform 1.3.x Fixed platform-tooling-support-tests for junit-platform-test module Refactor to junit-platform-testkit from junit-platform-test Fixed top level build.gradle reference for junit-platform-testkit name Fixed a few typos Fixed expected package in jar-describe-module test Fixed another package typo for jar-describe-module test Suppress warnings in tests Rename test class to DefaultParallelExecutionConfigurationStrategyTests - Added the "s" to "Test" Disable broken tests in DefaultParallelExecutionConfigurationStrategyTests This commit disables tests in DefaultParallelExecutionConfigurationStrategyTests that are broken due to issues with Mockito on Java 10. Issue: junit-team#1478 Fix DefaultParallelExecutionConfigurationStrategyTests Move test task config to testing.gradle - Remove duplication - Rename test classes to adhere to naming convention Simplify test include config in Gradle build Upgrade to Mockito 2.19.0 Upgrade to Maven Surefire 2.22.0 Upgrade to Univocity 2.6.4 Upgrade to Spotless 3.13.0 Upgrade to Dependency Versions plugin 0.20.0 Release 5.3 M1 Back to snapshots for further development Add missing date for 5.3 M1 release Upgrade to Gradle build scan 1.14 Upgrade to Fast Classpath Scanner 3.1.0 Upgrade to JRuby 9.2.0.0 Upgrade to JMH Gradle plugin 0.4.6 Upgrade to Nemerosa Versioning plugin 2.7.1 Upgrade to git-publish plugin 1.0.1 Upgrade to Kotlin 1.2.50 Upgrade to ktlint 0.24.0 Introduce log4j config for platform-tooling-support-tests project Avoid Eclipse build path cycles This commit modifies the test dependencies for the junit-jupiter-engine project so that there are no build path cycles when importing all of the projects into Eclipse. Switch to different mirror for Ant downloads Use latest Ant version Upgrade to Ant 1.10.4 Ant now supports printing the test run summary via the built-in junitlauncher task. This supersedes the need for the SummaryPrintingListener class, which is now removed from the example project. Revert JRuby upgrade due to incompatibility with asciidoctorj This reverts commit 1c3899e. Do not publish to local Maven repo unless necessary This commit reintroduces the change in 84e2abe and adds a comment to explain the rationale. Issue: junit-team#1429 Polish JavaDoc for @ResourceLock Polish JavaDoc for @ResourceLocks Remove default value for @ResourceLocks.value() Added back API functions Refactored method usage in most tests to adhere to original API Add missing @since/@API to Store.getOrComputeIfAbsent(Class) This commit adds the missing @SInCE and @API declarations to ExtensionContext.Store.getOrComputeIfAbsent(Class<V>) that were accidentally omitted in the JUnit 5.1 release. Issue: junit-team#1110 Polishing Polishing Prune Release Notes for 5.3 Closes: junit-team#1480 Introduce skeleton for 5.3 RC1 release notes Introduce "Release Notes" template Improve error message for 0 arg sets for a @ParameterizedTest Issue: junit-team#1477 Fix broken test Use simple class name as display name in JUnit Vintage Prior to this commit, there was a discrepancy between how Jupiter and Vintage generated display names for test classes. Specifically, Jupiter provided short class names as display names; whereas, Vintage provided fully qualified class names. This commit addresses this by using the simple name of the test class as the display in Vintage as well. Issue: junit-team#970 Document change in Release Notes Issue: junit-team#970 Ensure project is built successfully before publishing Polish HierarchicalTestEngine and collaborators Introduce TestInstanceFactory extension API This commit introduces a new TestInstanceFactory extension API that allows for custom creation of test instances. - TestInstanceFactory must be applied at the class level. - If multiple TestInstanceFactory extensions are registered on a single test class, an exception is thrown. - TestInstanceFactory may also be used with @nested test classes, in which case only one TestInstanceFactory is registered on each level. Issue: junit-team#672 Polish TestInstanceFactory implementation Issue: junit-team#672 Move TestInstanceFactory entry to 5.3 RC1 Release Notes Issue: junit-team#672 Polish TestInstanceFactory section in User Guide Issue: junit-team#672 Instantiate converters and aggregators only once per parameterized test Prior to this commit `ArgumentConverter` and `ArgumentsAggregator` classes were instantiated once per invocation of the `@ParameterizedTest`. With this commit, instances will be created during the first invocation of the `@ParameterizedTest` method, stored and reused on subsequent invocations. If the same converter of aggregator classes are used on other `@ParameterizedTest` methods a new instance will be created on first invocation and reused for subsequent invocations of that method. Of the two proposed options to implement this, either eager instantiation or reuse I chose reuse. The rationale was if the instantiation of a Converter of Aggregator failed with an exception this error could be caught with the existing safe mechanism and also for the failure to be very close to the impacting test invocation. Issue: junit-team#1397 Polishing Document junit-team#1397 in Release Notes Resolves junit-team#1397. Closes junit-team#1433. Make TestInstancePostProcessor a @FunctionalInterface Polish TestInstanceFactoryTests Issue: junit-team#672 Ensure TestInstanceFactory can be registered as a lambda expression Issue: junit-team#672 Test exception handling for TestInstanceFactory errors Issue: junit-team#672 Ensure TestInstanceFactory creates object of correct type Issue: junit-team#672 Resolve TestInstanceFactory at class level Prior to this commit, TestInstanceFactory extensions were resolved lazily on-demand. This led to the undesired side effect that configuration errors were reported for each test method when executing with per-method lifecycle semantics. In addition, this also meant that an attempt was made to invoke all test methods even though the test class could not be instantiated. This commit addresses these issues by resolving the TestInstanceFactory during the "prepare" phase for each ClassTestDescriptor. This commit also introduces the following methods in ExtensionRegistry in order to make the above change possible. - getParent() - getLocalExtensions(Class) Issue: junit-team#672 Handle exception thrown by TestInstanceFactory This commit ensures that an exception thrown by a TestInstanceFactory is wrapped in a TestInstantiationException, unless the exception is already a TestInstantiationException in which case it is simply rethrown. Issue: junit-team#672 Allow Assertions and Assumptions classes to be extended Prior to this commit, it was not possible to extend the Assertions and Assumptions classes since they both were final and had private constructors. This commit removes this restriction in order to support unusual circumstances in which it may be "forbidden" to use static imports. Note, however, that a note has been added to the class-level JavaDoc for these classes in order to voice the opinion of the JUnit Team that it is a best practice to use static imports for methods defined in the Assertions and Assumptions classes. Resolves: junit-team#1484 Upgrade Gradle to 4.9-rc-1 Introduce ParameterizedTestMethodContext `ParameterizedTestMethodContext` encapsulates access to the test method parameters' annotations to ensure it is only done once per `@ParameterizedTest` which significantly improves performance for methods with more than a few declared parameters. Resolves junit-team#1483. Upgrade to Gradle build-scan 1.15 Make AnnotationConsumer a @FunctionalInterface Fix and polish JavaDoc for AnnotationConsumer Polish JavaDoc Polish JavaDoc Upgrade to Kotlin 1.2.51 Use single quotes where feasible in Gradle build scripts Print arrays in failure msg for assertThrows(ThrowingSupplier) Prior to this commit, if a lambda expression provided to assertThrows() returned an array instead of throwing an exception, the array included in the failure message was not human-readable. This commit addresses this by delegating to StringUtils.nullSafeToString() in order to properly print arrays as well. Issue: junit-team#1401 Add note regarding CommonParserSettings.setNumberOfRowsToSkip() Improve exception handling for CSV errors This commit introduces custom exception handling for errors that occur while processing the @CsvSource and @CsvFileSource annotations for parameterized tests. In addition to custom error messages, this commit also introduces a dedicated CsvParsingException. Attempt to fix build on Windows Use archive.apache.org to download Ant binaries Prior to this commit a mirror was used to get the current version, 1.10.4, of the Ant binaries. When the next version, like 1.10.5, is released the mirrors usually switch to provide only the new version. This will fail our build. Now, using archive.apache.org to download Ant binaries, the build will continue to work: the archive provides the current and also former releases. Note: Tool.MAVEN already uses the same source. Refine DoD wrt. test coverage Team Decision: Make it more clear to contributors that not just happy paths should be covered by tests Safely parse FilePosition from query string Prior to this commit, FilePosition.fromQuery() would fail (and return an empty Optional) if any query parameter had a non-integer value. This commit fixes this issue by only attempting to parse a query parameter's value as an integer if the parameter's name is "line" or "column". In addition, the parsing algorithm now stops as soon as it has encountered entries for the line and column. Fixes: junit-team#1489 Implement first-one-wins algorithm in FilePosition.fromQuery() Remove unnecessary @API declaration Support classpath resource for custom TestSource in dynamic tests Issue junit-team#1178 introduced support for creating a UriSource (specifically a FileSource, DirectorySource, or DefaultUriSource) from a URI supplied for a dynamic container or dynamic test. This commit further extends that feature by introducing support for creating a ClasspathResourceSource from a URI supplied for a dynamic container or dynamic test if the URI contains the "classpath" scheme. Otherwise, the behavior introduced in junit-team#1178 is used. Issue: junit-team#1467 Add missing Release Notes entry for 5.3 M1 Issue: junit-team#1178 Polish Release Notes for 5.3 M1 Polishing Exceptions in `@AfterEach` methods fail aborted tests Issue: junit-team#1313 Exceptions in `@AfterAll` methods fail aborted containers Issue: junit-team#1313 Make ThrowableCollector configurable This commit generalizes `ThrowableCollector` to take a predicate that is used to decide whether a `Throwable` is aborted or failed execution. The Jupiter engines uses a specialized implementation that treats OTA's `TestAbortedExceptions` as aborting and everything else as failing: `OpenTest4JAwareThrowableCollector`. In addition, this commit introduces `ThrowableCollector.Factory` and lets `HierarchicalTestEngines` create them in order to allow the engine to decide how to configure its `ThrowableCollectors`. For backwards compatibility, the default implementation returns a factory that always creates instances of `OpenTest4JAwareThrowableCollector`. Issue: junit-team#1313 Use diamond notation where possible Polishing Rename keepAlive to keepAliveSeconds Prior to this commit, the Javadoc for `getKeepAlive()` in `ParallelExecutionConfiguration` mistakenly documented that the timeout's unit was milliseconds when in fact `ForkJoinPoolHierarchicalTestExecutorService` used seconds. Now, the property has been renamed to `getKeepSecondsAlive()` and the documentation has been adapted accordingly. Compile tests using Java 10 Simplify configuration of Java compile tasks Spotless! Document result of ArgumentsProvider throwing TestAbortedException Issue: junit-team#1477 Enable all recommended compiler warnings Prior to this commit, we maintained an explicit list of compiler warnings via `-Xlint:<key>` arguments. The list of possible keys grew to 26, as of javac shipped with JDK 10, and we already missed to activate some new warnings. Now all recommended compiler warnings are enabled via the `-Xlint` argument. Disable "method overrides" warnings for test sources Avoid redundant type argument warning on JDK 9+ This commit introduces a concrete implementation of the internal ResultAwareThrowingSupplierAdapter API in order to avoid a warning that results from the use of an anonymous inner class with "redundant type information" that is flagged on JDK 9 and above. Since the code base is still released using JDK 8 as the target environment, we cannot simply use the diamond notation for the anonymous inner class declaration. Upgrade to Gradle 4.9-rc-2 Reuse config from testing.gradle Polishing Limit queue size of dynamic tests Prior to this commit, concurrently executed dynamic tests (e.g. from a Jupiter `@ParameterizedTest`) were forked immediately after being reported even though all worker threads of the `ForkJoinPool` were already busy. Now, we limit the amount of queued work by using `ForkJoinTask.getSurplusQueuedTaskCount()`. We only fork execution if, according to this heuristic, the task can be executed right away. Otherwise, we execute the task in the current thread which effectively blocks further dynamic tests from being reported and queued. Issue: junit-team#1491 Add tests for additional supported @MethodSource return types A discussion in junit-team#1492 made us aware that @MethodSource factory methods can return 2D object arrays as well as other types that were neither tested nor documented. This commit introduces tests for additionally supported return types for @MethodSource factory methods. Issue: junit-team#1492 Starting refactor to minimize code diff noise based on CR Correcting some formatting problems Minimize the codediff by providing existing method calls, preserving order of method declarations, and indentation

This commit introduces the support to add an instance of `java.net.URI` to a dynamic container or test by adding static factory methods that take a test source uri as a parameter. Prior to this commit the source for dynamic tests was always a `MethodSource` pointing to the `@TestFactory` annotated method. Now, for example, it is possible to refer to external files, that are the underlying sources of generated containers and tests. Addresses part of: junit-team#1178

Issue: junit-team#1178

@SInCE

…arate package with basic event timing Refactored to -tck format with package name changes + added concept of TestExecution + TerminationInfo SpotlessApply for checkstyle fixes Corrected README Refactored to junit-platform-test and pulled in testArtifact classes from junit-platform-engine Starting javadocs Corrected imports and styling Consolidated some of the API calls for execution recorder + javadocs / API docs ./gradlew spotlessApply Switched to using Instant for execution timings and minor refactoring Fix javadoc NPE by removing {@link} references to generics. Relevant ticket here: https://bugs.openjdk.java.net/browse/JDK-8188248 Bumped API version to tagged PR version of platform 1.3.x Fixed platform-tooling-support-tests for junit-platform-test module Refactor to junit-platform-testkit from junit-platform-test Fixed top level build.gradle reference for junit-platform-testkit name Fixed a few typos Fixed expected package in jar-describe-module test Fixed another package typo for jar-describe-module test Suppress warnings in tests Rename test class to DefaultParallelExecutionConfigurationStrategyTests - Added the "s" to "Test" Disable broken tests in DefaultParallelExecutionConfigurationStrategyTests This commit disables tests in DefaultParallelExecutionConfigurationStrategyTests that are broken due to issues with Mockito on Java 10. Issue: junit-team#1478 Fix DefaultParallelExecutionConfigurationStrategyTests Move test task config to testing.gradle - Remove duplication - Rename test classes to adhere to naming convention Simplify test include config in Gradle build Upgrade to Mockito 2.19.0 Upgrade to Maven Surefire 2.22.0 Upgrade to Univocity 2.6.4 Upgrade to Spotless 3.13.0 Upgrade to Dependency Versions plugin 0.20.0 Release 5.3 M1 Back to snapshots for further development Add missing date for 5.3 M1 release Upgrade to Gradle build scan 1.14 Upgrade to Fast Classpath Scanner 3.1.0 Upgrade to JRuby 9.2.0.0 Upgrade to JMH Gradle plugin 0.4.6 Upgrade to Nemerosa Versioning plugin 2.7.1 Upgrade to git-publish plugin 1.0.1 Upgrade to Kotlin 1.2.50 Upgrade to ktlint 0.24.0 Introduce log4j config for platform-tooling-support-tests project Avoid Eclipse build path cycles This commit modifies the test dependencies for the junit-jupiter-engine project so that there are no build path cycles when importing all of the projects into Eclipse. Switch to different mirror for Ant downloads Use latest Ant version Upgrade to Ant 1.10.4 Ant now supports printing the test run summary via the built-in junitlauncher task. This supersedes the need for the SummaryPrintingListener class, which is now removed from the example project. Revert JRuby upgrade due to incompatibility with asciidoctorj This reverts commit 1c3899e. Do not publish to local Maven repo unless necessary This commit reintroduces the change in 84e2abe and adds a comment to explain the rationale. Issue: junit-team#1429 Polish JavaDoc for @ResourceLock Polish JavaDoc for @ResourceLocks Remove default value for @ResourceLocks.value() Added back API functions Refactored method usage in most tests to adhere to original API Add missing @since/@API to Store.getOrComputeIfAbsent(Class) This commit adds the missing @SInCE and @API declarations to ExtensionContext.Store.getOrComputeIfAbsent(Class<V>) that were accidentally omitted in the JUnit 5.1 release. Issue: junit-team#1110 Polishing Polishing Prune Release Notes for 5.3 Closes: junit-team#1480 Introduce skeleton for 5.3 RC1 release notes Introduce "Release Notes" template Improve error message for 0 arg sets for a @ParameterizedTest Issue: junit-team#1477 Fix broken test Use simple class name as display name in JUnit Vintage Prior to this commit, there was a discrepancy between how Jupiter and Vintage generated display names for test classes. Specifically, Jupiter provided short class names as display names; whereas, Vintage provided fully qualified class names. This commit addresses this by using the simple name of the test class as the display in Vintage as well. Issue: junit-team#970 Document change in Release Notes Issue: junit-team#970 Ensure project is built successfully before publishing Polish HierarchicalTestEngine and collaborators Introduce TestInstanceFactory extension API This commit introduces a new TestInstanceFactory extension API that allows for custom creation of test instances. - TestInstanceFactory must be applied at the class level. - If multiple TestInstanceFactory extensions are registered on a single test class, an exception is thrown. - TestInstanceFactory may also be used with @nested test classes, in which case only one TestInstanceFactory is registered on each level. Issue: junit-team#672 Polish TestInstanceFactory implementation Issue: junit-team#672 Move TestInstanceFactory entry to 5.3 RC1 Release Notes Issue: junit-team#672 Polish TestInstanceFactory section in User Guide Issue: junit-team#672 Instantiate converters and aggregators only once per parameterized test Prior to this commit `ArgumentConverter` and `ArgumentsAggregator` classes were instantiated once per invocation of the `@ParameterizedTest`. With this commit, instances will be created during the first invocation of the `@ParameterizedTest` method, stored and reused on subsequent invocations. If the same converter of aggregator classes are used on other `@ParameterizedTest` methods a new instance will be created on first invocation and reused for subsequent invocations of that method. Of the two proposed options to implement this, either eager instantiation or reuse I chose reuse. The rationale was if the instantiation of a Converter of Aggregator failed with an exception this error could be caught with the existing safe mechanism and also for the failure to be very close to the impacting test invocation. Issue: junit-team#1397 Polishing Document junit-team#1397 in Release Notes Resolves junit-team#1397. Closes junit-team#1433. Make TestInstancePostProcessor a @FunctionalInterface Polish TestInstanceFactoryTests Issue: junit-team#672 Ensure TestInstanceFactory can be registered as a lambda expression Issue: junit-team#672 Test exception handling for TestInstanceFactory errors Issue: junit-team#672 Ensure TestInstanceFactory creates object of correct type Issue: junit-team#672 Resolve TestInstanceFactory at class level Prior to this commit, TestInstanceFactory extensions were resolved lazily on-demand. This led to the undesired side effect that configuration errors were reported for each test method when executing with per-method lifecycle semantics. In addition, this also meant that an attempt was made to invoke all test methods even though the test class could not be instantiated. This commit addresses these issues by resolving the TestInstanceFactory during the "prepare" phase for each ClassTestDescriptor. This commit also introduces the following methods in ExtensionRegistry in order to make the above change possible. - getParent() - getLocalExtensions(Class) Issue: junit-team#672 Handle exception thrown by TestInstanceFactory This commit ensures that an exception thrown by a TestInstanceFactory is wrapped in a TestInstantiationException, unless the exception is already a TestInstantiationException in which case it is simply rethrown. Issue: junit-team#672 Allow Assertions and Assumptions classes to be extended Prior to this commit, it was not possible to extend the Assertions and Assumptions classes since they both were final and had private constructors. This commit removes this restriction in order to support unusual circumstances in which it may be "forbidden" to use static imports. Note, however, that a note has been added to the class-level JavaDoc for these classes in order to voice the opinion of the JUnit Team that it is a best practice to use static imports for methods defined in the Assertions and Assumptions classes. Resolves: junit-team#1484 Upgrade Gradle to 4.9-rc-1 Introduce ParameterizedTestMethodContext `ParameterizedTestMethodContext` encapsulates access to the test method parameters' annotations to ensure it is only done once per `@ParameterizedTest` which significantly improves performance for methods with more than a few declared parameters. Resolves junit-team#1483. Upgrade to Gradle build-scan 1.15 Make AnnotationConsumer a @FunctionalInterface Fix and polish JavaDoc for AnnotationConsumer Polish JavaDoc Polish JavaDoc Upgrade to Kotlin 1.2.51 Use single quotes where feasible in Gradle build scripts Print arrays in failure msg for assertThrows(ThrowingSupplier) Prior to this commit, if a lambda expression provided to assertThrows() returned an array instead of throwing an exception, the array included in the failure message was not human-readable. This commit addresses this by delegating to StringUtils.nullSafeToString() in order to properly print arrays as well. Issue: junit-team#1401 Add note regarding CommonParserSettings.setNumberOfRowsToSkip() Improve exception handling for CSV errors This commit introduces custom exception handling for errors that occur while processing the @CsvSource and @CsvFileSource annotations for parameterized tests. In addition to custom error messages, this commit also introduces a dedicated CsvParsingException. Attempt to fix build on Windows Use archive.apache.org to download Ant binaries Prior to this commit a mirror was used to get the current version, 1.10.4, of the Ant binaries. When the next version, like 1.10.5, is released the mirrors usually switch to provide only the new version. This will fail our build. Now, using archive.apache.org to download Ant binaries, the build will continue to work: the archive provides the current and also former releases. Note: Tool.MAVEN already uses the same source. Refine DoD wrt. test coverage Team Decision: Make it more clear to contributors that not just happy paths should be covered by tests Safely parse FilePosition from query string Prior to this commit, FilePosition.fromQuery() would fail (and return an empty Optional) if any query parameter had a non-integer value. This commit fixes this issue by only attempting to parse a query parameter's value as an integer if the parameter's name is "line" or "column". In addition, the parsing algorithm now stops as soon as it has encountered entries for the line and column. Fixes: junit-team#1489 Implement first-one-wins algorithm in FilePosition.fromQuery() Remove unnecessary @API declaration Support classpath resource for custom TestSource in dynamic tests Issue junit-team#1178 introduced support for creating a UriSource (specifically a FileSource, DirectorySource, or DefaultUriSource) from a URI supplied for a dynamic container or dynamic test. This commit further extends that feature by introducing support for creating a ClasspathResourceSource from a URI supplied for a dynamic container or dynamic test if the URI contains the "classpath" scheme. Otherwise, the behavior introduced in junit-team#1178 is used. Issue: junit-team#1467 Add missing Release Notes entry for 5.3 M1 Issue: junit-team#1178 Polish Release Notes for 5.3 M1 Polishing Exceptions in `@AfterEach` methods fail aborted tests Issue: junit-team#1313 Exceptions in `@AfterAll` methods fail aborted containers Issue: junit-team#1313 Make ThrowableCollector configurable This commit generalizes `ThrowableCollector` to take a predicate that is used to decide whether a `Throwable` is aborted or failed execution. The Jupiter engines uses a specialized implementation that treats OTA's `TestAbortedExceptions` as aborting and everything else as failing: `OpenTest4JAwareThrowableCollector`. In addition, this commit introduces `ThrowableCollector.Factory` and lets `HierarchicalTestEngines` create them in order to allow the engine to decide how to configure its `ThrowableCollectors`. For backwards compatibility, the default implementation returns a factory that always creates instances of `OpenTest4JAwareThrowableCollector`. Issue: junit-team#1313 Use diamond notation where possible Polishing Rename keepAlive to keepAliveSeconds Prior to this commit, the Javadoc for `getKeepAlive()` in `ParallelExecutionConfiguration` mistakenly documented that the timeout's unit was milliseconds when in fact `ForkJoinPoolHierarchicalTestExecutorService` used seconds. Now, the property has been renamed to `getKeepSecondsAlive()` and the documentation has been adapted accordingly. Compile tests using Java 10 Simplify configuration of Java compile tasks Spotless! Document result of ArgumentsProvider throwing TestAbortedException Issue: junit-team#1477 Enable all recommended compiler warnings Prior to this commit, we maintained an explicit list of compiler warnings via `-Xlint:<key>` arguments. The list of possible keys grew to 26, as of javac shipped with JDK 10, and we already missed to activate some new warnings. Now all recommended compiler warnings are enabled via the `-Xlint` argument. Disable "method overrides" warnings for test sources Avoid redundant type argument warning on JDK 9+ This commit introduces a concrete implementation of the internal ResultAwareThrowingSupplierAdapter API in order to avoid a warning that results from the use of an anonymous inner class with "redundant type information" that is flagged on JDK 9 and above. Since the code base is still released using JDK 8 as the target environment, we cannot simply use the diamond notation for the anonymous inner class declaration. Upgrade to Gradle 4.9-rc-2 Reuse config from testing.gradle Polishing Limit queue size of dynamic tests Prior to this commit, concurrently executed dynamic tests (e.g. from a Jupiter `@ParameterizedTest`) were forked immediately after being reported even though all worker threads of the `ForkJoinPool` were already busy. Now, we limit the amount of queued work by using `ForkJoinTask.getSurplusQueuedTaskCount()`. We only fork execution if, according to this heuristic, the task can be executed right away. Otherwise, we execute the task in the current thread which effectively blocks further dynamic tests from being reported and queued. Issue: junit-team#1491 Add tests for additional supported @MethodSource return types A discussion in junit-team#1492 made us aware that @MethodSource factory methods can return 2D object arrays as well as other types that were neither tested nor documented. This commit introduces tests for additionally supported return types for @MethodSource factory methods. Issue: junit-team#1492 Starting refactor to minimize code diff noise based on CR Correcting some formatting problems Minimize the codediff by providing existing method calls, preserving order of method declarations, and indentation

Issue: junit-team#1178

sbrannen added component: Jupiter theme: programming model type: enhancement labels Nov 22, 2017

sbrannen mentioned this issue Feb 12, 2018

Enhance DynamicNode with optional TestSource property #1293

Closed

sbrannen added the status: team discussion label Feb 12, 2018

sbrannen added this to the 5.2 Backlog milestone Feb 12, 2018

sormuras added the theme: dynamic tests label Feb 12, 2018

sormuras added a commit that referenced this issue Apr 28, 2018

[WIP] Add tag and test source support to dynamic containers and tests

b6cd523

Addresses #1178

sormuras mentioned this issue Apr 28, 2018

[WIP] Add tag and test source support to dynamic containers and tests #1396

Closed

7 tasks

marcphilipp modified the milestones: 5.3 Backlog, 5.3 M1 May 14, 2018

marcphilipp added the status: in progress label May 14, 2018

marcphilipp assigned sormuras May 14, 2018

sormuras added a commit that referenced this issue May 22, 2018

Add test source support to dynamic containers and tests

4dffdcd

Addresses #1178

sormuras mentioned this issue May 22, 2018

Add TestSource support to dynamic containers and tests #1428

Merged

7 tasks

sormuras added a commit that referenced this issue May 23, 2018

Add test source support to dynamic containers and tests

6aa162f

Addresses #1178

sormuras added a commit that referenced this issue May 25, 2018

Add TestSource support to dynamic containers and tests

7aff417

Addresses #1178

sormuras added a commit that referenced this issue May 25, 2018

Add TestSource support to dynamic containers and tests

fbc26a2

Addresses #1178

sormuras added a commit that referenced this issue May 27, 2018

Add TestSource support to dynamic containers and tests

273b5ea

Addresses #1178

sormuras added a commit that referenced this issue Jun 13, 2018

Polishing

6c2f912

Issue: #1178

sormuras added a commit that referenced this issue Jun 14, 2018

Polishing

5cb71ec

Issue: #1178

sormuras closed this as completed Jun 14, 2018

ghost removed the status: in progress label Jun 14, 2018

sbrannen added a commit that referenced this issue Jun 14, 2018

Polishing

b80fbab

Issue: #1178

sbrannen added component: Platform and removed status: team discussion labels Jun 14, 2018

sbrannen mentioned this issue Jun 15, 2018

Support classpath resource for custom test source in dynamic tests #1467

Closed

1 task

Andrei94 pushed a commit to Andrei94/junit5 that referenced this issue Jun 23, 2018

Polishing

732caf2

Issue: junit-team#1178

Andrei94 pushed a commit to Andrei94/junit5 that referenced this issue Jun 23, 2018

Polishing

3fea0c7

Issue: junit-team#1178

sbrannen mentioned this issue Jul 6, 2018

FilePosition.fromQuery() fails if any query parameter has a non-integer value #1489

Closed

1 task

sbrannen added a commit that referenced this issue Jul 6, 2018

Add missing Release Notes entry for 5.3 M1

3531f82

Issue: #1178

dotCipher pushed a commit to dotCipher/junit5 that referenced this issue Sep 18, 2018

Polishing

11dad09

Issue: junit-team#1178

dotCipher pushed a commit to dotCipher/junit5 that referenced this issue Sep 18, 2018

Polishing

d8226d3

Issue: junit-team#1178

dotCipher pushed a commit to dotCipher/junit5 that referenced this issue Sep 18, 2018

Add missing Release Notes entry for 5.3 M1

4da10e4

Issue: junit-team#1178

magictractor mentioned this issue Apr 8, 2019

Support MethodSource as custom test source in dynamic tests #1850

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DynamicTest should be able to provide tags and source information to TestDescriptor #1178

DynamicTest should be able to provide tags and source information to TestDescriptor #1178

jhorstmann commented Nov 22, 2017 •

edited by sbrannen

Loading

sbrannen commented Nov 22, 2017

sbrannen commented Feb 12, 2018

robeden commented Mar 14, 2018

marcphilipp commented Mar 15, 2018

robeden commented Mar 15, 2018 •

edited

Loading

marcphilipp commented Mar 16, 2018

robeden commented Mar 16, 2018

marcphilipp commented Mar 16, 2018

robeden commented Mar 16, 2018

jhorstmann commented Apr 23, 2018 •

edited

Loading

sormuras commented Jun 14, 2018

sbrannen commented Jun 14, 2018

sbrannen commented Jun 14, 2018

sbrannen commented Jun 15, 2018

DynamicTest should be able to provide tags and source information to TestDescriptor #1178

DynamicTest should be able to provide tags and source information to TestDescriptor #1178

Comments

jhorstmann commented Nov 22, 2017 • edited by sbrannen Loading

Overview

Example

Related Issues

Deliverables

sbrannen commented Nov 22, 2017

sbrannen commented Feb 12, 2018

robeden commented Mar 14, 2018

marcphilipp commented Mar 15, 2018

robeden commented Mar 15, 2018 • edited Loading

marcphilipp commented Mar 16, 2018

robeden commented Mar 16, 2018

marcphilipp commented Mar 16, 2018

robeden commented Mar 16, 2018

jhorstmann commented Apr 23, 2018 • edited Loading

sormuras commented Jun 14, 2018

sbrannen commented Jun 14, 2018

sbrannen commented Jun 14, 2018

sbrannen commented Jun 15, 2018

jhorstmann commented Nov 22, 2017 •

edited by sbrannen

Loading

robeden commented Mar 15, 2018 •

edited

Loading

jhorstmann commented Apr 23, 2018 •

edited

Loading