[Spark] Set spark.testing flag in tests #4119
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
Currently, the
Utils.isTesting
check in tests is not reliable because thespark.testing
system property is not set before the suites start, so the tests are relying on this line inSparkFunSuite
to set the test indicator. This is causing issues in CI because it means whetherUtils.isTesting
returns correct result depends on whether it is called afterSparkFunSuite::beforeAll
is called.For example, some deletion vector-related suites are relying on the
TEST_DV_NAME_PREFIX
config to be correctly initialized in tests. However, if a previous test in the same JVM group initialized this config beforespark.testing
is correctly set bySparkFunSuite
, the deletion vector suites would see the incorrect value ofTEST_DV_NAME_PREFIX
and fail (This is currently not an issue in master by luck but if a PR added new tests that changed the JVM grouping in CI job (#4039, #4040), this would incorrectly fail the PR's spark test job).This PR fixes this by setting
spark.testing
to true in build file. Note that this flag is already set in the Spark repo but not here yet.How was this patch tested?
Check that the
TEST_DV_NAME_PREFIX
is correctly set for tests that runUtils.testing
beforeSparkFunSuite::beforeAll
(e.g.,org.apache.spark.sql.delta.DeltaWithNewTransactionWithCoordinatedCommitsBatch100Suite
)Does this PR introduce any user-facing changes?
No