Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] SearchableSnapshotsPrewarmingIntegTests testConcurrentPrewarming failing #99124

Closed
original-brownbear opened this issue Sep 1, 2023 · 3 comments · Fixed by #101203
Closed

[CI] SearchableSnapshotsPrewarmingIntegTests testConcurrentPrewarming failing #99124

original-brownbear opened this issue Sep 1, 2023 · 3 comments · Fixed by #101203
Assignees
@arteam
Labels
blocker :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed Meta label for distributed team (obsolete) >test-failure Triaged test failures from CI

Comments

@original-brownbear
Copy link
Member

Build scan:
https://gradle-enterprise.elastic.co/s/5g3re76rgzz4o/tests/:x-pack:plugin:searchable-snapshots:internalClusterTest/org.elasticsearch.xpack.searchablesnapshots.cache.full.SearchableSnapshotsPrewarmingIntegTests/testConcurrentPrewarming

Reproduction line:

./gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.cache.full.SearchableSnapshotsPrewarmingIntegTests.testConcurrentPrewarming" -Dtests.seed=367AEC923A22F7A8 -Dtests.locale=tr-TR -Dtests.timezone=Etc/GMT+3 -Druntime.java=20

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.searchablesnapshots.cache.full.SearchableSnapshotsPrewarmingIntegTests&tests.test=testConcurrentPrewarming

Failure excerpt:

java.lang.AssertionError: 
Expected: <21>
     but: was <20>

  at __randomizedtesting.SeedInfo.seed([367AEC923A22F7A8:7E197EE4537CC3F7]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.junit.Assert.assertThat(Assert.java:956)
  at org.junit.Assert.assertThat(Assert.java:923)
  at org.elasticsearch.xpack.searchablesnapshots.cache.full.SearchableSnapshotsPrewarmingIntegTests.testConcurrentPrewarming(SearchableSnapshotsPrewarmingIntegTests.java:271)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1623)
@original-brownbear original-brownbear added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Sep 1, 2023
@elasticsearchmachine elasticsearchmachine added the Team:Distributed Meta label for distributed team (obsolete) label Sep 1, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@piergm
Copy link
Member

piergm commented Oct 11, 2023

Occurred again today: https://gradle-enterprise.elastic.co/s/6o2csevwuijhc

@davidkyle
Copy link
Member

Another https://gradle-enterprise.elastic.co/s/iplqhi36iw642/tests/task/:x-pack:plugin:searchable-snapshots:internalClusterTest/details/org.elasticsearch.xpack.searchablesnapshots.cache.full.SearchableSnapshotsPrewarmingIntegTests/testConcurrentPrewarming?top-execution=1

@arteam arteam self-assigned this Oct 18, 2023
arteam added a commit to arteam/elasticsearch that referenced this issue Oct 23, 2023
@arteam
Add a latch to sync when all max upload tasks have been submitted
67eca02 
Before we check the amount of active tasks on the prewarming executor,
we need to verify that all the tasks have been actually submitted.

Otherwise, we have a race in and amount of active tasks can be lower
then the amount of submitted tasks.

Fixes elastic#99124
@arteam arteam mentioned this issue Oct 23, 2023
Merged
arteam added a commit that referenced this issue Oct 23, 2023
@arteam @DaveCTurner
Add a latch to sync when all max upload tasks have been submitted (#1…
13bf223 
…01203)

Before we check the amount of active tasks on the prewarming executor,
we need to verify that all the tasks have been actually submitted.

Otherwise, we have a race in and amount of active tasks can be lower
then the amount of submitted tasks.

Fixes #99124

---------

Co-authored-by: David Turner <[email protected]>
@nicktindall nicktindall mentioned this issue Sep 4, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arteam arteam

Labels
blocker :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed Meta label for distributed team (obsolete) >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add a latch to sync when all max upload tasks have been submitted arteam/elasticsearch
5 participants
@arteam @davidkyle @original-brownbear @elasticsearchmachine @piergm