Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] MlDistributedFailureIT testFullClusterRestart failing #108757

Open
cbuescher opened this issue May 17, 2024 · 3 comments
Open

[CI] MlDistributedFailureIT testFullClusterRestart failing #108757

cbuescher opened this issue May 17, 2024 · 3 comments
Labels
:ml Machine learning needs:risk Requires assignment of a risk label (low, medium, blocker) Team:ML Meta label for the ML team >test-failure Triaged test failures from CI

Comments

@cbuescher
Copy link
Member

Could be related to #97574 but the error looks a bit different at first glance.

Build scan:
https://gradle-enterprise.elastic.co/s/opcpchdbcpjqu/tests/:x-pack:plugin:ml:internalClusterTest/org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT/testFullClusterRestart

Reproduction line:

./gradlew ':x-pack:plugin:ml:internalClusterTest' --tests "org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT.testFullClusterRestart" -Dtests.seed=4F35CBC70E44FC4A -Dtests.locale=ar-JO -Dtests.timezone=Asia/Yekaterinburg -Druntime.java=17 -Dtests.fips.enabled=true

Applicable branches:
8.13

Reproduces locally?:
No

Failure history:
Failure dashboard for org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT#testFullClusterRestart

Failure excerpt:

java.lang.AssertionError: 
Expected: an empty collection
     but: <[LEAK: resource was not cleaned up before it was garbage-collected.
Recent access records: 
Created at:
	org.elasticsearch.action.search.ArraySearchPhaseResults.<init>(ArraySearchPhaseResults.java:27)
	org.elasticsearch.action.search.QueryPhaseResultConsumer.<init>(QueryPhaseResultConsumer.java:85)
	org.elasticsearch.action.search.SearchPhaseController.newSearchPhaseResults(SearchPhaseController.java:813)
	org.elasticsearch.action.search.TransportSearchAction$AsyncSearchActionProvider.newSearchPhase(TransportSearchAction.java:1324)
	org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1153)
	org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:914)
	org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$8(TransportSearchAction.java:342)
	org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
	org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)
	org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)
	org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:455)
	org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:309)
	org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:113)
	org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)
	org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)
	org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)
	org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68)
	org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)
	org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105)
	org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:83)
	org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:356)
	org.elasticsearch.client.internal.support.AbstractClient.search(AbstractClient.java:491)
	org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:221)
	org.elasticsearch.xpack.ml.job.persistence.JobConfigProvider.expandJobs(JobConfigProvider.java:585)
	org.elasticsearch.xpack.ml.job.JobManager.expandJobBuilders(JobManager.java:170)
	org.elasticsearch.xpack.ml.action.TransportGetJobsAction.masterOperation(TransportGetJobsAction.java:75)
	org.elasticsearch.xpack.ml.action.TransportGetJobsAction.masterOperation(TransportGetJobsAction.java:34)
	org.elasticsearch.action.support.master.TransportMasterNodeAction.executeMasterOperation(TransportMasterNodeAction.java:125)
	org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:236)
	org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95)
	org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:241)
	org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:236)
	org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:173)
	org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:55)
	org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)
	org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)
	org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)
	org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68)
	org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)
	org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105)
	org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:83)
	org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:356)
	org.elasticsearch.xpack.core.ClientHelper.lambda$executeAsyncWithOrigin$3(ClientHelper.java:236)
	org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:221)
	org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:236)
	org.elasticsearch.xpack.ml.job.task.OpenJobPersistentTasksExecutor$RevertToCurrentSnapshotAction.tryAction(OpenJobPersistentTasksExecutor.java:543)
	org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:111)
	org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	java.base/java.lang.Thread.run(Thread.java:833)]>

  at __randomizedtesting.SeedInfo.seed([4F35CBC70E44FC4A:1E1ADAD6FB68B4E5]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
  at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2119)
  at org.elasticsearch.test.ESTestCase.checkStaticState(ESTestCase.java:729)
  at org.elasticsearch.test.ESTestCase.after(ESTestCase.java:520)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1004)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:833)
@cbuescher cbuescher added :ml Machine learning >test-failure Triaged test failures from CI labels May 17, 2024
@elasticsearchmachine elasticsearchmachine added Team:ML Meta label for the ML team needs:risk Requires assignment of a risk label (low, medium, blocker) labels May 17, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

@pxsalehi
Copy link
Member

More failures: https://gradle-enterprise.elastic.co/s/2grtkssvn4x2g/tests/task/:x-pack:plugin:ml:internalClusterTest/details/org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT/testFullClusterRestart?top-execution=1

@elasticsearchmachine
Copy link
Collaborator

This has been muted on branch 8.13.

Mute Reasons:

  • [8.13] 1.0% failure rate in test testFullClusterRestart (1.0% fail rate)

Build Scans:

elasticsearchmachine added a commit that referenced this issue May 31, 2024
@elasticsearchmachine
Mute org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT te…
d909e25 
…stFullClusterRestart #108757
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:ml Machine learning needs:risk Requires assignment of a risk label (low, medium, blocker) Team:ML Meta label for the ML team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pxsalehi @cbuescher @elasticsearchmachine