Loading data: Failed UK Monolith RF2 SNAPSHOT import on branch MAIN (Resolved)

[Dylan-c-93]

Hi,

I'm having issues when running the first load of data into snowstorm.

• Snowstorm version: 10.3.1
• Data: UK Monolith edition May24
• Machine: AWS EC2 t2.2xlarge
• Elasticsearch version: 8.11.1
• Java 17
• Using the release .jar and loading from the command line

Using:
sudo java -Xms8g -Xmx16g snowstorm-jar.10.3.1.jar --delete-indices --debug --elasticsearch.index.max.terms.count=1000000 --import=/home/ubuntu/snowstorm/data/monolith.zip

The load errors on "o.s.s.core.rf2.rf2import.ImportService : Failed RF2 SNAPSHOT import on branch MAIN".

I've tried also following the guidance helpfully shared here (just updating to use Elastic 8.11.1, the latest version of snowstorm, and more memory as the load was crashing at the query concept stage), and can't see where the issue creeps in. I think it's likely user error rather than a bug, but I thought it was worth checking here anyway.

I've attached the debug output.
Snowstorm_Evaluation_Report.txt

Update:

• The import works fine with the international edition
• The import works fine as per the linked comment (Elasticsearch 7.10.2, Snowstorm 8.1.0, UK Monolith edition 36.4).
• I'm not able to figure out whether a change in Elasticsearch, Snowstorm, or the data is the cause of the issue, or if it is still somehow user error.

Thanks in advance.

Error Message:
org.springframework.data.elasticsearch.UncategorizedElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed at org.springframework.data.elasticsearch.client.elc.ElasticsearchExceptionTranslator.translateExceptionIfPossible(ElasticsearchExceptionTranslator.java:105) at org.springframework.data.elasticsearch.client.elc.ElasticsearchExceptionTranslator.translateException(ElasticsearchExceptionTranslator.java:64) at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.execute(ElasticsearchTemplate.java:635) at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.searchScrollStart(ElasticsearchTemplate.java:395) at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.searchForStream(AbstractElasticsearchTemplate.java:426) at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.searchForStream(AbstractElasticsearchTemplate.java:413) at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.buildRelevantPartsOfExistingGraph(SemanticIndexUpdateService.java:549) at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.updateSemanticIndex(SemanticIndexUpdateService.java:206) at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.updateStatedAndInferredSemanticIndex(SemanticIndexUpdateService.java:131) at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.preCommitCompletion(SemanticIndexUpdateService.java:95) at io.kaicode.elasticvc.api.BranchService.completeCommit(BranchService.java:416) at io.kaicode.elasticvc.domain.Commit.close(Commit.java:61) at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl.completeImportCommit(ImportComponentFactoryImpl.java:231) at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl.loadingComponentsCompleted(ImportComponentFactoryImpl.java:220) at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.doLoadReleaseFiles(ReleaseImporter.java:251) at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.doLoadReleaseFiles(ReleaseImporter.java:203) at org.ihtsdo.otf.snomedboot.ReleaseImporter.loadSnapshotReleaseFiles(ReleaseImporter.java:46) at org.ihtsdo.otf.snomedboot.ReleaseImporter.loadSnapshotReleaseFiles(ReleaseImporter.java:80) at org.snomed.snowstorm.core.rf2.rf2import.ImportService.snapshotImport(ImportService.java:233) at org.snomed.snowstorm.core.rf2.rf2import.ImportService.importFiles(ImportService.java:180) at org.snomed.snowstorm.core.rf2.rf2import.ImportService.importArchive(ImportService.java:133) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:352) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:713) at org.snomed.snowstorm.core.rf2.rf2import.ImportService$$SpringCGLIB$$0.importArchive(<generated>) at org.snomed.snowstorm.SnowstormApplication.importEditionRF2FromDisk(SnowstormApplication.java:150) at org.snomed.snowstorm.SnowstormApplication.run(SnowstormApplication.java:110) at org.springframework.boot.SpringApplication.lambda$callRunner$4(SpringApplication.java:794) at org.springframework.util.function.ThrowingConsumer$1.acceptWithException(ThrowingConsumer.java:83) at org.springframework.util.function.ThrowingConsumer.accept(ThrowingConsumer.java:60) at org.springframework.util.function.ThrowingConsumer$1.accept(ThrowingConsumer.java:88) at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:806) at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:794) at org.springframework.boot.SpringApplication.lambda$callRunners$3(SpringApplication.java:782) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:782) at org.springframework.boot.SpringApplication.run(SpringApplication.java:341) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1358) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1347) at org.snomed.snowstorm.SnowstormApplication.main(SnowstormApplication.java:63) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:91) at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:53) at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:58) Caused by: co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed at co.elastic.clients.transport.ElasticsearchTransportBase.getApiResponse(ElasticsearchTransportBase.java:345) at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:147) at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1897) at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.lambda$searchScrollStart$17(ElasticsearchTemplate.java:395) at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.execute(ElasticsearchTemplate.java:633) ... 54 common frames omitted

[kaicode]

Thanks for reporting this.

Has anyone else in the community tried importing the UK Monolith package on the latest Snowstorm and Elasticsearch 8?

[mohammada-huma]

Hi, yes I did the same and it failed with the same error.

[kaicode]

The "all shards failed" message points towards an Elasticsearch issue.
Errors and Warnings from the Elasticsearch log are needed to help debug this.

[mohammada-huma]

I don't have access now, but I tested the same dataset with old v7.2.0 and it passed that phase without any errors.
I think it's great if you can check UK monolith 38.4.0 with the latest docker-compose from the repo.

[kaicode]

I attempted to reproduce the issue.
In my case after about an hour of loading the monolith package the Elasticsearch dies with this error:

2024-08-12 18:51:00 ERROR: Elasticsearch exited unexpectedly, with exit code 137

This indicates that Elasticsearch ran out of memory.
It's not clear if this error is due to the 5g memory limit on the Elasticsearch container in the docker compose file or the 6g overall Docker limit I have set on my machine.

I will just increase the overall Docker limit to 8g, without changing the Elasticsearch container limit, and run again.

[mohammada-huma]

I tried with 6gb elasticsearch in docker and server with -xmx8g [running snowstorm from local IDE to connect to elastic in docker], aka 14gb total and I still received out of memory for snowstorm.

In the end I succeeded to import data with 26gb total docker memory [elastic+snowstorm] on these versions:

• elasticsearch:8.11.1
• snomedinternational/snowstorm:10.2.1

Settings: java -Xms2g -Xmx16g --add-opens java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED -cp @/app/jib-classpath-file org.snomed.snowstorm.SnowstormApplication --elasticsearch.urls=http://es:9200 --elasticsearch.index.max.terms.count=2147483647

[kaicode]

Without setting the minimum memory it's hard to tell how much memory was actually used by that Snowstorm process when it ran out. With -Xms2g -Xmx16g we know that it had 2g because otherwise the jvm would have failed to start the process, but unless there was another 14g free on the machine it may not have been able to reserve much more. Also running in debug from an IDE can cause much more memory to be used.
Do you happen to know how much memory Snowstorm actually used before running out when importing the UK monolith package?

[kaicode]

I got the same issue with Elasticsearch exiting with out of memory exit code when Elastic had 5g Docker 8g.
Increasing memory for another run...

[mohammada-huma]

I cannot say with certainty, all I know is that 8gb for snowstorm alone was not enough for me, so minimum should be something more than 8gb for uk monolith.

For elastic, I used this setting: "ES_JAVA_OPTS=-Xms4g -Xmx8g" which worked for me.
The issue is that I don't own a 32gb system myself, and I borrowed it, so I cannot test with that anymore.

[kaicode]

My latest import of the UK Monolith package completed.

• Snowstorm version: 10.3.1
• Data: UK Monolith 38.4.0
• Machine: MacBook Pro M1
• Elasticsearch version: 8.11.1
• Java 17
• Docker Overall Memory Limit 12g

Docker Compose changes:

services:
  elasticsearch:
  ...
    environment:
    ...
      - "ES_JAVA_OPTS=-Xms6g -Xmx6g"
    ...
    mem_reservation: 6g

  snowstorm:
  ...
    entrypoint: java -Xms4g -Xmx4g --add-opens java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED -cp @/app/jib-classpath-file org.snomed.snowstorm.SnowstormApplication --elasticsearch.urls=http://es:9200 --elasticsearch.index.max.terms.count=1000000
  ...
    mem_reservation: 4g

Log:

snowstorm      | 2024-08-13T17:02:21.492Z  INFO 1 --- [pool-3-thread-1] o.s.s.core.rf2.rf2import.ImportService   : Completed RF2 SNAPSHOT import on branch MAIN in 4788 seconds. ID 07a3e926-6636-4b2b-a975-d30a6ba99c7f

It seems to be okay with a bit more memory.

Once the import has completed the memory requirements can be reduced again. 2-4g for Elastic and the same for Snowstorm should be plenty.