Support ES 7.0 Issue #39

[lucaswin-amzn]

Issue: #39

Description of changes: Update plugin to work with Elasticsearch 7.0
Breaking changes in 7.0

Build output:

alerting lucaswin$ ./gradlew clean

> Configure project :alerting
=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 5.2.1
  OS Info               : Mac OS X 10.12.6 (x86_64)
  JDK Version           : 12 (Oracle Corporation 12.0.1 [Java HotSpot(TM) 64-Bit Server VM 12.0.1+12])
  JAVA_HOME             : /Library/Java/JavaVirtualMachines/jdk-12.0.1.jdk/Contents/Home
  Random Testing Seed   : AFB3B27A7E2DB611
=======================================

Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 3s
4 actionable tasks: 2 executed, 2 up-to-date

alerting lucaswin$ ./gradlew build

> Configure project :alerting
=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 5.2.1
  OS Info               : Mac OS X 10.12.6 (x86_64)
  JDK Version           : 12 (Oracle Corporation 12.0.1 [Java HotSpot(TM) 64-Bit Server VM 12.0.1+12])
  JAVA_HOME             : /Library/Java/JavaVirtualMachines/jdk-12.0.1.jdk/Contents/Home
  Random Testing Seed   : C61BB471DAE4C0B3
=======================================

> Task :alerting-notification:compileJava
Note: /Users/lucaswin/Desktop/github/alerting-backend/alerting/notification/src/main/java/com/amazon/opendistroforelasticsearch/alerting/destination/Notification.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :alerting:unitTest
==> Test Info: seed=C61BB471DAE4C0B3; jvms=2; suites=4
==> Test Summary: 4 suites, 17 tests

> Task :alerting:integTestRunner
==> Test Info: seed=C61BB471DAE4C0B3; jvm=1; suites=4
==> Test Summary: 4 suites, 69 tests

> Task :alerting-notification:test
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.easymock.cglib.core.ReflectUtils$1 (file:/Users/lucaswin/.gradle/caches/modules-2/files-2.1/org.easymock/easymock/4.0.1/26f1c39bd323b6ac7fe0c4c8e33bfdffcdde03a2/easymock-4.0.1.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.easymock.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 2m 21s
50 actionable tasks: 49 executed, 1 up-to-date

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jinsoor-amzn]

I see that we don't need to have doc types such as ScheduledJob.SCHEDULED_JOB_TYPE and
AlertIndices.MAPPING_TYPE. Can we remove them now?

[abbashus]

Just a side note: Do not create branches for pull request on the main repository. Use forks to create pull requests.
Sample PR workflow: https://gist.github.com/Chaser324/ce0505fbed06b947d962

[lucaswin-amzn]

Just a side note: Do not create branches for pull request on the main repository. Use forks to create pull requests.
Sample PR workflow: https://gist.github.com/Chaser324/ce0505fbed06b947d962

Thanks will do that for the next PR.

[alolita]

+1 to using forks for all PRs. Thanks @lucaswin-amzn.

[ghost]

alerting/alerting/src/main/kotlin/com/amazon/opendistroforelasticsearch/alerting/alerts/AlertMover.kt

Lines 98 to 104 in 10d51ef

	IndexRequest(AlertIndices.HISTORY_WRITE_INDEX)
	.routing(monitorId)
	.source(Alert.parse(alertContentParser(hit.sourceRef), hit.id, hit.version)
	.copy(state = Alert.State.DELETED)
	.toXContent(XContentFactory.jsonBuilder(), ToXContent.EMPTY_PARAMS))
	.setIfSeqNo(hit.seqNo)
	.setIfPrimaryTerm(hit.primaryTerm)

This doesn't seem correct - how will the seqNo and primaryTerm be retained when moving from the active index to the history index?

[lucaswin-amzn]

I am not sure what you are asking. We were keeping the version prior, which is still supported since we are using an external versioning system, but since versions are going away, we might want to keep the document the same as a direct copy from the ACTIVE state to DELETED state between the indexes.

[ghost]

Sequence numbers and primary terms are specific to an index. They cannot be carried over from the hit in the active index to the copy in the history index. To avoid stale writes on the alert in the history index we need to continue to use external versioning which is still supported in ES 7: https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#_internal_versioning_is_no_longer_supported_for_optimistic_concurrency_control

[lucaswin-amzn]

@jinsoor-amzn The _doc has been removed. I thought it was still needed but in fact is not.

[mihirsoni]

LGTM, Thanks for the changes.

[lucaswin-amzn]

@jinsoor-amzn The _doc has been removed. I thought it was still needed but in fact is not.

My mistake @jinsoor-amzn, ScheduledJob.SCHEDULED_JOB_TYPE is still needed for initializing mappings in ScheduledJobIndices during the CreateIndexRequest with mappings as .mappings() required a doc type.

It was removed from Alert.kt as we create the index at indexing time of the first alert.