Elasticsearch 7.0 Removal of mapping types

[paul]

Re: https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

I'm unable to use chewy to create indexes in Elasticsearch 7.0. Chewy attempts to create indexes with mappings that looks like this:

PUT /posts
{
  "settings": { "analysis": { ... } },
  "mappings" : {
    "post": { 
      "properties": { "id": { "type": "keyword" } , etc }
    }
  }
}

ES7 has removed the "mapping type" level from the "mappings" field, so now expects documents that look like this:

{
  "settings": { "analysis": { ... } },
  "mappings" : {
    "properties": { "id": { "type": "keyword" } , etc }
  }
}

For the 7.x series, the old way is still supported, but only my appending a ?include_type_name=true query param to the create index URL.

PUT posts?include_type_name=true

I tried a few ways to set it in the body, but that doesn't seem possible. It also doesn't seem possible for chewy to provide a query param when creating the index.

[noellabo]

The following is an example of a workaround with compatibility parameters.
noellabo@94e8a6c

As it is possible to respond by rewriting the URL, some people have adopted a method to solve using nginx.
https://gist.github.com/Vincanote/2b18ba8aa33d068a9cc012b9a4d891af

[Dee-2015]

Just linking to my other comment

mastodon/mastodon#10664 (comment)

[StephRicardo]

Is there a plan to update Chewy to work with ES 7 or to use the workaround for now?

[pyromaniac]

Hello everyone. There is definitely a plan to update Chewy. Unfortunately, I can't tell when it is going to happen exactly. Hopefully, sooner than later. I'm sorry for it.

[silva96]

Hi, is there at least a branch where we can coordinate this? I would like to contribute.

[darkleaf]

# config/initializers/elastic_7.rb
# https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#hits-total-now-object-search-response
# https://www.elastic.co/blog/moving-from-types-to-typeless-apis-in-elasticsearch-7-0
# https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html#_schedule_for_removal_of_mapping_types

Elasticsearch::Transport::Client.prepend Module.new {
  def search(arguments = {})
    arguments[:rest_total_hits_as_int] = true
    super arguments
  end
}
Elasticsearch::API::Indices::IndicesClient.prepend Module.new {
  def create(arguments = {})
    arguments[:include_type_name] = true
    super arguments
  end

  def put_mapping(arguments = {})
    arguments[:include_type_name] = true
    super arguments
  end
}

It works with Chewy 5.1.0 and ElasticSearch 7.7.0 or 6.8.1

[subelsky]

this PR fixes by @przbadu fixes the issue. I've tested it locally and works with Elasticsearch 7 API: #712

[silva96]

Elasticsearch 6.8.x (pre 7.x) is EOL tomorrow, https://www.elastic.co/es/support/eol

any plans to make chewy compatible?

[rapito]

Disclaimer: This only works for multiple mappings on an Index that share the same properties.

On my case I had indexes with multiple types but same attribute structure, so #673 (comment) and #712 didn't work for me by themselves.

I had to go further to make sure that the Index Create statements would only send a single mapping to ES but keep separate type definitions on Chewy. This is very important as it allows Active Record imports for separate Tables/Models.

Additionally, in order to allow the actual imports, Chewy::Type::Import was monkey patched as well to always send the first defined type as the mapping to ES.

The overall structure looks something like this:

class AnimalsIndex < Chewy::Index

	define_type Cats.clever_query, name: :cats do
		field :name, value: ->(obj) { obj.name }
		field :lives, value: ->(obj) { obj.owner.lives * 100 }
		field :owner, value: ->(obj) { obj.devil.name }
	end

	define_type Dogs.clever_query, name: :dogs do
		field :name, value: ->(obj) { obj.name }
		field :lives, value: ->(obj) { obj.alive ? 1 : 0 }
		field :owner, value: ->(obj) { obj.owner.name }
	end

end

In addition to #673 (comment), this is the additional monkey patched we added on initializers/elastic.rb:

# Replaced create statement statement of indexes to only send first mapping to ES
module Chewy
	class Index
		def self.create!(suffix = nil, **options)
			options.reverse_merge!(alias: true)
			general_name = index_name
			suffixed_name = index_name(suffix: suffix)
			body = specification_hash
			body[:aliases] = {general_name => {}} if options[:alias] && suffixed_name != general_name

			# Monkey Patch: Let's just remove any mappings and use the first one we have
			body[:mappings] = {body[:mappings].keys.first => body[:mappings].values.first} if body[:mappings].try(:count).to_i > 1

			result = client.indices.create(index: suffixed_name, body: body)

			Chewy.wait_for_status if result
			result
		end
	end
end

# Replaced import object to always use the first mapping for all types of an index
module Chewy
	class Type
		module Import
			class BulkRequest
				def initialize(type, suffix: nil, bulk_size: nil, **bulk_options)
					@type = type
					@suffix = suffix
					@bulk_size = bulk_size - 1.kilobyte if bulk_size # 1 kilobyte for request header and newlines
					@bulk_options = bulk_options
					# Monkey Patch: 
					@type = @type.index.types.first #always replace type with first type
					raise ArgumentError, '`bulk_size` can\'t be less than 1 kilobyte' if @bulk_size && @bulk_size <= 0
				end
			end
		end
	end
end

This is definitely very hacky and prone to serious issues in the future, but waiting for official support on ES 7 is no longer an option so had to workaround it. Hope it helps someone.

[rabotyaga]

Please take a look at Chewy 7.0.0 release. Stay tuned for the 7.1 release, with dropped support for ES6.8 and removed deprecation warnings.