Dynamic collection name in loadElasticsearch command

[metmajer]

Thanks for providing this awesome lib! I am currently testing the loadElasticsearch command with the following configuration:

loadElasticsearch {
    collection: "myCollection"
    type: "myType"
    documentLoader: "REST" 
    elasticsearchConfig {
        hosts: ["localhost"]
    }
    ttl: 0
}

Is there a way to provide a dynamic collection name, e.g. with a date, or could you think of a simple solution to this?

Thanks and regards, Martin

[prog8]

Hi, I think I can easily provide such a solution. It shouldn't take long time to implement it.

[metmajer]

That would be great. Could you already think of an example on how this could look like? I am still a newbie to morphlines and I cannot think of single command that would allow you to specify something like this:

loadElasticsearch {
    ...
    collector: "myIndexPrefix-${DATE}"
    ...
}

How would you approach that?

Cheers, Martin

[otisg]

Side comment: is this really referred to as "collection"? If yes, that's bad because ES doesn't have a notion of "collection" (SolrCloud does). ES uses the term "index".

[prog8]

I think Martin writes about index. Morplines-ES use collection parameter and that's why Martin wrote that.
And you are right Otis that collection is not good word here. I'll change that name to "indexName" instead of collection.

[prog8]

I pushed a change (collection -> index) here: 4dcb709

[metmajer]

Btw, I have solved this issue for now by providing the values for the index and the type name in dedicated fields. The problem with this solution is, however, that these temporary fields will also be index into elasticsearch. Still hoping for a better solution.

[prog8]

Hi Martin,
I'll prepare some patch that will let you prepare dynamic index names. I think by default it can be some TimeBasedIndexNameBuilder which will implement interface IndexNameBuilder. Name builder class should configurable and if you think there should be another name builder than time based you can just provide class implementing IndexNameBuilder to classpath and configuring it. Do you like that solution?

[metmajer]

Hi Pawel,

I can't really imagine how this would look like in the end, maybe you can scribble an example configuration?
Honestly, I believe that the idea of reusing record fields to make up an index name is quite powerful.

Example:

loadElasticsearch {
    index: "@{indexName}"
    type: "@{indexType}"
    documentLoader: "REST" 
    elasticsearchConfig {
        hosts: ["localhost"]
    }
    ttl: 0
}

Since the fields 'indexName' and 'indexType' should not actually show up in the ES index, a configuration property 'ignoreFields' would come in handy to filter those fields out.

Example:

loadElasticsearch {
    ...
    ignoreFields: ["indexName", "indexType"]
    ...
}

Think this would be an easy solution that also does not require a proprietary naming solution.

What do you think?

Regards, Martin

[otisg]

@metmajer if you maybe have a patch, we'll take the PR :)

[fiserro]

I've made a pull request for this issue.
#6

[metmajer]

@otisg Sorry, missed your comment. I haven't got the patch. Hope @fiserro's pull request helps.