How to specify ElasticSearch IndexName?

[sarunasil]

I've been previously using the [https://github.com/serilog/serilog-sinks-elasticsearch](ElasticSearch Sink) and it has a neat feature to be able to specify the index name that ElasticSearch needs to log the message into. Therefore, I want to do the same thing using this sink via logstash.

Essentially, how can one set output{ elasticsearch { index => } } from the default "%{ [@metadata][beat] } ..." ?
Is it possible to add additional @metadata using this sink to then use it as an index name?

Thanks in advance

[FantasticFiasco]

I don't know since I've never tried it.

This is a simple HTTP sink, that happens to be able to send log events to the HTTP input of Logstash. For your suggestion to work there has to be some interaction between Logstash and ElasticSearch, and if there are any requirements on the payload, you can either modify the payload of the HTTP POST using a batch formatter or a event formatter.

Will that get you closer to a solution?

[sarunasil]

Hmm, I'll try to clarify:
I think 'special interaction' between Logstash and ElasticSearch already exists as the index name for the incoming event can be specified in the Logstash e.g.:

output {
  elasticsearch {
    hosts => "http://xx.xx.xx.xx:9200 3"
    index => "hashes-%{+YYYY.MM.dd}"
  }
}

It also appears that the default index name is index => "%{[@metadata][beat]}-%{[@metadata][version]}" where [@ metadata][beat] and [@ metadata][version] comes from the Event that's being logged. source

Then, it seems to me that, by only using the existing HTTP sink code, it is impossible to send events to multiple indexes via the same Logstash instance because all the events would go to the one index. That is, e.g. I want to use the same Logstash (and ElasticSearch) instance to log Event messages from 2 separate applications into 2 separate indexes. Is this the case?

So the question then, essentially, boils down to: how can I add a new static field in JSON HTTP POST payload which would be called 'metadata' (or smth else that's custom) and accessible from Logstash so that the field or it's child field be used as part of the index name? My guess would be that I need to implement my own instance of ITextFormatter or extend one of the existing Event formatters but I'm not certain. Any ideas?

[FantasticFiasco]

Ok, adding a new property to the JSON event is pretty simple, I commited the necesary code here. Don't pay any respect to the fact that I changed from DurableHttpUsingFileSizeRolledBuffersto Http and from netcoreapp2.0 to netcoreapp5.0. It has no relevance for your use-case.

What you will need to do is to create your own implementation of ITextFormatter, I named my class MetadataTextFormatter. It's basically a copy of NormalRenderedTextFormatter. You can base your implementation of any of the other formatters if you wan't to.

The only thing I added was the property @metadata, and it's just one line of code. The JSON event now looks something like this.

{
  "Timestamp": "2020-12-28T09:37:39.1158230+01:00",
  "Level": "Information",
  "MessageTemplate": "{@customer} placed {@order}",
  "RenderedMessage": "Customer { FirstName: \"Jalon\", Surname: \"Wiza\", SocialSecurityNumber: \"970-57-4369\" } placed Order { Id: 0, Item: \"apple\", Quantity: 8 }",
  "Properties": { 
    "customer": {
      "FirstName": "Jalon",
      "Surname": "Wiza",
      "SocialSecurityNumber": "970-57-4369",
      "_typeTag": "Customer"
    },
      "order": {
      "Id": 0,
      "Item": "apple",
      "Quantity": 8,
      "_typeTag": "Order"
    },
    "SourceContext": "SerilogExample.Program"
  },
  "@metadata": {
    "app": "my-app",
    "version": "1.2.3"
  }
}

Please update your application accordingly and verify that it works as expected.

[sarunasil]

Yes, it does seem to work. Ofc, it depends what is the index format on logstash, thus I ended up passing parameters through the constructor to set the index name from a config file.

So all seems to be good, thank you!