filter_lua: add support for log metadata handling

[edsiper]

DRAFT PR

Introduces a v2 APi for the Lua functions that allows the handling of Log metadata, e.g:

pipeline:
  inputs:
    - name: dummy
      processors:
        logs:
          - name: lua
            api: v2
            call: test_v2
            code: |
              function test_v2(tag, timestamp, metadata, body)
                metadata['meta_test'] = 'ok'
                body['body_test'] = 'ok'
                return 2, timestamp, metadata, body
              end
  outputs:
    - name : stdout
      match: '*'

output:

bin/fluent-bit -c ../conf/fluent-bit-lua.yaml
Fluent Bit v3.2.3
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____ 
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /  
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2024/12/09 22:50:57] [ info] [fluent bit] version=3.2.3, commit=412d3ea818, pid=447239
[2024/12/09 22:50:57] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/12/09 22:50:57] [ info] [simd    ] disabled
[2024/12/09 22:50:57] [ info] [cmetrics] version=0.9.9
[2024/12/09 22:50:57] [ info] [ctraces ] version=0.5.7
[2024/12/09 22:50:57] [ info] [input:dummy:dummy.0] initializing
[2024/12/09 22:50:57] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/12/09 22:50:57] [ info] [sp] stream processor started
[2024/12/09 22:50:57] [ info] [output:stdout:stdout.0] worker #0 started
[0] dummy.0: [[1733806258.333027665, {"meta_test"=>"ok"}], {"body_test"=>"ok", "message"=>"dummy"}]
[0] dummy.0: [[1733806259.332974330, {"meta_test"=>"ok"}], {"body_test"=>"ok", "message"=>"dummy"}]

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[tarruda]

Hi @edsiper !

Not sure if you saw, but a few months ago I created a PR which exposes log metadata to Lua in a backwards compatible way: #9323 . #9323 adds a new processor, so internally it is not compatible with filter API. In any case I thought you might find it useful to look at how the public Lua API looks like there, so it could potentially be implemented in a similar manner.