Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added fluentd-record-modifier plugin #349

Merged
merged 3 commits into from
Feb 7, 2020

Conversation

alexxed
Copy link
Contributor

@alexxed alexxed commented Feb 7, 2020

because it's faster than record-transformer

Q A
Bug fix? no
New feature? yes
API breaks? no
Deprecations? no
Related tickets fixes #348
License Apache 2.0

What's in this PR?

Added the fluentd record-modifier plugin.

Why?

It's at least twice as fast compared to record-transformer

Checklist

  • Error handling code meets the guideline
  • Logging code meets the guideline (TODO)
  • User guide and development docs updated (if needed)
  • Related Helm chart(s) updated (if needed)

@alexxed
Copy link
Contributor Author

alexxed commented Feb 7, 2020

Needs a manual test.

@tarokkk tarokkk requested review from pepov, tarokkk and ahma and removed request for pepov and tarokkk February 7, 2020 13:47
@tarokkk tarokkk added community enhancement New feature or request labels Feb 7, 2020
Copy link
Contributor

@tarokkk tarokkk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ;) needs some testing thought

Copy link
Contributor

@ahma ahma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great Job Thanks @alexxed LGTM

@ahma ahma added this to the 3.0.0 milestone Feb 7, 2020
@alexxed
Copy link
Contributor Author

alexxed commented Feb 7, 2020

Tested with this logging configuration and a cluster flow that sends to sumo logic:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: logging
  namespace: logging
spec:
  enableRecreateWorkloadOnImmutableFieldChange: true
  fluentd:
    image:
        repository: alexandruszaszacquia/fluentd
        tag: record-modifier
        pullPolicy: IfNotPresent
    metrics:
      port: 24231
  fluentbit:
    metrics:
      port: 2020
  controlNamespace: logging

cluster flow

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: clusterflow-forward-logs-to-sumologic
  namespace: logging
spec:
  filters:
    - record_modifier:
       records:
       - stage: dev
       - aws_local_ip: ${record.dig("kubernetes", "host").to_s.gsub(/ip-([^\.]+)\.ec2\.internal/, '\1').gsub(/-/, '.')}
  selectors: {}
  outputRefs:
    - output-acquia-sumologic

and got this record:
image

I haven't manually tested the other options, e.g. replace, whitelist; but there's a unit test with all of them. As long as that is cross referenced with the plugin's documentation, that should be good to go.

@tarokkk tarokkk merged commit 4c6156a into kube-logging:master Feb 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add the fluentd record_modifier plugin
3 participants