Add out_kafka daemonset #11
Conversation
with Yolean/kubernetes-kafka brokers.
but beware of the aggregation recursion with the test, you'll see escaped escaped ... escaped json.
|
Note: there is a Message_Key optional config parameter available
…On Nov 11, 2017 09:06, "solsson" ***@***.***> wrote:
Based on fluent/fluent-bit#94
<fluent/fluent-bit#94>, currently the dev build
from fluent/fluent-bit#94 (comment)
<fluent/fluent-bit#94 (comment)>
A typical record (pretty-printed using jq):
{
"kubernetes": {
"docker_id": "dd50658b5d1110070dfa24511b7c2e6095bc210b7807617bc9c7e9705c64be28",
"container_name": "fluent-bit",
"host": "minikube",
"annotations": {
"kubernetes.io/created-by": "{\\\"kind\\\":\\\"SerializedReference\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"reference\\\":{\\\"kind\\\":\\\"DaemonSet\\\",\\\"namespace\\\":\\\"logging\\\",\\\"name\\\":\\\"fluent-bit\\\",\\\"uid\\\":\\\"7b5e7b6d-c6ef-11e7-ada5-080027039766\\\",\\\"apiVersion\\\":\\\"extensions\\\",\\\"resourceVersion\\\":\\\"1695\\\"}}\\n"
},
"labels": {
"version": "v1",
"pod-template-generation": "1",
"kubernetes.io/cluster-service": "true",
"k8s-app": "fluent-bit-logging",
"controller-revision-hash": "2050036262"
},
"pod_id": "af021eb0-c6ef-11e7-ada5-080027039766",
"namespace_name": "logging",
"pod_name": "fluent-bit-qpz69"
},
"time": "2017-11-11T15:00:40.427239619Z",
"stream": "stderr",
"log": "[2017/11/11 15:00:40] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-1.broker.kafka.svc.cluster.local:9092/bootstrap]: kafka-1.broker.kafka.svc.cluster.local:9092/1: Receive failed: Disconnected\\n",
***@***.***": 1510412440.42724
}
Notably, compared to filebeat 6.0.0-rc (Yolean/kubernetes-kafka#88
<Yolean/kubernetes-kafka#88>), it contains node
name and annotations. Also notably, like filebeat, no key is set.
------------------------------
You can view, comment on, or merge this pull request online at:
#11
Commit Summary
- Copies output/elasticsearch/
- Configures current dev build of out_kafka,
File Changes
- *A* output/kafka/fluent-bit-configmap.yaml
<https://github.com/fluent/fluent-bit-kubernetes-logging/pull/11/files#diff-0>
(93)
- *A* output/kafka/fluent-bit-ds-minikube.yaml
<https://github.com/fluent/fluent-bit-kubernetes-logging/pull/11/files#diff-1>
(50)
- *A* output/kafka/fluent-bit-ds.yaml
<https://github.com/fluent/fluent-bit-kubernetes-logging/pull/11/files#diff-2>
(44)
Patch Links:
- https://github.com/fluent/fluent-bit-kubernetes-logging/pull/11.patch
- https://github.com/fluent/fluent-bit-kubernetes-logging/pull/11.diff
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#11>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAWkNtINQLbvtcT5TxNm--GKVlVkSQmdks5s1bfdgaJpZM4QahKX>
.
|
How can I use record fields with that option, for example |
|
@solsson that feature is not yet available, meaning the "ability to compose a value based on record keys" |
| Match * | ||
| Brokers bootstrap.kafka:9092 | ||
| Topics ops.kube-logs-fluentbit.stream.json.001 | ||
| Timestamp_Key @timestamp |
There was a problem hiding this comment.
There is an option called Retry_Limit set to False, that means if Fluent Bit cannot flush the records to Kafka it will re-try indefinitely until it succeeds.
What are your thoughts?
There was a problem hiding this comment.
I was unaware of this but now I found fluent/fluent-bit#309 (comment). I guess the daemonset pods need some monitoring, like a gauge for the number of messages in buffer.
If one message fails it's quite unlikely that the next will be sent, so I think Retry_Limit=False makes sense to preserve ordering, in combination with a very narrow memory limit. It's a decent alternative to buffer monitoring, because if the pod hits the memory limit it will be replaced and generic monitoring on readiness (for example using prometheus-operator and this example) will alert about the issue.
|
Hrm... @solsson with your recent changes to use the bootstrap service (and @edsiper adding the Retry_Limit False ) I'm getting occasional messages: I'm going to look into it a bit more with debugging on. |
|
@StevenACoffman That's quite odd. Unless I'm mistaken Kafka producers will only use the boostrap servers string for the initial connect to brokers, then immediately get proper advertised addresses. I'm prepared to revert that change, but I would like to find the cause first. |
|
@edsiper I'm not sure the Just for testing, I locally reverted bootstrap service change in this pull request, but left my addition of the After I recreate the topic in kafka (I have autorecreate disabled), the pods continue to complain indefinitely until I manually restart them. Then they behave again. By the way, I'm using kail to follow this, and it's pretty great. |
|
I've been running in a QA cluster for a couple of days with Retry_Limit false, without memory limits. Haven't had time to analyze results, but memory usage is suspiciously high on two pods now (45 and 60 MB while the rest are below 20 MB). I also have lots of logs like: I haven't seen any indication that our other Kafka clients have this much issues. |
|
hmm those messages are generated by librdkafka, now using 0.11.1, I will upgrade to 0.11.3 (https://github.com/edenhill/librdkafka/releases/tag/v0.11.3) |
|
rdkafka updated on image fluent/fluent-bit-kafka-dev:0.5 |
|
@edsiper @solsson I updated the Yolean/kubernetes-kafka:single-node branch and dumped the files from here in this branch of this repo from 7945b66 together into my own StevenACoffman/kubernetes-kafka#2 This lets me run in minikube a single node kafka cluster, with a fluent-bit daemonset for testing purposes. Hopefully helpful to one of you as well! Unfortunately, I'm out for the rest of the year, so I'll not be able to give this more attention until Jan 2, 2018. |
|
It's worth noting about the QA cluster I've been running on that it's multi-zone (Yolean/kubernetes-kafka#41). With log aggregation being a per-node affair, I wonder if kafka clients too can be "rack-aware". If I set acks to |
but beware of the aggregation recursion with the test, you'll see escaped escaped ... escaped json.
but beware of the aggregation recursion with the test, you'll see escaped escaped ... escaped json.
|
#16 replaces this |
Based on fluent/fluent-bit#94, currently the dev build from fluent/fluent-bit#94 (comment)
A typical record (pretty-printed using jq):
{ "kubernetes": { "docker_id": "dd50658b5d1110070dfa24511b7c2e6095bc210b7807617bc9c7e9705c64be28", "container_name": "fluent-bit", "host": "minikube", "annotations": { "kubernetes.io/created-by": "{\\\"kind\\\":\\\"SerializedReference\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"reference\\\":{\\\"kind\\\":\\\"DaemonSet\\\",\\\"namespace\\\":\\\"logging\\\",\\\"name\\\":\\\"fluent-bit\\\",\\\"uid\\\":\\\"7b5e7b6d-c6ef-11e7-ada5-080027039766\\\",\\\"apiVersion\\\":\\\"extensions\\\",\\\"resourceVersion\\\":\\\"1695\\\"}}\\n" }, "labels": { "version": "v1", "pod-template-generation": "1", "kubernetes.io/cluster-service": "true", "k8s-app": "fluent-bit-logging", "controller-revision-hash": "2050036262" }, "pod_id": "af021eb0-c6ef-11e7-ada5-080027039766", "namespace_name": "logging", "pod_name": "fluent-bit-qpz69" }, "time": "2017-11-11T15:00:40.427239619Z", "stream": "stderr", "log": "[2017/11/11 15:00:40] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-1.broker.kafka.svc.cluster.local:9092/bootstrap]: kafka-1.broker.kafka.svc.cluster.local:9092/1: Receive failed: Disconnected\\n", "@timestamp": 1510412440.42724 }Notably, compared to filebeat 6.0.0-rc (Yolean/kubernetes-kafka#88), it contains node name and annotations. Also notably, like filebeat, no key is set.