why redis input event value twice

[ivanwa]

When I switched to use redis to be MQ. I fount a problem than event value record twice.

Problems with log

Indexing debug log:

{
"message" => "<6>May 26 17:11:43 test-yum-10-59 kernel: device eth0 entered promiscuous mode",
"@Version" => "1",
"@timestamp" => "2016-05-26T09:11:43.000Z",
"host" => "172.26.10.59",
"port" => 17599,
"type" => "syslog",
"syslog_timestamp" => [
[0] "May 26 17:11:43",
[1] "May 26 17:11:43"
],
"syslog_hostname" => [
[0] "test-yum-10-59",
[1] "test-yum-10-59"
],
"syslog_program" => [
[0] "kernel",
[1] "kernel"
],
"syslog_message" => [
[0] "device eth0 entered promiscuous mode",
[1] "device eth0 entered promiscuous mode"
],
"received_at" => [
[0] "2016-05-26T09:11:43.843Z",
[1] "2016-05-26T09:11:43.000Z"
],
"received_from" => [
[0] "172.26.10.59",
[1] "172.26.10.59"
...

But my shipping log is:
{
"message" => "<6>May 26 17:11:43 test-yum-10-59 kernel: device eth0 entered promiscuous mode",
"@Version" => "1",
"@timestamp" => "2016-05-26T09:11:43.000Z",
"host" => "172.26.10.59",
"port" => 17599,
"type" => "syslog",
"syslog_timestamp" => "May 26 17:11:43",
"syslog_hostname" => "test-yum-10-59",
"syslog_program" => "kernel",
"syslog_message" => "device eth0 entered promiscuous mode",
"received_at" => "2016-05-26T09:11:43.843Z",
"received_from" => "172.26.10.59",
"syslog_severity_code" => 5,
"syslog_facility_code" => 1,
"syslog_facility" => "user-level",
"syslog_severity" => "notice"
}

Then I fount redis subscribe messages it seems to be ok!

172.26.10.74:6379> SUBSCRIBE logstash-chan-2016.05.26
Reading messages... (press Ctrl-C to quit)

1. "subscribe"
2. "logstash-chan-2016.05.26"
3. (integer) 1
4. "message"
5. "logstash-chan-2016.05.26"
6. "{"message":"<6>May 26 17:11:43 test-yum-10-59 kernel: device eth0 entered promiscuous mode","@Version":"1","@timestamp":"2016-05-26T09:11:43.000Z","host":"172.26.10.59","port":17599,"type":"syslog","syslog_timestamp":"May 26 17:11:43","syslog_hostname":"test-yum-10-59","syslog_program":"kernel","syslog_message":"device eth0 entered promiscuous mode","received_at":"2016-05-26T09:11:43.843Z","received_from":"172.26.10.59","syslog_severity_code":5,"syslog_facility_code":1,"syslog_facility":"user-level","syslog_severity":"notice"}"

Is my configuration errors or other problems?

Config

shipping

logstash-syslog.conf
input {
tcp {
port => 5000
type => syslog
}
udp {
port => 5000
type => syslog
}
}

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

output {
redis {
host => "172.26.10.74"
data_type => "channel"
key => "logstash-chan-%{+yyyy.MM.dd}"
}
#elasticsearch { hosts => ["172.26.10.74:9200"] }
stdout { codec => rubydebug }
}

indexing

redis-chan-input.conf
input {
redis {
data_type => "pattern_channel"
key => "logstash-chan-*"
host => "172.26.10.74"
port => "6379"
threads => 5
}
}
output {
elasticsearch { hosts => ["172.26.10.74:9200"] }
stdout { codec => rubydebug }
}

Package version

logstash-2.3.2-1.noarch
elasticsearch-2.3.3-1.noarch
kibana-4.5.1-1.x86_64