Support Redis cluster in input and output plugins

[jordansissel]

(This issue was originally filed by @liuxingyishi at elastic/logstash#2979)

---

When I set an output to a node in redis-cluster(Redis 3.0.0),it can not direct to other nodes in redis-cluster automatically.Port 7000 to 7005 are used by redis cluster.
logstash logstash-simple.conf:
input { stdin { } }
output {
redis {
host => ["localhost:7000"]
data_type => "list"
key => "key_count"}
stdout { codec => rubydebug }
}
logstash logs:exception=>#Redis::CommandError: MOVED 15454 127.0.0.1:7002
I supposed that it may be caused by logstash(redis-client role) was not started in cluster mode.

[ChastinaLi]

👍

[splitice]

Just thought I would share a little bit of an experiement with cluster support.

Heres the hacked together plugin (https://github.com/splitice/logstash-input-redis), do not even think of running this is in production. Its just some early exploration of what would be needed (and some fiddling in ruby as I have never touched the language before). After I get our log producer to support redis-cluster I'll do some more work.

Only the queue/list based method has been tested.

Configure like:

redis_cluster {
host => "127.0.0.1"
data_type => "list"
keys => ["rsyslog_1"]
}

[guyboertje]

OK - All the bazillion plus ones have convinced me. I am also swayed by the fine experiment that @splitice has done with Jedis.

I will build a Jedis JRuby extension that supports pipelined lrange ltrim on clustered Redis.

[mp911de]

@guyboertje @splitice Did you test the redis input also with lettuce? Lettuce does not require connection pooling and pipelines commands by default.

[huseyincotuk]

@guyboertje @splitice Any progress on the development? I can test it as a volunteer right now. Please let me know when you finish.

[splitice]

We are now using a different intermediary queue, my work is complete / idle.

Imho redid cluster tooling still needs some work before it's ready for us.

[guyboertje]

Please do not keep adding +1 type comments. I (we, the Logstash team) are not yet able to code for Redis Clustering at the moment. We are in the process of defining the Java Plugin mechanism and selecting suitable plugins to move to Java - this plugin is one candidate - its just not a high priority one at the moment.

[berglh]

@guyboertje Redis-rb is looking like it's close to adding support for Redis Cluster. I'm not sure if this is directly portable to jruby for use in the logstash Redis plugins; but it sounded promising so I thought I'd add the ref here: redis/redis-rb#716

I've been blowing off the dust, getting this https://github.com/berglh/logstash-filter-redis plugin working for use as a dynamic look-up/write-out table as an alternative to the jdbc and translate filters. I've been testing this plugin against a Redis cluster only to find the Redis client didn't seem to like writes being redirected to other Redis server instances.

I'll keep an eye on that PR and figure out if it's possible to use it for logtash Redis plugins.

[diablo39]

Hello,
Anything changed for last 2 years?

[thisisusernameforgithub]

or in the last 4 years?

[shuaizhang1997]

or in the last 4 years?

[berglh]

🧟