input-redis does not know ssl-flag, although it is mentioned in documentation

[0asp0]

• Version: logstash 7.2.0
• Operating System: docker container / centos as host
• Config File (if you have sensitive info, please remove it):

I am using redis behind stunnel to gain TLS encryption for redis. Filebeat is running fine, ships it's data sucessfully to redis database.

Now I want to establish secured connection from logstash to redis.
I updated logstash to version 7.2.0 where the docs are offering a flag for ssl:

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html#plugins-inputs-redis-ssl

Sounds like it is what I need.

So my input looks like this:

input
{
	redis
	{
		data_type => "list"
		db   => "${REDIS_DB}"
		host => "${REDIS_HOST}"
		port => "${REDIS_PORT}"
		ssl  => "${REDIS_SSL}"
		key => "timer"
	}
}

When I check the environment variables on container level, they are looking correctly to me:

[root@poc-logstash-5ddf9b77db-756d6 logstash]# echo $REDIS_SSL
true

Here are the logfiles. Looks like ssl flag is unknown for logstash, but it is documented...

kubectl logs poc-logstash-5ddf9b77db-228n7
2019/07/12 09:01:22 Setting 'xpack.monitoring.elasticsearch.hosts' from environment.
2019/07/12 09:01:22 Setting 'xpack.monitoring.enabled' from environment.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-07-12T09:01:42,207][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2019-07-12T09:01:42,228][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2019-07-12T09:01:42,763][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.2.0"}
[2019-07-12T09:01:42,795][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"d5a7bc34-522e-4afb-bfa9-170640a783ba", :path=>"/usr/share/logstash/data/uuid"}
[2019-07-12T09:01:44,429][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://poc-es-master:9200/]}}
[2019-07-12T09:01:44,705][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://poc-es-master:9200/"}
[2019-07-12T09:01:44,758][INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>7}
[2019-07-12T09:01:44,761][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2019-07-12T09:01:44,893][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2019-07-12T09:01:44,894][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[aschampe@kubernetes01 elasticsearch]$ kubectl logs poc-logstash-5ddf9b77db-228n7
2019/07/12 09:01:22 Setting 'xpack.monitoring.elasticsearch.hosts' from environment.
2019/07/12 09:01:22 Setting 'xpack.monitoring.enabled' from environment.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-07-12T09:01:42,207][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2019-07-12T09:01:42,228][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2019-07-12T09:01:42,763][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.2.0"}
[2019-07-12T09:01:42,795][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"d5a7bc34-522e-4afb-bfa9-170640a783ba", :path=>"/usr/share/logstash/data/uuid"}
[2019-07-12T09:01:44,429][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://poc-es-master:9200/]}}
[2019-07-12T09:01:44,705][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://poc-es-master:9200/"}
[2019-07-12T09:01:44,758][INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>7}
[2019-07-12T09:01:44,761][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2019-07-12T09:01:44,893][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2019-07-12T09:01:44,894][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2019-07-12T09:02:12,925][ERROR][logstash.inputs.redis    ] Unknown setting 'ssl' for redis
[2019-07-12T09:02:12,925][ERROR][logstash.inputs.redis    ] Unknown setting 'ssl' for redis

[lifeofguenter]

this worked in logstash 6.x

[lifeofguenter]

works when I switch back to docker.elastic.co/logstash/logstash:6.8.2 so this is definitely a regression, but it might have to do with the docker team rather than with this plugin itself - as the code looks good and untouched.

Unfortunately the docker-team does not want any issue reporting: https://github.com/elastic/dockerfiles

[thomaspeitz]

Workaround Dockerfile if you only use redis for input.

FROM docker.elastic.co/logstash/logstash-oss:7.3.1 # or different 7.x.x image

RUN logstash-plugin remove logstash-input-redis
RUN logstash-plugin remove logstash-output-redis
RUN logstash-plugin install logstash-input-redis
RUN logstash-plugin list --verbose logstash-input-redis

[chelnak]

Any update on this? Looks like the issue has been around for a while ago and is a pain for those of us that are using something like helm to manage logstash.

[jsvd]

this input has been published recently so it doesn't have a version conflict with the output anymore.

doing bin/logstash-plugin update logstash-input-redis logstash-output-redis will bring the input to a version above 3.2.0 that has ssl support:

/tmp/logstash-7.2.0 % bin/logstash-plugin list --verbose redis
logstash-input-redis (3.1.4)
logstash-output-redis (5.0.0)
/tmp/logstash-7.2.0 % bin/logstash-plugin update logstash-input-redis logstash-output-redis
Updating logstash-input-redis, logstash-output-redis
Updated logstash-input-redis 3.1.4 to 3.5.0

[chelnak]

@jsvd Do you know if this has been included in the latest container release?