Allow Filebeat to read from pipes

[dooblem]

This can be considered as an enhancement.

When Filebeat is trying reading from a fifo file:
Aug 11 12:31:23 XXXXXXXXXXXXXXX /usr/bin/filebeat[20879]: log.go:103: Stop Harvesting. Unexpected file opening error: Given file is not a regular file

Old issue from Logstash Forwarder, and steps to reproduce, here:
elastic/logstash-forwarder#525

Thanks in advance,
Marc

[ruflin]

It seems like this would require a new input_type?

[dooblem]

Maybe. Basically a pipe can be read like a log file, but no offset should be recorded in the registry I think.
You may forget this issue, as I seem to be the only one to need this. And this is to work around that:
https://discuss.elastic.co/t/can-filebeat-watch-for-new-files-created-in-a-directory/57811/1

[dooblem]

Here: #2246

[apelsina]

I seem to be the only one to need this.

I need this, too.

[hezor]

Yes, I would find this handy too.

[meerkampj]

Could this be used to grab Journald logs ?

Like :
Journald :
ForwardToConsole=yes
TTYPath=/dev/console

Filebeat :
filebeat.prospectors:

• input_type: fifo
  paths:
  • /dev/console

[vacri]

This would solve a problem of mine as well.

[ruflin]

I'm currently finally refactoring the prospector part which will allow additional prospectors. This is good news for this feature request above as it should make it much easier to add (potentially). If someone is interested to contribute it, ping me here in about 1-2 weeks from now and I can point to the code parts which must be touched.

[ledav-net]

I would be interested in such feature too... Had some progress done already for this ?

[ruflin]

Progress has been made on the refactoring and the prospectors / input or now properly separated: https://github.com/elastic/beats/tree/master/filebeat/input On the implementation of the feature itself nothing has happened but the above change should make a contribution much eaiser.

[ledav-net]

Good to know, thanks for your answer :-)

[endlesslight0]

Need this feature as well.

[gaetanquentin]

readding from pipe could be a solution for filebeat not reading compressed files (.gz).

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue doesn't have a Team:<team> label.