Make current journal log file distinguishable from rotated journal log files

[lanzafame]

Checklist

• This is not a new feature or an enhancement to the Filecoin protocol. If it is, please open an FIP issue.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not brainstorming ideas. If you have an idea you'd like to discuss, please open a new discussion on the lotus forum and select the category as Ideas.
• I have a specific, actionable, and well motivated improvement to propose.

Lotus component

Other

Improvement Suggestion

I am trying to get Loki to scrape journal logs atm and it is difficult with the current naming of the journal log files as there is no way to distinguish from the current log file and which files have been rotated.

Currently, they all look like the following:

lotus-journal-2021-08-04T112641+1000.ndjson
lotus-journal-2021-08-04T164634+1000.ndjson
lotus-journal-2021-08-05T105711+1000.ndjson
lotus-journal-2021-08-05T111201+1000.ndjson
lotus-journal-2021-08-05T113522+1000.ndjson
lotus-journal-2021-08-07T135550+1000.ndjson
lotus-journal-2021-08-10T140251+1000.ndjson
lotus-journal-2021-08-10T163444+1000.ndjson
lotus-journal-2021-08-10T163858+1000.ndjson <-- current file

Ideally, they would look like the following:

lotus-journal-2021-08-04T112641+1000.ndjson
lotus-journal-2021-08-04T164634+1000.ndjson
lotus-journal-2021-08-05T105711+1000.ndjson
lotus-journal-2021-08-05T111201+1000.ndjson
lotus-journal-2021-08-05T113522+1000.ndjson
lotus-journal-2021-08-07T135550+1000.ndjson
lotus-journal-2021-08-10T140251+1000.ndjson
lotus-journal-2021-08-10T163444+1000.ndjson
lotus-journal.ndjson

This way a simple pattern match can be used to capture only the current log file.

[f8-ptrk]

can you explain how you get that journal?

[lanzafame]

@f8-ptrk you can find a journal directory in both the .lotus and .lotusminer directories.

[f8-ptrk]

thanks. fascinating

the file date doesn't work for loki to find the newest one?

[lanzafame]

Well the file date is incremented based on build date of the binary... or something like that, note the file creation dates and the dates in the log filenames:

10 Aug 14:02 lotus-journal-2021-08-07T135550+1000.ndjson
10 Aug 16:34 lotus-journal-2021-08-10T140251+1000.ndjson
10 Aug 16:38 lotus-journal-2021-08-10T163444+1000.ndjson
14 Aug 12:00 lotus-journal-2021-08-10T163858+1000.ndjson

@f8-ptrk Also you may want to follow: #7040

[lanzafame]

If you are rotating logs, be careful when using a wildcard pattern like *.log, and make sure it doesn’t match the rotated log file. For example, if you move your logs from server.log to server.01-01-1970.log in the same directory every night, a static config with a wildcard search pattern like *.log will pick up that new file and read it, effectively causing the entire days logs to be re-ingested.

src: https://grafana.com/docs/loki/latest/clients/promtail/configuration/#example-static-config

[f8-ptrk]

bigger problem i see is that promtail maybe will block log files from being rotated at all if running on the output file itself

we currently just custom parse them into influx format and wire them over to the db with custom go code. the golog files. but it's pretty high up the list to get the full log complex parsed including systemd journal etc

[lanzafame]

I haven't seen promtail have an issue with rotated files. https://grafana.com/docs/loki/latest/clients/promtail/troubleshooting/#a-tailed-file-is-truncated-while-promtail-is-not-running