Journalctl

[gouthamve]

TODO:

• [ ] Vendoring is broken, tom has custom version of cortex vendored that is not pushed anywhere, I cp -r'ed required deps.
• [ ] Static linking is not possible, cannot make docker image work.

[multun]

@gouthamve @tomwilkie awesome work, thanks !
it merging this still planned? without any equivalent feature, loki won't be usable for journald-enabled infrastructures, which is a shame considering how big of an improvement it seems to be compared to existing log aggregation software

[hikhvar]

@multun I have build a small PoC for shipping journald logs to loki. (https://github.com/hikhvar/journaldtail) However my current problem is, that I get Entry out of order Errors while pushing to loki :-(

[multun]

@hikhvar

loki/pkg/chunkenc/dumb_chunk.go

Line 40 in c0a6644

if len(c.entries) > 0 && c.entries[len(c.entries)-1].Timestamp.After(entry.Timestamp) {

Then your logs must must somehow be out of order ;-) It may be that 1) the logs appears out of order inside journald in the first place 2) logs aren't fetched in order 3) some conversion is needed 4) ?

[hikhvar]

I did a bit more debugging. I did not clean the loki instance between my test runs. Since I start at the top of the journal in each run, I tried to ingest duplicate logs.

[SvenDowideit]

I've been playing with @hikhvar 's journaldtail - and I can get the out of order error basically at will by doing a docker run

sudo journalctl | grep dockerd results in:

Jan 11 16:17:57 TOWER-SL sudo[10986]:   dow184 : TTY=pts/7 ; PWD=/home/dow184/go/src/github.com/hikhvar/journaldtail ; USER=root ; COMMAND=/bin/journalctl dockerd
Jan 11 16:18:53 TOWER-SL dockerd[1881]: time="2019-01-11T16:18:53.863285304+10:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Jan 11 16:18:53 TOWER-SL dockerd[1881]: time="2019-01-11T16:18:53.863328047+10:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Jan 11 16:18:53 TOWER-SL dockerd[1881]: time="2019-01-11T16:18:53+10:00" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/b23a8c72c33d2921c8be98dd8f7ce16a9d17417617f5602c934f55b014aede15/shim.sock" debug=false pid=11371
Jan 11 16:18:54 TOWER-SL dockerd[1881]: time="2019-01-11T16:18:54+10:00" level=info msg="shim reaped" id=b23a8c72c33d2921c8be98dd8f7ce16a9d17417617f5602c934f55b014aede15
Jan 11 16:18:54 TOWER-SL dockerd[1881]: time="2019-01-11T16:18:54.791609894+10:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

loki onthe other hand is missing the Jan 11 16:18:53 TOWER-SL dockerd[1881]: time="2019-01-11T16:18:53.863328047+10:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]" line

I presume this is really a bug in dockerd's logging, given that some times have milliseconds, and some don't :(

upgrading from Docker 18.06 to 18.09 may also have stopped this - possibly because the containerd shim entries aren't logged - I'll circle back to this on Monday