docs: Write documentation on collecting systemd logs

[andrzej-stencel]

We need high-level documentation for collecting systemd logs, akin to https://github.com/SumoLogic/sumologic-kubernetes-collection/blob/v2.11.0/deploy/docs/collecting-kubernetes-events.md for collecting Kubernetes events.

It should describe the frequent use cases, like:

• disabling collection for specific units, only enabling it for specific units, etc.
• customizing the collected data - decorating with metadata, filtering, etc.

This might (and hopefully should) result in changes to configuration of the collection, so that the common use cases do not force the user to mess with the raw OTC configuration, or even prevent that configuration from being exposed in the values.yaml file.

[ballensans]

The ability to gather kernel messages would be fantastic too - they have no "unit" so they kinda fall through the gaps there. In particular, gathering the logs generated by IPTables "LOG" rules and being able to enrich the logs the log prefix and level would be amazing!

Use case - Calico, and I suspect other k8s network policy engines, create underlying IPTables rules to implement the network policy.

Example command to create IPTables LOG rule:

iptables -I OUTPUT -m comment --comment "log all outbound" -j LOG --log-prefix "packet-outbound: " --log-level 5

Results of reading with journalctl:

journalctl --follow --priority 5 | head -5
-- Logs begin at Fri 2023-03-10 19:25:09 EST. --
Mar 16 12:35:35 node01 kernel: packet-outbound: IN= OUT=eth0 SRC=10.38.91.220 DST=10.38.90.210 LEN=88 TOS=0x10 PREC=0x00 TTL=64 ID=19173 DF PROTO=TCP SPT=22 DPT=53153 WINDOW=547 RES=0x00 ACK PSH URGP=0
Mar 16 12:35:35 node01 kernel: packet-outbound: IN= OUT=eth0 SRC=10.38.91.220 DST=10.38.90.210 LEN=88 TOS=0x10 PREC=0x00 TTL=64 ID=19174 DF PROTO=TCP SPT=22 DPT=53153 WINDOW=547 RES=0x00 ACK PSH URGP=0
Mar 16 12:35:38 node01 kernel: packet-outbound: IN= OUT=eth0 SRC=10.38.91.220 DST=10.38.90.210 LEN=88 TOS=0x10 PREC=0x00 TTL=64 ID=19175 DF PROTO=TCP SPT=22 DPT=53153 WINDOW=547 RES=0x00 ACK PSH URGP=0
Mar 16 12:35:38 node01 kernel: packet-outbound: IN= OUT=eth0 SRC=10.38.91.220 DST=10.38.90.210 LEN=88 TOS=0x10 PREC=0x00 TTL=64 ID=19176 DF PROTO=TCP SPT=22 DPT=53153 WINDOW=547 RES=0x00 ACK PSH URGP=0

Selecting / enriching the logs baed on the prefix (packet-outbound: above) would be great.

[andrzej-stencel]

This is done, docs are here: https://help.sumologic.com/docs/send-data/kubernetes/collecting-logs/#systemd-logs