[ECS] [request]: Support FireLens container as non-root user

[PettitWesley]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Which service(s) is this request for?
ECS

Why does FireLens currently require Fluent Bit to run as root?

See here for an explainer: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#why-does-fluent-bit-run-as-root-user-inside-the-container

Why do customers want to run FireLens container as non-root?

Running as non-root is generally considered a security best practice. If there is a container breakout/escape or some un-planned privileges given to a containerized process, being root might allow it to obtain further access. Whereas, if the process is non-root, an additional privilege escalation exploit may be needed.

For code that is vended and vetted by a trusted provider, such as amazon, and/or code that is open source and is widely used, this risk may not be a real concern as the code can be trusted not to attempt any container breakouts/escapes/malicious activity.

Background

In linux systems, unix sockets are a type of socket represented by a file path. Unix sockets require the following permissions:

• Creation: A process needs write & execute permissions in the directory to create a socket.
• Connection: A process needs write permissions on the socket file to connect to it.

See here for key info: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#why-does-fluent-bit-run-as-root-user-inside-the-container

What would it take to run FireLens container as non-root?

The primary problem is that the FireLens container creates a unix socket in the ECS_HOST_DATA_DIR. That directory and sub-directories is owned by root, since the ECS Agent runs as root:

$ ls -l
drwxr-xr-x 2 root root 22 Aug 10 18:24 data

I performed a local test, where I set the permissions for this directory and any sub-directories to be 0777 permissions.

If you do this, then the Fluent Bit FireLens container can run as non-root and create the unix socket file.

The unix socket created is owned by the Fluent Bit process user and group. Then Docker connects to it, to send Fluent Bit the logs.

Since Docker runs as root, in my test, it was able to connect to the unix socket regardless of which user the Fluent Bit process ran as.

If this is a problem then, Fluent Bit supports specifying the socket permissions: fluent/fluent-bit#5511

Thus, to support this, the current summary of required changes would be:

1. ECS Agent needs to create the directory used by Fluent Bit to store its unix socket with wide permissions that allow Fluent Bit to write and execute in it.
2. We need to remove the current validation in the ECS API that prevents customers from setting the user field on the FireLens container.
3. We'd have to think carefully about the threat model of the FireLens sub-directories in the ECS_HOST_DATA_DIR. It might be safe to open up the permissions to all users in the specific FireLens sub-directories but keep the parent path ECS_HOST_DATA_DIR root owned.

[PettitWesley]

One question I realized one should ask is whether there's a way to workaround this by adding linux capabilities.

Recall the explanation here: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#why-does-fluent-bit-run-as-root-user-inside-the-container

I tried a simulated FireLens setup, where it tries to create the unix socket in a root owned directory, and it fails even after I added all of the linux capabilities that seem file related:

$ docker run -it --cap-add CAP_CHOWN --cap-add CAP_DAC_OVERRIDE --cap-add CAP_DAC_READ_SEARCH --cap-add CAP_FOWNER  --user 1000:1000  -v $(pwd)/config:/fluent-bit/etc -v $(pwd)/data:/var/run public.ecr.aws/aws-observability/aws-for-fluent-bit:stable
AWS for Fluent Bit Container Image Version 2.31.11Fluent Bit v1.9.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/09/29 23:24:16] [ info] [fluent bit] version=1.9.10, commit=1f4d09e087, pid=1
[2023/09/29 23:24:16] [ info] [storage] version=1.3.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2023/09/29 23:24:16] [ info] [cmetrics] version=0.3.7
[2023/09/29 23:24:16] [error] [plugins/in_forward/fw.c:59 errno=13] Permission denied
[2023/09/29 23:24:16] [error] [input:forward:forward.0] could not listen on unix:///var/run/fluent.sock
[2023/09/29 23:24:16] [error] Failed initialize input forward.0
[2023/09/29 23:24:16] [error] [lib] backend failed

Those capabilities can be used in known container breakout attacks, so even if adding them worked, this likely still wouldn't satisfy the true goal of non-root, which is to lock down containers. Its interesting that it doesn't work though.

I also tried the above with --privileged and got the same result.