[RFC 0143] Nix Store ACLs

[balsoft]

Summary

This RFC proposes a system of Access Control Lists in the Nix Store, allowing multiple users to share software in a single store without exposing it to each other.

Rendered

[edolstra]

You may want to have a look at https://gist.github.com/edolstra/afa5a41d4acbc0d6c8cccfede7fd4792 and its implementation (https://github.com/edolstra/nix/tree/acls). The core of that proposal is to maintain the closure invariant that if a user can read path P, they can read the paths in the closure of P. There is no __permission = [usernames] attribute because that doesn't work on something like NixOS, since the users might not exist yet.

[edolstra]

This RFC is open for shepherd nominations! (It already was but we never asked for nominations explicitly...)

[kevincox]

This RFC has not acquired enough shepherds. This typically shows lack of interest from the community. In order to progress a full shepherd team is required. Consider trying to raise interest by posting in Discourse, talking in Matrix or reaching out to people that you know.

If not enough shepherds can be found in the next month we will close this RFC until we can find enough interested participants. The PR can be reopened at any time if more shepherd nominations are made.

See more info on the Nix RFC process here

[Ericson2314]

I am not sure whether this is a good thing to prioritize at this moment, but I do want to get here eventually. I nominate myself,

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/avoiding-information-leakage-in-multi-host-deployments/29926/3

[thufschmitt]

I'd gladly nominate myself as a shepherd (with a conflict of interest warning as this comes from a client project for Tweag, so there shouldn't be only Tweagers in the shepherd team).

@edolstra, I'd also nominate yourself if you're willing since you're both knowledgeable and interested in that problem space

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-51/30870/1

[kevincox]

@thufschmitt what is the status of this RFC? Is there anything that is needed to move it towards FCP?

[thufschmitt]

@thufschmitt what is the status of this RFC? Is there anything that is needed to move it towards FCP?

Sorry for the long silence. It's ongoing, although for $REASONS it's hard to find a time to gather the team.

[thufschmitt]

Left an army of comments :)

Overall: I like the broad direction quite a lot, but there is a handful of things to iron out and details to clarify

[thufschmitt]

Suggested change

	Change the implementation of the Nix daemon (and, potentially, nix-serve, depending on the chosen "Remote Store" implementation) to store access control lists as metadata, automatically update them when users provide proof that they have the necessary source, and allow `trusted-user`s to manipulate those ACLs manually.
	Change the implementation of the Nix daemon to store access control lists as metadata, automatically update them when users provide proof that they have the necessary source, and allow `trusted-user`s to manipulate those ACLs manually.

I don't think we need to care about nix-serve for this (unless it's already implemented?)

[thufschmitt]

This might not matter too much, but shouldn't anyone with access to the path be allowed to share it further?

[thufschmitt]

(it looks like that's what L53 implies too)

[balsoft]

This might not matter too much, but shouldn't anyone with access to the path be allowed to share it further?

Yes, indeed

[thufschmitt]

It's not clear at which level these options acts: the first paragraph seems to talk about a specific path (with one access-control list), while the second one seems to be global to the store

[thufschmitt]

If I understand it correctly:

• Setting acl=true (as a global store option) means that every path MUST have an attached ACL (which is enforced)
• Setting selective-acl=true means that paths CAN have an attached ACL, and will be world-readable otherwise

Is that it?

[thufschmitt]

Probably irrelevant here, but this is somewhat reminescent of the mechanism for allowing access to paths in pure eval mode

[thufschmitt]

Does that mean that it's possible for someone to register a derivation without having access to its closure? How would that work?

[balsoft]

Yes, that would be theoretically possible; after all, derivations (.drv files) are pretty much just regular files added to the store, and one can add such files without having access to the closure.

[thufschmitt]

This whole section is a bit weird since the LocalStore doesn't have any notion of user (nor trusted-users).

Maybe this could be reorganized as

1. We add the following API to the store interface to manipulate ACLs with the following invariants: …
2. The local store implements it by setting filesystem-level ACLs (without any mention of access control)
3. The remote store adds some access control by enforcing blah, blah, and blah.

[thufschmitt]

Suggested change

	### Nix Daemon/local store implementation
	### Nix Daemon/remote store implementation

?

[thufschmitt]

IIRC attic has a similar problem, and some solution to it. It would be nice to see how they handle it (maybe Cachix has it too)

[thufschmitt]

nothing will get re-built, only the permissions will be updated

Re. my comment about whether __permissions should affect the hash: This means that it doesn't. If you go that way it should be spelled out explicitly

[thufschmitt]

I would be a bit stronger here (or somewhere else), and highlight that this isn't a good mechanism for storing secrets

[balsoft]

I'd say it depends on what is meant by "secrets"; after all, the only reason to have ACLs in the first place is to have some information which is only accessible to a select group, and thus "secret" in some sense. Of course, it should be highlighted that this is intended for "secret" (proprietary or otherwise protected) source-code and build artifacts, rather than runtime secrets such as credentials.

[infinisil]

2023-10-10 Meeting 1

Attendees: @balsoft @edolstra @thufschmitt @infinisil
Notes by: @infinisil

• Eelco: Closure invariant: Need to have access to the whole closure of a path
  • Alex: Sometimes doesn't make sense, you might want to have a library be proprietary and link it to, but have the result be public.
  • Theophane: Generally inputs may be private, output may be public
• Eelco: User need to exist before they can be given access
  • Alex: Indeed, users need to exist at eval time
• Silvan: How about just agreeing on design constraints in the RFC, not go into a specific implementation
  • Alex: Probably better that way
• Theophane:
  • Closure invariant, very fundamental to Nix, probably don't want to break that
  • How to specify permissions, currently a derivation builtin attribute
• Eelco: Should take a closer look at my design
  • Privacy is not a property of a derivation, but rather how the user builds it with the CLI
  • Alex: Mostly the case as well. It's convenient to specify which users get access to paths in a company. The __permissions is like syntactic sugar for a similar CLI interface.
• Alex: Regarding runtime closure requirement, doesn't feel secure, might leak details you don't want to have exposed
  • Theophane: Should err on the side of being cautious, deny access by default.
  • Reminds of requiredReferences/allowedReferences, something similar could work
  • Alex: Have a constraint when setting acls on a store path, the entities must also be able to access the runtime closure. Set permissions recursively.
• Alex: Another thought: If a user has access to the build-time closure, should they get access to the resulting path as well?
  • Theophane: E.g. remote builders have extra sandbox paths that could provide extra secret stuff
  • Alex: Non-reproducible derivations don't work since the hash doesn't work.
  • Theophane: Poison the closure to indicate impurities in the build graph. Having to rebuild to give more people access is expensive
  • Alex: How would you poison the closure? Not sure it could work
  • Theophane: We have a somewhat weak security model
  • Alex: Impure derivations probably make this work
  • Eelco: You need to prove that you can instantiate the derivation, then you get added to the ACLs. If you have access to a .drv you get access to the output paths.
    • Alex: Something in the build inputs might be protected.
    • Eelco: Combining the drv closure invariant and drv giving you access to the outputs, then you get access the entire graph
    • Alex: What about non-derivation paths?
    • Eelco: Also would be in the drv inputs, included in drv closure
    • Alex: Need the hash to instantiate the derivation
    • Theophane: To make it private you need to prove you have the content
    • Eelco: If you try to create a drv but some references aren't readable it would be rejected. Need to provide proof to the daemon that you have access to them.
    • Alex: Yeah that would also work. One of these might be more efficient
• Eelco: Fear with ACLs: Requires auditing of Nix to ensure no leaks. E.g. daemon operations that return a NAR for a store path
  • Theophane: Only a fixed number of daemon protocol operations, the boundary would be there
  • Alex: Not many places where you read from the store.
  • Daemon vs store boundary, maybe both
• Eelco: How does the RFC handle building, temporary build user access?
  • Alex: check if the user has access to the inputs; grant access to the build user, revoke access afterwards
  • Eelco: Risk of killing the build not revoking access. Write a sort of recovery log that shows what access was granted so it can be cleaned up
  • Alex: Maybe better, use extended file system acls to mark the build user that has access
  • Eelco: Doesn't work, would require scanning the store
  • Eelco: Not a problem with the sandbox
  • Theophane: Could require the sandbox for private paths
  • Silvan: Safe default to require it
  • Eelco: See https://github.com/edolstra/nix/blob/e00259c7c83abc62c9221b065a636edac2e55beb/src/libstore/build/local-derivation-goal.cc#L665-L687 and https://github.com/edolstra/nix/blob/e00259c7c83abc62c9221b065a636edac2e55beb/src/libstore/build/local-derivation-goal.cc#L993C27-L1006 for how to handle this
• Alex: Consider dropping ACLs in the build, would be an impurity
  • Might require a fuse layer, expensive
  • Silvan: Probably not a big deal to have this impurity
• Eelco: What are the use cases?
  • E.g. teams not being able to access each others closures
  • E.g. NixOS use case: Files with passwords in them
    • E.g. user would build it as a private closure, at boot time the activation script would make paths accessible somehow, haven't gotten to implementing that
    • Alex: Might not be a good idea because everything is hashed
    • Silvan: E.g. imagine having a pin, could be brute-forced checking against the hash
    • Eelco: Could have a salt
    • Eelco: Theoretical concern, would require a lot of brute forcing to exploit
    • Theophane: People could put password as a standalone file in the store
    • Alex: Should be able to store small secrets as well
    • Silvan: Trivial example of storing a single bit in the store, could be easily brute-forced
  • Eelco: Need to have good use cases to justify the complexity
• Theophane: Should also go through the encryption RFC
  • Deterministic encryption
• Alex: Current loc: 2.5k lines total, 300 lines test, maybe 200 docs
• Silvan: We can all agree that we want something like ACLs (only need to make sure it's secure, worth it, doable, etc.)
• Alex: Will look at Eelco's previous design

[infinisil]

2023-11-07 Meeting 2

Attendees: @infinisil @edolstra @Ericson2314 @thufschmitt @balsoft
Notes by: @infinisil

Notes

• PR open: Add granular access control for nix store nix#9287
  • Needs more tests
• Updated RFC after the last meeting
• Looked at Eelco's previous ACL proposal
  • Remove some bits from the RFC so it could be closer to that
• Don't have the variant "If you have access to a path, have access to the closure"
  • Should list arguments for and against that in the RFC
  • Eelco: Very critical imo
  • Alex: Use case: Give access to the output but not to the input, e.g. not giving access to the source code
  • Alex: What if it's only the default but can be changed?
    • Eelco: Doesn't sound reasonable, kind of like a dangling reference
    • Alex: Good point, will rethink
• Add use cases in the RFC
  • Include non-supported use cases, e.g. like storing small use cases
• No need to pin down details in the RFC
  • Best remove everything that's not necessary
  • RFC should specify the expected semantics and use cases
• Eelco: Would like secrets to be supported
  • Alex: Does work, but can be attempted to brute-force using the store hash
    • E.g. storing a secret 4-digit pin in the store
  • Lots of discussion in the previous RFC 005
    • Use salt, but where does it come from?
  • Make the store not readable by everybody?
    • Mentioned in the RFC
  • Only a problem with fixed-output/CA derivations?
    • Such secrets would often be added from local paths -> fixed-output/CA
  • Attack vector of being able to test whether specific secrets are in the store
  • Alternatively just make it clear that people shouldn't use brute-forceable secrets
• John: Consider fuse filesystem whose interface really hides things, provide separate view of the store to every user
  • Could make adding features much easier and more flexible
  • Comparatively, using unix features to implement all of these features is very tricky
  • Eelco: Ideally ACL-based and fuse-based store interface would be the same
  • John: Soon in linux: fuse passthru, makes things a lot faster
  • Eelco: Not sure if fuse is easier or harder, requires a daemon to be running
    • Might be hard for NixOS, since it would need the daemon running perhaps already in the initrd
• John: ACLs as a security tool seems kind of wrong
  • Don't want to write down a username in Nix code
  • Eelco: How does that work if users don't exist yet?
    • Alex: Fails, mainly for convenience when on the local system
  • Doesn't really make sense on NixOS where users are only created during activation
    • Alex: Would give access to the user with the Nix command in the activation script, no need to specify the user in Nix code
  • Alternatively could support that in some post build hook
    • Alex: Doesn't work, it's only metadata and doesn't end up in the store, wouldn't want to rebuild to change it
  • John: In capability systems, it's not an intrinsic property who has access to what, you always need a reference to give you access. In the very end of e.g. a NixOS system you have some export list about access
    • Map from user to path rather than path to user?
  • Is it really a good use case to explicitly give somebody else access to a store path, instead of everybody just building and getting access themselves?
    • Useful in teams
• What if two users have the same expression but with different access control? Would access be revoked when the other builds it?
  • Alex: Yes, makes sense to me
    • Silvan: Please put it as an example into the RFC
  • John: Revoking permissions is hard to do modularity
    • Initially had a grant tree but didn't end up working out well
  • Maybe don't support declarative revoking of permissions
• John: Coming back to capability systems, what if we didn't have to worry about using ACLs underneath
  • Maybe ACLs is a mismatch with Nix
  • Alex: I don't think it's a mismatch, just an implementation detail
  • How would a capability system look like?
  • John: Might be able to write down some thoughts on that
• John: Can the rootless daemon be finished soon? Having that and the no o-r on the /nix/store (RFC 97) would help a lot
  • Alex: These seem orthogonal
  • John: Implementation-wise orthogonal, but conceptually very related, first step to not think of it as an open world

Action items

• Alex: Write down use cases
• Alex: Add runtime closure invariant and implement it
• Alex: Write advantages/disadvantages about __permissions

[YorikSar]

I see FUSE mentioned in the discussion here. I'd like to note that on macOS FUSE is hard to set up (you have to mangle with System Integrity Protection stuff), so it would not work for people who just need Nix. ACLs on the other hand are supported pretty much everywhere.

[Ericson2314]

I would rather not support macOS than support I feature that I feel is in the wrong direction. macOS Nix talking to Linux Nix still can get benefits.

---

(Adding more detail/polish to a few things I said in the meeting)

I am a proponent of Capability-based security as generally better, and in particular "the Nix way" giving the shared overlaps with functional programming, etc. While it is possible have some capability system and then set and unset file ACLs based on that, the result amplifies reads into writes (O(1) share capability becomes O(files in closure) setfacl calls), which is bad for performance, and bad for introducing more state. I am thus very skeptical that good things can come from file ACLs.

The order I am thinking of is:

1. Rootless daemon, and using rootless daemon with NixOS. Results:

   • The bulk of Nix is out of the trusted compute base. This is sort of table stakes about thinking about security at all.
   • Store objects that NixOS needs should be root owned. Either that means different children of /nix/store have different owners/groups --- making Nix cope with that is IMO "MVP ACL store", or another solution such as "baking" separate stores for NixOS.

2. RFC-97. Exactly what this looks like is different based on the paths taken above (e.g. is there one store or multiple). Results:

• No more unprivileged figuring out what all the store paths in the store are
• Potential problem: store paths granting permissions may not be good enough if store paths are too easy to forge (e.g. because of various side channels)

3. Non-forgable capabilities: Guarding access to the store with stronger capabilities than store paths. Examples:

• Special daemon will hand out directory or mount namespace file descriptors with just the closure of root path(s) available.
• Systemd services sandboxed in various ways (dynamic UIDs, Linux namespeaces, etc.) only get access to their closures, regardless of what store paths will be guessed.

A basic theme here is that with the exception of the rootless daemon, which is almost finished, I think most of the work doesn't need to happen with Nix itself. I think its is more of an integration-y / dev-ops exercise in making various bits of software continue working as we take away more and more permissions. And dev work probably goes into separate little high-privilege daemons (like the GC roots scanner that comes with the rootless daemon), not the "main bulk" of Nix.

[thufschmitt]

@Ericson2314 this is giving up on an existing batteries-included cross-platform implementation in favour of a hypothetical build-your-own-cake Linux-only one. As much as I like your final vision, I don't see it happening any time soon (if ever).

[Ericson2314]

As always, we want to make sure Nix is going in the right direction. Some experimental features are alright, but also nicely self contained. NixOS/nix#9287 looks pretty deeply involved some some core interfaces, making a lot more code away of Unix users for example. For me, the more deeply entwined something is, the higher the bar is that it be for sure in the "right" direction.

[roberth]

I'm sorry to have to say this, but I feel strongly that the Nix team does not have the bandwidth to work on this concept. It vastly complicates the Nix model, which we simply can't afford, considering the state of the other unfinished projects.

For what it's worth, the existence of a proof of concept PR doesn't change anything about that, because usually the responsibility to actually complete the project falls on the Nix team, which has to respond to issues, fix bugs and generally put in a lot of effort to make sure invariants are upheld. In this case those a security properties. Nothing to sneeze at.

I very much appreciate @Ericson2314's suggestion to explore what can be done by simplifying and decoupling what we already have, which is actually feasible now, because it does not require any changes to the model.

@thufschmitt No such implementation exists, because we can not support it.

I urge the Nix team to focus on finalizing existing projects first and postpone this RFC until we have time to consider a significant feature like this

[infinisil]

2023-11-21 Meeting 3

Attendees: @infinisil, @thufschmitt, @edolstra, @balsoft
Notes by: @infinisil

• @balsoft: Update: Added closure invariant to the RFC
• Regarding @Ericson2314's ideas in [RFC 0143] Nix Store ACLs #143 (comment)
  • @balsoft: Should be a separate RFC
• Regarding @roberth's concerns in [RFC 0143] Nix Store ACLs #143 (comment)
  • Can it be separated more from core Nix? Like a separate store implementation
  • @balsoft: Partially done already, but could be extended
  • Eventually upstream into local store
  • Problem with local derivation goal, no abstraction layer currently, needed to set ACLs on the fly
  • @infinisil: Would this be in-tree or a plugin?
    • Just a single separate file, but in-tree
  • @balsoft: Can we merge the RFC even if we aren't implementing it yet?
    • @thufschmitt: It's going to be a maintenance burden, RFC's shouldn't promise what can't currently be done
    • @balsoft: But it's gonna be unblocked in the future
    • @infinisil: RFC acts as a blocker against the Nix team having to care about it
  • @thufschmitt: Change the RFC to enable Nix to have out-of-tree store implementations?
    • @balsoft: Is that going to help with with @roberth's concerns? Invariants are the tricky bit, that would have to be in-tree
    • @thufschmitt: Store interface is reasonably stable, might work
  • @balsoft: Can try implementing ACLs as a subclass of local store. Trying to move as much functionality outside of local derivation goal
    • @thufschmitt: LocalDerivationGoal changes needed?
    • @balsoft: Yes for getting access to paths
    • @thufschmitt: What if paths are accessible by default to all build users?
    • @balsoft: Sounds problematic, builders change, requires changes on the store
    • @balsoft: Use case of enabling ACLs, requires going through the entire store
      • @thufschmitt: Not a problem, only needed for ACLs paths, store is world-readable by default
    • @balsoft: Have to figure out which paths have ACLs, require expensive traversing
    • @balsoft: Feels hacky generally, how to detect change in build users?
      • @thufschmitt: Build group might work
      • @balsoft: What if build user gets removed?
      • @edolstra: What about auto uid allocation
      • Would not work, could be a follow-up
    • @infinisil: Sounds like more experimentation needed to see if it works with little maintenance burden
  • @edolstra: I thinks it's unavoidable to touch the core of Nix
    • @thufschmitt: Question is how much it can be avoided in a first version
    • @balsoft: Looks very difficult, would probably have to drop the primop
    • @thufschmitt: Best way to succeed: Shrink it to its minimum, add things on top later
    • @infinisil: Still requires the same maintenance amount in the end
    • @thufschmitt: But more manageable with smaller PRs
    • @infinisil: Should always have small PRs irrespective of the RFC
    • @balsoft: Reasonable small commits already, can be split up
    • @infinisil: Not worth splitting up if it's not gonna get merged. Should talk to Nix team about whether to accept this or not.
    • @infinisil: Already half the Nix team generally against
    • @thufschmitt: Room for negotiation if it's smaller and more split up

[infinisil]

2023-12-19 Meeting 4

Attendees: @infinisil, @thufschmitt, @edolstra, @balsoft
Notes by: @infinisil

• @edolstra: Seems hard to do without deeply involving the store
  • @balsoft: Either extending plugins to have a local-store subclass, changing a couple things and adding functions, though wouldn't be composable
  • Or integrate into Nix, but keep diff very small by creating separate files for everything
• @edolstra: Users need to be passed around in the daemon?
  • @balsoft: There could be a way to hook plugin data into the worker protocol
• @edolstra: Mark it as experimental to future-proof?
  • @infinisil: Probably good to break it regularly, otherwise users rely on it
  • @edolstra: Doesn't mean people will use it, encourage more testing, set expectations
  • @infinisil: Trial period of e.g. 1 year. If success -> stable, otherwise remove again.
  • @thufschmitt: What if people don't try it out?
  • @infinisil: Have a warning after half a year that it will get removed again unless people give positive feedback in a linked issue
  • @balsoft: What is the criteria?
  • @infinisil: Has to be decided on a case-by-case basis
• @thufschmitt: Would the new plugin infra for worker protocol be useful for non-ACL use cases?
  • @edolstra: How would that worker protocol plugin look like?
  • @balsoft: Register plugin as a worker protocol plugin, allow operations that pass data between client/daemon based on plugin name
  • @thufschmitt: JSON?
  • @balsoft: Any serialisation works
• @infinisil: Nix team has little resources but the RFC is generally a good idea, go towards accepting it?
  • @thufschmitt/@balsoft: Important to make it as easy as possible on the Nix team
  • @balsoft: Maybe RFC could be simplified to just "We want ACLs and they should look something like this", probably not controversial at all
  • @edolstra: Semantics is important, more like my original proposal, focusing more on use cases
  • @edolstra: Prototype could be okay if minimal, plugins seem overcomplicated for this
  • Make it as minimal as possible? But how minimal can it be made?
  • @balsoft: Could drop some parts of the RFC
  • @balsoft: Drop CLI commands? Probably doesn't work
    • @edolstra: CLI command can be in a separate file, marked as experimental
  • @thufschmitt: Daemon changes could be dropped by having ACL stuff be run as root
• @infinisil: Sounds fine to merge if we properly figure out how to merge it as experimental
  • @thufschmitt: John and Robert made good points against merging it. Would be hard to undo.
  • @infinisil: Combine with the idea of making the diff minimal
  • @thufschmitt: Not easy
• @thufschmitt: In favor of merging it as minimally as possible, expand upon that, discussing on the changes as we go
  • @balsoft: Need to prevent an eternal experimental feature situation
  • @infinisil: Combine with an experimental feature timeline
    • @thufschmitt: Not sure if it needs to be in the RFC
    • @infinisil: This RFC could lay the groundwork
    • @thufschmitt: +1
  • @thufschmitt: Doesn't solve the problem of limited maintainer time, but might be more okay very incrementally
  • @infinisil: Justify maintainer cost by generalising the work to as much of the codebase as possible, e.g. a general implementation of experimental feature handling
  • General agreement on this
  • Should ask Robert on that, but after the holidays

[balsoft]

Drafted due to no progress being possible at the moment, particularly in light of @roberth 's comment and steering committee discussions.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/secrets-in-nix-suck-and-how-to-fix-them/43822/5