Handling large, sensitive files of arbitrary data in Pillar

[jaloren]

When entire files are sensitive, you can store the entire file contents in a pillar variable, and then use file.managed's contents_pillar variable to pull the entire file contents from a pillar variable. And once feature #3790 is implemented, this will be quite user friendly to use.

However, what's not clear is if this pillar implementation can handle large files and binary data.

Here's the use case.

I want to set up saltstack as a cloud controller replacement for openstack. This means serving up VM images from the saltstack file server. However, these images could contain sensitive data that only a specific set of minions (comptue nodes/hypervisors) are entitled to. Given the current functionality, this means I would need to literally load an entire VM image into a pillar variable. Even the most stripped down VMs are usually 2 GB and some will be much larger.

Here are my questions:

1. Would you expect pillar to be able to handle this in a reasonably performant manner?
2. The VM image is binary data. I suspect that pillar is expecting simple data types like strings and integers. Would pillar be able consume/handle that data without corrupting it or accidentally modifying it?

[basepi]

This would be cool to have, though I'm not sure it will be added anytime soon. Pillar should be able to handle any sensitive data needs you may have.

[pille]

duplicate of #636, which was closed wrt pillar

[basepi]

Yep, good find, @pille. And the points made on that thread are valid -- there's nothing that's desired here that isn't already satisfied by pillar, and if we were to try to make fileserver access control, it would cause a lot of unnecessary overhead (considering we already have this with pillar).

I'm going to close this one, as "solved via pillar", just like #636.

[jaloren]

Can someone explain to me how pillar comes close to doing any of the items i listed above let alone all of them?

Is pillar a file server itself? If so i really haven't seen that documented anywhere. If it is a file server, how do I control access to those files? By "publishing" the path to targeted minions I would guess? Based on my use of pillar and all the documentation I have read, the only thing I can do with pillar is store arbitrary static data in a python dictionary which is then pushed out to specific minions. This to put it mildly is not even close to granting minions access to files which they can remotely copy down. I have not seen any documentation that explains how to "publish a file url which only entitles a specific set of minions to access it".

If pillar can do that, then great! This means there's a huge gaping hole in the documentation OR there's a horrible problem with locating this documentation.

[basepi]

Sorry for the abrupt close on this one, @jaloren. You didn't really specify any use cases, just what the ACL would look like.

There are two common patterns right now. The first is to store just the pieces of files that are sensitive in pillar, and then you create templates in the main fileserver which are filled with these values when they're sent down via file.managed (which has full templating abilities).

When entire files are sensitive, you can store the entire file contents in a pillar variable, and then use file.managed's contents_pillar variable to pull the entire file contents from a pillar variable.

Now, I will agree that this is not as convenient as being able to apply an ACL on the actual fileserver itself. But it keeps the fileserver backends much more simple and efficient, something that is important for a system which is accessed so often.

I'm open to more discussion, just saw that this had been brought up before and it had been decided pillar met the requirements, so I closed it.

[jaloren]

basepi...no worries I am just trying to understand how people are supposed to do ACLs with files in salt.

It sounds like this was what i was looking for:

"When entire files are sensitive, you can store the entire file contents in a pillar variable, and then use file.managed's contents_pillar variable to pull the entire file contents from a pillar variable."

So i get that but this seems awfully ugly for my use case. I want to set up saltstack as a cloud controller replacement for openstack. This means serving up VM images from the saltstack file server. However, these images could contain sensitive data that only a specific set of minions (comptue nodes/hypervisors) are entitled to. Given the current functionality, this means I would need to literally load an entire VM image into a pillar variable. Even the most stripped down VMs are usually 2 GB and some will be much larger.

Would you expect pillar to be able to handle this in a reasonably performant manner and not accidently corrupt or mangle the binary data?

[basepi]

Hrm, large, but sensitive, files. You make a good point. I'll have to think on this use case. I'm going to re-open for the moment until I get a chance to talk with other engineers and see what they think of this use case.

[jaloren]

I'd also like to clarify that I understand the concern about the increased overhead that ACLs would poise on the salt file server implementation. It could become a big bottleneck with a lot of files and/or a lot of minions, which since its salt is actually going to happen.

Consequently, my original proposal could be to costly in terms of performance. I actually liked tf198 comment in #1543 for a private file server environment but it looks like that didn't go anywhere.

In any case, i guess i didn't write this issue up as clearly as i wished I had. The ticket should be "how do i securely share sensitive files only with minions that are entitled to the data".

[jaloren]

Basepi: I hope this was kosher and not confusing. i updated the git issue to reflect what the actual problem is.

[cachedout]

This definitely sounds like a valid use-case to me, though I recommend that
we disable ACLs by default to avoid the performance penalty that might
otherwise be incurred.

On Fri, Jun 20, 2014 at 8:50 AM, jaloren [email protected] wrote:

I'd also like to clarify that I understand the concern about the increased
overhead that ACLs would poise on the salt file server implementation. It
could become a big bottleneck with a lot of files and/or a lot of minions,
which since its salt is actually going to happen.

Consequently, my original proposal could be to costly in terms of
performance. I actually liked tf198 comment in #1543
#1543 for a private file server
environment but it looks like that didn't go anywhere.

In any case, i guess i didn't write this issue up as clearly as i wished I
had. The ticket should be "how do i securely share sensitive files only
with minions that are entitled to the data".

—
Reply to this email directly or view it on GitHub
#13510 (comment).

[pille]

@jaloren: what about encrypting your VM images and distribute the key via pillar?

[jaloren]

@pille That's a possibility. Though it brings to mind the following question.

In regards to exposing sensitive files on the file server, Is there a way to offload key management and cryptographic operations to salt?

Given current functionality, this is the best that I can come with but it seems kinda clunky.

1. write a custom salt runner that takes as an input a directory on the master, gets the files from the directory, encrypts them with keys stored in pillar, and then copies those files onto the salt file server. The runner would need to maintain a list of currently encrypted files and the ability to reverse the encryption.
2. write a custom state module that 1) identifies what encrypted files a minion is entitled to access 2) gets the files from the file server 3) gets the key from pillar 4) decrypts the file

[pille]

@jaloren: in the short term you could add an additional step for the delivery via fileserver.
just deploy the encrypted file as you would by applying a state. then add another one that decrypts the file, which requires the first one. the drawback is, that you need twice the disk space.

in the long term this could become a feature of the fileserver itself. e.g. when a key attribute is specified, the file is decrypted and updated directly with out an intermediate copy. salt may cache checksums of the decrypted file for detecting file-changes.

[jaloren]

@pille okay that sounds good to me.

[terminalmage]

So, for storing the binary data, what about base64-encoding it to store it in pillar, and then adding an option to go alongside contents_pillar (maybe contents_pillar_base64?) to denote that the pillar contents should be base64 decoded. The pillar entry would end up looking something like this:

my_binary_file: |
    f0VMRgIBAQAAAAAAAAAAAAQAPgABAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOACW
    AAAAAAAAAAQAAAAAAAAAECEAAAAAAAAAAAAAAAAAAAAAAAAAAAAArB8AAAAAAAAAAAAAAAAAAAAA
    AAAAAAAAAQAAAAUAAAAAUAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAABADQAAAAAAABAA
    AAAAAAABAAAABAAAAABQAAAAAAAAAEBtAAAAAAAAAAAAAAAAAAAQAAAAAAAAABAAAAAAAAAAEAAA
    AAAAAAEAAAAGAAAAAGAAAAAAAAAAUG0AAAAAAAAAAAAAAAAAAFAAAAAAAAAAUAAAAAAAAAAQAAAA
    .....
    .....
    zMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzM
    zMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzMzM
    zMzMzMzMzMzMzMzMzMzMzMzM

The only obstacle that I can see is the fact that base64 strings contain newlines, and YAML multiline strings must be indented as shown above. This is easily accomplished, however, with a little VIM-fu, or with sed:

base64 mybinaryfile | sed 's/^/    /' >indented.txt

We would also probably want to make contents_pillar_base64 a string so we could differentiate between base64-encoded ascii and binary data, because as soon as we add a feature like this someone is going to want to deploy base64-encoded ascii data.

[basepi]

I like the idea, but do not like the idea of manually encoding it and putting it in a pillar file, I feel like that's asking for trouble. Instead, why not define a new external pillar that's just designed to pull files into pillar as base64 encoded strings? The contents_pillar_base64 still makes sense.

[terminalmage]

Ooh, I like the external pillar idea

[cachedout]

Another +1 for the external pillar idea.

[leonardinius]

Hi,

I accidentally stumbled across this issue and wondered - how could someone get the same hierarchy traversal but rather than getting file contents - would get ability to override minion pillars.

E.g. we store our pillar states in Git / clone them over to master. But, I would love to have easy way to override database credentials pillar for minion_id in production. Maybe it's something what is already possible to get in Salt which I'm not aware of.

Should I ask the question on salt-users mail list instead?

[cachedout]

Hi @leonardinius. I recommend you toss that one over to salt-users and see what ideas come back. Much more likely that more eyes will see it than just here in this issue. Thanks!

[whiteinge]

Not appropriate for large files, but an initial base64 implementation is in #19935.

[Diaoul]

Looking forward for 2015.2 with this feature.

[igorwidlinski]

+1

My company could use this for:

1. storing self and externally signed SSL certificates and their keys
2. kerberos keytabs
3. i am sure i can come up with other things..

Especially #2 as us login into servers is quite important. It is simply a show stopper for now.

[arnisoph]

Anyone considered https://github.com/saltstack/salt/blob/develop/salt/pillar/file_tree.py so far?

[baniobloom]

This issue appears to be closed. Is this in the current version 2015.05? I have a current need for this feature.

[basepi]

The file_tree external pillar is available. There's also ongoing discussion in #9569 about other solutions.

[windoverwater]

I know this issue is closed, but I am interested in the original use case (as I understand it - though maybe I have rose colored glassed at the moment). Basically, I too would like to distribute large binary secure files selectively to select sets of minions. And having #13510 (pillar file_tree.py) is great, but I do not know the efficacy of that solution compared to what I tried earlier this year and sadly did not achieve.

This is what I tried (and this was after a few email discussions said to try it):

Create a set of salt environments along the lines of security_level_1 thru security_level_4 with the more secure 'salt environments' associated with the more secure minions. For example, servers may get the most restricted environment (security_level_4) while desktops get the least restricted (base - everything can read from the base environment). The various environment inherited the lower security environments.

With this setup server class minions were associated with the security_level_4 environment (via the top.sls file) while desktop minions where only associated with the base environment (top.sls).

Everything was going well security model wise until it was discovered that any minion could set 'saltenv=security_level_4' and grab the secure files out of the salt tree.

Sigh.

If only the minions were NOT allowed to reference any environment and instead were limited to the environments specified in the master's config file. If that restriction was enabled, then users with use cases like the above could decide to use salt environments to implement various security models and leverage the performance and storage of the salt file system to manage large secure files.

Thoughts/comments?

Thanks!

[whiteinge]

@windoverwater the only way to do that securely using file_roots is using separate syndics. So, a tad heavy. You're very right the "large" requirement in this issue as originally written was mostly overlooked in this discussion. It is worth filing another issue IMO to track a "Pillar-like" system that can distribute large binaries. I think such an addition will necessarily be a major one rather than a bolt-on to existing functionality like the base64-encoding workaround.

[windoverwater]

Nice suggestion - never read up on syndic details until just now! Thanks! Yes it looks like that could work, and yes it would be a, uh, a tad heavy. But that is the idea - file_roots access with effective RBAC.

But the good news is that for now given the details we should be able to leverage the pillar solution and solve the RBAC problem that way.