[RFC 0078] System-agnostic configuration file generators #78
Conversation
7c6f434c
changed the title
|
This is based on brainstorming during NixCon mainly with @svanderburg . Sander, do you accept to co-sign this as a co-author? (Also thanks to aanderse jtojnar edolstra 0x4A6F LnL7 for discussion) |
edolstra
changed the title
|
Anyone who wants to shepherd this RFC? |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/why-isnt-more-of-home-manager-merged-into-nixpkgs/6096/29 |
|
I like this RFC, thank you! |
|
@7c6f434c sure! RFC looks great so far. I'm going to read it in detail. |
| There is currently a lot of code duplication between | ||
| * NixOS | ||
| * home-manager | ||
| * nix-darwin |
There was a problem hiding this comment.
Another target that would benefit from this are containers that need configuration independent from NixOS modules since they don't support systemd etc.
There was a problem hiding this comment.
True; I am not sure what publically available codebase should be referenced as an example. My real use case is not even in the list and it is closer to what you say about containers; on the other hand, highly-visible existing parallel efforts are more convenient to discuss.
There was a problem hiding this comment.
Option generation is also useful for override, such as pkgs.rofi.override = { theme = <file>; }
There was a problem hiding this comment.
True. But the packages are shared just fine already anyway, so not a strong motivation. Wrapper packages can already be done easily enough. If implementation turns out to be a huge success we might discuss this on a case by case basis whether to move stuff to the configuration file generator library, but let's get there first.
| port = 1234; | ||
| nodeName = "localhost"; | ||
| content = "Hello"; |
There was a problem hiding this comment.
Yes, the type is provided next to the generator function.
| Output: | ||
| ``` | ||
| { | ||
| "subdir/listen.conf" = { |
There was a problem hiding this comment.
An example how this would be integrated into NixOS modules might be helpful.
There was a problem hiding this comment.
Hmm, I am not sure it can be done easily without integrating an entire example from #42 and then extending it…
| listenPort = "1234"; | ||
| serverGreet = "I am localhost"; | ||
| }; | ||
| type = …; |
There was a problem hiding this comment.
Well, the output is intended for use with an RFCs#42 implementing NixOS module, to put into (in the simplest case) services.<service>.settings. In the simplest cases one can just have a static type for the settings entry, but I guess it doesn't hurt to repeat it here and to allow, when necessary, some variation (depending on configuration format input option, I guess)
There was a problem hiding this comment.
That deserves introduction and propper reasoning in the detailed section, then?
There was a problem hiding this comment.
Well, we promise to provide type specification and serialiser.
There was a problem hiding this comment.
A great motion to factor config management, define abstract interfaces and reduce the current bundling! ❤️ I hope some of my comments prompt you to do a clarifying fixup commit.
I'd also like to see a broader perspective on config management incorporated (my stated use case is just a top-of-my-head thought, nothing duly researched)
| [summary]: #summary | ||
|
|
||
| Provide the configuration file generation functionality separately from NixOS | ||
| as a whole, and with scoped use of the module system. |
There was a problem hiding this comment.
What is a "scoped use of the module system"? Maybe affording another sentence for the message would do.
There was a problem hiding this comment.
Scope in the sense of visibility… reworded
| # Summary | ||
| [summary]: #summary | ||
|
|
||
| Provide the configuration file generation functionality separately from NixOS |
There was a problem hiding this comment.
I want to add, that in my use case, I would have an interest in that configuration management (as a shared resource) in nix would compound different APIs, like:
- filesystem API (configuration files)
- socket based APIs (fully dynamic service configuration)
→ It might still be useful to apply fully dynamic service configuration declaratively through nix.
Examples of such configuration APIs are:
- network gear TCP endpoints that respond to
yangornetconf - k8s APIs
- ldap configuration (set of boostrapping data after handing off data management to the ldap specific UIs)
It might be out of scope for this particular RFC, though it might be worth including in future work.
There was a problem hiding this comment.
I am not even sure it is in scope for future work!
Sure, that's a good idea in general, but this RFC is about sharing something that is already written now, and in a format that has seen some use by now. Inventing on-start configuration for things without persistent configuration is a good direction of work, but it is somewhat different type of work than described here.
There was a problem hiding this comment.
You are right maybe "future work" is too normative. Would you like to add a "broader context" (or a better naming!) section at the end (as an after thought)?
| functions implementing program configuration file generation. .As the design | ||
| uses a module system, putting configuration generators next to packages is also | ||
| not a perfect solution; also a configuration generator might end up generating | ||
| a set of config for a combination of packages. |
There was a problem hiding this comment.
As the design uses a module system, putting configuration generators next to packages is also
not a perfect solution; also a configuration generator might end up generating
a set of config for a combination of packages.
→ for terseness sake, that might entirely go into the alternative section?
| Abstract generation of configuration files with package-like flat arguments and | ||
| plain text file outputs. |
There was a problem hiding this comment.
This is not clear to me. Would that be something comparable to a templating approach with variable substitution?
There was a problem hiding this comment.
What you say is a reasonable implementation of this abstract interface.
| This RFC proposal builds upon RFCs#42 defining a configuration file abstraction | ||
| approach. | ||
|
|
||
| A subtree is established in the Nixpkgs repository serving as a library of |
There was a problem hiding this comment.
| A subtree is established in the Nixpkgs repository serving as a library of | |
| `NixOs/nixpkgs:./configs` is established serving as a library of |
(or some ore appropriate folder name)
There was a problem hiding this comment.
I do not think configs is a perfect name, but maybe it is good enough, so let's see if there are other opinions. In principle, that could be left to implementation step, should only lead to a little bit of bikeshedding
There was a problem hiding this comment.
Agree, maybe you can change it to "folder subtree" to differenciate the token from "attribute subtree" below.
There was a problem hiding this comment.
Hmmm. Indeed an identifier conflict, thanks. Reworded a bit.
| Each such generator takes as an input a NixOS module system based «subtree», | ||
| e.g. the values typically bound to `cfg` variable in typical modules. |
There was a problem hiding this comment.
NixOS module system based «subtree»
Does it take the subtree as input (as I understand it: it's path — since above sibtree referred to a path) or rather does it take the (lazily) evaluated attribute set corresponding to current config? (I'd induce the latter is what's intended here).
There was a problem hiding this comment.
Well, the value of the subtree? A possibly nested attribute set, yes. But not to entire config, just to the service.
There was a problem hiding this comment.
So here you could use "attribute subtree" to differenciate from "folder subtree" above. That would help to not get confused.
There was a problem hiding this comment.
Hm, maybe indeed worth expanding.
| file names of configuration files, and the values are attribute sets of RFCs#42 | ||
| settings abstraction and serialiser. |
There was a problem hiding this comment.
| file names of configuration files, and the values are attribute sets of RFCs#42 | |
| settings abstraction and serialiser. | |
| file names of configuration files, and the values are attribute sets with one or several attributes conforming to RFCs#42 settings notation and other required metadata. |
There was a problem hiding this comment.
«well-known» is factually wrong here for some of the more interesting configuration file formats.
There was a problem hiding this comment.
Oh, that was my interpretation of logSettings question on RFC42.
I changed the suggestion and eliminated well-known. It should be still a little clearer that way.
| Along with the function the corresponding type specifications for input and | ||
| output modules are provided. |
There was a problem hiding this comment.
| Along with the function the corresponding type specifications for input and | |
| output modules are provided. | |
| Along with the function definition, its corresponding input and output types are specified. |
Move this up to where the detailed idea is presented. At this position, I got almost confused as if it was an afterthought.
There was a problem hiding this comment.
It is still detailed-design section. We define the function, and what else is provided. In a sense, it is an afterthought, or supporting scaffolding.
There was a problem hiding this comment.
With the removed non-terse interlude this notion is now close enough to it's object (the functions). I reformulated for conciseness sake.
"Provided along with the function." can mean many things (including providing when calling).
"specified along with the function definition" is much less open, though still not perfect.
There was a problem hiding this comment.
Somewhat reshuffled everything in the paragraph.
| approach. | ||
|
|
||
| A subtree is established in the Nixpkgs repository serving as a library of | ||
| functions implementing program configuration file generation. .As the design |
There was a problem hiding this comment.
| functions implementing program configuration file generation. .As the design | |
| functions implementing program configuration file generation. Alongside these functions their respective input and output types are specified. As the design |
|
I nominate myself as a shepherd. I don't have practical experience using Nix with other inits, but I am currently working on this. |
|
I’d be interested maybe helping out when it’s in progress. Maybe a flakes-based setup will help realizing this. |
| Define generic service abstraction for non-`systemd` systems, possibly with a | ||
| convertor from `systemd` units. | ||
|
|
||
| Consider some way of referring from package to related configuration file |
There was a problem hiding this comment.
Why? (what for) → you might want to clarify to improve intelligibility here.
There was a problem hiding this comment.
Anyone interested in this future work item now is likely to be aware of the context like passthru.tests; those who are not familiar with that will wait anyway until some detailed work is started
There was a problem hiding this comment.
Your response is a little bit enigmatic with regard to rhe matter of the question, but let's make some assumptions at the risk of requiring further clarification:
Do you consider here the possibility of moving service configuration still closer to the packaging itself?
If that is the case, the (new) proposed module system for packaging might come into play. And (very) early coordination with this RFC might benefit future development.
There was a problem hiding this comment.
We'll see from feedback, but currently the RFC argues against moving configuration generators close to packages. However, we might consider developing a way of having some kind of cross-references available at evaluation time.
|
https://github.com/svanderburg/nix-processmgmt/ is my favourite prototype so far for abstracting services I know the example is listed as "a minimalistic silly example" but looking at it I have no idea how the input results in the output |
|
agree on I think how the input gets converted into the output is an implementation detail? I expect whatever is described before implementation will get even the interface slightly changed once implemented; I do not think specifying internals of the function implementation is a good idea. I guess at first it should be similar to how #42 is prototyped now. |
|
@7c6f434c I'm unclear as to whether this RFC is focusing on just configuration files or also the definiton of services, both of which having different implementations across nixos, home-manager, nix-darwin as you've listed. I get not wanting to define implementation details but in the output there are file names like With #42 I can see how things were before, a proposed structure, and a full example module. |
|
I wouldn't really want to switch to a new way of defining services, only to then adopt a nix-processmgmt approach and need to do it all over again.
Nah, would be out of character for NixOS to adopt init configurability any time soon. On the other hand, #42 is a good idea anyway, and this RFC is more about making the code that would be written anyway available separately from the rest of the things, os not too much of a change of how things are written, just about at what interfaces they are split.
I'm unclear as to whether this RFC is focusing on just configuration files or also the definiton of services, both of which having different implementations across nixos, home-manager, nix-darwin as you've listed.
As summary and detailed design say, this is about configuration files, as there the difference in desired functionality for different targets is pretty small.
There are a lot of mentions of systemd in the RFC but I think the problem of configuration files and services having different implementations in different projects are two separate but related issues.
The mentions about RFC are:
* explanation why the scope does not include complete service generation, only configuration files (systemd unit reuse)
* future work on possible service generation for non-NixOS
If the RFC's goal is "Extracting configuration file generation out of NixOS service modules in a more reusable way." then I think the RFC should avoid changing the actual systemd service much/at all and leave that to another RFC.
� which is why the second mention is future work and not unresolved questions.
I get not wanting to define implementation details but in the output there are file names like `listen.conf` and `content.conf`. where are they defined? what determines that `listen.conf` is JSON and `content.conf` is INI? where is it defined that the input `port` then becomes `listenPort` for `listen.conf`?
Well, the implementation needs to provide all these facts, which are generally determined by inspecting the program being configured.
With #42 I can see how things were before, a proposed structure, and a full example module.
I am still not sure whether there will be feedback leading to a slight course correction, so I am trying to delay making any kind of implementation prototypes.
|
|
As one data point for usecases; I recently started wanting to be able - in nix-on-droid - to start web servers for apps from f-droid that are just frontend clients. My understanding is Termux doesn't support systemd, or something, ...and then I ended up going doing a slight bit more research: There appears to be a bit of infra for launching service scripts: A bit of a todo/note to self:
You can observe me naively running into idiosyncrasies of the proot environment in nix-community/nix-on-droid#75 (comment) . IIRC the main issues were debugging difficulties and x11 needing patches from termux to handle setuid/setgid exiting with failure because proot refuses to emulate them - so nothing immediately relevant to service generation I think. |
|
Potential use case: ttacon/glorious#49 / numtide/devshell#47 |
edolstra
added
the
status: open for nominations
rfcs/0072-commonmark-docs.md
Outdated
| @@ -0,0 +1,334 @@ | |||
| --- | |||
There was a problem hiding this comment.
For some reason 0072-commonmark-docs.md is part of this RFC, could you remove it?
There was a problem hiding this comment.
Squashed the edits until now. Seems to have helped.
|
@Profpatsch would you be interested in being a shepherd on this RFC? Generally this RFC is in need for more shepherds before being able to move onwards. |
|
Not only you, to be fair… (I am also still in doubt whether there is something to discuss that benefits from synchronicity) |
|
@7c6f434c Well, a meeting can be a good way to get the discussion unstuck and decide on steps forward. So I think it would be a good idea. |
|
***@***.*** Well, a meeting can be a good way to get the discussion unstuck and decide on steps forward. So I think it would be a good idea.
Well, it is not that discussion is stuck as a discussion, it is that I am mostly convinced that some code should be tried out to see how it would look like / work like with a real module, but then I need to experiment a bit around the parts of the module system I never touched before to gain the confidence in what the hell I would be doing. Which I haven't got around to yet,
I guess I could quickly _spec_ stuff based on these import-as-subtree without checking if it works, but that sounds wrong.
Should this be converted to draft then? Closed with a plan to reopen once a import-as-subtree implementation has been tried by someone?
|
|
@7c6f434c Okay, thanks. I've marked the RFC as "on hold". It can be moved back to under discussion at any time, of course, when you're ready. |
|
👋 it's nearly been 2 years - anybody willing to pick this RFC back up? I'm relatively new to nix but RFC 42 seems to be the "default" now (at least, in home-manager and nix-darwin), and I'd love to see something like this happen! |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/is-nixbsd-a-posibility/29612/19 |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/is-nixbsd-a-posibility/29612/22 |
A helper for module system applications. Naming compatible with NixOS/nix#6257 To be supported by nix flake check in NixOS/nix#8332 Concept is in the spirit of a proposed module based solution to [RFC 0078 System-agnostic configuration file generators](NixOS/rfcs#78).
|
Maybe we could start by providing information about the system (e.g. |
|
We don't use the |
Extracting configuration file generation out of NixOS service modules in a more reusable way. Based on a discussion during NxCon 2020 Hackday.