-
-
Notifications
You must be signed in to change notification settings - Fork 159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC 0148] Pipe operator #148
base: master
Are you sure you want to change the base?
Conversation
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nix is already considered to be overly complex by many. Of course we know that Nix solves hard problems and can't do so without exposing intrinsic complexity, but this needed complexity is already more than most people expect to have to deal with, coming from the broken but "easy" traditional methods.
This makes the addition of accidental complexity to Nix disproportionately harmful. It makes newcomers more likely to reject Nix, depriving themselves of a real solution to their packaging and deployment problems, while depriving us from valuable contributions.
I am opposed to the pipe function, and skeptical of a function application operator; especially one that reverses the evaluation order compared to normal function application.
rfcs/0148-pipe-operator.md
Outdated
|
||
## `builtins.pipe` | ||
|
||
`lib.pipe`'s functionality is implemented as a built-in function. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pipe
is not a good function because the order is not obvious. It's already available practically everywhere as lib.pipe
. It's also hardly ever used, so doing this for performance strips it of the last argument for making it a builtin.
Builtins need to satisfy more stringent requirements, they take resources from the Nix team, and they can never be changed or removed.
I would recommend to always use a library, which can change things arbitrarily, thanks to being locked or pinned.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's also hardly ever used
I think this is due to documentation issues rather than due to this function itself being bad. While it has a lot of potential of misuse, it can actually make code more readable if you put in the effort for your code to be readable from top to bottom.
I would recommend to always use a library, which can change things arbitrarily, thanks to being locked or pinned
This... makes sense, but then I'd make an argument that parts of nixpkgs.lib
that manipulate generic data should exist outside of builtins
and outside of nixpkgs
- in a separate repository. It is not weird: we already have library flakes like flake-utils
or flake-parts
.
This discussion is outside of the scrope of this RFC, though; I merely disagree that lib.pipe
not seeing much usage is due to technical reasons.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shepherds currently believe this conversation is resolved, outcome: inclined to leave pipe
in lib
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pipe is not a good function because the order is not obvious.
@roberth for what it’s worth, the commit where I introduced it has a rationale for the order in its message.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not a problem with your contribution, but with the constraints of the Nix syntax.
lib.pipe
is a necessary step, and its adoption shows that it is useful to many, so thank you for creating it!
rfcs/0148-pipe-operator.md
Outdated
## Change the `pipe` function signature | ||
|
||
There are many equivalent ways to declare this function, instead of just using the current design. | ||
For example, one could flip its arguments to allow a partially-applied point-free style (see above). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you could call this curried style.
While it is point free in the technical sense of not having a variable name to carry the data flow, point-free is generally only used in a context where higher order functions are used for more arbitrary data flows. For many, it also carries the connotation of the synonymously used pointless style.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've never heard that term and the resources I've read so far all talked about "point-free" programming style. Google does not seem to know much either, so some links would be appreciated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The flipped pipe function is just function composition on a list, eta expanded. Eta expansion / reduction do not change the meaning, so these two functions would be indistinguishable.
That means that we can divide the problem and drop some unnecessary terminology.
pipe = flip composeFunctions
(andcomposeFunctions = flip pipe
by virtue of aflip
law)composeFunctions
is isomorphic topipe
composeFunctions
returns a function, which makes it easier to use where a function is expected: the application of higher order functions such asmap
.
A lot of that is just overly formal thought. Instead we can simplify this section to:
flip pipe
is equivalent to function composition applied to a list. By using a function that composes a list of functions instead of pipe, we get back a function, which can be readily used in higher order functions such as map
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
point-free is also point-less, in a semantic sense 😛
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shepherds currently believe this conversation can be resolved by accepting or rejecting the suggestion below.
I am very well aware of the language's complexity. I am also aware of the fact that some of its users are very technical and into functional programming languages, and for many others this is a first contact with these concepts. This is why I suggest picking only one of these operators as a compromise. I don't have any overly strong preferences here, but I make a point against function composition instead of argument piping, because the former leads to the so-called "point-free" programming style which tends to be more confusing for new users. I picked I strongly oppose calling this proposal "accidental" complexity. I spent a lot of time thinking about the options and trying to balance expressiveness and complexity, finding a compromise that makes solving real Nix problems easier. The Nix language is full of weird quirks users eventually have to face: we even have a wiki article collecting such instances, and it is far from exhaustive. Feel free to call these accidental complexity. But nothing about this proposal is accidental. |
To be clear, I don't use accidental to describe this RFC. I only used it in accidental complexity, which I've used as a synonym for extrinsic complexity. I can see that you've put a lot of thought into it, and I respect that, but that does not necessarily make the change a net positive.
A language with more syntax may be easier to write, but is not more expressive unless the syntax comes with semantics that were not already covered by existing features and combinations of them.
What is a real Nix problem? At least the lack of a syntax won't end up on the quirks page. Are the quirks a problem? Probably. I'd be happy to see a proposal that simplifies the language or makes the syntax easier to learn, but those are hard problems with a lot of inertia, and up-front costs that aren't "repaid" for years into the future. |
And more people consider Nix awkward to read and write rather than being complex. Mostly because the standard library is poor and not discoverable.
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/pre-rfc-pipe-operator/28387/10 |
I disagree with your interpretation of the word "expressiveness". Because you can (almost) always already solve all existing problems with existing syntax in (almost) any programming language. That's the point of being Turing complete. Therefore the question then becomes, how well can a problem be expressed in the language. And I think this holds true even for changes that are purely syntactical, like this one. When adding new syntax to the language, one goal may indeed be to make it easier to write. Another one, and IMO much more important, is does it make the language easier to read. Depending on the feature and the language those may coincide, or one may have the other as a side benefit, or they may contradict each other. I recently learned Haskell, so I am very well aware of the trap of having a lot of powerful operators that allow it to write programs concisely and without parentheses, but which comes at the cost of readability.
Problems that one might face when writing Nix code, either for some flake/shell/system configuration or when contributing to Nixpkgs. What I mean with this, is that I don't want to solve problems that for example mostly occur when trying to solve AdventOfCode in Nix. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While the lib.pipe
function is pretty cool (thanks for introducing it to me!), I think going through with this RFC would take gigantic effort, and the positive results (if any) will be seen only after a few years. And I'm thinking those positive results aren't worth it, because they won't make Nixlang significantly easier to read for beginners, and they don't solve any technical issues with Nixpkgs or Nix.
rfcs/0148-pipe-operator.md
Outdated
|
||
## `builtins.pipe` | ||
|
||
`lib.pipe`'s functionality is implemented as a built-in function. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's also hardly ever used
I think this is due to documentation issues rather than due to this function itself being bad. While it has a lot of potential of misuse, it can actually make code more readable if you put in the effort for your code to be readable from top to bottom.
I would recommend to always use a library, which can change things arbitrarily, thanks to being locked or pinned
This... makes sense, but then I'd make an argument that parts of nixpkgs.lib
that manipulate generic data should exist outside of builtins
and outside of nixpkgs
- in a separate repository. It is not weird: we already have library flakes like flake-utils
or flake-parts
.
This discussion is outside of the scrope of this RFC, though; I merely disagree that lib.pipe
not seeing much usage is due to technical reasons.
rfcs/0148-pipe-operator.md
Outdated
like line numbers when some part of the pipeline fails. | ||
Additionally, it allows easy usage outside of Nixpkgs and increases discoverability. | ||
|
||
While Nixpkgs is bounds to minimum Nix versions and thus `|>` won't be available until |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is a great argument against going through with this RFC.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would be an argument for never making any changes to improve the language at all, ever. Unless you see some shiny alternative language to which we'll realistically all migrate to within the next five years, any improvements to the Nix language are still expected to be beneficial.
I've actually thought about proposing this change for quite some time now, and the main thing that held me back was the idea is that Nix is a "lost cause" and that I should rather spending my energy towards some successor. But eventually, I realized that realistically, we'll be stuck with Nix for a long time to come, so better invest the energy to make it a bit more comfortable …
Also note that outside of Nixpkgs, the feature will be available almost immediately. There is a lot of Nix code outside of Nixpkgs, all these flakes, shells, and users' system configurations. Projects like home-manager, devenv, nur, etc. are all free to migrate to newer Nix versions at their own pace. We shouldn't ignore these.
Honestly, the most effort in this RFC is convincing people, and maybe also the coordination across many projects. But from the technical side of things, this is actually pretty easy to implement.
This only applies to Nixpkgs, see #148 (comment)
They also won't make it significantly harder for beginners to read Nix code IMO, while bringing quality of life improvements to everyday Nix development. I don't understand why only "technical" problems should be worth solving. |
This RFC is now open for shepherd nominations! |
@AndersonTorres because I don't know every programming language and just didn't come across OCaml during my research. Furthermore, the related work section is intended to roughly cover the design space, not to be an exhaustive list of every programming language. |
Updated the text according to some of the initial feedback. The more I think of it, the more I'm leaning towards also having |
I was thinking on it, precisely, two pipe operators! It makes the language more orthogonal: if we have The problems are:
|
I nominate @maralorn, who has a lot of experience with functional programming languages and programming language design. |
@roberth, what about you, would you like to be a shepherd? I'd rather like to have critical voices on board from the beginning … |
I am pretty certain there are a lot of people with more experience on this in the wider community. That doesn’t prevent me from having opinions on this particular bikeshed, though. 🤣 My availability depends on the timeframe here. I will be very busy in the next few weeks, after that I’d be on board. |
I've procrastinated writing this RFC for at least half a year, so waiting two more months won't be the end of the world |
I can do that |
Applied all suggestions; about the evaluation order point I'd like to delay that discussion until we have some real-world experience with implementations. |
Nix implementation: NixOS/nix#11131 |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2024-08-05/50170/1 |
Lix implementation is now merged into main as well. There is one small difference to the semantics of the implementation in Nix: Mixing both operators is not forbidden, instead |
@@ -0,0 +1,330 @@ | |||
--- |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Plan for moving forward/stabilizing feature in Nix
Now that the experimental feature has been released, it may be time to start talking about a plan for evaluating it.
I see a number of possibilities:
- We do nothing and hope it happens organically that enough enthusiasts try the feature out to give it a good trial. In all likelihood, people will only comment if they dislike something about the feature, which means we could receive little or no feedback and be left wondering if enough people have looked at it. This requires least effort but is likely to take the longest.
- We promote the feature in official Nix channels and set up some sort of polling for both positive and negative feedback, with an expectation of how long the poll will be running before moving this RFC to FCP. This requires a bit of up-front work and may be biased towards people who pay attention to official Nix channels, but has the advantage of resolving faster than the first possibility.
- We devise an actual experiment to measure people's ability to learn the new operators. This might take the form of a survey asking people to rate the readability of various Nix expressions, some with the new operators and some without, and testing if there is a statistically significant difference in the ratings. We could send the survey to a representative sample of community members instead of (or in addition to) the public polling from the previous possibility, to try to control for bias. I'm not aware of the Nix community undertaking this level of rigor in the past but it would help us bring a stronger case for assuaging people's concerns about the mental cost of adding new operators. It would be a substantial investment of time from a few people (I'm happy to be one of those people but it probably shouldn't be me alone).
Anyone else have thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do nothing and hope it happens organically that enough enthusiasts try the feature out to give it a good trial. In all likelihood, people will only comment if they dislike something about the feature, which means we could receive little or no feedback and be left wondering if enough people have looked at it. This requires least effort but is likely to take the longest.
I think it would be beneficial to give the new feature more visibility and promotion, rather than leaving it as an experimental feature indefinitely.
Collecting feedback through some form of polling or voting mechanism could provide valuable insights.
Additionally, introducing new features in a programming language is a significant event, especially for a community-driven project like Nix. It's likely that experienced PL experts would be needed to assess its impact and stability.
This comment was marked as duplicate.
This comment was marked as duplicate.
Latest observations:
Based on these observations and considering that a single operator could bring most of the value, I'm leaning towards accepting If we take this direction, perhaps |
Here are some of my thoughts after using this operator for a while: Partially applied pipesConsider nix-repl> inc = x: x+1
nix-repl> x = i: i |> inc |> toString
nix-repl> x 3
"4" It would be nice to drop the Lambdas inside pipenix-repl> double = x: x*2
# current syntax
nix-repl> 3 |> (x: 1 + x) |> double |> toString
"8"
# proposed alternative
nix-repl> 3 |> x: 1 + x |> double |> toString
# current syntax
nix-repl> 3 |> (x: 1 + (x |> double)) |> toString
"7"
# proposed alternative
nix-repl> 3 |> x: 1 + (x |> double) |> toString The brackets needed around the function feel redundant. |
@@ -0,0 +1,330 @@ | |||
--- |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Attempting to corral feedback into a thread.)
nix-repl> double = x: x*2 # current syntax nix-repl> 3 |> (x: 1 + x) |> double |> toString "8" # proposed alternative nix-repl> 3 |> x: 1 + x |> double |> toString # current syntax nix-repl> 3 |> (x: 1 + (x |> double)) |> toString "7" # proposed alternative nix-repl> 3 |> x: 1 + (x |> double) |> toString
Both of those proposals seem, to me, to be ‘more likely’ to mean something else:
3 |> (x: (1 + x |> double |> toString))
3 |> (x: (1 + (x |> double) |> toString))
In other words, if |>
binds more loosely than the lambda-forming :
, it would be the first thing in the language to do so and thus would be very surprising to me. So far in the language, a lambda always extends as far rightward as it can.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Partially applied pipes
To me this smells like function composition with extra steps. Function composition is discussed in the RFC as well, and while I am not completely opposed to it, my fear is that it reduces code readability given a lack of type annotations. In x = |> inc |> toString
it is a lot less intuitively clear that x
is a function.
This is one of these questions where I'd like to see more usage to find out whether this is a thing that comes up sufficiently often in practice that it is worth dealing with.
(Side note: I think we would get this feature for free if Nix had operator sections)
Lambdas inside pipe
While I agree that not requiring parentheses here would be nice, especially given that one goal of the pipe operator is to reduce parentheses, I don't see any way this would be realistically implementable without throwing the entire language under the bus. Making the binding strength of abstractions context dependent just doesn't sound great
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Making the binding strength of abstractions context dependent just doesn't sound great
agreed, it’s a recipe for disaster
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lambdas inside pipe
Making the binding strength of abstractions context dependent
Right. I didn't think that through. This is a bad idea.
Partially applied pipes
In
x = |> inc |> toString
it is a lot less intuitively clear thatx
is a function
In my opinion it is slightly more clear than, for example,
concatLines = concatStringsSep "\n"
getBin = getOutput "bin"
or any other partially applied function. The open |>
(or |
) at the start suggests to me that this is a function.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2024-08-19/50831/1 |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2024-09-02/51514/1 |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2024-09-16/52224/1 |
Implements: NixOS/rfcs#148 Closes: #554
Implements: NixOS/rfcs#148 Closes: #554
Implements: NixOS/rfcs#148 Closes: #554
Implements: NixOS/rfcs#148 Closes: #554
|
||
## Nixpkgs interaction | ||
|
||
As soon as the Nixpkgs minimum version contains `|>`, using it will be allowed and encouraged in the documentation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it allowed to backport the pipe operator to the Nixpkgs minimum nix version?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do major feature additions like this usually get a backport?
Even if it was backported, wouldn't it defeat the purpose of having a minimal version that's likely to be installed by most users?
A backport would still be a new release, even if it was based on an old major version number.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sometimes. Support for zstd cache compression was backported to 2.3, so it's within the realms of possibility.
The reason we have a minimum version of 2.3 not so much to support people who haven't updated Nix in several years, it's because later versions of Nix have unresolved regressions in things that some people depend on.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2024-09-30/53690/1 |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2024-10-28/55095/1 |
RFCSC: @rhendric, what's the status of this RFC? What is the next step to move this RFC forward? |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2024-11-11/55888/1 |
Per these notes, we are nominally now in a period of experimentation, collecting feedback, and iterating. I had ventured some thoughts in this direction here but to my knowledge nothing formal has actually been done. |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2024-11-25/56591/1 |
Rendered
Discussion notice: please try to attach all discussions to a thread by using the code review feature. If your comment doesn't refer a specific line to attach to, use the header line instead.