Function chaining

[shelakel]

Would it be possible to allow calling any function on Any so that the value is passed to the function as the first parameter and the parameters passed to the function call on the value is added afterwards?
ex.

sum(a::Int, b::Int) -> a + b

a = 1
sum(1, 2) # = 3
a.sum(2) # = 3 or
1.sum(2) # = 3

Is it possible to indicate in a deterministic way what a function will return in order to avoid run time exceptions?

[JeffBezanson]

The . syntax is very useful, so we aren't going to make it just a synonym for function call. I don't understand the advantage of 1.sum(2) over sum(1,2). To me it seems to confuse things.

Is the question about exceptions a separate issue? i think the answer is no, aside from wrapping a function body in try..catch.

[shelakel]

The 1.sum(2) example is trivial (I also prefer sum(1,2)) but it's just to demonstrate that a function isn't owned per se by that type ex. 1 can be passed to a function with the first parameter being a Real, not just to functions that expect the first parameter to be an Int.

Edit: I might have misunderstood your comment. Dot functions will be useful when applying certain design patterns such as the builder pattern commonly used for configuration. ex.

validate_for(name).required().gt(3) 
# vs 
gt(required(validate_for(name)), 3) 

The exceptions I was just referring to is due to functions returning non-deterministic results (which is anyway bad practice). An example would be calling a.sum(2).sum(4) where .sum(2) sometimes return a String instead of an Int but .sum(4) expects an Int. I take it the compiler/runtime is already smart enough to evaluate such circumstances - which would be same when nesting the function sum(sum(1, 2), 4) - but the feature request would require extending said functionality to enforce type constraints on dot functions.

[ssfrr]

One of the use cases people seem to like is the "fluent interface". It's sometimes nice in OOP APIs when methods return the object, so you can do things like some_obj.move(4, 5).scale(10).display()

For me I think that this is better expressed as function composition, but the |> doesn't work with arguments unless you use anon. functions, e.g. some_obj |> x -> move(x, 4, 5) |> x -> scale(x, 10) |> display, which is pretty ugly.

One option to support this sort of thing would be if |> shoved the LHS as the first argument to the RHS before evaluating, but then it couldn't be implemented as a simple function as it is now.

Another option would be some sort of @composed macro that would add this sort of behavior to the following expression

You could also shift responsibility for supporting this to library designers, where they could define

function move(obj, x, y)
    # move the object
end

move(x, y) = obj -> move(obj, x, y)

so when you don't supply an object it does partial function application (by returning a function of 1 argument) which you could then use inside a normal |> chain.

[kmsquire]

Actually, the definition of |> could probably be changed right now to the
behavior your asking for. I'd be for it.

On Monday, January 27, 2014, Spencer Russell [email protected]
wrote:

One of the use cases people seem to like is the "fluent interface". It's
sometimes nice in OOP APIs when methods return the object, so you can do
things like some_obj.move(4, 5).scale(10).display()

For me I think that this is better expressed as function composition, but
the |> doesn't work with arguments unless you use anon. functions, e.g. some_obj
|> x -> move(x, 4, 5) |> x -> scale(x, 10) |> display, which is pretty
ugly.

One option to support this sort of thing would be if |> shoved the LHS as
the first argument to the RHS before evaluating, but then it couldn't be
implemented as a simple function as it is now.

Another option would be some sort of @Composed macro that would add this
sort of behavior to the following expression

You could also shift responsibility for supporting this to library
designers, where they could define

function move(obj, x, y)
# move the object
end

move(x, y) = obj -> move(obj, x, y)

so when you don't supply an object it does partial function application
(by returning a function of 1 argument) which you could then use inside a
normal |> chain.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5571#issuecomment-33408448
.

[shelakel]

ssfrr I like the way you think! I was unaware of the function composition |>. I see there's recently been a similar discussion [https://github.com//issues/4963].

kmsquire I like the idea of extending the current function composition to allow you to specify parameters on the calling function ex. some_obj |> move(4, 5) |> scale(10) |> display. Native support would mean one less closure, but what ssfrr suggested is a viable way for now and as an added benefit it should also be forward compatible with the extended function composition functionality if it gets implemented.

Thanks for the prompt responses :)

[kmsquire]

Actually, @ssfrr was correct--it isn't possible to implement this as a simple function.

[jakebolewski]

What you want are threading macros (ex. http://clojuredocs.org/clojure_core/clojure.core/-%3E). Unfortunate that @-> @->> @-?>> is not viable syntax in Julia.

[ssfrr]

Yeah, I was thinking that infix macros would be a way to implement this. I'm not familiar enough with macros to know what the limitations are.

[kmsquire]

I think this works for @ssfrr's compose macro:

Edit: This might be a little clearer:

import Base.Meta.isexpr
_ispossiblefn(x) = isa(x, Symbol) || isexpr(x, :call)

function _compose(x)
    if !isa(x, Expr)
        x
    elseif isexpr(x, :call) &&    #
        x.args[1] == :(|>) &&     # check for `expr |> fn`
        length(x.args) == 3 &&    # ==> (|>)(expr, fn)
        _ispossiblefn(x.args[3])  #

        f = _compose(x.args[3])
        arg = _compose(x.args[2])
        if isa(f, Symbol)
            Expr(:call, f, arg) 
        else
            insert!(f.args, 2, arg)
            f
        end
    else
        Expr(x.head, [_compose(y) for y in x.args]...)
    end
end

macro compose(x)
    _compose(x)
end

julia> macroexpand(:(@compose x |> f |> g(1) |> h('a',"B",d |> c(fred |> names))))
:(h(g(f(x),1),'a',"B",c(d,names(fred))))

[StefanKarpinski]

If we're going to have this |> syntax, I'd certainly be all for making it more useful than it is right now. Using just to allow putting the function to apply on the right instead of the left has always seemed like a colossal waste of syntax.

[malmaud]

+1. It's especially important when you are using Julia for data analysis, where you commonly have data transformation pipelines. In particular, Pandas in Python is convenient to use because you can write things like df.groupby("something").aggregate(sum).std().reset_index(), which is a nightmare to write with the current |> syntax.

[cdsousa]

👍 for this.

(I'd already thought in suggesting the use of the .. infix operator for this (obj..move(4,5)..scale(10)..display), but the operator |> will be nice too)

[malmaud]

Another possibility is adding syntactic sugar for currying, like
f(a,~,b) translating to x->f(a,x,b). Then |> could keep its current meaning.

[ssfrr]

Oooh, that would be a really nice way to turn any expression into a function.

Possibly something like Clojure's anonymous function literals, where #(% + 5) is shorthand for x -> x + 5. This also generalizes to multiple arguments with %1, %2, etc. so #(myfunc(2, %1, 5, %2) is shorthand for x, y -> myfunc(2, x, 5, y)

Aesthetically I don't think that syntax fits very well into otherwise very readable julia, but I like the general idea.

To use my example above (and switching to @malmaud's tilde instead of %), you could do

some_obj |> move(~, 4, 5) |> scale(~, 10) |> display

which looks pretty nice.

This is nice in that it doesn't give the first argument any special treatment. The downside is that used this way we're taking up a symbol.

Perhaps this is another place where you could use a macro, so the substitution only happens within the context of the macro.

[StefanKarpinski]

We obviously can't do this with ~ since that's already a standard function in Julia. Scala does this with _, which we could also do, but there's a significant problem with figuring out what part of the expression is the anonymous function. For example:

map(f(_,a), v)

Which one does this mean?

map(f(x->x,a), v)
map(x->f(x,a), v)
x->map(f(x,a), v)

They're all valid interpretations. I seem to recall that Scala uses the type signatures of functions to determine this, which strikes me as unfortunate since it means that you can't really parse Scala without knowing the types of everything. We don't want to do that (and couldn't even if we wanted to), so there has to be a purely syntactic rule to determine which meaning is intended.

[ssfrr]

Right, I see your point on the ambiguity of how far to go out. In Clojure the whole expression is wrapped in #(...) so it's unambiguous.

In Julia is it idiomatic to use _ as don't-care value? Like x, _ = somfunc() if somefunc returns two values and you only want the first one?

To solve that I think we'd need macro with an interpolation-like usage:

some_obj |> @$(move($, 4, 5)) |> @$(scale($, 10)) |> display

but again, I think it's getting pretty noisy at that point, and I don't think that @$(move($, 4, 5)) gives us anything over the existing syntax x -> move(x, 4, 5), which is IMO both prettier and more explicit.

I think this would be a good application of an infix macro. As with #4498, if whatever rule defines functions as infix applied to macros as well, we could have a @-> or @|> macro that would have the threading behavior.

[malmaud]

Ya, I like the infix macro idea, although a new operator could just be introduced for this use in lieu of having a whole system for inplace macros. For example,
some_obj ||> move($,4,5) ||> scale($, 10) |> disp
or maybe just keep |> but have a rule that
x |> f implicitly transforms into x |> f($):
some_obj |> scale($,10) |> disp

[meglio]

Folks, it all really looks ugly: |> ||> etc.
So far I found out Julia's syntax to be so clear that these things discussed above doesn't look so pretty if compared to anything else.

In Scala it's probably the worst thing - they have so much operators like ::, :, <<, >> +:: and so on - it just makes any code ugly and not readable for one without a few months of experience in using the language.

[johnmyleswhite]

Sorry to hear you don't like the proposals, Anton. It would be helpful if you made an alternative proposal.

[meglio]

Oh sorry, I am not trying to be unkind. And yes - critics without proposals
are useless.

Unfortunately I am not a scientist constructing languages so I just do not
know what to propose... well , except making methods optionally owned by
objects as it is in some languages.

[malmaud]

I like the phrase "scientist constructing languages" - it sounds much more grandiose than numerical programmers sick of Matlab.

I feel that almost every language has a way to chain functions - either by repeated application of . in OO languages, or special syntax just for that purpose in more functional languages (Haskell, Scala, Mathematica, etc.). Those latter languages also have special syntax for anonymous function arguments, but I don't think Julia is really going to go there.

I'll reiterate support for Spencer's proposal - x |> f(a) get translated into f(x, a), very analogously to how do blocks works (and it reinforces a common theme that the first argument of a function is privileged in Julia for syntactic sugar purposes). x |> f is then seen as short-hand for x |> f(). It's simple, doesn't introduce any new operators, handles the vast majority of cases that we want function chaining for, is backwards-compatible, and fits with existing Julia design principles.

[JeffBezanson]

I also think that is the best proposal here, main problem being that it seems to preclude defining |> for things like I/O redirection or other custom purposes.

[ssfrr]

Just to note, . is not a special function chaining syntax, but it happens to work that way if the function on the left returns the object it just modified, which is something that the library developer has to do intentionally.

Analogously, in Julia a library developer can already support chaining with |> by defining their functions of N arguments to return a function of 1 argument when given N-1 arguments, as mentioned here

That would seem to cause problems if you want your function to support variable number of args, however, so having an operator that could perform the argument stuffing would be nice.

@JeffBezanson, it seems that this operator could be implemented if there was a way to do infix macros. Do you know if there's an ideological issue with that, or is just not implemented?

[kmsquire]

Recently, ~ was special-cased so that it quoted its arguments and calls
the macro @~ by default. |> could be made to do the same thing.

Of course, in a few months, someone will ask for <| to do the same...

On Thursday, February 6, 2014, Spencer Russell [email protected]
wrote:

Just to note, . is not a special function chaining syntax, but it happens
to work that way if the function on the left returns the object it just
modified, which is something that the library developer has to do
intentionally.

Analogously, in Julia a library developer can already support chaining
with |> by defining their functions of N arguments to return a function
of 1 argument when given N-1 arguments, as mentioned herehttps://github.com//issues/5571#issuecomment-33408448

That would seem to cause problems if you want your function to support
variable number of args, however, so having an operator that could perform
the argument stuffing would be nice.

@JeffBezanson https://github.com/JeffBezanson, it seems that this
operator could be implemented if there was a way to do infix macros. Do you
know if there's an ideological issue with that, or is just not implemented?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5571#issuecomment-34374347
.

[ssfrr]

right, I definitely wouldn't want this to be a special case. Handling it in your API design is actually not that bad, and even the variable arguments limitation isn't too much of an issue if you have type annotations to disambiguate.

function move(obj::MyType, x, y, args...)
    # do stuff
    obj
end

move(args...) = obj::MyType -> move(obj, args...)

I think this behavior could be handled by a @composable macro that would handle the 2nd declaration.

The infix macro idea is attractive to me in the situation where it would be unified with declaring infix functions, which is discussed in #4498.

[meglio]

Why Julia creators are so much against allowing objects to contain their own methods? Where could I read more about that decision? Which thoughts and theory are behind that decision?

[ihnorton]

@meglio a more useful place for general questions is the mailing list or the StackOverflow julia-lang tag. See Stefan's talk and the archives of the users and dev lists for previous discussions on this topic.

[porterjamesj]

Just chiming in, to me the most intuitive thing is to have some placeholder be replaced by the
value of the previous expression in the sequence of things you're trying to compose, similar to clojure's as-> macro. So this:

@as _ begin
    3+3
    f(_,y)
    g(_) * h(_,z)
end

would be expanded to:

g(f(3+3,y)) * h(f(3+3,y),z)

You can think of the expression on the previous line "dropping down" to fill the underscore hole on the next line.

I started sketching a tiny something like this last quarter in a bout of finals week procrastination.

We could also support a oneliner version using |>:

@as _ 3+3 |> f(_,y) |> g(_) * h(_,z)

[kmsquire]

@porterjamesj, I like that idea!

[JeffBezanson]

I agree; that is pretty nice, and has an appealing generality.
On Feb 7, 2014 3:19 PM, "Kevin Squire" [email protected] wrote:

@porterjamesj https://github.com/porterjamesj, I like that idea!

Reply to this email directly or view it on GitHubhttps://github.com//issues/5571#issuecomment-34497703
.

[jaynagpaul]

The way elixir does it where the pipe operator always passes in the left-hand side as the first argument and allows extra arguments afterward, has been pretty useful, I would love to see something like "elixir" |> String.ends_with?("ixir") as a first class citizen in Julia.

[enerdgumen]

Other languages define it as Uniform Function Call Syntax.
This feature offers several advantages (see Wikipedia), it would be nice if Julia support it.

[javadba]

So is there a fluent interface to Julia at this point?

[StefanKarpinski]

Please post questions to the Julia discourse discussion forum.

[c42f]

In a fit of hacking (and questionable judgement!?) I've created another possible solution to the tightness of binding of function placeholders:

https://github.com/c42f/MagicUnderscores.jl

As noted over at #24990, this is based on the observation that one often wants certain slots of a given function to bind an _ placeholder expression tightly, and others loosely. MagicUnderscores makes this extensible for any user defined function (very much in the spirit of the broadcast machinery). Thus we can have such things as

julia> @_ [1,2,3,4] |> filter(_>2, _)
2-element Array{Int64,1}:
 3
 4

julia> @_ [1,2,3,4] |> filter(_>2, _) |> length
2

"just work". (With the @_ obviously going away if it's possible to make this a general solution.)

[richiejp]

Some variation @MikeInnes suggestion would seem adequate for my needs (usually long chains of filter, map, reduce, enumerate, zip etc. using do syntax).

c(f) = (a...) -> (b...) -> f(a..., b...)

1:10 |> c(map)() do x
    x^2
end |> c(filter)() do x
    x > 50
end

This works, although I can't get ' to work anymore. It is slightly shorter than:

1:10 |> x -> map(x) do x
    x^2
end |> x -> filter(x) do x
    x > 50
end

Also I guess one could just do

cmap = c(map)
cfilter = c(filter)
cetc = c(etc)
...

1:10 |> cmap() do x
    x^2
end |> cfilter() do x
    x > 50
end |> cetc() do ...

[MikeInnes]

As of 1.0 you'll need to overload adjoint instead of ctranspose. You can also do:

julia> Base.getindex(f::Function, x...) = (y...) -> f(x..., y...)

julia> 1:10 |> map[x -> x^2] |> filter[x -> x>50]
3-element Array{Int64,1}:
  64
  81
 100

If we could overload apply_type then we could get map{x -> x^2} :)

[t0mpr1c3]

@MikeInnes I just stole that

[t0mpr1c3]

A late and slightly frivolous contribution -- how about piping data to any location in the argument list using a combination of left and right curry operators:

VERSION==v"0.6.2"
import Base: ctranspose, transpose  
ctranspose(f::Function) = (a...) -> ((b...) -> f(a..., b...))  
 transpose(f::Function) = (a...) -> ((b...) -> f(b..., a...))

"little" |> (*)'''("Mary ")("had ")("a ") |> (*).'(" lamb")

[NightMachinery]

Clojure has some nice threading macros. Do we have those in the Julia ecosystem somewhere?

[t0mpr1c3]

Clojure has some nice threading macros. Do we have those in the Julia ecosystem somewhere?

https://github.com/MikeInnes/Lazy.jl

[oxinabox]

Clojure has some nice threading macros. Do we have those in the Julia ecosystem somewhere?

we have at least 10 of them.
I posted a list further up in the thread.
#5571 (comment)

[bramtayl]

Can you edit the list to have LightQuery instead of the other two packages of mine?

[encendre]

Since the |> operator come from elixir why not take inspiration from one of the ways they have to create anonymous functions ?
in elixir you can use &expr for defining a new anonymous function and &n for capturing positional arguments (&1 is the first arguments, &2 is the second, etc.)
In elixir there are extra stuff to write, (for example you need a dot before the parenthesis to call an anonymous function &(&1 + 1).(10))

But here's what it could look like in julia

&(&1 * 10)        # same as: v -> v * 10
&(&2 + 2*&5)      # same as: (_, x, _, _, y) -> x + 2*y
&map(sqrt, &1)    # same as: v -> map(sqtr, v)

So we can use the |> operator more nicely

1:9 |> &map(&1) do x
  x^2
end |> &filter(&1) do x
  x in 25:50
end

instead of

1:9 |> v -> map(v) do x
  x^2
end |> v -> filter(v) do x
  x in 25:50
end

note you can replace line 2 and 3 by .|> &(&1^2) or .|> (v -> v^2)

The main difference with the propositions with _ placeholder is that here it is possible to use positional arguments, and the & in front of the expressions makes the scope of the placeholders obvious (to the reader and the compiler).

Note that I have taken & in my examples, but the use of ?, _, $ or something else instead, would not change anything to the case.

[jpivarski]

Scala uses _ for the first argument, _ for the second argument, etc., which is concise, but you quickly run out of situations where you can apply it (can't repeat or reverse the order of arguments). It also doesn't have a prefix (& in the suggestion above) which disambiguates functions from expressions, and that, in practice, is another issue that prevents its use. As a practitioner, you end up wrapping intended inline functions in extra parentheses and curly brackets, hoping that it will be recognized.

So I'd say that the highest priority when introducing a syntax like this is that it be unambiguous.

But as for prefixes for arguments, $ has a tradition in the shell scripting world. It's always good to use familiar characters. If the |> is from Elixir, then that could be an argument to take & from Elixir, too, with the idea that users are already thinking in that mode. (Assuming there are a lot of former Elixir users out there...)

One thing that a syntax like this can probably never capture is creating a function that takes N arguments but uses fewer than N. The $1, $2, $3 in the body implies the existence of 3 arguments, but if you want to put this in a position where it will be called with 4 arguments (the last to be ignored), then there isn't a natural way to express it. (Other than predefining identity functions for every N and wrapping the expression with one of those.) This isn't relevant for the motivating case of putting it after a |>, which has only one argument, though.

[flipgthb]

I extended the @MikeInnes trick of overloading getindex, using Colon as if functions were arrays:

struct LazyCall{F} <: Function
	func::F
	args::Tuple
	kw::Dict
end

Base.getindex(f::Function,args...;kw...) = LazyCall{typeof(f)}(f,args,kw)

function (lf::LazyCall)(vals...; kwvals...)
	
	# keywords are free
	kw = merge(lf.kw, kwvals)
	
	# indices of free variables
	x_ = findall(x->isa(x,Colon),lf.args)
	# indices of fixed variables
	x! = setdiff(1:length(lf.args),x_)
	
	# the calling order is aligned with the empty spots
	xs = vcat(zip(x_,vals)...,zip(x!,lf.args[x!])...)
	args = map(x->x[2],sort(xs;by=x->x[1]))

	# unused vals go to the end
	callit = lf.func(args...,vals[length(x_)+1:end]...; kw...)
	
	return callit
end

[1,2,3,4,1,1,5]|> replace![ : , 1=>10, 3=>300, count=2]|> filter[>(50)]  # == [300]

log[2](2) == log[:,2](2) == log[2][2]() == log[2,2]()  # == true

It is much slower than lambdas or threading macros, but I think its super cool :p

[c42f]

To remind people commenting here, do have a look at relevant discussion at #24990.

Also, I'd encourage you to try out https://github.com/c42f/Underscores.jl which gives a function-chaining-friendly implementation of _ syntax for placeholders. @jpivarski based on your examples, you may find it fairly familiar and comfortable.

[vtjnash]

This seems to be a morass of ideas, better suited now to discourse (since github search and pagination is awful), or the sub-issues it has spawned (such as #24990)

[fatihaltindis]

Let's say, we have couple of functions named as f!(x,y,z), g!(x,y,z) and h!(x,y,z). Each function takes the input arguments, do some operation and returns nothing but make changes on input arguments. For instance;

function f!(x,y,z)
    x = x+1
    y = y+2
    z = z+3
    return nothing
 end

For the sake of simplicity, we can assume g! and h! function does similar operations, and return nothing.

My question is How can we create a pipeline in Julia to call these 3 functions in order? I tried the code below. But it didn't work. How can we make it work?
@pipe (x,y,z) |> f! |> g! |> h!

Thanks in advance for your help and suggestions.