Tracking Issue for RFC 3216: "Allow using for<'a> syntax when declaring closures"

[pnkfelix]

This is a tracking issue for the RFC "Allow using for<'a> syntax when declaring closures" (rust-lang/rfcs#3216).
The feature gate for the issue is #![feature(closure_lifetime_binder)].

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Steps

• Implement the RFC - implemented in Implement for<> lifetime binder for closures #98705
• Adjust documentation (see instructions on rustc-dev-guide)
• Stabilization PR (see instructions on rustc-dev-guide)

Unresolved Questions

None at this time

Implementation history

• Initial implementation in Implement for<> lifetime binder for closures #98705

[WaffleLapkin]

I'd like to work on implementing this 👀

@rustbot claim

[Jules-Bertholet]

Found an ICE involving this feature: #103736

(Maybe we could use an F-closure_lifetime_binder tag?)

[Jules-Bertholet]

Suggestion: could it be allowed to elide the return type of an explicitly higher-ranked closure when it is ()? Currently, Rust allows eliding () returns in all other function signatures, even when nothing else can be elided.

[WaffleLapkin]

@Jules-Bertholet I don't think it's a good idea, we may want to allow inferring return type in the future, just like with normal closures.

[Jules-Bertholet]

@WaffleLapkin "Infer return type when it is ()" should be forward-compatible with "infer return type always."

[WaffleLapkin]

@Jules-Bertholet oh, I misunderstood this as "make for<> || { some expression... } return type always (), same as with normal function". Inferring it is probably fine.

[zirconium-n]

Is this feature blocked by any ongoing process? The implementation seems completed for a while, maybe we can push this to stabilization?

[opsnull]

Is this feature blocked by any ongoing process? The implementation seems completed for a while, maybe we can push this to stabilization?

same question here.

[runiq]

A question out of interest: Is this in any way a prerequisite for #68923?

[safinaskar]

Unfortunately, for<'a> |x: &'a ()| -> &'a () { x } and |x: &()| -> &() { x } currently are not equivalent. Here is minimal example:

#![feature(closure_lifetime_binder)]
#![feature(unboxed_closures)]

fn foo<F>(f: F)
where
    F: for<'a> FnOnce<(&'a (),)>,
{}

fn main() {
    foo(for<'a> |x: &'a ()| -> &'a () { x }); // COMPILES
    foo(|x: &()| -> &() { x }); // DOESN'T COMPILE
}

Here is playground:
https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=47b1d8af93240be0cc1326a892dc01fd

This was discovered because of #34162 . Here is (larger) code, which tries to fix sort_by_key (by Jules-Bertholet):

#![feature(closure_lifetime_binder)]
#![feature(unboxed_closures)]

fn sort_by_key<T, F>(s: &mut [T], mut f: F)
where
    F: for<'a> FnMut<(&'a T,)>,
    // instead of `B: Ord`
    for<'a> <F as FnOnce<(&'a T,)>>::Output: Ord,
{
    s.sort_by(|a, b| f(a).cmp(&f(b)))
}

#[derive(Debug)]
struct Client(String);

impl Client {
    fn key(&self) -> &str {
        &self.0
    }
}

fn main() {
    let mut test = vec![
        Client("c".to_string()),
        Client("a".to_string()),
        Client("b".to_string()),
    ];
    sort_by_key(&mut test, for<'a> |c: &'a Client| -> &'a str { c.key() }); // compiles
    //sort_by_key(&mut test, |c: &Client| -> &str { c.key() }); // doesn't compile
    dbg!(test);
}

The code above compiles on nightly without any additional flags (assuming you choose for<'a> version)

Also, here is another problematic code, which doesn't require unboxed_closures:

#![feature(closure_lifetime_binder)]

trait X<'a> {
}

impl<'a, T, K> X<'a> for T
where
    T: FnOnce(&'a ()) -> K
{}

fn foo<F>(f: F)
where
    F: for<'a> X<'a>,
{}

fn main() {
    foo(for<'a> |x: &'a ()| -> &'a () { x }); // COMPILES
    foo(|x: &()| -> &() { x }); // DOESN'T COMPILE
}

playground: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=bfcdede24f2e7b27380e2990df1af328

[workingjubilee]

So you're saying we need the syntactic ability to specifically annotate all closure lifetimes in order to be able to write the programs we want to write? Some kind of for<'lt> syntax, maybe?

[safinaskar]

@workingjubilee , no. We already have such syntax, it is available with #![feature(closure_lifetime_binder)]. I'm just saying for<'a> |x: &'a ()| -> &'a () { x } should work exactly like |x: &()| -> &() { x }, but it is currently not. And this causes problems as I showed above, which prevent us from having proper ergonomic sort_by_key

[Jules-Bertholet]

Ideally, the compiler would infer the most permissive signature possible, and so the for<'a> syntax would not be necessary (except as documentation).

[workingjubilee]

I'm just saying for<'a> |x: &'a ()| -> &'a () { x } should work exactly like |x: &()| -> &() { x }, but it is currently not.

It is my understanding that the binding of a specific lifetime and a for<'lt> (which, as I understand it, is effectively the ∀ lifetime) are not interchangeable in all cases, for the precise reason that ∀ and ∃ are different quantifiers.

[WanderLanz]

@safinaskar As far as I understand compiler internals, a closure is inherently not invariant in regards to the lifetime of the arguments, and you need strong hints in order to make a closure such similar to a normal fn. For example, the higher ranked closure macros in the lending_iterator crate.

In other words, the compiler is very conservative when closures could possibly have something like a for<'all> where Closure: 'all bound and instead only inherently satisfies FnOnce(&'1 ()) -> &'2 ().

#![feature(closure_lifetime_binder)]

trait X<'a> {
}

impl<'a, T, K> X<'a> for T
where
    T: FnOnce(&'a ()) -> K
{}

fn foo<F>(f: F)
where
    F: for<'a> X<'a>,
{}

fn main() {
    foo(for<'a> |x: &'a ()| -> &'a () { x }); // COMPILES
    // foo(|x: &()| -> &() { x }); // DOESN'T COMPILE
    
    fn hrc_hint<F>(f: F) -> F
    where
        F: for<'all> FnOnce(&'all ()) -> &'all ()
    {
        f
    }
    foo(hrc_hint(|x: &()| -> &() { x })); // NOW COMPILES
}

Finally, to avoid this, as far as I can tell, either: the compiler becomes less conservative on closures (possibly E.2024?), we get bounded HRTBs (for<'all> where *: 'all) (very unlikely), or closure_lifetime_binder becomes popular enough to be stabilized.

[Kixunil]

@runiq I don't know if it's required, it certainly looks helpful but FYI it's not suffficient - I tried using this and the code still doesn't compile.

[vojtechkral]

Hi!
I thought the code from #129787 could be fixed with this feature, but it looks like I'm getting the same error still:

fn f() -> impl Future<Output = ()> + Send {
    async {
        let data: &[_] = &[0u8];
        stream::iter([data])
            .map(for<'x> |_data: &'x [u8]| -> future::Ready<()> { future::ready(()) })
            .buffer_unordered(1)
            .for_each_concurrent(4, |_| async {})
            .await;
    }
}

-> implementation of `FnOnce` is not general enough.

though when I pass the lambda through an identity function that explicitly declares/requires a HRTB lifetime, it then builds:

fn lifetime_hrtb<F, R>(f: F) -> F
where
    F: for<'a> FnMut(&'a [u8]) -> R,
{
    f
}

fn f() -> impl Future<Output = ()> + Send {
    async {
        let data: &[_] = &[0u8];
        stream::iter([data])
            .map(lifetime_hrtb(|_data| async {}))
            .buffer_unordered(1)
            .for_each_concurrent(4, |_| async {})
            .await;
    }
}

any clues as to what's going on? 🙏

edit: this was on rustc 1.82.0-nightly (1f12b9b0f 2024-08-27)

[compiler-errors]

This is a tracking issue, for the record. If you have a bug with a specific feature, please open a new issue! If you want to discuss the implementation, please also open a new issue and it can be tagged C-discussion. Tracking issues have a linear history so they are not conducive to any kind of discussion.