-
Notifications
You must be signed in to change notification settings - Fork 12.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Stabilization] Pin APIs #55766
Comments
@rfcbot fcp merge |
Team member @withoutboats has proposed to merge this. The next step is review by the rest of the tagged team members: Concerns:
Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up! See this document for info about what commands tagged team members can give me. |
Thanks for the detailed writeup here @withoutboats! I've also sort of been historically confused by the various guarantees of In trying to start writing this down though I keep running up against a wall of "what is |
if i have got this correct, every type that is not self referential (ie: not a generator) is Unpin |
It's not just self-referentiality, there are some other use-cases for stable memory addresses that How I understand |
Hm I still don't really understand First off, it's probably helpful to know what types implement I keep trying to summarize or state what I think I understand the guarantees of |
@alexcrichton Thanks for the questions, I'm sure the pinning APIs can be a bit confusing for people who haven't been part of the group focusing on them.
Unpin is an auto trait like Send or Sync, so most types implement it automatically. Generators and the types of async functions are The explicit impls for pointer types is to make the
Here is the sort of fundamental idea of the pinning APIs. First, given a pointer type
The implication of all of this is that if you construct a So if you implement Moving a pointer type like |
This was one of the trickiest parts of the pinning API, and we got it wrong at first. Unpin means "even once something has been put into a pin, it is safe to get a mutable reference to it." There is another trait that exists today that gives you the same access: To actually implement a self-referential type would require unsafe code - in practice, the only self-referential types anyone cares about are those that the compiler generates for you: generators and async function state machines. These explicitly say they don't implement The problem emerges once you have a struct containing one of these anonymous types, like a future combinator. In order go from a For this reason, pin projection is unsafe! In order to perform a pin projection without violating the pinning invariants, you have to guarantee that you never do several things, which I listed in the stabilization proposal. So, the tl;dr: |
Shouldn't an async state machine have a |
I guess what matters in this context is whether an This is not the same as "drop glue", which can be called through |
Also, while a generator (and therefore an async state machine) has custom drop glue, it's just to drop the correct set of fields based on the current state, it promises to never move any of the fields during drop. |
The terminology I use (though I don't think there's any standard): "drop glue" is the compiler generated recursive walking of fields, calling their destructors; "Drop implementation" is an implementation of the |
Is someone on the hook to write a nomicon chapter on futures? It seems incredibly necessary given how subtle this is. In particular I think examples, especially examples of buggy/bad implementations and how they're unsound, would be illuminating. |
@gankro Sure, you can put me down for that. |
Thanks for the explanation everyone! I personally am super new to async Rust and the Pin APIs, but I have played a bit around with it during the last days (rust-lang/futures-rs#1315 - where I tried to exploit pinning for intrusive collections in async code). During experimentation I got some concerns with these APIs:
While getting things to compile, I often thought about C++, where most of these issues are avoided: Types can be declared unmovable by deleting the move constructor and assignment operator. When a type is not moveable, types which contain that type are also not. Thereby the property and requirements flow natively through the type hierarchy and get checked by the compiler, instead of forwarding the property inside some calls (not all, e.g. not Regarding Alex What I haven't yet fully understood is, why getting Another thought that I had is whether it's possible to get a simpler system if some limitations are applied. E.g. if things that require pinning could only be used inside async methods and not with future combinators. Maybe that could simplify things to one of On the positive side: Being able to write async/await code with "stack" borrows is pretty cool and much needed! And ability to use pinning for other use-cases, like intrusive collections, will help performance as well as targets like embedded systems or kernels. So I'm really looking forward to a solution on this. |
Minor nit w.r.t. the stabilization report:
I'm guessing this is actually an |
@Matthias247 you can't safely get &mut T from One thing I have trouble explaining to myself is what makes Future fundamentally different from other traits such that Pin needs to be part of its API. I mean, I know intuitively it's because async/await requires pinning, but does that say something specifically about Futures that's different from, say, Iterators? Could poll take &mut self and only implement Future for |
This actually makes me wonder whether there's a couple of missing methods on impl<'a, T: ?Sized> Pin<&'a T> {
fn map<U: Unpin, F: FnOnce(&T) -> &U>(self, f: F) -> Pin<&'a U>
}
impl<'a, T: ?Sized> Pin<&'a mut T> {
fn map<U: Unpin, F: FnOnce(&mut T) -> &mut U>(self, f: F) -> Pin<&'a mut U>
} These could be used in the I'm trying to work out why this macro claims to be unsafe. If we are The only situation where I can see it being unsound is implementing a custom To make the previous situation safer there could be a wrapper built on pub struct MustPin<T: Unpin>(T, Pinned);
impl<T: Unpin> MustPin<T> {
pub const fn new(t: T) -> Self { ... }
pub fn get(self: Pin<&Self>) -> *const T { ... }
pub fn get_mut(self: Pin<&mut Self>) -> *mut T { ... }
} This all seems backwards compatible with the current API, it could remove some of the unsafety from |
@Nemo157 Those map functions are unsafe because I could EDIT: Also the pin-utils macro is different, |
There's no theoretical difference, but there are pragmatic ones. For one, Another important pragmatic difference is that the code patterns between |
Ah, damn, forgot about that part of it 😦 |
Should these be called
Drop guaranteeI think we have to codify the
"Invalidate" here can mean "deallocation", but also "repurposing": When changing This has the consequence, for example, that it becomes illegal to deallocate a Repurposing
|
This is because, so far, all iterators are defined using a type and an However, even if this is not the primary motivation for including generators in the language, it would be very nice to eventually be able to use generator |
|
Ok thanks for the explanations all! I agree with @gankro that a nomicon-style chapter about In an effort to help myself understand this though I wanted to try again to write down why each function is safe or why it's
|
@HeroicKatora you're correct that that function would be safe. I don't mind adding it but I'd like to avoid expanding the API we're stabilizing in this moment, since this thread is already hundreds of comments long. We can always add it later as the need arises just as we do with all std APIs. |
I would just say that both naming and functionality of Edit: It would mostly just generalize what's already there.
has the instantiation for
|
Expand std::pin module docs and rename std::pin::Pinned to PhantomPinned cc #49150, #55766 r? @withoutboats
…richton Pin stabilization This implements the changes suggested in rust-lang#55766 (comment) and stabilizes the `pin` feature. @alexcrichton also listed several "blockers" in that issue, but then in [this comment](rust-lang#55766 (comment)) mentioned that they're more "TODO items": > In that vein I think it's fine for a stabilization PR to be posted at any time now with FCP lapsed for a week or so now. The final points about self/pin/pinned can be briefly discussed there (if even necessary, they could be left as the proposal above). Let's settle these last bits here and get this thing stabilized! :) r? @alexcrichton cc @withoutboats
…richton Pin stabilization This implements the changes suggested in rust-lang#55766 (comment) and stabilizes the `pin` feature. @alexcrichton also listed several "blockers" in that issue, but then in [this comment](rust-lang#55766 (comment)) mentioned that they're more "TODO items": > In that vein I think it's fine for a stabilization PR to be posted at any time now with FCP lapsed for a week or so now. The final points about self/pin/pinned can be briefly discussed there (if even necessary, they could be left as the proposal above). Let's settle these last bits here and get this thing stabilized! :) r? @alexcrichton cc @withoutboats
…richton Pin stabilization This implements the changes suggested in rust-lang#55766 (comment) and stabilizes the `pin` feature. @alexcrichton also listed several "blockers" in that issue, but then in [this comment](rust-lang#55766 (comment)) mentioned that they're more "TODO items": > In that vein I think it's fine for a stabilization PR to be posted at any time now with FCP lapsed for a week or so now. The final points about self/pin/pinned can be briefly discussed there (if even necessary, they could be left as the proposal above). Let's settle these last bits here and get this thing stabilized! :) r? @alexcrichton cc @withoutboats
Now that the stabilization has gone through, the point of the PFCP seems moot, so therefore: @rfcbot cancel |
Is there any tracking to make sure the comments developed starting at #55766 (comment) get turned into docs? Seems we are already too late for the release as beta got branched off... even though that was explicitly called out here to happen before stabilization :/ |
Is there any reason that this is still open? @Centril you closed and reopened this, was that deliberate? |
@RalfJung I tried to cancel the FCP but it didn't listen; @alexcrichton can you cancel the FCP? |
This is stable, so closing. |
The `Unpin` bound was originally added in rust-lang#56939 following the recommendation of @withoutboats in rust-lang#55766 (comment) That comment does not give explicit justification for why the bound should be added. The relevant context was: > [ ] Remove `impl<P> Unpin for Pin<P>` > > This impl is not justified by our standard justification for unpin > impls: there is no pointer direction between `Pin<P>` and `P`. Its > usefulness is covered by the impls for pointers themselves. > > This futures impl (link to the impl changed in this PR) will need to > change to add a `P: Unpin` bound. The decision to remove the unconditional impl of `Unpin for Pin` is sound (these days there is just an auto-impl for when `P: Unpin`). But, I think the decision to also add the `Unpin` bound for `impl Future` may have been unnecessary. Or if that's not the case, I'd be very interested to have the argument for why written down somewhere. The bound _appears_ to not be needed, since the presence of a `Pin<P>` should indicate that it's safe to project to `Pin<&mut P::Target>` just like for `Pin::as_mut`.
… r=m-ou-se Remove P: Unpin bound on impl Future for Pin We can safely produce a `Pin<&mut P::Target>` without moving out of the `Pin` by using `Pin::as_mut` directly. The `Unpin` bound was originally added in rust-lang#56939 following the recommendation of `@withoutboats` in rust-lang#55766 (comment) That comment does not give explicit justification for why the bound should be added. The relevant context was: > [ ] Remove `impl<P> Unpin for Pin<P>` > > This impl is not justified by our standard justification for unpin impls: there is no pointer direction between `Pin<P>` and `P`. Its usefulness is covered by the impls for pointers themselves. > > This futures impl (link to the impl changed in this PR) will need to change to add a `P: Unpin` bound. The decision to remove the unconditional impl of `Unpin for Pin` is sound (these days there is just an auto-impl for when `P: Unpin`). But, I think the decision to also add the `Unpin` bound for `impl Future` may have been unnecessary. Or if that's not the case, I'd be very interested to have the argument for why written down somewhere. The bound _appears_ to not be needed, as demonstrated by the change requiring no unsafe code and by the existence of `Pin::as_mut`.
… r=m-ou-se Remove P: Unpin bound on impl Future for Pin We can safely produce a `Pin<&mut P::Target>` without moving out of the `Pin` by using `Pin::as_mut` directly. The `Unpin` bound was originally added in rust-lang#56939 following the recommendation of ``@withoutboats`` in rust-lang#55766 (comment) That comment does not give explicit justification for why the bound should be added. The relevant context was: > [ ] Remove `impl<P> Unpin for Pin<P>` > > This impl is not justified by our standard justification for unpin impls: there is no pointer direction between `Pin<P>` and `P`. Its usefulness is covered by the impls for pointers themselves. > > This futures impl (link to the impl changed in this PR) will need to change to add a `P: Unpin` bound. The decision to remove the unconditional impl of `Unpin for Pin` is sound (these days there is just an auto-impl for when `P: Unpin`). But, I think the decision to also add the `Unpin` bound for `impl Future` may have been unnecessary. Or if that's not the case, I'd be very interested to have the argument for why written down somewhere. The bound _appears_ to not be needed, as demonstrated by the change requiring no unsafe code and by the existence of `Pin::as_mut`.
Remove P: Unpin bound on impl Future for Pin We can safely produce a `Pin<&mut P::Target>` without moving out of the `Pin` by using `Pin::as_mut` directly. The `Unpin` bound was originally added in #56939 following the recommendation of ``@withoutboats`` in rust-lang/rust#55766 (comment) That comment does not give explicit justification for why the bound should be added. The relevant context was: > [ ] Remove `impl<P> Unpin for Pin<P>` > > This impl is not justified by our standard justification for unpin impls: there is no pointer direction between `Pin<P>` and `P`. Its usefulness is covered by the impls for pointers themselves. > > This futures impl (link to the impl changed in this PR) will need to change to add a `P: Unpin` bound. The decision to remove the unconditional impl of `Unpin for Pin` is sound (these days there is just an auto-impl for when `P: Unpin`). But, I think the decision to also add the `Unpin` bound for `impl Future` may have been unnecessary. Or if that's not the case, I'd be very interested to have the argument for why written down somewhere. The bound _appears_ to not be needed, as demonstrated by the change requiring no unsafe code and by the existence of `Pin::as_mut`.
@rfcbot fcp merge
Feature name:
pin
Stabilization target: 1.32.0
Tracking issue: #49150
Related RFCs: rust-lang/rfcs#2349
This is a proposal to stabilize the
pin
library feature, making the "pinning"APIs for manipulating pinned memory usable on stable.
(I've tried to write this proposal as a comprehensive "stabilization report.")
Stabilized feature or APIs
[std|core]::pin::Pin
This stabilizes the
Pin
type in thepin
submodule ofstd
/core
.Pin
isa fundamental, transparent wrapper around a generic type
P
, which is intendedto be a pointer type (for example,
Pin<&mut T>
andPin<Box<T>>
are bothvalid, intended constructs). The
Pin
wrapper modifies the pointer to "pin"the memory it refers to in place, preventing the user from moving objects out
of that memory.
The usual way to use the
Pin
type is to construct a pinned variant of somekind of owning pointer (
Box
,Rc
, etc). The std library owning pointers allprovide a
pinned
constructor which returns this. Then, to manipulate thevalue inside, all of these pointers provide a way to degrade toward
Pin<&T>
and
Pin<&mut T>
. Pinned pointers can deref, giving you back&T
, but cannotsafely mutably deref: this is only possible using the unsafe
get_mut
function.
As a result, anyone mutating data through a pin will be required to uphold the
invariant that they never move out of that data. This allows other code to
safely assume that the data is never moved, allowing it to contain (for
example) self references.
The
Pin
type will have these stabilized APIs:impl<P> Pin<P> where P: Deref, P::Target: Unpin
fn new(pointer: P) -> Pin<P>
impl<P> Pin<P> where P: Deref
unsafe fn new_unchecked(pointer: P) -> Pin<P>
fn as_ref(&self) -> Pin<&P::Target>
impl<P> Pin<P> where P: DerefMut
fn as_mut(&mut self) -> Pin<&mut P::Target>
fn set(&mut self, P::Target);
impl<'a, T: ?Sized> Pin<&'a T>
unsafe fn map_unchecked<U, F: FnOnce(&T) -> &U>(self, f: F) -> Pin<&'a U>
fn get_ref(self) -> &'a T
impl<'a, T: ?Sized> Pin<&'a mut T>
fn into_ref(self) -> Pin<&'a T>
unsafe fn get_unchecked_mut(self) -> &'a mut T
unsafe fn map_unchecked_mut<U, F: FnOnce(&mut T) -> &mut U>(self, f: F) -> Pin<&'a mut U>
impl<'a, T: ?Sized> Pin<&'a mut T> where T: Unpin
fn get_mut(self) -> &'a mut T
Trait impls
Most of the trait impls on
Pin
are fairly rote, these two are important toits operation:
impl<P: Deref> Deref for Pin<P> { type Target = P::Target }
impl<P: DerefMut> DerefMut for Pin<P> where P::Target: Unpin { }
std::marker::Unpin
Unpin is a safe auto trait which opts out of the guarantees of pinning. If the
target of a pinned pointer implements
Unpin
, it is safe to mutablydereference to it.
Unpin
types do not have any guarantees that they will notbe moved out of a
Pin
.This makes it as ergonomic to deal with a pinned reference to something that
does not contain self-references as it would be to deal with a non-pinned
reference. The guarantees of
Pin
only matter for special case types likeself-referential structures: those types do not implement
Unpin
, so they havethe restrictions of the
Pin
type.Notable implementations of
Unpin
in std:impl<'a, T: ?Sized> Unpin for &'a T
impl<'a, T: ?Sized> Unpin for &'a mut T
impl<T: ?Sized> Unpin for Box<T>
impl<T: ?Sized> Unpin for Rc<T>
impl<T: ?Sized> Unpin for Arc<T>
These codify the notion that pinnedness is not transitive across pointers. That
is, a
Pin<&T>
only pins the actual memory block represented byT
in aplace. Users have occassionally been confused by this and expected that a type
like
Pin<&mut Box<T>>
pins the data ofT
in place, but it only pins thememory the pinned reference actually refers to: in this case, the
Box
'srepresentation, which a pointer into the heap.
std::marker::Pinned
The
Pinned
type is a ZST which does not implementUnpin
; it allows you tosupress the auto-implementation of
Unpin
on stable, where!Unpin
implswould not be stable yet.
Smart pointer constructors
Constructors are added to the std smart pointers to create pinned references:
Box::pinned(data: T) -> Pin<Box<T>>
Rc::pinned(data: T) -> Pin<Rc<T>>
Arc::pinned(data: T) -> Pin<Arc<T>>
Notes on pinning & safety
Over the last 9 months the pinning APIs have gone through several iterations as
we have investigated their expressive power and also the soundness of their
guarantees. I would now say confidently that the pinning APIs stabilized here
are sound and close enough to the local maxima in ergonomics and
expressiveness; that is, ready for stabilization.
One of the trickier issues of pinning is determining when it is safe to perform
a pin projection: that is, to go from a
Pin<P<Target = Foo>>
to aPin<P<Target = Bar>>
, whereBar
is a field ofFoo
. Fortunately, we havebeen able to codify a set of rules which can help users determine if such a
projection is safe:
(Foo: Unpin) implies (Bar: Unpin)
: thatis, if it is never the case that
Foo
(the containing type) isUnpin
whileBar
(the projected type) is notUnpin
.Bar
is never moved during the destruction ofFoo
,meaning that either
Foo
has no destructor, or the destructor is carefullychecked to make sure that it never moves out of the field being projected to.
Foo
(the containing type) is notrepr(packed)
,because this causes fields to be moved around to realign them.
Additionally, the std APIs provide no safe way to pin objects to the stack.
This is because there is no way to implement that safely using a function API.
However, users can unsafely pin things to the stack by guaranteeing that they
never move the object again after creating the pinned reference.
The
pin-utils
crate on crates.io contains macros to assist with both stackpinning and pin projection. The stack pinning macro safely pins objects to the
stack using a trick involving shadowing, whereas a macro for projection exists
which is unsafe, but avoids you having to write the projection boilerplate in
which you could possibly introduce other incorrect unsafe code.
Implementation changes prior to stabilization
Unpin
from the prelude, removepin::Unpin
re-exportAs a general rule, we don't re-export things from multiple places in std unless
one is a supermodule of the real definition (e.g. shortening
std::collections::hash_map::HashMap
tostd::collections::HashMap
). For thisreason, the re-export of
std::marker::Unpin
fromstd::pin::Unpin
is out ofplace.
At the same time, other important marker traits like Send and Sync are included
in the prelude. So instead of re-exporting
Unpin
from thepin
module, byputting in the prelude we make it unnecessary to import
std::marker::Unpin
,the same reason it was put into
pin
.Currently, a lot of the associated function of
Pin
do not use method syntax.In theory, this is to avoid conflicting with derefable inner methods. However,
this rule has not been applied consistently, and in our experience has mostly
just made things more inconvenient. Pinned pointers only implement immutable
deref, not mutable deref or deref by value, limiting the ability to deref
anyway. Moreover, many of these names are fairly unique (e.g.
map_unchecked
)and unlikely to conflict.
Instead, we prefer to give the
Pin
methods their due precedence; users whoneed to access an interior method always can using UFCS, just as they would be
required to to access the Pin methods if we did not use method syntax.
get_mut_unchecked
toget_unchecked_mut
The current ordering is inconsistent with other uses in the standard library.
impl<P> Unpin for Pin<P>
This impl is not justified by our standard justification for unpin impls: there is no pointer direction between
Pin<P>
andP
. Its usefulness is covered by the impls for pointers themselves.This futures impl will need to change to add a
P: Unpin
bound.Pin
asrepr(transparent)
Pin should be a transparent wrapper around the pointer inside of it, with the same representation.
Connected features and larger milestones
The pin APIs are important to safely manipulating sections of memory which can
be guaranteed not to be moved out. If the objects in that memory do not
implement
Unpin
, their address will never change. This is necessary forcreating self-referential generators and asynchronous functions. As a result,
the
Pin
type appears in the standard libraryfuture
APIs and will soonappear in the APIs for generators as well (#55704).
Stabilizing the
Pin
type and its APIs is a necessary precursor to stabilizingthe
Future
APIs, which is itself a necessary precursor to stabilizing theasync/await
syntax and moving the entirefutures 0.3
async IO ecosystemonto stable Rust.
cc @cramertj @RalfJung
The text was updated successfully, but these errors were encountered: