Tracking Issue for pointer_bytes_offsets

[Gankra]

Feature gates:

#![feature(pointer_byte_offsets)]

#![feature(const_pointer_byte_offsets)]

This is a tracking issue for the pointer_byte_offsets raw pointer conveniences like ptr.byte_add(offset)

Public API

impl<T: ?Sized> *const T {
    // feature gates `pointer_byte_offsets` and `const_pointer_byte_offsets`
    pub const unsafe fn byte_offset(self, count: isize) -> Self;
    pub const unsafe fn byte_add(self, count: usize) -> Self;
    pub const unsafe fn byte_sub(self, count: usize) -> Self;

    pub const fn wrapping_byte_offset(self, count: isize) -> Self;
    pub const fn wrapping_byte_add(self, count: usize) -> Self;
    pub const fn wrapping_byte_sub(self, count: usize) -> Self;

    pub const unsafe fn byte_offset_from<U: ?Sized>(self, origin: *const U) -> isize;
}

// ... and the same for` *mut T`

Steps / History

• Implementation: Add convenience byte offset/check align functions to pointers #95643
• Making byte_offset_from more generic: Make pointer::byte_offset_from more generic #103489
• Final comment period (FCP): Tracking Issue for pointer_bytes_offsets #96283 (comment)
• Stabilization PR: Stabilize [const_]pointer_byte_offsets #116205

Unresolved Questions

• Should these operations actually accomodate DSTs? This seems deeply semantically dubious/broken but idk, you can plausibly use them right if you are extremely careful and understand the implications of stacked borrows for slices/projections.

[Gankra]

To clarify the open question's concerns a bit: if you make a ref-slice and then cast it to a raw-slice, offsetting the raw pointer will not change the slice's len, but you can't actually turn the offset slice back into a ref, because (at least under the strictest rules) the original ref-slice has provenance over only its exact range, and so derived pointers cannot access outside the range. Creating the offset ref-slice is trying to assert provenance over additional bytes you don't have permission to access.

Similarly, trait-object-refs will make you sad if you do that.

This can only be soundly used (meaningfully) if you from_raw_parts the raw-DST from a raw pointer with provenance over the whole "array" you want to stride through.

So like, in general you need to write something like (pseudocode of the general pattern):

let array = [...];

// Steal metadata from an example or make the metadata from whole cloth
let dst = &array[something];
let metadata = get_metadata(dst);

// Now build a raw DST from a pointer to the whole array
let raw_array_ptr = array.as_ptr();
let raw_dst = ptr::from_raw_parts(raw_array_ptr, metadata);

// Yay this actually can be strided through the whole array now
let next_dst = raw_dst.byte_add(offset_to_next_dst);

(CC @RalfJung since this is an interesting stacked borrows example, no need to reply if there's nothing more to say.)

[WaffleLapkin]

@Gankra actually, I think allowing these methods take pointers to DST unlocks a lot of abilities for custom DSTs. In particular byte_sub is required to implement Arc::from_raw (as it accepts a possibly fat pointer to T that needs to be offseted back to ArcInner<T>).

[WaffleLapkin]

I want to propose to stabilize these methods! (can someone open an fcp?)

Stabilization report

Implementation History

• Initially these methods were implemented in Add convenience byte offset/check align functions to pointers #95643
• The only unresolved question raised since then is whatever to allow calling these methods on pointers to unsized types. I propose to resolve this to "yes, we should actually", this will unlock a lot of options for working with unsized types (see expitience report for an example).

API Summary

impl *const T {
    pub const unsafe fn byte_offset(self, count: isize) -> Self;
    pub const unsafe fn byte_add(self, count: usize) -> Self;
    pub const unsafe fn byte_sub(self, count: usize) -> Self;

    pub const fn wrapping_byte_offset(self, count: isize) -> Self;
    pub const fn wrapping_byte_add(self, count: usize) -> Self;
    pub const fn wrapping_byte_sub(self, count: usize) -> Self;

    pub const unsafe fn byte_offset_from<U: ?Sized>(self, origin: *const U) -> isize;
}

// ... and the same for` *mut T`

Experience Report

#99113, #100819 and #100030 refactored some std code into using these methods simplifying the code.

These methods allow custom Arc implementations to work with unsized types just as well as std::arc::Arc, namely they allow to offset a pointer by <size of arc metadata> while keeping pointer metadata, which is required for example for Arc::from_raw.

This also allows to polyfil with_metadata_of by using wrapping_byte_offset, which is again very useful when working with DSTs.

[scottmcm]

Pondering: does byte_offset_from really need its argument to be a *const T? Arguably it doesn't care what type you pass it, so it could be *const U instead (or *const impl Sized, I suppose).

That would allow things like ptr::addr_of!(x.b).byte_offset_from(ptr::addr_of!(x)) to compile without needing an arbitrary choice of cast.

EDIT: this was updated in #103489

[dennis-hamester]

I have a question regarding the offset_from family of functions, stabilized or not. In all those cases where it is safe to use, why not use as integer casts, ignoring the fact that conversion from bytes to units of T might be required?

I just had the situation today, where I needed the offset between 2 bytes slices (one is a subslice of the other) and failed to see why I should use the unsafe offset_from. In either case, I have to be able to reason about why my code is correct, but not having to reason about why my code doesn't invoke UB seemed like a +1 for integer casts. Obviously this is just one example, and certainly not "all those cases".

But I'm wondering, what are safe examples in which I should use offset_from as opposed to integer casts?

[scottmcm]

offset_from (and particularly sub_ptr) allows LLVM to optimize more based on the unsafety. If your code is not a place where you get a material advantage from those extra optimizations, then by all means use the safe phrasings. (sub_ptr is very important for slice::Iter::len, but there's very little code that's anywhere close to as perf-critical as that.)

[RalfJung]

Another advantage of offset_from is that it can be used in const fn.

For some time we had safe variants of offset_from, maybe it is worth thinking about bringing them back.

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

[dtolnay]

We don't need to restart an FCP clock for this but I'd still like to be sure the team has seen the important discussion that took place after the 3 currently checked boxes. Thank you for raising this @tgross35. @rust-lang/libs-api: please have a look at #96283 (comment) which is an insightful breakdown.

I think the current set of names (see top of this issue) continue to be my first choice for word order.

@rfcbot poll libs-api Proceed with current word order?

[rfcbot]

Team member @dtolnay has asked teams: T-libs-api, for consensus on:

Proceed with current word order?

• @Amanieu
• @BurntSushi
• @dtolnay
• @joshtriplett
• @m-ou-se

[tgross35]

This didn’t yet finish the poll no? I don’t expect the results to change but it should be pretty easy to get the last checkbox

[m-ou-se]

I don't think it matters much, but I think of these methods as "byte-based addition" rather than "adding bytes", so that makes me slightly prefer byte_add.