Inconsistent vtable layout with projections in supertrait bounds, making upcasting unsound #135315
Assignees
Labels
A-coercions
Area: implicit and explicit `expr as Type` coercions
A-trait-objects
Area: trait objects, vtable layout
C-bug
Category: This is a bug.
F-trait_upcasting
`#![feature(trait_upcasting)]`
I-unsound
Issue: A soundness hole (worst kind of bug), see: https://en.wikipedia.org/wiki/Soundness
requires-nightly
This issue requires a nightly compiler in some way.
T-compiler
Relevant to the compiler team, which will review and decide on the PR/issue.
T-types
Relevant to the types team, which will review and decide on the PR/issue.
Comments
rustbot
added
needs-triage
|
yeah im already looking at it |
saethlin
removed
the
needs-triage
lcnr
added
the
T-types
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jan 29, 2025
Fix deduplication mismatches in vtables leading to upcasting unsoundness We currently have two cases where subtleties in supertraits can trigger disagreements in the vtable layout, e.g. leading to a different vtable layout being accessed at a callsite compared to what was prepared during unsizing. Namely: ### rust-lang#135315 In this example, we were not normalizing supertraits when preparing vtables. In the example, ``` trait Supertrait<T> { fn _print_numbers(&self, mem: &[usize; 100]) { println!("{mem:?}"); } } impl<T> Supertrait<T> for () {} trait Identity { type Selff; } impl<Selff> Identity for Selff { type Selff = Selff; } trait Middle<T>: Supertrait<()> + Supertrait<T> { fn say_hello(&self, _: &usize) { println!("Hello!"); } } impl<T> Middle<T> for () {} trait Trait: Middle<<() as Identity>::Selff> {} impl Trait for () {} fn main() { (&() as &dyn Trait as &dyn Middle<()>).say_hello(&0); } ``` When we prepare `dyn Trait`, we see a supertrait of `Middle<<() as Identity>::Selff>`, which itself has two supertraits `Supertrait<()>` and `Supertrait<<() as Identity>::Selff>`. These two supertraits are identical, but they are not duplicated because we were using structural equality and *not* considering normalization. This leads to a vtable layout with two trait pointers. When we upcast to `dyn Middle<()>`, those two supertraits are now the same, leading to a vtable layout with only one trait pointer. This leads to an offset error, and we call the wrong method. ### rust-lang#135316 This one is a bit more interesting, and is the bulk of the changes in this PR. It's a bit similar, except it uses binder equality instead of normalization to make the compiler get confused about two vtable layouts. In the example, ``` trait Supertrait<T> { fn _print_numbers(&self, mem: &[usize; 100]) { println!("{mem:?}"); } } impl<T> Supertrait<T> for () {} trait Trait<T, U>: Supertrait<T> + Supertrait<U> { fn say_hello(&self, _: &usize) { println!("Hello!"); } } impl<T, U> Trait<T, U> for () {} fn main() { (&() as &'static dyn for<'a> Trait<&'static (), &'a ()> as &'static dyn Trait<&'static (), &'static ()>) .say_hello(&0); } ``` When we prepare the vtable for `dyn for<'a> Trait<&'static (), &'a ()>`, we currently consider the PolyTraitRef of the vtable as the key for a supertrait. This leads two two supertraits -- `Supertrait<&'static ()>` and `for<'a> Supertrait<&'a ()>`. However, we can upcast[^up] without offsetting the vtable from `dyn for<'a> Trait<&'static (), &'a ()>` to `dyn Trait<&'static (), &'static ()>`. This is just instantiating the principal trait ref for a specific `'a = 'static`. However, when considering those supertraits, we now have only one distinct supertrait -- `Supertrait<&'static ()>` (which is deduplicated since there are two supertraits with the same substitutions). This leads to similar offsetting issues, leading to the wrong method being called. [^up]: I say upcast but this is a cast that is allowed on stable, since it's not changing the vtable at all, just instantiating the binder of the principal trait ref for some lifetime. The solution here is to recognize that a vtable isn't really meaningfully higher ranked, and to just treat a vtable as corresponding to a `TraitRef` so we can do this deduplication more faithfully. That is to say, the vtable for `dyn for<'a> Tr<'a>` and `dyn Tr<'x>` are always identical, since they both would correspond to a set of free regions on an impl... Do note that `Tr<for<'a> fn(&'a ())>` and `Tr<fn(&'static ())>` are still distinct. ---- There's a bit more that can be cleaned up. In codegen, we can stop using `PolyExistentialTraitRef` basically everywhere. We can also fix SMIR to stop storing `PolyExistentialTraitRef` in its vtable allocations. As for testing, it's difficult to actually turn this into something that can be tested with `rustc_dump_vtable`, since having multiple supertraits that are identical is a recipe for ambiguity errors. Maybe someone else is more creative with getting that attr to work, since the tests I added being run-pass tests is a bit unsatisfying. Miri also doesn't help here, since it doesn't really generate vtables that are offset by an index in the same way as codegen. r? `@lcnr` for the vibe check? Or reassign, idk. Maybe let's talk about whether this makes sense. <sup>(I guess an alternative would also be to not do any deduplication of vtable supertraits (or only a really conservative subset) rather than trying to normalize and deduplicate more faithfully here. Not sure if that works and is sufficient tho.)</sup> cc `@steffahn` -- ty for the minimizations cc `@WaffleLapkin` -- since you're overseeing the feature stabilization :3 Fixes rust-lang#135315 Fixes rust-lang#135316
github-project-automation
bot
moved this from unblocked
to new solver everywhere
in T-types unsound issues
bjorn3
pushed a commit
to bjorn3/rust
that referenced
this issue
Feb 7, 2025
Fix deduplication mismatches in vtables leading to upcasting unsoundness We currently have two cases where subtleties in supertraits can trigger disagreements in the vtable layout, e.g. leading to a different vtable layout being accessed at a callsite compared to what was prepared during unsizing. Namely: ### rust-lang#135315 In this example, we were not normalizing supertraits when preparing vtables. In the example, ``` trait Supertrait<T> { fn _print_numbers(&self, mem: &[usize; 100]) { println!("{mem:?}"); } } impl<T> Supertrait<T> for () {} trait Identity { type Selff; } impl<Selff> Identity for Selff { type Selff = Selff; } trait Middle<T>: Supertrait<()> + Supertrait<T> { fn say_hello(&self, _: &usize) { println!("Hello!"); } } impl<T> Middle<T> for () {} trait Trait: Middle<<() as Identity>::Selff> {} impl Trait for () {} fn main() { (&() as &dyn Trait as &dyn Middle<()>).say_hello(&0); } ``` When we prepare `dyn Trait`, we see a supertrait of `Middle<<() as Identity>::Selff>`, which itself has two supertraits `Supertrait<()>` and `Supertrait<<() as Identity>::Selff>`. These two supertraits are identical, but they are not duplicated because we were using structural equality and *not* considering normalization. This leads to a vtable layout with two trait pointers. When we upcast to `dyn Middle<()>`, those two supertraits are now the same, leading to a vtable layout with only one trait pointer. This leads to an offset error, and we call the wrong method. ### rust-lang#135316 This one is a bit more interesting, and is the bulk of the changes in this PR. It's a bit similar, except it uses binder equality instead of normalization to make the compiler get confused about two vtable layouts. In the example, ``` trait Supertrait<T> { fn _print_numbers(&self, mem: &[usize; 100]) { println!("{mem:?}"); } } impl<T> Supertrait<T> for () {} trait Trait<T, U>: Supertrait<T> + Supertrait<U> { fn say_hello(&self, _: &usize) { println!("Hello!"); } } impl<T, U> Trait<T, U> for () {} fn main() { (&() as &'static dyn for<'a> Trait<&'static (), &'a ()> as &'static dyn Trait<&'static (), &'static ()>) .say_hello(&0); } ``` When we prepare the vtable for `dyn for<'a> Trait<&'static (), &'a ()>`, we currently consider the PolyTraitRef of the vtable as the key for a supertrait. This leads two two supertraits -- `Supertrait<&'static ()>` and `for<'a> Supertrait<&'a ()>`. However, we can upcast[^up] without offsetting the vtable from `dyn for<'a> Trait<&'static (), &'a ()>` to `dyn Trait<&'static (), &'static ()>`. This is just instantiating the principal trait ref for a specific `'a = 'static`. However, when considering those supertraits, we now have only one distinct supertrait -- `Supertrait<&'static ()>` (which is deduplicated since there are two supertraits with the same substitutions). This leads to similar offsetting issues, leading to the wrong method being called. [^up]: I say upcast but this is a cast that is allowed on stable, since it's not changing the vtable at all, just instantiating the binder of the principal trait ref for some lifetime. The solution here is to recognize that a vtable isn't really meaningfully higher ranked, and to just treat a vtable as corresponding to a `TraitRef` so we can do this deduplication more faithfully. That is to say, the vtable for `dyn for<'a> Tr<'a>` and `dyn Tr<'x>` are always identical, since they both would correspond to a set of free regions on an impl... Do note that `Tr<for<'a> fn(&'a ())>` and `Tr<fn(&'static ())>` are still distinct. ---- There's a bit more that can be cleaned up. In codegen, we can stop using `PolyExistentialTraitRef` basically everywhere. We can also fix SMIR to stop storing `PolyExistentialTraitRef` in its vtable allocations. As for testing, it's difficult to actually turn this into something that can be tested with `rustc_dump_vtable`, since having multiple supertraits that are identical is a recipe for ambiguity errors. Maybe someone else is more creative with getting that attr to work, since the tests I added being run-pass tests is a bit unsatisfying. Miri also doesn't help here, since it doesn't really generate vtables that are offset by an index in the same way as codegen. r? `@lcnr` for the vibe check? Or reassign, idk. Maybe let's talk about whether this makes sense. <sup>(I guess an alternative would also be to not do any deduplication of vtable supertraits (or only a really conservative subset) rather than trying to normalize and deduplicate more faithfully here. Not sure if that works and is sufficient tho.)</sup> cc `@steffahn` -- ty for the minimizations cc `@WaffleLapkin` -- since you're overseeing the feature stabilization :3 Fixes rust-lang#135315 Fixes rust-lang#135316
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
(playground)
example output (shortened):
so apparently, we're calling
_print_numbersactually, because the vtable ofdyn Traitanddyn Middle<()>aren't compatible.@rustbot label F-trait_upcasting, I-unsound, T-compiler, requires-nightly, A-trait-objects, A-coercions
The text was updated successfully, but these errors were encountered: