A/V with from an unrestored MethodTable used in coreclr!JIT_NewArr1

[briansull]

Running the MusicStore App with everything crossgened I get this A/V:

Here is the stack trace:

:000> k
# Child-SP          RetAddr           Call Site
00 000000c3`1777cac0 00007ffc`8c286400 coreclr!MethodTable::IsFullyLoaded+0x29 [d:\fxkit\baseline\coreclr\src\vm\methodtable.h @ 1223] 
01 000000c3`1777cb00 00007ffc`ac63efca coreclr!JIT_NewArr1+0x290 [d:\fxkit\baseline\coreclr\src\vm\jithelpers.cpp @ 3155] 
02 000000c3`1777d030 00007ffc`ac63993e System.Collections.Generic.EnumerableHelpers.ToArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, Int32 ByRef)+0x7a
03 000000c3`1777d0b0 00007ffc`ac62fab8 System_Linq!System.Linq.Buffer`1[[System.__Canon, System.Private.CoreLib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>)+0x4e
04 000000c3`1777d100 00007ffc`cbef6420 System_Linq!System.Linq.Enumerable+ReverseIterator`1[[System.__Canon, System.Private.CoreLib]].MoveNext()+0x48
05 000000c3`1777d150 00007ffc`ae80173c Microsoft_Extensions_Configuration!Microsoft.Extensions.Configuration.ConfigurationRoot.get_Item(System.String)+0x50
06 000000c3`1777d1b0 00007ffc`2d2b1af2 Microsoft_AspNetCore_Hosting!Microsoft.AspNetCore.Hosting.WebHostBuilder..ctor()+0xec
07 000000c3`1777d1f0 00007ffc`8c94e523 MusicStore!MusicStore.Program.Main(System.String[])+0x112*** WARNING: Unable to verify checksum for D:\fxkit\JitBench\src\MusicStore\pubdir.new\MusicStore.dll

Here is the A/V location

First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
coreclr!MethodTable::IsFullyLoaded+0x29:
00007ffc`8c148729 8b00            mov     eax,dword ptr [rax] ds:00010000`00000000=????????

0:000> u coreclr!MethodTable::IsFullyLoaded l 12
coreclr!MethodTable::IsFullyLoaded [d:\fxkit\baseline\coreclr\src\vm\methodtable.h @ 1220]:
00007ffc`8c148700 48894c2408      mov     qword ptr [rsp+8],rcx
00007ffc`8c148705 4883ec38        sub     rsp,38h
00007ffc`8c148709 33c0            xor     eax,eax
00007ffc`8c14870b 85c0            test    eax,eax
00007ffc`8c14870d 7402            je      coreclr!MethodTable::IsFullyLoaded+0x11 (00007ffc`8c148711)
00007ffc`8c14870f eb37            jmp     coreclr!MethodTable::IsFullyLoaded+0x48 (00007ffc`8c148748)
00007ffc`8c148711 488b4c2440      mov     rcx,qword ptr [rsp+40h]
00007ffc`8c148716 e8d526efff      call    coreclr!MethodTable::IsPreRestored (00007ffc`8c03adf0)
00007ffc`8c14871b 85c0            test    eax,eax
00007ffc`8c14871d 751d            jne     coreclr!MethodTable::IsFullyLoaded+0x3c (00007ffc`8c14873c)
00007ffc`8c14871f 488b4c2440      mov     rcx,qword ptr [rsp+40h]
00007ffc`8c148724 e8d77ff5ff      call    coreclr!MethodTable::GetWriteableData (00007ffc`8c0a0700)
00007ffc`8c148729 8b00            mov     eax,dword ptr [rax]
00007ffc`8c14872b 83e040          and     eax,40h
00007ffc`8c14872e 85c0            test    eax,eax
00007ffc`8c148730 740a            je      coreclr!MethodTable::IsFullyLoaded+0x3c (00007ffc`8c14873c)
00007ffc`8c148732 c744242000000000 mov     dword ptr [rsp+20h],0
00007ffc`8c14873a eb08            jmp     coreclr!MethodTable::IsFullyLoaded+0x44 (00007ffc`8c148744)
0:000> r
rax=0001000000000000 rbx=0000000000000001 rcx=00007ffc8d315430
rdx=00007ffc2d387a7a rsi=00007ffc2d3f3898 rdi=000000c31777d130
rip=00007ffc8c148729 rsp=000000c31777cac0 rbp=000000c31777d0a0
r8=0000000000000000  r9=00007ffc8cfa8440 r10=000000c31777cb00
r11=00007ffc8bf00000 r12=0000000000000000 r13=0000000000000000
r14=0000027e15b33a18 r15=0000000000000000
iopl=0         nv up ei pl nz na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206
coreclr!MethodTable::IsFullyLoaded+0x29:
00007ffc`8c148729 8b00            mov     eax,dword ptr [rax] ds:00010000`00000000=????????
0:000> dv
           this = 0x00007ffc`2d387a7a
...
Switching the active frame to coreclr!JIT_NewArr1:
...
0:000> dv
                        pArrayMT = 0x00007ffc`2d387a7a
                        elemType = 0n32764 (No matching enumerant)
                 __pUnCException = 0x00000000`00000000
                __pUnCEntryFrame = 0x000000c3`1777cd28
             __fExceptionCatched = false
                  CURRENT_THREAD = 0x0000027b`13ade680
        CURRENT_THREAD_AVAILABLE = true
___PermitHelperMethodFrameState = class PermitHelperMethodFrameState
                   __helperframe = class FrameWithCookie<HelperMethodFrame>
___CompletedFCallTransitionState = class CompletedFCallTransitionState
       __haveCheckedRestoreState = 0n0
                        newArray = class OBJECTREF
                      alwaysZero = 0n0
___PermitHelperMethodFrameState = class PermitHelperMethodFrameState
                         arrayMT = 0x00007ffc`2d387a7a
                            size = 0n1
                    __fCallCheck = class FCallCheck
                     __lastError = 0xa5b8
         ___FCallTransitionState = class FCallTransitionState
                         __cache = 0x00000000`00000000
                            __me = 0x00000000`00000000
0:000> k
# Child-SP          RetAddr           Call Site
00 000000c3`1777cac0 00007ffc`8c286400 coreclr!MethodTable::IsFullyLoaded+0x29 [d:\fxkit\baseline\coreclr\src\vm\methodtable.h @ 1223] 
01 000000c3`1777cb00 00007ffc`ac63efca coreclr!JIT_NewArr1+0x290 [d:\fxkit\baseline\coreclr\src\vm\jithelpers.cpp @ 3155] 
02 000000c3`1777d030 00007ffc`ac63993e System.Collections.Generic.EnumerableHelpers.ToArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, Int32 ByRef)+0x7a
03 000000c3`1777d0b0 00007ffc`ac62fab8 System_Linq!System.Linq.Buffer`1[[System.__Canon, System.Private.CoreLib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>)+0x4e
04 000000c3`1777d100 00007ffc`cbef6420 System_Linq!System.Linq.Enumerable+ReverseIterator`1[[System.__Canon, System.Private.CoreLib]].MoveNext()+0x48
05 000000c3`1777d150 00007ffc`ae80173c Microsoft_Extensions_Configuration!Microsoft.Extensions.Configuration.ConfigurationRoot.get_Item(System.String)+0x50
06 000000c3`1777d1b0 00007ffc`2d2b1af2 Microsoft_AspNetCore_Hosting!Microsoft.AspNetCore.Hosting.WebHostBuilder..ctor()+0xec
07 000000c3`1777d1f0 00007ffc`8c94e523 MusicStore!MusicStore.Program.Main(System.String[])+0x112

[briansull]

It is likely a regression from dotnet/coreclr#12369
@ruben-ayrapetyan can you PTAL

@jkotas FYI

Note that the MethodTable is mis-aligned at a 0x2 odd address:

pArrayMT = 0x00007ffc`2d387a7a

[briansull]

The callsite in this Jitted generic method:

System.Collections.Generic.EnumerableHelpers.ToArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, Int32 ByRef)+0x7a

00007ffc`ac67efb2 85db            test    ebx,ebx
00007ffc`ac67efb4 7442            je      System.Collections.Generic.EnumerableHelpers.ToArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, Int32 ByRef)+0xa8 (00007ffc`ac67eff8)
00007ffc`ac67efb6 488bce          mov     rcx,rsi
00007ffc`ac67efb9 ff156143fbff    call    qword ptr [System_Linq+0x3320 (00007ffc`ac633320)]
00007ffc`ac67efbf 488bc8          mov     rcx,rax
00007ffc`ac67efc2 8bd3            mov     edx,ebx
>>> 00007ffc`ac67efc4 ff15b620fbff    call    qword ptr [System_Linq+0x1080 (00007ffc`ac631080)]
00007ffc`ac67efca 4c8bf8          mov     r15,rax

Note that

call qword ptr [System_Linq+0x1080 (00007ffc'ac631080)] actually calls to coreclr!JIT_NewArr1

Stopping at the start of coreclr!JIT_NewArr1:

Breakpoint 5 hit
coreclr!JIT_NewArr1:
00007ffc`8c286170 4889542410      mov     qword ptr [rsp+10h],rdx 

0:000> r
rax=00007ffc2d3a7a7a rbx=0000000000000001 rcx=00007ffc2d3a7a7a
rdx=0000000000000001 rsi=00007ffc2d413898 rdi=000000f237b7d1e0
rip=00007ffc8c286170 rsp=000000f237b7d0d8 rbp=000000f237b7d150
 r8=0000000000000f78  r9=0000000000000f78 r10=000000f237b7c790
r11=00007ffc8bf00000 r12=0000000000000000 r13=0000000000000000
r14=000002c3eceb3e98 r15=0000000000000000

We can see that ebx/rbx is 1

So we previously called:

00007ffc`ac67efb6 488bce          mov     rcx,rsi
00007ffc`ac67efb9 ff156143fbff    call    qword ptr [System_Linq+0x3320 (00007ffc`ac633320)]

This will call this stub/thunk to (possibly) load a new value for rcx:

0:000> dp 00007ffc`ae7a3320
00007ffc`ae7a3320  00007ffc`2d2d7898 00007ffc`ae7a570d

0:000> u 07ffc`2d2d7898
00007ffc`2d2d7898 4889c8          mov     rax,rcx
00007ffc`2d2d789b 488b4938        mov     rcx,qword ptr [rcx+38h]
00007ffc`2d2d789f 488b4918        mov     rcx,qword ptr [rcx+18h]
00007ffc`2d2d78a3 4885c9          test    rcx,rcx
00007ffc`2d2d78a6 7404            je      00007ffc`2d2d78ac
00007ffc`2d2d78a8 4889c8          mov     rax,rcx
00007ffc`2d2d78ab c3              ret

So dumping rsi:

0:000> dp 00007ffc2d413898 l 8
00007ffc`2d413898  00007ffc`2d40f0f8 00007ffc`2d413468
00007ffc`2d4138a8  00007ffc`2d40f0f8 00007ffc`2d413660
00007ffc`2d4138b8  00000000`00000000 002d0007`0300038d
00007ffc`2d4138c8  ffffffff`fffffef0 00007ffc`2d413830

So [rcx+0x38] is 00007ffc'2d413830

0:000> dp 00007ffc`2d413830 l 8
00007ffc`2d413830  00007ffc`2d39e458 00007ffc`2d39e908
00007ffc`2d413840  00007ffc`2d1d00c0 00007ffc`2d3a7a7a

resulting in: 00007ffc'2d3a7a7a

[jkotas]

This looks like a R2R versioning problem: The convention for READYTORUN_HELPER_NewArray have changed with the above change. It will crash if you run R2R images compiled against old runtime against new runtime.

I think we should keep the convention the same - for now at least:

• Add new JIT_NewArr1_R2R helper to jithelpers.cpp that still takes TypeDesc. It should fetch template MT from the typedesc and call JIT_NewArr1. The extra overhead should matter because of this is slow path anyway.
• Map READYTORUN_HELPER_NewArray to this JIT_NewArr1_R2R

[ruben-ayrapetyan]

@briansull, @jkotas,

I plan to open pull request with fix today.

[ruben-ayrapetyan]

Small failing test case:

using System;                                                                                                          
using System.Runtime.CompilerServices;                                                                                 
                                                                                                                       
namespace Application                                                                                                  
{                                                                                                                      
    public class Program                                                                                               
    {                                                                                                                  
        [MethodImpl(MethodImplOptions.NoInlining)]                                                                     
        static void f<T>()                                                                                           
        {                                                                                                              
            T [] arr = new T[1000];                                                                                      
        }                                                                                                              
                                                                                                                       
        public static void Main(string[] args)                                                                         
        {                                                                                                              
            f<object>();                                                                                             
        }                                                                                                              
    }                                                                                                                  
}

Dump:

G_M21697_IG01:        ; func=00, offs=000000H, size=0005H, gcrefRegs=00000000 {}, byrefRegs=00000000 {}, byref, nogc <-- Prolog IG

IN0006: 000000 push     rax
IN0007: 000001 mov      qword ptr [rsp], rdi

G_M21697_IG02:        ; offs=000005H, size=0015H, gcrefRegs=00000000 {}, byrefRegs=00000000 {}, byref

IN0001: 000005 call     [CORINFO_HELP_READYTORUN_GENERIC_HANDLE]
IN0002: 00000B mov      rdi, rax
IN0003: 00000E mov      esi, 0x3E8
IN0004: 000013 call     [CORINFO_HELP_NEWARR_1_DIRECT]
IN0005: 000019 nop      

G_M21697_IG03:        ; offs=00001AH, size=0005H, epilog, nogc, emitadd

IN0008: 00001A add      rsp, 8
IN0009: 00001E ret

Issue is in calling CORINFO_HELP_NEWARR_1_DIRECT.
Other new-array helpers seem to be working in R2R mode (several cases were checked for same runtime, older runtime - newer image, newer runtime - older image).
The CORINFO_HELP_NEWARR_1_DIRECT fails on same runtime, and also in "newer runtime - older image" case.

[briansull]

I did rebuild all of the native images using the latest build. So its more than just a version to version compat issue, I believe.

[briansull]

Fixed by
Implement JIT_NewArr1_R2R wrapper dotnet/coreclr#12475