Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possible issue with guard pages on Windows ARM64 #65058

Closed
MichalStrehovsky opened this issue Feb 9, 2022 · 8 comments · Fixed by #65088
Closed

Possible issue with guard pages on Windows ARM64 #65058

MichalStrehovsky opened this issue Feb 9, 2022 · 8 comments · Fixed by #65088
Assignees
@echesakov
Labels
arch-arm64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI bug

Comments

@MichalStrehovsky
Copy link
Member

I'm seeing an (unfortunately non-deterministic) access violation early in the NativeAOT-compiled System.Runtime.Tests startup on Windows ARM64:

>	System.Runtime.Tests.exe!S_P_CoreLib_System_IO_StreamWriter__Flush_0() Line 308	Unknown
 	System.Runtime.Tests.exe!System_Console_System_Console__CreateOutputWriter() Line 237	Unknown
 	System.Runtime.Tests.exe!System_Console_System_Console___get_Out_g__EnsureInitialized_26_0() Line 207	Unknown
 	System.Runtime.Tests.exe!System_Console_System_Console__WriteLine_12() Line 807	Unknown
 	System.Runtime.Tests.exe!System_Runtime_Tests_SingleFileTestRunner__Main() Line 26	Unknown
 	System.Runtime.Tests.exe!System_Runtime_Tests__Module___StartupCodeMain()	Unknown
 	System.Runtime.Tests.exe!wmain(int argc, wchar_t * * argv) Line 205	C++

The crash happens as the code is trying to touch an SP-relative address a "little bit" (less than a page) further away from the current and obviously written to SP:

00007FF751DB86FC B94403FF             ldr         wzr,[sp,#0x400]
Click here to see full disassembly - the problem instruction is highlighted 
        private void Flush(bool flushStream, bool flushEncoder)
        {
            // flushEncoder should be true at the end of the file and if
            // the user explicitly calls Flush (though not if AutoFlush is true).
            // This is required to flush any dangling characters from our UTF-7
            // and UTF-8 encoders.
            ThrowIfDisposed();
00007FF751DB85E0 D10103FF             sub         sp,sp,S_P_CoreLib_System_IO_StreamWriter__Flush_0 (00h)  
00007FF751DB85E4 A900D3F3             stp         x19,x20,[sp,#8]  
00007FF751DB85E8 A901DBF5             stp         x21,x22,[sp,#0x18]  
00007FF751DB85EC F90017F7             str         x23,[sp,#0x28]  
00007FF751DB85F0 A9037BFD             stp         fp,lr,[sp,#0x30]  
00007FF751DB85F4 9100C3FD             add         fp,sp,#0x30  
00007FF751DB85F8 D28D2903             mov         x3,#0x6948  
00007FF751DB85FC F2A40583             movk        x3,#0x202C,lsl #0x10  
00007FF751DB8600 F2CDEDA3             movk        x3,#0x6F6D,lsl #0x20  
00007FF751DB8604 F2E42DA3             movk        x3,#0x216D,lsl #0x30  
00007FF751DB8608 F81D03A3             stur        x3,[fp,#-0x30]  
00007FF751DB860C AA0003F3             mov         x19,x0  
00007FF751DB8610 2A0103F4             mov         w20,w1  
00007FF751DB8614 2A0203F5             mov         w21,w2  
00007FF751DB8618 39416E60             ldrb        w0,[x19,#0x5B]  
00007FF751DB861C 35001040             cbnz        w0,S_P_CoreLib_System_IO_StreamWriter__Flush_0+244h (07FF751DB8824h)  

            // Perf boost for Flush on non-dirty writers.
            if (_charPos == 0 && !flushStream && !flushEncoder)
00007FF751DB8620 B9405260             ldr         w0,[x19,#0x50]  
00007FF751DB8624 53001E81             uxtb        w1,w20  
00007FF751DB8628 2A010000             orr         w0,w0,w1  
00007FF751DB862C 53001EA1             uxtb        w1,w21  
00007FF751DB8630 2A010000             orr         w0,w0,w1  
00007FF751DB8634 35000200             cbnz        w0,S_P_CoreLib_System_IO_StreamWriter__Flush_0+94h (07FF751DB8674h)  
            {
                return;
00007FF751DB8638 D28D2909             mov         x9,#0x6948  
00007FF751DB863C F2A40589             movk        x9,#0x202C,lsl #0x10  
00007FF751DB8640 F2CDEDA9             movk        x9,#0x6F6D,lsl #0x20  
00007FF751DB8644 F2E42DA9             movk        x9,#0x216D,lsl #0x30  
00007FF751DB8648 F85D03AA             ldur        x10,[fp,#-0x30]  
00007FF751DB864C EB0A013F             cmp         x9,x10  
00007FF751DB8650 54000040             beq         S_P_CoreLib_System_IO_StreamWriter__Flush_0+78h (07FF751DB8658h)  
00007FF751DB8654 97DFCA9B             bl          RhpFallbackFailFast (07FF7515AB0C0h)  
00007FF751DB8658 D100C3BF             sub         sp,fp,#0x30  
00007FF751DB865C A9437BFD             ldp         fp,lr,[sp,#0x30]  
00007FF751DB8660 F94017F7             ldr         x23,[sp,#0x28]  
00007FF751DB8664 A941DBF5             ldp         x21,x22,[sp,#0x18]  
00007FF751DB8668 A940D3F3             ldp         x19,x20,[sp,#8]  
00007FF751DB866C 910103FF             add         sp,sp,#0x40  
00007FF751DB8670 D65F03C0             ret  
            }

            if (!_haveWrittenPreamble)
00007FF751DB8674 39416660             ldrb        w0,[x19,#0x59]  
00007FF751DB8678 35000220             cbnz        w0,S_P_CoreLib_System_IO_StreamWriter__Flush_0+0DCh (07FF751DB86BCh)  
            {
                _haveWrittenPreamble = true;
00007FF751DB867C 52800020             mov         w0,#1  
00007FF751DB8680 39016660             strb        w0,[x19,#0x59]  
00007FF751DB8684 F9401660             ldr         x0,[x19,#0x28]  
00007FF751DB8688 F9400001             ldr         x1,[x0]  
00007FF751DB868C F9402021             ldr         x1,[x1,#0x40]  
00007FF751DB8690 D63F0020             blr         x1  
00007FF751DB8694 AA0003E2             mov         x2,x0  
00007FF751DB8698 2A0103E3             mov         w3,w1  
                ReadOnlySpan<byte> preamble = _encoding.Preamble;
                if (preamble.Length > 0)
00007FF751DB869C 7100007F             cmp         w3,#0  
00007FF751DB86A0 540000ED             ble         S_P_CoreLib_System_IO_StreamWriter__Flush_0+0DCh (07FF751DB86BCh)  
                {
                    _stream.Write(preamble);
00007FF751DB86A4 F9401260             ldr         x0,[x19,#0x20]  
00007FF751DB86A8 AA0203E1             mov         x1,x2  
00007FF751DB86AC 2A0303E2             mov         w2,w3  
00007FF751DB86B0 F9400003             ldr         x3,[x0]  
00007FF751DB86B4 F940A063             ldr         x3,[x3,#0x140]  
00007FF751DB86B8 D63F0060             blr         x3  
                }
            }

            // For sufficiently small char data being flushed, try to encode to the stack.
            // For anything else, fall back to allocating the byte[] buffer.
            Span<byte> byteBuffer = stackalloc byte[0];
00007FF751DB86BC F9401E60             ldr         x0,[x19,#0x38]  
00007FF751DB86C0 B4000100             cbz         x0,S_P_CoreLib_System_IO_StreamWriter__Flush_0+100h (07FF751DB86E0h)  
            if (_byteBuffer is not null)
            {
                byteBuffer = _byteBuffer;
00007FF751DB86C4 B5000080             cbnz        x0,S_P_CoreLib_System_IO_StreamWriter__Flush_0+0F4h (07FF751DB86D4h)  
00007FF751DB86C8 AA1F03F6             mov         x22,xzr  
00007FF751DB86CC 2A1F03F7             mov         w23,wzr  
00007FF751DB86D0 14000003             b           S_P_CoreLib_System_IO_StreamWriter__Flush_0+0FCh (07FF751DB86DCh)  
00007FF751DB86D4 91004016             add         x22,x0,#0x10  
00007FF751DB86D8 B9400817             ldr         w23,[x0,#8]  
00007FF751DB86DC 1400001C             b           S_P_CoreLib_System_IO_StreamWriter__Flush_0+16Ch (07FF751DB874Ch)  
            }
            else
            {
                int maxBytesForCharPos = _encoding.GetMaxByteCount(_charPos);
                byteBuffer = maxBytesForCharPos <= 1024 ? // arbitrary threshold
00007FF751DB86E0 F9401660             ldr         x0,[x19,#0x28]  
00007FF751DB86E4 B9405261             ldr         w1,[x19,#0x50]  
00007FF751DB86E8 F9400002             ldr         x2,[x0]  
00007FF751DB86EC F940B442             ldr         x2,[x2,#0x168]  
00007FF751DB86F0 D63F0040             blr         x2  
00007FF751DB86F4 7110001F             cmp         w0,#0x400  
00007FF751DB86F8 540000CC             bgt         S_P_CoreLib_System_IO_StreamWriter__Flush_0+130h (07FF751DB8710h)  

////////////////////////////////////////////////////////////////////////////////////////////////////
////////////////////////////////////////////////////////////////////////////////////////////////////
////
////
////                               AV at the next instruction
////
////
////////////////////////////////////////////////////////////////////////////////////////////////////
////////////////////////////////////////////////////////////////////////////////////////////////////
00007FF751DB86FC B94403FF             ldr         wzr,[sp,#0x400]  

00007FF751DB8700 D11003FF             sub         sp,sp,#0x400  
00007FF751DB8704 910003F6             mov         x22,sp  
00007FF751DB8708 52808017             mov         w23,#0x400  
00007FF751DB870C 14000010             b           S_P_CoreLib_System_IO_StreamWriter__Flush_0+16Ch (07FF751DB874Ch)  
00007FF751DB8710 F9401660             ldr         x0,[x19,#0x28]  
00007FF751DB8714 F9402261             ldr         x1,[x19,#0x40]  
00007FF751DB8718 B9400821             ldr         w1,[x1,#8]  
00007FF751DB871C F9400002             ldr         x2,[x0]  
00007FF751DB8720 F940B442             ldr         x2,[x2,#0x168]  
00007FF751DB8724 D63F0040             blr         x2  
00007FF751DB8728 93407C01             sxtw        x1,w0  
00007FF751DB872C F000DD60             adrp        x0,__Array<System_ComponentModel_System_ComponentModel_IEditableObject>::`vftable'+40h (07FF753967000h)  
00007FF751DB8730 91304000             add         x0,x0,#0xC10  
00007FF751DB8734 97DFC140             bl          RhpNewArray (07FF7515A8C34h)  
00007FF751DB8738 9100E26E             add         x14,x19,#0x38  
00007FF751DB873C AA0003EF             mov         x15,x0  
00007FF751DB8740 94285230             bl          RhpAssignRefAVLocation (07FF7527CD000h)  
00007FF751DB8744 91004016             add         x22,x0,#0x10  
00007FF751DB8748 B9400817             ldr         w23,[x0,#8]  
00007FF751DB874C F9401A60             ldr         x0,[x19,#0x30]  
00007FF751DB8750 F9402261             ldr         x1,[x19,#0x40]  
00007FF751DB8754 B9405262             ldr         w2,[x19,#0x50]  
00007FF751DB8758 B50000A1             cbnz        x1,S_P_CoreLib_System_IO_StreamWriter__Flush_0+18Ch (07FF751DB876Ch)  
00007FF751DB875C 350006A2             cbnz        w2,S_P_CoreLib_System_IO_StreamWriter__Flush_0+250h (07FF751DB8830h)  
00007FF751DB8760 AA1F03E1             mov         x1,xzr  
00007FF751DB8764 2A1F03E2             mov         w2,wzr  
00007FF751DB8768 14000006             b           S_P_CoreLib_System_IO_StreamWriter__Flush_0+1A0h (07FF751DB8780h)  
00007FF751DB876C B9400823             ldr         w3,[x1,#8]  
00007FF751DB8770 2A0203E4             mov         w4,w2  
00007FF751DB8774 EB04007F             cmp         x3,x4  
00007FF751DB8778 540005C3             blo         S_P_CoreLib_System_IO_StreamWriter__Flush_0+250h (07FF751DB8830h)  
00007FF751DB877C 91004021             add         x1,x1,#0x10  
00007FF751DB8780 AA1603E3             mov         x3,x22  
00007FF751DB8784 2A1703E4             mov         w4,w23  
00007FF751DB8788 53001EA5             uxtb        w5,w21  
00007FF751DB878C F9400006             ldr         x6,[x0]  
00007FF751DB8790 F9402CC6             ldr         x6,[x6,#0x58]  
00007FF751DB8794 D63F00C0             blr         x6  
                    stackalloc byte[1024] :
                    (_byteBuffer = new byte[_encoding.GetMaxByteCount(_charBuffer.Length)]);
            }

            int count = _encoder.GetBytes(new ReadOnlySpan<char>(_charBuffer, 0, _charPos), byteBuffer, flushEncoder);
            _charPos = 0;
00007FF751DB8798 B900527F             str         wzr,[x19,#0x50]  
            if (count > 0)
00007FF751DB879C 7100001F             cmp         w0,#0  
00007FF751DB87A0 5400018D             ble         S_P_CoreLib_System_IO_StreamWriter__Flush_0+1F0h (07FF751DB87D0h)  
            {
                _stream.Write(byteBuffer.Slice(0, count));
00007FF751DB87A4 F9401263             ldr         x3,[x19,#0x20]  
00007FF751DB87A8 2A0003E1             mov         w1,w0  
00007FF751DB87AC 2A1703E2             mov         w2,w23  
00007FF751DB87B0 EB02003F             cmp         x1,x2  
00007FF751DB87B4 540003E8             bhi         S_P_CoreLib_System_IO_StreamWriter__Flush_0+250h (07FF751DB8830h)  
00007FF751DB87B8 AA1603E1             mov         x1,x22  
00007FF751DB87BC 2A0003E2             mov         w2,w0  
00007FF751DB87C0 AA0303E0             mov         x0,x3  
00007FF751DB87C4 F9400063             ldr         x3,[x3]  
00007FF751DB87C8 F940A063             ldr         x3,[x3,#0x140]  
00007FF751DB87CC D63F0060             blr         x3  
            }

            if (flushStream)
00007FF751DB87D0 72001E9F             tst         w20,#0xFF  
00007FF751DB87D4 540000A0             beq         S_P_CoreLib_System_IO_StreamWriter__Flush_0+208h (07FF751DB87E8h)  
            {
                _stream.Flush();
00007FF751DB87D8 F9401260             ldr         x0,[x19,#0x20]  
00007FF751DB87DC F9400001             ldr         x1,[x0]  
00007FF751DB87E0 F9405C21             ldr         x1,[x1,#0xB8]  
00007FF751DB87E4 D63F0020             blr         x1  
            }
        }
00007FF751DB87E8 D28D2909             mov         x9,#0x6948  
00007FF751DB87EC F2A40589             movk        x9,#0x202C,lsl #0x10  
00007FF751DB87F0 F2CDEDA9             movk        x9,#0x6F6D,lsl #0x20  
00007FF751DB87F4 F2E42DA9             movk        x9,#0x216D,lsl #0x30  
00007FF751DB87F8 F85D03AA             ldur        x10,[fp,#-0x30]  
00007FF751DB87FC EB0A013F             cmp         x9,x10  
00007FF751DB8800 54000040             beq         S_P_CoreLib_System_IO_StreamWriter__Flush_0+228h (07FF751DB8808h)  
00007FF751DB8804 97DFCA2F             bl          RhpFallbackFailFast (07FF7515AB0C0h)  
00007FF751DB8808 D100C3BF             sub         sp,fp,#0x30  
00007FF751DB880C A9437BFD             ldp         fp,lr,[sp,#0x30]  
00007FF751DB8810 F94017F7             ldr         x23,[sp,#0x28]  
00007FF751DB8814 A941DBF5             ldp         x21,x22,[sp,#0x18]  
00007FF751DB8818 A940D3F3             ldp         x19,x20,[sp,#8]  
00007FF751DB881C 910103FF             add         sp,sp,#0x40  
00007FF751DB8820 D65F03C0             ret  
        }

        private void Flush(bool flushStream, bool flushEncoder)
        {
            // flushEncoder should be true at the end of the file and if
            // the user explicitly calls Flush (though not if AutoFlush is true).
            // This is required to flush any dangling characters from our UTF-7
            // and UTF-8 encoders.
            ThrowIfDisposed();
00007FF751DB8824 AA1303E0             mov         x0,x19  
00007FF751DB8828 940005D2             bl          S_P_CoreLib_System_IO_StreamWriter___ThrowIfDisposed_g__ThrowObjectDisposedException_76_0 (07FF751DB9F70h)  
00007FF751DB882C D43E0000             brk         #0xF000  
            {
                _stream.Write(byteBuffer.Slice(0, count));
00007FF751DB8830 97FD31A8             bl          S_P_CoreLib_System_ThrowHelper__ThrowArgumentOutOfRangeException (07FF751D04ED0h)  
00007FF751DB8834 D43E0000             brk         #0xF000

The SP looks legit at the time of crash and the relative address seems to be on the next page but for some reason it doesn't appear to be triggering the guard page logic within the OS and we get an AV instead.

Looking at __chkstk implementation in the VC++ runtime it seems to be touching things at page-aligned boundaries instead of just blindly reaching into the middle of the next page 4 kB away from the current SP. Aligning to the page boundary looks like "unnecessary extra work" so maybe it has a meaning.

Crash dump, symbols, and EXE here: https://github.com/MichalStrehovsky/NativeAOT5/releases/download/vBlah/System.Runtime.Tests.zip
Note this is NativeAOT, so no SOS extensions needed, and the EXE is fully self contained and can be executed under debugger as-is. The issue is non-deterministic and happens before anything gets written to the console. Once you see console output, you missed it.
@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Feb 9, 2022
@MichalStrehovsky MichalStrehovsky added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI and removed area-NativeAOT-coreclr labels Feb 9, 2022
@ghost
Copy link

ghost commented Feb 9, 2022

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

I'm seeing an (unfortunately non-deterministic) access violation early in the NativeAOT-compiled System.Runtime.Tests startup on Windows ARM64:

>	System.Runtime.Tests.exe!S_P_CoreLib_System_IO_StreamWriter__Flush_0() Line 308	Unknown
 	System.Runtime.Tests.exe!System_Console_System_Console__CreateOutputWriter() Line 237	Unknown
 	System.Runtime.Tests.exe!System_Console_System_Console___get_Out_g__EnsureInitialized_26_0() Line 207	Unknown
 	System.Runtime.Tests.exe!System_Console_System_Console__WriteLine_12() Line 807	Unknown
 	System.Runtime.Tests.exe!System_Runtime_Tests_SingleFileTestRunner__Main() Line 26	Unknown
 	System.Runtime.Tests.exe!System_Runtime_Tests__Module___StartupCodeMain()	Unknown
 	System.Runtime.Tests.exe!wmain(int argc, wchar_t * * argv) Line 205	C++

The crash happens as the code is trying to touch an SP-relative address a "little bit" (less than a page) further away from the current and obviously written to SP:

00007FF751DB86FC B94403FF             ldr         wzr,[sp,#0x400]
Click here to see full disassembly - the problem instruction is highlighted 
        private void Flush(bool flushStream, bool flushEncoder)
        {
            // flushEncoder should be true at the end of the file and if
            // the user explicitly calls Flush (though not if AutoFlush is true).
            // This is required to flush any dangling characters from our UTF-7
            // and UTF-8 encoders.
            ThrowIfDisposed();
00007FF751DB85E0 D10103FF             sub         sp,sp,S_P_CoreLib_System_IO_StreamWriter__Flush_0 (00h)  
00007FF751DB85E4 A900D3F3             stp         x19,x20,[sp,#8]  
00007FF751DB85E8 A901DBF5             stp         x21,x22,[sp,#0x18]  
00007FF751DB85EC F90017F7             str         x23,[sp,#0x28]  
00007FF751DB85F0 A9037BFD             stp         fp,lr,[sp,#0x30]  
00007FF751DB85F4 9100C3FD             add         fp,sp,#0x30  
00007FF751DB85F8 D28D2903             mov         x3,#0x6948  
00007FF751DB85FC F2A40583             movk        x3,#0x202C,lsl #0x10  
00007FF751DB8600 F2CDEDA3             movk        x3,#0x6F6D,lsl #0x20  
00007FF751DB8604 F2E42DA3             movk        x3,#0x216D,lsl #0x30  
00007FF751DB8608 F81D03A3             stur        x3,[fp,#-0x30]  
00007FF751DB860C AA0003F3             mov         x19,x0  
00007FF751DB8610 2A0103F4             mov         w20,w1  
00007FF751DB8614 2A0203F5             mov         w21,w2  
00007FF751DB8618 39416E60             ldrb        w0,[x19,#0x5B]  
00007FF751DB861C 35001040             cbnz        w0,S_P_CoreLib_System_IO_StreamWriter__Flush_0+244h (07FF751DB8824h)  

            // Perf boost for Flush on non-dirty writers.
            if (_charPos == 0 && !flushStream && !flushEncoder)
00007FF751DB8620 B9405260             ldr         w0,[x19,#0x50]  
00007FF751DB8624 53001E81             uxtb        w1,w20  
00007FF751DB8628 2A010000             orr         w0,w0,w1  
00007FF751DB862C 53001EA1             uxtb        w1,w21  
00007FF751DB8630 2A010000             orr         w0,w0,w1  
00007FF751DB8634 35000200             cbnz        w0,S_P_CoreLib_System_IO_StreamWriter__Flush_0+94h (07FF751DB8674h)  
            {
                return;
00007FF751DB8638 D28D2909             mov         x9,#0x6948  
00007FF751DB863C F2A40589             movk        x9,#0x202C,lsl #0x10  
00007FF751DB8640 F2CDEDA9             movk        x9,#0x6F6D,lsl #0x20  
00007FF751DB8644 F2E42DA9             movk        x9,#0x216D,lsl #0x30  
00007FF751DB8648 F85D03AA             ldur        x10,[fp,#-0x30]  
00007FF751DB864C EB0A013F             cmp         x9,x10  
00007FF751DB8650 54000040             beq         S_P_CoreLib_System_IO_StreamWriter__Flush_0+78h (07FF751DB8658h)  
00007FF751DB8654 97DFCA9B             bl          RhpFallbackFailFast (07FF7515AB0C0h)  
00007FF751DB8658 D100C3BF             sub         sp,fp,#0x30  
00007FF751DB865C A9437BFD             ldp         fp,lr,[sp,#0x30]  
00007FF751DB8660 F94017F7             ldr         x23,[sp,#0x28]  
00007FF751DB8664 A941DBF5             ldp         x21,x22,[sp,#0x18]  
00007FF751DB8668 A940D3F3             ldp         x19,x20,[sp,#8]  
00007FF751DB866C 910103FF             add         sp,sp,#0x40  
00007FF751DB8670 D65F03C0             ret  
            }

            if (!_haveWrittenPreamble)
00007FF751DB8674 39416660             ldrb        w0,[x19,#0x59]  
00007FF751DB8678 35000220             cbnz        w0,S_P_CoreLib_System_IO_StreamWriter__Flush_0+0DCh (07FF751DB86BCh)  
            {
                _haveWrittenPreamble = true;
00007FF751DB867C 52800020             mov         w0,#1  
00007FF751DB8680 39016660             strb        w0,[x19,#0x59]  
00007FF751DB8684 F9401660             ldr         x0,[x19,#0x28]  
00007FF751DB8688 F9400001             ldr         x1,[x0]  
00007FF751DB868C F9402021             ldr         x1,[x1,#0x40]  
00007FF751DB8690 D63F0020             blr         x1  
00007FF751DB8694 AA0003E2             mov         x2,x0  
00007FF751DB8698 2A0103E3             mov         w3,w1  
                ReadOnlySpan<byte> preamble = _encoding.Preamble;
                if (preamble.Length > 0)
00007FF751DB869C 7100007F             cmp         w3,#0  
00007FF751DB86A0 540000ED             ble         S_P_CoreLib_System_IO_StreamWriter__Flush_0+0DCh (07FF751DB86BCh)  
                {
                    _stream.Write(preamble);
00007FF751DB86A4 F9401260             ldr         x0,[x19,#0x20]  
00007FF751DB86A8 AA0203E1             mov         x1,x2  
00007FF751DB86AC 2A0303E2             mov         w2,w3  
00007FF751DB86B0 F9400003             ldr         x3,[x0]  
00007FF751DB86B4 F940A063             ldr         x3,[x3,#0x140]  
00007FF751DB86B8 D63F0060             blr         x3  
                }
            }

            // For sufficiently small char data being flushed, try to encode to the stack.
            // For anything else, fall back to allocating the byte[] buffer.
            Span<byte> byteBuffer = stackalloc byte[0];
00007FF751DB86BC F9401E60             ldr         x0,[x19,#0x38]  
00007FF751DB86C0 B4000100             cbz         x0,S_P_CoreLib_System_IO_StreamWriter__Flush_0+100h (07FF751DB86E0h)  
            if (_byteBuffer is not null)
            {
                byteBuffer = _byteBuffer;
00007FF751DB86C4 B5000080             cbnz        x0,S_P_CoreLib_System_IO_StreamWriter__Flush_0+0F4h (07FF751DB86D4h)  
00007FF751DB86C8 AA1F03F6             mov         x22,xzr  
00007FF751DB86CC 2A1F03F7             mov         w23,wzr  
00007FF751DB86D0 14000003             b           S_P_CoreLib_System_IO_StreamWriter__Flush_0+0FCh (07FF751DB86DCh)  
00007FF751DB86D4 91004016             add         x22,x0,#0x10  
00007FF751DB86D8 B9400817             ldr         w23,[x0,#8]  
00007FF751DB86DC 1400001C             b           S_P_CoreLib_System_IO_StreamWriter__Flush_0+16Ch (07FF751DB874Ch)  
            }
            else
            {
                int maxBytesForCharPos = _encoding.GetMaxByteCount(_charPos);
                byteBuffer = maxBytesForCharPos <= 1024 ? // arbitrary threshold
00007FF751DB86E0 F9401660             ldr         x0,[x19,#0x28]  
00007FF751DB86E4 B9405261             ldr         w1,[x19,#0x50]  
00007FF751DB86E8 F9400002             ldr         x2,[x0]  
00007FF751DB86EC F940B442             ldr         x2,[x2,#0x168]  
00007FF751DB86F0 D63F0040             blr         x2  
00007FF751DB86F4 7110001F             cmp         w0,#0x400  
00007FF751DB86F8 540000CC             bgt         S_P_CoreLib_System_IO_StreamWriter__Flush_0+130h (07FF751DB8710h)  

////////////////////////////////////////////////////////////////////////////////////////////////////
////////////////////////////////////////////////////////////////////////////////////////////////////
////
////
////                               AV at the next instruction
////
////
////////////////////////////////////////////////////////////////////////////////////////////////////
////////////////////////////////////////////////////////////////////////////////////////////////////
00007FF751DB86FC B94403FF             ldr         wzr,[sp,#0x400]  

00007FF751DB8700 D11003FF             sub         sp,sp,#0x400  
00007FF751DB8704 910003F6             mov         x22,sp  
00007FF751DB8708 52808017             mov         w23,#0x400  
00007FF751DB870C 14000010             b           S_P_CoreLib_System_IO_StreamWriter__Flush_0+16Ch (07FF751DB874Ch)  
00007FF751DB8710 F9401660             ldr         x0,[x19,#0x28]  
00007FF751DB8714 F9402261             ldr         x1,[x19,#0x40]  
00007FF751DB8718 B9400821             ldr         w1,[x1,#8]  
00007FF751DB871C F9400002             ldr         x2,[x0]  
00007FF751DB8720 F940B442             ldr         x2,[x2,#0x168]  
00007FF751DB8724 D63F0040             blr         x2  
00007FF751DB8728 93407C01             sxtw        x1,w0  
00007FF751DB872C F000DD60             adrp        x0,__Array<System_ComponentModel_System_ComponentModel_IEditableObject>::`vftable'+40h (07FF753967000h)  
00007FF751DB8730 91304000             add         x0,x0,#0xC10  
00007FF751DB8734 97DFC140             bl          RhpNewArray (07FF7515A8C34h)  
00007FF751DB8738 9100E26E             add         x14,x19,#0x38  
00007FF751DB873C AA0003EF             mov         x15,x0  
00007FF751DB8740 94285230             bl          RhpAssignRefAVLocation (07FF7527CD000h)  
00007FF751DB8744 91004016             add         x22,x0,#0x10  
00007FF751DB8748 B9400817             ldr         w23,[x0,#8]  
00007FF751DB874C F9401A60             ldr         x0,[x19,#0x30]  
00007FF751DB8750 F9402261             ldr         x1,[x19,#0x40]  
00007FF751DB8754 B9405262             ldr         w2,[x19,#0x50]  
00007FF751DB8758 B50000A1             cbnz        x1,S_P_CoreLib_System_IO_StreamWriter__Flush_0+18Ch (07FF751DB876Ch)  
00007FF751DB875C 350006A2             cbnz        w2,S_P_CoreLib_System_IO_StreamWriter__Flush_0+250h (07FF751DB8830h)  
00007FF751DB8760 AA1F03E1             mov         x1,xzr  
00007FF751DB8764 2A1F03E2             mov         w2,wzr  
00007FF751DB8768 14000006             b           S_P_CoreLib_System_IO_StreamWriter__Flush_0+1A0h (07FF751DB8780h)  
00007FF751DB876C B9400823             ldr         w3,[x1,#8]  
00007FF751DB8770 2A0203E4             mov         w4,w2  
00007FF751DB8774 EB04007F             cmp         x3,x4  
00007FF751DB8778 540005C3             blo         S_P_CoreLib_System_IO_StreamWriter__Flush_0+250h (07FF751DB8830h)  
00007FF751DB877C 91004021             add         x1,x1,#0x10  
00007FF751DB8780 AA1603E3             mov         x3,x22  
00007FF751DB8784 2A1703E4             mov         w4,w23  
00007FF751DB8788 53001EA5             uxtb        w5,w21  
00007FF751DB878C F9400006             ldr         x6,[x0]  
00007FF751DB8790 F9402CC6             ldr         x6,[x6,#0x58]  
00007FF751DB8794 D63F00C0             blr         x6  
                    stackalloc byte[1024] :
                    (_byteBuffer = new byte[_encoding.GetMaxByteCount(_charBuffer.Length)]);
            }

            int count = _encoder.GetBytes(new ReadOnlySpan<char>(_charBuffer, 0, _charPos), byteBuffer, flushEncoder);
            _charPos = 0;
00007FF751DB8798 B900527F             str         wzr,[x19,#0x50]  
            if (count > 0)
00007FF751DB879C 7100001F             cmp         w0,#0  
00007FF751DB87A0 5400018D             ble         S_P_CoreLib_System_IO_StreamWriter__Flush_0+1F0h (07FF751DB87D0h)  
            {
                _stream.Write(byteBuffer.Slice(0, count));
00007FF751DB87A4 F9401263             ldr         x3,[x19,#0x20]  
00007FF751DB87A8 2A0003E1             mov         w1,w0  
00007FF751DB87AC 2A1703E2             mov         w2,w23  
00007FF751DB87B0 EB02003F             cmp         x1,x2  
00007FF751DB87B4 540003E8             bhi         S_P_CoreLib_System_IO_StreamWriter__Flush_0+250h (07FF751DB8830h)  
00007FF751DB87B8 AA1603E1             mov         x1,x22  
00007FF751DB87BC 2A0003E2             mov         w2,w0  
00007FF751DB87C0 AA0303E0             mov         x0,x3  
00007FF751DB87C4 F9400063             ldr         x3,[x3]  
00007FF751DB87C8 F940A063             ldr         x3,[x3,#0x140]  
00007FF751DB87CC D63F0060             blr         x3  
            }

            if (flushStream)
00007FF751DB87D0 72001E9F             tst         w20,#0xFF  
00007FF751DB87D4 540000A0             beq         S_P_CoreLib_System_IO_StreamWriter__Flush_0+208h (07FF751DB87E8h)  
            {
                _stream.Flush();
00007FF751DB87D8 F9401260             ldr         x0,[x19,#0x20]  
00007FF751DB87DC F9400001             ldr         x1,[x0]  
00007FF751DB87E0 F9405C21             ldr         x1,[x1,#0xB8]  
00007FF751DB87E4 D63F0020             blr         x1  
            }
        }
00007FF751DB87E8 D28D2909             mov         x9,#0x6948  
00007FF751DB87EC F2A40589             movk        x9,#0x202C,lsl #0x10  
00007FF751DB87F0 F2CDEDA9             movk        x9,#0x6F6D,lsl #0x20  
00007FF751DB87F4 F2E42DA9             movk        x9,#0x216D,lsl #0x30  
00007FF751DB87F8 F85D03AA             ldur        x10,[fp,#-0x30]  
00007FF751DB87FC EB0A013F             cmp         x9,x10  
00007FF751DB8800 54000040             beq         S_P_CoreLib_System_IO_StreamWriter__Flush_0+228h (07FF751DB8808h)  
00007FF751DB8804 97DFCA2F             bl          RhpFallbackFailFast (07FF7515AB0C0h)  
00007FF751DB8808 D100C3BF             sub         sp,fp,#0x30  
00007FF751DB880C A9437BFD             ldp         fp,lr,[sp,#0x30]  
00007FF751DB8810 F94017F7             ldr         x23,[sp,#0x28]  
00007FF751DB8814 A941DBF5             ldp         x21,x22,[sp,#0x18]  
00007FF751DB8818 A940D3F3             ldp         x19,x20,[sp,#8]  
00007FF751DB881C 910103FF             add         sp,sp,#0x40  
00007FF751DB8820 D65F03C0             ret  
        }

        private void Flush(bool flushStream, bool flushEncoder)
        {
            // flushEncoder should be true at the end of the file and if
            // the user explicitly calls Flush (though not if AutoFlush is true).
            // This is required to flush any dangling characters from our UTF-7
            // and UTF-8 encoders.
            ThrowIfDisposed();
00007FF751DB8824 AA1303E0             mov         x0,x19  
00007FF751DB8828 940005D2             bl          S_P_CoreLib_System_IO_StreamWriter___ThrowIfDisposed_g__ThrowObjectDisposedException_76_0 (07FF751DB9F70h)  
00007FF751DB882C D43E0000             brk         #0xF000  
            {
                _stream.Write(byteBuffer.Slice(0, count));
00007FF751DB8830 97FD31A8             bl          S_P_CoreLib_System_ThrowHelper__ThrowArgumentOutOfRangeException (07FF751D04ED0h)  
00007FF751DB8834 D43E0000             brk         #0xF000

The SP looks legit at the time of crash and the relative address seems to be on the next page but for some reason it doesn't appear to be triggering the guard page logic within the OS and we get an AV instead.

Looking at __chkstk implementation in the VC++ runtime it seems to be touching things at page-aligned boundaries instead of just blindly reaching into the middle of the next page 4 kB away from the current SP. Aligning to the page boundary looks like "unnecessary extra work" so maybe it has a meaning.

Crash dump, symbols, and EXE here: https://github.com/MichalStrehovsky/NativeAOT5/releases/download/vBlah/System.Runtime.Tests.zip
Note this is NativeAOT, so no SOS extensions needed, and the EXE is fully self contained and can be executed under debugger as-is. The issue is non-deterministic and happens before anything gets written to the console. Once you see console output, you missed it.

Author: MichalStrehovsky
Assignees: -
Labels:

area-CodeGen-coreclr, untriaged
Milestone: -

@jkotas
Copy link
Member

jkotas commented Feb 9, 2022

ldr wzr,[sp,#0x400]

This is accessing memory at sp plus 0x400. Should it be sp minus 0x400 instead?

@MichalStrehovsky
Copy link
Member Author

This is accessing memory at sp plus 0x400. Should it be sp minus 0x400 instead?

Oh yeah, that's odd. The next instruction is:

00007FF751DB86FC B94403FF             ldr         wzr,[sp,#0x400]  
00007FF751DB8700 D11003FF             sub         sp,sp,#0x400

So we probably wanted a minus.

@EgorBo

This comment was marked as off-topic.

Sign in to view
@EgorBo

This comment was marked as off-topic.

Sign in to view
@EgorBo EgorBo self-assigned this Feb 9, 2022
@EgorBo
Copy link
Member

EgorBo commented Feb 9, 2022

Ah actually since it's a stack probe we better forward it to @echesakovMSFT

@EgorBo EgorBo assigned echesakov and unassigned EgorBo Feb 9, 2022
@echesakov
Copy link
Contributor

There is a bug in #64481 I recently merged - I will fix it in a moment.

@echesakov echesakov mentioned this issue Feb 9, 2022
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Feb 9, 2022
@echesakov echesakov added arch-arm64 bug and removed untriaged New issue has not been triaged by the area owner labels Feb 9, 2022
@echesakov
Copy link
Contributor

This is accessing memory at sp plus 0x400. Should it be sp minus 0x400 instead?

In fact, it should be at [sp]. There is a chance that [sp] already points at a guard page and [sp,#-0x400] to a reserved page below that.
This is what we did before #64481 where I accidentally changed the logic to touch [sp, #amount] instead.

@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Feb 9, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Mar 12, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@echesakov echesakov

Labels
arch-arm64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

For LclHeap touch a page at [originalSp] not at [originalSp+amount] echesakov/runtime
4 participants
@EgorBo @echesakov @jkotas @MichalStrehovsky