Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mmap() should handle collisions with areas mapped with mmu::linear_map() #1135

Closed
wkozaczuk opened this issue May 6, 2021 · 1 comment
Closed

mmap() should handle collisions with areas mapped with mmu::linear_map() #1135

wkozaczuk opened this issue May 6, 2021 · 1 comment

Comments

@wkozaczuk
Copy link
Collaborator

While investigating crash when trying to run Node.JS on OSv on aarch64 (this problem does not seem to be arch specific), I have come to realize that OSv mmap() does not check for any potential collisions with areas of memory mapped with mmu::linear_map().

For example, is case of this crash with extra debug info added one can see clear collision:

OSv v0.55.0-248-g504b5423
linear_map: mapped from: ffff8000407bc000
                     to: ffff8000c0000000
linear_map: mapped from: ffff9000407bc000
                     to: ffff9000c0000000
linear_map: mapped from: ffffa000407bc000
                     to: ffffa000c0000000
linear_map: mapped from: 0000000040000000
                     to: 00000000407bc000
linear_map: mapped from: 0000000009000000
                     to: 0000000009001000
linear_map: mapped from: 0000000008000000
                     to: 0000000008010000
linear_map: mapped from: 0000000008010000
                     to: 0000000008020000
linear_map: mapped from: 0000004010000000
                     to: 0000004020000000
linear_map: mapped from: 000000003eff0000
                     to: 000000003f000000
linear_map: mapped from: 0000000010000000
                     to: 000000003eff0000
linear_map: mapped from: 0000000009010000
                     to: 0000000009011000
getauxval() stubbed
PSCI: version 65536.0 detected.
2 CPUs detected
getauxval() stubbed
Firmware vendor: Unknown
ELF [tid:0, mod:0, ]: The base set to: 000000000000000000 and end: 0x00000000407bab50
bsd: initializing - done
VFS: mounting ramfs at /
VFS: mounting devfs at /dev
net: initializing - done
linear_map: mapped from: ffff80000a000000
                     to: ffff80000a000200
Version 1 not supported!
linear_map: mapped from: ffff80000a000200
                     to: ffff80000a000400
Version 1 not supported!
linear_map: mapped from: ffff80000a000400
                     to: ffff80000a000600
Version 1 not supported!
linear_map: mapped from: ffff80000a000600
                     to: ffff80000a000800
Version 1 not supported!
linear_map: mapped from: ffff80000a000800
                     to: ffff80000a000a00
Version 1 not supported!
linear_map: mapped from: ffff80000a000a00
                     to: ffff80000a000c00
Version 1 not supported!
linear_map: mapped from: ffff80000a000c00
                     to: ffff80000a000e00
Version 1 not supported!
linear_map: mapped from: ffff80000a000e00
                     to: ffff80000a001000
Version 1 not supported!
eth0: ethernet address: 52:54:00:12:34:56
virtio-blk: Add blk device instances 0 as vblk0, devsize=72177664
random: virtio-rng registered as a source.
random: <Software, Yarrow> initialized
VFS: unmounting /dev
VFS: mounting rofs at /rofs
VFS: mounting devfs at /dev
VFS: mounting procfs at /proc
VFS: mounting sysfs at /sys
VFS: mounting ramfs at /tmp
[I/31 dhcp]: Broadcasting DHCPDISCOVER message with xid: [870938799]
[I/31 dhcp]: Waiting for IP...
[I/39 dhcp]: Received DHCPOFFER message from DHCP server: 192.168.122.1 regarding offerred IP address: 192.168.122.15
[I/39 dhcp]: Broadcasting DHCPREQUEST message with xid: [870938799] to SELECT offered IP: 192.168.122.15
[I/39 dhcp]: Received DHCPACK message from DHCP server: 192.168.122.1 regarding offerred IP address: 192.168.122.15
[I/39 dhcp]: Server acknowledged IP 192.168.122.15 for interface eth0 with time to lease in seconds: 86400
eth0: 192.168.122.15
[I/39 dhcp]: Configuring eth0: ip 192.168.122.15 subnet mask 255.255.255.0 gateway 192.168.122.1 MTU 1500
Booted up in 93.50 ms
Cmdline: /node !
ELF [tid:31, mod:1, /libvdso.so]: The base set to: 0x0000100000000000 and end: 0x0000100000011018
ELF [tid:31, mod:2, /node]: The base set to: 0x0000100000020000 and end: 0x0000100000032018
ELF [tid:31, mod:3, /usr/lib/libnode.so.72]: The base set to: 0x0000100000040000 and end: 0x000010000211e8d8
ELF [tid:31, mod:4, /usr/lib/libz.so.1]: The base set to: 0x0000100002120000 and end: 0x00001000021490b0
ELF [tid:31, mod:5, /usr/lib/libbrotlidec.so.1]: The base set to: 0x0000100002150000 and end: 0x000010000216a068
ELF [tid:31, mod:6, /usr/lib/libbrotlicommon.so.1]: The base set to: 0x0000100002170000 and end: 0x00001000021a0030
ELF [tid:31, mod:7, /usr/lib/libbrotlienc.so.1]: The base set to: 0x00001000021b0000 and end: 0x0000100002244070
ELF [tid:31, mod:8, /usr/lib/libcares.so.2]: The base set to: 0x0000100002250000 and end: 0x00001000022740f0
ELF [tid:31, mod:9, /usr/lib/libnghttp2.so.14]: The base set to: 0x0000100002280000 and end: 0x00001000022b70d0
ELF [tid:31, mod:10, /usr/lib/libcrypto.so.1.1]: The base set to: 0x00001000022c0000 and end: 0x0000100002554c68
ELF [tid:31, mod:11, /usr/lib/libssl.so.1.1]: The base set to: 0x0000100002560000 and end: 0x00001000025fa898
ELF [tid:31, mod:12, /usr/lib/libicui18n.so.67]: The base set to: 0x0000100002600000 and end: 0x0000100002904250
ELF [tid:31, mod:13, /usr/lib/libicuuc.so.67]: The base set to: 0x0000100002910000 and end: 0x0000100002b04140
ELF [tid:31, mod:14, /usr/lib/libicudata.so.67]: The base set to: 0x0000100002b10000 and end: 0x0000100004636010
ELF [tid:31, mod:15, /usr/lib/libgcc_s.so.1]: The base set to: 0x0000100004640000 and end: 0x0000100004664458
--> mmap MAP_ANONYMOUS: trying to map from: 0x36e0d000 to: 0x3ee0d000
--> mmap MAP_ANONYMOUS: trying to map from: 0x42d00000 to: 0x42d7f000
--> mmap MAP_ANONYMOUS: trying to map from: 0x3cd00000 to: 0x3cd7f000

error: kvm run failed Function not implemented
 PC=0000100000afb694 X00=0000000000000001 X01=0000000000001000
X02=000000003ee40118 X03=000000003ee80000 X04=0000000000040000
X05=0000000000000000 X06=ffffa000426ff4d8 X07=0000200000700310
X08=00000000000092d8 X09=0000000000000080 X10=0000000000001728
X11=0000200000700150 X12=000000003ee80000 X13=0000000000000000
X14=0000100002481448 X15=0000000000000000 X16=00001000020d5a70
X17=0000100000afb670 X18=000000000000005c X19=000000003ee40000
X20=000000003ee80000 X21=0000200000700310 X22=ffffa000426ff4d8
X23=000000003ee40118 X24=ffff8000428d0318 X25=0000000000000000
X26=000000003ee7ffff X27=00002000007002f0 X28=000000003ee40118
X29=00002000007001e0 X30=0000100000b9ed40  SP=00002000007001e0
PSTATE=00000345 ---- EL1h
qemu failed.

The key lines are here:

linear_map: mapped from: 000000003eff0000
                     to: 000000003f000000
linear_map: mapped from: 0000000010000000
                     to: 000000003eff0000

--> mmap MAP_ANONYMOUS: trying to map from: 0x36e0d000 to: 0x3ee0d000
--> mmap MAP_ANONYMOUS: trying to map from: 0x42d00000 to: 0x42d7f000
--> mmap MAP_ANONYMOUS: trying to map from: 0x3cd00000 to: 0x3cd7f000

One can see 1st and 3rd MAP_ANONYMOUS mmap() received an addr hint that collides the memory mapped from 0x0000000010000000 to 0x000000003f000000. And our mmap implementation only checks the addr hint against vma_list not what was mapped by mmu::linear_map.

I am surprised this has not been an issue so far in the past.
@wkozaczuk
Copy link
Collaborator Author

The osv mmap in gdb session from another run looks like this:

(gdb) osv mmap
0x0000000000000000 0x0000000000000000 [0.0 kB]         flags=none     perm=none
0x000000002d18e000 0x000000003518e000 [131072.0 kB]    flags=none     perm=none
0x00000000351c0000 0x0000000035200000 [256.0 kB]       flags=none     perm=none
0x0000000035200000 0x000000003520d000 [52.0 kB]        flags=none     perm=none
0x000000005b880000 0x000000005b8c0000 [256.0 kB]       flags=none     perm=r   
0x0000100000000000 0x0000100000001000 [4.0 kB]         flags=fmF      perm=rx   offset=0x00000000 path=/libvdso.so
0x0000100000010000 0x0000100000011000 [4.0 kB]         flags=fmF      perm=r    offset=0x00000000 path=/libvdso.so
0x0000100000011000 0x0000100000012000 [4.0 kB]         flags=fmF      perm=rw   offset=0x00001000 path=/libvdso.so
0x0000100000020000 0x0000100000022000 [8.0 kB]         flags=fmF      perm=rx   offset=0x00000000 path=/node
0x0000100000031000 0x0000100000032000 [4.0 kB]         flags=fmF      perm=r    offset=0x00001000 path=/node
0x0000100000032000 0x0000100000033000 [4.0 kB]         flags=fmF      perm=rw   offset=0x00002000 path=/node
0x0000100000040000 0x0000100002059000 [32868.0 kB]     flags=fmF      perm=rx   offset=0x00000000 path=/usr/lib/libnode.so.72
0x0000100002068000 0x00001000020fb000 [588.0 kB]       flags=fmF      perm=r    offset=0x02018000 path=/usr/lib/libnode.so.72
0x00001000020fb000 0x0000100002107000 [48.0 kB]        flags=fmF      perm=rw   offset=0x020ab000 path=/usr/lib/libnode.so.72
0x0000100002107000 0x000010000211f000 [96.0 kB]        flags=f        perm=rw  
0x0000100002120000 0x0000100002139000 [100.0 kB]       flags=fmF      perm=rx   offset=0x00000000 path=/usr/lib/libz.so.1
0x0000100002148000 0x0000100002149000 [4.0 kB]         flags=fmF      perm=r    offset=0x00018000 path=/usr/lib/libz.so.1
0x0000100002149000 0x000010000214a000 [4.0 kB]         flags=fmF      perm=rw   offset=0x00019000 path=/usr/lib/libz.so.1
0x0000100002150000 0x000010000215a000 [40.0 kB]        flags=fmF      perm=rx   offset=0x00000000 path=/usr/lib/libbrotlidec.so.1
0x0000100002169000 0x000010000216a000 [4.0 kB]         flags=fmF      perm=r    offset=0x00009000 path=/usr/lib/libbrotlidec.so.1
0x000010000216a000 0x000010000216b000 [4.0 kB]         flags=fmF      perm=rw   offset=0x0000a000 path=/usr/lib/libbrotlidec.so.1
0x0000100002170000 0x0000100002190000 [128.0 kB]       flags=fmF      perm=rx   offset=0x00000000 path=/usr/lib/libbrotlicommon.so.1
0x000010000219f000 0x00001000021a0000 [4.0 kB]         flags=fmF      perm=r    offset=0x0001f000 path=/usr/lib/libbrotlicommon.so.1
0x00001000021a0000 0x00001000021a1000 [4.0 kB]         flags=fmF      perm=rw   offset=0x00020000 path=/usr/lib/libbrotlicommon.so.1
0x00001000021b0000 0x0000100002234000 [528.0 kB]       flags=fmF      perm=rx   offset=0x00000000 path=/usr/lib/libbrotlienc.so.1
0x0000100002243000 0x0000100002244000 [4.0 kB]         flags=fmF      perm=r    offset=0x00083000 path=/usr/lib/libbrotlienc.so.1
0x0000100002244000 0x0000100002245000 [4.0 kB]         flags=fmF      perm=rw   offset=0x00084000 path=/usr/lib/libbrotlienc.so.1
0x0000100002250000 0x0000100002264000 [80.0 kB]        flags=fmF      perm=rx   offset=0x00000000 path=/usr/lib/libcares.so.2
0x0000100002273000 0x0000100002274000 [4.0 kB]         flags=fmF      perm=r    offset=0x00013000 path=/usr/lib/libcares.so.2
0x0000100002274000 0x0000100002275000 [4.0 kB]         flags=fmF      perm=rw   offset=0x00014000 path=/usr/lib/libcares.so.2
0x0000100002280000 0x00001000022a5000 [148.0 kB]       flags=fmF      perm=rx   offset=0x00000000 path=/usr/lib/libnghttp2.so.14
0x00001000022b4000 0x00001000022b7000 [12.0 kB]        flags=fmF      perm=r    offset=0x00024000 path=/usr/lib/libnghttp2.so.14
0x00001000022b7000 0x00001000022b8000 [4.0 kB]         flags=fmF      perm=rw   offset=0x00027000 path=/usr/lib/libnghttp2.so.14
0x00001000022c0000 0x0000100002517000 [2396.0 kB]      flags=fmF      perm=rx   offset=0x00000000 path=/usr/lib/libcrypto.so.1.1
0x0000100002526000 0x000010000254f000 [164.0 kB]       flags=fmF      perm=r    offset=0x00256000 path=/usr/lib/libcrypto.so.1.1
0x000010000254f000 0x0000100002551000 [8.0 kB]         flags=fmF      perm=rw   offset=0x0027f000 path=/usr/lib/libcrypto.so.1.1
0x0000100002551000 0x0000100002555000 [16.0 kB]        flags=f        perm=rw  
0x0000100002560000 0x00001000025de000 [504.0 kB]       flags=fmF      perm=rx   offset=0x00000000 path=/usr/lib/libssl.so.1.1
0x00001000025ee000 0x00001000025f7000 [36.0 kB]        flags=fmF      perm=r    offset=0x0007e000 path=/usr/lib/libssl.so.1.1
0x00001000025f7000 0x00001000025fb000 [16.0 kB]        flags=fmF      perm=rw   offset=0x00087000 path=/usr/lib/libssl.so.1.1
0x0000100002600000 0x00001000028e2000 [2952.0 kB]      flags=fmF      perm=rx   offset=0x00000000 path=/usr/lib/libicui18n.so.67
0x00001000028f2000 0x0000100002903000 [68.0 kB]        flags=fmF      perm=r    offset=0x002e2000 path=/usr/lib/libicui18n.so.67
0x0000100002903000 0x0000100002904000 [4.0 kB]         flags=fmF      perm=rw   offset=0x002f3000 path=/usr/lib/libicui18n.so.67
0x0000100002904000 0x0000100002905000 [4.0 kB]         flags=f        perm=rw  
0x0000100002910000 0x0000100002ae0000 [1856.0 kB]      flags=fmF      perm=rx   offset=0x00000000 path=/usr/lib/libicuuc.so.67
0x0000100002af0000 0x0000100002b02000 [72.0 kB]        flags=fmF      perm=r    offset=0x001d0000 path=/usr/lib/libicuuc.so.67
0x0000100002b02000 0x0000100002b03000 [4.0 kB]         flags=fmF      perm=rw   offset=0x001e2000 path=/usr/lib/libicuuc.so.67
0x0000100002b03000 0x0000100002b05000 [8.0 kB]         flags=f        perm=rw  
0x0000100002b10000 0x0000100004625000 [27732.0 kB]     flags=fmF      perm=rx   offset=0x00000000 path=/usr/lib/libicudata.so.67
0x0000100004635000 0x0000100004636000 [4.0 kB]         flags=fmF      perm=r    offset=0x01b15000 path=/usr/lib/libicudata.so.67
0x0000100004636000 0x0000100004637000 [4.0 kB]         flags=fmF      perm=rw   offset=0x01b16000 path=/usr/lib/libicudata.so.67
0x0000100004640000 0x0000100004653000 [76.0 kB]        flags=fmF      perm=rx   offset=0x00000000 path=/usr/lib/libgcc_s.so.1
0x0000100004663000 0x0000100004664000 [4.0 kB]         flags=fmF      perm=r    offset=0x00013000 path=/usr/lib/libgcc_s.so.1
0x0000100004664000 0x0000100004665000 [4.0 kB]         flags=fmF      perm=rw   offset=0x00014000 path=/usr/lib/libgcc_s.so.1
0x0000200000000000 0x0000200000001000 [4.0 kB]         flags=p        perm=none
0x0000200000001000 0x0000200000002000 [4.0 kB]         flags=p        perm=none
0x0000200000002000 0x0000200000101000 [1020.0 kB]      flags=p        perm=rw  
0x0000200000200000 0x0000200000601000 [4100.0 kB]      flags=p        perm=rw  
0x0000200000601000 0x0000200000602000 [4.0 kB]         flags=p        perm=none
0x0000200000602000 0x0000200000701000 [1020.0 kB]      flags=p        perm=rw  
0x0000200000701000 0x0000200000702000 [4.0 kB]         flags=p        perm=none
0x0000200000702000 0x0000200000801000 [1020.0 kB]      flags=p        perm=rw  
0x0000200000801000 0x0000200000802000 [4.0 kB]         flags=p        perm=none
0x0000200000802000 0x0000200000901000 [1020.0 kB]      flags=p        perm=rw  
0x0000200000901000 0x0000200000902000 [4.0 kB]         flags=p        perm=none
0x0000200000902000 0x0000200000a01000 [1020.0 kB]      flags=p        perm=rw  
0x0000200000a01000 0x0000200000a02000 [4.0 kB]         flags=p        perm=none
0x0000200000a02000 0x0000200000b01000 [1020.0 kB]      flags=p        perm=rw  
0x0000200000b01000 0x0000200000b02000 [4.0 kB]         flags=p        perm=none
0x0000200000b02000 0x0000200000c01000 [1020.0 kB]      flags=p        perm=rw  
0x0000800000000000 0x0000800000000000 [0.0 kB]         flags=none     perm=none

In this case 3 anonymous areas collide:

0x000000002d18e000 0x000000003518e000 [131072.0 kB]    flags=none     perm=none
0x00000000351c0000 0x0000000035200000 [256.0 kB]       flags=none     perm=none
0x0000000035200000 0x000000003520d000 [52.0 kB]        flags=none     perm=none

This simple hack fixes the issue but it is not really a proper solution:

diff --git a/libc/mman.cc b/libc/mman.cc
index add8ef15..b6454dfd 100644
--- a/libc/mman.cc
+++ b/libc/mman.cc
@@ -144,6 +144,10 @@ void *mmap(void *addr, size_t length, int prot, int flags,
     }
 #endif
     if (flags & MAP_ANONYMOUS) {
+        if (addr && addr < (void*)0x0000100000000000) {
+           addr = addr + (u64)0x0000100000000000;
+        }
         // We have already determined (see below) the region where the heap must be located. Now the JVM will request
         // fixed mappings inside that region
         if (jvm_heap_size && (addr >= jvm_heap_region) && (addr + length <= jvm_heap_region_end) && (mmap_flags & mmu::mmap_fixed)) {

Should we register areas mapped by mmu::linear_map() in vma_list?

@wkozaczuk wkozaczuk mentioned this issue Jun 13, 2021
Closed
wkozaczuk added a commit that referenced this issue Mar 31, 2022
@wkozaczuk
mmu: track linear mmap
913fedf 
Sometimes while debugging problems related to how kernel and devices are
memory-mapped it is helpful to see it in some form in gdb or by reading
a procfs file just like we can do with mmap-ed VMAs. In addition we
need to know where linear VMAs are located so that we can avoid
collisions with mmap() as described by the issue #1135.

To that end this patch adds new struct - linear_vma and collection
of those - linear_vma_set - to track how memory gets mapped using
linear_map(). It also modifies all places calling linear_map() to
pass new argument - name.

Please note that we can not re-use existing vma class, as it holds
much richer information and lots of it is not applicable to linear map
which is quite static and is simply a pre-populated mapping between
some area of virtual and physical memory.

Upcoming patches will add new 'osv linear_mmap' to loader.py
and implementation of new sysfs pseudo file.

Signed-off-by: Waldemar Kozaczuk <jwkozaczuk@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@wkozaczuk