Error importing ubuntu distribution (out of memory)

[ErmolaevNV]

Version

10.0.19045.2364

WSL Version

• WSL 2
• WSL 1

Kernel Version

No response

Distro Version

No response

Other Software

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
C:/Program Files/Git  238G  178G   61G  75% /
W:                    931G  312G  620G  34% /w

.wslconfig:

# Settings apply across all Linux distros running on WSL 2
[wsl2]

# Limits VM memory to use no more than 4 GB, this can be set as whole numbers using GB or MB
memory=4GB 

# Sets the VM to use two virtual processors
processors=4

# Specify a custom Linux kernel to use with your installed distros. The default kernel used can be found at https://github.com/microsoft/WSL2-Linux-Kernel
# kernel=W:\\temp\\myCustomKernel

# Sets additional kernel parameters, in this case enabling older Linux base images such as Centos 6
kernelCommandLine = vsyscall=emulate

# Sets amount of swap storage space to 8GB, default is 25% of available RAM
swap=8ф

# Sets swapfile path location, default is %USERPROFILE%\AppData\Local\Temp\swap.vhdx
swapfile=W:\\temp\\wsl-swap.vhdx

# Disable page reporting so WSL retains all allocated memory claimed from Windows and releases none back when free
pageReporting=false  

# Turn off default connection to bind WSL 2 localhost to Windows localhost
localhostforwarding=true

# Disables nested virtualization
nestedVirtualization=false

# Turns on output console showing contents of dmesg when opening a WSL 2 distro for debugging
debugConsole=true

[network]
generateResolvConf = false

Repro Steps

1. wsl.exe --export Ubuntu w:\ubuntu.tar
2. wsl.exe --import Ubuntu w:\wsl\Ubuntu w:\ubuntu.tar --version 2

Expected Behavior

Imported distribution to a new location

Actual Behavior

[ 49.801488] hv_balloon: Max. dynamic memory size: 3072 MB
[ 1247.237139] bsdtar invoked oom-killer: gfp_mask=0x108c40(GFP_NOFS|__GFP_NOFAIL|__GFP_HARDWALL), order=0, oom_score_adj=0
[ 1247.238027] CPU: 3 PID: 133 Comm: bsdtar Not tainted 5.15.79.1-microsoft-standard-WSL2 #1
[ 1247.238548] Call Trace:
[ 1247.238712]
[ 1247.238888] dump_stack_lvl+0x34/0x48
[ 1247.239169] dump_header+0x4a/0x19a
[ 1247.239433] oom_kill_process.cold+0xb/0x10
[ 1247.239716] out_of_memory+0x1fb/0x2a0
[ 1247.239978] mem_cgroup_out_of_memory+0xe5/0x100
[ 1247.240338] try_charge_memcg+0x66e/0x730
[ 1247.240621] charge_memcg+0x3f/0x90
[ 1247.240881] __mem_cgroup_charge+0x2c/0x90
[ 1247.241145] __add_to_page_cache_locked+0x2eb/0x360
[ 1247.241533] ? scan_shadow_nodes+0x30/0x30
[ 1247.241777] add_to_page_cache_lru+0x48/0xd0
[ 1247.242078] pagecache_get_page+0x18e/0x4a0
[ 1247.242315] __getblk_gfp+0xe0/0x270
[ 1247.242548] ? submit_bh_wbc+0x160/0x190
[ 1247.242792] ext4_sb_breadahead_unmovable+0x18/0x50
[ 1247.243110] __ext4_get_inode_loc+0x392/0x420
[ 1247.243392] ? kmem_cache_alloc+0x108/0x370
[ 1247.243641] __ext4_get_inode_loc_noinmem+0x3d/0xa0
[ 1247.243945] __ext4_iget+0x149/0xdb0
[ 1247.244191] ? d_alloc_parallel+0x232/0x4b0
[ 1247.244470] ext4_lookup+0x11f/0x1e0
[ 1247.244716] __lookup_slow+0x88/0x150
[ 1247.244969] walk_component+0x13f/0x1b0
[ 1247.245197] ? path_init+0x1e5/0x390
[ 1247.245399] path_lookupat.isra.0+0x6e/0x150
[ 1247.245696] path_openat+0xbee/0x1090
[ 1247.245908] ? filename_lookup+0xd9/0x150
[ 1247.246166] do_filp_open+0xb2/0x120
[ 1247.246401] ? cp_new_stat+0x150/0x190
[ 1247.246625] ? __check_object_size+0x146/0x160
[ 1247.246938] do_sys_openat2+0x24a/0x320
[ 1247.247193] do_sys_open+0x44/0x80
[ 1247.247428] do_syscall_64+0x3b/0xc0
[ 1247.247677] entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 1247.247980] RIP: 0033:0x7f66ed079ae6
[ 1247.248194] Code: 58 f6 c2 40 75 3e 89 d0 45 31 d2 25 00 00 41 00 3d 00 00 41 00 74 2d 64 8b 04 25 18 00 00 00 85 c0 75 51 b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 8e 00 00 00 48 83 c4 78 c3 66 0f 1f 84 00
[ 1247.249325] RSP: 002b:00007ffefe5bea90 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[ 1247.249776] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f66ed079ae6
[ 1247.250220] RDX: 0000000000290000 RSI: 000055ce88c752dc RDI: 0000000000000003
[ 1247.250782] RBP: 000055ce88c752dc R08: 0000000000000000 R09: 00007ffefe5bec40
[ 1247.251299] R10: 0000000000000000 R11: 0000000000000246 R12: 000055ce88c752e0
[ 1247.251753] R13: 0000000000000003 R14: 000000000000002f R15: 0000000000000000
[ 1247.252145]
[ 1247.252339] memory: usage 65536kB, limit 65536kB, failcnt 9242697
[ 1247.252670] memory+swap: usage 65536kB, limit 9007199254740988kB, failcnt 0
[ 1247.252992] kmem: usage 1452kB, limit 9007199254740988kB, failcnt 0
[ 1247.253314] Memory cgroup stats for /64M:
[ 1247.253330] anon 62971904
[ 1247.253330] file 2641920
[ 1247.253330] kernel_stack 16384
[ 1247.253330] pagetables 176128
[ 1247.253330] percpu 0
[ 1247.253330] sock 0
[ 1247.253330] shmem 0
[ 1247.253330] file_mapped 24576
[ 1247.253330] file_dirty 122880
[ 1247.253330] file_writeback 0
[ 1247.253330] swapcached 0
[ 1247.253330] anon_thp 54525952
[ 1247.253330] file_thp 0
[ 1247.253330] shmem_thp 0
[ 1247.253330] inactive_anon 62967808
[ 1247.253330] active_anon 4096
[ 1247.253330] inactive_file 1306624
[ 1247.253330] active_file 1298432
[ 1247.253330] unevictable 0
[ 1247.253330] slab_reclaimable 1074664
[ 1247.253330] slab_unreclaimable 199800
[ 1247.253330] slab 1274464
[ 1247.253330] workingset_refault_anon 0
[ 1247.253330] workingset_refault_file 12424701
[ 1247.253330] workingset_activate_anon 0
[ 1247.253330] workingset_activate_file 1925150
[ 1247.253330] workingset_restore_anon 0
[ 1247.253330] workingset_restore_file 1499122
[ 1247.253330] workingset_nodereclaim 10792
[ 1247.258518] Tasks state (memory values in pages):
[ 1247.258836] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 1247.259399] [ 128] 0 128 538 262 40960 0 0 init
[ 1247.259844] [ 133] 0 133 18344 15711 188416 0 0 bsdtar
[ 1247.260361] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/64M,task_memcg=/64M,task=bsdtar,pid=133,uid=0
[ 1247.261092] Memory cgroup out of memory: Killed process 133 (bsdtar) total-vm:73376kB, anon-rss:61484kB, file-rss:1360kB, shmem-rss:0kB, UID:0 pgtables:184kB oom_score_adj:0
[ 1247.268921] WSL (1) ERROR: UtilProcessChildExitCode:2056: /usr/bin/bsdtar killed by signal
[ 1247.268927] 9
[ 1247.269843] WSL (1) ERROR: ProcessMessage:2869: import failed
[ 7395.798619] sd 0:0:0:1: [sdb] Synchronizing SCSI cache

Diagnostic Logs

No response

[Biswa96]

Try to use smaller tarball file for importing.

[elsaco]

@ErmolaevNV comment out memory=4GB in wslconfig, do a wsl --shutdown and try again. You can enable the memory limit after successful import. Also check your swap=8ф config. That must be a typo.

[OneBlue]

@ErmolaevNV: Looking at your .wslconfig, this line will cause a parsing issue:

swap=8ф

Which might just disable swap completely, explaining the error you're seeing. If the tarball is really big, 4GB of memory might not be enough for tar.

I'll close for now but feel free to reopen if importing without the memory & swap limit doesn't solve the issue.

[msci-horvrud]

I had the same issue, and I think it is not directly related to the memory configuration.
If I execute the same import 3 times, it fails with the same error 2 times, it is ad-hoc (same wslconfig for all run).

Relevant config:

[wsl2]
memory=16GB
processors=6
swap=0

I build a unique tar for import, it's size is ~6GB.

Exact error log:

Writing superblocks and filesystem accounting information: done

[    3.203305] EXT4-fs (sdb): mounted filesystem with ordered data mode. Opts: discard,errors=remount-ro,data=ordered. Quota mode: none.
[   47.253763] bsdtar invoked oom-killer: gfp_mask=0x108c40(GFP_NOFS|__GFP_NOFAIL|__GFP_HARDWALL), order=0, oom_score_adj=0
[   47.255546] CPU: 1 PID: 150 Comm: bsdtar Not tainted 5.15.79.1-microsoft-standard-WSL2 #1
[   47.256282] Call Trace:
[   47.256504]  <TASK>
[   47.256690]  dump_stack_lvl+0x34/0x48
[   47.256851]  dump_header+0x4a/0x19a
[   47.257101]  oom_kill_process.cold+0xb/0x10
[   47.257301]  out_of_memory+0x1fb/0x2a0
[   47.257472]  mem_cgroup_out_of_memory+0xe5/0x100
[   47.257672]  try_charge_memcg+0x66e/0x730
[   47.257815]  ? sbitmap_get+0x82/0x1a0
[   47.257964]  charge_memcg+0x3f/0x90
[   47.258113]  __mem_cgroup_charge+0x2c/0x90
[   47.258253]  __add_to_page_cache_locked+0x2eb/0x360
[   47.258429]  ? scan_shadow_nodes+0x30/0x30
[   47.258573]  add_to_page_cache_lru+0x48/0xd0
[   47.258746]  pagecache_get_page+0x18e/0x4a0
[   47.258884]  __getblk_gfp+0xe0/0x270
[   47.259032]  ? submit_bh_wbc+0x160/0x190
[   47.259170]  ext4_sb_breadahead_unmovable+0x18/0x50
[   47.259361]  __ext4_get_inode_loc+0x392/0x420
[   47.259537]  ? kmem_cache_alloc+0x108/0x370
[   47.259682]  __ext4_get_inode_loc_noinmem+0x3d/0xa0
[   47.259870]  __ext4_iget+0x149/0xdb0
[   47.260011]  ? d_alloc_parallel+0x232/0x4b0
[   47.260154]  ext4_lookup+0x11f/0x1e0
[   47.260315]  __lookup_slow+0x85/0x150
[   47.260448]  walk_component+0x13f/0x1b0
[   47.260590]  link_path_walk.part.0+0x236/0x3a0
[   47.260774]  ? __alloc_file+0x98/0x110
[   47.260913]  path_openat+0xb5/0x1090
[   47.261064]  ? __mod_lruvec_page_state+0x53/0xa0
[   47.261252]  ? page_add_file_rmap+0x8b/0x150
[   47.261420]  ? filemap_map_pages+0x111/0x580
[   47.261600]  do_filp_open+0xb2/0x120
[   47.261738]  ? __check_object_size+0x146/0x160
[   47.261932]  do_sys_openat2+0x24a/0x320
[   47.262075]  do_sys_open+0x44/0x80
[   47.262213]  do_syscall_64+0x38/0xc0
[   47.262352]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[   47.262532] RIP: 0033:0x7f99558f898b
[   47.262680] Code: 25 00 00 41 00 3d 00 00 41 00 74 33 64 8b 04 25 18 00 00 00 85 c0 75 57 44 89 e2 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 81 00 00 00 48 83 c4 68 5d 41 5c c3 90 48
[   47.263334] RSP: 002b:00007ffd190a9c90 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[   47.263607] RAX: ffffffffffffffda RBX: 000055e00836d070 RCX: 00007f99558f898b
[   47.263899] RDX: 00000000000b0000 RSI: 000055e0085d80f0 RDI: 00000000ffffff9c
[   47.264176] RBP: 000055e0085d80f0 R08: 000055e0085d8df0 R09: 0000000000000000
[   47.264429] R10: 0000000000000000 R11: 0000000000000246 R12: 00000000000b0000
[   47.264679] R13: 000055e0085d8d40 R14: 000055e0085d8d48 R15: 000055e0085d8d40
[   47.265052]  </TASK>
[   47.265238] memory: usage 65536kB, limit 65536kB, failcnt 691952
[   47.265642] memory+swap: usage 65536kB, limit 9007199254740988kB, failcnt 0
[   47.265961] kmem: usage 5960kB, limit 9007199254740988kB, failcnt 0
[   47.266197] Memory cgroup stats for /64M:
[   47.266207] anon 5738496
[   47.266207] file 55267328
[   47.266207] kernel_stack 16384
[   47.266207] pagetables 65536
[   47.266207] percpu 0
[   47.266207] sock 0
[   47.266207] shmem 0
[   47.266207] file_mapped 0
[   47.266207] file_dirty 0
[   47.266207] file_writeback 0
[   47.266207] swapcached 0
[   47.266207] anon_thp 2097152
[   47.266207] file_thp 0
[   47.266207] shmem_thp 0
[   47.266207] inactive_anon 5734400
[   47.266207] active_anon 4096
[   47.266207] inactive_file 27635712
[   47.266207] active_file 27631616
[   47.266207] unevictable 0
[   47.266207] slab_reclaimable 5965416
[   47.266207] slab_unreclaimable 38232
[   47.266207] slab 6003648
[   47.266207] workingset_refault_anon 0
[   47.266207] workingset_refault_file 195285
[   47.266207] workingset_activate_anon 0
[   47.266207] workingset_activate_file 24754
[   47.266207] workingset_restore_anon 0
[   47.266207] workingset_restore_file 17636
[   47.266207] workingset_nodereclaim 7871
[   47.266207] pgfault 16670
[   47.269331] Tasks state (memory values in pages):
[   47.269511] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[   47.269831] [    142]     0   142      573      263    40960        0             0 init
[   47.270113] [    150]     0   150     4371     1872    73728        0             0 bsdtar
[   47.270405] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/64M,task_memcg=/64M,task=bsdtar,pid=150,uid=0
[   47.270889] Memory cgroup out of memory: Killed process 150 (bsdtar) total-vm:17484kB, anon-rss:5584kB, file-rss:1904kB, shmem-rss:0kB, UID:0 pgtables:72kB oom_score_adj:0
[   48.251626] WSL (1) ERROR: UtilProcessChildExitCode:2056: /usr/bin/bsdtar killed by signal
[   48.251629] 9
[   48.252130] WSL (1) ERROR: ProcessMessage:2869: import failed
[   48.330375] sd 0:0:0:1: [sdb] Synchronizing SCSI cache

[elsaco]

@msci-horvrud you're not using swap. Try setting a swap partition, like adding a 6GB swap vhdx drive, and try importing again. The out of memory killer (oom-kill) was triggered and terminated some of your processes. I run into oom-kill-er while compiling a large chunk of code and adding swap fixed the issue for me.

[msci-horvrud]

I did a recording (can not attach), where it is visible that the windows process of the VM of WSL is not exceeding the 1GB memory usage during the import, even as I set the memory limit to 32GB (the imported image can be loaded into it 5 times!).

As I increase the swap size it is working, but I would consider that as a workaround, I think a Bug is still there in WSL to cope with.
Why is this line appearing in my error log?

memory: usage 65536kB, limit 65536kB, failcnt 621204

The importer uses a different wslconfig, with different limits? Or some other containerization/isolation is causing that the available memory is not available to that given process?

I was able to import the same image, with 4GB of RAM and 1GB of swap with a success rate of 1 from 2.
I have 64GB, so during all my operations there is no memory pressure from other windows processes.

[msci-horvrud]

I was "able" to get OOMKill with the following wslconfig as well:

[wsl2]
memory=32GB
processors=6
swap=8GB

So the previously recommended workaround of using more swap is not even a real solution.

I double-checked that if I start an existing wsl distro, I can verify that the config changes were applied:

asdasd:/mnt/c/Users/myname# cat /proc/swaps
Filename                                Type            Size            Used            Priority
/dev/sdb                                partition       8388608         0               -2

asdasd:/mnt/c/Users/myname# cat /proc/meminfo
MemTotal:       32872832 kB
MemFree:        32201464 kB
MemAvailable:   32181388 kB

I will open a new ticket, as I am not able to re-open this.