Prevent overlapped decompression of embedded assemblies #7732

grendello · 2023-01-24T11:50:51Z

Commit d236af5 introduced embedded assembly compression,
using the lz4 algorithm, which speeds up startup and reduces final package size.

Assemblies are compressed at build time and, at the same time, pre-allocated buffers for
the decompressed data are allocated in libxamarin-app.so. The buffers are then passed
to the LZ4 APIs, all threads using the same output buffer. The assumption was that we can
do fine without locking as even if overlapped decompression happens, the output data will be
the same and so even if two threads do the same thing at the same time, the data will be valid
at all times, so long as at least one thread completes the decompression.

This assumption proved to be largely true, but it appears that in high concurrency cases it
is possible that the data in decompression buffer differs. My guess is that LZ4 either uses the
output buffer as a scratchpad area when decompressing or that it initializes/modifies the buffer
before writing actual data in it. With overlapped decompression, it may lead to one thread
overwriting valid data previously written by another thread, so that when the latter returns the
buffer it thought to have had valid data may contain certain bytes temporarily overwritten by the
decompression session in the other, still running, thread. It may happen that MonoVM reads the
corrupted data just when it is still invalid (before the still running decompression session actually
writes the valid data), a classic race condition.

To fix this, the decompression block is now protected with a startup-aware mutex. Mutex will be held
only after the initial startup phase is completed, so there should not be much loss of startup
performance.

jonathanpeppers · 2023-01-24T16:56:20Z

src/monodroid/jni/embedded-assemblies.cc

-				internal_timing->end_event (decompress_time_index, true /* uses_more_info */);
-				internal_timing->add_more_info (decompress_time_index, name);


Was this timing message not that helpful?

Was this timing message not that helpful?

Yeah, I used it initially to see how much decompression itself costs us, but right now it's just wasted CPU cycles.

jonpryor · 2023-01-24T17:05:31Z

src/monodroid/jni/embedded-assemblies.cc

@@ -81,31 +89,32 @@ EmbeddedAssemblies::get_assembly_data (uint8_t *data, uint32_t data_size, [[mayb
 	if (header->magic == COMPRESSED_DATA_MAGIC) {
 		if (XA_UNLIKELY (compressed_assemblies.descriptors == nullptr)) {
 			log_fatal (LOG_ASSEMBLY, "Compressed assembly found but no descriptor defined");
-			exit (FATAL_EXIT_MISSING_ASSEMBLY);
+			abort ();


Why abort() and not exit()? I forget what abort() does on Android…

exit() causes the application to restart, abort() kills it with a native SIGABRT stack trace

There are a lot of places that users complain about a "white screen", and it ended up their app was crashing and restarting.

Should we audit all of the exit() calls?

Yes, we should, I've been meaning to do it for quite a while now. I'll open a separate PR for this.

PR #7734 opened

grendello · 2023-01-25T16:43:50Z

Let's wait for PR #7734 to be merged first.

* main: [monodroid] Replace `exit()` with `abort()` in native code (dotnet#7734) Bump to xamarin/Java.Interop/main@8a1ae57 (dotnet#7738) [build] bump `$(AndroidNet7Version)` (dotnet#7737) Bump to xamarin/Java.Interop/main@1366d99 (dotnet#7718) [Xamarin.Android.Build.Tasks] fix AndroidGenerateResourceDesigner (dotnet#7721) Bump to xamarin/monodroid@50faac94 (dotnet#7725)

src/monodroid/jni/embedded-assemblies.cc

jonpryor · 2023-01-26T19:37:30Z

[monodroid] Prevent overlapped decompression of embedded assemblies (#7732)

Fixes: https://github.com/xamarin/xamarin-android/issues/7335

Context: d236af54538ef1fe62d0125798cdde78e46dcd8e

Commit d236af54 introduced embedded assembly compression, using LZ4,
which speeds up startup and reduces final package size.

Assemblies are compressed at build time and, at the same time, pre-
allocated buffers for the **decompressed** data are allocated in
`libxamarin-app.so`.  The buffers are then passed to the LZ4 APIs,
all threads using the same output buffer.  The assumption was that we
can do fine without locking as even if overlapped decompression
happens, the output data will be the same and so even if two threads
do the same thing at the same time, the data will be valid at all
times, so long as at least one thread completes the decompression.

This assumption proved to be **largely** true, but it appears that in
high concurrency cases it is possible that the data in the
decompression buffer differs.  This can result in app crashes:

	A/libc: Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 3092 (.NET ThreadPool), pid 2727 (myapp.name)
	A/DEBUG: pid: 2727, tid: 3092, name: .NET ThreadPool  >>> myapp.name <<<
	A/DEBUG:       #01 pc 0000000000029b1c  /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmono-android.release.so (offset 0x103d000) (xamarin::android::internal::MonodroidRuntime::mono_log_handler(char const*, char const*, char const*, int, void*)+144) (BuildId: 29c5a3805a0bedee1eede9b6668d7c676aa63371)
	A/DEBUG:       #02 pc 00000000002680bc  /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
	A/DEBUG:       #03 pc 00000000002681e8  /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
	A/DEBUG:       #04 pc 000000000008555c  /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (mono_metadata_string_heap+188) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
	…

My guess is that LZ4 either uses the output buffer as a scratchpad
area when decompressing or that it initializes/modifies the buffer
before writing actual data in it.  With overlapped decompression, it
may lead to one thread overwriting valid data previously written by
another thread, so that when the latter returns the buffer it thought
to have had valid data may contain certain bytes temporarily
overwritten by the decompression session in the other, still running,
thread.  It may happen that MonoVM reads the corrupted data just when
it is still invalid (before the still running decompression session
actually writes the valid data), a classic race condition.

To fix this, the decompression block is now protected with a startup-
aware mutex.  Mutex will be held only after the initial startup phase
is completed, so there should not be much loss of startup performance.

…otnet#7732) Fixes: dotnet#7335 Context: d236af5 Commit d236af5 introduced embedded assembly compression, using LZ4, which speeds up startup and reduces final package size. Assemblies are compressed at build time and, at the same time, pre- allocated buffers for the **decompressed** data are allocated in `libxamarin-app.so`. The buffers are then passed to the LZ4 APIs, all threads using the same output buffer. The assumption was that we can do fine without locking as even if overlapped decompression happens, the output data will be the same and so even if two threads do the same thing at the same time, the data will be valid at all times, so long as at least one thread completes the decompression. This assumption proved to be **largely** true, but it appears that in high concurrency cases it is possible that the data in the decompression buffer differs. This can result in app crashes: A/libc: Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 3092 (.NET ThreadPool), pid 2727 (myapp.name) A/DEBUG: pid: 2727, tid: 3092, name: .NET ThreadPool >>> myapp.name <<< A/DEBUG: #1 pc 0000000000029b1c /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmono-android.release.so (offset 0x103d000) (xamarin::android::internal::MonodroidRuntime::mono_log_handler(char const*, char const*, char const*, int, void*)+144) (BuildId: 29c5a3805a0bedee1eede9b6668d7c676aa63371) A/DEBUG: #2 pc 00000000002680bc /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b) A/DEBUG: dotnet#3 pc 00000000002681e8 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b) A/DEBUG: dotnet#4 pc 000000000008555c /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (mono_metadata_string_heap+188) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b) … My guess is that LZ4 either uses the output buffer as a scratchpad area when decompressing or that it initializes/modifies the buffer before writing actual data in it. With overlapped decompression, it may lead to one thread overwriting valid data previously written by another thread, so that when the latter returns the buffer it thought to have had valid data may contain certain bytes temporarily overwritten by the decompression session in the other, still running, thread. It may happen that MonoVM reads the corrupted data just when it is still invalid (before the still running decompression session actually writes the valid data), a classic race condition. To fix this, the decompression block is now protected with a startup- aware mutex. Mutex will be held only after the initial startup phase is completed, so there should not be much loss of startup performance.

* main: Bump r8 from 3.3.75 to 4.0.48 (dotnet#7700) [monodroid] Prevent overlapped decompression of embedded assemblies (dotnet#7732) [xaprepare] Support arm64 emulator components (dotnet#7743)

…otnet#7732) Fixes: dotnet#7335 Context: d236af5 Commit d236af5 introduced embedded assembly compression, using LZ4, which speeds up startup and reduces final package size. Assemblies are compressed at build time and, at the same time, pre- allocated buffers for the **decompressed** data are allocated in `libxamarin-app.so`. The buffers are then passed to the LZ4 APIs, all threads using the same output buffer. The assumption was that we can do fine without locking as even if overlapped decompression happens, the output data will be the same and so even if two threads do the same thing at the same time, the data will be valid at all times, so long as at least one thread completes the decompression. This assumption proved to be **largely** true, but it appears that in high concurrency cases it is possible that the data in the decompression buffer differs. This can result in app crashes: A/libc: Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 3092 (.NET ThreadPool), pid 2727 (myapp.name) A/DEBUG: pid: 2727, tid: 3092, name: .NET ThreadPool >>> myapp.name <<< A/DEBUG: #1 pc 0000000000029b1c /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmono-android.release.so (offset 0x103d000) (xamarin::android::internal::MonodroidRuntime::mono_log_handler(char const*, char const*, char const*, int, void*)+144) (BuildId: 29c5a3805a0bedee1eede9b6668d7c676aa63371) A/DEBUG: #2 pc 00000000002680bc /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b) A/DEBUG: dotnet#3 pc 00000000002681e8 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b) A/DEBUG: dotnet#4 pc 000000000008555c /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (mono_metadata_string_heap+188) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b) … My guess is that LZ4 either uses the output buffer as a scratchpad area when decompressing or that it initializes/modifies the buffer before writing actual data in it. With overlapped decompression, it may lead to one thread overwriting valid data previously written by another thread, so that when the latter returns the buffer it thought to have had valid data may contain certain bytes temporarily overwritten by the decompression session in the other, still running, thread. It may happen that MonoVM reads the corrupted data just when it is still invalid (before the still running decompression session actually writes the valid data), a classic race condition. To fix this, the decompression block is now protected with a startup- aware mutex. Mutex will be held only after the initial startup phase is completed, so there should not be much loss of startup performance.

…7732) Fixes: #7335 Context: d236af5 Commit d236af5 introduced embedded assembly compression, using LZ4, which speeds up startup and reduces final package size. Assemblies are compressed at build time and, at the same time, pre- allocated buffers for the **decompressed** data are allocated in `libxamarin-app.so`. The buffers are then passed to the LZ4 APIs, all threads using the same output buffer. The assumption was that we can do fine without locking as even if overlapped decompression happens, the output data will be the same and so even if two threads do the same thing at the same time, the data will be valid at all times, so long as at least one thread completes the decompression. This assumption proved to be **largely** true, but it appears that in high concurrency cases it is possible that the data in the decompression buffer differs. This can result in app crashes: A/libc: Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 3092 (.NET ThreadPool), pid 2727 (myapp.name) A/DEBUG: pid: 2727, tid: 3092, name: .NET ThreadPool >>> myapp.name <<< A/DEBUG: #1 pc 0000000000029b1c /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmono-android.release.so (offset 0x103d000) (xamarin::android::internal::MonodroidRuntime::mono_log_handler(char const*, char const*, char const*, int, void*)+144) (BuildId: 29c5a3805a0bedee1eede9b6668d7c676aa63371) A/DEBUG: #2 pc 00000000002680bc /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b) A/DEBUG: #3 pc 00000000002681e8 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b) A/DEBUG: #4 pc 000000000008555c /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (mono_metadata_string_heap+188) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b) … My guess is that LZ4 either uses the output buffer as a scratchpad area when decompressing or that it initializes/modifies the buffer before writing actual data in it. With overlapped decompression, it may lead to one thread overwriting valid data previously written by another thread, so that when the latter returns the buffer it thought to have had valid data may contain certain bytes temporarily overwritten by the decompression session in the other, still running, thread. It may happen that MonoVM reads the corrupted data just when it is still invalid (before the still running decompression session actually writes the valid data), a classic race condition. To fix this, the decompression block is now protected with a startup- aware mutex. Mutex will be held only after the initial startup phase is completed, so there should not be much loss of startup performance.

* main: [Xamarin.Android.Build.Tasks] Fix issue where app will not install (dotnet#7719) Bump to dotnet/installer@779a644 8.0.100-alpha.1.23070.23 (dotnet#7728) LEGO: Merge pull request 7751 [Mono.Android] Wrap connection exceptions in HttpRequestException (dotnet#7661) [Mono.Android] Fix View.SystemUiVisibility enumification (dotnet#7730) Bump r8 from 3.3.75 to 4.0.48 (dotnet#7700) [monodroid] Prevent overlapped decompression of embedded assemblies (dotnet#7732) [xaprepare] Support arm64 emulator components (dotnet#7743) Bump SQLite to 3.40.1 (dotnet#7733) Bump to xamarin/xamarin-android-binutils/L_15.0.7-5.0.3@6721af4b (dotnet#7742) [monodroid] Replace `exit()` with `abort()` in native code (dotnet#7734) Bump to xamarin/Java.Interop/main@8a1ae57 (dotnet#7738) [build] bump `$(AndroidNet7Version)` (dotnet#7737) Bump to xamarin/Java.Interop/main@1366d99 (dotnet#7718) [Xamarin.Android.Build.Tasks] fix AndroidGenerateResourceDesigner (dotnet#7721) Bump to xamarin/monodroid@50faac94 (dotnet#7725)

* main: (112 commits) [ci] Remove classic Mono.Android-Tests runs (dotnet#7778) $(AndroidPackVersionSuffix)=preview.2; net8 is 34.0.0-preview.2 (dotnet#7761) Localized file check-in by OneLocBuild Task (dotnet#7758) Bump to dotnet/installer@dec1209 8.0.100-alpha.1.23080.11 (dotnet#7755) LEGO: Merge pull request 7756 Localized file check-in by OneLocBuild (dotnet#7752) [Xamarin.Android.Build.Tasks] Fix issue where app will not install (dotnet#7719) Bump to dotnet/installer@779a644 8.0.100-alpha.1.23070.23 (dotnet#7728) LEGO: Merge pull request 7751 [Mono.Android] Wrap connection exceptions in HttpRequestException (dotnet#7661) [Mono.Android] Fix View.SystemUiVisibility enumification (dotnet#7730) Bump r8 from 3.3.75 to 4.0.48 (dotnet#7700) [monodroid] Prevent overlapped decompression of embedded assemblies (dotnet#7732) [xaprepare] Support arm64 emulator components (dotnet#7743) Bump SQLite to 3.40.1 (dotnet#7733) Bump to xamarin/xamarin-android-binutils/L_15.0.7-5.0.3@6721af4b (dotnet#7742) [monodroid] Replace `exit()` with `abort()` in native code (dotnet#7734) Bump to xamarin/Java.Interop/main@8a1ae57 (dotnet#7738) [build] bump `$(AndroidNet7Version)` (dotnet#7737) Bump to xamarin/Java.Interop/main@1366d99 (dotnet#7718) ...

…otnet#7732) Fixes: dotnet#7335 Context: d236af5 Commit d236af5 introduced embedded assembly compression, using LZ4, which speeds up startup and reduces final package size. Assemblies are compressed at build time and, at the same time, pre- allocated buffers for the **decompressed** data are allocated in `libxamarin-app.so`. The buffers are then passed to the LZ4 APIs, all threads using the same output buffer. The assumption was that we can do fine without locking as even if overlapped decompression happens, the output data will be the same and so even if two threads do the same thing at the same time, the data will be valid at all times, so long as at least one thread completes the decompression. This assumption proved to be **largely** true, but it appears that in high concurrency cases it is possible that the data in the decompression buffer differs. This can result in app crashes: A/libc: Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 3092 (.NET ThreadPool), pid 2727 (myapp.name) A/DEBUG: pid: 2727, tid: 3092, name: .NET ThreadPool >>> myapp.name <<< A/DEBUG: #1 pc 0000000000029b1c /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmono-android.release.so (offset 0x103d000) (xamarin::android::internal::MonodroidRuntime::mono_log_handler(char const*, char const*, char const*, int, void*)+144) (BuildId: 29c5a3805a0bedee1eede9b6668d7c676aa63371) A/DEBUG: #2 pc 00000000002680bc /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b) A/DEBUG: #3 pc 00000000002681e8 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b) A/DEBUG: dotnet#4 pc 000000000008555c /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (mono_metadata_string_heap+188) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b) … My guess is that LZ4 either uses the output buffer as a scratchpad area when decompressing or that it initializes/modifies the buffer before writing actual data in it. With overlapped decompression, it may lead to one thread overwriting valid data previously written by another thread, so that when the latter returns the buffer it thought to have had valid data may contain certain bytes temporarily overwritten by the decompression session in the other, still running, thread. It may happen that MonoVM reads the corrupted data just when it is still invalid (before the still running decompression session actually writes the valid data), a classic race condition. To fix this, the decompression block is now protected with a startup- aware mutex. Mutex will be held only after the initial startup phase is completed, so there should not be much loss of startup performance.

grendello added 2 commits January 24, 2023 00:04

Tentative fix for issue dotnet#7335, needs more testing

9d2740d

Remove dead code

171c170

grendello requested a review from jonpryor as a code owner January 24, 2023 11:50

grendello mentioned this pull request Jan 24, 2023

Crashes on startup significantly increased after migration to .NET6 #7335

Closed

Update apkdesc files

8d11a8d

grendello requested review from dellis1972 and jonathanpeppers as code owners January 24, 2023 16:28

dellis1972 approved these changes Jan 24, 2023

View reviewed changes

jonathanpeppers reviewed Jan 24, 2023

View reviewed changes

jonpryor reviewed Jan 24, 2023

View reviewed changes

Another batch of apkdesc updates

3b1c93b

grendello added the do-not-merge PR should not be merged. label Jan 25, 2023

grendello removed the do-not-merge PR should not be merged. label Jan 25, 2023

Fix after-merge SNAFU

efc88b5

dellis1972 reviewed Jan 26, 2023

View reviewed changes

src/monodroid/jni/embedded-assemblies.cc Show resolved Hide resolved

jonpryor merged commit 2ff72d7 into dotnet:main Jan 26, 2023

tipa mentioned this pull request Feb 15, 2023

InvalidCastException thrown after TimerQueueTimer.CallCallback dotnet/runtime#81211

Closed

akoeplinger mentioned this pull request Feb 21, 2023

Cherry-pick #7732 to d17-5 #7746

Merged

grendello mentioned this pull request Feb 21, 2023

[monodroid] Prevent overlapped decompression of embedded assemblies #7817

Merged

github-actions bot locked and limited conversation to collaborators Jan 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prevent overlapped decompression of embedded assemblies #7732

Prevent overlapped decompression of embedded assemblies #7732

grendello commented Jan 24, 2023 •

edited

Loading

jonathanpeppers Jan 24, 2023

grendello Jan 24, 2023

jonpryor Jan 24, 2023

grendello Jan 24, 2023

jonathanpeppers Jan 24, 2023

grendello Jan 24, 2023

grendello Jan 24, 2023

grendello commented Jan 25, 2023

jonpryor commented Jan 26, 2023 •

edited

Loading

		internal_timing->end_event (decompress_time_index, true /* uses_more_info */);
		internal_timing->add_more_info (decompress_time_index, name);

Prevent overlapped decompression of embedded assemblies #7732

Prevent overlapped decompression of embedded assemblies #7732

Conversation

grendello commented Jan 24, 2023 • edited Loading

jonathanpeppers Jan 24, 2023

Choose a reason for hiding this comment

grendello Jan 24, 2023

Choose a reason for hiding this comment

jonpryor Jan 24, 2023

Choose a reason for hiding this comment

grendello Jan 24, 2023

Choose a reason for hiding this comment

jonathanpeppers Jan 24, 2023

Choose a reason for hiding this comment

grendello Jan 24, 2023

Choose a reason for hiding this comment

grendello Jan 24, 2023

Choose a reason for hiding this comment

grendello commented Jan 25, 2023

jonpryor commented Jan 26, 2023 • edited Loading

grendello commented Jan 24, 2023 •

edited

Loading

jonpryor commented Jan 26, 2023 •

edited

Loading