[android] Instrumentation run failed due to 'Process crashed.'

[friedbunny]

In this android-gnustl-arm-v7 job (for #14591), instrumentation tests appear to have completed successfully:

Instrumentation testing complete.

More details are available at [https://console.firebase.google.com/project/android-gl-native/testlab/histories/bh.1470d2ab45df6f1f/matrices/8751784270342670967].
+---------+-----------------------+--------------------------------------------+
| OUTCOME |    TEST_AXIS_VALUE    |                TEST_DETAILS                |
+---------+-----------------------+--------------------------------------------+
| Failed  | htc_m8-19-en-portrait | 0 test cases failed, 112 passed, 2 skipped |
+---------+-----------------------+--------------------------------------------+
Exited with code 10

... but the ultimate result on firebase was a failure with "Instrumentation run failed due to 'Process crashed.’"

I don’t see any obvious clues as to why this happened and maybe it’s just provider instability, but I figured I’d ticket it nonetheless.

/cc @mapbox/maps-android

[tobrun]

Thank you for the ticket @friedbunny. This is indeed a weird issue as I'm not seeing any crashes in the logging of firebase 🤔

[friedbunny]

Saw another job report “0 test cases failed” but crash on firebase. This one has a bit more information:

Build fingerprint: 'htc/brightstarus_wwe/htc_m8:4.4.4/KTU84P/389838.5:user/release-keys'
Revision: '0'
pid: 13352, tid: 14373, name: OkHttp Dispatch  >>> com.mapbox.mapboxsdk.testapp <<<
debuggerd: checkTellHTCSettings
debuggerd: buildType: user, roAaReport: com, ro.sf: 1
debuggerd: rosf: 1
debuggerd: [New ROM] isShippingROM: true
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr deadbaad
Abort message: 'invalid address or address of corrupt block 0x783007b0 passed to dlfree’

[tobrun]

Looking at all runs of latest week, it seems this issue has starting popping up 3-4 days ago. Comparing that to commits on master. It seems most plausible that the SoLoader changes are responsible from 685a9e5.

cc @zugaldia

[tobrun]

Will run tests locally on older handset to try symbolicating the crashtrace. Will schedule CI runs with and without 685a9e5 on a variety of devices on Firebase test lab.

[tobrun]

Nurbot $ cat ~/crash.txt | ndk-stack -sym platform/android/MapboxGLAndroidSDK/build/intermediates/cmake/debug/obj/armeabi-v7a/
********** Crash dump: **********
Build fingerprint: 'motorola/victara_reteu/victara:4.4.4/KXE21.187-42/41:user/release-keys'
Abort message: 'invalid address or address of corrupt block 0x73e738e0 passed to dlfree'
#00 0x000118b4 /system/lib/libc.so (dlfree+1191)
#01 0x0000dd4b /system/lib/libc.so (free+10)
#02 0x000c5585 /system/lib/libcrypto.so (CRYPTO_free+24)
#03 0x0002f6bf /system/lib/libssl.so (ssl_parse_serverhello_tlsext+738)
#04 0x000182c1 /system/lib/libssl.so (ssl3_get_server_hello+1008)
#05 0x000176bf /system/lib/libssl.so (ssl3_connect+566)
#06 0x00027f4b /system/lib/libssl.so (SSL_do_handshake+50)
#07 0x0000af49 /system/lib/libjavacrypto.so
#08 0x000203cc /system/lib/libdvm.so (dvmPlatformInvoke+112)
#09 0x00050ef3 /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+398)
#10 0x00029860 /system/lib/libdvm.so
#11 0x00030b68 /system/lib/libdvm.so (dvmMterpStd(Thread*)+76)
#12 0x0002e200 /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+184)
#13 0x00063519 /system/lib/libdvm.so (dvmCallMethodV(Thread*, Method const*, Object*, bool, JValue*, std::__va_list)+336)
#14 0x0006353d /system/lib/libdvm.so (dvmCallMethod(Thread*, Method const*, Object*, JValue*, ...)+20)
#15 0x0005820b /system/lib/libdvm.so
#16 0x0000d2b0 /system/lib/libc.so (__thread_entry+72)
#17 0x0000d448 /system/lib/libc.so (pthread_create+240)
Crash dump is completed

[zugaldia]

@tobrun thanks for cutting square/okhttp#5198.

After reading the linked ticket square/okhttp#2679, these are two things we should try next:

• See if we can reproduce the issue when reducing setMaxRequestsPerHost() in the dispatcher. It seems the issue is mostly happening when doing a lot of request at the same time.
• Looks like this is limited to Android 4.x devices but this might be mitigated making sure they're updated to the latest version. I.e. we need to confirm this isn't happening with updated 4.x devices.

[tobrun]

See if we can reproduce the issue when reducing setMaxRequestsPerHost() in the dispatcher. It seems the issue is mostly happening when doing a lot of request at the same time.

Did a batch of tests with:

  @NonNull
  private static Dispatcher getDispatcher() {
    Dispatcher dispatcher = new Dispatcher();
    // Matches core limit set on
    // https://github.com/mapbox/mapbox-gl-native/blob/master/platform/android/src/http_file_source.cpp#L192
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
      dispatcher.setMaxRequestsPerHost(20);
    }
    return dispatcher;
  }

not seeing any improvement

Looks like this is limited to Android 4.x devices but this might be mitigated making sure they're updated to the latest version. I.e. we need to confirm this isn't happening with updated 4.x devices.

Validated that the devices I'm using for testing on Firebase are running the latest available update.

[langsmith]

Final follow up about this ticket:

This rare bug that was discovered via instrumented unit tests on Android 4.4 (aka KitKat). This bug isn’t tied to a code change in the Mapbox Maps SDK for Android. Rather, the root cause is on the Android OS (and how it interacts with OkHttp).

The Maps SDK doesn’t directly communicate with this system library that the bug originates from, but goes through an external dependency that handles the http communication for the Maps SDK. Capturing from the upstream repository that the code path to this system library isn’t thread safe in Android 4.4 and can result in the native crash above. The only way to fix this issue is by upgrading the end-user device to Android 5.0 or higher which isn’t possible if the device manufacturer hasn’t released an update for it.

#14971 and #14986 address this ticket in the best way we can at this current moment.