diff --git a/.github/workflows/codespell.yml b/.github/workflows/codespell.yml index a36f3c620a..450175ad6c 100644 --- a/.github/workflows/codespell.yml +++ b/.github/workflows/codespell.yml @@ -14,4 +14,4 @@ jobs: - uses: actions/checkout@v2 - uses: codespell-project/actions-codespell@master with: - ignore_words_list: ba,compliancy,firt,ist,keypair,ligh,ser,synopsys,zuser + ignore_words_list: ba,compliancy,firt,ist,keypair,ligh,ser,synopsys,theses,zuser diff --git a/Document/0x04c-Tampering-and-Reverse-Engineering.md b/Document/0x04c-Tampering-and-Reverse-Engineering.md index a73ac20754..c9f20877a2 100644 --- a/Document/0x04c-Tampering-and-Reverse-Engineering.md +++ b/Document/0x04c-Tampering-and-Reverse-Engineering.md @@ -69,10 +69,69 @@ In theory, the mapping between assembly and machine code should be one-to-one, a - Position independent code (PIC) sequences. - Hand crafted assembly code. -On a similar vein, decompilation is a very complicated process, involving many deterministic and heuristic based approaches. As a consequence, decompilation is usually not really accurate, but nevertheless very helpful in getting a quick understanding of the function being analyzed. The accuracy of decompilation depends on the amount of information available in the code being decompiled and the sophistication of the decompiler. In addition, many compilation and post-compilation tools introduce additional complexity to the compiled code in order to increase the difficulty of comprehension and/or even decompilation itself. Such code referred to as _obfuscated code_. +Similarly, decompilation is a very complicated process, involving many deterministic and heuristic based approaches. As a consequence, decompilation is usually not really accurate, but nevertheless very helpful in getting a quick understanding of the function being analyzed. The accuracy of decompilation depends on the amount of information available in the code being decompiled and the sophistication of the decompiler. In addition, many compilation and post-compilation tools introduce additional complexity to the compiled code in order to increase the difficulty of comprehension and/or even decompilation itself. Such code referred to as [_obfuscated code_](#obfuscation). Over the past decades many tools have perfected the process of disassembly and decompilation, producing output with high fidelity. Advanced usage instructions for any of the available tools can often easily fill a book of their own. The best way to get started is to simply pick up a tool that fits your needs and budget and get a well-reviewed user guide. In this section, we will provide an introduction to some of those tools and in the subsequent "Reverse Engineering and Tampering" Android and iOS chapters we'll focus on the techniques themselves, especially those that are specific to the platform at hand. +### Obfuscation + +Obfuscation is the process of transforming code and data to make it more difficult to comprehend (and sometimes even difficult to disassemble). It is usually an integral part of the software protection scheme. Obfuscation isn't something that can be simply turned on or off, programs can be made incomprehensible, in whole or in part, in many ways and to different degrees. + +> Note: All presented techniques below will not stop someone with enough time and budget from reverse engineering your app. However, combining these techniques will make their job significantly harder. The aim is thus to discourage reverse engineers from performing further analysis and not making it worth the effort. + +The following techniques can be used to obfuscate an application: + +- Name obfuscation +- Instruction substitution +- Control flow flattening +- Dead code injection +- String encryption +- Packing + +#### Name Obfuscation + +The standard compiler generates binary symbols based on class and function names from the source code. Therefore, if no obfuscation is applied, symbol names remain meaningful and can easily be extracted from the app binary. For instance, a function which detects a jailbreak can be located by searching for relevant keywords (e.g. "jailbreak"). The listing below shows the disassembled function `JailbreakDetectionViewController.jailbreakTest4Tapped` from the Damn Vulnerable iOS App (DVIA-v2). + +```assembly +__T07DVIA_v232JailbreakDetectionViewControllerC20jailbreakTest4TappedyypF: +stp x22, x21, [sp, #-0x30]! +mov rbp, rsp +``` + +After the obfuscation we can observe that the symbol’s name is no longer meaningful as shown on the listing below. + +```assembly +__T07DVIA_v232zNNtWKQptikYUBNBgfFVMjSkvRdhhnbyyFySbyypF: +stp x22, x21, [sp, #-0x30]! +mov rbp, rsp +``` + +Nevertheless, this only applies to the names of functions, classes and fields. The actual code remains unmodified, so an attacker can still read the disassembled version of the function and try to understand its purpose (e.g. to retrieve the logic of a security algorithm). + +#### Instruction Substitution + +This technique replaces standard binary operators like addition or subtraction with more complex representations. For example, an addition `x = a + b` can be represented as `x = -(-a) - (-b)`. However, using the same replacement representation could be easily reversed, so it is recommended to add multiple substitution techniques for a single case and introduce a random factor. This technique can be reversed during decompilation, but depending on the complexity and depth of the substitutions, reversing it can still be time consuming. + +#### Control Flow Flattening + +Control flow flattening replaces original code with a more complex representation. The transformation breaks the body of a function into basic blocks and puts them all inside a single infinite loop with a switch statement that controls the program flow. This makes the program flow significantly harder to follow because it removes the natural conditional constructs that usually make the code easier to read. + +![control-flow-flattening](./Images/Chapters/0x06j/control-flow-flattening.png) \ + +The image shows how control flow flattening alters code (see "[Obfuscating C++ programs via control flow flattening](http://ac.inf.elte.hu/Vol_030_2009/003.pdf)") + +#### Dead Code Injection + +This technique makes the program's control flow more complex by injecting dead code into the program. Dead code is a stub of code that doesn’t affect the original program’s behavior but increases the overhead of the reverse engineering process. + +#### String Encryption + +Applications are often compiled with hardcoded keys, licences, tokens and endpoint URLs. By default, all of them are stored in plaintext in the data section of an application’s binary. This technique encrypts these values and injects stubs of code into the program that will decrypt that data before it is used by the program. + +#### Packing + +[Packing](https://attack.mitre.org/techniques/T1027/002/) is a dynamic rewriting obfuscation technique which compresses or encrypts the original executable into data and dynamically recovers it during execution. Packing an executable changes the file signature in an attempt to avoid signature-based detection. + ### Debugging and Tracing In the traditional sense, debugging is the process of identifying and isolating problems in a program as part of the software development life cycle. The same tools used for debugging are valuable to reverse engineers even when identifying bugs is not the primary goal. Debuggers enable program suspension at any point during runtime, inspection of the process' internal state, and even register and memory modification. These abilities simplify program inspection. diff --git a/Document/0x04h-Testing-Code-Quality.md b/Document/0x04h-Testing-Code-Quality.md index fbbee0e028..8b07a8ba4c 100644 --- a/Document/0x04h-Testing-Code-Quality.md +++ b/Document/0x04h-Testing-Code-Quality.md @@ -271,6 +271,38 @@ Fuzz testing techniques or scripts (often called "fuzzers") will typically gener For more information on fuzzing, refer to the [OWASP Fuzzing Guide](https://owasp.org/www-community/Fuzzing "OWASP Fuzzing Guide"). +## Binary Protection Mechanisms + +### Position Independent Code + +[PIC (Position Independent Code)](https://en.wikipedia.org/wiki/Position-independent_code) is code that, being placed somewhere in the primary memory, executes properly regardless of its absolute address. PIC is commonly used for shared libraries, so that the same library code can be loaded in a location in each program address space where it does not overlap with other memory in use (for example, other shared libraries). + +PIE (Position Independent Executable) are executable binaries made entirely from PIC. PIE binaries are used to enable [ASLR (Address Space Layout Randomization)](https://en.wikipedia.org/wiki/Address_space_layout_randomization) which randomly arranges the address space positions of key data areas of a process, including the base of the executable and the positions of the stack, heap and libraries. + +### Memory management + +#### Automatic Reference Counting + +[ARC (Automatic Reference Counting)](https://en.wikipedia.org/wiki/Automatic_Reference_Counting) is a memory management feature of the Clang compiler exclusive to [Objective-C](https://developer.apple.com/library/content/releasenotes/ObjectiveC/RN-TransitioningToARC/Introduction/Introduction.html) and [Swift](https://docs.swift.org/swift-book/LanguageGuide/AutomaticReferenceCounting.html). ARC automatically frees up the memory used by class instances when those instances are no longer needed. ARC differs from tracing garbage collection in that there is no background process that deallocates the objects asynchronously at runtime. + +Unlike tracing garbage collection, ARC does not handle reference cycles automatically. This means that as long as there are "strong" references to an object, it will not be deallocated. Strong cross-references can accordingly create deadlocks and memory leaks. It is up to the developer to break cycles by using weak references. You can learn more about how it differs from Garbage Collection [here](https://fragmentedpodcast.com/episodes/064/). + +#### Garbage Collection + +[Garbage Collection (GC)](https://en.wikipedia.org/wiki/Garbage_collection_(computer_science)) is an automatic memory management feature of some languages such as Java/Kotlin/Dart. The garbage collector attempts to reclaim memory which was allocated by the program, but is no longer referenced—also called garbage. The Android runtime (ART) makes use of an [improved version of GC](https://source.android.com/devices/tech/dalvik#Improved_GC). You can learn more about how it differs from ARC [here](https://fragmentedpodcast.com/episodes/064/). + +#### Manual Memory Management + +[Manual memory management](https://en.wikipedia.org/wiki/Manual_memory_management) is typically required in native libraries written in C/C++ where ARC and GC do not apply. The developer is responsible for doing proper memory management. Manual memory management is known to enable several major classes of bugs into a program when used incorrectly, notably violations of [memory safety](https://en.wikipedia.org/wiki/Memory_safety) or [memory leaks](https://en.wikipedia.org/wiki/Memory_leak). + +More information can be found in ["Memory Corruption Bugs (MSTG-CODE-8)"](#memory-corruption-bugs-mstg-code-8). + +### Stack Smashing Protection + +[Stack canaries](https://en.wikipedia.org/wiki/Stack_buffer_overflow#Stack_canaries) help prevent stack buffer overflow attacks by storing a hidden integer value on the stack right before the return pointer. This value is then validated before the return statement of the function is executed. A buffer overflow attack often overwrites a region of memory in order to overwrite the return pointer and take over the program flow. If stack canaries are enabled, they will be overwritten as well and the CPU will know that the memory has been tampered with. + +Stack buffer overflow is a type of the more general programming vulnerability known as [buffer overflow](https://en.wikipedia.org/wiki/Buffer_overflow) (or buffer overrun). Overfilling a buffer on the stack is more likely to **derail program execution** than overfilling a buffer on the heap because the stack contains the return addresses for all active function calls. + ## References ### OWASP MASVS diff --git a/Document/0x05a-Platform-Overview.md b/Document/0x05a-Platform-Overview.md index bab56b9c37..72a6ab5a3a 100644 --- a/Document/0x05a-Platform-Overview.md +++ b/Document/0x05a-Platform-Overview.md @@ -28,6 +28,10 @@ The current version of Android executes this bytecode on the Android runtime (AR In the DVM, bytecode is translated into machine code at execution time, a process known as _just-in-time_ (JIT) compilation. This enables the runtime to benefit from the speed of compiled code while maintaining the flexibility of code interpretation. To further improve performance, Android introduced the [Android Runtime (ART)](https://source.android.com/devices/tech/dalvik/configure#how_art_works) to replace the DVM. ART uses a hybrid combination of _ahead-of-time_ (AOT), JIT and profile-guided compilation. Apps are recompiled on the device when they are installed, or when the OS undergoes a major update. While the code is being recompiled, device-specific and advanced code optimizations techniques can be applied. The final recompiled code is then used for all subsequent executions. AOT improves performance by a factor of two while reducing power consumption, due to the device-specific optimizations. +![OWASP MSTG](Images/Chapters/0x05a/java2oat.png) \ + +Source: + Android apps don't have direct access to hardware resources, and each app runs in its own virtual machine or sandbox. This enables the OS to have precise control over resources and memory access on the device. For instance, a crashing app doesn't affect other apps running on the same device. Android controls the maximum number of system resources allocated to apps, preventing any one app from monopolizing too many resources. At the same time, this sandbox design can be considered as one of the many principles in Android's global defense-in-depth strategy. A malicious third-party application, with low privileges, shouldn't be able to escape its own runtime and read the memory of a victim application on the same device. In the following section we take a closer look at the different defense layers in the Android operating system. ## Android Security: Defense-in-Depth Approach diff --git a/Document/0x05b-Basic-Security_Testing.md b/Document/0x05b-Basic-Security_Testing.md index 7b59b6377c..abc1e7a845 100644 --- a/Document/0x05b-Basic-Security_Testing.md +++ b/Document/0x05b-Basic-Security_Testing.md @@ -511,6 +511,26 @@ As seen above in "[Exploring the App Package](#exploring-the-app-package "Explor Refer to the section "[Reviewing Decompiled Java Code](0x05c-Reverse-Engineering-and-Tampering.md#reviewing-decompiled-java-code "Reviewing Decompiled Java Code")" in the chapter "[Tampering and Reverse Engineering on Android](0x05c-Reverse-Engineering-and-Tampering.md)" for more information about how to reverse engineer DEX files. +##### Compiled App Binary + +In some cases it might be useful to retrieve the compiled app binary (.odex). + +First get the path to the app's data directory: + +```bash +adb shell pm path com.example.myapplication +package:/data/app/~~DEMFPZh7R4qfUwwwh1czYA==/com.example.myapplication-pOslqiQkJclb_1Vk9-WAXg==/base.apk +``` + +Remove the `/base.apk` part, add `/oat/arm64/base.odex` and use the resulting path to pull the base.odex from the device: + +```bash +adb root +adb pull /data/app/~~DEMFPZh7R4qfUwwwh1czYA==/com.example.myapplication-pOslqiQkJclb_1Vk9-WAXg==/oat/arm64/base.odex +``` + +Note that the exact directory will be different based on your Android version. If the `/oat/arm64/base.odex` file can't be found, manually search in the directory returned by `pm path`. + ##### Native Libraries You can inspect the `lib` folder in the APK: diff --git a/Document/0x05c-Reverse-Engineering-and-Tampering.md b/Document/0x05c-Reverse-Engineering-and-Tampering.md index d9f5c1ac54..aef73f7ea5 100644 --- a/Document/0x05c-Reverse-Engineering-and-Tampering.md +++ b/Document/0x05c-Reverse-Engineering-and-Tampering.md @@ -22,9 +22,13 @@ Nevertheless, if the code has been purposefully obfuscated (or some tool-breakin #### Decompiling Java Code -If you don't mind looking at Smali instead of Java, you can simply [open your APK in Android Studio](https://developer.android.com/studio/debug/apk-debugger "Debug pre-built APKs") by clicking **Profile or debug APK** from the Welcome screen (even if you don't intend to debug it you can take a look at the smali code). +**Java Disassembled Code (smali):** -Alternatively you can use [apktool](0x08-Testing-Tools.md#apktool) to extract and disassemble resources directly from the APK archive and disassemble Java bytecode to Smali. apktool allows you to reassemble the package, which is useful for patching and applying changes to e.g. the Android Manifest. +If you want to inspect the app's smali code (instead of Java), you can [open your APK in Android Studio](https://developer.android.com/studio/debug/apk-debugger "Debug pre-built APKs") by clicking **Profile or debug APK** from the "Welcome screen" (even if you don't intend to debug it you can take a look at the smali code). + +Alternatively you can use [apktool](0x08-Testing-Tools.md#apktool) to extract and disassemble resources directly from the APK archive and disassemble Java bytecode to smali. apktool allows you to reassemble the package, which is useful for [patching](#patching-repackaging-and-re-signing) the app or applying changes to e.g. the Android Manifest. + +**Java Decompiled Code:** If you want to look directly into Java source code on a GUI, simply open your APK using [jadx](0x08-Testing-Tools.md#jadx) or [Bytecode Viewer](0x08-Testing-Tools.md#bytecode-viewer). @@ -299,7 +303,7 @@ Remember that in most of the cases, just using static analysis will not be enoug #### Reviewing Decompiled Java Code -Following the example from "Decompiling Java Code", we assume that you've successfully decompiled and opened the crackme app in IntelliJ. As soon as IntelliJ has indexed the code, you can browse it just like you'd browse any other Java project. Note that many of the decompiled packages, classes, and methods have weird one-letter names; this is because the bytecode has been "minified" with ProGuard at build time. This is a basic type of obfuscation that makes the bytecode a little more difficult to read, but with a fairly simple app like this one, it won't cause you much of a headache. When you're analyzing a more complex app, however, it can get quite annoying. +Following the example from ["Decompiling Java Code"](#decompiling-java-code), we assume that you've successfully decompiled and opened the crackme app in IntelliJ. As soon as IntelliJ has indexed the code, you can browse it just like you'd browse any other Java project. Note that many of the decompiled packages, classes, and methods have weird one-letter names; this is because the bytecode has been "minified" with ProGuard at build time. This is a basic type of [obfuscation](0x04c-Tampering-and-Reverse-Engineering.md#obfuscation) that makes the bytecode a little more difficult to read, but with a fairly simple app like this one, it won't cause you much of a headache. When you're analyzing a more complex app, however, it can get quite annoying. When analyzing obfuscated code, annotating class names, method names, and other identifiers as you go along is a good practice. Open the `MainActivity` class in the package `sg.vantagepoint.uncrackable1`. The method `verify` is called when you tap the "verify" button. This method passes the user input to a static method called `a.a`, which returns a boolean value. It seems plausible that `a.a` verifies user input, so we'll refactor the code to reflect this. diff --git a/Document/0x05i-Testing-Code-Quality-and-Build-Settings.md b/Document/0x05i-Testing-Code-Quality-and-Build-Settings.md index c00f9f4863..6749404d03 100644 --- a/Document/0x05i-Testing-Code-Quality-and-Build-Settings.md +++ b/Document/0x05i-Testing-Code-Quality-and-Build-Settings.md @@ -484,88 +484,59 @@ There are various steps to take: ### Overview -Because decompiling Java classes is trivial, applying some basic obfuscation to the release byte-code is recommended. [ProGuard](0x08-Testing-Tools.md#proguard) offers an easy way to shrink and obfuscate code and to strip unneeded debugging information from the byte-code of Android Java apps. It replaces identifiers, such as class names, method names, and variable names, with meaningless character strings. This is a type of layout obfuscation, which is "free" in that it doesn't impact the program's performance. +The tests used to detect the presence of [binary protection mechanisms](0x04h-Testing-Code-Quality.md#binary-protection-mechanisms) heavily depend on the language used for developing the application. -Since most Android applications are Java-based, they are [immune to buffer overflow vulnerabilities](https://owasp.org/www-community/vulnerabilities/Buffer_Overflow "Java Buffer Overflows"). Nevertheless, a buffer overflow vulnerability may still be applicable when you're using the Android NDK; therefore, consider secure compiler settings. +In general all binaries should be tested, which includes both the main app executable as well as all libraries/dependencies. However, on Android we will focus on native libraries since the main executables are considered safe as we will see next. -### Static Analysis +Android optimizes its Dalvik bytecode from the app DEX files (e.g. classes.dex) and generates a new file containing the native code, usually with an .odex, .oat extension. This [Android compiled binary](0x05b-Basic-Security_Testing.md#compiled-app-binary) is wrapped using the [ELF format](https://refspecs.linuxfoundation.org/elf/gabi4+/contents.html) which is the format used by Linux and Android to package assembly code. -If source code is provided, you can check the build.gradle file to see whether obfuscation settings have been applied. In the example below, you can see that `minifyEnabled` and `proguardFiles` are set. Creating exceptions to protect some classes from obfuscation (with `-keepclassmembers` and `-keep class`) is common. Therefore, auditing the ProGuard configuration file to see what classes are exempted is important. The `getDefaultProguardFile('proguard-android.txt')` method gets the default ProGuard settings from the `/tools/proguard/` folder. +The app's [NDK native libraries](0x05b-Basic-Security_Testing.md#native-libraries) also [use the ELF format](https://developer.android.com/ndk/guides/abis). -Further information on how to shrink, obfuscate, and optimize your app can be found in the [Android developer documentation](https://developer.android.com/studio/build/shrink-code "Shrink, obfuscate, and optimize your app"). +- [**PIE (Position Independent Executable)**](0x04h-Testing-Code-Quality.md#position-independent-code): + - Since Android 7.0 (API level 24), PIC compilation is [enabled by default](https://source.android.com/devices/tech/dalvik/configure) for the main executables. + - With Android 5.0 (API level 21), support for non-PIE enabled native libraries was [dropped](https://source.android.com/security/enhancements/enhancements50) and since then, PIE is [enforced by the linker](https://cs.android.com/android/platform/superproject/+/master:bionic/linker/linker_main.cpp;l=430). +- [**Memory management**](0x04h-Testing-Code-Quality.md#memory-management): + - Garbage Collection will simply run for the main binaries and there's nothing to be checked on the binaries themselves. + - Garbage Collection does not apply to Android native libraries. The developer is responsible for doing proper [manual memory management](0x04h-Testing-Code-Quality.md#manual-memory-management). See ["Memory Corruption Bugs (MSTG-CODE-8)"](#memory-corruption-bugs-mstg-code-8). +- [**Stack Smashing Protection**](0x04h-Testing-Code-Quality.md#stack-smashing-protection): + - Android apps get compiled to Dalvik bytecode which is considered memory safe (at least for mitigating buffer overflows). Other frameworks such as Flutter will not compile using stack canaries because of the way their language, in this case Dart, mitigates buffer overflows. + - It must be enabled for Android native libraries but it might be difficult to fully determine it. + - NDK libraries should have it enabled since the compiler does it by default. + - Other custom C/C++ libraries might not have it enabled. -> When you build you project using Android Studio 3.4 or Android Gradle plugin 3.4.0 or higher, the plugin no longer uses ProGuard to perform compile-time code optimization. Instead, the plugin works with the R8 compiler. R8 works with all of your existing ProGuard rules files, so updating the Android Gradle plugin to use R8 should not require you to change your existing rules. +Learn more: -R8 is the new code shrinker from Google and was introduced in Android Studio 3.3 beta. By default, R8 removes attributes that are useful for debugging, including line numbers, source file names, and variable names. R8 is a free Java class file shrinker, optimizer, obfuscator, and pre-verifier and is faster than ProGuard, see also an [Android Developer blog post for further details](https://android-developers.googleblog.com/2018/11/r8-new-code-shrinker-from-google-is.html "R8"). It is shipped with Android's SDK tools. To activate shrinking for the release build, add the following to build.gradle: +- [Android executable formats](https://lief-project.github.io/doc/latest/tutorials/10_android_formats.html) +- [Android runtime (ART)](https://source.android.com/devices/tech/dalvik/configure#how_art_works) +- [Android NDK](https://developer.android.com/ndk/guides) +- [Android linker changes for NDK developers](https://android.googlesource.com/platform/bionic/+/master/android-changes-for-ndk-developers.md) -```default -android { - buildTypes { - release { - // Enables code shrinking, obfuscation, and optimization for only - // your project's release build type. - minifyEnabled true - - // Includes the default ProGuard rules files that are packaged with - // the Android Gradle plugin. To learn more, go to the section about - // R8 configuration files. - proguardFiles getDefaultProguardFile( - 'proguard-android-optimize.txt'), - 'proguard-rules.pro' - } - } - ... -} -``` - -The file `proguard-rules.pro` is where you define custom ProGuard rules. With the flag `-keep` you can keep certain code that is not being removed by R8, which might otherwise produce errors. For example to keep common Android classes, as in our sample configuration `proguard-rules.pro` file: +### Static Analysis -```default -... --keep public class * extends android.app.Activity --keep public class * extends android.app.Application --keep public class * extends android.app.Service -... -``` +Test the app native libraries to determine if they have the PIE and stack smashing protections enabled. -You can define this more granularly on specific classes or libraries in your project with the [following syntax](https://developer.android.com/studio/build/shrink-code#configuration-files "Customize which code to keep"): +You can use [radare2's rabin2](0x08-Testing-Tools.md#radare2) to get the binary information. We'll use [r2pay-v1.0.apk](https://github.com/OWASP/owasp-mstg/blob/master/Crackmes/Android/Level_04/r2pay-v1.0.apk) as an example. -```default --keep public class MyClass -``` +All native libraries must have `canary` and `pic` both set to `true`. -### Dynamic Analysis +That's the case for `libnative-lib.so`: -If source code has not been provided, an APK can be decompiled to determine whether the codebase has been obfuscated. Several tools are available for converting DEX code to a JAR file (e.g. dex2jar). The JAR file can be opened with tools such as JD-GUI that can be used to make sure that class, method, and variable names are not human-readable. +```sh +rabin2 -I lib/x86_64/libnative-lib.so | grep -E "canary|pic" +canary true +pic true +``` -Below you can find a sample for an obfuscated code block: +But not for `libtool-checker.so`: -```java -package com.a.a.a; - -import com.a.a.b.a; -import java.util.List; - -class a$b - extends a -{ - public a$b(List paramList) - { - super(paramList); - } - - public boolean areAllItemsEnabled() - { - return true; - } - - public boolean isEnabled(int paramInt) - { - return true; - } -} +```sh +rabin2 -I lib/x86_64/libtool-checker.so | grep -E "canary|pic" +canary false +pic true ``` +In this example, `libtool-checker.so` must be recompiled with stack smashing protection support. + ## References ### OWASP MASVS diff --git a/Document/0x05j-Testing-Resiliency-Against-Reverse-Engineering.md b/Document/0x05j-Testing-Resiliency-Against-Reverse-Engineering.md index f58aa91428..1b6e5b5e3c 100644 --- a/Document/0x05j-Testing-Resiliency-Against-Reverse-Engineering.md +++ b/Document/0x05j-Testing-Resiliency-Against-Reverse-Engineering.md @@ -1000,24 +1000,129 @@ Refer to the "[Tampering and Reverse Engineering on Android](0x05c-Reverse-Engin ### Overview -Obfuscation is the process of transforming code and data to make it more difficult to comprehend. It is an integral part of every software protection scheme. What's important to understand is that obfuscation isn't something that can be simply turned on or off. Programs can be made incomprehensible, in whole or in part, in many ways and to different degrees. +The chapter ["Mobile App Tampering and Reverse Engineering"](0x04c-Tampering-and-Reverse-Engineering.md#obfuscation) introduces several well-known obfuscation techniques that can be used in mobile apps in general. -In the test case "Make Sure That Free Security Features Are Activated (MSTG-CODE-9)" in chapter "Code Quality and Build Settings of Android Apps", we describe a few basic obfuscation techniques that are commonly used on Android with R8 and Pro-Guard. +Android apps can implement some of those obfuscation techniques using different tooling. For example, [ProGuard](0x08-Testing-Tools.md#proguard) offers an easy way to shrink and obfuscate code and to strip unneeded debugging information from the bytecode of Android Java apps. It replaces identifiers, such as class names, method names, and variable names, with meaningless character strings. This is a type of layout obfuscation, which doesn't impact the program's performance. -### Effectiveness Assessment +> Decompiling Java classes is trivial, therefore it is recommended to always applying some basic obfuscation to the production bytecode. + +Learn more about Android obfuscation techniques: + +- ["Security Hardening of Android Native Code"](https://darvincitech.wordpress.com/2020/01/07/security-hardening-of-android-native-code/) by Gautam Arvind +- ["APKiD: Fast Identification of AppShielding Products"](https://github.com/enovella/cve-bio-enovella/blob/master/slides/APKiD-NowSecure-Connect19-enovella.pdf) by Eduardo Novella +- ["Challenges of Native Android Applications: Obfuscation and Vulnerabilities"](https://www.theses.fr/2020REN1S047.pdf) by Pierre Graux + +#### Using ProGuard + +Developers use the build.gradle file to enable obfuscation. In the example below, you can see that `minifyEnabled` and `proguardFiles` are set. Creating exceptions to protect some classes from obfuscation (with `-keepclassmembers` and `-keep class`) is common. Therefore, auditing the ProGuard configuration file to see what classes are exempted is important. The `getDefaultProguardFile('proguard-android.txt')` method gets the default ProGuard settings from the `/tools/proguard/` folder. + +Further information on how to shrink, obfuscate, and optimize your app can be found in the [Android developer documentation](https://developer.android.com/studio/build/shrink-code "Shrink, obfuscate, and optimize your app"). + +> When you build your project using Android Studio 3.4 or Android Gradle plugin 3.4.0 or higher, the plugin no longer uses ProGuard to perform compile-time code optimization. Instead, the plugin uses the R8 compiler. R8 works with all of your existing ProGuard rules files, so updating the Android Gradle plugin to use R8 should not require you to change your existing rules. + +R8 is the new code shrinker from Google and was introduced in Android Studio 3.3 beta. By default, R8 removes attributes that are useful for debugging, including line numbers, source file names, and variable names. R8 is a free Java class file shrinker, optimizer, obfuscator, and pre-verifier and is faster than ProGuard, see also an [Android Developer blog post for further details](https://android-developers.googleblog.com/2018/11/r8-new-code-shrinker-from-google-is.html "R8"). It is shipped with Android's SDK tools. To activate shrinking for the release build, add the following to build.gradle: + +```default +android { + buildTypes { + release { + // Enables code shrinking, obfuscation, and optimization for only + // your project's release build type. + minifyEnabled true + + // Includes the default ProGuard rules files that are packaged with + // the Android Gradle plugin. To learn more, go to the section about + // R8 configuration files. + proguardFiles getDefaultProguardFile( + 'proguard-android-optimize.txt'), + 'proguard-rules.pro' + } + } + ... +} +``` + +The file `proguard-rules.pro` is where you define custom ProGuard rules. With the flag `-keep` you can keep certain code that is not being removed by R8, which might otherwise produce errors. For example to keep common Android classes, as in our sample configuration `proguard-rules.pro` file: + +```default +... +-keep public class * extends android.app.Activity +-keep public class * extends android.app.Application +-keep public class * extends android.app.Service +... +``` + +You can define this more granularly on specific classes or libraries in your project with the [following syntax](https://developer.android.com/studio/build/shrink-code#configuration-files "Customize which code to keep"): + +```default +-keep public class MyClass +``` + +Obfuscation often carries a cost in runtime performance, therefore it is usually only applied to certain very specific parts of the code, typically those dealing with security and runtime protection. + +### Static Analysis + +[Decompile the APK](0x05c-Reverse-Engineering-and-Tampering.md#decompiling-java-code) and [review it](0x05c-Reverse-Engineering-and-Tampering.md#reviewing-decompiled-java-code) to determine whether the codebase has been obfuscated. -Attempt to decompile the bytecode, disassemble any included library files and try to understand it. When doing so, consider the following: +Below you can find a sample for an obfuscated code block: + +```java +package com.a.a.a; + +import com.a.a.b.a; +import java.util.List; + +class a$b + extends a +{ + public a$b(List paramList) + { + super(paramList); + } + + public boolean areAllItemsEnabled() + { + return true; + } + + public boolean isEnabled(int paramInt) + { + return true; + } +} +``` + +Here are some considerations: -- Obfuscation often carries a cost in runtime performance, therefore it might have been only applied to certain very specific parts of the code, typically those dealing with security and runtime protection. - Meaningful identifiers, such as class names, method names, and variable names, might have been discarded. - String resources and strings in binaries might have been encrypted. - Code and data related to the protected functionality might be encrypted, packed, or otherwise concealed. -- For native code, [libc APIs](https://man7.org/linux/man-pages/dir_section_3.html) (e.g open, read) might have been replaced with OS [syscalls](https://man7.org/linux/man-pages/man2/syscalls.2.html). -- Additional obfuscation techniques such as ["Control Flow Flattening"](https://github.com/obfuscator-llvm/obfuscator/wiki/Control-Flow-Flattening) or ["Bogus Control Flow"](https://github.com/obfuscator-llvm/obfuscator/wiki/Bogus-Control-Flow) might have been applied using e.g. [Obfuscator-LLVM](https://github.com/obfuscator-llvm/obfuscator "Obfuscator-LLVM"). -Some of these techniques are discussed and analyzed in the blog post ["Security hardening of Android native code"](https://darvincitech.wordpress.com/2020/01/07/security-hardening-of-android-native-code/) by Gautam Arvind. +For native code: + +- [libc APIs](https://man7.org/linux/man-pages/dir_section_3.html) (e.g open, read) might have been replaced with OS [syscalls](https://man7.org/linux/man-pages/man2/syscalls.2.html). +- [Obfuscator-LLVM](https://github.com/obfuscator-llvm/obfuscator "Obfuscator-LLVM") might have been applied to perform ["Control Flow Flattening"](https://github.com/obfuscator-llvm/obfuscator/wiki/Control-Flow-Flattening) or ["Bogus Control Flow"](https://github.com/obfuscator-llvm/obfuscator/wiki/Bogus-Control-Flow). + +Some of these techniques are discussed and analyzed in the blog post ["Security hardening of Android native code"](https://darvincitech.wordpress.com/2020/01/07/security-hardening-of-android-native-code/) by Gautam Arvind and in the ["APKiD: Fast Identification of AppShielding Products"](https://github.com/enovella/cve-bio-enovella/blob/master/slides/APKiD-NowSecure-Connect19-enovella.pdf) presentation by Eduardo Novella. + +For a more detailed assessment, you need a detailed understanding of the relevant threats and the obfuscation methods used. Tools such as [APKiD](0x08-Testing-Tools.md#apkid) may give you additional indications about which techniques were used for the target app such as obfuscators, packers and anti-debug measures. + +### Dynamic Analysis + +You can use [APKiD](0x08-Testing-Tools.md#apkid) to detect if the app has been obfuscated. + +Example using the [Android Crackme Level 4](https://github.com/OWASP/owasp-mstg/blob/master/Crackmes/Android/Level_04/r2pay-v1.0.apk): + +```sh +apkid owasp-mstg/Crackmes/Android/Level_04/r2pay-v1.0.apk +[+] APKiD 2.1.2 :: from RedNaga :: rednaga.io +[*] owasp-mstg/Crackmes/Android/Level_04/r2pay-v1.0.apk!classes.dex + |-> anti_vm : Build.TAGS check, possible ro.secure check + |-> compiler : r8 + |-> obfuscator : unreadable field names, unreadable method names +``` -For a more detailed assessment, you need a detailed understanding of the relevant threats and the obfuscation methods used. There are some tools such as [APKiD](https://github.com/rednaga/APKiD) that might be able to give you some indications about the kind of obfuscators being used. +In this case it detects that the app has unreadable field names and method names, among other things. ## Testing Device Binding (MSTG-RESILIENCE-10) diff --git a/Document/0x06b-Basic-Security-Testing.md b/Document/0x06b-Basic-Security-Testing.md index 20f2976732..57d59dbde9 100644 --- a/Document/0x06b-Basic-Security-Testing.md +++ b/Document/0x06b-Basic-Security-Testing.md @@ -663,9 +663,16 @@ Refer to the chapter [Tampering and Reverse Engineering on iOS](0x06c-Reverse-En ##### Native Libraries -iOS native libraries are known as Frameworks. +iOS apps can make their codebase modular by using different elements. In the MSTG we will refer to all of them as native libraries, but they can come in different forms: -You can easily visualize them from Passionfruit by clicking on "Modules": +- [Static and Dynamic Libraries](https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/OverviewOfDynamicLibraries.html#//apple_ref/doc/uid/TP40001873-SW1): + - Static Libraries can be used and will be compiled in the app binary. + - Dynamic Libraries (typically having the `.dylib` extension) are also used but must be part of a framework bundle. Standalone Dynamic Libraries are [not supported](https://developer.apple.com/library/archive/technotes/tn2435/_index.html#//apple_ref/doc/uid/DTS40017543-CH1-PROJ_CONFIG-APPS_WITH_DEPENDENCIES_BETWEEN_FRAMEWORKS) on iOS, watchOS, or tvOS, except for the system Swift libraries provided by Xcode. +- [Frameworks](https://developer.apple.com/library/archive/technotes/tn2435/_index.html#//apple_ref/doc/uid/DTS40017543-CH1-PROJ_CONFIG-APPS_WITH_DEPENDENCIES_BETWEEN_FRAMEWORKS) (since iOS 8). A Framework is a hierarchical directory that encapsulates a dynamic library, header files, and resources, such as storyboards, image files, and localized strings, into a single package. +- [Binary Frameworks (`XCFrameworks`)](https://developer.apple.com/videos/play/wwdc2019/416/): Xcode 11 supports distributing binary libraries using the `XCFrameworks` format which is a new way to bundle up multiple variants of a Framework, e.g. for any of the platforms that Xcode supports (including simulator and devices). They can also bundle up static libraries (and their corresponding headers) and support binary distribution of Swift and C-based code. `XCFrameworks` can be [distributed as Swift Packages](https://developer.apple.com/documentation/swift_packages/distributing_binary_frameworks_as_swift_packages). +- [Swift Packages](https://developer.apple.com/documentation/swift_packages): Xcode 11 add supports for Swift packages, which are reusable components of Swift, Objective-C, Objective-C++, C, or C++ code that developers can use in their projects and are distributed as source code. Since Xcode 12 they can also [bundle resources](https://developer.apple.com/videos/play/wwdc2020/10169/), such as images, storyboards, and other files. Since Package libraries are [static by default](https://developer.apple.com/videos/play/wwdc2019/408/?time=739). Xcode compiles them, and the packages they depend on, and then links and combines everything into the application. + +You can visualize native libraries in Passionfruit by clicking on "Modules": diff --git a/Document/0x06i-Testing-Code-Quality-and-Build-Settings.md b/Document/0x06i-Testing-Code-Quality-and-Build-Settings.md index 9b628a37d2..0fd9391d07 100644 --- a/Document/0x06i-Testing-Code-Quality-and-Build-Settings.md +++ b/Document/0x06i-Testing-Code-Quality-and-Build-Settings.md @@ -614,15 +614,26 @@ There are various well written explanations which can help with taking care of m ### Overview -Although Xcode enables all binary security features by default, it may be relevant to verify this for an old application or to check for the misconfiguration of compilation options. The following features are applicable: +The tests used to detect the presence of [binary protection mechanisms](0x04h-Testing-Code-Quality.md#binary-protection-mechanisms) heavily depend on the language used for developing the application. -- **ARC** - Automatic Reference Counting - A memory management feature that adds retain and release messages when required -- **Stack Canary** - Stack-smashing protection - Helps prevent buffer overflow attacks by means of having a small integer right before the return pointer. A buffer overflow attack often overwrites a region of memory in order to overwrite the return pointer and take over the process-control. In that case, the canary gets overwritten as well. Therefore, the value of the canary is always checked to make sure it has not changed before a routine uses the return pointer on the stack. -- **PIE** - Position Independent Executable - enables full ASLR for the executable binary (not applicable for libraries). +Although Xcode enables all binary security features by default, it may be relevant to verify this for old applications or to check for compiler flag misconfigurations. The following features are applicable: -Tests to detect the presence of these protection mechanisms heavily depend on the language used for developing the application. For example, existing techniques for detecting the presence of stack canaries do not work for pure Swift apps. For more details, please check the online article "[On iOS Binary Protections](https://sensepost.com/blog/2021/on-ios-binary-protections/ "On iOS Binary Protection")". +- [**PIE (Position Independent Executable)**](0x04h-Testing-Code-Quality.md#position-independent-code): + - PIE applies to executable binaries (Mach-O type `MH_EXECUTE`). + - However it's not applicable for libraries (Mach-O type `MH_DYLIB`). +- [**Memory management**](0x04h-Testing-Code-Quality.md#memory-management): + - Both pure Objective-C, Swift and hybrid binaries should have ARC (Automatic Reference Counting) enabled. + - For C/C++ libraries, the developer is responsible for doing proper [manual memory management](0x04h-Testing-Code-Quality.md#manual-memory-management). See ["Memory Corruption Bugs (MSTG-CODE-8)"](#memory-corruption-bugs-mstg-code-8). +- [**Stack Smashing Protection**](0x04h-Testing-Code-Quality.md#stack-smashing-protection): For pure Objective-C binaries, this should always be enabled. Since Swift is designed to be memory safe, if a library is purely written in Swift, and stack canaries weren’t enabled, the risk will be minimal. -### Static Analysis +Learn more: + +- [OS X ABI Mach-O File Format Reference](https://github.com/aidansteele/osx-abi-macho-file-format-reference) +- [On iOS Binary Protections](https://sensepost.com/blog/2021/on-ios-binary-protections/) +- [Security of runtime process in iOS and iPadOS](https://support.apple.com/en-gb/guide/security/sec15bfe098e/web) +- [Mach-O Programming Topics - Position-Independent Code](https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/MachOTopics/1-Articles/dynamic_code.html) + +Tests to detect the presence of these protection mechanisms heavily depend on the language used for developing the application. For example, existing techniques for detecting the presence of stack canaries do not work for pure Swift apps. #### Xcode Project Settings @@ -652,9 +663,9 @@ ARC is automatically enabled for Swift apps by the `swiftc` compiler. However, f See the [Technical Q&A QA1788 Building a Position Independent Executable](https://developer.apple.com/library/mac/qa/qa1788/_index.html "Technical Q&A QA1788 Building a Position Independent Executable"). -#### With otool +### Static Analysis -Below are procedures for checking the binary security features described above. All the features are enabled in these examples. +You can use [otool](0x08-Testing-Tools.md#otool) to check the binary security features described above. All the features are enabled in these examples. - PIE: diff --git a/Document/0x06j-Testing-Resiliency-Against-Reverse-Engineering.md b/Document/0x06j-Testing-Resiliency-Against-Reverse-Engineering.md index 726c9c3b06..1b8dbb1c32 100644 --- a/Document/0x06j-Testing-Resiliency-Against-Reverse-Engineering.md +++ b/Document/0x06j-Testing-Resiliency-Against-Reverse-Engineering.md @@ -641,7 +641,7 @@ However, this is not a concern on iOS. As discussed in the section [Testing on t ### Overview -Obfuscation is a process of transforming code into a form that is difficult to disassemble and understand and is an integral part of every software protection scheme. The application preserves the original functionality after obfuscation. What's important to understand is that obfuscation isn't something that can be simply turned on or off. Programs can be made incomprehensible, in whole or in part, in many ways and to different degrees. +The chapter ["Mobile App Tampering and Reverse Engineering"](0x04c-Tampering-and-Reverse-Engineering.md#obfuscation) introduces several well-known obfuscation techniques that can be used in mobile apps in general. > Note: All presented techniques below may not stop reverse engineers, but combining all of those techniques will make their job significantly harder. The aim of those techniques is to discourage reverse engineers from performing further analysis. @@ -698,7 +698,9 @@ Applications are often compiled with hardcoded keys, licences, tokens and endpoi - [SwiftShield](https://github.com/rockbruno/swiftshield) can be used to perform name obfuscation. It reads the source code of the Xcode project and replaces all names of classes, methods and fields with random values before the compiler is used. - [obfuscator-llvm](https://github.com/obfuscator-llvm) operates on the Intermediate Representation (IR) instead of the source code. It can be used for symbol obfuscation, string encryption and control flow flattening. Since it's based on IR, it can hide out significantly more information about the application as compared to SwiftShield. -### How to use SwiftShield +Learn more about iOS obfuscation techniques [here](https://faculty.ist.psu.edu/wu/papers/obf-ii.pdf). + +#### How to use SwiftShield > Warning: SwiftShield irreversibly overwrites all your source files. Ideally, you should have it run only on your CI server, and on release builds. @@ -764,7 +766,7 @@ This is needed for [deobfuscating encrypted crash logs](https://github.com/rockb Another example project is available in SwiftShield's [Github repo](https://github.com/rockbruno/swiftshield/tree/master/ExampleProject "SwiftShieldExample"), that can be used to test the execution of SwiftShield. -### Effectiveness Assessment +### Static Analysis Attempt to disassemble the Mach-O in the IPA and any included library files in the "Frameworks" directory (.dylib or .framework files), and perform static analysis. At the very least, the app's core functionality (i.e., the functionality meant to be obfuscated) shouldn't be easily discerned. Verify that: diff --git a/Document/0x08-Testing-Tools.md b/Document/0x08-Testing-Tools.md index 87f947c860..accc5c76e1 100644 --- a/Document/0x08-Testing-Tools.md +++ b/Document/0x08-Testing-Tools.md @@ -1114,6 +1114,17 @@ The official IDE for Google's Android operating system, built on JetBrains' Inte Android-SSL-TrustKiller is a Cydia Substrate Module acting as a blackbox tool to bypass SSL certificate pinning for most applications running on a device - +### APKiD + +[APKiD](https://github.com/rednaga/APKiD) gives you information about how an APK was made. It identifies many compilers, packers, obfuscators, and other weird stuff. + +For more information on what this tool can be used for, check out: + +- [Android Compiler Fingerprinting](http://hitcon.org/2016/CMT/slide/day1-r0-e-1.pdf) +- [Detecting Pirated and Malicious Android Apps with APKiD](http://rednaga.io/2016/07/31/detecting_pirated_and_malicious_android_apps_with_apkid/) +- [APKiD: PEiD for Android Apps](https://github.com/enovella/cve-bio-enovella/blob/master/slides/bheu18-enovella-APKID.pdf) +- [APKiD: Fast Identification of AppShielding Products](https://github.com/enovella/cve-bio-enovella/blob/master/slides/APKiD-NowSecure-Connect19-enovella.pdf) + ### Apktool [Apktool](https://github.com/iBotPeaches/Apktool) is used to unpack Android app packages (APKs). Simply unzipping APKs with the standard `unzip` utility leaves some files unreadable. `AndroidManifest.xml` is encoded into binary XML format which isn’t readable with a text editor. Also, the app resources are still packaged into a single archive file. diff --git a/Document/Images/Chapters/0x05a/java2oat.png b/Document/Images/Chapters/0x05a/java2oat.png new file mode 100644 index 0000000000..7b948c2262 Binary files /dev/null and b/Document/Images/Chapters/0x05a/java2oat.png differ