Merge pull request #214 from DP-3T/develop

SDK Version 2.0

CALIBRATION_APP_USAGE.md

@@ -0,0 +1,72 @@
+# How to use the calibration app for experiments
+
+## Setup
+
+Make sure that the calibration app is the only installed ExposureNotifications app (check Settings > Google > COVID-19 Exposure Notifications), otherwise clearing the exposure history is not possible over adb.
+
+### Make sure your device supports ExposureNotification v1.5
+Go to Settings > Google > COVID-19 exposure notifications and scroll to the bottom of the screen. Verify that the version number starts with 15... (Be aware that version 15xxxxxxxxx is newer than version 20xxxxxxx.)
+
+Alternatively you can use the following adb command:
+```
+adb shell dumpsys activity provider com.google.android.gms.chimera.container.GmsModuleProvider | grep 'nearby_en'
+```
+This will show you the EN version, but here the leading 15 for v1.5 is not printed. Known v1.5 versions are: v203004001, v202902002
+
+### Enable ExposureNotification Debug Mode
+Go to Settings > Google > COVID-19 exposure notifications > Debug mode and make sure that the following options are enabled:
+- Bypass app signature check
+- Return all TEKs immediately
+The option "Enable diagnosis key file signature check" should not be enabled.
+
+### Reset app and exposure history
+Run the following adb command to clear app and exposure history:
+```
+adb shell pm clear org.dpppt.android.calibration
+```
+Alternatively you can go to Settings > Apps > Calibration App and select "Clear data".
+
+### Configure app
+Set experiment name and device name with the following adb command (replace expName and devName):
+```
+adb shell am broadcast -a org.dpppt.android.calibration.adb --es experimentName "expName" --es deviceName "devName" -n org.dpppt.android.calibration/.handshakes.ADBBroadcastReceiver
+```
+Alternatively you can also open the app and enter experiment and device name in the parameters tab (second icon at the bottom).
+
+Now you have a clean setup of the app where device and experiment name are already set.
+
+
+## Run experiment
+To start an experiment open the app and press the red "START TRACING" button. An additional popup will show up where you have to agree to enable Exposure Notifications.
+
+At the end of the experiment go to the parameters tab (second icon at the bottom) and press "UPLOAD KEYS FOR EXPERIMENT". Make sure that the parameters from the setup phase are set correctly.
+
+To stop tracing and prevent any side-effects go back to the controls tab (first icon at the bottom) and press "STOP TRACING".
+
+## Execute matching
+
+Make sure tracing is enabled on the device for the matching to work (this is a restriction of the Google ExposureNotification framework). To do this, go to the controls tab (first icon at the bottom) of the app and press "START TRACING" if not already active.
+
+After all devices have uploaded their keys you have two options to execute the matching of a certain experiment:
+### Matching with ADB
+Run the following command to start the matching over adb (replace expName with the name of your experiment)
+```
+adb shell am broadcast -a org.dpppt.android.calibration.adb --es runMatching "expName" -n org.dpppt.android.calibration/.handshakes.ADBBroadcastReceiver
+```
+Check logcat to see when the matching finished (search for MatchingWorker), you will see the message "matching executed and uploaded successfully!".
+
+### Matching from the app
+Go to the handshakes tab (third icon at the bottom) of the app and you will see a list of all available experiments. Click an experiment to run the matching.
+The result will be displayed in the app but also uploaded to the server.
+
+### Check matching results
+The matching result will be automatically uploaded as JSON to the following URL:
+https://dp3tdemo.blob.core.windows.net/fileupload/result_experiment_{EXPERIMENT_NAME}_{DATE}_device_{DEVICE_NAME}.json
+Example:
+https://dp3tdemo.blob.core.windows.net/fileupload/result_experiment_1_2020-07-19_device_DD.json
+
+## Good to know
+
+- You can only execute one experiment and one (!) matching after a clean install. All subsequent experiments / matching runs will not work or lead to wrong results.
+
+- Battery optimization does not have to be deactivated for experiments, this is only needed to guarantee periodic backend syncs in the background, which is irrelevant for experiments. The error notification can thus be ignored.

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog for DP3T-SDK Android
 
+## version 2.0.0 (28.10.2020)
+
+- updated to play-services-nearby-exposurenotification-1.7.1-eap.aar, make sure to update this in your project as well!
+- use exposureWindows to compute attenuationDurations
+- exposureDays now returns all exposure days, not only the last one as in previous versions
+- updated defaults for attenuationBucketThresholds, new values are 55 and 63
+- add Experiment-Mode to calibration app to simplify experiments with multiple devices and the new ExposureWindows-API (see CALIBRATION_APP_USAGE.md for details)
+- expose EN-Module version, this can be used to extend the userAgent to be able to handle potential bugs in future EN versions from the backend
+- Version 2.0.0 of the SDK will require EN module version >= 1.6, if run on older versions a notification will be generated asking the user to update Google Play Services
+- add config option DP3T.setNumberOfDaysToConsiderForExposure() to define how many days after the exposure an exposure should be considered
+- add config option DP3T.setNumberOfDaysToKeepExposedDays() to define how many days after an exposure is reported, this should be kept
+
 ## version 1.0.5 (24.9.2020)
 
 - support location less scanning on Android 11

EXPOSURE_NOTIFICATION_API_USAGE.md

@@ -1,5 +1,5 @@
 # ExposureNotification API usage
-This document outlines the interaction of the SDK with the [Exposure Notification](https://www.google.com/covid19/exposurenotifications/) Framework by Google.
+This document outlines the interaction of the SDK with the [Exposure Notification](https://www.google.com/covid19/exposurenotifications/) Framework v1.5 by Google.
 
 ## Enabling Exposure Notifications
 
@@ -17,32 +17,39 @@ To check if Exposure Notifications are still enabled for our app (the user could
 
 To retrieve the Temporary Exposure Keys (TEKs) we need to call `exposureNotificationClient.getTemporaryExposureKeyHistory()`. This will result in an `ApiException` if Exposure Notifications are disabled. If enabled the method results in an `ApiException` that allows us to call `startResolutionForResult()` to trigger a system popup asking the user if he wants to share the TEKs of the last 14 days with the app. The result (user accepts or declines in the popup) is then returned as an activity result intent. If the user agrees to share the keys with the app in the popup the next call to `getTemporaryExposureKeyHistory()` will return the TEKs directly without an additional Exception or system popup.
 
-The TEK of the current day is never returned by `getTemporaryExposureKeyHistory()`, but only the keys of the previous 13 days. After the user agreed to share the keys we can call `getTemporaryExposureKeyHistory()` again on the following day and will then receive the TEK of the day the user agreed to share the keys as well. For this to work, Exposure Notifications must still be active for our app.
+There are two possibilities how the TEK of the current day can be retrieved. Which case is currently active is decided by a configuration of Google.
+### Same day TEK release with shortened rolling period
+In this case, the TEK of the current day and the previous 13 days are returned by `getTemporaryExposureKeyHistory()`. The TEK of the current day has a rolling period that is shorter than 24h, so this key is no longer valid after it was returned. If the device continues to have exposure notifications active a new TEK will be used on the same day. Therefore, it is possible that for the same date multiple TEKs are returned by `getTemporaryExposureKeyHistory()`, if the user allowed to export the keys already once or more in the previous 14 days.
+
+### Next day TEK release
+In this case, the TEK of the current day is not returned by `getTemporaryExposureKeyHistory()`, but only the keys of the previous 13 days. After the user agreed to share the keys we can call `getTemporaryExposureKeyHistory()` again on the following day and will then receive the TEK of the day the user agreed to share the keys as well. For this to work, Exposure Notifications must still be active for our app.
 
 ## Detecting Exposure
 
-For a contact to be counted as a possible exposure it must be longer than a certain number of minutes on a certain day. The current implementation of the EN-framework does not expose this information. Our way to overcome this limitation is to pass the published keys for each day individually to the framework.
+For a contact to be counted as a possible exposure it must be longer than a certain number of minutes on a certain day. The current implementation of the EN-framework does not expose this information directly, but as of version 1.5 of the exposure notification framework, we can use the new exposure windows feature to calculate the information ourselves.
 
-To check for exposure on a given day (we check the past 10 days) we need to call `exposureNotificationClient.provideDiagnosisKeys()`. This method has three parameters:
+To check for exposure we need to call `exposureNotificationClient.provideDiagnosisKeys()`. This method has only one parameter and takes a file list containing the TEKs.
 
-#### File list
-TEKs to check for exposure against must be provided in a [special file format](https://developers.google.com/android/exposure-notifications/exposure-key-file-format). The API would allow for multiple files being provided, but we always provide all available keys in a single file.
+These TEKs to check for exposure against must be provided in a [special file format](https://developers.google.com/android/exposure-notifications/exposure-key-file-format). The API would allow for multiple files being provided, but we always provide all available keys in a single file.
 
-#### Exposure Configuration
-The exposure configuration defines the configuration for the Google scoring of exposures. In our case we ignore most of the scoring methods and only provide the thresholds for the duration at attenuation buckets. The thresholds for the attenuation buckets are loaded from our [config server](https://github.com/DP-3T/dp3t-config-backend-ch/blob/master/dpppt-config-backend/src/main/java/org/dpppt/switzerland/backend/sdk/config/ws/model/GAENSDKConfig.java). This allows us to group the duration of a contact with another device into three buckets regarding the measured attenuation values that we then use to detect if the contact was long enough and close enough.
-To detect an exposure the following formula is used to compute the exposure duration:
-```
-durationAttenuationLow * factorLow + durationAttenuationMedium * factorMedium
-```
-If this duration is at least as much as defined in the `triggerThreshold` a notification is triggered for that day.
-
-#### Token
-Providing a token allows us to update an exposure check executed previously and only providing additional new TEKs in the file. The previously provided TEKs for the same token are stored internally by the framework and the new exposure result is the total exposure with all provided TEKs in the current and previous calls. When we download TEKs from our backend we receive a token (timestamp) that we can use for the next sync to only download the newly added TEKs. This reduces traffic between the app and the backend.
 
 ### Result
 
-The result of the `provideDiagnosisKeys()` call is provided as a broadcast with action `ExposureNotificationClient.ACTION_EXPOSURE_STATE_UPDATED`. In the Intent we directly get the `ExposureSummary` object, that allows us to check if the exposure limit for a notification was reached by checking the minutes of exposure per attenuation window. The duration per window has a maximum of 30 minutes, longer exposures are also returned as 30 minutes of exposure.
+The result of the `provideDiagnosisKeys()` call is provided as a broadcast with action `ExposureNotificationClient.ACTION_EXPOSURE_STATE_UPDATED`. After this we can call [`getExposureWindows()`](https://developers.google.com/android/exposure-notifications/exposure-notifications-api#exposurewindow) to get a list of `ExposureWindows` describing our current state of exposure.
+
+
+#### Calculation of exposure from ExposureWindows
+A `ExposureWindow` is a set of Bluetooth scan events from observed beacons within a timespan. A window contains multiple `ScanInstance` which are aggregations of attenuation of beacons during a scan.
+
+By grouping the ExposureWindows by day and then adding up all `secondsSinceLastScan` where `typicalAttenuationDb` lies between our defined attenuation thresholds we can compose three buckets.
+
+The thresholds for the attenuation buckets are loaded from our [config server](https://github.com/DP-3T/dp3t-config-backend-ch/blob/master/dpppt-config-backend/src/main/java/org/dpppt/switzerland/backend/sdk/config/ws/model/GAENSDKConfig.java).
+
+To detect an exposure the following formula is used to compute the exposure duration:
+```
+durationAttenuationLow * factorLow + durationAtttenuationMedium * factorMedium
+```
 
 ### Rate limit
 
-We are only allowed to call `provideDiagnosisKeys()` 20 times per UTC day. Because we check for every of the past 10 days individually, this allows us to check for exposure at most twice per day. These checks happen after 6am and 6pm (swiss time) when the SyncWorker is scheduled the next time or the app is opened. All 10 days are checked individually and if one fails it is retried on the next run. No checks are made between midnight UTC and 6am (swiss time) to prevent exceeding the rate limit per UTC day.
+We are only allowed to call `provideDiagnosisKeys()` 6 times per UTC day. Therefore, after an attempted call to `provideDiagnosisKeys()` we always wait 4h before doing the next call to guarantee to stay within the rate limit.

README.md

@@ -12,6 +12,8 @@ DP-3T is a free-standing effort started at EPFL and ETHZ that produced this prot
 ## Introduction
 This is the implementation of the DP-3T protocol using the [Exposure Notification](https://www.google.com/covid19/exposurenotifications/) Framework of Apple/Google. Only approved government public health authorities can access the APIs. Therefore, using this SDK will result in an API error unless either your account is whitelisted as test account or your app is approved by Google and signed with the production certificate.
 
+As of version 2.0 of this SDK we use features added in v1.5 of the Google framework and for iOS features that were added in v2.0 of the Apple framework. Make sure to also use at least version 2.0 of the [dp3t-sdk-backend](https://github.com/DP-3T/dp3t-sdk-backend) to be compatible with the DP3T Android SDK 2.0. See [EXPOSURE_NOTIFICATION_API_USAGE.md](EXPOSURE_NOTIFICATION_API_USAGE.md) for a detailed description of how we use the Google EN Api.
+
 Our prestandard solution that is not using the Apple/Google framework can be found under the [tag prestandard](https://github.com/DP-3T/dp3t-sdk-android/tree/prestandard).
 
 ## Repositories
@@ -27,6 +29,8 @@ The full set of documents for DP3T is at https://github.com/DP-3T/documents. Ple
 ## Calibration App
 Included in this repository is a Calibration App that can run, debug and test the SDK directly without implementing it in a new app first. It collects additional data and stores it locally into a database to allow for tests with phones from different vendors. Various parameters of the SDK are exposed and can be changed at runtime. Additionally it provides an overview of how to use the SDK.
 
+See [CALIBRATION_APP_USAGE.md ](CALIBRATION_APP_USAGE.md) for more information on how to use the calibration app.
+
 <p align="center">
   <img src="calibration-app/screenshots/1.png" width="256">
   <img src="calibration-app/screenshots/2.png" width="256">

calibration-app/app/build.gradle

@@ -30,8 +30,8 @@ android {
 		applicationId "org.dpppt.android.calibration"
 		minSdkVersion 23
 		targetSdkVersion 30
-		versionCode 2
-		versionName "0.2"
+		versionCode 3
+		versionName "0.3 (EN1.5)"
 
 		missingDimensionStrategy 'version', 'calibration'
 

calibration-app/app/src/main/AndroidManifest.xml

@@ -28,6 +28,14 @@
 			</intent-filter>
 		</activity>
 
+		<receiver
+			android:name=".handshakes.ADBBroadcastReceiver"
+			android:exported="true">
+			<intent-filter>
+				<action android:name="org.dpppt.android.calibration.adb" />
+			</intent-filter>
+		</receiver>
+
 	</application>
 
 </manifest>

calibration-app/app/src/main/java/org/dpppt/android/calibration/MainActivity.java

@@ -13,6 +13,7 @@
 import android.os.Bundle;
 import androidx.annotation.Nullable;
 import androidx.appcompat.app.AppCompatActivity;
+import androidx.fragment.app.Fragment;
 
 import com.google.android.material.bottomnavigation.BottomNavigationView;
 
@@ -74,6 +75,9 @@ protected void onActivityResult(int requestCode, int resultCode, @Nullable Inten
 		boolean handled = DP3T.onActivityResult(this, requestCode, resultCode, data);
 
 		if (!handled) {
+			for (Fragment fragment : getSupportFragmentManager().getFragments()) {
+				fragment.onActivityResult(requestCode, resultCode, data);
+			}
 			super.onActivityResult(requestCode, resultCode, data);
 		}
 	}

calibration-app/app/src/main/java/org/dpppt/android/calibration/MainApplication.java

@@ -48,7 +48,7 @@ public static void initDP3T(Context context) {
 						"RZ0FFdkxXZHVFWThqcnA4aWNSNEpVSlJaU0JkOFh2UgphR2FLeUg2VlFnTXV2Zk1JcmxrNk92QmtKeH" +
 						"dhbUdNRnFWYW9zOW11di9rWGhZdjF1a1p1R2RjREJBPT0KLS0tLS1FTkQgUFVCTElDIEtFWS0tLS0t");
 		DP3T.init(context,
-				new ApplicationInfo("org.dpppt.demo", BASE_URL, BASE_URL),
+				new ApplicationInfo(BASE_URL, BASE_URL),
 				signaturePublicKey);
 
 		if (!BuildConfig.DEBUG) {
@@ -58,12 +58,13 @@ public static void initDP3T(Context context) {
 			DP3T.setCertificatePinner(certificatePinner);
 		}
 
-		String userAgent = BuildConfig.APPLICATION_ID + ";" +
-				BuildConfig.VERSION_NAME + ";" +
-				BuildConfig.VERSION_CODE + ";" +
-				"Android;" +
-				Build.VERSION.SDK_INT;
-		DP3T.setUserAgent(userAgent);
+		DP3T.setUserAgent(() ->
+				BuildConfig.APPLICATION_ID + ";" +
+						BuildConfig.VERSION_NAME + ";" +
+						BuildConfig.VERSION_CODE + ";" +
+						"Android;" +
+						Build.VERSION.SDK_INT
+		);
 	}
 
 	@Override