[WIP] Add maze harness #246

fractalwrench · 2018-02-20T15:16:45Z

Adds a harnessed app for Maze Runner, and removes the old test harness module.

This also adds the boilerplate required to launch a test scenario that sends a payload.

…-android into add-maze-harness

coveralls · 2018-02-20T15:41:17Z

Pull Request Test Coverage Report for Build 1247

0 of 7 (0.0%) changed or added relevant lines in 1 file are covered.
105 unchanged lines in 8 files lost coverage.
Overall coverage decreased (-0.7%) to 70.447%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sdk/src/main/java/com/bugsnag/android/ExceptionHandler.java	0	7	0.0%

Files with Coverage Reduction	New Missed Lines	%
sdk/src/main/java/com/bugsnag/android/ExceptionHandler.java	1	46.34%
sdk/src/main/java/com/bugsnag/android/DefaultHttpClient.java	2	53.85%
sdk/src/main/java/com/bugsnag/android/DeviceData.java	3	77.44%
sdk/src/main/java/com/bugsnag/android/Error.java	4	79.67%
sdk/src/main/java/com/bugsnag/android/Configuration.java	4	92.06%
sdk/src/main/java/com/bugsnag/android/MetaData.java	6	93.0%
sdk/src/main/java/com/bugsnag/android/SessionTracker.java	42	55.64%
sdk/src/main/java/com/bugsnag/android/Client.java	43	53.81%

Totals
Change from base Build 1095:	-0.7%
Covered Lines:	1764
Relevant Lines:	2504

💛 - Coveralls

kattrali

Looks good, I'd make a few structural changes to better indicate what the new files are for. Both fakekeys.jks and mazerunner/ should be in features/fixtures/ since these files only exist for the sake of running the feature/ tests.

kattrali · 2018-02-20T15:47:05Z

Gemfile

@@ -0,0 +1,4 @@
+source "https://rubygems.org"
+
+gem 'bugsnag-maze-runner', path: '../../..'


You'll need to swap this with the git path:

- gem 'bugsnag-maze-runner', path: '../../..' + gem 'bugsnag-maze-runner', git: 'git@github.com:bugsnag/maze-runner'

kattrali · 2018-02-20T15:50:44Z

settings.gradle

@@ -1 +1 @@
-include ':example', ':testharness', ':sdk', ':examplelib'


The maze is likely to get pretty big. I'd think we wouldn't want to build/test it by default.

Running ./gradlew clean :mazerunner:assemble --profile currently completes in ~15s. I'd agree that we should keep an eye on how long it takes to build as we add more scenarios, but think that there's currently still a benefit to building this on CI as it will detect any compilation errors.

On CI, sure. I thought it was odd/long to run gradle build locally and automatically run the examples.

kattrali · 2018-02-23T23:56:11Z

Started merging these down so the larger changes make sense, but may have caused some conflicts as the test scenario generation signature seems to have changed and I'm not sure which one is right.

fractalwrench · 2018-02-27T17:48:30Z

Resolved the Scenario signature update to include the config + context (which allows for initialising Bugsnag with our own params before each scenario starts). Other comments still need to be addressed.

…-android into add-maze-harness

kattrali

Quick summary now that everything is merged up:

All scenarios should have names describing exactly what is being tested, i.e. "Attaching metadata to a report from a global callback"
Related scenarios should be merged into features "Reporting handled exceptions", "Filtering metadata", "Session tracking", "Reporting unhandled exceptions", "Release stages", "Recording user information", etc
Test-only projects and files should be moved out of the root of the repository into features/fixtures

fractalwrench · 2018-03-14T15:29:13Z

I have grouped the scenarios together which I thought made sense, it'd be good to confirm there aren't any others that would fit together. I've also moved the test files into features/fixtures and updated the scenario names.

kattrali · 2018-03-15T09:56:13Z

features/breadcrumb.feature

@@ -0,0 +1,18 @@
+Feature: Reporting Breadcrumbs
+
+Scenario: Test handled Android Exception


Missing a scenario name here

Updated to "Manually added breadcrumbs are sent in report"

kattrali · 2018-03-15T09:57:35Z

features/filtering_metadata.feature

+    And the event "metaData.custom.password" equals "[FILTERED]"
+    And the event "metaData.user.password" equals "[FILTERED]"
+
+Scenario: Manual Filter Tracking


What is manual filter tracking? Adding a custom metadata filter?

That's a much better name - I've updated the scenario title.

kattrali · 2018-03-15T10:45:21Z

features/strict_mode.feature

+    And the request is a valid for the error reporting API
+    And the exception "errorClass" equals "android.os.StrictMode$StrictModeViolation"
+    And the exception "message" equals "policy=262148 violation=4"
+    And the event "metaData.StrictMode.Violation" equals "NetworkOperation"


I occasionally get this:

Scenario: StrictMode Network on Main Thread violation # features/strict_mode.feature:12 When I run "StrictModeNetworkScenario" with the defaults # features/steps/build_steps.rb:1 Then I should receive a request # maze-runner-65bdcb7a135e/lib/features/steps/request_assertion_steps.rb:11 And the request is a valid for the error reporting API # maze-runner-65bdcb7a135e/lib/features/steps/error_reporting_steps.rb:1 the field 'events' must be a non-empty array (Minitest::Assertion) features/strict_mode.feature:15:in `And the request is a valid for the error reporting API' And the exception "errorClass" equals "android.os.StrictMode$StrictModeViolation" # maze-runner-65bdcb7a135e/lib/features/steps/error_reporting_steps.rb:47 And the exception "message" equals "policy=262148 violation=4" # maze-runner-65bdcb7a135e/lib/features/steps/error_reporting_steps.rb:47 And the event "metaData.StrictMode.Violation" equals "NetworkOperation" # maze-runner-65bdcb7a135e/lib/features/steps/error_reporting_steps.rb:25

Request output:

URI: http://10.0.2.2:9291/ HEADERS: content-type: 'application/json' bugsnag-sent-at: '2018-03-15T10:40:21Z' bugsnag-api-key: 'a35a2a72bd230ac0aa0f52715bbdc6aa' bugsnag-payload-version: '4.0' user-agent: 'Dalvik/2.1.0 (Linux; U; Android 7.0; Android SDK built for x86_64 Build/NYC)' host: '10.0.2.2:9291' connection: 'Keep-Alive' accept-encoding: 'gzip' BODY: { "apiKey": "a35a2a72bd230ac0aa0f52715bbdc6aa", "payloadVersion": "4.0", "notifier": { "name": "Android Bugsnag Notifier", "version": "4.3.1", "url": "https://bugsnag.com" }, "events": [ ] }

Any idea what's going on here?

Hm, not 100% sure, but could this be another manifestation of #270? If the errorFile is deleted prior to request serialisation then an empty array of events would be expected. Is only 1 request received by this scenario?

(I tried running this around 5 times and it seemed to pass consistently on my setup)

It seems to be inconsistent and could potentially be related to #270 - there seems to exist a case where the payload is sent but no events are present.

…-android into add-maze-harness

fractalwrench · 2018-03-19T09:12:03Z

@kattrali are there any further actions on this one? I've been unable to reproduce the StrictMode issue.

Increase test matrix for E2E scenarios

feat (NativeInterface): add SetAutoDetectAnrs method to Bugsnag interface

$fractalwrench$

fractalwrench added 7 commits February 19, 2018 17:00

$@fractalwrench$

remove unused testharness module

f9d64db

$@fractalwrench$

add basic mazerunner app

0e986ae

$@fractalwrench$

setup features

4c0078e

$@fractalwrench$

remove features which will be moved to global lib

319753f

$@fractalwrench$

remove features which will be moved to global lib

018deff

$@fractalwrench$

Merge branch 'add-maze-harness' of https://github.com/bugsnag/bugsnag…

1459233

…-android into add-maze-harness

$@fractalwrench$

use same gradle deps

83883c2

$@fractalwrench$ fractalwrench requested a review from kattrali February 20, 2018 15:16

$@fractalwrench$

reduce amount of code required by using reflection to load scenario

c28a4ae

kattrali reviewed Feb 20, 2018

View reviewed changes

fractalwrench added 7 commits February 22, 2018 12:09

$@fractalwrench$

Add unhandled exception scenarios (#249)

e47f902

$@fractalwrench$

Add handled exception scenario for java/kotlin (#247)

bf33f2a

$@fractalwrench$

Add ignored exception and disabled auto notify scenarios (#252)

4412d68

$@fractalwrench$

Add scenarios for different combinations of release stages (#253)

839b33c

$@fractalwrench$

Add scenarios for user metadata (#251)

d0ff588

$@fractalwrench$

Add Strictmode scenarios (#254)

045d97c

$@fractalwrench$

Add unusual crash scenarios (#250)

3d2e2fb

$@fractalwrench$

update scenario signature

2b8f528

$@fractalwrench$ fractalwrench changed the title ~~Add maze harness~~ [WIP] Add maze harness Feb 27, 2018

fractalwrench added 2 commits February 28, 2018 14:45

$@fractalwrench$

fix bad merge for unhandled

f9a2eb8

$@fractalwrench$

increase wait for stackoverflow scenario

9545c4f

$@fractalwrench$ fractalwrench mentioned this pull request Mar 13, 2018

Add metadata scenario #261

Merged

fractalwrench added 5 commits March 13, 2018 17:23

$@fractalwrench$

add breadcrumb scenario

701e766

$@fractalwrench$

update gemfile to point at mazerunner master

fa48804

$@fractalwrench$

Merge branch 'add-maze-harness' of https://github.com/bugsnag/bugsnag…

a407a1d

…-android into add-maze-harness

$@fractalwrench$

test app version override

db99752

$@fractalwrench$

add cache scenario for report

604b20c

fractalwrench added 4 commits March 13, 2018 17:37

$@fractalwrench$

add manual/auto context scenarios

a36a984

$@fractalwrench$

Add Session scenarios (#255)

4383bf2

$@fractalwrench$

add scenarios for manual + auto filtering of sensitive data (#262)

e871300

$@fractalwrench$

Add metadata scenario (#261)

d845ee6

kattrali reviewed Mar 13, 2018

View reviewed changes

fractalwrench added 2 commits March 14, 2018 14:34

$@fractalwrench$

move mazerunner keystore to fixtures dir

6a55d23

$@fractalwrench$

improve scenario descriptions, group related features together

822968f

kattrali reviewed Mar 15, 2018

View reviewed changes

fractalwrench added 4 commits March 15, 2018 13:20

$@fractalwrench$

Add scenarios for asynchronously flushing error store (#274)

97954e4

$@fractalwrench$

update wording for custom metadata filter scenario

dadc133

$@fractalwrench$

update scenario name for breadcrumbs

0f69097

$@fractalwrench$

Merge branch 'add-maze-harness' of https://github.com/bugsnag/bugsnag…

1bb6f6a

…-android into add-maze-harness

kattrali merged commit e5589c7 into master Mar 20, 2018

kattrali deleted the add-maze-harness branch March 20, 2018 08:49

lemnik pushed a commit that referenced this pull request Jun 2, 2021

$@fractalwrench$

Merge pull request #246 from bugsnag/v5-fixture-test-matrix

694835a

Increase test matrix for E2E scenarios

rich-bugsnag added a commit that referenced this pull request Sep 3, 2021

Merge pull request #246 from bugsnag/6501-SetAutoDetectAnrs-method-added

f8b7f44

feat (NativeInterface): add SetAutoDetectAnrs method to Bugsnag interface

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add maze harness #246

[WIP] Add maze harness #246

$@fractalwrench$ fractalwrench commented Feb 20, 2018

coveralls commented Feb 20, 2018 •

edited

Loading

kattrali left a comment

kattrali Feb 20, 2018

kattrali Feb 20, 2018

$@fractalwrench$ fractalwrench Feb 28, 2018

kattrali Mar 5, 2018

kattrali commented Feb 23, 2018

fractalwrench commented Feb 27, 2018

kattrali left a comment

fractalwrench commented Mar 14, 2018

kattrali Mar 15, 2018

$@fractalwrench$ fractalwrench Mar 16, 2018

kattrali Mar 15, 2018

$@fractalwrench$ fractalwrench Mar 16, 2018

kattrali Mar 15, 2018

$@fractalwrench$ fractalwrench Mar 16, 2018

$@fractalwrench$ fractalwrench Mar 16, 2018

kattrali Mar 16, 2018

fractalwrench commented Mar 19, 2018

		@@ -0,0 +1,4 @@
		source "https://rubygems.org"

		gem 'bugsnag-maze-runner', path: '../../..'

		@@ -1 +1 @@
		include ':example', ':testharness', ':sdk', ':examplelib'

		@@ -0,0 +1,18 @@
		Feature: Reporting Breadcrumbs

		Scenario: Test handled Android Exception

[WIP] Add maze harness #246

[WIP] Add maze harness #246

Conversation

fractalwrench commented Feb 20, 2018

coveralls commented Feb 20, 2018 • edited Loading

Pull Request Test Coverage Report for Build 1247

💛 - Coveralls

kattrali left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kattrali commented Feb 23, 2018

fractalwrench commented Feb 27, 2018

kattrali left a comment

Choose a reason for hiding this comment

fractalwrench commented Mar 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fractalwrench commented Mar 19, 2018

$@fractalwrench$ fractalwrench commented Feb 20, 2018

coveralls commented Feb 20, 2018 •

edited

Loading