Demonstration of 2 bugs reproducible using UITest 2.2.7

Overview

This is a snapshot to reproduce two specific bugs more consistently. It's not recommended to use this version for everyday use.

Bug 1: App is not restarted between local tests on iOS

Behavior on iOS simulator

After initially running some tests with an app on the iOS simulator, if you start a new test run; the app is not automatically restarted as it normally would be in the default AppDataMode.

This can cause inconsistent test results, since queries in the 1st test executed will pass or fail based on the non-restarted app; rather than when the app has been restarted as will occur for subsequent tests, except for the very last test executed.

Xamarin.UITest will not restart the app on the iOS simulator for the last test executed. I suspect this may be intentional; so that a user viewing the simulator & completed tests will be able to directly see the final state of the app.

This doesn't directly interfere with testing, but since the 1st test doesn't reset it can cause the next test executed to have unexpected behavior due to this bug.

Behavior on iOS device

(Note: I have not explicitly reproduced this because I haven't had the chance to test this on a physical device, but I suspect the behavior is closely related or caused by the same issue we see in the simulator demo noted above)

Reported steps to reproduce:

1. Run a test
2. The app starts
3. The test runs and passes, but an error shows in the output window.
4. The app DOES NOT close
5. Run a new test
6. the app closes
7. The test runs but fails with an exception. The app does not start.
8. Repeat

Bug 2: App.Tap() cannot be performed when using ConnectToApp() on iOS

Description

I can reproduce the issue just by changing "StartApp()" to "ConnectToApp()" in UITestDemo, making sure the test app is launched on the target simulator; and then running a test that includes tapping.

There are some interesting additional observations when using the Repl() tool, specifically; if you set up this issue, I note that the tree command & app.Flash(x => x.Marked("Add")) command works normally as expected.

However, app.Tap(x => x.Marked("Add")) reproduces the issue, and in the Repl outputs a very large stacktrace. I've attached the output to this report as "raw stacktrace"

Because Tree & Flash work normally as expected; it seems like queries should be working, but I don't know much about how ConnectToApp() operates since I haven't used it many times in the past.

Demonstration of image rotation bug

Overview

This is a snapshot to reproduce a specific bug more consistently. It's not recommended to use this version for everyday use.

Description

Xamarin.UITest 2.2.6 is unable to detect nor interact with a Xamarin.Forms rotated image. The current format of the sample is set up to reproduce this issue, so it is expected the tests will fail with an error that the targeted element cannot be found:

        [Test]
        public void ImageRotationInteraction()
        {
            app.Tap(x => x.Marked("About"));
            app.Flash(x => x.Marked("logoWithClickEvent")); // will fail if image is rotated

        }

Comparison to normal behavior

If you comment out the rotation of the image on the AboutPage.Xaml.cs line 16:

https://github.com/King-of-Spades/AppCenter-Test-Samples/blob/image-rotation-experiment/Xamarin.UITest/UITestDemo/UITestDemo/Views/AboutPage.xaml.cs#L15-L16

Then run the test the test will pass. The element will also be visible to the REPL tool tree command.

Update & workaround

Given the age of this bug, I've tested and can reproduce it on the latest versions. However, we've discovered a fairly simple workaround as well. The cause of the issue is that when the image is rotated, Xamarin.UITest incorrectly determines it as being offscreen, so it can be found by appending .All() to an otherwise valid query for the element:

app.Tap(x=>x.All().Marked("logoWithClickEvent"));

Because .All() returns "all" matching elements, not just those currently visible in the UI, it's recommended that you check that the element you're searching for is either the 1st returned with this query at index zero (app.Tap() activates the 1st element found), and/or you add additional methods to make sure you're actually targeting the right element.

Complete samples for each supported framework

Overview

The repo now has completed and validated samples for Appium, Espresso, XCUITest, and Xamarin.UITest. In this snapshot generally everything should be working correctly; though we continue to monitor and maintain the samples in case breaking changes come through.

I'm also issuing this release in case something happens where we have a broken sample and aren't sure where to go back to to diagnose it.

Bug reproduction for UITests failing first test when iOS project is cleaned

Overview

This is a reproduction to confirm a bug where if an iOS project is cleaned, the first test run against it will fail with the following error:

SetUp : Xamarin.UITest.XDB.Exceptions.DeviceAgentException : DeviceAgent start session failed: An error occurred while sending the request
  ----> System.Net.Http.HttpRequestException : An error occurred while sending the request
  ----> System.Net.WebException : Error: ConnectFailure (Connection refused)
  ----> System.Net.Sockets.SocketException : Connection refused

Note: I do not know if this is the only cause of that particular error message.

Release for bug "VS 4 Mac beta - Socket error on UITest tests"

Overview

Drafting a release of the currently working solutions as a reference point for dotnet/android#1817

Adding Espresso Baseline sample

Appium (Android) & Xamarin.UITest samples are confirmed to be working consistently when set up properly. Espresso should work but still needs 3rd party confirmation that it will be set up properly on other systems.

Xamarin.UITest project complete

Overview

This release includes a Xamarin.Forms project for iOS & Android along with a Xamarin.UITest project & shell commands to upload to AppCenter/Test or testcloud.xamarin.com.

I'm tagging this as a pre-release until I've gotten confirmation from at least a few other users that they were successful using it.

Appium Android Sample complete

Overview

The Appium Android sample is finished and according to my testing should work for any users who have properly set up Appium locally. However I have not yet had anyone attempt to build or use this version of the sample; which is why I'm issuing it as a pre-release.

Also, this repo itself contains partially completed samples for the other test frameworks, which is why I'm designating this a 0.x release cycle at least until all the samples are complete in a basic form.