Manual testing on RTN14 and RTN15 #27

tiholic · 2020-07-31T14:42:34Z

This PR aims to acknowledge RTN14 and RTN15 are complied with in this package, however, few points are skipped considering nascent stage of this package.

TL;DR at bottom of this

RTN14a ✅

Actions done

Create realtime using an Invalid API Key
attach connectionStateChange listeners and channelStateChange listeners
connect to realtime instance using realtime.connect()

2020-07-31 17:31:29.424647 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 17:31:30.115235 : ConnectionStateChange event: ConnectionEvent.failed
                             Reason: ErrorInfo message=Invalid key in request: asdf_d231:fawefsd-afwesd. (See https://help.ably.io/error/40005 for help.) code=40005 statusCode=400 href=https://help.ably.io/error/40005

RTN14b, RTN14g ⛔

Ignoring as these spec items for now as they revolve around ProtocolMessage

RTN14c, RTN14d, RTN14e, RTN14f ✅

Turning wifi off - reconnect triggers every 15 seconds

loops between states: connecting and disconnected every 15s for 60s (i.e., until the default connectionStateTtl)
there-further, state changes between connecting and suspended continuously.

2020-07-31 16:44:48.912345 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 16:44:48.965277 : ConnectionStateChange event: ConnectionEvent.disconnected
                             Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000
2020-07-31 16:45:03.950481 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 16:45:04.014063 : ConnectionStateChange event: ConnectionEvent.disconnected
                             Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000
2020-07-31 16:45:18.988102 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 16:45:19.062352 : ConnectionStateChange event: ConnectionEvent.disconnected
                             Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000
2020-07-31 16:45:34.020740 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 16:45:34.109471 : ConnectionStateChange event: ConnectionEvent.disconnected
                             Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000
2020-07-31 16:45:49.056232 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 16:45:49.127278 : ConnectionStateChange event: ConnectionEvent.suspended
                             Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000
2020-07-31 16:46:49.101562 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 16:46:49.178212 : ConnectionStateChange event: ConnectionEvent.suspended
                             Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000
2020-07-31 16:47:49.139349 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 16:47:49.213962 : ConnectionStateChange event: ConnectionEvent.suspended
                             Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000

RTN15h, RTN15h1, RTN15h2 ✅

Temporary key ✅

Created a temporary key on web console and deleted that after connection is established.
Connection disconnected, and eventually status changed to failed after 1 retry.

2020-07-31 18:25:05.070597 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 18:25:05.296538 : ConnectionStateChange event: ConnectionEvent.connected
                             Reason: null
2020-07-31 18:25:12.123308 : ConnectionStateChange event: ConnectionEvent.disconnected
                             Reason: ErrorInfo message=Key/token status changed (revoke). (See https://help.ably.io/error/40131 for help.) code=40131 statusCode=401 href=https://help.ably.io/error/40131
2020-07-31 18:25:12.126130 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 18:25:12.408257 : ConnectionStateChange event: ConnectionEvent.failed
                             Reason: ErrorInfo message=Key revoked. (See https://help.ably.io/error/40131 for help.) code=40131 statusCode=401 href=https://help.ably.io/error/40131

Temporary app ✅

Created a temporary app on web console and deleted that after connection is established.
Connection disconnected, and eventually status changed to failed after 1 retry.

2020-07-31 18:14:26.818151 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 18:14:27.057491 : ConnectionStateChange event: ConnectionEvent.connected
                             Reason: null
2020-07-31 18:14:51.896204 : ConnectionStateChange event: ConnectionEvent.disconnected
                             Reason: ErrorInfo message=Application lEu0Vg disabled. (See https://help.ably.io/error/40300 for help.) code=40300 statusCode=403 href=https://help.ably.io/error/40300
2020-07-31 18:14:52.612866 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 18:14:52.926823 : ConnectionStateChange event: ConnectionEvent.failed
                             Reason: ErrorInfo message=Application lEu0Vg disabled. (See https://help.ably.io/error/40300 for help.) code=40300 statusCode=403 href=https://help.ably.io/error/40300

Restricted API key ✅

Trying to attach to a channel with an API key with restricted access

2020-07-31 19:40:59.586416 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 19:41:00.463867 : ConnectionStateChange event: ConnectionEvent.connected
                             Reason: null
//Here, trying to attach to channel that is not allowed with this API key
                             Attaching to channel test-channel: Current state ChannelState.initialized
                             ChannelStateChange: ChannelState.attaching
                             Reason: null
                             ChannelStateChange: ChannelState.suspended
                             Reason: ErrorInfo message=Channel denied access based on given capability; channelId = test-channel. (See https://help.ably.io/error/40160 for help.) code=40160 statusCode=401 href=https://help.ably.io/error/40160
                             Unable to attach to channel: ErrorInfo message=Channel denied access based on given capability; channelId = test-channel. (See https://help.ably.io/error/40160 for help.) code=40160 statusCode=401 href=https://help.ably.io/error/40160

Un-restricting the above API Key, and manually re-connecting ❓

The above, with updated API key settings to remove restrictions and trying a fresh connection again by realtime.connect() on same realtime instance:

2020-07-31 19:45:58.041846 : ConnectionStateChange event: ConnectionEvent.connecting
                             Reason: null
2020-07-31 19:45:58.315391 : ConnectionStateChange event: ConnectionEvent.connected
                             Reason: ErrorInfo message=Connection unavailable code=80002 statusCode=503 href=https://help.ably.io/error/80002
                             Attaching to channel test-channel: Current state ChannelState.suspended

⚠️ ⚠️ ⚠️ ⚠️ ⚠️
ConnectionState changes to connected but with a reason "Connection Unavailable" ❓
This sounds and feels fishy 🎣

If a fresh realtime instance is used to connect, it connects as expected.

Another observation ❗

If a channel is already connected, it can publish and subscribe to/from that channel even if API Key settings are updated with new restricted permissions. This feels a bit concerning to me, but could be technically costly.

RTN15a ❓

Token has expired => not sure how to create this scenario

RTN15g (RTN15g1, RTN15g2, RTN15g3) ✅

These items are closely related with logs shared under heading RTN14c, RTN14d, RTN14e, RTN14f above

RTN15i, RTN15b (RTN15b1, RTN15b2), RTN15c (RTN15c1, RTN15c2, RTN15c3, RTN15c4), RTN15e ⛔

Ignoring as these spec items for now as they revolve around ProtocolMessage

RTN15f ⛔

Ignoring as these spec items for now as they revolve around ACK and NACK

RTN15d ⛔

Integration Tests - unsure of the priority on integration tests at the moment.

TL;DR ⌚ 🏃‍♂️

Temporary connection issues ✅

Cause:

bad network

What's happening:

Automatic connection retry every 15 seconds
this connection goes into disconnected state until connectionStateTtl is reached
After connectionStateTtl is reached, connection goes into suspended state while connection re-establishment happens in background.

Permanent connection issues ✅

Cause:

permanent API Key expiration (deletion of API key from web console)
permanent App expiration (deletion of app from web console)
Restricted API key (adding restrictions on API key)

What's happening?

On permanent API key expiration and (2) On permanent App expiration

ConnectionState changes in the following order and halts: connecting -> connected -> disconnected -> connecting -> failed

On restricting API Key

Connection succeeds initially, and when attaching to a channel that is restricted with the API key, it fails with code 40160 and ChannelState changes to suspended while ConnectionState still stays connected

Additional Scenario: ❓

Un-restricting the restricted API Key - Connection on already available Realtime instance would not work. It requires us to use a fresh Realtime instance to be able to operate normally again.

example/lib/main.dart

example/lib/neseted-realtime-events.dart

#27 (comment)

after testing for Connection Failure and Connection Resume spec

#27 (comment)

QuintinWillison

Minor comment. Happy for you to resolve, or leave as is, as you see fit.

QuintinWillison · 2020-09-10T15:02:10Z

ManualTesting.md

+and feasible tests can be converted to automated tests. 
+
+This document currently targets **RTN14** and **RTN15**
+ sections of the [spec](https://docs.ably.io/client-lib-development-guide/features/#idl), partially


I'm not quite sure why this 'deep link' goes to the Interface Definition. I would suggest making both RTN14 and RTN15 above direct links to their spec points (i.e. RTN14 and RTN15).

Though I also acknowledge that you've linked titles below.

Done: 5c9c04e

tiholic · 2020-09-17T17:51:27Z

@QuintinWillison I'd like to merge this and discuss on the results later as they are already added to ManualTesting.md

#27 (comment)

tiholic requested review from QuintinWillison, mattheworiordan and paddybyers July 31, 2020 14:42

tiholic force-pushed the feature/realtime-publish branch from 9178040 to 9949795 Compare August 3, 2020 10:44

tiholic force-pushed the feature/connection-resume-failure branch from 5f5a8cb to 0a85c32 Compare August 3, 2020 10:46

tiholic requested a review from zoechi August 3, 2020 11:27

zoechi suggested changes Aug 3, 2020

View reviewed changes

example/lib/main.dart Show resolved Hide resolved

example/lib/main.dart Outdated Show resolved Hide resolved

example/lib/main.dart Show resolved Hide resolved

example/lib/neseted-realtime-events.dart Outdated Show resolved Hide resolved

zoechi self-requested a review August 4, 2020 19:14

tiholic mentioned this pull request Aug 4, 2020

Realtime Channel attach, detach and state events #20

Merged

tiholic force-pushed the feature/realtime-publish branch from 9949795 to c727e99 Compare August 4, 2020 20:19

tiholic force-pushed the feature/connection-resume-failure branch from 0a85c32 to 4af537a Compare August 4, 2020 20:29

tiholic added a commit that referenced this pull request Aug 4, 2020

using var and removing brittle code

86c4e6b

#27 (comment)

zoechi approved these changes Aug 5, 2020

View reviewed changes

QuintinWillison removed their request for review August 10, 2020 08:28

tiholic force-pushed the feature/realtime-publish branch from 59ff55b to 176d8c1 Compare August 12, 2020 15:13

tiholic added a commit that referenced this pull request Aug 18, 2020

using var and removing brittle code

df48b87

#27 (comment)

tiholic force-pushed the feature/connection-resume-failure branch from c61e063 to 5565a5f Compare August 18, 2020 05:51

Base automatically changed from feature/realtime-publish to main August 18, 2020 07:22

QuintinWillison requested review from QuintinWillison and removed request for mattheworiordan and paddybyers September 10, 2020 14:58

tiholic added 9 commits September 10, 2020 21:15

simplify example

3da1e14

after testing for Connection Failure and Connection Resume spec

using long conversion util instead of blind typecast >_<

f3d7dcc

using var and removing brittle code

cd0b0b3

#27 (comment)

moving realtime into it's try-catch nest!

b8b0b3e

optimizing example

04513ca

rename file [typo]

b3b118c

update file status comment

28b6723

Added ManualTesting.md

3d5de14

Added ManualTesting.md

4a104b3

tiholic added 2 commits September 10, 2020 21:15

Added ManualTesting.md

4d8f34f

Added ManualTesting.md

34db0d2

tiholic force-pushed the feature/connection-resume-failure branch from d992307 to 34db0d2 Compare September 10, 2020 15:45

QuintinWillison approved these changes Sep 10, 2020

View reviewed changes

update links

5c9c04e

zoechi approved these changes Sep 11, 2020

View reviewed changes

zoechi approved these changes Sep 12, 2020

View reviewed changes

tiholic merged commit 8702782 into main Sep 22, 2020

tiholic added a commit that referenced this pull request Sep 22, 2020

using var and removing brittle code

846f9e2

#27 (comment)

tiholic deleted the feature/connection-resume-failure branch September 22, 2020 11:21

tiholic added a commit that referenced this pull request Nov 3, 2020

using var and removing brittle code

1830330

#27 (comment)

tiholic added a commit that referenced this pull request Nov 3, 2020

using var and removing brittle code

99fb5e0

#27 (comment)

tiholic added a commit that referenced this pull request Nov 3, 2020

using var and removing brittle code

788510a

#27 (comment)

tiholic added a commit that referenced this pull request Nov 4, 2020

using var and removing brittle code

207021b

#27 (comment)

tiholic added a commit that referenced this pull request Nov 4, 2020

using var and removing brittle code

f129448

#27 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Manual testing on RTN14 and RTN15 #27

Manual testing on RTN14 and RTN15 #27

tiholic commented Jul 31, 2020

QuintinWillison left a comment

QuintinWillison Sep 10, 2020

tiholic Sep 11, 2020

tiholic commented Sep 17, 2020 •

edited

Loading

Manual testing on RTN14 and RTN15 #27

Manual testing on RTN14 and RTN15 #27

Conversation

tiholic commented Jul 31, 2020

RTN14a ✅

RTN14b, RTN14g ⛔

RTN14c, RTN14d, RTN14e, RTN14f ✅

RTN15h, RTN15h1, RTN15h2 ✅

Temporary key ✅

Temporary app ✅

Restricted API key ✅

Un-restricting the above API Key, and manually re-connecting ❓

Another observation ❗

RTN15a ❓

RTN15g (RTN15g1, RTN15g2, RTN15g3) ✅

RTN15i, RTN15b (RTN15b1, RTN15b2), RTN15c (RTN15c1, RTN15c2, RTN15c3, RTN15c4), RTN15e ⛔

RTN15f ⛔

RTN15d ⛔

TL;DR ⌚ 🏃‍♂️

Temporary connection issues ✅

Permanent connection issues ✅

QuintinWillison left a comment

Choose a reason for hiding this comment

QuintinWillison Sep 10, 2020

Choose a reason for hiding this comment

tiholic Sep 11, 2020

Choose a reason for hiding this comment

tiholic commented Sep 17, 2020 • edited Loading

tiholic commented Sep 17, 2020 •

edited

Loading