[HOLD] Implement phased rollouts of the New Expensify mobile app

[roryabraham]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Problem

There is no fast way to roll back a mobile production release, so if any bugs get through our QA process to production, we will have to scramble to fix the issue as fast as possible. We also don't have a way to deploy hotfixes or CP to production, so if we need to fix an issue on production, we'll need to first clear the checklist of any existing deploy blockers before running another production deploy including the fix.

Why this is important

As New Expensify is no longer a beta product, stability of the mobile app is now very important.

Solution

Start using phased rollouts on NewDot mobile. While this doesn't completely eliminate the problem, it allows us to reduce the number of affected users by pausing the rollout while we work on releasing a fix.

Platform:

Where is this issue occurring?

• iOS
• Android

Slack conversation: https://expensify.slack.com/archives/C07J32337/p1634940809050800

[roryabraham]

@Expensify/mobile-deployers is there anyone who's interested in looking into this? I think it's valuable for stability but I may need to drop it for higher priority things.

[roryabraham]

No update here, switching this to monthly for now.

[roryabraham]

No update yet

[roryabraham]

Still no update

[roryabraham]

No update, actually going to send this back into the pool for now.

[AndrewGable]

My suggestion is we shouldn't do this until we have two things:

1. A reliable way to monitor crashes, currently we do not really monitor any logs or crashes and just rely on bug reports.
2. More production users

[roryabraham]

A reliable way to monitor crashes, currently we do not really monitor any logs or crashes and just rely on bug reports.

I think we have crashlytics set up, right? So it would just be a matter of chorifying the monitoring of crashes, right?

More production users

This makes sense, but I have a few thoughts:

1. As we continue to invest in NewDot we should see a shift starting soon(ish), and we'll want this in place before the shift has really happened. My reasoning is that phased rollouts can prevent crashes from becoming fires (I know this because I once caused a fire by accidentally skipping phased rollout and deploying a crash 100%).
2. NewDot has stability issues, and to paraphrase DB "we need to start thinking about NewDot not as the future of Expensify, but as the present of Expensify"
3. We won't want to manually manage the rollout in Google Play Store like we do for OldDot mobile. So we'll need to come up with some CD solution that will bump the rollout on a schedule unless someone intervenes manually. Maybe we could use a Bedrock job that periodically executes a workflow using the GH API, similar to triggerWorkflowAndWait. 🤔

All that said, I agree we can sit on this until we're closer to that time. It's not super urgent, but imo as soon as someone has some spare cycles to work on this there's not really a strong reason not to implement phased rollouts.

[AndrewGable]

Agree with all the above in regards to making the NewDot shift, if we want to keep it on GitHub actions we can use schedule. I would think we could bump all existing builds by 5% M-F on Android. For iOS the "bumping" happens automatically.

However, I do think we really need to invest more into crashalytics. I personally have never looked into the dashboard, so we need to feed crashes into GitHub issues somehow (e.g. https://github.com/Expensify/Expensify/issues/144832), so we can at least monitor crashes when we do create fires, otherwise we are just bumping for no gain.

Once we have this in place, I could see us skipping all bumps when there are known deploy blockers (which hopefully we can create automatically via the above automatic issue creation from crashalytics).

[roryabraham]

Just discovered we can implement phased rollouts on the desktop app(s) too: https://www.electron.build/auto-update#staged-rollouts

[melvin-bot]

@roryabraham, this Monthly task hasn't been acted upon in 6 weeks; closing.

If you disagree, feel encouraged to reopen it -- but pick your least important issue to close instead.

[roryabraham]

We'll definitely still want this. It's in-scope for N9

[melvin-bot]

Triggered auto assignment to @abekkala (AutoAssignerTriage), see https://stackoverflow.com/c/expensify/questions/4749 for more details.

[roryabraham]

Marking this as a new feature

[roryabraham]

No update, not prioritizing deploy improvements at this time

[roryabraham]

On HOLD for WAQ

[roryabraham]

Dropping for now, see #12021 (comment)

[MelvinBot]

@roryabraham, this Monthly task hasn't been acted upon in 6 weeks; closing.

If you disagree, feel encouraged to reopen it -- but pick your least important issue to close instead.

[melvin-bot]

@roryabraham, this Monthly task hasn't been acted upon in 6 weeks; closing.

If you disagree, feel encouraged to reopen it -- but pick your least important issue to close instead.

[melvin-bot]

@roryabraham, this Monthly task hasn't been acted upon in 6 weeks; closing.

If you disagree, feel encouraged to reopen it -- but pick your least important issue to close instead.