[Tracking] NewDot Deploy Improvements

[roryabraham]

Problem: For the most part, the NewDot deploy system served us well over the last couple years. We've successfully completed over 2,000 staging deploys and hundreds of production deploys. :star-struck: However, experience has revealed a few issues with the current system:

• It's complex, which makes it harder to follow and diagnose problems
• It's brittle. It's a complex state machine that doesn't operate well when it gets into an "unexpected state". When it's in an unexpected state, it's difficult to predict what will happen when you take any given action to try and fix it. This has led to us making problems worse, and even doing an accidental production deploy before.
• It's unpredictable. The lack of idempotency in deploy actions means that the same action, taken twice, can end up with different results.
• It's weak. Which is to say, when our QA fails and we do ship bugs to customers, we are limited in our options to fix those, and/or otherwise prevent real users from experiencing those bugs.
• It's slow. The amount of time it takes to handle deploys and push out fixes eats up a lot of deployer time during their week.
• It's fundamentally flawed. There are a few rare cases in which the underlying git logic is flawed, and this can end up in the wrong code being deployed to staging and/or production.

These problems prevent daily deploys, waste deployer time, and result in a higher probability of NewDot users experiencing bugs.

Solution: Let's engage in a holistic rebuild of the NewDot deploy system to address these issues and focus on improving the simplicity, durability, idempotency, speed, and flexibility of the system, and include this project in the scope of #vip-waq-everywhere.

---

Known Issues

TODO:

• [HOLD] Move No QA PRs to a separate section of the checklist #9905
• Failed CP's can break our CI/CD #6863
• https://github.com/Expensify/Expensify/issues/195693
• Merging code to main can trigger deploy of unrelated systems #9725
• [HOLD] Multiple concurrent deploys are slow #9726
• [HOLD] Implement phased rollouts of the New Expensify mobile app #6306
• Get rid of deprecated set-output commands in GitHub Actions #11877
• https://github.com/Expensify/Expensify/issues/240551
• [$500] [HOLD for payment 2023-12-06] Local testing of GitHub Actions #13604
• [HOLD] Create a production QA checklist #13605
• [HOLD] Race condition between staging and production deploys #14834
• Production deploys can fail if a lower version is pending review #20196
• [Due 2023-11-01] Deprecate altool – use notarytool instead  #20763
• https://github.com/Expensify/Expensify/issues/299601
• Race condition can cause big checklists #27123
• https://github.com/Expensify/Expensify/issues/330131

In Progress:

Done:

• [HOLD] Manually CPing a pull request does not add the PR to the checklist #4379
• [$250] Race condition with the CP Staging label #7284
• Failed workflow comments in slack do not have the correct URL #12392
• [HOLD for payment 2023-01-30] Remove at-mentions in deploy comments #14404
• [HOLD for payment 2023-06-21] Cache node_modules to speed up CI/CD #20433
• [HOLD for payment 2023-05-23] Make it so we can run E2E performance tests on any branch #15605
• [HOLD for payment 2023-07-03] Investigate why some changes didn't make it to staging as expected #10214
• https://github.com/Expensify/Expensify/issues/280286
• [HOLD for payment 2023-07-06] [HOLD for payment 2023-07-05] Removing CP Staging label #20764
• https://github.com/Expensify/Expensify/issues/293688
• PRs already deployed to production included on staging checklist, PRs incorrectly commented on #23368
• • [HOLD for payment 2023-09-20] [$1250] Cache Pods to speed up iOS deploys #18623

[roryabraham]

Putting this on HOLD for WAQ

[roryabraham]

I am going to drop this for now with the intention of bringing it back up as a SWM project once they are done with the PR testing project.

[roryabraham]

Another potentially valuable addition here would be to deprecate .github/libs and provide a cleaner way to share code between the main application, tests, and GitHub Actions. Right now we have issues where CJS and ES6 modules don't know how to talk to eachother properly (example)

[MelvinBot]

@roryabraham, this Monthly task hasn't been acted upon in 6 weeks; closing.

If you disagree, feel encouraged to reopen it -- but pick your least important issue to close instead.

[melvin-bot]

@roryabraham, this Monthly task hasn't been acted upon in 6 weeks; closing.

If you disagree, feel encouraged to reopen it -- but pick your least important issue to close instead.

[roryabraham]

Made a number of improvements last week, including a fix for #27123

[roryabraham]

I think the next improvement I would make here would be to split up the build and upload steps in TestFlight such that:

1. There's a separate job or workflow for the build, and it generates the build as a build artifact
2. The upload step just grabs that build artifact and uploads it to TestFlight

I think this could provide a few benefits:

• If TestFlight upload is slow or flaky (as is pretty common), we can easily download the build artifact and upload it to TF manually
• As soon as one build is done, we can start the next one without waiting for an upload to complete

In short, I think separating the build and deploy steps might DRY things up, improve atomicity, and lend itself to more robust deploy process overall without additional complexity. It also might make the workflows more readable

[roryabraham]

No update – been OOO this week with jury duty

[roryabraham]

No upate

[roryabraham]

No update, but we're discussing how to hook in HybridApp deploys in the near future.

[roryabraham]

I think the last idea is a good one. I also think it's a bit of a problem that we can't retry a build on one platform until the builds for all platforms complete.

[roryabraham]

No update right now

[roryabraham]

Continued exploration in #22821

[roryabraham]

Last week I found and fixed some issues with our pod cache that was making our iOS builds unreliable. I'm hoping this will contribute to significantly reduced build failures and hopefully no need to ever go deleting caches manually (though, if you find you have to do this and it helps, please speak up so we can fix it)

• run pod install if node_modules changes
• run pod install if firebase.json changes

[roryabraham]

I think the time has come to close this long-standing tracking issue. There will always be more improvements to make, but overall the deploy system is in a much better place than it used to be stability-wise. Further improvements and new features would likely better be tracked in their own dedicated, focused issues going forward.