[META] Upgrade Testing Framework Issue List

[chelma]

Meta issue to track the work associated with the Upgrade Testing Framework.

Framework Workstream

• Create pexpect core library #1
• Set up Validation Framework Runner #4
• Set up in-Python Docker library #6
• Create Framework Step that sets up starting cluster #8
• Create Framework Step that executes Snapshot/Restore upgrade #7
• Create Python REST Library for Elasticsearch/OpenSearch #50
• Handle UTF Setup/Teardown Edge Cases #51
• Enable import/setup of test files inside Docker containers #65

Expectations Workstream

• Design "Expectation" Data Model #2
• Design/Prototype test library #3
• Write initial validation script to invoke against starting cluster #5

Miscellaneous

• Develop UX Doc for the Upgrade Testing Framework #14
• Determine OpenSearch Client SDK Strategy #48
• Prototype Ability To Downselect Tests #49

[wbeckler]

Some of these steps might be over-engineered. After building the simplest version of this, without its own Expectation data model, and without pexpect, we might find that the hard parts lie elsewhere. It might make sense to build the simplest version of a validator and then to work outwards from there.

[chelma]

I agree we need to be careful about over-engineering on the expectations side, but pexpect is a fundamental building block on which everything else depends (it's how Python applications invoke shell calls). Also - these issues have yet to be refined by the team, I cranked them out as a stop-gap to give the team an idea on where things might head before I had to leave for vacation last Monday. I expect we'll discuss/adjust them in the coming week.

[chelma]

From community demo of the UTF today - this tool exists and may be worth exploring and/or collaborating with its developers on.

https://github.com/smortex/es-to-os