[META] Automate performance testing

[gaiksaya]

Coming from #123, this is a meta issue for defining the performance testing framework as part of automated test infra. This issue focuses on performance regression testing and not scalability benchmarking

Tasks:

• Create single node cluster for performance testing #127
• Integrate rally(networking, client etc.) with the single node cluster #128
• Implement a PerformanceTestSuite class and wire it in to test.py #215
• Publish human-readable test run reports as a simple HTML document #130
• Onboard perf test to unified test workflow #506
• Automate performance testing for OpenSearch #129
• Automate performance tests for OpenSearch Dashboards #706
• [Bug]: Update perftest to the new repo structure in opensearch-infra #583
• Publish performance test results for every build  #707
• Automate Performance Testing for Plugins
  • Automate performance testing for CCR plugin #1428

Bugs:

• Request level logging during the tests #696

Exit Criteria:

• Performance test job, perf-test on the jenkins cluster completes without issue.
• Performance tests for Security Enabled/Disabled on both x64 and Arm64 finish successfully
• Performance test data is available for comparison by the OpenSearch-Core team.
• When test-orchestration-pipeline is triggered, it triggers the performance tests

[abhinavGupta16]

@anasalkouz - Can you update the status of this issue? Will this be ready by 1.3.0? Thanks!

[Bukhtawar]

@dblock are we also looking to gate PR merges if they cause regression. This is important since we want to discover this earlier. We do not want to wait for a release only to discover a regression at which point it would get harder to pin point the exact commit(s)

[bbarani]

@kavilla @tianleh @seanneumann Do you have plans to execute performance testing for Dashboards soon?

[bbarani]

Closing this META issue as we have automated performance testing for OpenSearch. We will track the automation of performance testing for OpenSearch dashboards in this issue.