Consideration for Onboarding Smoke Tests to OpenSearch Dashboards

[zelinh]

Summary

This issue explores the feasibility of onboarding OpenSearch Smoke Tests Framework for the OpenSearch Dashboards (OSD) distribution, based on a detailed analysis of its practicality and challenges.

[zelinh]

While we already have a backend smoke test framework for OpenSearch, the extension of a similar framework to OSD has been proposed. However, during the analysis, we identified several technical barriers that make this onboarding effort inefficient and redundant.

Analysis

1. Lack of a Source of Truth

• Unlike the backend, OSD lacks a centralized API specification repository (e.g., opensearch-api-specification) to define test cases and ensure API coverage.
• Feature request #487 highlights this gap.

2. Redundancy with Existing Functional Tests

• OSD functional tests already cover API calls alongside UI testing. Rebuilding similar capabilities specifically for smoke tests is an inefficient use of resources.
• Functional tests repo serves as a better alternative for comprehensive coverage.

3. Dependency on Backend APIs

• Most OSD API calls are proxies to backend APIs already tested by the backend smoke tests.
• Examples include:
  • Search Requests: Proxies backend _search endpoints.
  • Index Management: Relies on backend APIs for indices, mappings, and aliases.

4. Limited Value of Frontend Smoke Tests

• Frontend workflows involve UI, session handling, and rendering logic, which are outside the scope of backend-style smoke tests.
• Edge cases unique to frontend operations would remain untested by these smoke tests.

[zelinh]

Decision

After reviewing the challenges and the existing testing ecosystem for OSD, the team has decided to skip onboarding smoke tests for OpenSearch Dashboards.
Instead, we should focus on:

• Enhancing the existing functional tests to cover any identified gaps.
• Reassessing the need for OSD-specific API test coverage when a centralized API specification becomes available.

[zelinh]

I will close out this issue for now and we may re-visit this in the future.