[FEATURE] Add support for large free text workloads

[kotwanikunal]

Is your feature request related to a problem?

• The current workloads do not have a large free text documents which is representative of the real world scenarios
• This issue was highlighted in [BUG] Indexing Performance Degraded in OpenSearch 1.3.+ OpenSearch#2916 where there was a big drop in indexing performance which was not uncovered by the recommended workloads during the 1.3.0 launch as well as recently run tests for 1.2.4 and 1.3.0 with the same configuration
• We would like to identify such anomalies to further strengthen our coverage and impact analysis for releases and feature additions to the OpenSearch codebase

What solution would you like?

• Addition of new customer dataset representative workloads with large free text

What alternatives have you considered?

• Running performance tests with multiple existing workloads which have even smaller documents than nyc_taxis

Do you have any additional context?

• N/A

[anasalkouz]

@treddeni-amazon this is really important to support. Having such performance degradation issues will decrease our trust of the performance benchmark results.

[travisbenedict]

@kotwanikunal would the so workload fit this use case? It has freeform text fields - example doc.

[kotwanikunal]

@kotwanikunal would the so workload fit this use case? It has freeform text fields - example doc.

Thanks for the update. I did schedule some tests over the weekend for so and http-logs. We should have the results by tomorrow to see if these tests detect the mentioned performance drop.

[kotwanikunal]

The results for so and http-logs on 1.2.4 and 1.3.0 are similar in nature to our original findings. 1.3.0 seems to perform better in general.
We will need an additional workload with larger free text fields which is able to detect the performance drops.

[CEHENKLE]

@opensearch-project/benchmark-core Hey folks -- What are your thoughts on this? Is this an improvement we can get added?

[IanHoang]

@CEHENKLE @kotwanikunal Thanks for pointing this out. We will generate an additional workload with larger text fields as soon as possible. However, due to a high volume of tasks that we need to attend to, please expect delays.

[ankitkala]

@CEHENKLE @kotwanikunal Can you guys also verify whether the regression mentioned above could've been caught with 50% heap instead of the current 1 GB.
I'm asking this since elastic has been completely relying on these datasets for catching any regression. Nothing against adding new workloads but we should also improve our existing test setup.
We can also look for more thorough perf testing with different workloads and cluster configuration.