[Discover-Next] Adding datasources support for dataframes

[sejli]

Description

Parses user string between ::datasource:: to get the data source Then appends the dataSourceId in the meta.

Also creates the data frame before the interceptor to keep the meta info passed around.

If this gets accepted follow ups need to happen:

• should be ensuring the virtual index pattern includes the data source name with the index pattern name to avoid conflicts and a weird state.
• some reason dataframe is being always wiped out the first call in the search source. need to fix so that the schema is persisted
• weird initial load state. likely just too many things at once so we should make sure the usual and fresh experience is respected
• better indicator on the datasource, or just autocomplete that displays all the datasource connections when user types initial ::.

Rebase of #7092

Issues Resolved

#6957

Screenshot

Testing the changes

Changelog

• feat: onboard dataframes support to MDS and create dataframe before request

Check List

• All tests pass
  • yarn test:jest
  • yarn test:jest_integration
• New functionality includes testing.
• New functionality has been documented.
• Update CHANGELOG.md
• Commits are signed per the DCO using --signoff

[github-actions]

❌ Invalid Prefix For Manual Changeset Creation

Invalid description prefix. Found "feat". Only "skip" entry option is permitted for manual commit of changeset files.

If you were trying to skip the changelog entry, please use the "skip" entry option in the ##Changelog section of your PR description.

[github-actions]

❌ Invalid Prefix For Manual Changeset Creation

Invalid description prefix. Found "feat". Only "skip" entry option is permitted for manual commit of changeset files.

If you were trying to skip the changelog entry, please use the "skip" entry option in the ##Changelog section of your PR description.

[github-actions]

❌ Invalid Prefix For Manual Changeset Creation

Invalid description prefix. Found "feat". Only "skip" entry option is permitted for manual commit of changeset files.

If you were trying to skip the changelog entry, please use the "skip" entry option in the ##Changelog section of your PR description.

[ashwin-pc]

@sejli This is temporary right? Can we mark it inthe code and link back to an issue or this PR to indicate that if thats the case

[codecov]

Codecov Report

Attention: Patch coverage is 3.44828% with 28 lines in your changes missing coverage. Please review.

Project coverage is 67.42%. Comparing base (7f0e9d0) to head (af192b9).
Report is 365 commits behind head on main.

Files with missing lines	Patch %	Lines
.../data/common/search/search_source/search_source.ts	0.00%	9 Missing ⚠️
src/plugins/data/common/data_frames/utils.ts	11.11%	8 Missing ⚠️
src/plugins/data/public/search/search_service.ts	0.00%	4 Missing ⚠️
src/plugins/data/server/search/search_service.ts	0.00%	4 Missing ⚠️
...on/index_patterns/index_patterns/index_patterns.ts	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7106      +/-   ##
==========================================
- Coverage   67.45%   67.42%   -0.03%     
==========================================
  Files        3448     3448              
  Lines       67916    67945      +29     
  Branches    11042    11052      +10     
==========================================
+ Hits        45812    45813       +1     
- Misses      19435    19463      +28     
  Partials     2669     2669              

Flag	Coverage Δ
Linux_1	33.05% <3.44%> (-0.02%)	⬇️
Linux_2	55.05% <4.00%> (-0.06%)	⬇️
Linux_3	45.23% <3.44%> (-0.05%)	⬇️
Linux_4	34.83% <3.44%> (-0.03%)	⬇️
Windows_1	33.08% <3.44%> (-0.02%)	⬇️
Windows_2	55.00% <4.00%> (-0.06%)	⬇️
Windows_3	45.25% <3.44%> (-0.04%)	⬇️
Windows_4	34.83% <3.44%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-7106-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 d3d1c43124cc51e0199a70c38874490f0fd6d82a
# Push it to GitHub
git push --set-upstream origin backport/backport-7106-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7106-to-2.x.

[kavilla]

@sejli This is temporary right? Can we mark it inthe code and link back to an issue or this PR to indicate that if thats the case

I believe so. One thing though it could help to trigger some suggestions based on datasources

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-7106-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 d3d1c43124cc51e0199a70c38874490f0fd6d82a
# Push it to GitHub
git push --set-upstream origin backport/backport-7106-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7106-to-2.x.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-7106-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 d3d1c43124cc51e0199a70c38874490f0fd6d82a
# Push it to GitHub
git push --set-upstream origin backport/backport-7106-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7106-to-2.x.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-7106-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 d3d1c43124cc51e0199a70c38874490f0fd6d82a
# Push it to GitHub
git push --set-upstream origin backport/backport-7106-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7106-to-2.x.

[AMoo-Miki]

Manually backported in #7139