[ML] APM Latency Correlations #99905

walterra · 2021-05-12T11:39:10Z

Summary

This PR replaces the previous latency tab with a new Latency correlations tab with the main goal to help profiling a set of transactions, and to understand which transaction field values are mainly responsible for slow transactions. A different type of statistical analysis is used for this workflow:

Screen.Recording.2021-06-28.at.16.27.50.mov

Switch the visualization/latency chart in the correlations tab to use log-log axis
Leverage the new pipeline aggregations that calculate the correlation values and Kolmogorov–Smirnov test to find fields with relevant/significant correlation
Overlay the overall latency distribution as well as the individual latency distribution
Add pagination to the Correlations table
Remove duplicates (where different fields may refer to the same thing and therefore yield the same significant/correlation)
Apply transaction name filters when drilling down from Transactions page (currently support time range, environment, transaction type)
List names of detected duplicates (where correlation values & KS test p values & distributions are the same)
Exclude irrelevant fields and include high priority fields in the list of field candidates

Related issues:

UI/UX considerations

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

… ml-apm-correlations

…arts earlier)

qn895 · 2021-06-28T21:50:18Z

@elasticmachine merge upstream

qn895 · 2021-06-29T16:30:58Z

@elasticmachine merge upstream

kibanamachine · 2021-06-29T18:49:19Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`apm`	1557	1559	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	4.3MB	4.3MB	+12.3KB

History

💚 Build #134852 succeeded 76dffcf
💔 Build #134832 failed 1bff73e
💚 Build #134799 succeeded 034c680
💔 Build #134781 failed b616063
💚 Build #134600 succeeded 78ab411

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @walterra @qn895

* [ML] Async Search Service. * [ML] Log Log Area Charts Grid. Scatterplot with streaming data. * [ML] Remove scatterplot streaming demo. * [ML] Improved histogram bins for log scale. * [ML] Move client side code from Ml to APM plugin. * [ML] Adds correlation table. * [ML] Layout tweaks. * [ML] Move server side code from ML to APM. * [ML] Remove console.logs. * [ML] Fix types. * [ML] Chart area line type fix. Slowness %. * [ML] Comment unused vars. * Fix missing pluginsStart data in context * Add KS test value and rename table columns * Update percentiles to be correct * Make columns optional * Update fractionals/expectations to match with backend logic * Update so progress is 100% when all is completed * Make pre-processing steps smaller part of overall progress(to show charts earlier) * Add no correlations found message * Fix progress logic * Fix incorrect threshold, types * Add back APM filtering functionality to match with other table * Improve histogram equality check with random sampling * Show overall latency distribution right away * Rename demo tab to latencyCorrelationsTab * Update percentiles query to use hdr * [ML] Fix issue where apm-* might have multiple indices with different mappings where keyword is not the only type - Fix to check for keyword mapping more thoroughly - Add try catch if one of the es search fail, don't quit the entire fetch * Remove commented code * Remove kstest column, round correlation to 2 sigfig dec * Remove old latency tab, replace with ml latency correlations tab as first/default one * Set axis to start at 0 because agg may results in weird * Remove commented code for grouping duplicates * Update msg to mean significant correlations * Add i18n * Change correlations flyout back to medium size * Add name of service or transaction for clarity * Share i18n * Consolidate roundToDecimalPlace usage * Remove redundant isDuplicate * Create MAX_CORRELATIONS_TO_SHOW * Update mlCorrelationcolumns * Fix i18n quotations * Update query to include filter * Revert "Update query to include filter" This reverts commit 9a37eec * Rename MlCorrelations to MlLatencyCorrelations for clarity * Add pagination * Update include/exclude logic for field candidates and add ip field support * Add transactionName filter suport * Reorganize math utils * Group duplicates together * Fix typescript, better hasPrefixToInclude support check, * Remove Finished toast * Add title to y axis * Reduce number of tick labels to show * Highlight table row that is being used for graph * Add from to follow MDN guideline * Add APM style prefix * Fix i18n * [ML] Fix logic for tick format to only show power of 10 * Replace roundToDecimalPlace with asPreciseDecimal * Switch to lodash range * Clean up get_query_with_params * Prioritize candidates using field_terms * Update percentiles result type to be array instead of objects * Use observability's rangeQuery instead * Change arg format of query * Revert candidate_terms logic * Consolidate fractions, expectations, and ranges calc * Add tooltip for Correlation * Change terms size to 20 * Move env/service/transaction sticky header to top level, remove link * Add support for http.response.status_code * Replace histogram circular markers with bars * Delete unused roundToDecimalPlace * Add fractions calculation * Make notes of fractions and fix bucket correlation * Remove any, commented code, consolidate cancelFetch * Use es6 max * Align tooltip at the top * Get rid of getCoreServices, param docs, rename type, remove rangeQuery * Adjust range * Show all values without grouping duplicates * Fix pagination * Make flyout larger Co-authored-by: Quynh Nguyen <quynh.nguyen@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-06-29T19:26:15Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

* [ML] Async Search Service. * [ML] Log Log Area Charts Grid. Scatterplot with streaming data. * [ML] Remove scatterplot streaming demo. * [ML] Improved histogram bins for log scale. * [ML] Move client side code from Ml to APM plugin. * [ML] Adds correlation table. * [ML] Layout tweaks. * [ML] Move server side code from ML to APM. * [ML] Remove console.logs. * [ML] Fix types. * [ML] Chart area line type fix. Slowness %. * [ML] Comment unused vars. * Fix missing pluginsStart data in context * Add KS test value and rename table columns * Update percentiles to be correct * Make columns optional * Update fractionals/expectations to match with backend logic * Update so progress is 100% when all is completed * Make pre-processing steps smaller part of overall progress(to show charts earlier) * Add no correlations found message * Fix progress logic * Fix incorrect threshold, types * Add back APM filtering functionality to match with other table * Improve histogram equality check with random sampling * Show overall latency distribution right away * Rename demo tab to latencyCorrelationsTab * Update percentiles query to use hdr * [ML] Fix issue where apm-* might have multiple indices with different mappings where keyword is not the only type - Fix to check for keyword mapping more thoroughly - Add try catch if one of the es search fail, don't quit the entire fetch * Remove commented code * Remove kstest column, round correlation to 2 sigfig dec * Remove old latency tab, replace with ml latency correlations tab as first/default one * Set axis to start at 0 because agg may results in weird * Remove commented code for grouping duplicates * Update msg to mean significant correlations * Add i18n * Change correlations flyout back to medium size * Add name of service or transaction for clarity * Share i18n * Consolidate roundToDecimalPlace usage * Remove redundant isDuplicate * Create MAX_CORRELATIONS_TO_SHOW * Update mlCorrelationcolumns * Fix i18n quotations * Update query to include filter * Revert "Update query to include filter" This reverts commit 9a37eec * Rename MlCorrelations to MlLatencyCorrelations for clarity * Add pagination * Update include/exclude logic for field candidates and add ip field support * Add transactionName filter suport * Reorganize math utils * Group duplicates together * Fix typescript, better hasPrefixToInclude support check, * Remove Finished toast * Add title to y axis * Reduce number of tick labels to show * Highlight table row that is being used for graph * Add from to follow MDN guideline * Add APM style prefix * Fix i18n * [ML] Fix logic for tick format to only show power of 10 * Replace roundToDecimalPlace with asPreciseDecimal * Switch to lodash range * Clean up get_query_with_params * Prioritize candidates using field_terms * Update percentiles result type to be array instead of objects * Use observability's rangeQuery instead * Change arg format of query * Revert candidate_terms logic * Consolidate fractions, expectations, and ranges calc * Add tooltip for Correlation * Change terms size to 20 * Move env/service/transaction sticky header to top level, remove link * Add support for http.response.status_code * Replace histogram circular markers with bars * Delete unused roundToDecimalPlace * Add fractions calculation * Make notes of fractions and fix bucket correlation * Remove any, commented code, consolidate cancelFetch * Use es6 max * Align tooltip at the top * Get rid of getCoreServices, param docs, rename type, remove rangeQuery * Adjust range * Show all values without grouping duplicates * Fix pagination * Make flyout larger Co-authored-by: Quynh Nguyen <quynh.nguyen@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Walter Rafelsberger <walter@elastic.co> Co-authored-by: Quynh Nguyen <quynh.nguyen@elastic.co>

walterra self-assigned this May 12, 2021

walterra force-pushed the ml-apm-correlations branch from 81730ec to d0028dd Compare May 26, 2021 12:42

walterra added 10 commits June 1, 2021 12:34

[ML] Async Search Service.

f83ebad

[ML] Log Log Area Charts Grid. Scatterplot with streaming data.

0942b29

[ML] Remove scatterplot streaming demo.

e07aac1

[ML] Improved histogram bins for log scale.

a163b46

[ML] Move client side code from Ml to APM plugin.

8607e5d

[ML] Adds correlation table.

1bd7c24

[ML] Layout tweaks.

8c15c86

[ML] Move server side code from ML to APM.

14828f8

[ML] Remove console.logs.

9ccb65a

[ML] Fix types.

220bca7

walterra force-pushed the ml-apm-correlations branch from aab841e to 220bca7 Compare June 1, 2021 10:39

qn895 and others added 3 commits June 1, 2021 13:22

Merge remote-tracking branch 'upstream/master' into ml-apm-correlations

0682831

[ML] Chart area line type fix. Slowness %.

9cfbd13

Merge branch 'ml-apm-correlations' of github.com:walterra/kibana into…

6eca7d1

… ml-apm-correlations

walterra assigned qn895 Jun 3, 2021

walterra and others added 13 commits June 3, 2021 08:13

[ML] Comment unused vars.

b88c15d

Merge upstream/main into branch

6f70c3f

Fix missing pluginsStart data in context

0009dfe

Add KS test value and rename table columns

fdab271

Update percentiles to be correct

317f122

Merge upstream/main into branch

634f42c

Make columns optional

6e6cbaa

Merge remote-tracking branch 'upstream/master' into ml-apm-correlations

b30fe03

Update fractionals/expectations to match with backend logic

8cd33bb

Update so progress is 100% when all is completed

d03250a

Make pre-processing steps smaller part of overall progress(to show ch…

4447653

…arts earlier)

Add no correlations found message

513b6ad

Fix progress logic

190c9f4

qn895 added 3 commits June 28, 2021 15:50

Show all values without grouping duplicates

b2ad1a2

Fix pagination

6bb6079

Make flyout larger

1bff73e

qn895 changed the title ~~[ML] APM Latency Correlations (Part 1)~~ [ML] APM Latency Correlations Jun 28, 2021

Merge branch 'master' into ml-apm-correlations

76dffcf

lcawl mentioned this pull request Jun 29, 2021

[ML] APM latency correlations popover walterra/kibana#4

Closed

Merge branch 'master' into ml-apm-correlations

2d7f90e

smith approved these changes Jun 29, 2021

View reviewed changes

qn895 added the auto-backport Deprecated - use backport:version if exact versions are needed label Jun 29, 2021

qn895 merged commit 7405b93 into elastic:master Jun 29, 2021

kibanamachine mentioned this pull request Jun 29, 2021

[7.x] [ML] APM Latency Correlations (#99905) #103745

Merged

lcawl mentioned this pull request Jun 29, 2021

[ML] Fix APM latency order of sticky header properties, add help popover #103759

Merged

walterra mentioned this pull request Jun 30, 2021

[ML] Fractions API fix and jest unit tests for APM Latency Correlations. #103907

Merged

1 task

walterra deleted the ml-apm-correlations branch July 5, 2021 09:12

sorenlouv added the apm:test-plan-7.14.0 label Jul 5, 2021

lcawl mentioned this pull request Jul 6, 2021

[DOCS] APM latency correlations #104564

Merged

walterra mentioned this pull request Jul 7, 2021

[ML] API integration tests for APM latency correlation. #104644

Merged

1 task

walterra mentioned this pull request Jul 12, 2021

[ML] Fix custom index name settings, functional tests for APM Latency Correlation. #105200

Merged

3 tasks

peteharverson mentioned this pull request Jul 26, 2021

[APM] Latency correlations 7.15 meta issue #106381

Closed

24 tasks

walterra mentioned this pull request Aug 19, 2021

[APM] Latency correlations 7.16 meta issue #109220

Closed

11 tasks

sorenlouv mentioned this pull request Aug 23, 2021

[APM] "The selected trace can not be found" error in Kibana on sampled trace from Latency Distribution graph #103417

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] APM Latency Correlations #99905

[ML] APM Latency Correlations #99905

walterra commented May 12, 2021 •

edited by qn895

Loading

qn895 commented Jun 28, 2021

qn895 commented Jun 29, 2021

kibanamachine commented Jun 29, 2021

kibanamachine commented Jun 29, 2021

[ML] APM Latency Correlations #99905

[ML] APM Latency Correlations #99905

Conversation

walterra commented May 12, 2021 • edited by qn895 Loading

Summary

Related issues:

Checklist

qn895 commented Jun 28, 2021

qn895 commented Jun 29, 2021

kibanamachine commented Jun 29, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

History

kibanamachine commented Jun 29, 2021

💚 Backport successful

walterra commented May 12, 2021 •

edited by qn895

Loading