[Security Solution] Adhoc views in sourcerer #208372

lgestc · 2025-01-27T14:09:00Z

Summary

Fixes #201847

Up until now, whenever we were not able to identify the Data View to switch sourcerer to based on required patterns,
we would update the Default Security Solution data view with missing patterns, effectively altering the advanced setting.

This PR introduces basic Adhoc Data View fallback for cases like that, where when we are missing the pattern, the in-memory data view is created (not persisted as SO) that has required patterns configured in it.

The user is informed with some badge that this happened, and will be able to pick some other, existing data view instead if they wish.

Testing

Create a timeline and save it
Update the advanced settings by removing an index pattern.
Re-visit the saved timeline and refresh
Find the update message saying the dataview needs to be updated with the index pattern that was just removed.
Click update, and find that the advanced setting default index has been modified.

In short, any time there is a mismatch between a saved set of index patterns and the advanced setting (as it is would be in any timelines saved prior to the advanced setting change), then it will deem the data view to need an 'update', which when clicked, will then re-add those indices to the default index in the advanced settings.

Alternatively,
To test: save this json as a timelines.ndjson file and upload the timelines to the timelines page in security solution

…apping'

jbudz

packages/kbn-babel-preset/styled_components_files.js LGTM

logeekal

Did a code review. Left some thoughts ( not necessary ask for change ) which we can discuss more.

Doing desk testing and will report back.

logeekal · 2025-02-06T10:05:52Z

x-pack/solutions/security/plugins/security_solution/public/sourcerer/store/helpers.ts

@@ -62,6 +62,7 @@ export const validateSelectedPatterns = (
    const dedupeAllDefaultPatterns = ensurePatternFormat(
      (dataView ?? state.defaultDataView).title.split(',')
    );
+    // TODO: If we having missing patterns here, just create a new dataView...don't worry about missingPatterns anymore...


Is this is leftover or still to be done?

ah I will remove it. thanks

logeekal · 2025-02-06T10:08:10Z

...s/security/plugins/security_solution/public/sourcerer/components/use_pick_index_patterns.tsx

@@ -29,7 +29,7 @@ interface UsePickIndexPatternsProps {
  signalIndexName: string | null;
 }

-export type ModifiedTypes = 'modified' | 'alerts' | 'deprecated' | 'missingPatterns' | '';
+export type ModifiedTypes = 'modified' | 'alerts' | 'deprecated' | 'missingPatterns' | 'adhoc' | '';


Couldn't it be renamed to be more meaningful. I am not sure what does ModifiedTypes indicate.

Maybe DataviewModificationTypes?

logeekal · 2025-02-06T10:22:52Z

...ons/security/plugins/security_solution/public/sourcerer/components/use_data_view_fallback.ts

+  /**
+   * Missing patterns that need to be wrapped in an adhoc adhocDataView
+   */
+  missingPatterns: string[];


Do you mean these patterns should be included in the adhoc view?

yes, will update the docs. thank you

...ons/security/plugins/security_solution/public/sourcerer/components/use_data_view_fallback.ts

logeekal · 2025-02-06T10:29:53Z

...ecurity/plugins/security_solution/public/sourcerer/components/use_create_adhoc_data_view.tsx

+  const { addError } = useAppToasts();
+
+  const createAdhocDataView = useCallback(
+    async (missingPatterns: string[]): Promise<DataView | null> => {


nit

Does it makes sense to rename missingPatterns to patterns?

From the perspective of createAdhocDataView, missingPatterns does not mean anything. It only cares about the patterns.

I say this so that useCreateAdhocDataView can also be used in some other context apart from missingPatterns?

logeekal · 2025-02-06T10:58:29Z

...ecurity/plugins/security_solution/public/sourcerer/components/use_data_view_fallback.test.ts

+    expect(onApplyFallbackDataView).toHaveBeenCalledWith('mock', ['pattern1', 'pattern2'], false);
+    expect(onApplyFallbackDataView).toHaveBeenCalledTimes(1);
+  });
+});


Consider testing onResolveErrorManually as well.

logeekal · 2025-02-06T11:01:57Z

x-pack/solutions/security/plugins/security_solution/public/sourcerer/components/temporary.tsx

-  },
-};
-
-export const TemporarySourcererComp = React.memo<Props>(


Could you please give a little background on what was this TemporarySourcererComp and why is it now removed?

logeekal · 2025-02-06T11:12:40Z

x-pack/solutions/security/plugins/security_solution/public/sourcerer/components/misc.test.tsx

@@ -106,36 +102,36 @@ describe('No data', () => {
  });

  test('Hide sourcerer - default ', () => {
-    const wrapper = mount(
+    render(


Thanks for converting this file to RTL ✨ ✨

logeekal · 2025-02-06T11:19:53Z

x-pack/solutions/security/plugins/security_solution/public/sourcerer/components/misc.test.tsx

  });
 });

-describe('Update available', () => {
-  const state2 = {
+describe('Compat mode', () => {


I think Compat mode does not convey much in terms of what is happening to the data view. Thinking out loud some options can be.

Updated

Can be accompanied with a call out what Updated means.

Adhoc

Considering our customers already know what Adhoc dataviews are.

Timeline - Most Preferable imo

We always create an ad-hoc view with the name Security Solution Dataview (Timeline) which is either copy of the original data view or updated with missing patterns. From user's perspective, it does not matter. It will always be the same.

What do you think? I would like opinion of @michaelolo24 on it as well. It might also make sense to involve design as well.

I wouldn't event include Security Solution Dataview as I think that would be too close to what we already have and confuse users into thinking they can modify it in the advanced settings or something. I would be interested to see how Discover handles this problem, but IMO we could do some thing close to what @logeekal mentioned here like:

Timeline Temporary Dataview or Timeline Unpersisted Dataview. I lean towards stating exactly what's happening.

logeekal · 2025-02-06T11:25:12Z

x-pack/solutions/security/plugins/security_solution/public/sourcerer/components/index.tsx

-      isTriggerDisabled={isTriggerDisabled}
-      isModified={isModified}
+      isTriggerDisabled={false}
+      isModified={modificationStatus}


I know this was here before but isModified looks like a boolean but is actually a modificationStatus 🤣

logeekal

Desk Testing Observations

So I did Desk testing for this and i am encountering 3 issues. I am not sure if I am doing anything wrong here. Below is the demo and my observations

Observations

✅ Compat Mode appeared as expected when there are some missing patterns in defaultIndex
When in compat mode, field selector keeps spinning and never finds the fields.
There is no dataview selected in the dropdown. IMO we should select the Adhoc dataview and name it as well so that there is no confusion on UX front.
Edited Later - If you notice in the demo video, the unified field list keeps on loading as well.

Demo

Screen.Recording.2025-02-06.at.13.56.07.mov

…rcerer/components/use_data_view_fallback.ts Co-authored-by: Jatin Kathuria <jtn.kathuria@gmail.com>

lgestc · 2025-02-07T08:07:57Z

Desk Testing Observations

So I did Desk testing for this and i am encountering 3 issues. I am not sure if I am doing anything wrong here. Below is the demo and my observations

Observations

✅ Compat Mode appeared as expected when there are some missing patterns in defaultIndex

When in compat mode, field selector keeps spinning and never finds the fields.

There is no dataview selected in the dropdown. IMO we should select the Adhoc dataview and name it as well so that there is no confusion on UX front.

Demo

Screen.Recording.2025-02-06.at.13.56.07.mov

Thank you @logeekal , I will look into point #2, as for #3 - there is a change in the making on how we display the dataviews, and it will include proper listing for the adhoc thing - in a separate PR. Reason being this one would end up being 40 files long again. Wording, labels etc - all subject to change. Counting on @michaelolo24 here, also @paulewing - Jatin made some good suggestions about the label we could use insead of "Compat mode", liked the "Timeline" best but not sure if the case for missing patterns is just about the Timeline :D

elasticmachine · 2025-02-07T09:24:10Z

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

Click to trigger kibana-pull-request for this PR!
Click to trigger kibana-deploy-project-from-pr for this PR!
Click to trigger kibana-deploy-cloud-from-pr for this PR!

logeekal · 2025-02-07T09:48:19Z

@lgestc ,

Thanks for your comment. In my previous comment, I added a 4th point which I missed earlier.

elasticmachine · 2025-02-07T09:59:28Z

💚 Build Succeeded

Buildkite Build
Commit: ab33ec3

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	6695	6692	-3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	21.4MB	21.4MB	-7.8KB

History

💔 Build #274144 failed 3073315
💔 Build #273949 failed cfd1cb4
💚 Build #273633 succeeded a2fc77f

lgestc · 2025-02-07T10:48:36Z

just found an issue, field mapping seems to be missing

michaelolo24 · 2025-02-07T16:33:04Z

x-pack/solutions/security/plugins/security_solution/public/sourcerer/components/index.tsx

-      );
-      setIsTriggerDisabled(true);
+      dispatchChangeDataView(dataView.id, dataView.title?.split(',') || [], false, dataView);
+      setDataViewId(dataView.id);


I know this is additional work, but I think we should be pulling the dataViewId from sourcerer state rather than relying on some mix of local and redux state here 🤔. I wonder if we're seeing any bugs because of this...

michaelolo24 · 2025-02-07T21:12:46Z

When testing based on the original logic defined here: #120022 , I noticed a few things (to not re-iterate any of @logeekal 's points)

When loading a timeline with an adhoc dataview, we don't show all available dataviews in the dropdown list, sometimes we don't even show the dataview picker...

If sourcerer is in a bad state (i.e. from number # 1 above, then that state is persisted across all timelines opened...) The dropdown selector was missing for timeline templates when I opened them after # 1:

When navigating away from the security page with a timeline with an adhoc dv and navigating back, errors are thrown that the dataview can't be found:

Clicking show detection only and the unselecting it doesn't return you to the adhoc dataview you were on before, it just resets you to the security default dataview

michaelolo24 and others added 2 commits January 27, 2025 08:55

create saved dataview on sourcerer update

f53d18a

Merge branch 'main' into fix-sourcerer-bug

4daf3dc

lgestc mentioned this pull request Jan 27, 2025

[WIP][Sourcerer Bug] #208368

Closed

9 tasks

lgestc and others added 18 commits January 28, 2025 14:19

wip

0ba9132

wip

5740773

--wip-- [skip ci]

a0178b3

--wip-- [skip ci]

01c2796

--wip-- [skip ci]

cf436e6

--wip-- [skip ci]

98c1ef3

--wip-- [skip ci]

b9b1f53

--wip-- [skip ci]

4a5fdc7

--wip-- [skip ci]

9d5adca

--wip-- [skip ci]

21151e7

--wip-- [skip ci]

29eed10

--wip-- [skip ci]

53e81d0

--wip-- [skip ci]

bb777b0

--wip-- [skip ci]

ceb1240

add test skeleton

5a823df

add test for the dataview fallback hook

ca0013b

Merge branch 'main' into adhoc_views_in_sourcerer

f5b6860

--wip-- [skip ci]

36a9386

lgestc force-pushed the adhoc_views_in_sourcerer branch from 358ae5f to 36a9386 Compare February 3, 2025 09:29

lgestc and others added 3 commits February 3, 2025 10:29

Merge branch 'main' into adhoc_views_in_sourcerer

084831f

fix broken paths

2a70690

remove unused translations

8ab8c15

lgestc force-pushed the adhoc_views_in_sourcerer branch from 26e89a9 to 8ab8c15 Compare February 3, 2025 12:32

elastic deleted a comment from elasticmachine Feb 3, 2025

[CI] Auto-commit changed files from 'node scripts/styled_components_m…

1413cde

…apping'

elastic deleted a comment from elasticmachine Feb 3, 2025

--wip-- [skip ci]

5276cf4

jbudz approved these changes Feb 4, 2025

View reviewed changes

lgestc added 2 commits February 5, 2025 13:48

Merge branch 'main' into adhoc_views_in_sourcerer

648e6a5

Merge branch 'main' into adhoc_views_in_sourcerer

a2fc77f

elastic deleted a comment from elasticmachine Feb 5, 2025

Merge branch 'main' into adhoc_views_in_sourcerer

81fc498

lgestc mentioned this pull request Feb 6, 2025

[Security Solution][Sourcerer] Render Data View title in sourcerer #209974

Draft

logeekal reviewed Feb 6, 2025

View reviewed changes

Merge branch 'main' into adhoc_views_in_sourcerer

cfd1cb4

logeekal reviewed Feb 6, 2025

View reviewed changes

lgestc and others added 3 commits February 6, 2025 22:00

Merge branch 'main' into adhoc_views_in_sourcerer

3073315

Merge branch 'main' into adhoc_views_in_sourcerer

87b4f50

Update x-pack/solutions/security/plugins/security_solution/public/sou…

ab33ec3

…rcerer/components/use_data_view_fallback.ts Co-authored-by: Jatin Kathuria <jtn.kathuria@gmail.com>

lgestc added the 9.0 candidate label Feb 7, 2025

lgestc marked this pull request as draft February 7, 2025 09:24

fix fields loading

b40c747

lgestc force-pushed the adhoc_views_in_sourcerer branch from 3ac12ed to b40c747 Compare February 7, 2025 10:44

lgestc added 6 commits February 7, 2025 12:13

--wip-- [skip ci]

22413b9

--wip-- [skip ci]

33dfaf0

--wip-- [skip ci]

b857c2b

fix missing fields when regular dv is present

17aa643

document new scope field

22a727f

remove whitespace

5418f0f

michaelolo24 reviewed Feb 7, 2025

View reviewed changes

change fallback enable logic

eb8694f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Adhoc views in sourcerer #208372

[Security Solution] Adhoc views in sourcerer #208372

lgestc commented Jan 27, 2025 •

edited

Loading

jbudz left a comment

logeekal left a comment

logeekal Feb 6, 2025

lgestc Feb 7, 2025

logeekal Feb 6, 2025

logeekal Feb 6, 2025

lgestc Feb 7, 2025

logeekal Feb 6, 2025

logeekal Feb 6, 2025

logeekal Feb 6, 2025

logeekal Feb 6, 2025

logeekal Feb 6, 2025

michaelolo24 Feb 7, 2025

logeekal Feb 6, 2025

logeekal left a comment •

edited

Loading

lgestc commented Feb 7, 2025

Desk Testing Observations

Observations

Demo

elasticmachine commented Feb 7, 2025

logeekal commented Feb 7, 2025 •

edited

Loading

elasticmachine commented Feb 7, 2025

lgestc commented Feb 7, 2025

michaelolo24 Feb 7, 2025

michaelolo24 commented Feb 7, 2025

[Security Solution] Adhoc views in sourcerer #208372

Are you sure you want to change the base?

[Security Solution] Adhoc views in sourcerer #208372

Conversation

lgestc commented Jan 27, 2025 • edited Loading

Summary

Testing

jbudz left a comment

Choose a reason for hiding this comment

logeekal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

logeekal left a comment • edited Loading

Choose a reason for hiding this comment

Desk Testing Observations

Observations

Demo

lgestc commented Feb 7, 2025

Desk Testing Observations

Observations

Demo

elasticmachine commented Feb 7, 2025

logeekal commented Feb 7, 2025 • edited Loading

elasticmachine commented Feb 7, 2025

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

History

lgestc commented Feb 7, 2025

Choose a reason for hiding this comment

michaelolo24 commented Feb 7, 2025

lgestc commented Jan 27, 2025 •

edited

Loading

logeekal left a comment •

edited

Loading

logeekal commented Feb 7, 2025 •

edited

Loading