[Ingest] Data streams list page #64134

jen-huang · 2020-04-21T23:46:36Z

Resolves #60814

Summary

This PR adds the initial shell page for data streams. A new DataStream type and list API was added to support this, along with corresponding response types.

GET /api/ingest_manager/data_streams returns an array of DataStreams sorted by last activity (desc) similar to:

{
  "data_streams": [
    {
      "index": "metrics-system.cpu-tester",
      "dataset": "system.cpu",
      "namespace": "tester",
      "type": "metrics",
      "package": "system",
      "last_activity": "2020-04-21T23:23:02.770Z",
      "size_in_bytes": 298682
    },
    {
      "index": "metrics-system.load-tester",
      "dataset": "system.load",
      "namespace": "tester",
      "type": "metrics",
      "package": "system",
      "last_activity": "2020-04-21T23:23:02.770Z",
      "size_in_bytes": 235274
    },
    ...
  ]
}

Deferred

The following tasks still needs to be done in subsequent PRs for this page:

Add row actions like view in Dashboard etc
Add overall stats to header (total data sets & total size)
Update Integration column values to display package icon
Add support for paging to API, currently returns first 100,000 indices/streams
- Table will also need to be updated, currently paging/sorting/filtering is done in memory

Screenshots

…ypes

elasticmachine · 2020-04-21T23:49:03Z

Pinging @elastic/ingest-management (Team:Ingest Management)

ruflin · 2020-04-22T07:15:02Z

x-pack/plugins/ingest_manager/server/routes/data_streams/handlers.ts

+            must: [
+              {
+                exists: {
+                  field: 'fields.stream.namespace',


That it has a fields prefix is actually a bug. As soon as elastic/beats#17858 is merged, this must be adjusted. Same for the other fields below.

@michalpristas Seeing your PR is approved, could you get it in so perhaps we can adjust this here before merging?

Will adjust.

ruflin · 2020-04-22T07:15:58Z

x-pack/plugins/ingest_manager/server/routes/data_streams/handlers.ts

+              },
+              package: {
+                terms: {
+                  field: 'event.module',


Why is this needed? Sorting? Perhaps prefix query could help here? event.module is likely to disappear in the future.

This is to detect the package to display in the Integrations column of the table. I see the same value in service.type too? What should I use?

None of both :-( This brings up the question: Is it the data that links to the package or does the ingest manager know which package belongs to the data? I would say it is the second one. Lets assume someone configure the agent to ship system metrics. He will not have added any information around packages as the agent does not know about packages. But the data will still be processed correctly, stored and works with the dashboards. Basically for each stream the ingest manager must check if it belongs to an installed package to show the link and then also show the dashboards ...

If your current solution works, lets keep it and then iterate on it.

Ok, it sounds like the get the package info the "right" way will involve more work that overlaps with the work needed for linking to dashboard, so will defer this for now.

Does that mean this PR gets a 👍? 🙂

The query gets a 👍 Did not really check the rest of the code or did it run locally.

nchaulet

Tested it locally and it seems a few fields are missing, I am missing something for tests?

nchaulet · 2020-04-23T00:54:01Z

x-pack/plugins/ingest_manager/server/routes/data_streams/handlers.ts

+
+    const dataStreams: DataStream[] = [];
+
+    (indexResults as any[]).forEach(result => {


any reason of mapping directly results here to dataStream? const dataStreams: DataStream[] = (indexResults as any[]).map(result => { ...;

will update; was leftover logic from previous attempt :)

nchaulet · 2020-04-23T00:59:19Z

x-pack/plugins/ingest_manager/server/routes/data_streams/handlers.ts

+
+const DATA_STREAM_INDEX_PATTERN = 'logs-*-*,metrics-*-*';
+
+export const getListHandler: RequestHandler = async (context, request, response) => {


Do you think we can cover this API by an API integration tests?

I took a look into our api integration tests and we have setup for fleet and epm routes, but not the general ingest manager routes. This particular route will need ES data fixtures to get tested. As we have a ton of other things to get done by FF, I'm going to defer test coverage on this for now :(

jen-huang · 2020-04-23T13:36:07Z

@nchaulet The missing fields are due to changes on agent side that were recently merged, needs a build/snapshot with those changes: #64134 (comment)

jfsiii · 2020-04-23T14:13:15Z

x-pack/plugins/ingest_manager/server/routes/data_streams/handlers.ts

+export const getListHandler: RequestHandler = async (context, request, response) => {
+  const callCluster = context.core.elasticsearch.dataClient.callAsCurrentUser;
+
+  try {


This handler is more knowledgeable than most. Can we move this out to a service or function(s) which accept callCluster, etc?

Blocking for this PR? This page needs to be iterated on with more functionality and we can move it to a service then.

jfsiii · 2020-04-23T14:24:22Z

x-pack/plugins/ingest_manager/common/types/rest_spec/fleet_setup.ts

-// eslint-disable-next-line @typescript-eslint/no-empty-interface
-export interface GetFleetSetupRequest {}
-
-export interface CreateFleetSetupRequest {


Can you talk about these changes? It's not clear to me why we're also making changes to the Fleet setup behavior

No behavioral changes, just cleaning up typings: e0e62b1

GetFleetSetupRequest, CreateFleetSetupRequest, and CreateFleetSetupResponse were duplicated in server/types/rest_spec/fleet_setup.ts, so I removed that file and kept this one in /common (to match with our existing structure).

GetFleetSetupRequest is empty so I removed it. CreateFleetSetupRequest is invalid as we no longer require username/password as part of the setup endpoint request, so it's also empty.

I see. Thanks for the additional info. I wasn't aware of the duplication or that we don't require user/pass any longer for setup.

I feel like setting validate: false is a step backwards (platform/security encourage them for a reason). I'd prefer to keep that but it's nothing to hold up this change.

* Clean up fleet setup request/response typings * Add data stream model and list route, handler, and request/response types * Initial pass at data streams list * Table styling fixes * Fix types, fix field names * Change forEach to map

kibanamachine · 2020-04-23T23:06:08Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 0bbc12f

History

💛 Build #42816 was flaky 0bbc12f
💚 Build #42528 succeeded d8f9bc2
💔 Build #42318 failed a151754

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Clean up fleet setup request/response typings * Add data stream model and list route, handler, and request/response types * Initial pass at data streams list * Table styling fixes * Fix types, fix field names * Change forEach to map

* master: (70 commits) KQL removes leading zero and breaks query (elastic#62748) [FieldFormats] Cleanup: rename IFieldFormatType -> FieldFormatInstanceType (elastic#64193) [ML] Changes transforms wizard UI text (elastic#64150) [Alerting] change server log action type .log to .server-log in README (elastic#64124) [Metrics UI] Design Refresh: Inventory View, Episode 1 (elastic#64026) chore(NA): reduce siem bundle size using babel-plugin-transfor… (elastic#63269) chore(NA): use core-js instead of babel-polyfill on canvas sha… (elastic#63486) skip flaky suite (elastic#61173) skip flaky suite (elastic#62497) Renamed ilm policy for event log so it is not prefixed with dot (elastic#64262) [eslint] no_restricted_paths config cleanup (elastic#63741) Add Oil Rig Icon from @elastic/maki (elastic#64364) [Maps] Migrate Maps embeddables to NP (elastic#63976) [Ingest] Data streams list page (elastic#64134) chore(NA): add file-loader into jest moduleNameMapper (elastic#64330) [DOCS] Added images to automating report generation (elastic#64333) [SIEM][CASE] Api Integration Tests: Configuration (elastic#63948) Expose ability to check if API Keys are enabled (elastic#63454) [DOCS] Fixes formatting in alerting doc (elastic#64338) [data.search.aggs]: Create agg types function for terms agg. (elastic#63541) ...

jen-huang added 5 commits April 21, 2020 11:12

Clean up fleet setup request/response typings

e0e62b1

Add data stream model and list route, handler, and request/response t…

1fc656a

…ypes

Initial pass at data streams list

5699a1a

Table styling fixes

f0b0b39

Merge remote-tracking branch 'upstream/master' into ingest/data-streams

a151754

jen-huang requested a review from a team April 21, 2020 23:46

jen-huang self-assigned this Apr 21, 2020

jen-huang added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.8.0 v8.0.0 labels Apr 21, 2020

ruflin reviewed Apr 22, 2020

View reviewed changes

Fix types, fix field names

d8f9bc2

jen-huang requested a review from a team April 22, 2020 19:04

jen-huang mentioned this pull request Apr 22, 2020

[Ingest] Ingest Manager UI: Alpha 1 #60445

Closed

22 tasks

nchaulet reviewed Apr 23, 2020

View reviewed changes

jen-huang added 2 commits April 23, 2020 06:46

Merge remote-tracking branch 'upstream/master' into ingest/data-streams

f05b6ad

Change forEach to map

0bbc12f

jfsiii reviewed Apr 23, 2020

View reviewed changes

nchaulet approved these changes Apr 23, 2020

View reviewed changes

jen-huang merged commit 9ad8b8f into elastic:master Apr 23, 2020

jen-huang deleted the ingest/data-streams branch April 23, 2020 21:14

jen-huang mentioned this pull request Apr 23, 2020

[7.x] [Ingest] Data streams list page (#64134) #64366

Merged

jen-huang mentioned this pull request Apr 23, 2020

[Ingest] Link data streams to dashboards #64369

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest] Data streams list page #64134

[Ingest] Data streams list page #64134

jen-huang commented Apr 21, 2020 •

edited

Loading

elasticmachine commented Apr 21, 2020

ruflin Apr 22, 2020

jen-huang Apr 22, 2020

ruflin Apr 22, 2020

jen-huang Apr 22, 2020

ruflin Apr 22, 2020

jen-huang Apr 22, 2020

ruflin Apr 22, 2020

nchaulet left a comment •

edited

Loading

nchaulet Apr 23, 2020

jen-huang Apr 23, 2020

nchaulet Apr 23, 2020

jen-huang Apr 23, 2020

jen-huang Apr 23, 2020

jen-huang commented Apr 23, 2020

jfsiii Apr 23, 2020

jen-huang Apr 23, 2020

jfsiii Apr 23, 2020

jen-huang Apr 23, 2020

jfsiii Apr 23, 2020

kibanamachine commented Apr 23, 2020


		const dataStreams: DataStream[] = [];

		(indexResults as any[]).forEach(result => {


		const DATA_STREAM_INDEX_PATTERN = 'logs--,metrics--';

		export const getListHandler: RequestHandler = async (context, request, response) => {

[Ingest] Data streams list page #64134

[Ingest] Data streams list page #64134

Conversation

jen-huang commented Apr 21, 2020 • edited Loading

Summary

Deferred

Screenshots

elasticmachine commented Apr 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nchaulet left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jen-huang commented Apr 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Apr 23, 2020

💚 Build Succeeded

History

jen-huang commented Apr 21, 2020 •

edited

Loading

nchaulet left a comment •

edited

Loading