[Meta] Index Patterns Roadmap

[mattkime]

Summary

The index patterns service is part of the data plugin and owned by @elastic/kibana-app-services. It provides a way to define sets of indices, load field lists, and format values. IndexPattern instances are also used as arguments for buildEsQuery and similar tooling.

Current goal - Index Patterns should be flexible and easy to use for both developers and Kibana users.

7.10 - Index Pattern API overhaul - clarity in functionality and persistence.
7.11 - Complete server side support, remove field list cache, create REST API

7.12

• Field editor plugin with runtime field editor support - [index patterns] index pattern field editor with runtime field functionality #84786

7.13

• Plan to rename index patterns Plan for renaming index patterns #94244
• High level docs Index Patterns - high level docs #92125 (in progress, nearly complete)

7.14

• Index pattern creation dialog - [index patterns] Create index pattern flyout #84725
• Index pattern field editor - add accessibility tests - Index pattern field editor - add accessibility tests #92536
• Use deprecation service for scripted fields - Index patterns / scripted fields - use deprecation api #97947
• REST API for selecting default pattern - Kibana API for selecting default pattern #97337
• REST API for runtime field CRUD operations - Index patterns - REST api for runtime field CRUD operations #92126
• Switch to the new Elasticsearch client (tests) - [Index Patterns] Switch to the new Elasticsearch client #98608
• Support fields that start with underscore - fields starting with underscore ( _ ) doesn't show. #2551
• API for determining dependencies on a given index pattern - Index Patterns - Define API for determining dependencies on a given index pattern #97471
• Index patterns management - tab that shows matched data sources - Index patterns management - tab that shows matched data sources #72665
• (bugfix) url formatter stores base of url - Kibana URL field in index-pattern with relative URL stores the base URL at creation time in the saved object #63924
• (bugfix) [App services] Permission errors logged when opening Kibana for the first time [App services] Permission errors logged when opening Kibana for the first time #95094
• Index patterns REST API docs [Index Patterns] Public REST API Docs #99372

7.x

• Add Data field descriptions - Add Data field descriptions #89726
• REST API for list of all index patterns - Index Patterns REST API for getting list of all patterns #99928
• Reduce index pattern field editor bundle size - [index patterns] Lazy load field format editor to reduce index pattern field editor bundle size #100366

8.0

• Rename index patterns - Plan for renaming index patterns #94244 - 8.1 - 8.3 (tbd)
• Remove cached fields from old index patterns - [Index patterns] Remove fields cache in a migration #98383
• Saved Object type conversion - [Meta] Breaking saved object type conversions in 8.0 #100489

Future

Bug Fixes

• 7.10 / 7.9.3
  • Allow creation of cross cluster index patterns with cluster version <7.9 [Index Patterns] Allow creation of cross cluster index patterns when cluster is <7.9 #77825
• Excessive index patterns fetching on a dashboard - [Index Patterns] Excessive index patterns fetching on a dashboard #82153
• Improve text for Index Patterns UI - Improve text for Index Patterns UI #80781
• Kibana should display a message to user when there is no data in ES on index pattern management page - Kibana should display a message to user when there is no data in ES on index pattern management page #81001
• Space privilege "Index pattern management" read still shows delete button Space privilege "Index pattern management" read still shows delete button #53682
• Index pattern management field list UI needs to refresh after field refresh is performed - Index pattern management screen is stale (non-updating) after refreshing #60359
• Index pattern creation - properly handle errors on failure to create saved object - Index pattern creation - properly handle errors on failure to create saved object #96043

Improve tests

• [Management] Indices list snapshot is unruly #16194 - clean up snapshots

Refactoring

• Index Pattern API - clean up field index pattern and index patterns interfaces Index Pattern API - clean up field index pattern and index patterns interfaces #68020
  • Move indexPattern.popularizeField into discover (In progress) Move indexPattern.popularizeField into discover #77668
  • Index pattern edit field formatter API (in progress) Index pattern edit field formatter API #78352
• Simplify relationship between IndexPattern class, IIndexPattern, and IndexPatternSpec - Index patterns - simplify relationship between IndexPattern class, IIndexPattern, and IndexPatternSpec #78378
• Index Pattern API - type cleanup - Index Pattern API - Only export types, not classes #68012
• Data stream code to xpack plugin - Data streams - move index pattern creation code to xpack plugin #71341
• Replace StubIndexPattern with a more detailed jest mock - [data.indexPatterns] Replace StubIndexPattern with a more detailed jest mock #78519
• Index patterns scripted field type functionality removal (expressions script type no longer supported) - index patterns scripted fields script type functionality needs removal #74118
• Index Patterns API - should pass errors on to callers instead of swallowing - Index Patterns API - should pass errors on to callers instead of swallowing #95993
• Refactoring client / server to have more happen on the server, needs refinement - [Meta] Index Patterns Everywhere #90076

API improvements

• Data stream support - [Meta] Support data streams in Kibana #62941
• Index Pattern server API - Index patterns - Server API #69105
• Index Pattern toSpec / fromSpec - Index pattern serialize and de-serialize #68844
• Index Pattern API to server - Index Pattern API to server #68003
• Remove field list cache - Plan to remove index pattern field mapping cache #71787
  • Support for loading field list in server API - Index patterns api - server support for loading field list #71588
• REST API - REST-like CRUD+ HTTP API for index patterns #72360 - in progress
• Allow editing of index pattern title / pattern - Allow editing of index pattern title / pattern #79412
• Index patterns - allow title definition separate from pattern - Index patterns - allow title definition separate from pattern.  #79413
• Frozen index support on per index pattern basis - Move search.includeFrozen to be on a per-index pattern basis. #49799
• constant_keyword support - Support for constant_keyword ES field type #52635
• Index patterns do not support date_range ES type - Index patterns do not support date_range ES type #76980
• Disambiguate IFieldType [index patterns] type cleanup - disambigutate IFieldSubType #100186

User Interface

• Empty index pattern page redesign - Empty index patterns page re-design  #68819
• Index pattern creation dialog - [discuss] Index pattern creation dialog #67711
• Debounce missing index warnings - Debounce missing index warnings #62649

Index Pattern management

• Index pattern matched source list, make sortable, show more items - Index Pattern Management - allow sorting of matched data sources #70512
• Matched source summary header - Index pattern management - matched source summary header #71505
• Adding a field name length in index pattern display - Adding a field name length in index pattern display #74220
  • Would use EUI grid with optional column
• Display more information about conflicts in index pattern fields - Display more information about conflicts in index pattern fields #74218
• Index Patterns app doesn't unset its doc title- Index Patterns app doesn't unset its doc title #72668
• Tab that shows matched data sources - Index patterns management - tab that shows matched data sources #72665
• Index patterns management loads all field lists, lets optimize this - Index Patterns page gets fields for all index patterns when only needs titles #100139

8.0

• Remove expression-based scripted fields from Kibana. - [Breaking][data.indexPatterns] Remove expression-based scripted fields from Kibana. #77669

Needs investigation

• Removing getComputedFields - [data.indexPattern][data.search.searchSource] Remove indexPattern.getComputedFields. #85162
• Better nested field support - Add nested and object root field info to the index pattern #54957
• Tighter field semantics for ECS indices - Add ability to detect if index-pattern implements ECS #69856
• Field format editor via modal - investigate, design, and define tasks
• Only fields with data - Index pattern to take into consideration which fields have data #81756
• Provide a default field formatter based on the field metadata - [Index patterns] Provide a default field formatter based on the field metadata #82348
• Apply context filters when retrieving list of fields in application - Apply context filters when retrieving list of fields in application #95558
  • Related, but concerning CCS - Improve usability of remote cross cluster search in Kibana #99100
• Add telemetry to REST API - [data views] Add telemetry to REST APIs #100185

The long list of index pattern related problems - #35481

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[ppf2]

Would be nice to also catch potential misconfigurations: #80565

[ppisljar]

Thank you for contributing to this issue, however, we are closing this issue due to inactivity as part of a backlog grooming effort. If you believe this feature/bug should still be considered, please reopen with a comment.