diff --git a/.github/ISSUE_TEMPLATE/Question.md b/.github/ISSUE_TEMPLATE/Question.md deleted file mode 100644 index 38fcb7af30b47..0000000000000 --- a/.github/ISSUE_TEMPLATE/Question.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -name: Question -about: Who, what, when, where, and how? - ---- - -Hey, stop right there! - -We use GitHub to track feature requests and bug reports. Please do not submit issues for questions about how to use features of Kibana, how to set Kibana up, best practices, or development related help. - -However, we do want to help! Head on over to our official Kibana forums and ask your questions there. In additional to awesome, knowledgeable community contributors, core Kibana developers are on the forums every single day to help you out. - -The forums are here: https://discuss.elastic.co/c/kibana - -We can't stop you from opening an issue here, but it will likely linger without a response for days or weeks before it is closed and we ask you to join us on the forums instead. Save yourself the time, and ask on the forums today. diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000000000..aa68db29974a8 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,5 @@ +blank_issues_enabled: false +contact_links: + - name: Question + url: https://discuss.elastic.co/c/kibana + about: Please ask and answer questions here. diff --git a/dev_docs/assets/api_doc_pick.png b/dev_docs/assets/api_doc_pick.png new file mode 100644 index 0000000000000..825fa47b266cb Binary files /dev/null and b/dev_docs/assets/api_doc_pick.png differ diff --git a/dev_docs/assets/dev_docs_nested_object.png b/dev_docs/assets/dev_docs_nested_object.png new file mode 100644 index 0000000000000..a6b2f533b3858 Binary files /dev/null and b/dev_docs/assets/dev_docs_nested_object.png differ diff --git a/dev_docs/best_practices.mdx b/dev_docs/best_practices.mdx index 6156c05197289..4d51263f93372 100644 --- a/dev_docs/best_practices.mdx +++ b/dev_docs/best_practices.mdx @@ -12,6 +12,132 @@ tags: ['kibana', 'onboarding', 'dev', 'architecture'] First things first, be sure to review our and check out all the available platform that can simplify plugin development. +## Developer documentation + +### High-level documentation + +#### Structure + +Refer to [divio documentation](https://documentation.divio.com/) for guidance on where and how to structure our high-level documentation. + + and + sections are both _explanation_ oriented, + covers both _tutorials_ and _How to_, and +the section covers _reference_ material. + +#### Location + +If the information spans multiple plugins, consider adding it to the [dev_docs](https://github.com/elastic/kibana/tree/master/dev_docs) folder. If it is plugin specific, consider adding it inside the plugin folder. Write it in an mdx file if you would like it to show up in our new (beta) documentation system. + + + +To add docs into the new docs system, create an `.mdx` file that +contains . Read about the syntax . An extra step is needed to add a menu item. will walk you through how to set the docs system +up locally and edit the nav menu. + + + +#### Keep content fresh + +A fresh pair of eyes are invaluable. Recruit new hires to read, review and update documentation. Leads should also periodically review documentation to ensure it stays up to date. File issues any time you notice documentation is outdated. + +#### Consider your target audience + +Documentation in the Kibana Developer Guide is targeted towards developers building Kibana plugins. Keep implementation details about internal plugin code out of these docs. + +#### High to low level + +When a developer first lands in our docs, think about their journey. Introduce basic concepts before diving into details. The left navigation should be set up so documents on top are higher level than documents near the bottom. + +#### Think outside-in + +It's easy to forget what it felt like to first write code in Kibana, but do your best to frame these docs "outside-in". Don't use esoteric, internal language unless a definition is documented and linked. The fresh eyes of a new hire can be a great asset. + +### API documentation + +We automatically generate . The following guidelines will help ensure your are useful. + +#### Code comments + +Every publicly exposed function, class, interface, type, parameter and property should have a comment using JSDoc style comments. + +- Use `@param` tags for every function parameter. +- Use `@returns` tags for return types. +- Use `@throws` when appropriate. +- Use `@beta` or `@deprecated` when appropriate. +- Use `@internal` to indicate this API item is intended for internal use only, which will also remove it from the docs. + +#### Interfaces vs inlined types + +Prefer types and interfaces over complex inline objects. For example, prefer: + +```ts +/** +* The SearchSpec interface contains settings for creating a new SearchService, like +* username and password. +*/ +export interface SearchSpec { + /** + * Stores the username. Duh, + */ + username: string; + /** + * Stores the password. I hope it's encrypted! + */ + password: string; +} + + /** + * Retrieve search services + * @param searchSpec Configuration information for initializing the search service. + * @returns the id of the search service + */ +export getSearchService: (searchSpec: SearchSpec) => string; +``` + +over: + +```ts +/** + * Retrieve search services + * @param searchSpec Configuration information for initializing the search service. + * @returns the id of the search service + */ +export getSearchService: (searchSpec: { username: string; password: string }) => string; +``` + +In the former, there will be a link to the `SearchSpec` interface with documentation for the `username` and `password` properties. In the latter the object will render inline, without comments: + +![prefer interfaces documentation](./assets/dev_docs_nested_object.png) + +#### Export every type used in a public API + +When a publicly exported API items references a private type, this results in a broken link in our docs system. The private type is, by proxy, part of your public API, and as such, should be exported. + +Do: + +```ts +export interface AnInterface { bar: string }; +export type foo: string | AnInterface; +``` + +Don't: + +```ts +interface AnInterface { bar: string }; +export type foo: string | AnInterface; +``` + +#### Avoid “Pick” + +`Pick` not only ends up being unhelpful in our documentation system, but it's also of limited help in your IDE. For that reason, avoid `Pick` and other similarly complex types on your public API items. Using these semantics internally is fine. + +![pick api documentation](./assets/api_doc_pick.png) + +### Example plugins + +Running Kibana with `yarn start --run-examples` will include all [example plugins](https://github.com/elastic/kibana/tree/master/examples). These are tested examples of platform services in use. We strongly encourage anyone providing a platform level service or to include a tutorial that links to a tested example plugin. This is better than relying on copied code snippets, which can quickly get out of date. + ## Performance Build with scalability in mind. diff --git a/docs/development/core/server/kibana-plugin-core-server.savedobjectsincrementcounteroptions.md b/docs/development/core/server/kibana-plugin-core-server.savedobjectsincrementcounteroptions.md index 68e9bb09456cd..8da2458cf007e 100644 --- a/docs/development/core/server/kibana-plugin-core-server.savedobjectsincrementcounteroptions.md +++ b/docs/development/core/server/kibana-plugin-core-server.savedobjectsincrementcounteroptions.md @@ -8,7 +8,7 @@ Signature: ```typescript -export interface SavedObjectsIncrementCounterOptions extends SavedObjectsBaseOptions +export interface SavedObjectsIncrementCounterOptions extends SavedObjectsBaseOptions ``` ## Properties @@ -18,4 +18,5 @@ export interface SavedObjectsIncrementCounterOptions extends SavedObjectsBaseOpt | [initialize](./kibana-plugin-core-server.savedobjectsincrementcounteroptions.initialize.md) | boolean | (default=false) If true, sets all the counter fields to 0 if they don't already exist. Existing fields will be left as-is and won't be incremented. | | [migrationVersion](./kibana-plugin-core-server.savedobjectsincrementcounteroptions.migrationversion.md) | SavedObjectsMigrationVersion | [SavedObjectsMigrationVersion](./kibana-plugin-core-server.savedobjectsmigrationversion.md) | | [refresh](./kibana-plugin-core-server.savedobjectsincrementcounteroptions.refresh.md) | MutatingOperationRefreshSetting | (default='wait\_for') The Elasticsearch refresh setting for this operation. See [MutatingOperationRefreshSetting](./kibana-plugin-core-server.mutatingoperationrefreshsetting.md) | +| [upsertAttributes](./kibana-plugin-core-server.savedobjectsincrementcounteroptions.upsertattributes.md) | Attributes | Attributes to use when upserting the document if it doesn't exist. | diff --git a/docs/development/core/server/kibana-plugin-core-server.savedobjectsincrementcounteroptions.upsertattributes.md b/docs/development/core/server/kibana-plugin-core-server.savedobjectsincrementcounteroptions.upsertattributes.md new file mode 100644 index 0000000000000..d5657dd65771f --- /dev/null +++ b/docs/development/core/server/kibana-plugin-core-server.savedobjectsincrementcounteroptions.upsertattributes.md @@ -0,0 +1,13 @@ + + +[Home](./index.md) > [kibana-plugin-core-server](./kibana-plugin-core-server.md) > [SavedObjectsIncrementCounterOptions](./kibana-plugin-core-server.savedobjectsincrementcounteroptions.md) > [upsertAttributes](./kibana-plugin-core-server.savedobjectsincrementcounteroptions.upsertattributes.md) + +## SavedObjectsIncrementCounterOptions.upsertAttributes property + +Attributes to use when upserting the document if it doesn't exist. + +Signature: + +```typescript +upsertAttributes?: Attributes; +``` diff --git a/docs/development/core/server/kibana-plugin-core-server.savedobjectsrepository.incrementcounter.md b/docs/development/core/server/kibana-plugin-core-server.savedobjectsrepository.incrementcounter.md index eb18e064c84e2..59d98bf4d607b 100644 --- a/docs/development/core/server/kibana-plugin-core-server.savedobjectsrepository.incrementcounter.md +++ b/docs/development/core/server/kibana-plugin-core-server.savedobjectsrepository.incrementcounter.md @@ -9,7 +9,7 @@ Increments all the specified counter fields (by one by default). Creates the doc Signature: ```typescript -incrementCounter(type: string, id: string, counterFields: Array, options?: SavedObjectsIncrementCounterOptions): Promise>; +incrementCounter(type: string, id: string, counterFields: Array, options?: SavedObjectsIncrementCounterOptions): Promise>; ``` ## Parameters @@ -19,7 +19,7 @@ incrementCounter(type: string, id: string, counterFields: Arraystring | The type of saved object whose fields should be incremented | | id | string | The id of the document whose fields should be incremented | | counterFields | Array<string | SavedObjectsIncrementCounterField> | An array of field names to increment or an array of [SavedObjectsIncrementCounterField](./kibana-plugin-core-server.savedobjectsincrementcounterfield.md) | -| options | SavedObjectsIncrementCounterOptions | [SavedObjectsIncrementCounterOptions](./kibana-plugin-core-server.savedobjectsincrementcounteroptions.md) | +| options | SavedObjectsIncrementCounterOptions<T> | [SavedObjectsIncrementCounterOptions](./kibana-plugin-core-server.savedobjectsincrementcounteroptions.md) | Returns: @@ -52,5 +52,19 @@ repository 'stats.apiCalls', ]) +// Increment the apiCalls field counter by 4 +repository + .incrementCounter('dashboard_counter_type', 'counter_id', [ + { fieldName: 'stats.apiCalls' incrementBy: 4 }, + ]) + +// Initialize the document with arbitrary fields if not present +repository.incrementCounter<{ appId: string }>( + 'dashboard_counter_type', + 'counter_id', + [ 'stats.apiCalls'], + { upsertAttributes: { appId: 'myId' } } +) + ``` diff --git a/packages/kbn-test/src/legacy_es/legacy_es_test_cluster.js b/packages/kbn-test/src/legacy_es/legacy_es_test_cluster.js index 43b6c90452b81..d472f27395ffb 100644 --- a/packages/kbn-test/src/legacy_es/legacy_es_test_cluster.js +++ b/packages/kbn-test/src/legacy_es/legacy_es_test_cluster.js @@ -12,9 +12,9 @@ import { get, toPath } from 'lodash'; import { Cluster } from '@kbn/es'; import { CI_PARALLEL_PROCESS_PREFIX } from '../ci_parallel_process_prefix'; import { esTestConfig } from './es_test_config'; +import { Client } from '@elastic/elasticsearch'; import { KIBANA_ROOT } from '../'; -import * as legacyElasticsearch from 'elasticsearch'; const path = require('path'); const del = require('del'); @@ -102,8 +102,8 @@ export function createLegacyEsTestCluster(options = {}) { * Returns an ES Client to the configured cluster */ getClient() { - return new legacyElasticsearch.Client({ - host: this.getUrl(), + return new Client({ + node: this.getUrl(), }); } diff --git a/src/core/public/doc_links/doc_links_service.ts b/src/core/public/doc_links/doc_links_service.ts index 6279d62d2c40e..0bb5ddd29609e 100644 --- a/src/core/public/doc_links/doc_links_service.ts +++ b/src/core/public/doc_links/doc_links_service.ts @@ -191,6 +191,7 @@ export class DocLinksService { lens: `${ELASTIC_WEBSITE_URL}what-is/kibana-lens`, lensPanels: `${ELASTIC_WEBSITE_URL}guide/en/kibana/${DOC_LINK_VERSION}/lens.html`, maps: `${ELASTIC_WEBSITE_URL}maps`, + vega: `${ELASTIC_WEBSITE_URL}guide/en/kibana/${DOC_LINK_VERSION}/vega.html`, }, observability: { guide: `${ELASTIC_WEBSITE_URL}guide/en/observability/${DOC_LINK_VERSION}/index.html`, @@ -284,6 +285,7 @@ export class DocLinksService { registerSourceOnly: `${ELASTICSEARCH_DOCS}snapshots-register-repository.html#snapshots-source-only-repository`, registerUrl: `${ELASTICSEARCH_DOCS}snapshots-register-repository.html#snapshots-read-only-repository`, restoreSnapshot: `${ELASTICSEARCH_DOCS}snapshots-restore-snapshot.html`, + restoreSnapshotApi: `${ELASTICSEARCH_DOCS}restore-snapshot-api.html#restore-snapshot-api-request-body`, }, ingest: { pipelines: `${ELASTICSEARCH_DOCS}ingest.html`, diff --git a/src/core/server/saved_objects/service/lib/repository.test.js b/src/core/server/saved_objects/service/lib/repository.test.js index 37572c83e4c88..ce48e8dc9a317 100644 --- a/src/core/server/saved_objects/service/lib/repository.test.js +++ b/src/core/server/saved_objects/service/lib/repository.test.js @@ -23,6 +23,7 @@ import { mockKibanaMigrator } from '../../migrations/kibana/kibana_migrator.mock import { elasticsearchClientMock } from '../../../elasticsearch/client/mocks'; import { esKuery } from '../../es_query'; import { errors as EsErrors } from '@elastic/elasticsearch'; + const { nodeTypes } = esKuery; jest.mock('./search_dsl/search_dsl', () => ({ getSearchDsl: jest.fn() })); @@ -3654,6 +3655,33 @@ describe('SavedObjectsRepository', () => { ); }); + it(`uses the 'upsertAttributes' option when specified`, async () => { + const upsertAttributes = { + foo: 'bar', + hello: 'dolly', + }; + await incrementCounterSuccess(type, id, counterFields, { namespace, upsertAttributes }); + expect(client.update).toHaveBeenCalledWith( + expect.objectContaining({ + body: expect.objectContaining({ + upsert: expect.objectContaining({ + [type]: { + foo: 'bar', + hello: 'dolly', + ...counterFields.reduce((aggs, field) => { + return { + ...aggs, + [field]: 1, + }; + }, {}), + }, + }), + }), + }), + expect.anything() + ); + }); + it(`prepends namespace to the id when providing namespace for single-namespace type`, async () => { await incrementCounterSuccess(type, id, counterFields, { namespace }); expect(client.update).toHaveBeenCalledWith( diff --git a/src/core/server/saved_objects/service/lib/repository.ts b/src/core/server/saved_objects/service/lib/repository.ts index aa1e62c1652ca..6e2a1d6ec0511 100644 --- a/src/core/server/saved_objects/service/lib/repository.ts +++ b/src/core/server/saved_objects/service/lib/repository.ts @@ -76,10 +76,16 @@ import { // BEWARE: The SavedObjectClient depends on the implementation details of the SavedObjectsRepository // so any breaking changes to this repository are considered breaking changes to the SavedObjectsClient. -// eslint-disable-next-line @typescript-eslint/consistent-type-definitions -type Left = { tag: 'Left'; error: Record }; -// eslint-disable-next-line @typescript-eslint/consistent-type-definitions -type Right = { tag: 'Right'; value: Record }; +interface Left { + tag: 'Left'; + error: Record; +} + +interface Right { + tag: 'Right'; + value: Record; +} + type Either = Left | Right; const isLeft = (either: Either): either is Left => either.tag === 'Left'; const isRight = (either: Either): either is Right => either.tag === 'Right'; @@ -98,7 +104,8 @@ export interface SavedObjectsRepositoryOptions { /** * @public */ -export interface SavedObjectsIncrementCounterOptions extends SavedObjectsBaseOptions { +export interface SavedObjectsIncrementCounterOptions + extends SavedObjectsBaseOptions { /** * (default=false) If true, sets all the counter fields to 0 if they don't * already exist. Existing fields will be left as-is and won't be incremented. @@ -111,6 +118,10 @@ export interface SavedObjectsIncrementCounterOptions extends SavedObjectsBaseOpt * operation. See {@link MutatingOperationRefreshSetting} */ refresh?: MutatingOperationRefreshSetting; + /** + * Attributes to use when upserting the document if it doesn't exist. + */ + upsertAttributes?: Attributes; } /** @@ -1694,6 +1705,20 @@ export class SavedObjectsRepository { * .incrementCounter('dashboard_counter_type', 'counter_id', [ * 'stats.apiCalls', * ]) + * + * // Increment the apiCalls field counter by 4 + * repository + * .incrementCounter('dashboard_counter_type', 'counter_id', [ + * { fieldName: 'stats.apiCalls' incrementBy: 4 }, + * ]) + * + * // Initialize the document with arbitrary fields if not present + * repository.incrementCounter<{ appId: string }>( + * 'dashboard_counter_type', + * 'counter_id', + * [ 'stats.apiCalls'], + * { upsertAttributes: { appId: 'myId' } } + * ) * ``` * * @param type - The type of saved object whose fields should be incremented @@ -1706,7 +1731,7 @@ export class SavedObjectsRepository { type: string, id: string, counterFields: Array, - options: SavedObjectsIncrementCounterOptions = {} + options: SavedObjectsIncrementCounterOptions = {} ): Promise> { if (typeof type !== 'string') { throw new Error('"type" argument must be a string'); @@ -1728,12 +1753,16 @@ export class SavedObjectsRepository { throw SavedObjectsErrorHelpers.createUnsupportedTypeError(type); } - const { migrationVersion, refresh = DEFAULT_REFRESH_SETTING, initialize = false } = options; + const { + migrationVersion, + refresh = DEFAULT_REFRESH_SETTING, + initialize = false, + upsertAttributes, + } = options; const normalizedCounterFields = counterFields.map((counterField) => { const fieldName = typeof counterField === 'string' ? counterField : counterField.fieldName; const incrementBy = typeof counterField === 'string' ? 1 : counterField.incrementBy || 1; - return { fieldName, incrementBy: initialize ? 0 : incrementBy, @@ -1757,11 +1786,14 @@ export class SavedObjectsRepository { type, ...(savedObjectNamespace && { namespace: savedObjectNamespace }), ...(savedObjectNamespaces && { namespaces: savedObjectNamespaces }), - attributes: normalizedCounterFields.reduce((acc, counterField) => { - const { fieldName, incrementBy } = counterField; - acc[fieldName] = incrementBy; - return acc; - }, {} as Record), + attributes: { + ...(upsertAttributes ?? {}), + ...normalizedCounterFields.reduce((acc, counterField) => { + const { fieldName, incrementBy } = counterField; + acc[fieldName] = incrementBy; + return acc; + }, {} as Record), + }, migrationVersion, updated_at: time, }); diff --git a/src/core/server/server.api.md b/src/core/server/server.api.md index 73f8a44075162..cf1647ef5cec3 100644 --- a/src/core/server/server.api.md +++ b/src/core/server/server.api.md @@ -2735,11 +2735,12 @@ export interface SavedObjectsIncrementCounterField { } // @public (undocumented) -export interface SavedObjectsIncrementCounterOptions extends SavedObjectsBaseOptions { +export interface SavedObjectsIncrementCounterOptions extends SavedObjectsBaseOptions { initialize?: boolean; // (undocumented) migrationVersion?: SavedObjectsMigrationVersion; refresh?: MutatingOperationRefreshSetting; + upsertAttributes?: Attributes; } // @public @@ -2839,7 +2840,7 @@ export class SavedObjectsRepository { // (undocumented) find(options: SavedObjectsFindOptions): Promise>; get(type: string, id: string, options?: SavedObjectsBaseOptions): Promise>; - incrementCounter(type: string, id: string, counterFields: Array, options?: SavedObjectsIncrementCounterOptions): Promise>; + incrementCounter(type: string, id: string, counterFields: Array, options?: SavedObjectsIncrementCounterOptions): Promise>; openPointInTimeForType(type: string | string[], { keepAlive, preference }?: SavedObjectsOpenPointInTimeOptions): Promise; removeReferencesTo(type: string, id: string, options?: SavedObjectsRemoveReferencesToOptions): Promise; resolve(type: string, id: string, options?: SavedObjectsBaseOptions): Promise>; diff --git a/src/plugins/index_pattern_management/server/routes/resolve_index.ts b/src/plugins/index_pattern_management/server/routes/resolve_index.ts index 851a2578231aa..22c214f2adee2 100644 --- a/src/plugins/index_pattern_management/server/routes/resolve_index.ts +++ b/src/plugins/index_pattern_management/server/routes/resolve_index.ts @@ -31,19 +31,11 @@ export function registerResolveIndexRoute(router: IRouter): void { }, }, async (context, req, res) => { - const queryString = req.query.expand_wildcards - ? { expand_wildcards: req.query.expand_wildcards } - : null; - const result = await context.core.elasticsearch.legacy.client.callAsCurrentUser( - 'transport.request', - { - method: 'GET', - path: `/_resolve/index/${encodeURIComponent(req.params.query)}${ - queryString ? '?' + new URLSearchParams(queryString).toString() : '' - }`, - } - ); - return res.ok({ body: result }); + const { body } = await context.core.elasticsearch.client.asCurrentUser.indices.resolveIndex({ + name: req.params.query, + expand_wildcards: req.query.expand_wildcards || 'open', + }); + return res.ok({ body }); } ); } diff --git a/src/plugins/kibana_usage_collection/server/collectors/application_usage/constants.ts b/src/plugins/kibana_usage_collection/server/collectors/application_usage/constants.ts index 1910ba054bf8e..f072f044925bf 100644 --- a/src/plugins/kibana_usage_collection/server/collectors/application_usage/constants.ts +++ b/src/plugins/kibana_usage_collection/server/collectors/application_usage/constants.ts @@ -12,15 +12,9 @@ export const ROLL_TOTAL_INDICES_INTERVAL = 24 * 60 * 60 * 1000; /** - * Roll daily indices every 30 minutes. - * This means that, assuming a user can visit all the 44 apps we can possibly report - * in the 3 minutes interval the browser reports to the server, up to 22 users can have the same - * behaviour and we wouldn't need to paginate in the transactional documents (less than 10k docs). - * - * Based on a more normal expected use case, the users could visit up to 5 apps in those 3 minutes, - * allowing up to 200 users before reaching the limit. + * Roll daily indices every 24h */ -export const ROLL_DAILY_INDICES_INTERVAL = 30 * 60 * 1000; +export const ROLL_DAILY_INDICES_INTERVAL = 24 * 60 * 60 * 1000; /** * Start rolling indices after 5 minutes up diff --git a/src/plugins/kibana_usage_collection/server/collectors/application_usage/index.ts b/src/plugins/kibana_usage_collection/server/collectors/application_usage/index.ts index 676f5fddc16e1..2d2d07d9d1894 100644 --- a/src/plugins/kibana_usage_collection/server/collectors/application_usage/index.ts +++ b/src/plugins/kibana_usage_collection/server/collectors/application_usage/index.ts @@ -7,3 +7,4 @@ */ export { registerApplicationUsageCollector } from './telemetry_application_usage_collector'; +export { rollDailyData as migrateTransactionalDocs } from './rollups'; diff --git a/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups.test.ts b/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/daily.test.ts similarity index 51% rename from src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups.test.ts rename to src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/daily.test.ts index 7d86bc41e0b90..5acd1fb9c9c3a 100644 --- a/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups.test.ts +++ b/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/daily.test.ts @@ -6,21 +6,16 @@ * Side Public License, v 1. */ -import { rollDailyData, rollTotals } from './rollups'; -import { savedObjectsRepositoryMock, loggingSystemMock } from '../../../../../core/server/mocks'; -import { MAIN_APP_DEFAULT_VIEW_ID } from '../../../../usage_collection/common/constants'; -import { SavedObjectsErrorHelpers } from '../../../../../core/server'; -import { - SAVED_OBJECTS_DAILY_TYPE, - SAVED_OBJECTS_TOTAL_TYPE, - SAVED_OBJECTS_TRANSACTIONAL_TYPE, -} from './saved_objects_types'; +import { savedObjectsRepositoryMock, loggingSystemMock } from '../../../../../../core/server/mocks'; +import { SavedObjectsErrorHelpers } from '../../../../../../core/server'; +import { SAVED_OBJECTS_DAILY_TYPE, SAVED_OBJECTS_TRANSACTIONAL_TYPE } from '../saved_objects_types'; +import { rollDailyData } from './daily'; describe('rollDailyData', () => { const logger = loggingSystemMock.createLogger(); - test('returns undefined if no savedObjectsClient initialised yet', async () => { - await expect(rollDailyData(logger, undefined)).resolves.toBe(undefined); + test('returns false if no savedObjectsClient initialised yet', async () => { + await expect(rollDailyData(logger, undefined)).resolves.toBe(false); }); test('handle empty results', async () => { @@ -33,7 +28,7 @@ describe('rollDailyData', () => { throw new Error(`Unexpected type [${type}]`); } }); - await expect(rollDailyData(logger, savedObjectClient)).resolves.toBe(undefined); + await expect(rollDailyData(logger, savedObjectClient)).resolves.toBe(true); expect(savedObjectClient.get).not.toBeCalled(); expect(savedObjectClient.bulkCreate).not.toBeCalled(); expect(savedObjectClient.delete).not.toBeCalled(); @@ -101,7 +96,7 @@ describe('rollDailyData', () => { throw SavedObjectsErrorHelpers.createGenericNotFoundError(type, id); }); - await expect(rollDailyData(logger, savedObjectClient)).resolves.toBe(undefined); + await expect(rollDailyData(logger, savedObjectClient)).resolves.toBe(true); expect(savedObjectClient.get).toHaveBeenCalledTimes(2); expect(savedObjectClient.get).toHaveBeenNthCalledWith( 1, @@ -196,7 +191,7 @@ describe('rollDailyData', () => { throw new Error('Something went terribly wrong'); }); - await expect(rollDailyData(logger, savedObjectClient)).resolves.toBe(undefined); + await expect(rollDailyData(logger, savedObjectClient)).resolves.toBe(false); expect(savedObjectClient.get).toHaveBeenCalledTimes(1); expect(savedObjectClient.get).toHaveBeenCalledWith( SAVED_OBJECTS_DAILY_TYPE, @@ -206,185 +201,3 @@ describe('rollDailyData', () => { expect(savedObjectClient.delete).toHaveBeenCalledTimes(0); }); }); - -describe('rollTotals', () => { - const logger = loggingSystemMock.createLogger(); - - test('returns undefined if no savedObjectsClient initialised yet', async () => { - await expect(rollTotals(logger, undefined)).resolves.toBe(undefined); - }); - - test('handle empty results', async () => { - const savedObjectClient = savedObjectsRepositoryMock.create(); - savedObjectClient.find.mockImplementation(async ({ type, page = 1, perPage = 10 }) => { - switch (type) { - case SAVED_OBJECTS_DAILY_TYPE: - case SAVED_OBJECTS_TOTAL_TYPE: - return { saved_objects: [], total: 0, page, per_page: perPage }; - default: - throw new Error(`Unexpected type [${type}]`); - } - }); - await expect(rollTotals(logger, savedObjectClient)).resolves.toBe(undefined); - expect(savedObjectClient.bulkCreate).toHaveBeenCalledTimes(0); - expect(savedObjectClient.delete).toHaveBeenCalledTimes(0); - }); - - test('migrate some documents', async () => { - const savedObjectClient = savedObjectsRepositoryMock.create(); - savedObjectClient.find.mockImplementation(async ({ type, page = 1, perPage = 10 }) => { - switch (type) { - case SAVED_OBJECTS_DAILY_TYPE: - return { - saved_objects: [ - { - id: 'appId-2:2020-01-01', - type, - score: 0, - references: [], - attributes: { - appId: 'appId-2', - timestamp: '2020-01-01T10:31:00.000Z', - minutesOnScreen: 0.5, - numberOfClicks: 1, - }, - }, - { - id: 'appId-1:2020-01-01', - type, - score: 0, - references: [], - attributes: { - appId: 'appId-1', - timestamp: '2020-01-01T11:31:00.000Z', - minutesOnScreen: 2.5, - numberOfClicks: 2, - }, - }, - { - id: 'appId-1:2020-01-01:viewId-1', - type, - score: 0, - references: [], - attributes: { - appId: 'appId-1', - viewId: 'viewId-1', - timestamp: '2020-01-01T11:31:00.000Z', - minutesOnScreen: 1, - numberOfClicks: 1, - }, - }, - ], - total: 3, - page, - per_page: perPage, - }; - case SAVED_OBJECTS_TOTAL_TYPE: - return { - saved_objects: [ - { - id: 'appId-1', - type, - score: 0, - references: [], - attributes: { - appId: 'appId-1', - minutesOnScreen: 0.5, - numberOfClicks: 1, - }, - }, - { - id: 'appId-1___viewId-1', - type, - score: 0, - references: [], - attributes: { - appId: 'appId-1', - viewId: 'viewId-1', - minutesOnScreen: 4, - numberOfClicks: 2, - }, - }, - { - id: 'appId-2___viewId-1', - type, - score: 0, - references: [], - attributes: { - appId: 'appId-2', - viewId: 'viewId-1', - minutesOnScreen: 1, - numberOfClicks: 1, - }, - }, - ], - total: 3, - page, - per_page: perPage, - }; - default: - throw new Error(`Unexpected type [${type}]`); - } - }); - await expect(rollTotals(logger, savedObjectClient)).resolves.toBe(undefined); - expect(savedObjectClient.bulkCreate).toHaveBeenCalledTimes(1); - expect(savedObjectClient.bulkCreate).toHaveBeenCalledWith( - [ - { - type: SAVED_OBJECTS_TOTAL_TYPE, - id: 'appId-1', - attributes: { - appId: 'appId-1', - viewId: MAIN_APP_DEFAULT_VIEW_ID, - minutesOnScreen: 3.0, - numberOfClicks: 3, - }, - }, - { - type: SAVED_OBJECTS_TOTAL_TYPE, - id: 'appId-1___viewId-1', - attributes: { - appId: 'appId-1', - viewId: 'viewId-1', - minutesOnScreen: 5.0, - numberOfClicks: 3, - }, - }, - { - type: SAVED_OBJECTS_TOTAL_TYPE, - id: 'appId-2___viewId-1', - attributes: { - appId: 'appId-2', - viewId: 'viewId-1', - minutesOnScreen: 1.0, - numberOfClicks: 1, - }, - }, - { - type: SAVED_OBJECTS_TOTAL_TYPE, - id: 'appId-2', - attributes: { - appId: 'appId-2', - viewId: MAIN_APP_DEFAULT_VIEW_ID, - minutesOnScreen: 0.5, - numberOfClicks: 1, - }, - }, - ], - { overwrite: true } - ); - expect(savedObjectClient.delete).toHaveBeenCalledTimes(3); - expect(savedObjectClient.delete).toHaveBeenCalledWith( - SAVED_OBJECTS_DAILY_TYPE, - 'appId-2:2020-01-01' - ); - expect(savedObjectClient.delete).toHaveBeenCalledWith( - SAVED_OBJECTS_DAILY_TYPE, - 'appId-1:2020-01-01' - ); - expect(savedObjectClient.delete).toHaveBeenCalledWith( - SAVED_OBJECTS_DAILY_TYPE, - 'appId-1:2020-01-01:viewId-1' - ); - }); -}); diff --git a/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups.ts b/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/daily.ts similarity index 55% rename from src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups.ts rename to src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/daily.ts index df7e7662b49cf..a7873c7d5dfe9 100644 --- a/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups.ts +++ b/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/daily.ts @@ -6,18 +6,20 @@ * Side Public License, v 1. */ -import { ISavedObjectsRepository, SavedObject, Logger } from 'kibana/server'; import moment from 'moment'; +import type { Logger } from '@kbn/logging'; +import { + ISavedObjectsRepository, + SavedObject, + SavedObjectsErrorHelpers, +} from '../../../../../../core/server'; +import { getDailyId } from '../../../../../usage_collection/common/application_usage'; import { ApplicationUsageDaily, - ApplicationUsageTotal, ApplicationUsageTransactional, SAVED_OBJECTS_DAILY_TYPE, - SAVED_OBJECTS_TOTAL_TYPE, SAVED_OBJECTS_TRANSACTIONAL_TYPE, -} from './saved_objects_types'; -import { SavedObjectsErrorHelpers } from '../../../../../../src/core/server'; -import { MAIN_APP_DEFAULT_VIEW_ID } from '../../../../usage_collection/common/constants'; +} from '../saved_objects_types'; /** * For Rolling the daily data, we only care about the stored attributes and the version (to avoid overwriting via concurrent requests) @@ -27,18 +29,17 @@ type ApplicationUsageDailyWithVersion = Pick< 'version' | 'attributes' >; -export function serializeKey(appId: string, viewId: string) { - return `${appId}___${viewId}`; -} - /** * Aggregates all the transactional events into daily aggregates * @param logger * @param savedObjectsClient */ -export async function rollDailyData(logger: Logger, savedObjectsClient?: ISavedObjectsRepository) { +export async function rollDailyData( + logger: Logger, + savedObjectsClient?: ISavedObjectsRepository +): Promise { if (!savedObjectsClient) { - return; + return false; } try { @@ -58,10 +59,7 @@ export async function rollDailyData(logger: Logger, savedObjectsClient?: ISavedO } = doc; const dayId = moment(timestamp).format('YYYY-MM-DD'); - const dailyId = - !viewId || viewId === MAIN_APP_DEFAULT_VIEW_ID - ? `${appId}:${dayId}` - : `${appId}:${dayId}:${viewId}`; + const dailyId = getDailyId({ dayId, appId, viewId }); const existingDoc = toCreate.get(dailyId) || @@ -103,9 +101,11 @@ export async function rollDailyData(logger: Logger, savedObjectsClient?: ISavedO } } } while (toCreate.size > 0); + return true; } catch (err) { logger.debug(`Failed to rollup transactional to daily entries`); logger.debug(err); + return false; } } @@ -125,7 +125,11 @@ async function getDailyDoc( dayId: string ): Promise { try { - return await savedObjectsClient.get(SAVED_OBJECTS_DAILY_TYPE, id); + const { attributes, version } = await savedObjectsClient.get( + SAVED_OBJECTS_DAILY_TYPE, + id + ); + return { attributes, version }; } catch (err) { if (SavedObjectsErrorHelpers.isNotFoundError(err)) { return { @@ -142,91 +146,3 @@ async function getDailyDoc( throw err; } } - -/** - * Moves all the daily documents into aggregated "total" documents as we don't care about any granularity after 90 days - * @param logger - * @param savedObjectsClient - */ -export async function rollTotals(logger: Logger, savedObjectsClient?: ISavedObjectsRepository) { - if (!savedObjectsClient) { - return; - } - - try { - const [ - { saved_objects: rawApplicationUsageTotals }, - { saved_objects: rawApplicationUsageDaily }, - ] = await Promise.all([ - savedObjectsClient.find({ - perPage: 10000, - type: SAVED_OBJECTS_TOTAL_TYPE, - }), - savedObjectsClient.find({ - perPage: 10000, - type: SAVED_OBJECTS_DAILY_TYPE, - filter: `${SAVED_OBJECTS_DAILY_TYPE}.attributes.timestamp < now-90d`, - }), - ]); - - const existingTotals = rawApplicationUsageTotals.reduce( - ( - acc, - { - attributes: { appId, viewId = MAIN_APP_DEFAULT_VIEW_ID, numberOfClicks, minutesOnScreen }, - } - ) => { - const key = viewId === MAIN_APP_DEFAULT_VIEW_ID ? appId : serializeKey(appId, viewId); - - return { - ...acc, - // No need to sum because there should be 1 document per appId only - [key]: { appId, viewId, numberOfClicks, minutesOnScreen }, - }; - }, - {} as Record< - string, - { appId: string; viewId: string; minutesOnScreen: number; numberOfClicks: number } - > - ); - - const totals = rawApplicationUsageDaily.reduce((acc, { attributes }) => { - const { - appId, - viewId = MAIN_APP_DEFAULT_VIEW_ID, - numberOfClicks, - minutesOnScreen, - } = attributes; - const key = viewId === MAIN_APP_DEFAULT_VIEW_ID ? appId : serializeKey(appId, viewId); - const existing = acc[key] || { minutesOnScreen: 0, numberOfClicks: 0 }; - - return { - ...acc, - [key]: { - appId, - viewId, - numberOfClicks: numberOfClicks + existing.numberOfClicks, - minutesOnScreen: minutesOnScreen + existing.minutesOnScreen, - }, - }; - }, existingTotals); - - await Promise.all([ - Object.entries(totals).length && - savedObjectsClient.bulkCreate( - Object.entries(totals).map(([id, entry]) => ({ - type: SAVED_OBJECTS_TOTAL_TYPE, - id, - attributes: entry, - })), - { overwrite: true } - ), - ...rawApplicationUsageDaily.map( - ({ id }) => savedObjectsClient.delete(SAVED_OBJECTS_DAILY_TYPE, id) // There is no bulkDelete :( - ), - ]); - } catch (err) { - logger.debug(`Failed to rollup daily entries to totals`); - logger.debug(err); - } -} diff --git a/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/index.ts b/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/index.ts new file mode 100644 index 0000000000000..8f3d83613aa9d --- /dev/null +++ b/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/index.ts @@ -0,0 +1,11 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +export { rollDailyData } from './daily'; +export { rollTotals } from './total'; +export { serializeKey } from './utils'; diff --git a/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/total.test.ts b/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/total.test.ts new file mode 100644 index 0000000000000..9fea955ab5d8a --- /dev/null +++ b/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/total.test.ts @@ -0,0 +1,194 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import { savedObjectsRepositoryMock, loggingSystemMock } from '../../../../../../core/server/mocks'; +import { MAIN_APP_DEFAULT_VIEW_ID } from '../../../../../usage_collection/common/constants'; +import { SAVED_OBJECTS_DAILY_TYPE, SAVED_OBJECTS_TOTAL_TYPE } from '../saved_objects_types'; +import { rollTotals } from './total'; + +describe('rollTotals', () => { + const logger = loggingSystemMock.createLogger(); + + test('returns undefined if no savedObjectsClient initialised yet', async () => { + await expect(rollTotals(logger, undefined)).resolves.toBe(undefined); + }); + + test('handle empty results', async () => { + const savedObjectClient = savedObjectsRepositoryMock.create(); + savedObjectClient.find.mockImplementation(async ({ type, page = 1, perPage = 10 }) => { + switch (type) { + case SAVED_OBJECTS_DAILY_TYPE: + case SAVED_OBJECTS_TOTAL_TYPE: + return { saved_objects: [], total: 0, page, per_page: perPage }; + default: + throw new Error(`Unexpected type [${type}]`); + } + }); + await expect(rollTotals(logger, savedObjectClient)).resolves.toBe(undefined); + expect(savedObjectClient.bulkCreate).toHaveBeenCalledTimes(0); + expect(savedObjectClient.delete).toHaveBeenCalledTimes(0); + }); + + test('migrate some documents', async () => { + const savedObjectClient = savedObjectsRepositoryMock.create(); + savedObjectClient.find.mockImplementation(async ({ type, page = 1, perPage = 10 }) => { + switch (type) { + case SAVED_OBJECTS_DAILY_TYPE: + return { + saved_objects: [ + { + id: 'appId-2:2020-01-01', + type, + score: 0, + references: [], + attributes: { + appId: 'appId-2', + timestamp: '2020-01-01T10:31:00.000Z', + minutesOnScreen: 0.5, + numberOfClicks: 1, + }, + }, + { + id: 'appId-1:2020-01-01', + type, + score: 0, + references: [], + attributes: { + appId: 'appId-1', + timestamp: '2020-01-01T11:31:00.000Z', + minutesOnScreen: 2.5, + numberOfClicks: 2, + }, + }, + { + id: 'appId-1:2020-01-01:viewId-1', + type, + score: 0, + references: [], + attributes: { + appId: 'appId-1', + viewId: 'viewId-1', + timestamp: '2020-01-01T11:31:00.000Z', + minutesOnScreen: 1, + numberOfClicks: 1, + }, + }, + ], + total: 3, + page, + per_page: perPage, + }; + case SAVED_OBJECTS_TOTAL_TYPE: + return { + saved_objects: [ + { + id: 'appId-1', + type, + score: 0, + references: [], + attributes: { + appId: 'appId-1', + minutesOnScreen: 0.5, + numberOfClicks: 1, + }, + }, + { + id: 'appId-1___viewId-1', + type, + score: 0, + references: [], + attributes: { + appId: 'appId-1', + viewId: 'viewId-1', + minutesOnScreen: 4, + numberOfClicks: 2, + }, + }, + { + id: 'appId-2___viewId-1', + type, + score: 0, + references: [], + attributes: { + appId: 'appId-2', + viewId: 'viewId-1', + minutesOnScreen: 1, + numberOfClicks: 1, + }, + }, + ], + total: 3, + page, + per_page: perPage, + }; + default: + throw new Error(`Unexpected type [${type}]`); + } + }); + await expect(rollTotals(logger, savedObjectClient)).resolves.toBe(undefined); + expect(savedObjectClient.bulkCreate).toHaveBeenCalledTimes(1); + expect(savedObjectClient.bulkCreate).toHaveBeenCalledWith( + [ + { + type: SAVED_OBJECTS_TOTAL_TYPE, + id: 'appId-1', + attributes: { + appId: 'appId-1', + viewId: MAIN_APP_DEFAULT_VIEW_ID, + minutesOnScreen: 3.0, + numberOfClicks: 3, + }, + }, + { + type: SAVED_OBJECTS_TOTAL_TYPE, + id: 'appId-1___viewId-1', + attributes: { + appId: 'appId-1', + viewId: 'viewId-1', + minutesOnScreen: 5.0, + numberOfClicks: 3, + }, + }, + { + type: SAVED_OBJECTS_TOTAL_TYPE, + id: 'appId-2___viewId-1', + attributes: { + appId: 'appId-2', + viewId: 'viewId-1', + minutesOnScreen: 1.0, + numberOfClicks: 1, + }, + }, + { + type: SAVED_OBJECTS_TOTAL_TYPE, + id: 'appId-2', + attributes: { + appId: 'appId-2', + viewId: MAIN_APP_DEFAULT_VIEW_ID, + minutesOnScreen: 0.5, + numberOfClicks: 1, + }, + }, + ], + { overwrite: true } + ); + expect(savedObjectClient.delete).toHaveBeenCalledTimes(3); + expect(savedObjectClient.delete).toHaveBeenCalledWith( + SAVED_OBJECTS_DAILY_TYPE, + 'appId-2:2020-01-01' + ); + expect(savedObjectClient.delete).toHaveBeenCalledWith( + SAVED_OBJECTS_DAILY_TYPE, + 'appId-1:2020-01-01' + ); + expect(savedObjectClient.delete).toHaveBeenCalledWith( + SAVED_OBJECTS_DAILY_TYPE, + 'appId-1:2020-01-01:viewId-1' + ); + }); +}); diff --git a/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/total.ts b/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/total.ts new file mode 100644 index 0000000000000..e27c7b897d995 --- /dev/null +++ b/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/total.ts @@ -0,0 +1,106 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import type { Logger } from '@kbn/logging'; +import type { ISavedObjectsRepository } from 'kibana/server'; +import { MAIN_APP_DEFAULT_VIEW_ID } from '../../../../../usage_collection/common/constants'; +import { + ApplicationUsageDaily, + ApplicationUsageTotal, + SAVED_OBJECTS_DAILY_TYPE, + SAVED_OBJECTS_TOTAL_TYPE, +} from '../saved_objects_types'; +import { serializeKey } from './utils'; + +/** + * Moves all the daily documents into aggregated "total" documents as we don't care about any granularity after 90 days + * @param logger + * @param savedObjectsClient + */ +export async function rollTotals(logger: Logger, savedObjectsClient?: ISavedObjectsRepository) { + if (!savedObjectsClient) { + return; + } + + try { + const [ + { saved_objects: rawApplicationUsageTotals }, + { saved_objects: rawApplicationUsageDaily }, + ] = await Promise.all([ + savedObjectsClient.find({ + perPage: 10000, + type: SAVED_OBJECTS_TOTAL_TYPE, + }), + savedObjectsClient.find({ + perPage: 10000, + type: SAVED_OBJECTS_DAILY_TYPE, + filter: `${SAVED_OBJECTS_DAILY_TYPE}.attributes.timestamp < now-90d`, + }), + ]); + + const existingTotals = rawApplicationUsageTotals.reduce( + ( + acc, + { + attributes: { appId, viewId = MAIN_APP_DEFAULT_VIEW_ID, numberOfClicks, minutesOnScreen }, + } + ) => { + const key = viewId === MAIN_APP_DEFAULT_VIEW_ID ? appId : serializeKey(appId, viewId); + + return { + ...acc, + // No need to sum because there should be 1 document per appId only + [key]: { appId, viewId, numberOfClicks, minutesOnScreen }, + }; + }, + {} as Record< + string, + { appId: string; viewId: string; minutesOnScreen: number; numberOfClicks: number } + > + ); + + const totals = rawApplicationUsageDaily.reduce((acc, { attributes }) => { + const { + appId, + viewId = MAIN_APP_DEFAULT_VIEW_ID, + numberOfClicks, + minutesOnScreen, + } = attributes; + const key = viewId === MAIN_APP_DEFAULT_VIEW_ID ? appId : serializeKey(appId, viewId); + const existing = acc[key] || { minutesOnScreen: 0, numberOfClicks: 0 }; + + return { + ...acc, + [key]: { + appId, + viewId, + numberOfClicks: numberOfClicks + existing.numberOfClicks, + minutesOnScreen: minutesOnScreen + existing.minutesOnScreen, + }, + }; + }, existingTotals); + + await Promise.all([ + Object.entries(totals).length && + savedObjectsClient.bulkCreate( + Object.entries(totals).map(([id, entry]) => ({ + type: SAVED_OBJECTS_TOTAL_TYPE, + id, + attributes: entry, + })), + { overwrite: true } + ), + ...rawApplicationUsageDaily.map( + ({ id }) => savedObjectsClient.delete(SAVED_OBJECTS_DAILY_TYPE, id) // There is no bulkDelete :( + ), + ]); + } catch (err) { + logger.debug(`Failed to rollup daily entries to totals`); + logger.debug(err); + } +} diff --git a/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/utils.ts b/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/utils.ts new file mode 100644 index 0000000000000..8be00e6287883 --- /dev/null +++ b/src/plugins/kibana_usage_collection/server/collectors/application_usage/rollups/utils.ts @@ -0,0 +1,11 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +export function serializeKey(appId: string, viewId: string) { + return `${appId}___${viewId}`; +} diff --git a/src/plugins/kibana_usage_collection/server/collectors/application_usage/saved_objects_types.ts b/src/plugins/kibana_usage_collection/server/collectors/application_usage/saved_objects_types.ts index 9e71b5c3b032e..f2b996f3af97a 100644 --- a/src/plugins/kibana_usage_collection/server/collectors/application_usage/saved_objects_types.ts +++ b/src/plugins/kibana_usage_collection/server/collectors/application_usage/saved_objects_types.ts @@ -6,7 +6,7 @@ * Side Public License, v 1. */ -import { SavedObjectAttributes, SavedObjectsServiceSetup } from 'kibana/server'; +import type { SavedObjectAttributes, SavedObjectsServiceSetup } from 'kibana/server'; /** * Used for accumulating the totals of all the stats older than 90d @@ -17,6 +17,7 @@ export interface ApplicationUsageTotal extends SavedObjectAttributes { minutesOnScreen: number; numberOfClicks: number; } + export const SAVED_OBJECTS_TOTAL_TYPE = 'application_usage_totals'; /** @@ -25,6 +26,8 @@ export const SAVED_OBJECTS_TOTAL_TYPE = 'application_usage_totals'; export interface ApplicationUsageTransactional extends ApplicationUsageTotal { timestamp: string; } + +/** @deprecated transactional type is no longer used, and only preserved for backward compatibility */ export const SAVED_OBJECTS_TRANSACTIONAL_TYPE = 'application_usage_transactional'; /** @@ -62,6 +65,7 @@ export function registerMappings(registerType: SavedObjectsServiceSetup['registe }); // Type for storing ApplicationUsageTransactional (declaring empty mappings because we don't use the internal fields for query/aggregations) + // Remark: this type is deprecated and only here for BWC reasons. registerType({ name: SAVED_OBJECTS_TRANSACTIONAL_TYPE, hidden: false, diff --git a/src/plugins/kibana_usage_collection/server/collectors/application_usage/schema.ts b/src/plugins/kibana_usage_collection/server/collectors/application_usage/schema.ts index 062d751ef454c..693e9132fe536 100644 --- a/src/plugins/kibana_usage_collection/server/collectors/application_usage/schema.ts +++ b/src/plugins/kibana_usage_collection/server/collectors/application_usage/schema.ts @@ -7,7 +7,7 @@ */ import { MakeSchemaFrom } from 'src/plugins/usage_collection/server'; -import { ApplicationUsageTelemetryReport } from './telemetry_application_usage_collector'; +import { ApplicationUsageTelemetryReport } from './types'; const commonSchema: MakeSchemaFrom = { appId: { type: 'keyword', _meta: { description: 'The application being tracked' } }, diff --git a/src/plugins/kibana_usage_collection/server/collectors/application_usage/telemetry_application_usage_collector.test.ts b/src/plugins/kibana_usage_collection/server/collectors/application_usage/telemetry_application_usage_collector.test.ts index 3e8434d446033..f1b21af5506e6 100644 --- a/src/plugins/kibana_usage_collection/server/collectors/application_usage/telemetry_application_usage_collector.test.ts +++ b/src/plugins/kibana_usage_collection/server/collectors/application_usage/telemetry_application_usage_collector.test.ts @@ -11,74 +11,99 @@ import { Collector, createUsageCollectionSetupMock, } from '../../../../usage_collection/server/usage_collection.mock'; - +import { MAIN_APP_DEFAULT_VIEW_ID } from '../../../../usage_collection/common/constants'; import { createCollectorFetchContextMock } from 'src/plugins/usage_collection/server/mocks'; -import { ROLL_TOTAL_INDICES_INTERVAL, ROLL_INDICES_START } from './constants'; import { registerApplicationUsageCollector, transformByApplicationViews, - ApplicationUsageViews, } from './telemetry_application_usage_collector'; -import { MAIN_APP_DEFAULT_VIEW_ID } from '../../../../usage_collection/common/constants'; -import { - SAVED_OBJECTS_DAILY_TYPE, - SAVED_OBJECTS_TOTAL_TYPE, - SAVED_OBJECTS_TRANSACTIONAL_TYPE, -} from './saved_objects_types'; +import { ApplicationUsageViews } from './types'; -describe('telemetry_application_usage', () => { - jest.useFakeTimers(); +import { SAVED_OBJECTS_DAILY_TYPE, SAVED_OBJECTS_TOTAL_TYPE } from './saved_objects_types'; - const logger = loggingSystemMock.createLogger(); +// use fake timers to avoid triggering rollups during tests +jest.useFakeTimers(); +describe('telemetry_application_usage', () => { + let logger: ReturnType; let collector: Collector; + let usageCollectionMock: ReturnType; + let savedObjectClient: ReturnType; + let getSavedObjectClient: jest.MockedFunction<() => undefined | typeof savedObjectClient>; - const usageCollectionMock = createUsageCollectionSetupMock(); - usageCollectionMock.makeUsageCollector.mockImplementation((config) => { - collector = new Collector(logger, config); - return createUsageCollectionSetupMock().makeUsageCollector(config); - }); - - const getUsageCollector = jest.fn(); const registerType = jest.fn(); const mockedFetchContext = createCollectorFetchContextMock(); - beforeAll(() => - registerApplicationUsageCollector(logger, usageCollectionMock, registerType, getUsageCollector) - ); - afterAll(() => jest.clearAllTimers()); + beforeEach(() => { + logger = loggingSystemMock.createLogger(); + usageCollectionMock = createUsageCollectionSetupMock(); + savedObjectClient = savedObjectsRepositoryMock.create(); + getSavedObjectClient = jest.fn().mockReturnValue(savedObjectClient); + usageCollectionMock.makeUsageCollector.mockImplementation((config) => { + collector = new Collector(logger, config); + return createUsageCollectionSetupMock().makeUsageCollector(config); + }); + registerApplicationUsageCollector( + logger, + usageCollectionMock, + registerType, + getSavedObjectClient + ); + }); + + afterEach(() => { + jest.clearAllTimers(); + }); test('registered collector is set', () => { expect(collector).not.toBeUndefined(); }); test('if no savedObjectClient initialised, return undefined', async () => { + getSavedObjectClient.mockReturnValue(undefined); + expect(collector.isReady()).toBe(false); expect(await collector.fetch(mockedFetchContext)).toBeUndefined(); - jest.runTimersToTime(ROLL_INDICES_START); }); - test('when savedObjectClient is initialised, return something', async () => { - const savedObjectClient = savedObjectsRepositoryMock.create(); - savedObjectClient.find.mockImplementation( - async () => - ({ - saved_objects: [], - total: 0, - } as any) + test('calls `savedObjectsClient.find` with the correct parameters', async () => { + savedObjectClient.find.mockResolvedValue({ + saved_objects: [], + total: 0, + per_page: 20, + page: 0, + }); + + await collector.fetch(mockedFetchContext); + + expect(savedObjectClient.find).toHaveBeenCalledTimes(2); + + expect(savedObjectClient.find).toHaveBeenCalledWith( + expect.objectContaining({ + type: SAVED_OBJECTS_TOTAL_TYPE, + }) + ); + expect(savedObjectClient.find).toHaveBeenCalledWith( + expect.objectContaining({ + type: SAVED_OBJECTS_DAILY_TYPE, + }) ); - getUsageCollector.mockImplementation(() => savedObjectClient); + }); - jest.runTimersToTime(ROLL_TOTAL_INDICES_INTERVAL); // Force rollTotals to run + test('when savedObjectClient is initialised, return something', async () => { + savedObjectClient.find.mockResolvedValue({ + saved_objects: [], + total: 0, + per_page: 20, + page: 0, + }); expect(collector.isReady()).toBe(true); expect(await collector.fetch(mockedFetchContext)).toStrictEqual({}); expect(savedObjectClient.bulkCreate).not.toHaveBeenCalled(); }); - test('it only gets 10k even when there are more documents (ES limitation)', async () => { - const savedObjectClient = savedObjectsRepositoryMock.create(); - const total = 10000; + test('it aggregates total and daily data', async () => { savedObjectClient.find.mockImplementation(async (opts) => { switch (opts.type) { case SAVED_OBJECTS_TOTAL_TYPE: @@ -95,18 +120,6 @@ describe('telemetry_application_usage', () => { ], total: 1, } as any; - case SAVED_OBJECTS_TRANSACTIONAL_TYPE: - const doc = { - id: 'test-id', - attributes: { - appId: 'appId', - timestamp: new Date().toISOString(), - minutesOnScreen: 0.5, - numberOfClicks: 1, - }, - }; - const savedObjects = new Array(total).fill(doc); - return { saved_objects: savedObjects, total: total + 1 }; case SAVED_OBJECTS_DAILY_TYPE: return { saved_objects: [ @@ -125,122 +138,21 @@ describe('telemetry_application_usage', () => { } }); - getUsageCollector.mockImplementation(() => savedObjectClient); - - jest.runTimersToTime(ROLL_TOTAL_INDICES_INTERVAL); // Force rollTotals to run - expect(await collector.fetch(mockedFetchContext)).toStrictEqual({ appId: { appId: 'appId', viewId: 'main', - clicks_total: total + 1 + 10, - clicks_7_days: total + 1, - clicks_30_days: total + 1, - clicks_90_days: total + 1, - minutes_on_screen_total: (total + 1) * 0.5 + 10, - minutes_on_screen_7_days: (total + 1) * 0.5, - minutes_on_screen_30_days: (total + 1) * 0.5, - minutes_on_screen_90_days: (total + 1) * 0.5, + clicks_total: 1 + 10, + clicks_7_days: 1, + clicks_30_days: 1, + clicks_90_days: 1, + minutes_on_screen_total: 0.5 + 10, + minutes_on_screen_7_days: 0.5, + minutes_on_screen_30_days: 0.5, + minutes_on_screen_90_days: 0.5, views: [], }, }); - expect(savedObjectClient.bulkCreate).toHaveBeenCalledWith( - [ - { - id: 'appId', - type: SAVED_OBJECTS_TOTAL_TYPE, - attributes: { - appId: 'appId', - viewId: 'main', - minutesOnScreen: 10.5, - numberOfClicks: 11, - }, - }, - ], - { overwrite: true } - ); - expect(savedObjectClient.delete).toHaveBeenCalledTimes(1); - expect(savedObjectClient.delete).toHaveBeenCalledWith( - SAVED_OBJECTS_DAILY_TYPE, - 'appId:YYYY-MM-DD' - ); - }); - - test('old transactional data not migrated yet', async () => { - const savedObjectClient = savedObjectsRepositoryMock.create(); - savedObjectClient.find.mockImplementation(async (opts) => { - switch (opts.type) { - case SAVED_OBJECTS_TOTAL_TYPE: - case SAVED_OBJECTS_DAILY_TYPE: - return { saved_objects: [], total: 0 } as any; - case SAVED_OBJECTS_TRANSACTIONAL_TYPE: - return { - saved_objects: [ - { - id: 'test-id', - attributes: { - appId: 'appId', - timestamp: new Date(0).toISOString(), - minutesOnScreen: 0.5, - numberOfClicks: 1, - }, - }, - { - id: 'test-id-2', - attributes: { - appId: 'appId', - viewId: 'main', - timestamp: new Date(0).toISOString(), - minutesOnScreen: 2, - numberOfClicks: 2, - }, - }, - { - id: 'test-id-3', - attributes: { - appId: 'appId', - viewId: 'viewId-1', - timestamp: new Date(0).toISOString(), - minutesOnScreen: 1, - numberOfClicks: 1, - }, - }, - ], - total: 1, - }; - } - }); - - getUsageCollector.mockImplementation(() => savedObjectClient); - - expect(await collector.fetch(mockedFetchContext)).toStrictEqual({ - appId: { - appId: 'appId', - viewId: 'main', - clicks_total: 3, - clicks_7_days: 0, - clicks_30_days: 0, - clicks_90_days: 0, - minutes_on_screen_total: 2.5, - minutes_on_screen_7_days: 0, - minutes_on_screen_30_days: 0, - minutes_on_screen_90_days: 0, - views: [ - { - appId: 'appId', - viewId: 'viewId-1', - clicks_total: 1, - clicks_7_days: 0, - clicks_30_days: 0, - clicks_90_days: 0, - minutes_on_screen_total: 1, - minutes_on_screen_7_days: 0, - minutes_on_screen_30_days: 0, - minutes_on_screen_90_days: 0, - }, - ], - }, - }); }); }); diff --git a/src/plugins/kibana_usage_collection/server/collectors/application_usage/telemetry_application_usage_collector.ts b/src/plugins/kibana_usage_collection/server/collectors/application_usage/telemetry_application_usage_collector.ts index ee1b42e61a6ca..a01f1bca4f0e0 100644 --- a/src/plugins/kibana_usage_collection/server/collectors/application_usage/telemetry_application_usage_collector.ts +++ b/src/plugins/kibana_usage_collection/server/collectors/application_usage/telemetry_application_usage_collector.ts @@ -11,57 +11,21 @@ import { timer } from 'rxjs'; import { ISavedObjectsRepository, Logger, SavedObjectsServiceSetup } from 'kibana/server'; import { UsageCollectionSetup } from 'src/plugins/usage_collection/server'; import { MAIN_APP_DEFAULT_VIEW_ID } from '../../../../usage_collection/common/constants'; -import { serializeKey } from './rollups'; - import { ApplicationUsageDaily, ApplicationUsageTotal, - ApplicationUsageTransactional, registerMappings, SAVED_OBJECTS_DAILY_TYPE, SAVED_OBJECTS_TOTAL_TYPE, - SAVED_OBJECTS_TRANSACTIONAL_TYPE, } from './saved_objects_types'; import { applicationUsageSchema } from './schema'; -import { rollDailyData, rollTotals } from './rollups'; +import { rollTotals, rollDailyData, serializeKey } from './rollups'; import { ROLL_TOTAL_INDICES_INTERVAL, ROLL_DAILY_INDICES_INTERVAL, ROLL_INDICES_START, } from './constants'; - -export interface ApplicationViewUsage { - appId: string; - viewId: string; - clicks_total: number; - clicks_7_days: number; - clicks_30_days: number; - clicks_90_days: number; - minutes_on_screen_total: number; - minutes_on_screen_7_days: number; - minutes_on_screen_30_days: number; - minutes_on_screen_90_days: number; -} - -export interface ApplicationUsageViews { - [serializedKey: string]: ApplicationViewUsage; -} - -export interface ApplicationUsageTelemetryReport { - [appId: string]: { - appId: string; - viewId: string; - clicks_total: number; - clicks_7_days: number; - clicks_30_days: number; - clicks_90_days: number; - minutes_on_screen_total: number; - minutes_on_screen_7_days: number; - minutes_on_screen_30_days: number; - minutes_on_screen_90_days: number; - views?: ApplicationViewUsage[]; - }; -} +import { ApplicationUsageTelemetryReport, ApplicationUsageViews } from './types'; export const transformByApplicationViews = ( report: ApplicationUsageViews @@ -92,6 +56,21 @@ export function registerApplicationUsageCollector( ) { registerMappings(registerType); + timer(ROLL_INDICES_START, ROLL_TOTAL_INDICES_INTERVAL).subscribe(() => + rollTotals(logger, getSavedObjectsClient()) + ); + + const dailyRollingSub = timer(ROLL_INDICES_START, ROLL_DAILY_INDICES_INTERVAL).subscribe( + async () => { + const success = await rollDailyData(logger, getSavedObjectsClient()); + // we only need to roll the transactional documents once to assure BWC + // once we rolling succeeds, we can stop. + if (success) { + dailyRollingSub.unsubscribe(); + } + } + ); + const collector = usageCollection.makeUsageCollector( { type: 'application_usage', @@ -105,7 +84,6 @@ export function registerApplicationUsageCollector( const [ { saved_objects: rawApplicationUsageTotals }, { saved_objects: rawApplicationUsageDaily }, - { saved_objects: rawApplicationUsageTransactional }, ] = await Promise.all([ savedObjectsClient.find({ type: SAVED_OBJECTS_TOTAL_TYPE, @@ -115,10 +93,6 @@ export function registerApplicationUsageCollector( type: SAVED_OBJECTS_DAILY_TYPE, perPage: 10000, // We can have up to 44 apps * 91 days = 4004 docs. This limit is OK }), - savedObjectsClient.find({ - type: SAVED_OBJECTS_TRANSACTIONAL_TYPE, - perPage: 10000, // If we have more than those, we won't report the rest (they'll be rolled up to the daily soon enough to become a problem) - }), ]); const applicationUsageFromTotals = rawApplicationUsageTotals.reduce( @@ -156,10 +130,7 @@ export function registerApplicationUsageCollector( const nowMinus30 = moment().subtract(30, 'days'); const nowMinus90 = moment().subtract(90, 'days'); - const applicationUsage = [ - ...rawApplicationUsageDaily, - ...rawApplicationUsageTransactional, - ].reduce( + const applicationUsage = rawApplicationUsageDaily.reduce( ( acc, { @@ -224,11 +195,4 @@ export function registerApplicationUsageCollector( ); usageCollection.registerCollector(collector); - - timer(ROLL_INDICES_START, ROLL_DAILY_INDICES_INTERVAL).subscribe(() => - rollDailyData(logger, getSavedObjectsClient()) - ); - timer(ROLL_INDICES_START, ROLL_TOTAL_INDICES_INTERVAL).subscribe(() => - rollTotals(logger, getSavedObjectsClient()) - ); } diff --git a/src/plugins/kibana_usage_collection/server/collectors/application_usage/types.ts b/src/plugins/kibana_usage_collection/server/collectors/application_usage/types.ts new file mode 100644 index 0000000000000..bef835e922d8d --- /dev/null +++ b/src/plugins/kibana_usage_collection/server/collectors/application_usage/types.ts @@ -0,0 +1,40 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +export interface ApplicationViewUsage { + appId: string; + viewId: string; + clicks_total: number; + clicks_7_days: number; + clicks_30_days: number; + clicks_90_days: number; + minutes_on_screen_total: number; + minutes_on_screen_7_days: number; + minutes_on_screen_30_days: number; + minutes_on_screen_90_days: number; +} + +export interface ApplicationUsageViews { + [serializedKey: string]: ApplicationViewUsage; +} + +export interface ApplicationUsageTelemetryReport { + [appId: string]: { + appId: string; + viewId: string; + clicks_total: number; + clicks_7_days: number; + clicks_30_days: number; + clicks_90_days: number; + minutes_on_screen_total: number; + minutes_on_screen_7_days: number; + minutes_on_screen_30_days: number; + minutes_on_screen_90_days: number; + views?: ApplicationViewUsage[]; + }; +} diff --git a/src/plugins/timelion/server/plugin.ts b/src/plugins/timelion/server/plugin.ts index 66348c572117d..226a978fe5d88 100644 --- a/src/plugins/timelion/server/plugin.ts +++ b/src/plugins/timelion/server/plugin.ts @@ -47,6 +47,7 @@ export class TimelionPlugin implements Plugin { core.capabilities.registerProvider(() => ({ timelion: { save: true, + show: true, }, })); core.savedObjects.registerType(timelionSheetSavedObjectType); diff --git a/src/plugins/timelion/server/saved_objects/timelion_sheet.ts b/src/plugins/timelion/server/saved_objects/timelion_sheet.ts index 52d7f59a7c734..231e049280bb1 100644 --- a/src/plugins/timelion/server/saved_objects/timelion_sheet.ts +++ b/src/plugins/timelion/server/saved_objects/timelion_sheet.ts @@ -12,6 +12,20 @@ export const timelionSheetSavedObjectType: SavedObjectsType = { name: 'timelion-sheet', hidden: false, namespaceType: 'single', + management: { + icon: 'visTimelion', + defaultSearchField: 'title', + importableAndExportable: true, + getTitle(obj) { + return obj.attributes.title; + }, + getInAppUrl(obj) { + return { + path: `/app/timelion#/${encodeURIComponent(obj.id)}`, + uiCapabilitiesPath: 'timelion.show', + }; + }, + }, mappings: { properties: { description: { type: 'text' }, diff --git a/src/plugins/usage_collection/common/application_usage.ts b/src/plugins/usage_collection/common/application_usage.ts new file mode 100644 index 0000000000000..c9dd489000d35 --- /dev/null +++ b/src/plugins/usage_collection/common/application_usage.ts @@ -0,0 +1,23 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import { MAIN_APP_DEFAULT_VIEW_ID } from './constants'; + +export const getDailyId = ({ + appId, + dayId, + viewId, +}: { + viewId: string; + appId: string; + dayId: string; +}) => { + return !viewId || viewId === MAIN_APP_DEFAULT_VIEW_ID + ? `${appId}:${dayId}` + : `${appId}:${dayId}:${viewId}`; +}; diff --git a/src/plugins/usage_collection/server/report/schema.ts b/src/plugins/usage_collection/server/report/schema.ts index 93203a33cd1e1..350ec8d90e765 100644 --- a/src/plugins/usage_collection/server/report/schema.ts +++ b/src/plugins/usage_collection/server/report/schema.ts @@ -9,6 +9,13 @@ import { schema, TypeOf } from '@kbn/config-schema'; import { METRIC_TYPE } from '@kbn/analytics'; +const applicationUsageReportSchema = schema.object({ + minutesOnScreen: schema.number(), + numberOfClicks: schema.number(), + appId: schema.string(), + viewId: schema.string(), +}); + export const reportSchema = schema.object({ reportVersion: schema.maybe(schema.oneOf([schema.literal(3)])), userAgent: schema.maybe( @@ -38,17 +45,8 @@ export const reportSchema = schema.object({ }) ) ), - application_usage: schema.maybe( - schema.recordOf( - schema.string(), - schema.object({ - minutesOnScreen: schema.number(), - numberOfClicks: schema.number(), - appId: schema.string(), - viewId: schema.string(), - }) - ) - ), + application_usage: schema.maybe(schema.recordOf(schema.string(), applicationUsageReportSchema)), }); export type ReportSchemaType = TypeOf; +export type ApplicationUsageReport = TypeOf; diff --git a/src/plugins/usage_collection/server/report/store_application_usage.test.ts b/src/plugins/usage_collection/server/report/store_application_usage.test.ts new file mode 100644 index 0000000000000..c4c9e5746e6cb --- /dev/null +++ b/src/plugins/usage_collection/server/report/store_application_usage.test.ts @@ -0,0 +1,115 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import moment from 'moment'; +import { savedObjectsRepositoryMock } from '../../../../core/server/mocks'; +import { getDailyId } from '../../common/application_usage'; +import { storeApplicationUsage } from './store_application_usage'; +import { ApplicationUsageReport } from './schema'; + +const createReport = (parts: Partial): ApplicationUsageReport => ({ + appId: 'appId', + viewId: 'viewId', + numberOfClicks: 0, + minutesOnScreen: 0, + ...parts, +}); + +describe('storeApplicationUsage', () => { + let repository: ReturnType; + let timestamp: Date; + + beforeEach(() => { + repository = savedObjectsRepositoryMock.create(); + timestamp = new Date(); + }); + + it('does not call `repository.incrementUsageCounters` when the report list is empty', async () => { + await storeApplicationUsage(repository, [], timestamp); + expect(repository.incrementCounter).not.toHaveBeenCalled(); + }); + + it('calls `repository.incrementUsageCounters` with the correct parameters', async () => { + const report = createReport({ + appId: 'app1', + viewId: 'view1', + numberOfClicks: 2, + minutesOnScreen: 5, + }); + + await storeApplicationUsage(repository, [report], timestamp); + + expect(repository.incrementCounter).toHaveBeenCalledTimes(1); + + expect(repository.incrementCounter).toHaveBeenCalledWith( + ...expectedIncrementParams(report, timestamp) + ); + }); + + it('aggregates reports with the same appId/viewId tuple', async () => { + const report1 = createReport({ + appId: 'app1', + viewId: 'view1', + numberOfClicks: 2, + minutesOnScreen: 5, + }); + const report2 = createReport({ + appId: 'app1', + viewId: 'view2', + numberOfClicks: 1, + minutesOnScreen: 7, + }); + const report3 = createReport({ + appId: 'app1', + viewId: 'view1', + numberOfClicks: 3, + minutesOnScreen: 9, + }); + + await storeApplicationUsage(repository, [report1, report2, report3], timestamp); + + expect(repository.incrementCounter).toHaveBeenCalledTimes(2); + + expect(repository.incrementCounter).toHaveBeenCalledWith( + ...expectedIncrementParams( + { + appId: 'app1', + viewId: 'view1', + numberOfClicks: report1.numberOfClicks + report3.numberOfClicks, + minutesOnScreen: report1.minutesOnScreen + report3.minutesOnScreen, + }, + timestamp + ) + ); + expect(repository.incrementCounter).toHaveBeenCalledWith( + ...expectedIncrementParams(report2, timestamp) + ); + }); +}); + +const expectedIncrementParams = ( + { appId, viewId, minutesOnScreen, numberOfClicks }: ApplicationUsageReport, + timestamp: Date +) => { + const dayId = moment(timestamp).format('YYYY-MM-DD'); + return [ + 'application_usage_daily', + getDailyId({ appId, viewId, dayId }), + [ + { fieldName: 'numberOfClicks', incrementBy: numberOfClicks }, + { fieldName: 'minutesOnScreen', incrementBy: minutesOnScreen }, + ], + { + upsertAttributes: { + appId, + viewId, + timestamp: moment(`${moment(dayId).format('YYYY-MM-DD')}T00:00:00Z`).toISOString(), + }, + }, + ]; +}; diff --git a/src/plugins/usage_collection/server/report/store_application_usage.ts b/src/plugins/usage_collection/server/report/store_application_usage.ts new file mode 100644 index 0000000000000..2058b054fda8c --- /dev/null +++ b/src/plugins/usage_collection/server/report/store_application_usage.ts @@ -0,0 +1,87 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import moment from 'moment'; +import { Writable } from '@kbn/utility-types'; +import { ISavedObjectsRepository } from 'src/core/server'; +import { ApplicationUsageReport } from './schema'; +import { getDailyId } from '../../common/application_usage'; + +type WritableApplicationUsageReport = Writable; + +export const storeApplicationUsage = async ( + repository: ISavedObjectsRepository, + appUsages: ApplicationUsageReport[], + timestamp: Date +) => { + if (!appUsages.length) { + return; + } + + const dayId = getDayId(timestamp); + const aggregatedReports = aggregateAppUsages(appUsages); + + return Promise.allSettled( + aggregatedReports.map(async (report) => incrementUsageCounters(repository, report, dayId)) + ); +}; + +const aggregateAppUsages = (appUsages: ApplicationUsageReport[]) => { + return [ + ...appUsages + .reduce((map, appUsage) => { + const key = getKey(appUsage); + const aggregated: WritableApplicationUsageReport = map.get(key) ?? { + appId: appUsage.appId, + viewId: appUsage.viewId, + minutesOnScreen: 0, + numberOfClicks: 0, + }; + + aggregated.minutesOnScreen += appUsage.minutesOnScreen; + aggregated.numberOfClicks += appUsage.numberOfClicks; + + map.set(key, aggregated); + return map; + }, new Map()) + .values(), + ]; +}; + +const incrementUsageCounters = ( + repository: ISavedObjectsRepository, + { appId, viewId, numberOfClicks, minutesOnScreen }: WritableApplicationUsageReport, + dayId: string +) => { + const dailyId = getDailyId({ appId, viewId, dayId }); + + return repository.incrementCounter( + 'application_usage_daily', + dailyId, + [ + { fieldName: 'numberOfClicks', incrementBy: numberOfClicks }, + { fieldName: 'minutesOnScreen', incrementBy: minutesOnScreen }, + ], + { + upsertAttributes: { + appId, + viewId, + timestamp: getTimestamp(dayId), + }, + } + ); +}; + +const getKey = ({ viewId, appId }: ApplicationUsageReport) => `${appId}___${viewId}`; + +const getDayId = (timestamp: Date) => moment(timestamp).format('YYYY-MM-DD'); + +const getTimestamp = (dayId: string) => { + // Concatenating the day in YYYY-MM-DD form to T00:00:00Z to reduce the TZ effects + return moment(`${moment(dayId).format('YYYY-MM-DD')}T00:00:00Z`).toISOString(); +}; diff --git a/src/plugins/usage_collection/server/report/store_report.test.mocks.ts b/src/plugins/usage_collection/server/report/store_report.test.mocks.ts new file mode 100644 index 0000000000000..d151e7d7a5ddd --- /dev/null +++ b/src/plugins/usage_collection/server/report/store_report.test.mocks.ts @@ -0,0 +1,12 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +export const storeApplicationUsageMock = jest.fn(); +jest.doMock('./store_application_usage', () => ({ + storeApplicationUsage: storeApplicationUsageMock, +})); diff --git a/src/plugins/usage_collection/server/report/store_report.test.ts b/src/plugins/usage_collection/server/report/store_report.test.ts index 7174a54067246..dfcdd1f8e7e42 100644 --- a/src/plugins/usage_collection/server/report/store_report.test.ts +++ b/src/plugins/usage_collection/server/report/store_report.test.ts @@ -6,6 +6,8 @@ * Side Public License, v 1. */ +import { storeApplicationUsageMock } from './store_report.test.mocks'; + import { savedObjectsRepositoryMock } from '../../../../core/server/mocks'; import { storeReport } from './store_report'; import { ReportSchemaType } from './schema'; @@ -16,8 +18,17 @@ describe('store_report', () => { const momentTimestamp = moment(); const date = momentTimestamp.format('DDMMYYYY'); + let repository: ReturnType; + + beforeEach(() => { + repository = savedObjectsRepositoryMock.create(); + }); + + afterEach(() => { + storeApplicationUsageMock.mockReset(); + }); + test('stores report for all types of data', async () => { - const savedObjectClient = savedObjectsRepositoryMock.create(); const report: ReportSchemaType = { reportVersion: 3, userAgent: { @@ -53,9 +64,9 @@ describe('store_report', () => { }, }, }; - await storeReport(savedObjectClient, report); + await storeReport(repository, report); - expect(savedObjectClient.create).toHaveBeenCalledWith( + expect(repository.create).toHaveBeenCalledWith( 'ui-metric', { count: 1 }, { @@ -63,51 +74,45 @@ describe('store_report', () => { overwrite: true, } ); - expect(savedObjectClient.incrementCounter).toHaveBeenNthCalledWith( + expect(repository.incrementCounter).toHaveBeenNthCalledWith( 1, 'ui-metric', 'test-app-name:test-event-name', [{ fieldName: 'count', incrementBy: 3 }] ); - expect(savedObjectClient.incrementCounter).toHaveBeenNthCalledWith( + expect(repository.incrementCounter).toHaveBeenNthCalledWith( 2, 'ui-counter', `test-app-name:${date}:${METRIC_TYPE.LOADED}:test-event-name`, [{ fieldName: 'count', incrementBy: 1 }] ); - expect(savedObjectClient.incrementCounter).toHaveBeenNthCalledWith( + expect(repository.incrementCounter).toHaveBeenNthCalledWith( 3, 'ui-counter', `test-app-name:${date}:${METRIC_TYPE.CLICK}:test-event-name`, [{ fieldName: 'count', incrementBy: 2 }] ); - expect(savedObjectClient.bulkCreate).toHaveBeenNthCalledWith(1, [ - { - type: 'application_usage_transactional', - attributes: { - numberOfClicks: 3, - minutesOnScreen: 10, - appId: 'appId', - viewId: 'appId_view', - timestamp: expect.any(Date), - }, - }, - ]); + + expect(storeApplicationUsageMock).toHaveBeenCalledTimes(1); + expect(storeApplicationUsageMock).toHaveBeenCalledWith( + repository, + Object.values(report.application_usage as Record), + expect.any(Date) + ); }); test('it should not fail if nothing to store', async () => { - const savedObjectClient = savedObjectsRepositoryMock.create(); const report: ReportSchemaType = { reportVersion: 3, userAgent: void 0, uiCounter: void 0, application_usage: void 0, }; - await storeReport(savedObjectClient, report); + await storeReport(repository, report); - expect(savedObjectClient.bulkCreate).not.toHaveBeenCalled(); - expect(savedObjectClient.incrementCounter).not.toHaveBeenCalled(); - expect(savedObjectClient.create).not.toHaveBeenCalled(); - expect(savedObjectClient.create).not.toHaveBeenCalled(); + expect(repository.bulkCreate).not.toHaveBeenCalled(); + expect(repository.incrementCounter).not.toHaveBeenCalled(); + expect(repository.create).not.toHaveBeenCalled(); + expect(repository.create).not.toHaveBeenCalled(); }); }); diff --git a/src/plugins/usage_collection/server/report/store_report.ts b/src/plugins/usage_collection/server/report/store_report.ts index c3e04990d5793..0545a54792d45 100644 --- a/src/plugins/usage_collection/server/report/store_report.ts +++ b/src/plugins/usage_collection/server/report/store_report.ts @@ -10,6 +10,7 @@ import { ISavedObjectsRepository } from 'src/core/server'; import moment from 'moment'; import { chain, sumBy } from 'lodash'; import { ReportSchemaType } from './schema'; +import { storeApplicationUsage } from './store_application_usage'; export async function storeReport( internalRepository: ISavedObjectsRepository, @@ -17,11 +18,11 @@ export async function storeReport( ) { const uiCounters = report.uiCounter ? Object.entries(report.uiCounter) : []; const userAgents = report.userAgent ? Object.entries(report.userAgent) : []; - const appUsage = report.application_usage ? Object.values(report.application_usage) : []; + const appUsages = report.application_usage ? Object.values(report.application_usage) : []; const momentTimestamp = moment(); - const timestamp = momentTimestamp.toDate(); const date = momentTimestamp.format('DDMMYYYY'); + const timestamp = momentTimestamp.toDate(); return Promise.allSettled([ // User Agent @@ -64,21 +65,6 @@ export async function storeReport( ]; }), // Application Usage - ...[ - (async () => { - if (!appUsage.length) return []; - const { saved_objects: savedObjects } = await internalRepository.bulkCreate( - appUsage.map((metric) => ({ - type: 'application_usage_transactional', - attributes: { - ...metric, - timestamp, - }, - })) - ); - - return savedObjects; - })(), - ], + storeApplicationUsage(internalRepository, appUsages, timestamp), ]); } diff --git a/src/plugins/vis_type_vega/public/components/vega_help_menu.tsx b/src/plugins/vis_type_vega/public/components/vega_help_menu.tsx index efb41c470024b..f5b0f614458fd 100644 --- a/src/plugins/vis_type_vega/public/components/vega_help_menu.tsx +++ b/src/plugins/vis_type_vega/public/components/vega_help_menu.tsx @@ -11,6 +11,8 @@ import { EuiButtonIcon, EuiContextMenuPanel, EuiContextMenuItem, EuiPopover } fr import { FormattedMessage } from '@kbn/i18n/react'; import { i18n } from '@kbn/i18n'; +import { getDocLinks } from '../services'; + function VegaHelpMenu() { const [isPopoverOpen, setIsPopoverOpen] = useState(false); const onButtonClick = useCallback(() => setIsPopoverOpen(!isPopoverOpen), [isPopoverOpen]); @@ -30,7 +32,7 @@ function VegaHelpMenu() { const items = [ diff --git a/src/plugins/vis_type_vega/public/plugin.ts b/src/plugins/vis_type_vega/public/plugin.ts index 0204c2c90b71b..f935362d21604 100644 --- a/src/plugins/vis_type_vega/public/plugin.ts +++ b/src/plugins/vis_type_vega/public/plugin.ts @@ -19,6 +19,7 @@ import { setUISettings, setInjectedMetadata, setMapServiceSettings, + setDocLinks, } from './services'; import { createVegaFn } from './vega_fn'; @@ -96,5 +97,6 @@ export class VegaPlugin implements Plugin { setNotifications(core.notifications); setData(data); setInjectedMetadata(core.injectedMetadata); + setDocLinks(core.docLinks); } } diff --git a/src/plugins/vis_type_vega/public/services.ts b/src/plugins/vis_type_vega/public/services.ts index c47378282932b..f67fe4794e783 100644 --- a/src/plugins/vis_type_vega/public/services.ts +++ b/src/plugins/vis_type_vega/public/services.ts @@ -6,7 +6,7 @@ * Side Public License, v 1. */ -import { CoreStart, NotificationsStart, IUiSettingsClient } from 'src/core/public'; +import { CoreStart, NotificationsStart, IUiSettingsClient, DocLinksStart } from 'src/core/public'; import { DataPublicPluginStart } from '../../data/public'; import { createGetterSetter } from '../../kibana_utils/public'; @@ -35,3 +35,5 @@ export const [getInjectedVars, setInjectedVars] = createGetterSetter<{ }>('InjectedVars'); export const getEnableExternalUrls = () => getInjectedVars().enableExternalUrls; + +export const [getDocLinks, setDocLinks] = createGetterSetter('docLinks'); diff --git a/test/api_integration/apis/telemetry/telemetry_local.ts b/test/api_integration/apis/telemetry/telemetry_local.ts index a7b4da566b143..d0a09ee58d335 100644 --- a/test/api_integration/apis/telemetry/telemetry_local.ts +++ b/test/api_integration/apis/telemetry/telemetry_local.ts @@ -156,7 +156,7 @@ export default function ({ getService }: FtrProviderContext) { describe('application usage limits', () => { function createSavedObject(viewId?: string) { return supertest - .post('/api/saved_objects/application_usage_transactional') + .post('/api/saved_objects/application_usage_daily') .send({ attributes: { appId: 'test-app', @@ -184,7 +184,7 @@ export default function ({ getService }: FtrProviderContext) { await Promise.all( savedObjectIds.map((savedObjectId) => { return supertest - .delete(`/api/saved_objects/application_usage_transactional/${savedObjectId}`) + .delete(`/api/saved_objects/application_usage_daily/${savedObjectId}`) .expect(200); }) ); @@ -230,7 +230,7 @@ export default function ({ getService }: FtrProviderContext) { .post('/api/saved_objects/_bulk_create') .send( new Array(10001).fill(0).map(() => ({ - type: 'application_usage_transactional', + type: 'application_usage_daily', attributes: { appId: 'test-app', minutesOnScreen: 1, @@ -248,13 +248,12 @@ export default function ({ getService }: FtrProviderContext) { // The SavedObjects API does not allow bulk deleting, and deleting one by one takes ages and the tests timeout await es.deleteByQuery({ index: '.kibana', - body: { query: { term: { type: 'application_usage_transactional' } } }, + body: { query: { term: { type: 'application_usage_daily' } } }, conflicts: 'proceed', }); }); - // flaky https://github.com/elastic/kibana/issues/94513 - it.skip("should only use the first 10k docs for the application_usage data (they'll be rolled up in a later process)", async () => { + it("should only use the first 10k docs for the application_usage data (they'll be rolled up in a later process)", async () => { const stats = await retrieveTelemetry(supertest); expect(stats.stack_stats.kibana.plugins.application_usage).to.eql({ 'test-app': { diff --git a/test/functional/page_objects/common_page.ts b/test/functional/page_objects/common_page.ts index c6412f55dffbf..6d9641a1a920e 100644 --- a/test/functional/page_objects/common_page.ts +++ b/test/functional/page_objects/common_page.ts @@ -463,6 +463,21 @@ export function CommonPageProvider({ getService, getPageObjects }: FtrProviderCo async getWelcomeText() { return await testSubjects.getVisibleText('global-banner-item'); } + + /** + * Clicks on an element, and validates that the desired effect has taken place + * by confirming the existence of a validator + */ + async clickAndValidate( + clickTarget: string, + validator: string, + isValidatorCssString: boolean = false, + topOffset?: number + ) { + await testSubjects.click(clickTarget, undefined, topOffset); + const validate = isValidatorCssString ? find.byCssSelector : testSubjects.exists; + await validate(validator); + } } return new CommonPage(); diff --git a/test/functional/services/common/find.ts b/test/functional/services/common/find.ts index 2a86efad1ea9d..0cd4c14683f6e 100644 --- a/test/functional/services/common/find.ts +++ b/test/functional/services/common/find.ts @@ -79,11 +79,11 @@ export async function FindProvider({ getService }: FtrProviderContext) { return wrap(await driver.switchTo().activeElement()); } - public async setValue(selector: string, text: string): Promise { + public async setValue(selector: string, text: string, topOffset?: number): Promise { log.debug(`Find.setValue('${selector}', '${text}')`); return await retry.try(async () => { const element = await this.byCssSelector(selector); - await element.click(); + await element.click(topOffset); // in case the input element is actually a child of the testSubject, we // call clearValue() and type() on the element that is focused after @@ -413,14 +413,15 @@ export async function FindProvider({ getService }: FtrProviderContext) { public async clickByCssSelector( selector: string, - timeout: number = defaultFindTimeout + timeout: number = defaultFindTimeout, + topOffset?: number ): Promise { log.debug(`Find.clickByCssSelector('${selector}') with timeout=${timeout}`); await retry.try(async () => { const element = await this.byCssSelector(selector, timeout); if (element) { // await element.moveMouseTo(); - await element.click(); + await element.click(topOffset); } else { throw new Error(`Element with css='${selector}' is not found`); } diff --git a/test/functional/services/common/test_subjects.ts b/test/functional/services/common/test_subjects.ts index 28b37d9576e8c..111206ec9eafe 100644 --- a/test/functional/services/common/test_subjects.ts +++ b/test/functional/services/common/test_subjects.ts @@ -100,9 +100,13 @@ export function TestSubjectsProvider({ getService }: FtrProviderContext) { await find.clickByCssSelectorWhenNotDisabled(testSubjSelector(selector), { timeout }); } - public async click(selector: string, timeout: number = FIND_TIME): Promise { + public async click( + selector: string, + timeout: number = FIND_TIME, + topOffset?: number + ): Promise { log.debug(`TestSubjects.click(${selector})`); - await find.clickByCssSelector(testSubjSelector(selector), timeout); + await find.clickByCssSelector(testSubjSelector(selector), timeout, topOffset); } public async doubleClick(selector: string, timeout: number = FIND_TIME): Promise { @@ -187,12 +191,13 @@ export function TestSubjectsProvider({ getService }: FtrProviderContext) { public async setValue( selector: string, text: string, - options: SetValueOptions = {} + options: SetValueOptions = {}, + topOffset?: number ): Promise { return await retry.try(async () => { const { clearWithKeyboard = false, typeCharByChar = false } = options; log.debug(`TestSubjects.setValue(${selector}, ${text})`); - await this.click(selector); + await this.click(selector, undefined, topOffset); // in case the input element is actually a child of the testSubject, we // call clearValue() and type() on the element that is focused after // clicking on the testSubject diff --git a/test/functional/services/lib/web_element_wrapper/web_element_wrapper.ts b/test/functional/services/lib/web_element_wrapper/web_element_wrapper.ts index 1a45aee877e1f..b1561b29342da 100644 --- a/test/functional/services/lib/web_element_wrapper/web_element_wrapper.ts +++ b/test/functional/services/lib/web_element_wrapper/web_element_wrapper.ts @@ -182,9 +182,9 @@ export class WebElementWrapper { * * @return {Promise} */ - public async click() { + public async click(topOffset?: number) { await this.retryCall(async function click(wrapper) { - await wrapper.scrollIntoViewIfNecessary(); + await wrapper.scrollIntoViewIfNecessary(topOffset); await wrapper._webElement.click(); }); } @@ -693,11 +693,11 @@ export class WebElementWrapper { * @nonstandard * @return {Promise} */ - public async scrollIntoViewIfNecessary(): Promise { + public async scrollIntoViewIfNecessary(topOffset?: number): Promise { await this.driver.executeScript( scrollIntoViewIfNecessary, this._webElement, - this.fixedHeaderHeight + topOffset || this.fixedHeaderHeight ); } diff --git a/x-pack/plugins/apm/public/components/app/service_overview/service_overview.test.tsx b/x-pack/plugins/apm/public/components/app/service_overview/service_overview.test.tsx index f6ffec46f9f51..b30faac7a65af 100644 --- a/x-pack/plugins/apm/public/components/app/service_overview/service_overview.test.tsx +++ b/x-pack/plugins/apm/public/components/app/service_overview/service_overview.test.tsx @@ -91,7 +91,7 @@ describe('ServiceOverview', () => { isAggregationAccurate: true, }, 'GET /api/apm/services/{serviceName}/dependencies': [], - 'GET /api/apm/services/{serviceName}/service_overview_instances': [], + 'GET /api/apm/services/{serviceName}/service_overview_instances/primary_statistics': [], }; /* eslint-enable @typescript-eslint/naming-convention */ diff --git a/x-pack/plugins/apm/public/components/app/service_overview/service_overview_instances_chart_and_table.tsx b/x-pack/plugins/apm/public/components/app/service_overview/service_overview_instances_chart_and_table.tsx index 435def8bb9a91..c10ec1052f2a2 100644 --- a/x-pack/plugins/apm/public/components/app/service_overview/service_overview_instances_chart_and_table.tsx +++ b/x-pack/plugins/apm/public/components/app/service_overview/service_overview_instances_chart_and_table.tsx @@ -6,11 +6,17 @@ */ import { EuiFlexItem, EuiPanel } from '@elastic/eui'; -import React from 'react'; +import { orderBy } from 'lodash'; +import React, { useState } from 'react'; +import uuid from 'uuid'; import { useApmServiceContext } from '../../../context/apm_service/use_apm_service_context'; import { useUrlParams } from '../../../context/url_params_context/use_url_params'; -import { useFetcher } from '../../../hooks/use_fetcher'; -import { ServiceOverviewInstancesTable } from './service_overview_instances_table'; +import { FETCH_STATUS, useFetcher } from '../../../hooks/use_fetcher'; +import { getTimeRangeComparison } from '../../shared/time_comparison/get_time_range_comparison'; +import { + ServiceOverviewInstancesTable, + TableOptions, +} from './service_overview_instances_table'; // We're hiding this chart until these issues are resolved in the 7.13 timeframe: // @@ -24,17 +30,70 @@ interface ServiceOverviewInstancesChartAndTableProps { serviceName: string; } +const INITIAL_STATE = { + items: [] as Array<{ + serviceNodeName: string; + errorRate: number; + throughput: number; + latency: number; + cpuUsage: number; + memoryUsage: number; + }>, + requestId: undefined, + totalItems: 0, +}; + +const INITIAL_STATE_COMPARISON_STATISTICS = { + currentPeriod: {}, + previousPeriod: {}, +}; + +export type SortField = + | 'serviceNodeName' + | 'latency' + | 'throughput' + | 'errorRate' + | 'cpuUsage' + | 'memoryUsage'; + +export type SortDirection = 'asc' | 'desc'; +export const PAGE_SIZE = 5; +const DEFAULT_SORT = { + direction: 'desc' as const, + field: 'throughput' as const, +}; + export function ServiceOverviewInstancesChartAndTable({ chartHeight, serviceName, }: ServiceOverviewInstancesChartAndTableProps) { const { transactionType } = useApmServiceContext(); + const [tableOptions, setTableOptions] = useState({ + pageIndex: 0, + sort: DEFAULT_SORT, + }); + + const { pageIndex, sort } = tableOptions; + const { direction, field } = sort; const { - urlParams: { environment, kuery, latencyAggregationType, start, end }, + urlParams: { + environment, + kuery, + latencyAggregationType, + start, + end, + comparisonType, + }, } = useUrlParams(); - const { data = [], status } = useFetcher( + const { comparisonStart, comparisonEnd } = getTimeRangeComparison({ + start, + end, + comparisonType, + }); + + const { data = INITIAL_STATE, status } = useFetcher( (callApmApi) => { if (!start || !end || !transactionType || !latencyAggregationType) { return; @@ -42,7 +101,7 @@ export function ServiceOverviewInstancesChartAndTable({ return callApmApi({ endpoint: - 'GET /api/apm/services/{serviceName}/service_overview_instances', + 'GET /api/apm/services/{serviceName}/service_overview_instances/primary_statistics', params: { path: { serviceName, @@ -54,11 +113,32 @@ export function ServiceOverviewInstancesChartAndTable({ start, end, transactionType, - numBuckets: 20, }, }, + }).then((response) => { + const tableItems = orderBy( + // need top-level sortable fields for the managed table + response.map((item) => ({ + ...item, + latency: item.latency ?? 0, + throughput: item.throughput ?? 0, + errorRate: item.errorRate ?? 0, + cpuUsage: item.cpuUsage ?? 0, + memoryUsage: item.memoryUsage ?? 0, + })), + field, + direction + ).slice(pageIndex * PAGE_SIZE, (pageIndex + 1) * PAGE_SIZE); + + return { + requestId: uuid(), + items: tableItems, + totalItems: response.length, + }; }); }, + // comparisonType is listed as dependency even thought it is not used. This is needed to trigger the comparison api when it is changed. + // eslint-disable-next-line react-hooks/exhaustive-deps [ environment, kuery, @@ -67,24 +147,93 @@ export function ServiceOverviewInstancesChartAndTable({ end, serviceName, transactionType, + pageIndex, + field, + direction, + comparisonType, ] ); + const { items, requestId, totalItems } = data; + + const { + data: comparisonStatistics = INITIAL_STATE_COMPARISON_STATISTICS, + status: comparisonStatisticsStatus, + } = useFetcher( + (callApmApi) => { + if ( + !start || + !end || + !transactionType || + !latencyAggregationType || + !totalItems + ) { + return; + } + + return callApmApi({ + endpoint: + 'GET /api/apm/services/{serviceName}/service_overview_instances/comparison_statistics', + params: { + path: { + serviceName, + }, + query: { + environment, + kuery, + latencyAggregationType, + start, + end, + numBuckets: 20, + transactionType, + serviceNodeIds: JSON.stringify( + items.map((item) => item.serviceNodeName) + ), + comparisonStart, + comparisonEnd, + }, + }, + }); + }, + // only fetches comparison statistics when requestId is invalidated by primary statistics api call + // eslint-disable-next-line react-hooks/exhaustive-deps + [requestId], + { preservePreviousData: false } + ); + return ( <> {/* */} { + setTableOptions({ + pageIndex: newTableOptions.page?.index ?? 0, + sort: newTableOptions.sort + ? { + field: newTableOptions.sort.field as SortField, + direction: newTableOptions.sort.direction, + } + : DEFAULT_SORT, + }); + }} /> diff --git a/x-pack/plugins/apm/public/components/app/service_overview/service_overview_instances_table/get_columns.tsx b/x-pack/plugins/apm/public/components/app/service_overview/service_overview_instances_table/get_columns.tsx new file mode 100644 index 0000000000000..f2a169eb31f98 --- /dev/null +++ b/x-pack/plugins/apm/public/components/app/service_overview/service_overview_instances_table/get_columns.tsx @@ -0,0 +1,225 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { EuiBasicTableColumn } from '@elastic/eui'; +import { i18n } from '@kbn/i18n'; +import React from 'react'; +import { ValuesType } from 'utility-types'; +import { LatencyAggregationType } from '../../../../../common/latency_aggregation_types'; +import { isJavaAgentName } from '../../../../../common/agent_name'; +import { UNIDENTIFIED_SERVICE_NODES_LABEL } from '../../../../../common/i18n'; +import { SERVICE_NODE_NAME_MISSING } from '../../../../../common/service_nodes'; +import { + asMillisecondDuration, + asPercent, + asTransactionRate, +} from '../../../../../common/utils/formatters'; +import { APIReturnType } from '../../../../services/rest/createCallApmApi'; +import { px, unit } from '../../../../style/variables'; +import { SparkPlot } from '../../../shared/charts/spark_plot'; +import { MetricOverviewLink } from '../../../shared/Links/apm/MetricOverviewLink'; +import { ServiceNodeMetricOverviewLink } from '../../../shared/Links/apm/ServiceNodeMetricOverviewLink'; +import { TruncateWithTooltip } from '../../../shared/truncate_with_tooltip'; +import { getLatencyColumnLabel } from '../get_latency_column_label'; + +type ServiceInstancePrimaryStatisticItem = ValuesType< + APIReturnType<'GET /api/apm/services/{serviceName}/service_overview_instances/primary_statistics'> +>; + +type ServiceInstanceComparisonStatistics = APIReturnType<'GET /api/apm/services/{serviceName}/service_overview_instances/comparison_statistics'>; + +export function getColumns({ + serviceName, + agentName, + latencyAggregationType, + serviceInstanceComparisonStatistics, + comparisonEnabled, +}: { + serviceName: string; + agentName?: string; + latencyAggregationType?: LatencyAggregationType; + serviceInstanceComparisonStatistics?: ServiceInstanceComparisonStatistics; + comparisonEnabled?: boolean; +}): Array> { + return [ + { + field: 'serviceNodeName', + name: i18n.translate( + 'xpack.apm.serviceOverview.instancesTableColumnNodeName', + { defaultMessage: 'Node name' } + ), + render: (_, item) => { + const { serviceNodeName } = item; + const isMissingServiceNodeName = + serviceNodeName === SERVICE_NODE_NAME_MISSING; + const text = isMissingServiceNodeName + ? UNIDENTIFIED_SERVICE_NODES_LABEL + : serviceNodeName; + + const link = isJavaAgentName(agentName) ? ( + + {text} + + ) : ( + ({ + ...query, + kuery: isMissingServiceNodeName + ? `NOT (service.node.name:*)` + : `service.node.name:"${item.serviceNodeName}"`, + })} + > + {text} + + ); + + return ; + }, + sortable: true, + }, + { + field: 'latencyValue', + name: getLatencyColumnLabel(latencyAggregationType), + width: px(unit * 10), + render: (_, { serviceNodeName, latency }) => { + const currentPeriodTimestamp = + serviceInstanceComparisonStatistics?.currentPeriod?.[serviceNodeName] + ?.latency; + const previousPeriodTimestamp = + serviceInstanceComparisonStatistics?.previousPeriod?.[serviceNodeName] + ?.latency; + return ( + + ); + }, + sortable: true, + }, + { + field: 'throughputValue', + name: i18n.translate( + 'xpack.apm.serviceOverview.instancesTableColumnThroughput', + { defaultMessage: 'Throughput' } + ), + width: px(unit * 10), + render: (_, { serviceNodeName, throughput }) => { + const currentPeriodTimestamp = + serviceInstanceComparisonStatistics?.currentPeriod?.[serviceNodeName] + ?.throughput; + const previousPeriodTimestamp = + serviceInstanceComparisonStatistics?.previousPeriod?.[serviceNodeName] + ?.throughput; + return ( + + ); + }, + sortable: true, + }, + { + field: 'errorRateValue', + name: i18n.translate( + 'xpack.apm.serviceOverview.instancesTableColumnErrorRate', + { defaultMessage: 'Error rate' } + ), + width: px(unit * 8), + render: (_, { serviceNodeName, errorRate }) => { + const currentPeriodTimestamp = + serviceInstanceComparisonStatistics?.currentPeriod?.[serviceNodeName] + ?.errorRate; + const previousPeriodTimestamp = + serviceInstanceComparisonStatistics?.previousPeriod?.[serviceNodeName] + ?.errorRate; + return ( + + ); + }, + sortable: true, + }, + { + field: 'cpuUsageValue', + name: i18n.translate( + 'xpack.apm.serviceOverview.instancesTableColumnCpuUsage', + { defaultMessage: 'CPU usage (avg.)' } + ), + width: px(unit * 8), + render: (_, { serviceNodeName, cpuUsage }) => { + const currentPeriodTimestamp = + serviceInstanceComparisonStatistics?.currentPeriod?.[serviceNodeName] + ?.cpuUsage; + const previousPeriodTimestamp = + serviceInstanceComparisonStatistics?.previousPeriod?.[serviceNodeName] + ?.cpuUsage; + return ( + + ); + }, + sortable: true, + }, + { + field: 'memoryUsageValue', + name: i18n.translate( + 'xpack.apm.serviceOverview.instancesTableColumnMemoryUsage', + { defaultMessage: 'Memory usage (avg.)' } + ), + width: px(unit * 9), + render: (_, { serviceNodeName, memoryUsage }) => { + const currentPeriodTimestamp = + serviceInstanceComparisonStatistics?.currentPeriod?.[serviceNodeName] + ?.memoryUsage; + const previousPeriodTimestamp = + serviceInstanceComparisonStatistics?.previousPeriod?.[serviceNodeName] + ?.memoryUsage; + return ( + + ); + }, + sortable: true, + }, + ]; +} diff --git a/x-pack/plugins/apm/public/components/app/service_overview/service_overview_instances_table/index.tsx b/x-pack/plugins/apm/public/components/app/service_overview/service_overview_instances_table/index.tsx index 83ad506e8659b..3ca5f657795a7 100644 --- a/x-pack/plugins/apm/public/components/app/service_overview/service_overview_instances_table/index.tsx +++ b/x-pack/plugins/apm/public/components/app/service_overview/service_overview_instances_table/index.tsx @@ -6,208 +6,86 @@ */ import { - EuiBasicTableColumn, + EuiBasicTable, EuiFlexGroup, EuiFlexItem, - EuiInMemoryTable, EuiTitle, } from '@elastic/eui'; import { i18n } from '@kbn/i18n'; import React from 'react'; import { ValuesType } from 'utility-types'; -import { isJavaAgentName } from '../../../../../common/agent_name'; -import { UNIDENTIFIED_SERVICE_NODES_LABEL } from '../../../../../common/i18n'; -import { SERVICE_NODE_NAME_MISSING } from '../../../../../common/service_nodes'; -import { - asMillisecondDuration, - asPercent, - asTransactionRate, -} from '../../../../../common/utils/formatters'; import { useApmServiceContext } from '../../../../context/apm_service/use_apm_service_context'; import { useUrlParams } from '../../../../context/url_params_context/use_url_params'; import { FETCH_STATUS } from '../../../../hooks/use_fetcher'; import { APIReturnType } from '../../../../services/rest/createCallApmApi'; -import { px, unit } from '../../../../style/variables'; -import { SparkPlot } from '../../../shared/charts/spark_plot'; -import { MetricOverviewLink } from '../../../shared/Links/apm/MetricOverviewLink'; -import { ServiceNodeMetricOverviewLink } from '../../../shared/Links/apm/ServiceNodeMetricOverviewLink'; import { TableFetchWrapper } from '../../../shared/table_fetch_wrapper'; -import { TruncateWithTooltip } from '../../../shared/truncate_with_tooltip'; -import { getLatencyColumnLabel } from '../get_latency_column_label'; +import { + PAGE_SIZE, + SortDirection, + SortField, +} from '../service_overview_instances_chart_and_table'; import { ServiceOverviewTableContainer } from '../service_overview_table_container'; +import { getColumns } from './get_columns'; type ServiceInstanceItem = ValuesType< - APIReturnType<'GET /api/apm/services/{serviceName}/service_overview_instances'> + APIReturnType<'GET /api/apm/services/{serviceName}/service_overview_instances/primary_statistics'> >; +type ServiceInstanceComparisonStatistics = APIReturnType<'GET /api/apm/services/{serviceName}/service_overview_instances/comparison_statistics'>; + +export interface TableOptions { + pageIndex: number; + sort: { + direction: SortDirection; + field: SortField; + }; +} + interface Props { items?: ServiceInstanceItem[]; serviceName: string; status: FETCH_STATUS; + totalItems: number; + tableOptions: TableOptions; + onChangeTableOptions: (newTableOptions: { + page?: { index: number }; + sort?: { field: string; direction: SortDirection }; + }) => void; + serviceInstanceComparisonStatistics?: ServiceInstanceComparisonStatistics; + isLoading: boolean; } - export function ServiceOverviewInstancesTable({ items = [], + totalItems, serviceName, status, + tableOptions, + onChangeTableOptions, + serviceInstanceComparisonStatistics, + isLoading, }: Props) { const { agentName } = useApmServiceContext(); const { - urlParams: { latencyAggregationType }, + urlParams: { latencyAggregationType, comparisonEnabled }, } = useUrlParams(); - const columns: Array> = [ - { - field: 'name', - name: i18n.translate( - 'xpack.apm.serviceOverview.instancesTableColumnNodeName', - { - defaultMessage: 'Node name', - } - ), - render: (_, item) => { - const { serviceNodeName } = item; - const isMissingServiceNodeName = - serviceNodeName === SERVICE_NODE_NAME_MISSING; - const text = isMissingServiceNodeName - ? UNIDENTIFIED_SERVICE_NODES_LABEL - : serviceNodeName; + const { pageIndex, sort } = tableOptions; + const { direction, field } = sort; - const link = isJavaAgentName(agentName) ? ( - - {text} - - ) : ( - ({ - ...query, - kuery: isMissingServiceNodeName - ? `NOT (service.node.name:*)` - : `service.node.name:"${item.serviceNodeName}"`, - })} - > - {text} - - ); - - return ; - }, - sortable: true, - }, - { - field: 'latencyValue', - name: getLatencyColumnLabel(latencyAggregationType), - width: px(unit * 10), - render: (_, { latency }) => { - return ( - - ); - }, - sortable: true, - }, - { - field: 'throughputValue', - name: i18n.translate( - 'xpack.apm.serviceOverview.instancesTableColumnThroughput', - { defaultMessage: 'Throughput' } - ), - width: px(unit * 10), - render: (_, { throughput }) => { - return ( - - ); - }, - sortable: true, - }, - { - field: 'errorRateValue', - name: i18n.translate( - 'xpack.apm.serviceOverview.instancesTableColumnErrorRate', - { - defaultMessage: 'Error rate', - } - ), - width: px(unit * 8), - render: (_, { errorRate }) => { - return ( - - ); - }, - sortable: true, - }, - { - field: 'cpuUsageValue', - name: i18n.translate( - 'xpack.apm.serviceOverview.instancesTableColumnCpuUsage', - { - defaultMessage: 'CPU usage (avg.)', - } - ), - width: px(unit * 8), - render: (_, { cpuUsage }) => { - return ( - - ); - }, - sortable: true, - }, - { - field: 'memoryUsageValue', - name: i18n.translate( - 'xpack.apm.serviceOverview.instancesTableColumnMemoryUsage', - { - defaultMessage: 'Memory usage (avg.)', - } - ), - width: px(unit * 9), - render: (_, { memoryUsage }) => { - return ( - - ); - }, - sortable: true, - }, - ]; + const columns = getColumns({ + agentName, + serviceName, + latencyAggregationType, + serviceInstanceComparisonStatistics, + comparisonEnabled, + }); - // need top-level sortable fields for the managed table - const tableItems = items.map((item) => ({ - ...item, - latencyValue: item.latency?.value ?? 0, - throughputValue: item.throughput?.value ?? 0, - errorRateValue: item.errorRate?.value ?? 0, - cpuUsageValue: item.cpuUsage?.value ?? 0, - memoryUsageValue: item.memoryUsage?.value ?? 0, - })); - - const isLoading = status === FETCH_STATUS.LOADING; + const pagination = { + pageIndex, + pageSize: PAGE_SIZE, + totalItemCount: totalItems, + hidePerPageOptions: true, + }; return ( @@ -223,24 +101,15 @@ export function ServiceOverviewInstancesTable({ - diff --git a/x-pack/plugins/apm/public/components/shared/charts/instances_latency_distribution_chart/index.tsx b/x-pack/plugins/apm/public/components/shared/charts/instances_latency_distribution_chart/index.tsx index aa353b40d464a..5f24c1ee2495b 100644 --- a/x-pack/plugins/apm/public/components/shared/charts/instances_latency_distribution_chart/index.tsx +++ b/x-pack/plugins/apm/public/components/shared/charts/instances_latency_distribution_chart/index.tsx @@ -29,7 +29,7 @@ import { getResponseTimeTickFormatter } from '../transaction_charts/helper'; interface InstancesLatencyDistributionChartProps { height: number; - items?: APIReturnType<'GET /api/apm/services/{serviceName}/service_overview_instances'>; + items?: APIReturnType<'GET /api/apm/services/{serviceName}/service_overview_instances/primary_statistics'>; status: FETCH_STATUS; } @@ -44,15 +44,11 @@ export function InstancesLatencyDistributionChart({ const chartTheme = { ...useChartTheme(), bubbleSeriesStyle: { - point: { - strokeWidth: 0, - fill: theme.eui.euiColorVis1, - radius: 4, - }, + point: { strokeWidth: 0, fill: theme.eui.euiColorVis1, radius: 4 }, }, }; - const maxLatency = Math.max(...items.map((item) => item.latency?.value ?? 0)); + const maxLatency = Math.max(...items.map((item) => item.latency ?? 0)); const latencyFormatter = getDurationFormatter(maxLatency); return ( @@ -79,9 +75,9 @@ export function InstancesLatencyDistributionChart({ 'xpack.apm.instancesLatencyDistributionChartLegend', { defaultMessage: 'Instances' } )} - xAccessor={(item) => item.throughput?.value} + xAccessor={(item) => item.throughput} xScaleType={ScaleType.Linear} - yAccessors={[(item) => item.latency?.value]} + yAccessors={[(item) => item.latency]} yScaleType={ScaleType.Linear} /> +> { + return withApmSpan( + 'get_service_instances_comparison_statistics', + async () => { + const [transactionStats, systemMetricStats = []] = await Promise.all([ + getServiceInstancesTransactionStatistics({ + ...params, + isComparisonSearch: true, + }), + getServiceInstancesSystemMetricStatistics({ + ...params, + isComparisonSearch: true, + }), + ]); + + const stats = joinByKey( + [...transactionStats, ...systemMetricStats], + 'serviceNodeName' + ); + + return stats; + } + ); +} + +export async function getServiceInstancesComparisonStatisticsPeriods({ + environment, + kuery, + latencyAggregationType, + setup, + serviceName, + transactionType, + searchAggregatedTransactions, + numBuckets, + serviceNodeIds, + comparisonStart, + comparisonEnd, +}: { + environment?: string; + kuery?: string; + latencyAggregationType: LatencyAggregationType; + setup: Setup & SetupTimeRange; + serviceName: string; + transactionType: string; + searchAggregatedTransactions: boolean; + numBuckets: number; + serviceNodeIds: string[]; + comparisonStart?: number; + comparisonEnd?: number; +}) { + return withApmSpan( + 'get_service_instances_comparison_statistics_periods', + async () => { + const { start, end } = setup; + + const commonParams = { + environment, + kuery, + latencyAggregationType, + setup, + serviceName, + transactionType, + searchAggregatedTransactions, + numBuckets, + serviceNodeIds, + }; + + const currentPeriodPromise = getServiceInstancesComparisonStatistics({ + ...commonParams, + start, + end, + }); + + const previousPeriodPromise = + comparisonStart && comparisonEnd + ? getServiceInstancesComparisonStatistics({ + ...commonParams, + start: comparisonStart, + end: comparisonEnd, + }) + : []; + const [currentPeriod, previousPeriod] = await Promise.all([ + currentPeriodPromise, + previousPeriodPromise, + ]); + + const firtCurrentPeriod = currentPeriod.length + ? currentPeriod[0] + : undefined; + + return { + currentPeriod: keyBy(currentPeriod, 'serviceNodeName'), + previousPeriod: keyBy( + previousPeriod.map((data) => { + return { + ...data, + cpuUsage: offsetPreviousPeriodCoordinates({ + currentPeriodTimeseries: firtCurrentPeriod?.cpuUsage, + previousPeriodTimeseries: data.cpuUsage, + }), + errorRate: offsetPreviousPeriodCoordinates({ + currentPeriodTimeseries: firtCurrentPeriod?.errorRate, + previousPeriodTimeseries: data.errorRate, + }), + latency: offsetPreviousPeriodCoordinates({ + currentPeriodTimeseries: firtCurrentPeriod?.latency, + previousPeriodTimeseries: data.latency, + }), + memoryUsage: offsetPreviousPeriodCoordinates({ + currentPeriodTimeseries: firtCurrentPeriod?.memoryUsage, + previousPeriodTimeseries: data.memoryUsage, + }), + throughput: offsetPreviousPeriodCoordinates({ + currentPeriodTimeseries: firtCurrentPeriod?.throughput, + previousPeriodTimeseries: data.throughput, + }), + }; + }), + 'serviceNodeName' + ), + }; + } + ); +} diff --git a/x-pack/plugins/apm/server/lib/services/get_service_instances/get_service_instance_system_metric_stats.ts b/x-pack/plugins/apm/server/lib/services/get_service_instances/get_service_instance_system_metric_stats.ts deleted file mode 100644 index 6a72f817b3f69..0000000000000 --- a/x-pack/plugins/apm/server/lib/services/get_service_instances/get_service_instance_system_metric_stats.ts +++ /dev/null @@ -1,161 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import { AggregationOptionsByType } from '../../../../../../../typings/elasticsearch'; -import { - environmentQuery, - rangeQuery, - kqlQuery, -} from '../../../../server/utils/queries'; -import { SERVICE_NODE_NAME_MISSING } from '../../../../common/service_nodes'; -import { - METRIC_CGROUP_MEMORY_USAGE_BYTES, - METRIC_PROCESS_CPU_PERCENT, - METRIC_SYSTEM_FREE_MEMORY, - METRIC_SYSTEM_TOTAL_MEMORY, - SERVICE_NAME, - SERVICE_NODE_NAME, -} from '../../../../common/elasticsearch_fieldnames'; -import { ProcessorEvent } from '../../../../common/processor_event'; -import { ServiceInstanceParams } from '.'; -import { getBucketSize } from '../../helpers/get_bucket_size'; -import { - percentCgroupMemoryUsedScript, - percentSystemMemoryUsedScript, -} from '../../metrics/by_agent/shared/memory'; -import { withApmSpan } from '../../../utils/with_apm_span'; - -export async function getServiceInstanceSystemMetricStats({ - environment, - kuery, - setup, - serviceName, - size, - numBuckets, -}: ServiceInstanceParams) { - return withApmSpan('get_service_instance_system_metric_stats', async () => { - const { apmEventClient, start, end } = setup; - - const { intervalString } = getBucketSize({ start, end, numBuckets }); - - const systemMemoryFilter = { - bool: { - filter: [ - { exists: { field: METRIC_SYSTEM_FREE_MEMORY } }, - { exists: { field: METRIC_SYSTEM_TOTAL_MEMORY } }, - ], - }, - }; - - const cgroupMemoryFilter = { - exists: { field: METRIC_CGROUP_MEMORY_USAGE_BYTES }, - }; - - const cpuUsageFilter = { exists: { field: METRIC_PROCESS_CPU_PERCENT } }; - - function withTimeseries(agg: T) { - return { - avg: { avg: agg }, - timeseries: { - date_histogram: { - field: '@timestamp', - fixed_interval: intervalString, - min_doc_count: 0, - extended_bounds: { - min: start, - max: end, - }, - }, - aggs: { - avg: { avg: agg }, - }, - }, - }; - } - - const subAggs = { - memory_usage_cgroup: { - filter: cgroupMemoryFilter, - aggs: withTimeseries({ script: percentCgroupMemoryUsedScript }), - }, - memory_usage_system: { - filter: systemMemoryFilter, - aggs: withTimeseries({ script: percentSystemMemoryUsedScript }), - }, - cpu_usage: { - filter: cpuUsageFilter, - aggs: withTimeseries({ field: METRIC_PROCESS_CPU_PERCENT }), - }, - }; - - const response = await apmEventClient.search({ - apm: { - events: [ProcessorEvent.metric], - }, - body: { - size: 0, - query: { - bool: { - filter: [ - { term: { [SERVICE_NAME]: serviceName } }, - ...rangeQuery(start, end), - ...environmentQuery(environment), - ...kqlQuery(kuery), - ], - should: [cgroupMemoryFilter, systemMemoryFilter, cpuUsageFilter], - minimum_should_match: 1, - }, - }, - aggs: { - [SERVICE_NODE_NAME]: { - terms: { - field: SERVICE_NODE_NAME, - missing: SERVICE_NODE_NAME_MISSING, - size, - }, - aggs: subAggs, - }, - }, - }, - }); - - return ( - response.aggregations?.[SERVICE_NODE_NAME].buckets.map( - (serviceNodeBucket) => { - const hasCGroupData = - serviceNodeBucket.memory_usage_cgroup.avg.value !== null; - - const memoryMetricsKey = hasCGroupData - ? 'memory_usage_cgroup' - : 'memory_usage_system'; - - return { - serviceNodeName: String(serviceNodeBucket.key), - cpuUsage: { - value: serviceNodeBucket.cpu_usage.avg.value, - timeseries: serviceNodeBucket.cpu_usage.timeseries.buckets.map( - (dateBucket) => ({ - x: dateBucket.key, - y: dateBucket.avg.value, - }) - ), - }, - memoryUsage: { - value: serviceNodeBucket[memoryMetricsKey].avg.value, - timeseries: serviceNodeBucket[ - memoryMetricsKey - ].timeseries.buckets.map((dateBucket) => ({ - x: dateBucket.key, - y: dateBucket.avg.value, - })), - }, - }; - } - ) ?? [] - ); - }); -} diff --git a/x-pack/plugins/apm/server/lib/services/get_service_instances/get_service_instance_transaction_stats.ts b/x-pack/plugins/apm/server/lib/services/get_service_instances/get_service_instance_transaction_stats.ts deleted file mode 100644 index 94a5e54e9ace5..0000000000000 --- a/x-pack/plugins/apm/server/lib/services/get_service_instances/get_service_instance_transaction_stats.ts +++ /dev/null @@ -1,166 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import { EventOutcome } from '../../../../common/event_outcome'; -import { - environmentQuery, - rangeQuery, - kqlQuery, -} from '../../../../server/utils/queries'; -import { SERVICE_NODE_NAME_MISSING } from '../../../../common/service_nodes'; -import { - EVENT_OUTCOME, - SERVICE_NAME, - SERVICE_NODE_NAME, - TRANSACTION_TYPE, -} from '../../../../common/elasticsearch_fieldnames'; -import { ServiceInstanceParams } from '.'; -import { getBucketSize } from '../../helpers/get_bucket_size'; -import { - getProcessorEventForAggregatedTransactions, - getTransactionDurationFieldForAggregatedTransactions, -} from '../../helpers/aggregated_transactions'; -import { calculateThroughput } from '../../helpers/calculate_throughput'; -import { withApmSpan } from '../../../utils/with_apm_span'; -import { - getLatencyAggregation, - getLatencyValue, -} from '../../helpers/latency_aggregation_type'; - -export async function getServiceInstanceTransactionStats({ - environment, - kuery, - latencyAggregationType, - setup, - transactionType, - serviceName, - size, - searchAggregatedTransactions, - numBuckets, -}: ServiceInstanceParams) { - return withApmSpan('get_service_instance_transaction_stats', async () => { - const { apmEventClient, start, end } = setup; - - const { intervalString, bucketSize } = getBucketSize({ - start, - end, - numBuckets, - }); - - const field = getTransactionDurationFieldForAggregatedTransactions( - searchAggregatedTransactions - ); - - const subAggs = { - ...getLatencyAggregation(latencyAggregationType, field), - failures: { - filter: { - term: { - [EVENT_OUTCOME]: EventOutcome.failure, - }, - }, - }, - }; - - const response = await apmEventClient.search({ - apm: { - events: [ - getProcessorEventForAggregatedTransactions( - searchAggregatedTransactions - ), - ], - }, - body: { - size: 0, - query: { - bool: { - filter: [ - { term: { [SERVICE_NAME]: serviceName } }, - { term: { [TRANSACTION_TYPE]: transactionType } }, - ...rangeQuery(start, end), - ...environmentQuery(environment), - ...kqlQuery(kuery), - ], - }, - }, - aggs: { - [SERVICE_NODE_NAME]: { - terms: { - field: SERVICE_NODE_NAME, - missing: SERVICE_NODE_NAME_MISSING, - size, - }, - aggs: { - ...subAggs, - timeseries: { - date_histogram: { - field: '@timestamp', - fixed_interval: intervalString, - min_doc_count: 0, - extended_bounds: { - min: start, - max: end, - }, - }, - aggs: { - ...subAggs, - }, - }, - }, - }, - }, - }, - }); - - const bucketSizeInMinutes = bucketSize / 60; - - return ( - response.aggregations?.[SERVICE_NODE_NAME].buckets.map( - (serviceNodeBucket) => { - const { - doc_count: count, - latency, - key, - failures, - timeseries, - } = serviceNodeBucket; - - return { - serviceNodeName: String(key), - errorRate: { - value: failures.doc_count / count, - timeseries: timeseries.buckets.map((dateBucket) => ({ - x: dateBucket.key, - y: dateBucket.failures.doc_count / dateBucket.doc_count, - })), - }, - throughput: { - value: calculateThroughput({ start, end, value: count }), - timeseries: timeseries.buckets.map((dateBucket) => ({ - x: dateBucket.key, - y: dateBucket.doc_count / bucketSizeInMinutes, - })), - }, - latency: { - value: getLatencyValue({ - aggregation: latency, - latencyAggregationType, - }), - timeseries: timeseries.buckets.map((dateBucket) => ({ - x: dateBucket.key, - y: getLatencyValue({ - aggregation: dateBucket.latency, - latencyAggregationType, - }), - })), - }, - }; - } - ) ?? [] - ); - }); -} diff --git a/x-pack/plugins/apm/server/lib/services/get_service_instances/get_service_instances_system_metric_statistics.ts b/x-pack/plugins/apm/server/lib/services/get_service_instances/get_service_instances_system_metric_statistics.ts new file mode 100644 index 0000000000000..1a33e9810dd5e --- /dev/null +++ b/x-pack/plugins/apm/server/lib/services/get_service_instances/get_service_instances_system_metric_statistics.ts @@ -0,0 +1,208 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { AggregationOptionsByType } from 'typings/elasticsearch'; +import { + METRIC_CGROUP_MEMORY_USAGE_BYTES, + METRIC_PROCESS_CPU_PERCENT, + METRIC_SYSTEM_FREE_MEMORY, + METRIC_SYSTEM_TOTAL_MEMORY, + SERVICE_NAME, + SERVICE_NODE_NAME, +} from '../../../../common/elasticsearch_fieldnames'; +import { ProcessorEvent } from '../../../../common/processor_event'; +import { SERVICE_NODE_NAME_MISSING } from '../../../../common/service_nodes'; +import { Coordinate } from '../../../../typings/timeseries'; +import { environmentQuery, kqlQuery, rangeQuery } from '../../../utils/queries'; +import { getBucketSize } from '../../helpers/get_bucket_size'; +import { Setup } from '../../helpers/setup_request'; +import { + percentCgroupMemoryUsedScript, + percentSystemMemoryUsedScript, +} from '../../metrics/by_agent/shared/memory'; +import { withApmSpan } from '../../../utils/with_apm_span'; + +interface ServiceInstanceSystemMetricPrimaryStatistics { + serviceNodeName: string; + cpuUsage: number | null; + memoryUsage: number | null; +} + +interface ServiceInstanceSystemMetricComparisonStatistics { + serviceNodeName: string; + cpuUsage: Coordinate[]; + memoryUsage: Coordinate[]; +} + +type ServiceInstanceSystemMetricStatistics = T extends true + ? ServiceInstanceSystemMetricComparisonStatistics + : ServiceInstanceSystemMetricPrimaryStatistics; + +export async function getServiceInstancesSystemMetricStatistics< + T extends true | false +>({ + environment, + kuery, + setup, + serviceName, + size, + start, + end, + serviceNodeIds, + numBuckets, + isComparisonSearch, +}: { + setup: Setup; + serviceName: string; + start: number; + end: number; + numBuckets?: number; + serviceNodeIds?: string[]; + environment?: string; + kuery?: string; + size?: number; + isComparisonSearch: T; +}): Promise>> { + return withApmSpan( + 'get_service_instances_system_metric_statistics', + async () => { + const { apmEventClient } = setup; + + const { intervalString } = getBucketSize({ start, end, numBuckets }); + + const systemMemoryFilter = { + bool: { + filter: [ + { exists: { field: METRIC_SYSTEM_FREE_MEMORY } }, + { exists: { field: METRIC_SYSTEM_TOTAL_MEMORY } }, + ], + }, + }; + + const cgroupMemoryFilter = { + exists: { field: METRIC_CGROUP_MEMORY_USAGE_BYTES }, + }; + + const cpuUsageFilter = { exists: { field: METRIC_PROCESS_CPU_PERCENT } }; + + function withTimeseries( + agg: TParams + ) { + return { + ...(isComparisonSearch + ? { + avg: { avg: agg }, + timeseries: { + date_histogram: { + field: '@timestamp', + fixed_interval: intervalString, + min_doc_count: 0, + extended_bounds: { + min: start, + max: end, + }, + }, + aggs: { avg: { avg: agg } }, + }, + } + : { avg: { avg: agg } }), + }; + } + + const subAggs = { + memory_usage_cgroup: { + filter: cgroupMemoryFilter, + aggs: withTimeseries({ script: percentCgroupMemoryUsedScript }), + }, + memory_usage_system: { + filter: systemMemoryFilter, + aggs: withTimeseries({ script: percentSystemMemoryUsedScript }), + }, + cpu_usage: { + filter: cpuUsageFilter, + aggs: withTimeseries({ field: METRIC_PROCESS_CPU_PERCENT }), + }, + }; + + const response = await apmEventClient.search({ + apm: { + events: [ProcessorEvent.metric], + }, + body: { + size: 0, + query: { + bool: { + filter: [ + { term: { [SERVICE_NAME]: serviceName } }, + ...rangeQuery(start, end), + ...environmentQuery(environment), + ...kqlQuery(kuery), + ...(isComparisonSearch && serviceNodeIds + ? [{ terms: { [SERVICE_NODE_NAME]: serviceNodeIds } }] + : []), + ], + should: [cgroupMemoryFilter, systemMemoryFilter, cpuUsageFilter], + minimum_should_match: 1, + }, + }, + aggs: { + [SERVICE_NODE_NAME]: { + terms: { + field: SERVICE_NODE_NAME, + missing: SERVICE_NODE_NAME_MISSING, + ...(size ? { size } : {}), + ...(isComparisonSearch ? { include: serviceNodeIds } : {}), + }, + aggs: subAggs, + }, + }, + }, + }); + + return ( + (response.aggregations?.[SERVICE_NODE_NAME].buckets.map( + (serviceNodeBucket) => { + const serviceNodeName = String(serviceNodeBucket.key); + const hasCGroupData = + serviceNodeBucket.memory_usage_cgroup.avg.value !== null; + + const memoryMetricsKey = hasCGroupData + ? 'memory_usage_cgroup' + : 'memory_usage_system'; + + const cpuUsage = + // Timeseries is available when isComparisonSearch is true + 'timeseries' in serviceNodeBucket.cpu_usage + ? serviceNodeBucket.cpu_usage.timeseries.buckets.map( + (dateBucket) => ({ + x: dateBucket.key, + y: dateBucket.avg.value, + }) + ) + : serviceNodeBucket.cpu_usage.avg.value; + + const memoryUsageValue = serviceNodeBucket[memoryMetricsKey]; + const memoryUsage = + // Timeseries is available when isComparisonSearch is true + 'timeseries' in memoryUsageValue + ? memoryUsageValue.timeseries.buckets.map((dateBucket) => ({ + x: dateBucket.key, + y: dateBucket.avg.value, + })) + : serviceNodeBucket[memoryMetricsKey].avg.value; + + return { + serviceNodeName, + cpuUsage, + memoryUsage, + }; + } + ) as Array>) || [] + ); + } + ); +} diff --git a/x-pack/plugins/apm/server/lib/services/get_service_instances/get_service_instances_transaction_statistics.ts b/x-pack/plugins/apm/server/lib/services/get_service_instances/get_service_instances_transaction_statistics.ts new file mode 100644 index 0000000000000..ad54a231b52ef --- /dev/null +++ b/x-pack/plugins/apm/server/lib/services/get_service_instances/get_service_instances_transaction_statistics.ts @@ -0,0 +1,202 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ +import { + EVENT_OUTCOME, + SERVICE_NAME, + SERVICE_NODE_NAME, + TRANSACTION_TYPE, +} from '../../../../common/elasticsearch_fieldnames'; +import { EventOutcome } from '../../../../common/event_outcome'; +import { LatencyAggregationType } from '../../../../common/latency_aggregation_types'; +import { SERVICE_NODE_NAME_MISSING } from '../../../../common/service_nodes'; +import { Coordinate } from '../../../../typings/timeseries'; +import { environmentQuery, kqlQuery, rangeQuery } from '../../../utils/queries'; +import { + getProcessorEventForAggregatedTransactions, + getTransactionDurationFieldForAggregatedTransactions, +} from '../../helpers/aggregated_transactions'; +import { calculateThroughput } from '../../helpers/calculate_throughput'; +import { getBucketSize } from '../../helpers/get_bucket_size'; +import { + getLatencyAggregation, + getLatencyValue, +} from '../../helpers/latency_aggregation_type'; +import { Setup } from '../../helpers/setup_request'; +import { withApmSpan } from '../../../utils/with_apm_span'; + +interface ServiceInstanceTransactionPrimaryStatistics { + serviceNodeName: string; + errorRate: number; + latency: number; + throughput: number; +} + +interface ServiceInstanceTransactionComparisonStatistics { + serviceNodeName: string; + errorRate: Coordinate[]; + latency: Coordinate[]; + throughput: Coordinate[]; +} + +type ServiceInstanceTransactionStatistics = T extends true + ? ServiceInstanceTransactionComparisonStatistics + : ServiceInstanceTransactionPrimaryStatistics; + +export async function getServiceInstancesTransactionStatistics< + T extends true | false +>({ + environment, + kuery, + latencyAggregationType, + setup, + transactionType, + serviceName, + size, + searchAggregatedTransactions, + start, + end, + serviceNodeIds, + numBuckets, + isComparisonSearch, +}: { + latencyAggregationType: LatencyAggregationType; + setup: Setup; + serviceName: string; + transactionType: string; + searchAggregatedTransactions: boolean; + start: number; + end: number; + isComparisonSearch: T; + serviceNodeIds?: string[]; + environment?: string; + kuery?: string; + size?: number; + numBuckets?: number; +}): Promise>> { + return withApmSpan( + 'get_service_instances_transaction_statistics', + async () => { + const { apmEventClient } = setup; + + const { intervalString, bucketSize } = getBucketSize({ + start, + end, + numBuckets, + }); + + const field = getTransactionDurationFieldForAggregatedTransactions( + searchAggregatedTransactions + ); + + const subAggs = { + ...getLatencyAggregation(latencyAggregationType, field), + failures: { + filter: { + term: { + [EVENT_OUTCOME]: EventOutcome.failure, + }, + }, + }, + }; + + const query = { + bool: { + filter: [ + { term: { [SERVICE_NAME]: serviceName } }, + { term: { [TRANSACTION_TYPE]: transactionType } }, + ...rangeQuery(start, end), + ...environmentQuery(environment), + ...kqlQuery(kuery), + ...(isComparisonSearch && serviceNodeIds + ? [{ terms: { [SERVICE_NODE_NAME]: serviceNodeIds } }] + : []), + ], + }, + }; + + const aggs = { + [SERVICE_NODE_NAME]: { + terms: { + field: SERVICE_NODE_NAME, + missing: SERVICE_NODE_NAME_MISSING, + ...(size ? { size } : {}), + ...(isComparisonSearch ? { include: serviceNodeIds } : {}), + }, + aggs: isComparisonSearch + ? { + timeseries: { + date_histogram: { + field: '@timestamp', + fixed_interval: intervalString, + min_doc_count: 0, + extended_bounds: { min: start, max: end }, + }, + aggs: subAggs, + }, + } + : subAggs, + }, + }; + + const response = await apmEventClient.search({ + apm: { + events: [ + getProcessorEventForAggregatedTransactions( + searchAggregatedTransactions + ), + ], + }, + body: { size: 0, query, aggs }, + }); + + const bucketSizeInMinutes = bucketSize / 60; + + return ( + (response.aggregations?.[SERVICE_NODE_NAME].buckets.map( + (serviceNodeBucket) => { + const { doc_count: count, key } = serviceNodeBucket; + const serviceNodeName = String(key); + + // Timeseries is returned when isComparisonSearch is true + if ('timeseries' in serviceNodeBucket) { + const { timeseries } = serviceNodeBucket; + return { + serviceNodeName, + errorRate: timeseries.buckets.map((dateBucket) => ({ + x: dateBucket.key, + y: dateBucket.failures.doc_count / dateBucket.doc_count, + })), + throughput: timeseries.buckets.map((dateBucket) => ({ + x: dateBucket.key, + y: dateBucket.doc_count / bucketSizeInMinutes, + })), + latency: timeseries.buckets.map((dateBucket) => ({ + x: dateBucket.key, + y: getLatencyValue({ + aggregation: dateBucket.latency, + latencyAggregationType, + }), + })), + }; + } else { + const { failures, latency } = serviceNodeBucket; + return { + serviceNodeName, + errorRate: failures.doc_count / count, + latency: getLatencyValue({ + aggregation: latency, + latencyAggregationType, + }), + throughput: calculateThroughput({ start, end, value: count }), + }; + } + } + ) as Array>) || [] + ); + } + ); +} diff --git a/x-pack/plugins/apm/server/lib/services/get_service_instances/index.ts b/x-pack/plugins/apm/server/lib/services/get_service_instances/primary_statistics.ts similarity index 52% rename from x-pack/plugins/apm/server/lib/services/get_service_instances/index.ts rename to x-pack/plugins/apm/server/lib/services/get_service_instances/primary_statistics.ts index 838753890a8cd..3cd98558eff02 100644 --- a/x-pack/plugins/apm/server/lib/services/get_service_instances/index.ts +++ b/x-pack/plugins/apm/server/lib/services/get_service_instances/primary_statistics.ts @@ -9,10 +9,10 @@ import { LatencyAggregationType } from '../../../../common/latency_aggregation_t import { joinByKey } from '../../../../common/utils/join_by_key'; import { withApmSpan } from '../../../utils/with_apm_span'; import { Setup, SetupTimeRange } from '../../helpers/setup_request'; -import { getServiceInstanceSystemMetricStats } from './get_service_instance_system_metric_stats'; -import { getServiceInstanceTransactionStats } from './get_service_instance_transaction_stats'; +import { getServiceInstancesSystemMetricStatistics } from './get_service_instances_system_metric_statistics'; +import { getServiceInstancesTransactionStatistics } from './get_service_instances_transaction_statistics'; -export interface ServiceInstanceParams { +interface ServiceInstancePrimaryStatisticsParams { environment?: string; kuery?: string; latencyAggregationType: LatencyAggregationType; @@ -21,21 +21,37 @@ export interface ServiceInstanceParams { transactionType: string; searchAggregatedTransactions: boolean; size: number; - numBuckets: number; + start: number; + end: number; } -export async function getServiceInstances( - params: Omit -) { - return withApmSpan('get_service_instances', async () => { +export async function getServiceInstancesPrimaryStatistics( + params: Omit +): Promise< + Array<{ + serviceNodeName: string; + errorRate?: number; + latency?: number; + throughput?: number; + cpuUsage?: number | null; + memoryUsage?: number | null; + }> +> { + return withApmSpan('get_service_instances_primary_statistics', async () => { const paramsForSubQueries = { ...params, size: 50, }; const [transactionStats, systemMetricStats] = await Promise.all([ - getServiceInstanceTransactionStats(paramsForSubQueries), - getServiceInstanceSystemMetricStats(paramsForSubQueries), + getServiceInstancesTransactionStatistics({ + ...paramsForSubQueries, + isComparisonSearch: false, + }), + getServiceInstancesSystemMetricStatistics({ + ...paramsForSubQueries, + isComparisonSearch: false, + }), ]); const stats = joinByKey( diff --git a/x-pack/plugins/apm/server/routes/create_apm_api.ts b/x-pack/plugins/apm/server/routes/create_apm_api.ts index 2bd7e25e848c8..2b5fb0b516ab5 100644 --- a/x-pack/plugins/apm/server/routes/create_apm_api.ts +++ b/x-pack/plugins/apm/server/routes/create_apm_api.ts @@ -30,7 +30,8 @@ import { serviceDependenciesRoute, serviceMetadataDetailsRoute, serviceMetadataIconsRoute, - serviceInstancesRoute, + serviceInstancesPrimaryStatisticsRoute, + serviceInstancesComparisonStatisticsRoute, serviceProfilingStatisticsRoute, serviceProfilingTimelineRoute, } from './services'; @@ -134,7 +135,8 @@ const createApmApi = () => { .add(serviceDependenciesRoute) .add(serviceMetadataDetailsRoute) .add(serviceMetadataIconsRoute) - .add(serviceInstancesRoute) + .add(serviceInstancesPrimaryStatisticsRoute) + .add(serviceInstancesComparisonStatisticsRoute) .add(serviceErrorGroupsComparisonStatisticsRoute) .add(serviceProfilingTimelineRoute) .add(serviceProfilingStatisticsRoute) diff --git a/x-pack/plugins/apm/server/routes/services.ts b/x-pack/plugins/apm/server/routes/services.ts index bac970416792b..7ba19035a90b0 100644 --- a/x-pack/plugins/apm/server/routes/services.ts +++ b/x-pack/plugins/apm/server/routes/services.ts @@ -8,7 +8,13 @@ import Boom from '@hapi/boom'; import * as t from 'io-ts'; import { uniq } from 'lodash'; +import { + LatencyAggregationType, + latencyAggregationTypeRt, +} from '../../common/latency_aggregation_types'; +import { ProfilingValueType } from '../../common/profiling'; import { isoToEpochRt } from '../../common/runtime_types/iso_to_epoch_rt'; +import { jsonRt } from '../../common/runtime_types/json_rt'; import { toNumberRt } from '../../common/runtime_types/to_number_rt'; import { getSearchAggregatedTransactions } from '../lib/helpers/aggregated_transactions'; import { setupRequest } from '../lib/helpers/setup_request'; @@ -16,31 +22,26 @@ import { getServiceAnnotations } from '../lib/services/annotations'; import { getServices } from '../lib/services/get_services'; import { getServiceAgentName } from '../lib/services/get_service_agent_name'; import { getServiceDependencies } from '../lib/services/get_service_dependencies'; -import { getServiceErrorGroupPrimaryStatistics } from '../lib/services/get_service_error_groups/get_service_error_group_primary_statistics'; import { getServiceErrorGroupPeriods } from '../lib/services/get_service_error_groups/get_service_error_group_comparison_statistics'; -import { getServiceInstances } from '../lib/services/get_service_instances'; +import { getServiceErrorGroupPrimaryStatistics } from '../lib/services/get_service_error_groups/get_service_error_group_primary_statistics'; +import { getServiceInstancesComparisonStatisticsPeriods } from '../lib/services/get_service_instances/comparison_statistics'; +import { getServiceInstancesPrimaryStatistics } from '../lib/services/get_service_instances/primary_statistics'; import { getServiceMetadataDetails } from '../lib/services/get_service_metadata_details'; import { getServiceMetadataIcons } from '../lib/services/get_service_metadata_icons'; import { getServiceNodeMetadata } from '../lib/services/get_service_node_metadata'; import { getServiceTransactionTypes } from '../lib/services/get_service_transaction_types'; import { getThroughput } from '../lib/services/get_throughput'; -import { createRoute } from './create_route'; +import { getServiceProfilingStatistics } from '../lib/services/profiling/get_service_profiling_statistics'; +import { getServiceProfilingTimeline } from '../lib/services/profiling/get_service_profiling_timeline'; import { offsetPreviousPeriodCoordinates } from '../utils/offset_previous_period_coordinate'; -import { jsonRt } from '../../common/runtime_types/json_rt'; +import { withApmSpan } from '../utils/with_apm_span'; +import { createRoute } from './create_route'; import { comparisonRangeRt, environmentRt, kueryRt, rangeRt, } from './default_api_types'; -import { withApmSpan } from '../utils/with_apm_span'; -import { getServiceProfilingStatistics } from '../lib/services/profiling/get_service_profiling_statistics'; -import { getServiceProfilingTimeline } from '../lib/services/profiling/get_service_profiling_timeline'; -import { ProfilingValueType } from '../../common/profiling'; -import { - latencyAggregationTypeRt, - LatencyAggregationType, -} from '../../common/latency_aggregation_types'; export const servicesRoute = createRoute({ endpoint: 'GET /api/apm/services', @@ -433,8 +434,9 @@ export const serviceThroughputRoute = createRoute({ }, }); -export const serviceInstancesRoute = createRoute({ - endpoint: 'GET /api/apm/services/{serviceName}/service_overview_instances', +export const serviceInstancesPrimaryStatisticsRoute = createRoute({ + endpoint: + 'GET /api/apm/services/{serviceName}/service_overview_instances/primary_statistics', params: t.type({ path: t.type({ serviceName: t.string, @@ -443,11 +445,58 @@ export const serviceInstancesRoute = createRoute({ t.type({ latencyAggregationType: latencyAggregationTypeRt, transactionType: t.string, + }), + environmentRt, + kueryRt, + rangeRt, + ]), + }), + options: { tags: ['access:apm'] }, + handler: async ({ context, request }) => { + const setup = await setupRequest(context, request); + const { serviceName } = context.params.path; + const { environment, kuery, transactionType } = context.params.query; + const latencyAggregationType = (context.params.query + .latencyAggregationType as unknown) as LatencyAggregationType; + + const searchAggregatedTransactions = await getSearchAggregatedTransactions( + setup + ); + + const { start, end } = setup; + + return getServiceInstancesPrimaryStatistics({ + environment, + kuery, + latencyAggregationType, + serviceName, + setup, + transactionType, + searchAggregatedTransactions, + start, + end, + }); + }, +}); + +export const serviceInstancesComparisonStatisticsRoute = createRoute({ + endpoint: + 'GET /api/apm/services/{serviceName}/service_overview_instances/comparison_statistics', + params: t.type({ + path: t.type({ + serviceName: t.string, + }), + query: t.intersection([ + t.type({ + latencyAggregationType: latencyAggregationTypeRt, + transactionType: t.string, + serviceNodeIds: jsonRt.pipe(t.array(t.string)), numBuckets: toNumberRt, }), environmentRt, kueryRt, rangeRt, + comparisonRangeRt, ]), }), options: { tags: ['access:apm'] }, @@ -458,6 +507,9 @@ export const serviceInstancesRoute = createRoute({ environment, kuery, transactionType, + comparisonStart, + comparisonEnd, + serviceNodeIds, numBuckets, } = context.params.query; const latencyAggregationType = (context.params.query @@ -467,7 +519,7 @@ export const serviceInstancesRoute = createRoute({ setup ); - return getServiceInstances({ + return getServiceInstancesComparisonStatisticsPeriods({ environment, kuery, latencyAggregationType, @@ -476,6 +528,9 @@ export const serviceInstancesRoute = createRoute({ transactionType, searchAggregatedTransactions, numBuckets, + serviceNodeIds, + comparisonStart, + comparisonEnd, }); }, }); diff --git a/x-pack/plugins/cases/server/routes/api/cases/find_cases.ts b/x-pack/plugins/cases/server/routes/api/cases/find_cases.ts index bc6907f52b9eb..10406d0edcd46 100644 --- a/x-pack/plugins/cases/server/routes/api/cases/find_cases.ts +++ b/x-pack/plugins/cases/server/routes/api/cases/find_cases.ts @@ -66,11 +66,13 @@ export function initFindCasesApi({ caseService, router, logger }: RouteDeps) { return response.ok({ body: CasesFindResponseRt.encode( transformCases({ - ...cases, + casesMap: cases.casesMap, + page: cases.page, + perPage: cases.perPage, + total: cases.total, countOpenCases: openCases, countInProgressCases: inProgressCases, countClosedCases: closedCases, - total: cases.casesMap.size, }) ), }); diff --git a/x-pack/plugins/cases/server/routes/api/cases/helpers.ts b/x-pack/plugins/cases/server/routes/api/cases/helpers.ts index 8659ab02d6d53..4e6c07d05bc17 100644 --- a/x-pack/plugins/cases/server/routes/api/cases/helpers.ts +++ b/x-pack/plugins/cases/server/routes/api/cases/helpers.ts @@ -13,12 +13,13 @@ import { CaseConnector, ESCaseConnector, ESCasesConfigureAttributes, + ConnectorTypeFields, ConnectorTypes, CaseStatuses, CaseType, SavedObjectFindOptions, } from '../../../../common/api'; -import { ESConnectorFields, ConnectorTypeFields } from '../../../../common/api/connectors'; +import { ESConnectorFields } from '../../../../common/api/connectors'; import { CASE_SAVED_OBJECT, SUB_CASE_SAVED_OBJECT } from '../../../saved_object_types'; import { sortToSnake } from '../utils'; import { combineFilters } from '../../../common'; diff --git a/x-pack/plugins/cases/server/routes/api/cases/sub_case/find_sub_cases.ts b/x-pack/plugins/cases/server/routes/api/cases/sub_case/find_sub_cases.ts index c24dde1944f83..e7f9f8b4f2d73 100644 --- a/x-pack/plugins/cases/server/routes/api/cases/sub_case/find_sub_cases.ts +++ b/x-pack/plugins/cases/server/routes/api/cases/sub_case/find_sub_cases.ts @@ -78,12 +78,13 @@ export function initFindSubCasesApi({ caseService, router, logger }: RouteDeps) return response.ok({ body: SubCasesFindResponseRt.encode( transformSubCases({ - ...subCases, + page: subCases.page, + perPage: subCases.perPage, + total: subCases.total, + subCasesMap: subCases.subCasesMap, open, inProgress, closed, - // there should only be one entry in the map for the requested case ID - total: subCases.subCasesMap.get(request.params.case_id)?.length ?? 0, }) ), }); diff --git a/x-pack/plugins/cases/server/services/index.ts b/x-pack/plugins/cases/server/services/index.ts index 7c5f06d48bb05..48a1a1ed68432 100644 --- a/x-pack/plugins/cases/server/services/index.ts +++ b/x-pack/plugins/cases/server/services/index.ts @@ -171,6 +171,7 @@ interface SubCasesMapWithPageInfo { subCasesMap: Map; page: number; perPage: number; + total: number; } interface CaseCommentStats { @@ -194,6 +195,7 @@ interface CasesMapWithPageInfo { casesMap: Map; page: number; perPage: number; + total: number; } type FindCaseOptions = CasesFindRequest & SavedObjectFindOptions; @@ -348,6 +350,7 @@ export class CaseService implements CaseServiceSetup { casesMap: casesWithComments, page: cases.page, perPage: cases.per_page, + total: cases.total, }; } @@ -529,6 +532,7 @@ export class CaseService implements CaseServiceSetup { subCasesMap: new Map(), page: 0, perPage: 0, + total: 0, }; if (!options) { @@ -585,7 +589,7 @@ export class CaseService implements CaseServiceSetup { return accMap; }, new Map()); - return { subCasesMap, page: subCases.page, perPage: subCases.per_page }; + return { subCasesMap, page: subCases.page, perPage: subCases.per_page, total: subCases.total }; } /** diff --git a/x-pack/plugins/infra/common/log_sources/resolve_log_source_configuration.ts b/x-pack/plugins/infra/common/log_sources/resolve_log_source_configuration.ts new file mode 100644 index 0000000000000..ad4b2963a41bd --- /dev/null +++ b/x-pack/plugins/infra/common/log_sources/resolve_log_source_configuration.ts @@ -0,0 +1,19 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { LogSourceConfigurationProperties } from '../http_api/log_sources'; + +// NOTE: Type will change, see below. +type ResolvedLogsSourceConfiguration = LogSourceConfigurationProperties; + +// NOTE: This will handle real resolution for https://github.com/elastic/kibana/issues/92650, via the index patterns service, but for now just +// hands back properties from the saved object (and therefore looks pointless...). +export const resolveLogSourceConfiguration = ( + sourceConfiguration: LogSourceConfigurationProperties +): ResolvedLogsSourceConfiguration => { + return sourceConfiguration; +}; diff --git a/x-pack/plugins/infra/common/metrics_sources/index.ts b/x-pack/plugins/infra/common/metrics_sources/index.ts new file mode 100644 index 0000000000000..a697c65e5a0aa --- /dev/null +++ b/x-pack/plugins/infra/common/metrics_sources/index.ts @@ -0,0 +1,81 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import * as rt from 'io-ts'; +import { omit } from 'lodash'; +import { + SourceConfigurationRT, + SourceStatusRuntimeType, +} from '../source_configuration/source_configuration'; +import { DeepPartial } from '../utility_types'; + +/** + * Properties specific to the Metrics Source Configuration. + */ +export const metricsSourceConfigurationPropertiesRT = rt.strict({ + name: SourceConfigurationRT.props.name, + description: SourceConfigurationRT.props.description, + metricAlias: SourceConfigurationRT.props.metricAlias, + inventoryDefaultView: SourceConfigurationRT.props.inventoryDefaultView, + metricsExplorerDefaultView: SourceConfigurationRT.props.metricsExplorerDefaultView, + fields: rt.strict(omit(SourceConfigurationRT.props.fields.props, 'message')), + anomalyThreshold: rt.number, +}); + +export type MetricsSourceConfigurationProperties = rt.TypeOf< + typeof metricsSourceConfigurationPropertiesRT +>; + +export const partialMetricsSourceConfigurationPropertiesRT = rt.partial({ + ...metricsSourceConfigurationPropertiesRT.type.props, + fields: rt.partial({ + ...metricsSourceConfigurationPropertiesRT.type.props.fields.type.props, + }), +}); + +export type PartialMetricsSourceConfigurationProperties = rt.TypeOf< + typeof partialMetricsSourceConfigurationPropertiesRT +>; + +const metricsSourceConfigurationOriginRT = rt.keyof({ + fallback: null, + internal: null, + stored: null, +}); + +export const metricsSourceStatusRT = rt.strict({ + metricIndicesExist: SourceStatusRuntimeType.props.metricIndicesExist, + indexFields: SourceStatusRuntimeType.props.indexFields, +}); + +export type MetricsSourceStatus = rt.TypeOf; + +export const metricsSourceConfigurationRT = rt.exact( + rt.intersection([ + rt.type({ + id: rt.string, + origin: metricsSourceConfigurationOriginRT, + configuration: metricsSourceConfigurationPropertiesRT, + }), + rt.partial({ + updatedAt: rt.number, + version: rt.string, + status: metricsSourceStatusRT, + }), + ]) +); + +export type MetricsSourceConfiguration = rt.TypeOf; +export type PartialMetricsSourceConfiguration = DeepPartial; + +export const metricsSourceConfigurationResponseRT = rt.type({ + source: metricsSourceConfigurationRT, +}); + +export type MetricsSourceConfigurationResponse = rt.TypeOf< + typeof metricsSourceConfigurationResponseRT +>; diff --git a/x-pack/plugins/infra/common/http_api/source_api.ts b/x-pack/plugins/infra/common/source_configuration/source_configuration.ts similarity index 61% rename from x-pack/plugins/infra/common/http_api/source_api.ts rename to x-pack/plugins/infra/common/source_configuration/source_configuration.ts index f14151531ba35..ad68a7a019848 100644 --- a/x-pack/plugins/infra/common/http_api/source_api.ts +++ b/x-pack/plugins/infra/common/source_configuration/source_configuration.ts @@ -5,8 +5,19 @@ * 2.0. */ +/** + * These are the core source configuration types that represent a Source Configuration in + * it's entirety. There are then subsets of this configuration that form the Logs Source Configuration + * and Metrics Source Configuration. The Logs Source Configuration is further expanded to it's resolved form. + * -> Source Configuration + * -> Logs source configuration + * -> Resolved Logs Source Configuration + * -> Metrics Source Configuration + */ + /* eslint-disable @typescript-eslint/no-empty-interface */ +import { omit } from 'lodash'; import * as rt from 'io-ts'; import moment from 'moment'; import { pipe } from 'fp-ts/lib/pipeable'; @@ -29,121 +40,113 @@ export const TimestampFromString = new rt.Type( ); /** - * Stored source configuration as read from and written to saved objects + * Log columns */ -const SavedSourceConfigurationFieldsRuntimeType = rt.partial({ - container: rt.string, - host: rt.string, - pod: rt.string, - tiebreaker: rt.string, - timestamp: rt.string, -}); - -export type InfraSavedSourceConfigurationFields = rt.TypeOf< - typeof SavedSourceConfigurationFieldColumnRuntimeType ->; - -export const SavedSourceConfigurationTimestampColumnRuntimeType = rt.type({ +export const SourceConfigurationTimestampColumnRuntimeType = rt.type({ timestampColumn: rt.type({ id: rt.string, }), }); export type InfraSourceConfigurationTimestampColumn = rt.TypeOf< - typeof SavedSourceConfigurationTimestampColumnRuntimeType + typeof SourceConfigurationTimestampColumnRuntimeType >; -export const SavedSourceConfigurationMessageColumnRuntimeType = rt.type({ +export const SourceConfigurationMessageColumnRuntimeType = rt.type({ messageColumn: rt.type({ id: rt.string, }), }); export type InfraSourceConfigurationMessageColumn = rt.TypeOf< - typeof SavedSourceConfigurationMessageColumnRuntimeType + typeof SourceConfigurationMessageColumnRuntimeType >; -export const SavedSourceConfigurationFieldColumnRuntimeType = rt.type({ +export const SourceConfigurationFieldColumnRuntimeType = rt.type({ fieldColumn: rt.type({ id: rt.string, field: rt.string, }), }); -export const SavedSourceConfigurationColumnRuntimeType = rt.union([ - SavedSourceConfigurationTimestampColumnRuntimeType, - SavedSourceConfigurationMessageColumnRuntimeType, - SavedSourceConfigurationFieldColumnRuntimeType, +export type InfraSourceConfigurationFieldColumn = rt.TypeOf< + typeof SourceConfigurationFieldColumnRuntimeType +>; + +export const SourceConfigurationColumnRuntimeType = rt.union([ + SourceConfigurationTimestampColumnRuntimeType, + SourceConfigurationMessageColumnRuntimeType, + SourceConfigurationFieldColumnRuntimeType, ]); -export type InfraSavedSourceConfigurationColumn = rt.TypeOf< - typeof SavedSourceConfigurationColumnRuntimeType ->; +export type InfraSourceConfigurationColumn = rt.TypeOf; -export const SavedSourceConfigurationRuntimeType = rt.partial({ +/** + * Fields + */ + +const SourceConfigurationFieldsRT = rt.type({ + container: rt.string, + host: rt.string, + pod: rt.string, + tiebreaker: rt.string, + timestamp: rt.string, + message: rt.array(rt.string), +}); + +/** + * Properties that represent a full source configuration, which is the result of merging static values with + * saved values. + */ +export const SourceConfigurationRT = rt.type({ name: rt.string, description: rt.string, metricAlias: rt.string, logAlias: rt.string, inventoryDefaultView: rt.string, metricsExplorerDefaultView: rt.string, - fields: SavedSourceConfigurationFieldsRuntimeType, - logColumns: rt.array(SavedSourceConfigurationColumnRuntimeType), + fields: SourceConfigurationFieldsRT, + logColumns: rt.array(SourceConfigurationColumnRuntimeType), anomalyThreshold: rt.number, }); +/** + * Stored source configuration as read from and written to saved objects + */ +const SavedSourceConfigurationFieldsRuntimeType = rt.partial( + omit(SourceConfigurationFieldsRT.props, ['message']) +); + +export type InfraSavedSourceConfigurationFields = rt.TypeOf< + typeof SavedSourceConfigurationFieldsRuntimeType +>; + +export const SavedSourceConfigurationRuntimeType = rt.intersection([ + rt.partial(omit(SourceConfigurationRT.props, ['fields'])), + rt.partial({ + fields: SavedSourceConfigurationFieldsRuntimeType, + }), +]); + export interface InfraSavedSourceConfiguration extends rt.TypeOf {} export const pickSavedSourceConfiguration = ( value: InfraSourceConfiguration ): InfraSavedSourceConfiguration => { - const { - name, - description, - metricAlias, - logAlias, - fields, - inventoryDefaultView, - metricsExplorerDefaultView, - logColumns, - anomalyThreshold, - } = value; - const { container, host, pod, tiebreaker, timestamp } = fields; - - return { - name, - description, - metricAlias, - logAlias, - inventoryDefaultView, - metricsExplorerDefaultView, - fields: { container, host, pod, tiebreaker, timestamp }, - logColumns, - anomalyThreshold, - }; + return value; }; /** - * Static source configuration as read from the configuration file + * Static source configuration, the result of merging values from the config file and + * hardcoded defaults. */ -const StaticSourceConfigurationFieldsRuntimeType = rt.partial({ - ...SavedSourceConfigurationFieldsRuntimeType.props, - message: rt.array(rt.string), -}); - +const StaticSourceConfigurationFieldsRuntimeType = rt.partial(SourceConfigurationFieldsRT.props); export const StaticSourceConfigurationRuntimeType = rt.partial({ - name: rt.string, - description: rt.string, - metricAlias: rt.string, - logAlias: rt.string, - inventoryDefaultView: rt.string, - metricsExplorerDefaultView: rt.string, + ...SourceConfigurationRT.props, fields: StaticSourceConfigurationFieldsRuntimeType, - logColumns: rt.array(SavedSourceConfigurationColumnRuntimeType), - anomalyThreshold: rt.number, }); export interface InfraStaticSourceConfiguration @@ -153,18 +156,20 @@ export interface InfraStaticSourceConfiguration * Full source configuration type after all cleanup has been done at the edges */ -const SourceConfigurationFieldsRuntimeType = rt.type({ - ...StaticSourceConfigurationFieldsRuntimeType.props, -}); - -export type InfraSourceConfigurationFields = rt.TypeOf; +export type InfraSourceConfigurationFields = rt.TypeOf; export const SourceConfigurationRuntimeType = rt.type({ - ...SavedSourceConfigurationRuntimeType.props, - fields: SourceConfigurationFieldsRuntimeType, - logColumns: rt.array(SavedSourceConfigurationColumnRuntimeType), + ...SourceConfigurationRT.props, + fields: SourceConfigurationFieldsRT, + logColumns: rt.array(SourceConfigurationColumnRuntimeType), }); +export interface InfraSourceConfiguration + extends rt.TypeOf {} + +/** + * Source status + */ const SourceStatusFieldRuntimeType = rt.type({ name: rt.string, type: rt.string, @@ -175,12 +180,17 @@ const SourceStatusFieldRuntimeType = rt.type({ export type InfraSourceIndexField = rt.TypeOf; -const SourceStatusRuntimeType = rt.type({ +export const SourceStatusRuntimeType = rt.type({ logIndicesExist: rt.boolean, metricIndicesExist: rt.boolean, indexFields: rt.array(SourceStatusFieldRuntimeType), }); +export interface InfraSourceStatus extends rt.TypeOf {} + +/** + * Source configuration along with source status and metadata + */ export const SourceRuntimeType = rt.intersection([ rt.type({ id: rt.string, @@ -198,11 +208,6 @@ export const SourceRuntimeType = rt.intersection([ }), ]); -export interface InfraSourceStatus extends rt.TypeOf {} - -export interface InfraSourceConfiguration - extends rt.TypeOf {} - export interface InfraSource extends rt.TypeOf {} export const SourceResponseRuntimeType = rt.type({ diff --git a/x-pack/plugins/infra/public/alerting/inventory/components/expression.test.tsx b/x-pack/plugins/infra/public/alerting/inventory/components/expression.test.tsx index 88d72300c2d6d..b345e138accec 100644 --- a/x-pack/plugins/infra/public/alerting/inventory/components/expression.test.tsx +++ b/x-pack/plugins/infra/public/alerting/inventory/components/expression.test.tsx @@ -17,7 +17,7 @@ import { act } from 'react-dom/test-utils'; import { Comparator } from '../../../../server/lib/alerting/metric_threshold/types'; import { SnapshotCustomMetricInput } from '../../../../common/http_api/snapshot_api'; -jest.mock('../../../containers/source/use_source_via_http', () => ({ +jest.mock('../../../containers/metrics_source/use_source_via_http', () => ({ useSourceViaHttp: () => ({ source: { id: 'default' }, createDerivedIndexPattern: () => ({ fields: [], title: 'metricbeat-*' }), diff --git a/x-pack/plugins/infra/public/alerting/inventory/components/expression.tsx b/x-pack/plugins/infra/public/alerting/inventory/components/expression.tsx index b28c76d1cb374..c4f8b5a615b0f 100644 --- a/x-pack/plugins/infra/public/alerting/inventory/components/expression.tsx +++ b/x-pack/plugins/infra/public/alerting/inventory/components/expression.tsx @@ -43,7 +43,7 @@ import { AlertTypeParamsExpressionProps, } from '../../../../../triggers_actions_ui/public'; import { MetricsExplorerKueryBar } from '../../../pages/metrics/metrics_explorer/components/kuery_bar'; -import { useSourceViaHttp } from '../../../containers/source/use_source_via_http'; +import { useSourceViaHttp } from '../../../containers/metrics_source/use_source_via_http'; import { sqsMetricTypes } from '../../../../common/inventory_models/aws_sqs/toolbar_items'; import { ec2MetricTypes } from '../../../../common/inventory_models/aws_ec2/toolbar_items'; import { s3MetricTypes } from '../../../../common/inventory_models/aws_s3/toolbar_items'; @@ -124,14 +124,13 @@ export const Expressions: React.FC = (props) => { } = props; const { source, createDerivedIndexPattern } = useSourceViaHttp({ sourceId: 'default', - type: 'metrics', fetch: http.fetch, toastWarning: notifications.toasts.addWarning, }); const [timeSize, setTimeSize] = useState(1); const [timeUnit, setTimeUnit] = useState('m'); - const derivedIndexPattern = useMemo(() => createDerivedIndexPattern('metrics'), [ + const derivedIndexPattern = useMemo(() => createDerivedIndexPattern(), [ createDerivedIndexPattern, ]); diff --git a/x-pack/plugins/infra/public/alerting/metric_anomaly/components/expression.test.tsx b/x-pack/plugins/infra/public/alerting/metric_anomaly/components/expression.test.tsx index dd4cbe10b74ee..6b99aff9f903d 100644 --- a/x-pack/plugins/infra/public/alerting/metric_anomaly/components/expression.test.tsx +++ b/x-pack/plugins/infra/public/alerting/metric_anomaly/components/expression.test.tsx @@ -12,7 +12,7 @@ import React from 'react'; import { Expression, AlertContextMeta } from './expression'; import { act } from 'react-dom/test-utils'; -jest.mock('../../../containers/source/use_source_via_http', () => ({ +jest.mock('../../../containers/metrics_source/use_source_via_http', () => ({ useSourceViaHttp: () => ({ source: { id: 'default' }, createDerivedIndexPattern: () => ({ fields: [], title: 'metricbeat-*' }), diff --git a/x-pack/plugins/infra/public/alerting/metric_anomaly/components/expression.tsx b/x-pack/plugins/infra/public/alerting/metric_anomaly/components/expression.tsx index 12cc2bf9fb3a9..afbd6ffa8b5f7 100644 --- a/x-pack/plugins/infra/public/alerting/metric_anomaly/components/expression.tsx +++ b/x-pack/plugins/infra/public/alerting/metric_anomaly/components/expression.tsx @@ -27,7 +27,7 @@ import { AlertTypeParamsExpressionProps, // eslint-disable-next-line @kbn/eslint/no-restricted-paths } from '../../../../../triggers_actions_ui/public/types'; -import { useSourceViaHttp } from '../../../containers/source/use_source_via_http'; +import { useSourceViaHttp } from '../../../containers/metrics_source/use_source_via_http'; import { findInventoryModel } from '../../../../common/inventory_models'; import { InventoryItemType, SnapshotMetricType } from '../../../../common/inventory_models/types'; import { NodeTypeExpression } from './node_type'; @@ -75,12 +75,11 @@ export const Expression: React.FC = (props) => { } = props; const { source, createDerivedIndexPattern } = useSourceViaHttp({ sourceId: 'default', - type: 'metrics', fetch: http.fetch, toastWarning: notifications.toasts.addWarning, }); - const derivedIndexPattern = useMemo(() => createDerivedIndexPattern('metrics'), [ + const derivedIndexPattern = useMemo(() => createDerivedIndexPattern(), [ createDerivedIndexPattern, ]); diff --git a/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression.test.tsx b/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression.test.tsx index a6d74d4f461a6..667f5c061ce48 100644 --- a/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression.test.tsx +++ b/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression.test.tsx @@ -15,7 +15,7 @@ import { act } from 'react-dom/test-utils'; // eslint-disable-next-line @kbn/eslint/no-restricted-paths import { Comparator } from '../../../../server/lib/alerting/metric_threshold/types'; -jest.mock('../../../containers/source/use_source_via_http', () => ({ +jest.mock('../../../containers/metrics_source/use_source_via_http', () => ({ useSourceViaHttp: () => ({ source: { id: 'default' }, createDerivedIndexPattern: () => ({ fields: [], title: 'metricbeat-*' }), diff --git a/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression.tsx b/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression.tsx index 3b8afc173c2bd..8835a7cd55ce8 100644 --- a/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression.tsx +++ b/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression.tsx @@ -35,7 +35,7 @@ import { import { MetricsExplorerKueryBar } from '../../../pages/metrics/metrics_explorer/components/kuery_bar'; import { MetricsExplorerOptions } from '../../../pages/metrics/metrics_explorer/hooks/use_metrics_explorer_options'; import { MetricsExplorerGroupBy } from '../../../pages/metrics/metrics_explorer/components/group_by'; -import { useSourceViaHttp } from '../../../containers/source/use_source_via_http'; +import { useSourceViaHttp } from '../../../containers/metrics_source/use_source_via_http'; import { convertKueryToElasticSearchQuery } from '../../../utils/kuery'; import { ExpressionRow } from './expression_row'; @@ -73,14 +73,13 @@ export const Expressions: React.FC = (props) => { const { http, notifications } = useKibanaContextForPlugin().services; const { source, createDerivedIndexPattern } = useSourceViaHttp({ sourceId: 'default', - type: 'metrics', fetch: http.fetch, toastWarning: notifications.toasts.addWarning, }); const [timeSize, setTimeSize] = useState(1); const [timeUnit, setTimeUnit] = useState('m'); - const derivedIndexPattern = useMemo(() => createDerivedIndexPattern('metrics'), [ + const derivedIndexPattern = useMemo(() => createDerivedIndexPattern(), [ createDerivedIndexPattern, ]); diff --git a/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression_chart.test.tsx b/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression_chart.test.tsx index 7e4209e4253d7..caf8e32814fe5 100644 --- a/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression_chart.test.tsx +++ b/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression_chart.test.tsx @@ -10,7 +10,7 @@ import { mountWithIntl, nextTick } from '@kbn/test/jest'; import { coreMock as mockCoreMock } from 'src/core/public/mocks'; import { MetricExpression } from '../types'; import { IIndexPattern } from 'src/plugins/data/public'; -import { InfraSource } from '../../../../common/http_api/source_api'; +import { MetricsSourceConfiguration } from '../../../../common/metrics_sources'; import React from 'react'; import { ExpressionChart } from './expression_chart'; import { act } from 'react-dom/test-utils'; @@ -45,20 +45,17 @@ describe('ExpressionChart', () => { fields: [], }; - const source: InfraSource = { + const source: MetricsSourceConfiguration = { id: 'default', origin: 'fallback', configuration: { name: 'default', description: 'The default configuration', - logColumns: [], metricAlias: 'metricbeat-*', - logAlias: 'filebeat-*', inventoryDefaultView: 'host', metricsExplorerDefaultView: 'host', fields: { timestamp: '@timestamp', - message: ['message'], container: 'container.id', host: 'host.name', pod: 'kubernetes.pod.uid', diff --git a/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression_chart.tsx b/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression_chart.tsx index 2a274c4b6d50f..e5558b961ab20 100644 --- a/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression_chart.tsx +++ b/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression_chart.tsx @@ -11,7 +11,7 @@ import { first, last } from 'lodash'; import { EuiText } from '@elastic/eui'; import { FormattedMessage } from '@kbn/i18n/react'; import { IIndexPattern } from 'src/plugins/data/public'; -import { InfraSource } from '../../../../common/http_api/source_api'; +import { MetricsSourceConfiguration } from '../../../../common/metrics_sources'; import { Color } from '../../../../common/color_palette'; import { MetricsExplorerRow, MetricsExplorerAggregation } from '../../../../common/http_api'; import { MetricExplorerSeriesChart } from '../../../pages/metrics/metrics_explorer/components/series_chart'; @@ -35,7 +35,7 @@ import { ThresholdAnnotations } from '../../common/criterion_preview_chart/thres interface Props { expression: MetricExpression; derivedIndexPattern: IIndexPattern; - source: InfraSource | null; + source: MetricsSourceConfiguration | null; filterQuery?: string; groupBy?: string | string[]; } diff --git a/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression_row.test.tsx b/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression_row.test.tsx index 54477a39c2626..90f75e6a94022 100644 --- a/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression_row.test.tsx +++ b/x-pack/plugins/infra/public/alerting/metric_threshold/components/expression_row.test.tsx @@ -13,7 +13,7 @@ import { act } from 'react-dom/test-utils'; // eslint-disable-next-line @kbn/eslint/no-restricted-paths import { Comparator } from '../../../../server/lib/alerting/metric_threshold/types'; -jest.mock('../../../containers/source/use_source_via_http', () => ({ +jest.mock('../../../containers/metrics_source/use_source_via_http', () => ({ useSourceViaHttp: () => ({ source: { id: 'default' }, createDerivedIndexPattern: () => ({ fields: [], title: 'metricbeat-*' }), diff --git a/x-pack/plugins/infra/public/alerting/metric_threshold/hooks/use_metrics_explorer_chart_data.ts b/x-pack/plugins/infra/public/alerting/metric_threshold/hooks/use_metrics_explorer_chart_data.ts index 908372d13b6bc..e3006993216ae 100644 --- a/x-pack/plugins/infra/public/alerting/metric_threshold/hooks/use_metrics_explorer_chart_data.ts +++ b/x-pack/plugins/infra/public/alerting/metric_threshold/hooks/use_metrics_explorer_chart_data.ts @@ -7,7 +7,7 @@ import { IIndexPattern } from 'src/plugins/data/public'; import { useMemo } from 'react'; -import { InfraSource } from '../../../../common/http_api/source_api'; +import { MetricsSourceConfiguration } from '../../../../common/metrics_sources'; import { MetricExpression } from '../types'; import { MetricsExplorerOptions } from '../../../pages/metrics/metrics_explorer/hooks/use_metrics_explorer_options'; import { useMetricsExplorerData } from '../../../pages/metrics/metrics_explorer/hooks/use_metrics_explorer_data'; @@ -15,7 +15,7 @@ import { useMetricsExplorerData } from '../../../pages/metrics/metrics_explorer/ export const useMetricsExplorerChartData = ( expression: MetricExpression, derivedIndexPattern: IIndexPattern, - source: InfraSource | null, + source: MetricsSourceConfiguration | null, filterQuery?: string, groupBy?: string | string[] ) => { diff --git a/x-pack/plugins/infra/public/components/source_configuration/index.ts b/x-pack/plugins/infra/public/components/source_configuration/index.ts deleted file mode 100644 index 50db601234a8c..0000000000000 --- a/x-pack/plugins/infra/public/components/source_configuration/index.ts +++ /dev/null @@ -1,10 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -export * from './input_fields'; -export { SourceConfigurationSettings } from './source_configuration_settings'; -export { ViewSourceConfigurationButton } from './view_source_configuration_button'; diff --git a/x-pack/plugins/infra/public/components/source_configuration/log_columns_configuration_form_state.tsx b/x-pack/plugins/infra/public/components/source_configuration/log_columns_configuration_form_state.tsx deleted file mode 100644 index b5b28cb25b83b..0000000000000 --- a/x-pack/plugins/infra/public/components/source_configuration/log_columns_configuration_form_state.tsx +++ /dev/null @@ -1,156 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import React, { useCallback, useMemo, useState } from 'react'; - -import { FormattedMessage } from '@kbn/i18n/react'; - -import { - LogColumnConfiguration, - isTimestampLogColumnConfiguration, - isMessageLogColumnConfiguration, - TimestampLogColumnConfiguration, - MessageLogColumnConfiguration, - FieldLogColumnConfiguration, -} from '../../utils/source_configuration'; - -export interface TimestampLogColumnConfigurationProps { - logColumnConfiguration: TimestampLogColumnConfiguration['timestampColumn']; - remove: () => void; - type: 'timestamp'; -} - -export interface MessageLogColumnConfigurationProps { - logColumnConfiguration: MessageLogColumnConfiguration['messageColumn']; - remove: () => void; - type: 'message'; -} - -export interface FieldLogColumnConfigurationProps { - logColumnConfiguration: FieldLogColumnConfiguration['fieldColumn']; - remove: () => void; - type: 'field'; -} - -export type LogColumnConfigurationProps = - | TimestampLogColumnConfigurationProps - | MessageLogColumnConfigurationProps - | FieldLogColumnConfigurationProps; - -interface FormState { - logColumns: LogColumnConfiguration[]; -} - -type FormStateChanges = Partial; - -export const useLogColumnsConfigurationFormState = ({ - initialFormState = defaultFormState, -}: { - initialFormState?: FormState; -}) => { - const [formStateChanges, setFormStateChanges] = useState({}); - - const resetForm = useCallback(() => setFormStateChanges({}), []); - - const formState = useMemo( - () => ({ - ...initialFormState, - ...formStateChanges, - }), - [initialFormState, formStateChanges] - ); - - const logColumnConfigurationProps = useMemo( - () => - formState.logColumns.map( - (logColumn): LogColumnConfigurationProps => { - const remove = () => - setFormStateChanges((changes) => ({ - ...changes, - logColumns: formState.logColumns.filter((item) => item !== logColumn), - })); - - if (isTimestampLogColumnConfiguration(logColumn)) { - return { - logColumnConfiguration: logColumn.timestampColumn, - remove, - type: 'timestamp', - }; - } else if (isMessageLogColumnConfiguration(logColumn)) { - return { - logColumnConfiguration: logColumn.messageColumn, - remove, - type: 'message', - }; - } else { - return { - logColumnConfiguration: logColumn.fieldColumn, - remove, - type: 'field', - }; - } - } - ), - [formState.logColumns] - ); - - const addLogColumn = useCallback( - (logColumnConfiguration: LogColumnConfiguration) => - setFormStateChanges((changes) => ({ - ...changes, - logColumns: [...formState.logColumns, logColumnConfiguration], - })), - [formState.logColumns] - ); - - const moveLogColumn = useCallback( - (sourceIndex, destinationIndex) => { - if (destinationIndex >= 0 && sourceIndex <= formState.logColumns.length - 1) { - const newLogColumns = [...formState.logColumns]; - newLogColumns.splice(destinationIndex, 0, newLogColumns.splice(sourceIndex, 1)[0]); - setFormStateChanges((changes) => ({ - ...changes, - logColumns: newLogColumns, - })); - } - }, - [formState.logColumns] - ); - - const errors = useMemo( - () => - logColumnConfigurationProps.length <= 0 - ? [ - , - ] - : [], - [logColumnConfigurationProps] - ); - - const isFormValid = useMemo(() => (errors.length <= 0 ? true : false), [errors]); - - const isFormDirty = useMemo(() => Object.keys(formStateChanges).length > 0, [formStateChanges]); - - return { - addLogColumn, - moveLogColumn, - errors, - logColumnConfigurationProps, - formState, - formStateChanges, - isFormDirty, - isFormValid, - resetForm, - }; -}; - -const defaultFormState: FormState = { - logColumns: [], -}; diff --git a/x-pack/plugins/infra/public/containers/source/index.ts b/x-pack/plugins/infra/public/containers/metrics_source/index.ts similarity index 100% rename from x-pack/plugins/infra/public/containers/source/index.ts rename to x-pack/plugins/infra/public/containers/metrics_source/index.ts diff --git a/x-pack/plugins/infra/public/containers/source/source.tsx b/x-pack/plugins/infra/public/containers/metrics_source/source.tsx similarity index 79% rename from x-pack/plugins/infra/public/containers/source/source.tsx rename to x-pack/plugins/infra/public/containers/metrics_source/source.tsx index 8e2a8f29e03df..b730f8b007e43 100644 --- a/x-pack/plugins/infra/public/containers/source/source.tsx +++ b/x-pack/plugins/infra/public/containers/metrics_source/source.tsx @@ -9,27 +9,25 @@ import createContainer from 'constate'; import { useEffect, useMemo, useState } from 'react'; import { - InfraSavedSourceConfiguration, - InfraSource, - SourceResponse, -} from '../../../common/http_api/source_api'; + MetricsSourceConfigurationResponse, + MetricsSourceConfiguration, + PartialMetricsSourceConfigurationProperties, +} from '../../../common/metrics_sources'; + import { useTrackedPromise } from '../../utils/use_tracked_promise'; import { useKibana } from '../../../../../../src/plugins/kibana_react/public'; export const pickIndexPattern = ( - source: InfraSource | undefined, - type: 'logs' | 'metrics' | 'both' + source: MetricsSourceConfiguration | undefined, + type: 'metrics' ) => { if (!source) { return 'unknown-index'; } - if (type === 'logs') { - return source.configuration.logAlias; - } if (type === 'metrics') { return source.configuration.metricAlias; } - return `${source.configuration.logAlias},${source.configuration.metricAlias}`; + return `${source.configuration.metricAlias}`; }; const DEPENDENCY_ERROR_MESSAGE = 'Failed to load source: No fetch client available.'; @@ -39,7 +37,7 @@ export const useSource = ({ sourceId }: { sourceId: string }) => { const fetchService = kibana.services.http?.fetch; const API_URL = `/api/metrics/source/${sourceId}`; - const [source, setSource] = useState(undefined); + const [source, setSource] = useState(undefined); const [loadSourceRequest, loadSource] = useTrackedPromise( { @@ -49,7 +47,7 @@ export const useSource = ({ sourceId }: { sourceId: string }) => { throw new Error(DEPENDENCY_ERROR_MESSAGE); } - return await fetchService(`${API_URL}/metrics`, { + return await fetchService(`${API_URL}`, { method: 'GET', }); }, @@ -62,12 +60,12 @@ export const useSource = ({ sourceId }: { sourceId: string }) => { const [createSourceConfigurationRequest, createSourceConfiguration] = useTrackedPromise( { - createPromise: async (sourceProperties: InfraSavedSourceConfiguration) => { + createPromise: async (sourceProperties: PartialMetricsSourceConfigurationProperties) => { if (!fetchService) { throw new Error(DEPENDENCY_ERROR_MESSAGE); } - return await fetchService(API_URL, { + return await fetchService(API_URL, { method: 'PATCH', body: JSON.stringify(sourceProperties), }); @@ -83,12 +81,12 @@ export const useSource = ({ sourceId }: { sourceId: string }) => { const [updateSourceConfigurationRequest, updateSourceConfiguration] = useTrackedPromise( { - createPromise: async (sourceProperties: InfraSavedSourceConfiguration) => { + createPromise: async (sourceProperties: PartialMetricsSourceConfigurationProperties) => { if (!fetchService) { throw new Error(DEPENDENCY_ERROR_MESSAGE); } - return await fetchService(API_URL, { + return await fetchService(API_URL, { method: 'PATCH', body: JSON.stringify(sourceProperties), }); @@ -102,7 +100,7 @@ export const useSource = ({ sourceId }: { sourceId: string }) => { [fetchService, sourceId] ); - const createDerivedIndexPattern = (type: 'logs' | 'metrics' | 'both') => { + const createDerivedIndexPattern = (type: 'metrics') => { return { fields: source?.status ? source.status.indexFields : [], title: pickIndexPattern(source, type), @@ -129,9 +127,6 @@ export const useSource = ({ sourceId }: { sourceId: string }) => { const sourceExists = useMemo(() => (source ? !!source.version : undefined), [source]); - const logIndicesExist = useMemo(() => source && source.status && source.status.logIndicesExist, [ - source, - ]); const metricIndicesExist = useMemo( () => source && source.status && source.status.metricIndicesExist, [source] @@ -144,7 +139,6 @@ export const useSource = ({ sourceId }: { sourceId: string }) => { return { createSourceConfiguration, createDerivedIndexPattern, - logIndicesExist, isLoading, isLoadingSource: loadSourceRequest.state === 'pending', isUninitialized, diff --git a/x-pack/plugins/infra/public/containers/source/use_source_via_http.ts b/x-pack/plugins/infra/public/containers/metrics_source/use_source_via_http.ts similarity index 62% rename from x-pack/plugins/infra/public/containers/source/use_source_via_http.ts rename to x-pack/plugins/infra/public/containers/metrics_source/use_source_via_http.ts index 548e6b8aa9cd9..2947f8fb09847 100644 --- a/x-pack/plugins/infra/public/containers/source/use_source_via_http.ts +++ b/x-pack/plugins/infra/public/containers/metrics_source/use_source_via_http.ts @@ -13,51 +13,47 @@ import createContainer from 'constate'; import { HttpHandler } from 'src/core/public'; import { ToastInput } from 'src/core/public'; import { - SourceResponseRuntimeType, - SourceResponse, - InfraSource, -} from '../../../common/http_api/source_api'; + metricsSourceConfigurationResponseRT, + MetricsSourceConfigurationResponse, + MetricsSourceConfiguration, +} from '../../../common/metrics_sources'; import { useHTTPRequest } from '../../hooks/use_http_request'; import { throwErrors, createPlainError } from '../../../common/runtime_types'; export const pickIndexPattern = ( - source: InfraSource | undefined, - type: 'logs' | 'metrics' | 'both' + source: MetricsSourceConfiguration | undefined, + type: 'metrics' ) => { if (!source) { return 'unknown-index'; } - if (type === 'logs') { - return source.configuration.logAlias; - } if (type === 'metrics') { return source.configuration.metricAlias; } - return `${source.configuration.logAlias},${source.configuration.metricAlias}`; + return `${source.configuration.metricAlias}`; }; interface Props { sourceId: string; - type: 'logs' | 'metrics' | 'both'; fetch?: HttpHandler; toastWarning?: (input: ToastInput) => void; } -export const useSourceViaHttp = ({ - sourceId = 'default', - type = 'both', - fetch, - toastWarning, -}: Props) => { +export const useSourceViaHttp = ({ sourceId = 'default', fetch, toastWarning }: Props) => { const decodeResponse = (response: any) => { return pipe( - SourceResponseRuntimeType.decode(response), + metricsSourceConfigurationResponseRT.decode(response), fold(throwErrors(createPlainError), identity) ); }; - const { error, loading, response, makeRequest } = useHTTPRequest( - `/api/metrics/source/${sourceId}/${type}`, + const { + error, + loading, + response, + makeRequest, + } = useHTTPRequest( + `/api/metrics/source/${sourceId}`, 'GET', null, decodeResponse, @@ -71,15 +67,12 @@ export const useSourceViaHttp = ({ })(); }, [makeRequest]); - const createDerivedIndexPattern = useCallback( - (indexType: 'logs' | 'metrics' | 'both' = type) => { - return { - fields: response?.source.status ? response.source.status.indexFields : [], - title: pickIndexPattern(response?.source, indexType), - }; - }, - [response, type] - ); + const createDerivedIndexPattern = useCallback(() => { + return { + fields: response?.source.status ? response.source.status.indexFields : [], + title: pickIndexPattern(response?.source, 'metrics'), + }; + }, [response]); const source = useMemo(() => { return response ? response.source : null; diff --git a/x-pack/plugins/infra/public/containers/saved_view/saved_view.tsx b/x-pack/plugins/infra/public/containers/saved_view/saved_view.tsx index 4c4835cbe4cdb..56a2a13e31ff7 100644 --- a/x-pack/plugins/infra/public/containers/saved_view/saved_view.tsx +++ b/x-pack/plugins/infra/public/containers/saved_view/saved_view.tsx @@ -17,10 +17,10 @@ import { useUrlState } from '../../utils/use_url_state'; import { useFindSavedObject } from '../../hooks/use_find_saved_object'; import { useCreateSavedObject } from '../../hooks/use_create_saved_object'; import { useDeleteSavedObject } from '../../hooks/use_delete_saved_object'; -import { Source } from '../source'; +import { Source } from '../metrics_source'; import { metricsExplorerViewSavedObjectName } from '../../../common/saved_objects/metrics_explorer_view'; import { inventoryViewSavedObjectName } from '../../../common/saved_objects/inventory_view'; -import { useSourceConfigurationFormState } from '../../components/source_configuration/source_configuration_form_state'; +import { useSourceConfigurationFormState } from '../../pages/metrics/settings/source_configuration_form_state'; import { useGetSavedObject } from '../../hooks/use_get_saved_object'; import { useUpdateSavedObject } from '../../hooks/use_update_saved_object'; diff --git a/x-pack/plugins/infra/public/containers/with_source/with_source.tsx b/x-pack/plugins/infra/public/containers/with_source/with_source.tsx index 3b9f0d3e1eae2..f3ca57a40c4c7 100644 --- a/x-pack/plugins/infra/public/containers/with_source/with_source.tsx +++ b/x-pack/plugins/infra/public/containers/with_source/with_source.tsx @@ -9,17 +9,19 @@ import React, { useContext } from 'react'; import { IIndexPattern } from 'src/plugins/data/public'; import { - InfraSavedSourceConfiguration, - InfraSourceConfiguration, -} from '../../../common/http_api/source_api'; + MetricsSourceConfigurationProperties, + PartialMetricsSourceConfigurationProperties, +} from '../../../common/metrics_sources'; import { RendererFunction } from '../../utils/typed_react'; -import { Source } from '../source'; +import { Source } from '../metrics_source'; interface WithSourceProps { children: RendererFunction<{ - configuration?: InfraSourceConfiguration; - create: (sourceProperties: InfraSavedSourceConfiguration) => Promise | undefined; - createDerivedIndexPattern: (type: 'logs' | 'metrics' | 'both') => IIndexPattern; + configuration?: MetricsSourceConfigurationProperties; + create: ( + sourceProperties: PartialMetricsSourceConfigurationProperties + ) => Promise | undefined; + createDerivedIndexPattern: (type: 'metrics') => IIndexPattern; exists?: boolean; hasFailed: boolean; isLoading: boolean; @@ -29,7 +31,9 @@ interface WithSourceProps { metricAlias?: string; metricIndicesExist?: boolean; sourceId: string; - update: (sourceProperties: InfraSavedSourceConfiguration) => Promise | undefined; + update: ( + sourceProperties: PartialMetricsSourceConfigurationProperties + ) => Promise | undefined; version?: string; }>; } @@ -42,7 +46,6 @@ export const WithSource: React.FunctionComponent = ({ children sourceExists, sourceId, metricIndicesExist, - logIndicesExist, isLoading, loadSource, hasFailedLoadingSource, @@ -60,7 +63,6 @@ export const WithSource: React.FunctionComponent = ({ children isLoading, lastFailureMessage: loadSourceFailureMessage, load: loadSource, - logIndicesExist, metricIndicesExist, sourceId, update: updateSourceConfiguration, diff --git a/x-pack/plugins/infra/public/lib/lib.ts b/x-pack/plugins/infra/public/lib/lib.ts index 622e0c9d33845..4541eb6518788 100644 --- a/x-pack/plugins/infra/public/lib/lib.ts +++ b/x-pack/plugins/infra/public/lib/lib.ts @@ -14,7 +14,7 @@ import { SnapshotNodeMetric, SnapshotNodePath, } from '../../common/http_api/snapshot_api'; -import { InfraSourceConfigurationFields } from '../../common/http_api/source_api'; +import { MetricsSourceConfigurationProperties } from '../../common/metrics_sources'; import { WaffleSortOption } from '../pages/metrics/inventory_view/hooks/use_waffle_options'; export interface InfraWaffleMapNode { @@ -124,7 +124,7 @@ export enum InfraWaffleMapRuleOperator { } export interface InfraWaffleMapOptions { - fields?: InfraSourceConfigurationFields | null; + fields?: MetricsSourceConfigurationProperties['fields'] | null; formatter: InfraFormatterType; formatTemplate: string; metric: SnapshotMetricInput; diff --git a/x-pack/plugins/infra/public/metrics_overview_fetchers.ts b/x-pack/plugins/infra/public/metrics_overview_fetchers.ts index 45b17aeb1f724..bcc2eec504209 100644 --- a/x-pack/plugins/infra/public/metrics_overview_fetchers.ts +++ b/x-pack/plugins/infra/public/metrics_overview_fetchers.ts @@ -14,9 +14,7 @@ export const createMetricsHasData = ( ) => async () => { const [coreServices] = await getStartServices(); const { http } = coreServices; - const results = await http.get<{ hasData: boolean }>( - '/api/metrics/source/default/metrics/hasData' - ); + const results = await http.get<{ hasData: boolean }>('/api/metrics/source/default/hasData'); return results.hasData; }; diff --git a/x-pack/plugins/infra/public/pages/link_to/redirect_to_host_detail_via_ip.tsx b/x-pack/plugins/infra/public/pages/link_to/redirect_to_host_detail_via_ip.tsx index ea2e67abc4141..8377eadfbce1d 100644 --- a/x-pack/plugins/infra/public/pages/link_to/redirect_to_host_detail_via_ip.tsx +++ b/x-pack/plugins/infra/public/pages/link_to/redirect_to_host_detail_via_ip.tsx @@ -14,7 +14,7 @@ import { useHostIpToName } from './use_host_ip_to_name'; import { getFromFromLocation, getToFromLocation } from './query_params'; import { LoadingPage } from '../../components/loading_page'; import { Error } from '../error'; -import { useSource } from '../../containers/source/source'; +import { useSourceViaHttp } from '../../containers/metrics_source/use_source_via_http'; type RedirectToHostDetailType = RouteComponentProps<{ hostIp: string; @@ -26,7 +26,7 @@ export const RedirectToHostDetailViaIP = ({ }, location, }: RedirectToHostDetailType) => { - const { source } = useSource({ sourceId: 'default' }); + const { source } = useSourceViaHttp({ sourceId: 'default' }); const { error, name } = useHostIpToName( hostIp, diff --git a/x-pack/plugins/infra/public/pages/logs/settings/fields_configuration_panel.tsx b/x-pack/plugins/infra/public/pages/logs/settings/fields_configuration_panel.tsx index 13eea67fb2a5a..236817ce3890f 100644 --- a/x-pack/plugins/infra/public/pages/logs/settings/fields_configuration_panel.tsx +++ b/x-pack/plugins/infra/public/pages/logs/settings/fields_configuration_panel.tsx @@ -19,7 +19,7 @@ import { import { i18n } from '@kbn/i18n'; import { FormattedMessage } from '@kbn/i18n/react'; import React from 'react'; -import { InputFieldProps } from '../../../components/source_configuration'; +import { InputFieldProps } from '../../../components/source_configuration/input_fields'; interface FieldsConfigurationPanelProps { isLoading: boolean; diff --git a/x-pack/plugins/infra/public/pages/logs/settings/indices_configuration_panel.tsx b/x-pack/plugins/infra/public/pages/logs/settings/indices_configuration_panel.tsx index 72b5c35b958d6..e6f03e76255a2 100644 --- a/x-pack/plugins/infra/public/pages/logs/settings/indices_configuration_panel.tsx +++ b/x-pack/plugins/infra/public/pages/logs/settings/indices_configuration_panel.tsx @@ -16,7 +16,7 @@ import { } from '@elastic/eui'; import { FormattedMessage } from '@kbn/i18n/react'; import React from 'react'; -import { InputFieldProps } from '../../../components/source_configuration'; +import { InputFieldProps } from '../../../components/source_configuration/input_fields'; interface IndicesConfigurationPanelProps { isLoading: boolean; diff --git a/x-pack/plugins/infra/public/pages/logs/stream/page_no_indices_content.tsx b/x-pack/plugins/infra/public/pages/logs/stream/page_no_indices_content.tsx index 5d6ff9544e187..bc3bc22f3f1b2 100644 --- a/x-pack/plugins/infra/public/pages/logs/stream/page_no_indices_content.tsx +++ b/x-pack/plugins/infra/public/pages/logs/stream/page_no_indices_content.tsx @@ -10,7 +10,7 @@ import { i18n } from '@kbn/i18n'; import React from 'react'; import { NoIndices } from '../../../components/empty_states/no_indices'; -import { ViewSourceConfigurationButton } from '../../../components/source_configuration'; +import { ViewSourceConfigurationButton } from '../../../components/source_configuration/view_source_configuration_button'; import { useKibana } from '../../../../../../../src/plugins/kibana_react/public'; import { useLinkProps } from '../../../hooks/use_link_props'; diff --git a/x-pack/plugins/infra/public/pages/metrics/index.tsx b/x-pack/plugins/infra/public/pages/metrics/index.tsx index 240cb778275b1..51cc4ca098483 100644 --- a/x-pack/plugins/infra/public/pages/metrics/index.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/index.tsx @@ -12,7 +12,7 @@ import { Route, RouteComponentProps, Switch } from 'react-router-dom'; import { EuiErrorBoundary, EuiFlexItem, EuiFlexGroup, EuiButtonEmpty } from '@elastic/eui'; import { IIndexPattern } from 'src/plugins/data/common'; -import { InfraSourceConfiguration } from '../../../common/http_api/source_api'; +import { MetricsSourceConfigurationProperties } from '../../../common/metrics_sources'; import { DocumentTitle } from '../../components/document_title'; import { HelpCenterContent } from '../../components/help_center_content'; import { RoutedTabs } from '../../components/navigation/routed_tabs'; @@ -24,7 +24,7 @@ import { } from './metrics_explorer/hooks/use_metrics_explorer_options'; import { WithMetricsExplorerOptionsUrlState } from '../../containers/metrics_explorer/with_metrics_explorer_options_url_state'; import { WithSource } from '../../containers/with_source'; -import { Source } from '../../containers/source'; +import { Source } from '../../containers/metrics_source'; import { MetricsExplorerPage } from './metrics_explorer'; import { SnapshotPage } from './inventory_view'; import { MetricsSettingsPage } from './settings'; @@ -188,8 +188,8 @@ export const InfrastructurePage = ({ match }: RouteComponentProps) => { }; const PageContent = (props: { - configuration: InfraSourceConfiguration; - createDerivedIndexPattern: (type: 'logs' | 'metrics' | 'both') => IIndexPattern; + configuration: MetricsSourceConfigurationProperties; + createDerivedIndexPattern: (type: 'metrics') => IIndexPattern; }) => { const { createDerivedIndexPattern, configuration } = props; const { options } = useContext(MetricsExplorerOptionsContainer.Context); diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/layout.tsx b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/layout.tsx index 089ad9c237818..534132eb75fa1 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/layout.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/layout.tsx @@ -18,7 +18,7 @@ import { useSnapshot } from '../hooks/use_snaphot'; import { useWaffleTimeContext } from '../hooks/use_waffle_time'; import { useWaffleFiltersContext } from '../hooks/use_waffle_filters'; import { DEFAULT_LEGEND, useWaffleOptionsContext } from '../hooks/use_waffle_options'; -import { useSourceContext } from '../../../../containers/source'; +import { useSourceContext } from '../../../../containers/metrics_source'; import { InfraFormatterType } from '../../../../lib/lib'; import { euiStyled } from '../../../../../../../../src/plugins/kibana_react/common'; import { Toolbar } from './toolbars/toolbar'; diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/ml/anomaly_detection/anomalies_table/anomalies_table.tsx b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/ml/anomaly_detection/anomalies_table/anomalies_table.tsx index 7f0424cf48758..409c11cbbe897 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/ml/anomaly_detection/anomalies_table/anomalies_table.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/ml/anomaly_detection/anomalies_table/anomalies_table.tsx @@ -43,7 +43,7 @@ import { import { PaginationControls } from './pagination'; import { AnomalySummary } from './annomaly_summary'; import { AnomalySeverityIndicator } from '../../../../../../../components/logging/log_analysis_results/anomaly_severity_indicator'; -import { useSourceContext } from '../../../../../../../containers/source'; +import { useSourceContext } from '../../../../../../../containers/metrics_source'; import { createResultsUrl } from '../flyout_home'; import { useWaffleViewState, WaffleViewState } from '../../../../hooks/use_waffle_view_state'; type JobType = 'k8s' | 'hosts'; diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/ml/anomaly_detection/anomaly_detection_flyout.tsx b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/ml/anomaly_detection/anomaly_detection_flyout.tsx index 326689e945e1d..387e739fab43f 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/ml/anomaly_detection/anomaly_detection_flyout.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/ml/anomaly_detection/anomaly_detection_flyout.tsx @@ -13,7 +13,7 @@ import { JobSetupScreen } from './job_setup_screen'; import { useInfraMLCapabilities } from '../../../../../../containers/ml/infra_ml_capabilities'; import { MetricHostsModuleProvider } from '../../../../../../containers/ml/modules/metrics_hosts/module'; import { MetricK8sModuleProvider } from '../../../../../../containers/ml/modules/metrics_k8s/module'; -import { useSourceViaHttp } from '../../../../../../containers/source/use_source_via_http'; +import { useSourceViaHttp } from '../../../../../../containers/metrics_source/use_source_via_http'; import { useActiveKibanaSpace } from '../../../../../../hooks/use_kibana_space'; export const AnomalyDetectionFlyout = () => { @@ -23,7 +23,6 @@ export const AnomalyDetectionFlyout = () => { const [screenParams, setScreenParams] = useState(null); const { source } = useSourceViaHttp({ sourceId: 'default', - type: 'metrics', }); const { space } = useActiveKibanaSpace(); diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/ml/anomaly_detection/job_setup_screen.tsx b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/ml/anomaly_detection/job_setup_screen.tsx index 894f76318bcfe..a210831eef865 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/ml/anomaly_detection/job_setup_screen.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/ml/anomaly_detection/job_setup_screen.tsx @@ -17,7 +17,7 @@ import moment, { Moment } from 'moment'; import { EuiComboBox } from '@elastic/eui'; import { i18n } from '@kbn/i18n'; import { EuiLoadingSpinner } from '@elastic/eui'; -import { useSourceViaHttp } from '../../../../../../containers/source/use_source_via_http'; +import { useSourceViaHttp } from '../../../../../../containers/metrics_source/use_source_via_http'; import { useMetricK8sModuleContext } from '../../../../../../containers/ml/modules/metrics_k8s/module'; import { useMetricHostsModuleContext } from '../../../../../../containers/ml/modules/metrics_hosts/module'; import { FixedDatePicker } from '../../../../../../components/fixed_datepicker'; @@ -42,7 +42,6 @@ export const JobSetupScreen = (props: Props) => { const [filterQuery, setFilterQuery] = useState(''); const { createDerivedIndexPattern } = useSourceViaHttp({ sourceId: 'default', - type: 'metrics', }); const indicies = h.sourceConfiguration.indices; @@ -79,7 +78,7 @@ export const JobSetupScreen = (props: Props) => { } }, [props.jobType, k.jobSummaries, h.jobSummaries]); - const derivedIndexPattern = useMemo(() => createDerivedIndexPattern('metrics'), [ + const derivedIndexPattern = useMemo(() => createDerivedIndexPattern(), [ createDerivedIndexPattern, ]); diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/node_details/tabs/metrics/metrics.tsx b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/node_details/tabs/metrics/metrics.tsx index d89aaefe53fd1..5ab8eb380a657 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/node_details/tabs/metrics/metrics.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/node_details/tabs/metrics/metrics.tsx @@ -23,7 +23,7 @@ import { EuiLoadingChart, EuiSpacer, EuiFlexGrid, EuiFlexItem } from '@elastic/e import { TabContent, TabProps } from '../shared'; import { useSnapshot } from '../../../../hooks/use_snaphot'; import { useWaffleOptionsContext } from '../../../../hooks/use_waffle_options'; -import { useSourceContext } from '../../../../../../../containers/source'; +import { useSourceContext } from '../../../../../../../containers/metrics_source'; import { findInventoryFields } from '../../../../../../../../common/inventory_models'; import { convertKueryToElasticSearchQuery } from '../../../../../../../utils/kuery'; import { SnapshotMetricType } from '../../../../../../../../common/inventory_models/types'; diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/node_details/tabs/properties/index.tsx b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/node_details/tabs/properties/index.tsx index 9aa2cdfd90203..010a1a9941335 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/node_details/tabs/properties/index.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/node_details/tabs/properties/index.tsx @@ -9,7 +9,7 @@ import React, { useCallback, useContext, useMemo } from 'react'; import { i18n } from '@kbn/i18n'; import { EuiLoadingChart } from '@elastic/eui'; import { TabContent, TabProps } from '../shared'; -import { Source } from '../../../../../../../containers/source'; +import { Source } from '../../../../../../../containers/metrics_source'; import { findInventoryModel } from '../../../../../../../../common/inventory_models'; import { InventoryItemType } from '../../../../../../../../common/inventory_models/types'; import { useMetadata } from '../../../../../metric_detail/hooks/use_metadata'; diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/search_bar.tsx b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/search_bar.tsx index cae17c174772d..16f73734836d0 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/search_bar.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/search_bar.tsx @@ -7,7 +7,7 @@ import React, { useContext } from 'react'; import { i18n } from '@kbn/i18n'; -import { Source } from '../../../../containers/source'; +import { Source } from '../../../../containers/metrics_source'; import { AutocompleteField } from '../../../../components/autocomplete_field'; import { WithKueryAutocompletion } from '../../../../containers/with_kuery_autocompletion'; import { useWaffleFiltersContext } from '../hooks/use_waffle_filters'; diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/timeline/timeline.tsx b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/timeline/timeline.tsx index 0248241d616dc..0a657b5242427 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/timeline/timeline.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/timeline/timeline.tsx @@ -29,7 +29,7 @@ import { useUiSetting } from '../../../../../../../../../src/plugins/kibana_reac import { toMetricOpt } from '../../../../../../common/snapshot_metric_i18n'; import { MetricsExplorerAggregation } from '../../../../../../common/http_api'; import { colorTransformer, Color } from '../../../../../../common/color_palette'; -import { useSourceContext } from '../../../../../containers/source'; +import { useSourceContext } from '../../../../../containers/metrics_source'; import { useTimeline } from '../../hooks/use_timeline'; import { useWaffleOptionsContext } from '../../hooks/use_waffle_options'; import { useWaffleTimeContext } from '../../hooks/use_waffle_time'; diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/toolbars/toolbar.tsx b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/toolbars/toolbar.tsx index cd05341156831..1c79807f139c3 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/toolbars/toolbar.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/toolbars/toolbar.tsx @@ -7,7 +7,7 @@ import React, { FunctionComponent } from 'react'; import { EuiFlexItem } from '@elastic/eui'; -import { useSourceContext } from '../../../../../containers/source'; +import { useSourceContext } from '../../../../../containers/metrics_source'; import { SnapshotMetricInput, SnapshotGroupBy, @@ -24,7 +24,7 @@ import { WaffleOptionsState, WaffleSortOption } from '../../hooks/use_waffle_opt import { useInventoryMeta } from '../../hooks/use_inventory_meta'; export interface ToolbarProps extends Omit { - createDerivedIndexPattern: (type: 'logs' | 'metrics' | 'both') => IIndexPattern; + createDerivedIndexPattern: (type: 'metrics') => IIndexPattern; changeMetric: (payload: SnapshotMetricInput) => void; changeGroupBy: (payload: SnapshotGroupBy) => void; changeCustomOptions: (payload: InfraGroupByOptions[]) => void; diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/toolbars/toolbar_wrapper.tsx b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/toolbars/toolbar_wrapper.tsx index abc0089e4fc2e..7fc332ead45c7 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/toolbars/toolbar_wrapper.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/toolbars/toolbar_wrapper.tsx @@ -8,7 +8,7 @@ import React from 'react'; import { EuiFlexItem, EuiFlexGroup } from '@elastic/eui'; import { fieldToName } from '../../lib/field_to_display_name'; -import { useSourceContext } from '../../../../../containers/source'; +import { useSourceContext } from '../../../../../containers/metrics_source'; import { useWaffleOptionsContext } from '../../hooks/use_waffle_options'; import { WaffleInventorySwitcher } from '../waffle/waffle_inventory_switcher'; import { ToolbarProps } from './toolbar'; diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/waffle/conditional_tooltip.test.tsx b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/waffle/conditional_tooltip.test.tsx index 523fa5f013b5a..6dde53efae761 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/waffle/conditional_tooltip.test.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/waffle/conditional_tooltip.test.tsx @@ -17,7 +17,7 @@ import { InfraFormatterType, } from '../../../../../lib/lib'; -jest.mock('../../../../../containers/source', () => ({ +jest.mock('../../../../../containers/metrics_source', () => ({ useSourceContext: () => ({ sourceId: 'default' }), })); diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/waffle/conditional_tooltip.tsx b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/waffle/conditional_tooltip.tsx index d0aeeca9850c4..6e334f4fbca75 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/waffle/conditional_tooltip.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/components/waffle/conditional_tooltip.tsx @@ -11,7 +11,7 @@ import { first } from 'lodash'; import { getCustomMetricLabel } from '../../../../../../common/formatters/get_custom_metric_label'; import { SnapshotCustomMetricInput } from '../../../../../../common/http_api'; import { withTheme, EuiTheme } from '../../../../../../../../../src/plugins/kibana_react/common'; -import { useSourceContext } from '../../../../../containers/source'; +import { useSourceContext } from '../../../../../containers/metrics_source'; import { findInventoryModel } from '../../../../../../common/inventory_models'; import { InventoryItemType, diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/hooks/use_process_list.ts b/x-pack/plugins/infra/public/pages/metrics/inventory_view/hooks/use_process_list.ts index d12bef2f3cdc0..e74abb2ecc459 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/hooks/use_process_list.ts +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/hooks/use_process_list.ts @@ -13,7 +13,7 @@ import { useEffect, useState } from 'react'; import { ProcessListAPIResponse, ProcessListAPIResponseRT } from '../../../../../common/http_api'; import { throwErrors, createPlainError } from '../../../../../common/runtime_types'; import { useHTTPRequest } from '../../../../hooks/use_http_request'; -import { useSourceContext } from '../../../../containers/source'; +import { useSourceContext } from '../../../../containers/metrics_source'; export interface SortBy { name: string; diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/hooks/use_waffle_filters.test.ts b/x-pack/plugins/infra/public/pages/metrics/inventory_view/hooks/use_waffle_filters.test.ts index 8d7e516d50b57..cc1108cb91e6d 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/hooks/use_waffle_filters.test.ts +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/hooks/use_waffle_filters.test.ts @@ -17,7 +17,7 @@ jest.mock('react-router-dom', () => ({ }), })); -jest.mock('../../../../containers/source', () => ({ +jest.mock('../../../../containers/metrics_source', () => ({ useSourceContext: () => ({ createDerivedIndexPattern: () => 'jestbeat-*', }), diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/hooks/use_waffle_filters.ts b/x-pack/plugins/infra/public/pages/metrics/inventory_view/hooks/use_waffle_filters.ts index 30c15410e1199..90cf96330e758 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/hooks/use_waffle_filters.ts +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/hooks/use_waffle_filters.ts @@ -13,7 +13,7 @@ import { constant, identity } from 'fp-ts/lib/function'; import createContainter from 'constate'; import { useAlertPrefillContext } from '../../../../alerting/use_alert_prefill'; import { useUrlState } from '../../../../utils/use_url_state'; -import { useSourceContext } from '../../../../containers/source'; +import { useSourceContext } from '../../../../containers/metrics_source'; import { convertKueryToElasticSearchQuery } from '../../../../utils/kuery'; import { esKuery } from '../../../../../../../../src/plugins/data/public'; diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/index.tsx b/x-pack/plugins/infra/public/pages/metrics/inventory_view/index.tsx index 6b980d33c2559..57073fee13c18 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/index.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/index.tsx @@ -17,8 +17,8 @@ import { ColumnarPage } from '../../../components/page'; import { SourceErrorPage } from '../../../components/source_error_page'; import { SourceLoadingPage } from '../../../components/source_loading_page'; -import { ViewSourceConfigurationButton } from '../../../components/source_configuration'; -import { Source } from '../../../containers/source'; +import { ViewSourceConfigurationButton } from '../../../components/source_configuration/view_source_configuration_button'; +import { Source } from '../../../containers/metrics_source'; import { useTrackPageview } from '../../../../../observability/public'; import { useKibana } from '../../../../../../../src/plugins/kibana_react/public'; import { Layout } from './components/layout'; diff --git a/x-pack/plugins/infra/public/pages/metrics/inventory_view/lib/create_uptime_link.test.ts b/x-pack/plugins/infra/public/pages/metrics/inventory_view/lib/create_uptime_link.test.ts index 1e315f95dbd7c..dbe45a387891c 100644 --- a/x-pack/plugins/infra/public/pages/metrics/inventory_view/lib/create_uptime_link.test.ts +++ b/x-pack/plugins/infra/public/pages/metrics/inventory_view/lib/create_uptime_link.test.ts @@ -14,7 +14,6 @@ const options: InfraWaffleMapOptions = { container: 'container.id', pod: 'kubernetes.pod.uid', host: 'host.name', - message: ['@message'], timestamp: '@timestanp', tiebreaker: '@timestamp', }, diff --git a/x-pack/plugins/infra/public/pages/metrics/metric_detail/components/invalid_node.tsx b/x-pack/plugins/infra/public/pages/metrics/metric_detail/components/invalid_node.tsx index 6b9912346f396..2a436eac30b2c 100644 --- a/x-pack/plugins/infra/public/pages/metrics/metric_detail/components/invalid_node.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/metric_detail/components/invalid_node.tsx @@ -9,7 +9,7 @@ import { EuiButton, EuiEmptyPrompt, EuiFlexGroup, EuiFlexItem } from '@elastic/e import { FormattedMessage } from '@kbn/i18n/react'; import React from 'react'; import { euiStyled } from '../../../../../../../../src/plugins/kibana_react/common'; -import { ViewSourceConfigurationButton } from '../../../../components/source_configuration'; +import { ViewSourceConfigurationButton } from '../../../../components/source_configuration/view_source_configuration_button'; import { useLinkProps } from '../../../../hooks/use_link_props'; interface InvalidNodeErrorProps { diff --git a/x-pack/plugins/infra/public/pages/metrics/metric_detail/index.tsx b/x-pack/plugins/infra/public/pages/metrics/metric_detail/index.tsx index d174707d8b6c9..13fa5cf1f0667 100644 --- a/x-pack/plugins/infra/public/pages/metrics/metric_detail/index.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/metric_detail/index.tsx @@ -17,7 +17,7 @@ import { Header } from '../../../components/header'; import { ColumnarPage, PageContent } from '../../../components/page'; import { withMetricPageProviders } from './page_providers'; import { useMetadata } from './hooks/use_metadata'; -import { Source } from '../../../containers/source'; +import { Source } from '../../../containers/metrics_source'; import { InfraLoadingPanel } from '../../../components/loading'; import { findInventoryModel } from '../../../../common/inventory_models'; import { NavItem } from './lib/side_nav_context'; diff --git a/x-pack/plugins/infra/public/pages/metrics/metric_detail/page_providers.tsx b/x-pack/plugins/infra/public/pages/metrics/metric_detail/page_providers.tsx index ac90e488cea94..c4e1b6bf8ef16 100644 --- a/x-pack/plugins/infra/public/pages/metrics/metric_detail/page_providers.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/metric_detail/page_providers.tsx @@ -7,7 +7,7 @@ import { EuiErrorBoundary } from '@elastic/eui'; import React from 'react'; -import { Source } from '../../../containers/source'; +import { Source } from '../../../containers/metrics_source'; import { MetricsTimeProvider } from './hooks/use_metrics_time'; export const withMetricPageProviders = (Component: React.ComponentType) => ( diff --git a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/chart.tsx b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/chart.tsx index 442382010d78c..35265f0a462cf 100644 --- a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/chart.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/chart.tsx @@ -19,7 +19,7 @@ import { } from '@elastic/charts'; import { first, last } from 'lodash'; import moment from 'moment'; -import { InfraSourceConfiguration } from '../../../../../common/http_api/source_api'; +import { MetricsSourceConfigurationProperties } from '../../../../../common/metrics_sources'; import { MetricsExplorerSeries } from '../../../../../common/http_api/metrics_explorer'; import { MetricsExplorerOptions, @@ -47,7 +47,7 @@ interface Props { options: MetricsExplorerOptions; chartOptions: MetricsExplorerChartOptions; series: MetricsExplorerSeries; - source: InfraSourceConfiguration | undefined; + source: MetricsSourceConfigurationProperties | undefined; timeRange: MetricsExplorerTimeOptions; onTimeChange: (start: string, end: string) => void; } diff --git a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/chart_context_menu.tsx b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/chart_context_menu.tsx index f5970cffa157d..8f281bda0229d 100644 --- a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/chart_context_menu.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/chart_context_menu.tsx @@ -16,7 +16,7 @@ import { } from '@elastic/eui'; import DateMath from '@elastic/datemath'; import { Capabilities } from 'src/core/public'; -import { InfraSourceConfiguration } from '../../../../../common/http_api/source_api'; +import { MetricsSourceConfigurationProperties } from '../../../../../common/metrics_sources'; import { AlertFlyout } from '../../../../alerting/metric_threshold/components/alert_flyout'; import { MetricsExplorerSeries } from '../../../../../common/http_api/metrics_explorer'; import { @@ -33,14 +33,14 @@ export interface Props { options: MetricsExplorerOptions; onFilter?: (query: string) => void; series: MetricsExplorerSeries; - source?: InfraSourceConfiguration; + source?: MetricsSourceConfigurationProperties; timeRange: MetricsExplorerTimeOptions; uiCapabilities?: Capabilities; chartOptions: MetricsExplorerChartOptions; } const fieldToNodeType = ( - source: InfraSourceConfiguration, + source: MetricsSourceConfigurationProperties, groupBy: string | string[] ): InventoryItemType | undefined => { const fields = Array.isArray(groupBy) ? groupBy : [groupBy]; diff --git a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/charts.tsx b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/charts.tsx index e2e64a6758a29..68faaf1f45145 100644 --- a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/charts.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/charts.tsx @@ -9,7 +9,7 @@ import { EuiButton, EuiFlexGrid, EuiFlexItem, EuiText, EuiHorizontalRule } from import { i18n } from '@kbn/i18n'; import { FormattedMessage } from '@kbn/i18n/react'; import React from 'react'; -import { InfraSourceConfiguration } from '../../../../../common/http_api/source_api'; +import { MetricsSourceConfigurationProperties } from '../../../../../common/metrics_sources'; import { MetricsExplorerResponse } from '../../../../../common/http_api/metrics_explorer'; import { MetricsExplorerOptions, @@ -31,7 +31,7 @@ interface Props { onFilter: (filter: string) => void; onTimeChange: (start: string, end: string) => void; data: MetricsExplorerResponse | null; - source: InfraSourceConfiguration | undefined; + source: MetricsSourceConfigurationProperties | undefined; timeRange: MetricsExplorerTimeOptions; } export const MetricsExplorerCharts = ({ diff --git a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/helpers/create_tsvb_link.ts b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/helpers/create_tsvb_link.ts index d2eeada219fa4..1a549041823ec 100644 --- a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/helpers/create_tsvb_link.ts +++ b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/components/helpers/create_tsvb_link.ts @@ -8,7 +8,7 @@ import { encode } from 'rison-node'; import uuid from 'uuid'; import { set } from '@elastic/safer-lodash-set'; -import { InfraSourceConfiguration } from '../../../../../../common/http_api/source_api'; +import { MetricsSourceConfigurationProperties } from '../../../../../../common/metrics_sources'; import { colorTransformer, Color } from '../../../../../../common/color_palette'; import { MetricsExplorerSeries } from '../../../../../../common/http_api/metrics_explorer'; import { @@ -143,7 +143,7 @@ const createTSVBIndexPattern = (alias: string) => { }; export const createTSVBLink = ( - source: InfraSourceConfiguration | undefined, + source: MetricsSourceConfigurationProperties | undefined, options: MetricsExplorerOptions, series: MetricsExplorerSeries, timeRange: MetricsExplorerTimeOptions, diff --git a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/hooks/use_metric_explorer_state.ts b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/hooks/use_metric_explorer_state.ts index eb5a4633d4fa9..a304c81ca1298 100644 --- a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/hooks/use_metric_explorer_state.ts +++ b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/hooks/use_metric_explorer_state.ts @@ -7,7 +7,7 @@ import { useState, useCallback, useContext } from 'react'; import { IIndexPattern } from 'src/plugins/data/public'; -import { InfraSourceConfiguration } from '../../../../../common/http_api/source_api'; +import { MetricsSourceConfigurationProperties } from '../../../../../common/metrics_sources'; import { MetricsExplorerMetric, MetricsExplorerAggregation, @@ -28,7 +28,7 @@ export interface MetricExplorerViewState { } export const useMetricsExplorerState = ( - source: InfraSourceConfiguration, + source: MetricsSourceConfigurationProperties, derivedIndexPattern: IIndexPattern, shouldLoadImmediately = true ) => { diff --git a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/hooks/use_metrics_explorer_data.test.tsx b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/hooks/use_metrics_explorer_data.test.tsx index 3d09a907be12f..9a5e5fcf39ce4 100644 --- a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/hooks/use_metrics_explorer_data.test.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/hooks/use_metrics_explorer_data.test.tsx @@ -22,7 +22,7 @@ import { import { MetricsExplorerOptions, MetricsExplorerTimeOptions } from './use_metrics_explorer_options'; import { IIndexPattern } from '../../../../../../../../src/plugins/data/public'; import { HttpHandler } from 'kibana/public'; -import { InfraSourceConfiguration } from '../../../../../common/http_api/source_api'; +import { MetricsSourceConfigurationProperties } from '../../../../../common/metrics_sources'; const mockedFetch = jest.fn(); @@ -38,7 +38,7 @@ const renderUseMetricsExplorerDataHook = () => { return renderHook( (props: { options: MetricsExplorerOptions; - source: InfraSourceConfiguration | undefined; + source: MetricsSourceConfigurationProperties | undefined; derivedIndexPattern: IIndexPattern; timeRange: MetricsExplorerTimeOptions; afterKey: string | null | Record; diff --git a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/hooks/use_metrics_explorer_data.ts b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/hooks/use_metrics_explorer_data.ts index b6620e963217d..6689aedcd7209 100644 --- a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/hooks/use_metrics_explorer_data.ts +++ b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/hooks/use_metrics_explorer_data.ts @@ -9,7 +9,7 @@ import DateMath from '@elastic/datemath'; import { isEqual } from 'lodash'; import { useEffect, useState, useCallback } from 'react'; import { IIndexPattern } from 'src/plugins/data/public'; -import { InfraSourceConfiguration } from '../../../../../common/http_api/source_api'; +import { MetricsSourceConfigurationProperties } from '../../../../../common/metrics_sources'; import { MetricsExplorerResponse, metricsExplorerResponseRT, @@ -25,7 +25,7 @@ function isSameOptions(current: MetricsExplorerOptions, next: MetricsExplorerOpt export function useMetricsExplorerData( options: MetricsExplorerOptions, - source: InfraSourceConfiguration | undefined, + source: MetricsSourceConfigurationProperties | undefined, derivedIndexPattern: IIndexPattern, timerange: MetricsExplorerTimeOptions, afterKey: string | null | Record, diff --git a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/index.tsx b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/index.tsx index 3eb9bbacddd2e..0d1ac47812577 100644 --- a/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/index.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/metrics_explorer/index.tsx @@ -9,7 +9,7 @@ import { EuiErrorBoundary } from '@elastic/eui'; import { i18n } from '@kbn/i18n'; import React, { useEffect } from 'react'; import { IIndexPattern } from 'src/plugins/data/public'; -import { InfraSourceConfiguration } from '../../../../common/http_api/source_api'; +import { MetricsSourceConfigurationProperties } from '../../../../common/metrics_sources'; import { useTrackPageview } from '../../../../../observability/public'; import { DocumentTitle } from '../../../components/document_title'; import { NoData } from '../../../components/empty_states'; @@ -19,7 +19,7 @@ import { useMetricsExplorerState } from './hooks/use_metric_explorer_state'; import { useSavedViewContext } from '../../../containers/saved_view/saved_view'; interface MetricsExplorerPageProps { - source: InfraSourceConfiguration; + source: MetricsSourceConfigurationProperties; derivedIndexPattern: IIndexPattern; } diff --git a/x-pack/plugins/infra/public/pages/metrics/settings.tsx b/x-pack/plugins/infra/public/pages/metrics/settings.tsx index c9be4abcf9e5f..c54725ab39754 100644 --- a/x-pack/plugins/infra/public/pages/metrics/settings.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/settings.tsx @@ -8,7 +8,7 @@ import { EuiErrorBoundary } from '@elastic/eui'; import React from 'react'; import { useKibana } from '../../../../../../src/plugins/kibana_react/public'; -import { SourceConfigurationSettings } from '../../components/source_configuration/source_configuration_settings'; +import { SourceConfigurationSettings } from './settings/source_configuration_settings'; export const MetricsSettingsPage = () => { const uiCapabilities = useKibana().services.application?.capabilities; diff --git a/x-pack/plugins/infra/public/components/source_configuration/fields_configuration_panel.tsx b/x-pack/plugins/infra/public/pages/metrics/settings/fields_configuration_panel.tsx similarity index 98% rename from x-pack/plugins/infra/public/components/source_configuration/fields_configuration_panel.tsx rename to x-pack/plugins/infra/public/pages/metrics/settings/fields_configuration_panel.tsx index 2a8abdbc04f8e..7026f372ec7ff 100644 --- a/x-pack/plugins/infra/public/components/source_configuration/fields_configuration_panel.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/settings/fields_configuration_panel.tsx @@ -20,7 +20,7 @@ import { i18n } from '@kbn/i18n'; import { FormattedMessage } from '@kbn/i18n/react'; import React from 'react'; -import { InputFieldProps } from './input_fields'; +import { InputFieldProps } from '../../../components/source_configuration/input_fields'; interface FieldsConfigurationPanelProps { containerFieldProps: InputFieldProps; diff --git a/x-pack/plugins/infra/public/components/source_configuration/indices_configuration_form_state.ts b/x-pack/plugins/infra/public/pages/metrics/settings/indices_configuration_form_state.ts similarity index 91% rename from x-pack/plugins/infra/public/components/source_configuration/indices_configuration_form_state.ts rename to x-pack/plugins/infra/public/pages/metrics/settings/indices_configuration_form_state.ts index b4dede79d11f2..ad26c1b13b0e1 100644 --- a/x-pack/plugins/infra/public/components/source_configuration/indices_configuration_form_state.ts +++ b/x-pack/plugins/infra/public/pages/metrics/settings/indices_configuration_form_state.ts @@ -11,16 +11,14 @@ import { createInputFieldProps, createInputRangeFieldProps, validateInputFieldNotEmpty, -} from './input_fields'; +} from '../../../components/source_configuration/input_fields'; interface FormState { name: string; description: string; metricAlias: string; - logAlias: string; containerField: string; hostField: string; - messageField: string[]; podField: string; tiebreakerField: string; timestampField: string; @@ -56,16 +54,6 @@ export const useIndicesConfigurationFormState = ({ }), [formState.name] ); - const logAliasFieldProps = useMemo( - () => - createInputFieldProps({ - errors: validateInputFieldNotEmpty(formState.logAlias), - name: 'logAlias', - onChange: (logAlias) => setFormStateChanges((changes) => ({ ...changes, logAlias })), - value: formState.logAlias, - }), - [formState.logAlias] - ); const metricAliasFieldProps = useMemo( () => createInputFieldProps({ @@ -144,7 +132,6 @@ export const useIndicesConfigurationFormState = ({ const fieldProps = useMemo( () => ({ name: nameFieldProps, - logAlias: logAliasFieldProps, metricAlias: metricAliasFieldProps, containerField: containerFieldFieldProps, hostField: hostFieldFieldProps, @@ -155,7 +142,6 @@ export const useIndicesConfigurationFormState = ({ }), [ nameFieldProps, - logAliasFieldProps, metricAliasFieldProps, containerFieldFieldProps, hostFieldFieldProps, @@ -193,11 +179,9 @@ export const useIndicesConfigurationFormState = ({ const defaultFormState: FormState = { name: '', description: '', - logAlias: '', metricAlias: '', containerField: '', hostField: '', - messageField: [], podField: '', tiebreakerField: '', timestampField: '', diff --git a/x-pack/plugins/infra/public/components/source_configuration/indices_configuration_panel.tsx b/x-pack/plugins/infra/public/pages/metrics/settings/indices_configuration_panel.tsx similarity index 93% rename from x-pack/plugins/infra/public/components/source_configuration/indices_configuration_panel.tsx rename to x-pack/plugins/infra/public/pages/metrics/settings/indices_configuration_panel.tsx index cff9b78777aa3..c64ab2b0e9df5 100644 --- a/x-pack/plugins/infra/public/components/source_configuration/indices_configuration_panel.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/settings/indices_configuration_panel.tsx @@ -17,8 +17,8 @@ import { import { FormattedMessage } from '@kbn/i18n/react'; import React from 'react'; -import { METRICS_INDEX_PATTERN } from '../../../common/constants'; -import { InputFieldProps } from './input_fields'; +import { METRICS_INDEX_PATTERN } from '../../../../common/constants'; +import { InputFieldProps } from '../../../components/source_configuration/input_fields'; interface IndicesConfigurationPanelProps { isLoading: boolean; diff --git a/x-pack/plugins/infra/public/components/source_configuration/ml_configuration_panel.tsx b/x-pack/plugins/infra/public/pages/metrics/settings/ml_configuration_panel.tsx similarity index 96% rename from x-pack/plugins/infra/public/components/source_configuration/ml_configuration_panel.tsx rename to x-pack/plugins/infra/public/pages/metrics/settings/ml_configuration_panel.tsx index 3bd498d460391..abf25dde0ea99 100644 --- a/x-pack/plugins/infra/public/components/source_configuration/ml_configuration_panel.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/settings/ml_configuration_panel.tsx @@ -13,7 +13,7 @@ import { EuiDescribedFormGroup } from '@elastic/eui'; import { EuiForm } from '@elastic/eui'; import React from 'react'; import { FormattedMessage } from 'react-intl'; -import { InputRangeFieldProps } from './input_fields'; +import { InputRangeFieldProps } from '../../../components/source_configuration/input_fields'; interface MLConfigurationPanelProps { isLoading: boolean; diff --git a/x-pack/plugins/infra/public/components/source_configuration/source_configuration_form_state.tsx b/x-pack/plugins/infra/public/pages/metrics/settings/source_configuration_form_state.tsx similarity index 57% rename from x-pack/plugins/infra/public/components/source_configuration/source_configuration_form_state.tsx rename to x-pack/plugins/infra/public/pages/metrics/settings/source_configuration_form_state.tsx index c80235137eea6..37da4bd1aa1bd 100644 --- a/x-pack/plugins/infra/public/components/source_configuration/source_configuration_form_state.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/settings/source_configuration_form_state.tsx @@ -6,12 +6,12 @@ */ import { useCallback, useMemo } from 'react'; -import { InfraSourceConfiguration } from '../../../common/http_api/source_api'; - +import { MetricsSourceConfigurationProperties } from '../../../../common/metrics_sources'; import { useIndicesConfigurationFormState } from './indices_configuration_form_state'; -import { useLogColumnsConfigurationFormState } from './log_columns_configuration_form_state'; -export const useSourceConfigurationFormState = (configuration?: InfraSourceConfiguration) => { +export const useSourceConfigurationFormState = ( + configuration?: MetricsSourceConfigurationProperties +) => { const indicesConfigurationFormState = useIndicesConfigurationFormState({ initialFormState: useMemo( () => @@ -19,11 +19,9 @@ export const useSourceConfigurationFormState = (configuration?: InfraSourceConfi ? { name: configuration.name, description: configuration.description, - logAlias: configuration.logAlias, metricAlias: configuration.metricAlias, containerField: configuration.fields.container, hostField: configuration.fields.host, - messageField: configuration.fields.message, podField: configuration.fields.pod, tiebreakerField: configuration.fields.tiebreaker, timestampField: configuration.fields.timestamp, @@ -34,43 +32,26 @@ export const useSourceConfigurationFormState = (configuration?: InfraSourceConfi ), }); - const logColumnsConfigurationFormState = useLogColumnsConfigurationFormState({ - initialFormState: useMemo( - () => - configuration - ? { - logColumns: configuration.logColumns, - } - : undefined, - [configuration] - ), - }); - - const errors = useMemo( - () => [...indicesConfigurationFormState.errors, ...logColumnsConfigurationFormState.errors], - [indicesConfigurationFormState.errors, logColumnsConfigurationFormState.errors] - ); + const errors = useMemo(() => [...indicesConfigurationFormState.errors], [ + indicesConfigurationFormState.errors, + ]); const resetForm = useCallback(() => { indicesConfigurationFormState.resetForm(); - logColumnsConfigurationFormState.resetForm(); - }, [indicesConfigurationFormState, logColumnsConfigurationFormState]); + }, [indicesConfigurationFormState]); - const isFormDirty = useMemo( - () => indicesConfigurationFormState.isFormDirty || logColumnsConfigurationFormState.isFormDirty, - [indicesConfigurationFormState.isFormDirty, logColumnsConfigurationFormState.isFormDirty] - ); + const isFormDirty = useMemo(() => indicesConfigurationFormState.isFormDirty, [ + indicesConfigurationFormState.isFormDirty, + ]); - const isFormValid = useMemo( - () => indicesConfigurationFormState.isFormValid && logColumnsConfigurationFormState.isFormValid, - [indicesConfigurationFormState.isFormValid, logColumnsConfigurationFormState.isFormValid] - ); + const isFormValid = useMemo(() => indicesConfigurationFormState.isFormValid, [ + indicesConfigurationFormState.isFormValid, + ]); const formState = useMemo( () => ({ name: indicesConfigurationFormState.formState.name, description: indicesConfigurationFormState.formState.description, - logAlias: indicesConfigurationFormState.formState.logAlias, metricAlias: indicesConfigurationFormState.formState.metricAlias, fields: { container: indicesConfigurationFormState.formState.containerField, @@ -79,17 +60,15 @@ export const useSourceConfigurationFormState = (configuration?: InfraSourceConfi tiebreaker: indicesConfigurationFormState.formState.tiebreakerField, timestamp: indicesConfigurationFormState.formState.timestampField, }, - logColumns: logColumnsConfigurationFormState.formState.logColumns, anomalyThreshold: indicesConfigurationFormState.formState.anomalyThreshold, }), - [indicesConfigurationFormState.formState, logColumnsConfigurationFormState.formState] + [indicesConfigurationFormState.formState] ); const formStateChanges = useMemo( () => ({ name: indicesConfigurationFormState.formStateChanges.name, description: indicesConfigurationFormState.formStateChanges.description, - logAlias: indicesConfigurationFormState.formStateChanges.logAlias, metricAlias: indicesConfigurationFormState.formStateChanges.metricAlias, fields: { container: indicesConfigurationFormState.formStateChanges.containerField, @@ -98,25 +77,18 @@ export const useSourceConfigurationFormState = (configuration?: InfraSourceConfi tiebreaker: indicesConfigurationFormState.formStateChanges.tiebreakerField, timestamp: indicesConfigurationFormState.formStateChanges.timestampField, }, - logColumns: logColumnsConfigurationFormState.formStateChanges.logColumns, anomalyThreshold: indicesConfigurationFormState.formStateChanges.anomalyThreshold, }), - [ - indicesConfigurationFormState.formStateChanges, - logColumnsConfigurationFormState.formStateChanges, - ] + [indicesConfigurationFormState.formStateChanges] ); return { - addLogColumn: logColumnsConfigurationFormState.addLogColumn, - moveLogColumn: logColumnsConfigurationFormState.moveLogColumn, errors, formState, formStateChanges, isFormDirty, isFormValid, indicesConfigurationProps: indicesConfigurationFormState.fieldProps, - logColumnConfigurationProps: logColumnsConfigurationFormState.logColumnConfigurationProps, resetForm, }; }; diff --git a/x-pack/plugins/infra/public/components/source_configuration/source_configuration_settings.tsx b/x-pack/plugins/infra/public/pages/metrics/settings/source_configuration_settings.tsx similarity index 94% rename from x-pack/plugins/infra/public/components/source_configuration/source_configuration_settings.tsx rename to x-pack/plugins/infra/public/pages/metrics/settings/source_configuration_settings.tsx index e63f43470497d..71fa4e7600503 100644 --- a/x-pack/plugins/infra/public/components/source_configuration/source_configuration_settings.tsx +++ b/x-pack/plugins/infra/public/pages/metrics/settings/source_configuration_settings.tsx @@ -19,15 +19,15 @@ import { i18n } from '@kbn/i18n'; import { FormattedMessage } from '@kbn/i18n/react'; import React, { useCallback, useContext, useMemo } from 'react'; -import { Source } from '../../containers/source'; +import { Source } from '../../../containers/metrics_source'; import { FieldsConfigurationPanel } from './fields_configuration_panel'; import { IndicesConfigurationPanel } from './indices_configuration_panel'; -import { NameConfigurationPanel } from './name_configuration_panel'; +import { NameConfigurationPanel } from '../../../components/source_configuration/name_configuration_panel'; import { useSourceConfigurationFormState } from './source_configuration_form_state'; -import { SourceLoadingPage } from '../source_loading_page'; -import { Prompt } from '../../utils/navigation_warning_prompt'; +import { SourceLoadingPage } from '../../../components/source_loading_page'; +import { Prompt } from '../../../utils/navigation_warning_prompt'; import { MLConfigurationPanel } from './ml_configuration_panel'; -import { useInfraMLCapabilitiesContext } from '../../containers/ml/infra_ml_capabilities'; +import { useInfraMLCapabilitiesContext } from '../../../containers/ml/infra_ml_capabilities'; interface SourceConfigurationSettingsProps { shouldAllowEdit: boolean; diff --git a/x-pack/plugins/infra/public/utils/source_configuration.ts b/x-pack/plugins/infra/public/utils/source_configuration.ts index b7b45d1927711..a3e1741c7590b 100644 --- a/x-pack/plugins/infra/public/utils/source_configuration.ts +++ b/x-pack/plugins/infra/public/utils/source_configuration.ts @@ -6,14 +6,14 @@ */ import { - InfraSavedSourceConfigurationColumn, - InfraSavedSourceConfigurationFields, + InfraSourceConfigurationColumn, + InfraSourceConfigurationFieldColumn, InfraSourceConfigurationMessageColumn, InfraSourceConfigurationTimestampColumn, -} from '../../common/http_api/source_api'; +} from '../../common/source_configuration/source_configuration'; -export type LogColumnConfiguration = InfraSavedSourceConfigurationColumn; -export type FieldLogColumnConfiguration = InfraSavedSourceConfigurationFields; +export type LogColumnConfiguration = InfraSourceConfigurationColumn; +export type FieldLogColumnConfiguration = InfraSourceConfigurationFieldColumn; export type MessageLogColumnConfiguration = InfraSourceConfigurationMessageColumn; export type TimestampLogColumnConfiguration = InfraSourceConfigurationTimestampColumn; diff --git a/x-pack/plugins/infra/server/infra_server.ts b/x-pack/plugins/infra/server/infra_server.ts index 69595c90c7911..f42207e0ad142 100644 --- a/x-pack/plugins/infra/server/infra_server.ts +++ b/x-pack/plugins/infra/server/infra_server.ts @@ -32,7 +32,7 @@ import { } from './routes/log_entries'; import { initInventoryMetaRoute } from './routes/inventory_metadata'; import { initLogSourceConfigurationRoutes, initLogSourceStatusRoutes } from './routes/log_sources'; -import { initSourceRoute } from './routes/source'; +import { initMetricsSourceConfigurationRoutes } from './routes/metrics_sources'; import { initOverviewRoute } from './routes/overview'; import { initAlertPreviewRoute } from './routes/alerting'; import { initGetLogAlertsChartPreviewDataRoute } from './routes/log_alerts'; @@ -50,7 +50,7 @@ export const initInfraServer = (libs: InfraBackendLibs) => { initGetHostsAnomaliesRoute(libs); initSnapshotRoute(libs); initNodeDetailsRoute(libs); - initSourceRoute(libs); + initMetricsSourceConfigurationRoutes(libs); initValidateLogAnalysisDatasetsRoute(libs); initValidateLogAnalysisIndicesRoute(libs); initGetLogEntryExamplesRoute(libs); diff --git a/x-pack/plugins/infra/server/lib/adapters/metrics/kibana_metrics_adapter.ts b/x-pack/plugins/infra/server/lib/adapters/metrics/kibana_metrics_adapter.ts index e390d6525cd60..921634361f4a2 100644 --- a/x-pack/plugins/infra/server/lib/adapters/metrics/kibana_metrics_adapter.ts +++ b/x-pack/plugins/infra/server/lib/adapters/metrics/kibana_metrics_adapter.ts @@ -34,7 +34,7 @@ export class KibanaMetricsAdapter implements InfraMetricsAdapter { options: InfraMetricsRequestOptions, rawRequest: KibanaRequest ): Promise { - const indexPattern = `${options.sourceConfiguration.metricAlias},${options.sourceConfiguration.logAlias}`; + const indexPattern = `${options.sourceConfiguration.metricAlias}`; const fields = findInventoryFields(options.nodeType, options.sourceConfiguration.fields); const nodeField = fields.id; @@ -112,7 +112,7 @@ export class KibanaMetricsAdapter implements InfraMetricsAdapter { ); } - const indexPattern = `${options.sourceConfiguration.metricAlias},${options.sourceConfiguration.logAlias}`; + const indexPattern = `${options.sourceConfiguration.metricAlias}`; const timerange = { min: options.timerange.from, max: options.timerange.to, @@ -132,7 +132,7 @@ export class KibanaMetricsAdapter implements InfraMetricsAdapter { const calculatedInterval = await calculateMetricInterval( client, { - indexPattern: `${options.sourceConfiguration.logAlias},${options.sourceConfiguration.metricAlias}`, + indexPattern: `${options.sourceConfiguration.metricAlias}`, timestampField: options.sourceConfiguration.fields.timestamp, timerange: options.timerange, }, diff --git a/x-pack/plugins/infra/server/lib/alerting/inventory_metric_threshold/evaluate_condition.ts b/x-pack/plugins/infra/server/lib/alerting/inventory_metric_threshold/evaluate_condition.ts index 439764f80186e..5244b8a81e75f 100644 --- a/x-pack/plugins/infra/server/lib/alerting/inventory_metric_threshold/evaluate_condition.ts +++ b/x-pack/plugins/infra/server/lib/alerting/inventory_metric_threshold/evaluate_condition.ts @@ -23,6 +23,7 @@ import { InfraTimerangeInput, SnapshotRequest } from '../../../../common/http_ap import { InfraSource } from '../../sources'; import { UNGROUPED_FACTORY_KEY } from '../common/utils'; import { getNodes } from '../../../routes/snapshot/lib/get_nodes'; +import { LogQueryFields } from '../../../services/log_queries/get_log_query_fields'; type ConditionResult = InventoryMetricConditions & { shouldFire: boolean[]; @@ -36,6 +37,7 @@ export const evaluateCondition = async ( condition: InventoryMetricConditions, nodeType: InventoryItemType, source: InfraSource, + logQueryFields: LogQueryFields, esClient: ElasticsearchClient, filterQuery?: string, lookbackSize?: number @@ -58,6 +60,7 @@ export const evaluateCondition = async ( metric, timerange, source, + logQueryFields, filterQuery, customMetric ); @@ -101,6 +104,7 @@ const getData = async ( metric: SnapshotMetricType, timerange: InfraTimerangeInput, source: InfraSource, + logQueryFields: LogQueryFields, filterQuery?: string, customMetric?: SnapshotCustomMetricInput ) => { @@ -124,7 +128,7 @@ const getData = async ( includeTimeseries: Boolean(timerange.lookbackSize), }; try { - const { nodes } = await getNodes(client, snapshotRequest, source); + const { nodes } = await getNodes(client, snapshotRequest, source, logQueryFields); if (!nodes.length) return { [UNGROUPED_FACTORY_KEY]: null }; // No Data state diff --git a/x-pack/plugins/infra/server/lib/alerting/inventory_metric_threshold/inventory_metric_threshold_executor.ts b/x-pack/plugins/infra/server/lib/alerting/inventory_metric_threshold/inventory_metric_threshold_executor.ts index 632ba9cd6f282..d775a503d1d32 100644 --- a/x-pack/plugins/infra/server/lib/alerting/inventory_metric_threshold/inventory_metric_threshold_executor.ts +++ b/x-pack/plugins/infra/server/lib/alerting/inventory_metric_threshold/inventory_metric_threshold_executor.ts @@ -68,12 +68,18 @@ export const createInventoryMetricThresholdExecutor = (libs: InfraBackendLibs) = sourceId || 'default' ); + const logQueryFields = await libs.getLogQueryFields( + sourceId || 'default', + services.savedObjectsClient + ); + const results = await Promise.all( criteria.map((c) => evaluateCondition( c, nodeType, source, + logQueryFields, services.scopedClusterClient.asCurrentUser, filterQuery ) diff --git a/x-pack/plugins/infra/server/lib/alerting/inventory_metric_threshold/preview_inventory_metric_threshold_alert.ts b/x-pack/plugins/infra/server/lib/alerting/inventory_metric_threshold/preview_inventory_metric_threshold_alert.ts index 472f9d408694c..f254f1e68ae46 100644 --- a/x-pack/plugins/infra/server/lib/alerting/inventory_metric_threshold/preview_inventory_metric_threshold_alert.ts +++ b/x-pack/plugins/infra/server/lib/alerting/inventory_metric_threshold/preview_inventory_metric_threshold_alert.ts @@ -14,10 +14,11 @@ import { isTooManyBucketsPreviewException, } from '../../../../common/alerting/metrics'; import { ElasticsearchClient } from '../../../../../../../src/core/server'; -import { InfraSource } from '../../../../common/http_api/source_api'; +import { InfraSource } from '../../../../common/source_configuration/source_configuration'; import { getIntervalInSeconds } from '../../../utils/get_interval_in_seconds'; import { InventoryItemType } from '../../../../common/inventory_models/types'; import { evaluateCondition } from './evaluate_condition'; +import { LogQueryFields } from '../../../services/log_queries/get_log_query_fields'; interface InventoryMetricThresholdParams { criteria: InventoryMetricConditions[]; @@ -30,6 +31,7 @@ interface PreviewInventoryMetricThresholdAlertParams { esClient: ElasticsearchClient; params: InventoryMetricThresholdParams; source: InfraSource; + logQueryFields: LogQueryFields; lookback: Unit; alertInterval: string; alertThrottle: string; @@ -43,6 +45,7 @@ export const previewInventoryMetricThresholdAlert: ( esClient, params, source, + logQueryFields, lookback, alertInterval, alertThrottle, @@ -68,7 +71,7 @@ export const previewInventoryMetricThresholdAlert: ( try { const results = await Promise.all( criteria.map((c) => - evaluateCondition(c, nodeType, source, esClient, filterQuery, lookbackSize) + evaluateCondition(c, nodeType, source, logQueryFields, esClient, filterQuery, lookbackSize) ) ); diff --git a/x-pack/plugins/infra/server/lib/alerting/metric_threshold/lib/evaluate_alert.ts b/x-pack/plugins/infra/server/lib/alerting/metric_threshold/lib/evaluate_alert.ts index f6214edc5d0ab..87150aa134837 100644 --- a/x-pack/plugins/infra/server/lib/alerting/metric_threshold/lib/evaluate_alert.ts +++ b/x-pack/plugins/infra/server/lib/alerting/metric_threshold/lib/evaluate_alert.ts @@ -11,7 +11,7 @@ import { isTooManyBucketsPreviewException, TOO_MANY_BUCKETS_PREVIEW_EXCEPTION, } from '../../../../../common/alerting/metrics'; -import { InfraSource } from '../../../../../common/http_api/source_api'; +import { InfraSource } from '../../../../../common/source_configuration/source_configuration'; import { InfraDatabaseSearchResponse } from '../../../adapters/framework/adapter_types'; import { createAfterKeyHandler } from '../../../../utils/create_afterkey_handler'; import { getAllCompositeData } from '../../../../utils/get_all_composite_data'; diff --git a/x-pack/plugins/infra/server/lib/alerting/metric_threshold/preview_metric_threshold_alert.ts b/x-pack/plugins/infra/server/lib/alerting/metric_threshold/preview_metric_threshold_alert.ts index 064804b661b74..a4c207f4006d5 100644 --- a/x-pack/plugins/infra/server/lib/alerting/metric_threshold/preview_metric_threshold_alert.ts +++ b/x-pack/plugins/infra/server/lib/alerting/metric_threshold/preview_metric_threshold_alert.ts @@ -12,7 +12,7 @@ import { isTooManyBucketsPreviewException, } from '../../../../common/alerting/metrics'; import { ElasticsearchClient } from '../../../../../../../src/core/server'; -import { InfraSource } from '../../../../common/http_api/source_api'; +import { InfraSource } from '../../../../common/source_configuration/source_configuration'; import { getIntervalInSeconds } from '../../../utils/get_interval_in_seconds'; import { PreviewResult } from '../common/types'; import { MetricExpressionParams } from './types'; diff --git a/x-pack/plugins/infra/server/lib/domains/fields_domain.ts b/x-pack/plugins/infra/server/lib/domains/fields_domain.ts index b653351a34760..d5ffa56987666 100644 --- a/x-pack/plugins/infra/server/lib/domains/fields_domain.ts +++ b/x-pack/plugins/infra/server/lib/domains/fields_domain.ts @@ -18,21 +18,16 @@ export class InfraFieldsDomain { public async getFields( requestContext: InfraPluginRequestHandlerContext, sourceId: string, - indexType: 'LOGS' | 'METRICS' | 'ANY' + indexType: 'LOGS' | 'METRICS' ): Promise { const { configuration } = await this.libs.sources.getSourceConfiguration( requestContext.core.savedObjects.client, sourceId ); - const includeMetricIndices = ['ANY', 'METRICS'].includes(indexType); - const includeLogIndices = ['ANY', 'LOGS'].includes(indexType); const fields = await this.adapter.getIndexFields( requestContext, - [ - ...(includeMetricIndices ? [configuration.metricAlias] : []), - ...(includeLogIndices ? [configuration.logAlias] : []), - ].join(',') + indexType === 'LOGS' ? configuration.logAlias : configuration.metricAlias ); return fields; diff --git a/x-pack/plugins/infra/server/lib/domains/log_entries_domain/log_entries_domain.ts b/x-pack/plugins/infra/server/lib/domains/log_entries_domain/log_entries_domain.ts index e3c42c4dceede..278ae0e086cfc 100644 --- a/x-pack/plugins/infra/server/lib/domains/log_entries_domain/log_entries_domain.ts +++ b/x-pack/plugins/infra/server/lib/domains/log_entries_domain/log_entries_domain.ts @@ -17,7 +17,7 @@ import { LogColumn, LogEntryCursor, LogEntry } from '../../../../common/log_entr import { InfraSourceConfiguration, InfraSources, - SavedSourceConfigurationFieldColumnRuntimeType, + SourceConfigurationFieldColumnRuntimeType, } from '../../sources'; import { getBuiltinRules } from '../../../services/log_entries/message/builtin_rules'; import { @@ -349,7 +349,7 @@ const getRequiredFields = ( ): string[] => { const fieldsFromCustomColumns = configuration.logColumns.reduce( (accumulatedFields, logColumn) => { - if (SavedSourceConfigurationFieldColumnRuntimeType.is(logColumn)) { + if (SourceConfigurationFieldColumnRuntimeType.is(logColumn)) { return [...accumulatedFields, logColumn.fieldColumn.field]; } return accumulatedFields; diff --git a/x-pack/plugins/infra/server/lib/infra_types.ts b/x-pack/plugins/infra/server/lib/infra_types.ts index 65bb5f878b275..08e42279e4939 100644 --- a/x-pack/plugins/infra/server/lib/infra_types.ts +++ b/x-pack/plugins/infra/server/lib/infra_types.ts @@ -5,7 +5,7 @@ * 2.0. */ -import { InfraSourceConfiguration } from '../../common/http_api/source_api'; +import { InfraSourceConfiguration } from '../../common/source_configuration/source_configuration'; import { InfraFieldsDomain } from './domains/fields_domain'; import { InfraLogEntriesDomain } from './domains/log_entries_domain'; import { InfraMetricsDomain } from './domains/metrics_domain'; @@ -13,6 +13,7 @@ import { InfraSources } from './sources'; import { InfraSourceStatus } from './source_status'; import { InfraConfig } from '../plugin'; import { KibanaFramework } from './adapters/framework/kibana_framework_adapter'; +import { GetLogQueryFields } from '../services/log_queries/get_log_query_fields'; export interface InfraDomainLibs { fields: InfraFieldsDomain; @@ -25,6 +26,7 @@ export interface InfraBackendLibs extends InfraDomainLibs { framework: KibanaFramework; sources: InfraSources; sourceStatus: InfraSourceStatus; + getLogQueryFields: GetLogQueryFields; } export interface InfraConfiguration { diff --git a/x-pack/plugins/infra/server/lib/metrics/index.ts b/x-pack/plugins/infra/server/lib/metrics/index.ts index cb89c5a6b1bd3..e436ad2ba0b05 100644 --- a/x-pack/plugins/infra/server/lib/metrics/index.ts +++ b/x-pack/plugins/infra/server/lib/metrics/index.ts @@ -120,5 +120,5 @@ export const query = async ( ThrowReporter.report(HistogramResponseRT.decode(response.aggregations)); } - throw new Error('Elasticsearch responsed with an unrecoginzed format.'); + throw new Error('Elasticsearch responded with an unrecognized format.'); }; diff --git a/x-pack/plugins/infra/server/lib/sources/defaults.ts b/x-pack/plugins/infra/server/lib/sources/defaults.ts index 1b924619a905c..ff6d6a4f5514b 100644 --- a/x-pack/plugins/infra/server/lib/sources/defaults.ts +++ b/x-pack/plugins/infra/server/lib/sources/defaults.ts @@ -10,7 +10,7 @@ import { LOGS_INDEX_PATTERN, TIMESTAMP_FIELD, } from '../../../common/constants'; -import { InfraSourceConfiguration } from '../../../common/http_api/source_api'; +import { InfraSourceConfiguration } from '../../../common/source_configuration/source_configuration'; export const defaultSourceConfiguration: InfraSourceConfiguration = { name: 'Default', diff --git a/x-pack/plugins/infra/server/lib/sources/index.ts b/x-pack/plugins/infra/server/lib/sources/index.ts index 57852f7f3e4e6..27ad665be31a9 100644 --- a/x-pack/plugins/infra/server/lib/sources/index.ts +++ b/x-pack/plugins/infra/server/lib/sources/index.ts @@ -8,4 +8,4 @@ export * from './defaults'; export { infraSourceConfigurationSavedObjectType } from './saved_object_type'; export * from './sources'; -export * from '../../../common/http_api/source_api'; +export * from '../../../common/source_configuration/source_configuration'; diff --git a/x-pack/plugins/infra/server/lib/sources/migrations/7_9_0_add_new_indexing_strategy_index_names.ts b/x-pack/plugins/infra/server/lib/sources/migrations/7_9_0_add_new_indexing_strategy_index_names.ts index dbfe0f81c187a..e71994fe11517 100644 --- a/x-pack/plugins/infra/server/lib/sources/migrations/7_9_0_add_new_indexing_strategy_index_names.ts +++ b/x-pack/plugins/infra/server/lib/sources/migrations/7_9_0_add_new_indexing_strategy_index_names.ts @@ -6,7 +6,7 @@ */ import { SavedObjectMigrationFn } from 'src/core/server'; -import { InfraSourceConfiguration } from '../../../../common/http_api/source_api'; +import { InfraSourceConfiguration } from '../../../../common/source_configuration/source_configuration'; export const addNewIndexingStrategyIndexNames: SavedObjectMigrationFn< InfraSourceConfiguration, diff --git a/x-pack/plugins/infra/server/lib/sources/sources.ts b/x-pack/plugins/infra/server/lib/sources/sources.ts index fe005b04978da..7abbed0a9fbdd 100644 --- a/x-pack/plugins/infra/server/lib/sources/sources.ts +++ b/x-pack/plugins/infra/server/lib/sources/sources.ts @@ -23,7 +23,7 @@ import { SourceConfigurationSavedObjectRuntimeType, StaticSourceConfigurationRuntimeType, InfraSource, -} from '../../../common/http_api/source_api'; +} from '../../../common/source_configuration/source_configuration'; import { InfraConfig } from '../../../server'; interface Libs { diff --git a/x-pack/plugins/infra/server/plugin.ts b/x-pack/plugins/infra/server/plugin.ts index c80e012844c1e..50fec38b9f2df 100644 --- a/x-pack/plugins/infra/server/plugin.ts +++ b/x-pack/plugins/infra/server/plugin.ts @@ -9,7 +9,7 @@ import { Server } from '@hapi/hapi'; import { schema, TypeOf } from '@kbn/config-schema'; import { i18n } from '@kbn/i18n'; import { CoreSetup, PluginInitializerContext, Plugin } from 'src/core/server'; -import { InfraStaticSourceConfiguration } from '../common/http_api/source_api'; +import { InfraStaticSourceConfiguration } from '../common/source_configuration/source_configuration'; import { inventoryViewSavedObjectType } from '../common/saved_objects/inventory_view'; import { metricsExplorerViewSavedObjectType } from '../common/saved_objects/metrics_explorer_view'; import { LOGS_FEATURE, METRICS_FEATURE } from './features'; @@ -30,6 +30,7 @@ import { InfraSourceStatus } from './lib/source_status'; import { LogEntriesService } from './services/log_entries'; import { InfraPluginRequestHandlerContext } from './types'; import { UsageCollector } from './usage/usage_collector'; +import { createGetLogQueryFields } from './services/log_queries/get_log_query_fields'; export const config = { schema: schema.object({ @@ -123,6 +124,7 @@ export class InfraServerPlugin implements Plugin { sources, sourceStatus, ...domainLibs, + getLogQueryFields: createGetLogQueryFields(sources), }; plugins.features.registerKibanaFeature(METRICS_FEATURE); diff --git a/x-pack/plugins/infra/server/routes/alerting/preview.ts b/x-pack/plugins/infra/server/routes/alerting/preview.ts index 6622df1a8333a..4d980834d3a70 100644 --- a/x-pack/plugins/infra/server/routes/alerting/preview.ts +++ b/x-pack/plugins/infra/server/routes/alerting/preview.ts @@ -25,7 +25,11 @@ import { previewMetricAnomalyAlert } from '../../lib/alerting/metric_anomaly/pre import { InfraBackendLibs } from '../../lib/infra_types'; import { assertHasInfraMlPlugins } from '../../utils/request_context'; -export const initAlertPreviewRoute = ({ framework, sources }: InfraBackendLibs) => { +export const initAlertPreviewRoute = ({ + framework, + sources, + getLogQueryFields, +}: InfraBackendLibs) => { framework.registerRoute( { method: 'post', @@ -77,6 +81,10 @@ export const initAlertPreviewRoute = ({ framework, sources }: InfraBackendLibs) }); } case METRIC_INVENTORY_THRESHOLD_ALERT_TYPE_ID: { + const logQueryFields = await getLogQueryFields( + sourceId || 'default', + requestContext.core.savedObjects.client + ); const { nodeType, criteria, @@ -87,6 +95,7 @@ export const initAlertPreviewRoute = ({ framework, sources }: InfraBackendLibs) params: { criteria, filterQuery, nodeType }, lookback, source, + logQueryFields, alertInterval, alertThrottle, alertNotifyWhen, diff --git a/x-pack/plugins/infra/server/routes/source/index.ts b/x-pack/plugins/infra/server/routes/metrics_sources/index.ts similarity index 69% rename from x-pack/plugins/infra/server/routes/source/index.ts rename to x-pack/plugins/infra/server/routes/metrics_sources/index.ts index 5ab3275f9ea9e..0123e4678697c 100644 --- a/x-pack/plugins/infra/server/routes/source/index.ts +++ b/x-pack/plugins/infra/server/routes/metrics_sources/index.ts @@ -8,63 +8,49 @@ import { schema } from '@kbn/config-schema'; import Boom from '@hapi/boom'; import { createValidationFunction } from '../../../common/runtime_types'; -import { - InfraSourceStatus, - SavedSourceConfigurationRuntimeType, - SourceResponseRuntimeType, -} from '../../../common/http_api/source_api'; import { InfraBackendLibs } from '../../lib/infra_types'; import { hasData } from '../../lib/sources/has_data'; import { createSearchClient } from '../../lib/create_search_client'; import { AnomalyThresholdRangeError } from '../../lib/sources/errors'; +import { + partialMetricsSourceConfigurationPropertiesRT, + metricsSourceConfigurationResponseRT, + MetricsSourceStatus, +} from '../../../common/metrics_sources'; -const typeToInfraIndexType = (value: string | undefined) => { - switch (value) { - case 'metrics': - return 'METRICS'; - case 'logs': - return 'LOGS'; - default: - return 'ANY'; - } -}; - -export const initSourceRoute = (libs: InfraBackendLibs) => { +export const initMetricsSourceConfigurationRoutes = (libs: InfraBackendLibs) => { const { framework } = libs; framework.registerRoute( { method: 'get', - path: '/api/metrics/source/{sourceId}/{type?}', + path: '/api/metrics/source/{sourceId}', validate: { params: schema.object({ sourceId: schema.string(), - type: schema.string(), }), }, }, async (requestContext, request, response) => { - const { type, sourceId } = request.params; + const { sourceId } = request.params; - const [source, logIndexStatus, metricIndicesExist, indexFields] = await Promise.all([ + const [source, metricIndicesExist, indexFields] = await Promise.all([ libs.sources.getSourceConfiguration(requestContext.core.savedObjects.client, sourceId), - libs.sourceStatus.getLogIndexStatus(requestContext, sourceId), libs.sourceStatus.hasMetricIndices(requestContext, sourceId), - libs.fields.getFields(requestContext, sourceId, typeToInfraIndexType(type)), + libs.fields.getFields(requestContext, sourceId, 'METRICS'), ]); if (!source) { return response.notFound(); } - const status: InfraSourceStatus = { - logIndicesExist: logIndexStatus !== 'missing', + const status: MetricsSourceStatus = { metricIndicesExist, indexFields, }; return response.ok({ - body: SourceResponseRuntimeType.encode({ source: { ...source, status } }), + body: metricsSourceConfigurationResponseRT.encode({ source: { ...source, status } }), }); } ); @@ -77,7 +63,7 @@ export const initSourceRoute = (libs: InfraBackendLibs) => { params: schema.object({ sourceId: schema.string(), }), - body: createValidationFunction(SavedSourceConfigurationRuntimeType), + body: createValidationFunction(partialMetricsSourceConfigurationPropertiesRT), }, }, framework.router.handleLegacyErrors(async (requestContext, request, response) => { @@ -110,20 +96,18 @@ export const initSourceRoute = (libs: InfraBackendLibs) => { patchedSourceConfigurationProperties )); - const [logIndexStatus, metricIndicesExist, indexFields] = await Promise.all([ - libs.sourceStatus.getLogIndexStatus(requestContext, sourceId), + const [metricIndicesExist, indexFields] = await Promise.all([ libs.sourceStatus.hasMetricIndices(requestContext, sourceId), - libs.fields.getFields(requestContext, sourceId, typeToInfraIndexType('metrics')), + libs.fields.getFields(requestContext, sourceId, 'METRICS'), ]); - const status: InfraSourceStatus = { - logIndicesExist: logIndexStatus !== 'missing', + const status: MetricsSourceStatus = { metricIndicesExist, indexFields, }; return response.ok({ - body: SourceResponseRuntimeType.encode({ + body: metricsSourceConfigurationResponseRT.encode({ source: { ...patchedSourceConfiguration, status }, }), }); @@ -154,25 +138,23 @@ export const initSourceRoute = (libs: InfraBackendLibs) => { framework.registerRoute( { method: 'get', - path: '/api/metrics/source/{sourceId}/{type}/hasData', + path: '/api/metrics/source/{sourceId}/hasData', validate: { params: schema.object({ sourceId: schema.string(), - type: schema.string(), }), }, }, async (requestContext, request, response) => { - const { type, sourceId } = request.params; + const { sourceId } = request.params; const client = createSearchClient(requestContext, framework); const source = await libs.sources.getSourceConfiguration( requestContext.core.savedObjects.client, sourceId ); - const indexPattern = - type === 'metrics' ? source.configuration.metricAlias : source.configuration.logAlias; - const results = await hasData(indexPattern, client); + + const results = await hasData(source.configuration.metricAlias, client); return response.ok({ body: { hasData: results }, diff --git a/x-pack/plugins/infra/server/routes/snapshot/index.ts b/x-pack/plugins/infra/server/routes/snapshot/index.ts index aaf23085d0d60..cbadd26ccd4bf 100644 --- a/x-pack/plugins/infra/server/routes/snapshot/index.ts +++ b/x-pack/plugins/infra/server/routes/snapshot/index.ts @@ -41,9 +41,15 @@ export const initSnapshotRoute = (libs: InfraBackendLibs) => { snapshotRequest.sourceId ); + const logQueryFields = await libs.getLogQueryFields( + snapshotRequest.sourceId, + requestContext.core.savedObjects.client + ); + UsageCollector.countNode(snapshotRequest.nodeType); const client = createSearchClient(requestContext, framework); - const snapshotResponse = await getNodes(client, snapshotRequest, source); + + const snapshotResponse = await getNodes(client, snapshotRequest, source, logQueryFields); return response.ok({ body: SnapshotNodeResponseRT.encode(snapshotResponse), diff --git a/x-pack/plugins/infra/server/routes/snapshot/lib/calculate_index_pattern_based_on_metrics.ts b/x-pack/plugins/infra/server/routes/snapshot/lib/calculate_index_pattern_based_on_metrics.ts deleted file mode 100644 index 85c1ece1ca042..0000000000000 --- a/x-pack/plugins/infra/server/routes/snapshot/lib/calculate_index_pattern_based_on_metrics.ts +++ /dev/null @@ -1,23 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import { SnapshotRequest } from '../../../../common/http_api'; -import { InfraSource } from '../../../lib/sources'; - -export const calculateIndexPatterBasedOnMetrics = ( - options: SnapshotRequest, - source: InfraSource -) => { - const { metrics } = options; - if (metrics.every((m) => m.type === 'logRate')) { - return source.configuration.logAlias; - } - if (metrics.some((m) => m.type === 'logRate')) { - return `${source.configuration.logAlias},${source.configuration.metricAlias}`; - } - return source.configuration.metricAlias; -}; diff --git a/x-pack/plugins/infra/server/routes/snapshot/lib/get_nodes.ts b/x-pack/plugins/infra/server/routes/snapshot/lib/get_nodes.ts index 9dec21d3ab1c7..ff3cf048b99de 100644 --- a/x-pack/plugins/infra/server/routes/snapshot/lib/get_nodes.ts +++ b/x-pack/plugins/infra/server/routes/snapshot/lib/get_nodes.ts @@ -12,16 +12,24 @@ import { transformRequestToMetricsAPIRequest } from './transform_request_to_metr import { queryAllData } from './query_all_data'; import { transformMetricsApiResponseToSnapshotResponse } from './trasform_metrics_ui_response'; import { copyMissingMetrics } from './copy_missing_metrics'; +import { LogQueryFields } from '../../../services/log_queries/get_log_query_fields'; -export const getNodes = async ( +export interface SourceOverrides { + indexPattern: string; + timestamp: string; +} + +const transformAndQueryData = async ( client: ESSearchClient, snapshotRequest: SnapshotRequest, - source: InfraSource + source: InfraSource, + sourceOverrides?: SourceOverrides ) => { const metricsApiRequest = await transformRequestToMetricsAPIRequest( client, source, - snapshotRequest + snapshotRequest, + sourceOverrides ); const metricsApiResponse = await queryAllData(client, metricsApiRequest); const snapshotResponse = transformMetricsApiResponseToSnapshotResponse( @@ -32,3 +40,59 @@ export const getNodes = async ( ); return copyMissingMetrics(snapshotResponse); }; + +export const getNodes = async ( + client: ESSearchClient, + snapshotRequest: SnapshotRequest, + source: InfraSource, + logQueryFields: LogQueryFields +) => { + let nodes; + + if (snapshotRequest.metrics.find((metric) => metric.type === 'logRate')) { + // *Only* the log rate metric has been requested + if (snapshotRequest.metrics.length === 1) { + nodes = await transformAndQueryData(client, snapshotRequest, source, logQueryFields); + } else { + // A scenario whereby a single host might be shipping metrics and logs. + const metricsWithoutLogsMetrics = snapshotRequest.metrics.filter( + (metric) => metric.type !== 'logRate' + ); + const nodesWithoutLogsMetrics = await transformAndQueryData( + client, + { ...snapshotRequest, metrics: metricsWithoutLogsMetrics }, + source + ); + const logRateNodes = await transformAndQueryData( + client, + { ...snapshotRequest, metrics: [{ type: 'logRate' }] }, + source, + logQueryFields + ); + // Merge nodes where possible - e.g. a single host is shipping metrics and logs + const mergedNodes = nodesWithoutLogsMetrics.nodes.map((node) => { + const logRateNode = logRateNodes.nodes.find( + (_logRateNode) => node.name === _logRateNode.name + ); + if (logRateNode) { + // Remove this from the "leftovers" + logRateNodes.nodes.filter((_node) => _node.name !== logRateNode.name); + } + return logRateNode + ? { + ...node, + metrics: [...node.metrics, ...logRateNode.metrics], + } + : node; + }); + nodes = { + ...nodesWithoutLogsMetrics, + nodes: [...mergedNodes, ...logRateNodes.nodes], + }; + } + } else { + nodes = await transformAndQueryData(client, snapshotRequest, source); + } + + return nodes; +}; diff --git a/x-pack/plugins/infra/server/routes/snapshot/lib/transform_request_to_metrics_api_request.ts b/x-pack/plugins/infra/server/routes/snapshot/lib/transform_request_to_metrics_api_request.ts index 8804121fc4167..128137efa272e 100644 --- a/x-pack/plugins/infra/server/routes/snapshot/lib/transform_request_to_metrics_api_request.ts +++ b/x-pack/plugins/infra/server/routes/snapshot/lib/transform_request_to_metrics_api_request.ts @@ -12,13 +12,14 @@ import { InfraSource } from '../../../lib/sources'; import { createTimeRangeWithInterval } from './create_timerange_with_interval'; import { parseFilterQuery } from '../../../utils/serialized_query'; import { transformSnapshotMetricsToMetricsAPIMetrics } from './transform_snapshot_metrics_to_metrics_api_metrics'; -import { calculateIndexPatterBasedOnMetrics } from './calculate_index_pattern_based_on_metrics'; import { META_KEY } from './constants'; +import { SourceOverrides } from './get_nodes'; export const transformRequestToMetricsAPIRequest = async ( client: ESSearchClient, source: InfraSource, - snapshotRequest: SnapshotRequest + snapshotRequest: SnapshotRequest, + sourceOverrides?: SourceOverrides ): Promise => { const timeRangeWithIntervalApplied = await createTimeRangeWithInterval(client, { ...snapshotRequest, @@ -27,9 +28,9 @@ export const transformRequestToMetricsAPIRequest = async ( }); const metricsApiRequest: MetricsAPIRequest = { - indexPattern: calculateIndexPatterBasedOnMetrics(snapshotRequest, source), + indexPattern: sourceOverrides?.indexPattern ?? source.configuration.metricAlias, timerange: { - field: source.configuration.fields.timestamp, + field: sourceOverrides?.timestamp ?? source.configuration.fields.timestamp, from: timeRangeWithIntervalApplied.from, to: timeRangeWithIntervalApplied.to, interval: timeRangeWithIntervalApplied.interval, @@ -74,7 +75,7 @@ export const transformRequestToMetricsAPIRequest = async ( top_hits: { size: 1, _source: [inventoryFields.name], - sort: [{ [source.configuration.fields.timestamp]: 'desc' }], + sort: [{ [sourceOverrides?.timestamp ?? source.configuration.fields.timestamp]: 'desc' }], }, }, }, diff --git a/x-pack/plugins/infra/server/services/log_queries/get_log_query_fields.ts b/x-pack/plugins/infra/server/services/log_queries/get_log_query_fields.ts new file mode 100644 index 0000000000000..9497a8b442768 --- /dev/null +++ b/x-pack/plugins/infra/server/services/log_queries/get_log_query_fields.ts @@ -0,0 +1,32 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { SavedObjectsClientContract } from 'src/core/server'; +import { InfraSources } from '../../lib/sources'; + +// NOTE: TEMPORARY: This will become a subset of the new resolved KIP compatible log source configuration. +export interface LogQueryFields { + indexPattern: string; + timestamp: string; +} + +// NOTE: TEMPORARY: This will become a subset of the new resolved KIP compatible log source configuration. +export const createGetLogQueryFields = (sources: InfraSources) => { + return async ( + sourceId: string, + savedObjectsClient: SavedObjectsClientContract + ): Promise => { + const source = await sources.getSourceConfiguration(savedObjectsClient, sourceId); + + return { + indexPattern: source.configuration.logAlias, + timestamp: source.configuration.fields.timestamp, + }; + }; +}; + +export type GetLogQueryFields = ReturnType; diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/datapanel.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/datapanel.test.tsx index fef8ee171830d..669758c5193a6 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/datapanel.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/datapanel.test.tsx @@ -173,7 +173,7 @@ const initialState: IndexPatternPrivateState = { label: 'My Op', dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'memory', }, }, @@ -200,7 +200,7 @@ const initialState: IndexPatternPrivateState = { label: 'My Op', dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, }, diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/bucket_nesting_editor.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/bucket_nesting_editor.test.tsx index ccdb86d250962..c6ecdd73cb6ab 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/bucket_nesting_editor.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/bucket_nesting_editor.test.tsx @@ -134,7 +134,7 @@ describe('BucketNestingEditor', () => { layer={{ columnOrder: ['a', 'b', 'c'], columns: { - a: mockCol({ operationType: 'avg', isBucketed: false }), + a: mockCol({ operationType: 'average', isBucketed: false }), b: mockCol({ operationType: 'max', isBucketed: false }), c: mockCol({ operationType: 'min', isBucketed: false }), }, diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_panel.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_panel.test.tsx index d586818cb3c11..7d1644d07d2aa 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_panel.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_panel.test.tsx @@ -372,7 +372,7 @@ describe('IndexPatternDimensionEditorPanel', () => { label: 'Unique count of source', dataType: 'number', isBucketed: false, - operationType: 'cardinality', + operationType: 'unique_count', sourceField: 'source,', }, })} @@ -414,7 +414,7 @@ describe('IndexPatternDimensionEditorPanel', () => { const items: EuiListGroupItemProps[] = wrapper.find(EuiListGroup).prop('listItems') || []; - expect(items.find(({ id }) => id === 'derivative')!['data-test-subj']).toContain( + expect(items.find(({ id }) => id === 'differences')!['data-test-subj']).toContain( 'incompatible' ); expect(items.find(({ id }) => id === 'cumulative_sum')!['data-test-subj']).toContain( @@ -462,7 +462,7 @@ describe('IndexPatternDimensionEditorPanel', () => { 'incompatible' ); - expect(items.find(({ id }) => id === 'derivative')!['data-test-subj']).not.toContain( + expect(items.find(({ id }) => id === 'differences')!['data-test-subj']).not.toContain( 'incompatible' ); expect(items.find(({ id }) => id === 'moving_average')!['data-test-subj']).not.toContain( @@ -817,7 +817,7 @@ describe('IndexPatternDimensionEditorPanel', () => { it('should select compatible operation if field not compatible with selected operation', () => { wrapper = mount(); - wrapper.find('button[data-test-subj="lns-indexPatternDimension-avg"]').simulate('click'); + wrapper.find('button[data-test-subj="lns-indexPatternDimension-average"]').simulate('click'); expect(setState).toHaveBeenCalledWith( { ...state, @@ -825,7 +825,7 @@ describe('IndexPatternDimensionEditorPanel', () => { first: { ...state.layers.first, incompleteColumns: { - col2: { operationType: 'avg' }, + col2: { operationType: 'average' }, }, }, }, @@ -838,7 +838,7 @@ describe('IndexPatternDimensionEditorPanel', () => { .filter('[data-test-subj="indexPattern-dimension-field"]'); const options = comboBox.prop('options'); - // options[1][2] is a `source` field of type `string` which doesn't support `avg` operation + // options[1][2] is a `source` field of type `string` which doesn't support `average` operation act(() => { comboBox.prop('onChange')!([options![1].options![2]]); }); @@ -885,7 +885,7 @@ describe('IndexPatternDimensionEditorPanel', () => { // Transition to a field operation (incompatible) wrapper - .find('button[data-test-subj="lns-indexPatternDimension-avg incompatible"]') + .find('button[data-test-subj="lns-indexPatternDimension-average incompatible"]') .simulate('click'); // Now check that the dimension gets cleaned up on state update @@ -896,7 +896,7 @@ describe('IndexPatternDimensionEditorPanel', () => { first: { ...state.layers.first, incompleteColumns: { - col2: { operationType: 'avg' }, + col2: { operationType: 'average' }, }, }, }, @@ -914,7 +914,7 @@ describe('IndexPatternDimensionEditorPanel', () => { dataType: 'number', isBucketed: false, label: '', - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, })} @@ -1037,7 +1037,7 @@ describe('IndexPatternDimensionEditorPanel', () => { wrapper = shallow( @@ -1143,7 +1143,7 @@ describe('IndexPatternDimensionEditorPanel', () => { label: 'Sum of bytes per hour', }); wrapper = mount(); - wrapper.find('button[data-test-subj="lns-indexPatternDimension-avg"]').simulate('click'); + wrapper.find('button[data-test-subj="lns-indexPatternDimension-average"]').simulate('click'); expect(props.setState).toHaveBeenCalledWith( { ...props.state, @@ -1478,7 +1478,7 @@ describe('IndexPatternDimensionEditorPanel', () => { it('should support selecting the operation before the field', () => { wrapper = mount(); - wrapper.find('button[data-test-subj="lns-indexPatternDimension-avg"]').simulate('click'); + wrapper.find('button[data-test-subj="lns-indexPatternDimension-average"]').simulate('click'); expect(setState).toHaveBeenCalledWith( { @@ -1488,7 +1488,7 @@ describe('IndexPatternDimensionEditorPanel', () => { ...state.layers.first, incompleteColumns: { col2: { - operationType: 'avg', + operationType: 'average', }, }, }, @@ -1516,7 +1516,7 @@ describe('IndexPatternDimensionEditorPanel', () => { columns: { ...state.layers.first.columns, col2: expect.objectContaining({ - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }), }, @@ -1547,7 +1547,7 @@ describe('IndexPatternDimensionEditorPanel', () => { /> ); - wrapper.find('button[data-test-subj="lns-indexPatternDimension-avg"]').simulate('click'); + wrapper.find('button[data-test-subj="lns-indexPatternDimension-average"]').simulate('click'); expect(setState).toHaveBeenCalledWith( { @@ -1559,7 +1559,7 @@ describe('IndexPatternDimensionEditorPanel', () => { ...initialState.layers.first.columns, col2: expect.objectContaining({ sourceField: 'bytes', - operationType: 'avg', + operationType: 'average', // Other parts of this don't matter for this test }), }, @@ -1601,7 +1601,7 @@ describe('IndexPatternDimensionEditorPanel', () => { wrapper = mount(); act(() => { - wrapper.find('button[data-test-subj="lns-indexPatternDimension-avg"]').simulate('click'); + wrapper.find('button[data-test-subj="lns-indexPatternDimension-average"]').simulate('click'); }); const options = wrapper @@ -1790,7 +1790,7 @@ describe('IndexPatternDimensionEditorPanel', () => { dataType: 'number', isBucketed: false, // Private - operationType: 'avg', + operationType: 'average', sourceField: 'memory', }, }); @@ -1831,7 +1831,7 @@ describe('IndexPatternDimensionEditorPanel', () => { dataType: 'number', isBucketed: false, // Private - operationType: 'avg', + operationType: 'average', sourceField: 'memory', params: { format: { id: 'bytes', params: { decimals: 0 } }, @@ -1871,7 +1871,7 @@ describe('IndexPatternDimensionEditorPanel', () => { dataType: 'number', isBucketed: false, // Private - operationType: 'avg', + operationType: 'average', sourceField: 'memory', params: { format: { id: 'bytes', params: { decimals: 2 } }, @@ -1914,7 +1914,7 @@ describe('IndexPatternDimensionEditorPanel', () => { expect(wrapper.find('ReferenceEditor')).toHaveLength(0); wrapper - .find('button[data-test-subj="lns-indexPatternDimension-derivative incompatible"]') + .find('button[data-test-subj="lns-indexPatternDimension-differences incompatible"]') .simulate('click'); expect(wrapper.find('ReferenceEditor')).toHaveLength(1); @@ -1926,7 +1926,7 @@ describe('IndexPatternDimensionEditorPanel', () => { label: 'Differences of (incomplete)', dataType: 'number', isBucketed: false, - operationType: 'derivative', + operationType: 'differences', references: ['col2'], params: {}, }, @@ -1939,7 +1939,7 @@ describe('IndexPatternDimensionEditorPanel', () => { expect(wrapper.find('ReferenceEditor')).toHaveLength(1); wrapper - .find('button[data-test-subj="lns-indexPatternDimension-avg incompatible"]') + .find('button[data-test-subj="lns-indexPatternDimension-average incompatible"]') .simulate('click'); expect(wrapper.find('ReferenceEditor')).toHaveLength(0); @@ -1948,10 +1948,10 @@ describe('IndexPatternDimensionEditorPanel', () => { it('should show a warning when the current dimension is no longer configurable', () => { const stateWithInvalidCol: IndexPatternPrivateState = getStateWithColumns({ col1: { - label: 'Invalid derivative', + label: 'Invalid differences', dataType: 'number', isBucketed: false, - operationType: 'derivative', + operationType: 'differences', references: ['ref1'], }, }); @@ -1962,7 +1962,7 @@ describe('IndexPatternDimensionEditorPanel', () => { expect( wrapper - .find('[data-test-subj="lns-indexPatternDimension-derivative incompatible"]') + .find('[data-test-subj="lns-indexPatternDimension-differences incompatible"]') .find('EuiText[color="danger"]') .first() ).toBeTruthy(); @@ -1975,7 +1975,7 @@ describe('IndexPatternDimensionEditorPanel', () => { label: 'Avg', dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, }), @@ -2010,6 +2010,8 @@ describe('IndexPatternDimensionEditorPanel', () => { ); - expect(wrapper.find('[data-test-subj="lns-indexPatternDimension-derivative"]')).toHaveLength(0); + expect(wrapper.find('[data-test-subj="lns-indexPatternDimension-differences"]')).toHaveLength( + 0 + ); }); }); diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/droppable.test.ts b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/droppable.test.ts index 82b6434e50aac..4f73454b68811 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/droppable.test.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/droppable.test.ts @@ -1035,7 +1035,7 @@ describe('IndexPatternDimensionEditorPanel', () => { }, }, col3: { - operationType: 'avg', + operationType: 'average', sourceField: 'memory', label: 'average of memory', dataType: 'number', diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.test.tsx index 9ad6a2d20a4c2..f17adf9be39f3 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.test.tsx @@ -157,7 +157,7 @@ describe('reference editor', () => { label: 'Average of bytes', dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, }, @@ -192,7 +192,7 @@ describe('reference editor', () => { label: 'Average of bytes', dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, }, @@ -233,7 +233,7 @@ describe('reference editor', () => { label: 'Average of bytes', dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, }, @@ -276,7 +276,7 @@ describe('reference editor', () => { label: 'Average of bytes', dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, }, @@ -296,9 +296,9 @@ describe('reference editor', () => { expect(subFunctionSelect.prop('selectedOptions')).toEqual( expect.arrayContaining([ expect.objectContaining({ - 'data-test-subj': 'lns-indexPatternDimension-avg incompatible', + 'data-test-subj': 'lns-indexPatternDimension-average incompatible', label: 'Average', - value: 'avg', + value: 'average', }), ]) ); @@ -334,7 +334,7 @@ describe('reference editor', () => { label: 'Average of bytes', dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, }, @@ -352,7 +352,7 @@ describe('reference editor', () => { const fieldSelect = wrapper.find(FieldSelect); expect(fieldSelect.prop('fieldIsInvalid')).toEqual(true); expect(fieldSelect.prop('selectedField')).toEqual('bytes'); - expect(fieldSelect.prop('selectedOperationType')).toEqual('avg'); + expect(fieldSelect.prop('selectedOperationType')).toEqual('average'); expect(fieldSelect.prop('incompleteOperation')).toEqual('max'); expect(fieldSelect.prop('markAllFieldsCompatible')).toEqual(false); }); @@ -369,7 +369,7 @@ describe('reference editor', () => { label: 'Average of bytes', dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, }, @@ -387,7 +387,7 @@ describe('reference editor', () => { const fieldSelect = wrapper.find(FieldSelect); expect(fieldSelect.prop('fieldIsInvalid')).toEqual(false); expect(fieldSelect.prop('selectedField')).toEqual('timestamp'); - expect(fieldSelect.prop('selectedOperationType')).toEqual('avg'); + expect(fieldSelect.prop('selectedOperationType')).toEqual('average'); expect(fieldSelect.prop('incompleteOperation')).toBeUndefined(); }); @@ -423,7 +423,7 @@ describe('reference editor', () => { label: 'Average of missing', dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'missing', }, }, @@ -438,7 +438,7 @@ describe('reference editor', () => { const fieldSelect = wrapper.find(FieldSelect); expect(fieldSelect.prop('fieldIsInvalid')).toEqual(true); expect(fieldSelect.prop('selectedField')).toEqual('missing'); - expect(fieldSelect.prop('selectedOperationType')).toEqual('avg'); + expect(fieldSelect.prop('selectedOperationType')).toEqual('average'); expect(fieldSelect.prop('incompleteOperation')).toBeUndefined(); expect(fieldSelect.prop('markAllFieldsCompatible')).toEqual(false); }); diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern.test.ts b/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern.test.ts index c02515c2e7201..14834adfc33cc 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern.test.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern.test.ts @@ -546,7 +546,7 @@ describe('IndexPattern Data Source', () => { dataType: 'number', isBucketed: false, sourceField: 'bytes', - operationType: 'avg', + operationType: 'average', timeScale: 'h', }, col3: { @@ -659,7 +659,7 @@ describe('IndexPattern Data Source', () => { dataType: 'number', isBucketed: false, sourceField: 'bytes', - operationType: 'avg', + operationType: 'average', timeScale: 'h', }, col3: { @@ -835,7 +835,7 @@ describe('IndexPattern Data Source', () => { dataType: 'date', isBucketed: false, sourceField: 'timefield', - operationType: 'cardinality', + operationType: 'unique_count', }, col2: { label: 'Date', @@ -885,7 +885,7 @@ describe('IndexPattern Data Source', () => { dataType: 'date', isBucketed: false, sourceField: 'timefield', - operationType: 'cardinality', + operationType: 'unique_count', }, col2: { label: 'Reference', @@ -1183,7 +1183,7 @@ describe('IndexPattern Data Source', () => { dataType: 'number', isBucketed: false, label: 'Foo', - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, }, @@ -1210,7 +1210,7 @@ describe('IndexPattern Data Source', () => { columnOrder: [], columns: {}, incompleteColumns: { - col1: { operationType: 'avg' as const }, + col1: { operationType: 'average' as const }, col2: { operationType: 'sum' as const }, }, }, diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.test.tsx index 210716e7494e0..e742b6ba62aff 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.test.tsx @@ -735,7 +735,7 @@ describe('IndexPattern Data Source suggestions', () => { sourceField: 'bytes', label: 'Avg of bytes', customLabel: true, - operationType: 'avg', + operationType: 'average', }, }, columnOrder: ['cola', 'colb'], @@ -770,7 +770,7 @@ describe('IndexPattern Data Source suggestions', () => { sourceField: 'bytes', label: 'Avg of bytes', customLabel: true, - operationType: 'avg', + operationType: 'average', }, }, }, @@ -1060,7 +1060,7 @@ describe('IndexPattern Data Source suggestions', () => { customLabel: true, dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, ref: { @@ -1120,7 +1120,7 @@ describe('IndexPattern Data Source suggestions', () => { customLabel: true, dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, ref: { @@ -1468,7 +1468,7 @@ describe('IndexPattern Data Source suggestions', () => { customLabel: true, dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', scale: 'ratio', }, @@ -1537,7 +1537,7 @@ describe('IndexPattern Data Source suggestions', () => { customLabel: true, dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', scale: 'ratio', }, @@ -1601,7 +1601,7 @@ describe('IndexPattern Data Source suggestions', () => { isBucketed: false, sourceField: 'dest', label: 'Unique count of dest', - operationType: 'cardinality', + operationType: 'unique_count', }, colb: { label: 'My Op', @@ -1647,7 +1647,7 @@ describe('IndexPattern Data Source suggestions', () => { isBucketed: false, sourceField: 'dest', label: 'Unique count of dest', - operationType: 'cardinality', + operationType: 'unique_count', }, colb: { label: 'My Custom Range', @@ -1723,7 +1723,7 @@ describe('IndexPattern Data Source suggestions', () => { customLabel: true, dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', scale: 'ratio', }, @@ -1843,7 +1843,7 @@ describe('IndexPattern Data Source suggestions', () => { dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'field4', }, col5: { @@ -1951,7 +1951,7 @@ describe('IndexPattern Data Source suggestions', () => { dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'field1', }, }, @@ -2031,7 +2031,7 @@ describe('IndexPattern Data Source suggestions', () => { dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'field1', }, }, @@ -2091,7 +2091,7 @@ describe('IndexPattern Data Source suggestions', () => { dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, }, diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/layerpanel.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/layerpanel.test.tsx index 381fa4ca27a49..a7c1074ed4eef 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/layerpanel.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/layerpanel.test.tsx @@ -165,7 +165,7 @@ const initialState: IndexPatternPrivateState = { label: 'My Op', dataType: 'number', isBucketed: false, - operationType: 'avg', + operationType: 'average', sourceField: 'memory', }, }, diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/loader.test.ts b/x-pack/plugins/lens/public/indexpattern_datasource/loader.test.ts index f4fa8bd185b6d..a68f8ae310f3e 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/loader.test.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/loader.test.ts @@ -145,7 +145,7 @@ const indexPattern2 = ({ agg: 'histogram', interval: 1000, }, - avg: { + average: { agg: 'avg', }, max: { @@ -569,7 +569,7 @@ describe('loader', () => { dataType: 'number', isBucketed: false, label: '', - operationType: 'avg', + operationType: 'average', sourceField: 'myfield', }, }, @@ -582,7 +582,7 @@ describe('loader', () => { dataType: 'number', isBucketed: false, label: '', - operationType: 'avg', + operationType: 'average', sourceField: 'myfield2', }, }, diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/loader.ts b/x-pack/plugins/lens/public/indexpattern_datasource/loader.ts index 7052a69ee6fb7..ec7ef6a37a27a 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/loader.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/loader.ts @@ -17,7 +17,7 @@ import { IndexPatternField, IndexPatternLayer, } from './types'; -import { updateLayerIndexPattern } from './operations'; +import { updateLayerIndexPattern, translateToOperationName } from './operations'; import { DateRange, ExistingFields } from '../../common/types'; import { BASE_API_URL } from '../../common'; import { @@ -109,7 +109,7 @@ export async function loadIndexPatterns({ const restriction = typeMeta.aggs && typeMeta.aggs[agg] && typeMeta.aggs[agg][field.name]; if (restriction) { - restrictionsObj[agg] = restriction; + restrictionsObj[translateToOperationName(agg)] = restriction; } }); if (Object.keys(restrictionsObj).length) { diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/derivative.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/differences.tsx similarity index 95% rename from x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/derivative.tsx rename to x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/differences.tsx index 5fb0bc3a83528..c50e9270eaac1 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/derivative.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/differences.tsx @@ -19,6 +19,8 @@ import { adjustTimeScaleOnOtherColumnChange } from '../../time_scale_utils'; import { OperationDefinition } from '..'; import { getFormatFromPreviousColumn } from '../helpers'; +const OPERATION_NAME = 'differences'; + const ofName = buildLabelFunction((name?: string) => { return i18n.translate('xpack.lens.indexPattern.derivativeOf', { defaultMessage: 'Differences of {name}', @@ -34,14 +36,14 @@ const ofName = buildLabelFunction((name?: string) => { export type DerivativeIndexPatternColumn = FormattedIndexPatternColumn & ReferenceBasedIndexPatternColumn & { - operationType: 'derivative'; + operationType: typeof OPERATION_NAME; }; export const derivativeOperation: OperationDefinition< DerivativeIndexPatternColumn, 'fullReference' > = { - type: 'derivative', + type: OPERATION_NAME, priority: 1, displayName: i18n.translate('xpack.lens.indexPattern.derivative', { defaultMessage: 'Differences', @@ -78,7 +80,7 @@ export const derivativeOperation: OperationDefinition< previousColumn?.timeScale ), dataType: 'number', - operationType: 'derivative', + operationType: OPERATION_NAME, isBucketed: false, scale: 'ratio', references: referenceIds, diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/index.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/index.ts index f261a0e1e2005..815acb8c4169f 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/index.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/index.ts @@ -7,5 +7,5 @@ export { counterRateOperation, CounterRateIndexPatternColumn } from './counter_rate'; export { cumulativeSumOperation, CumulativeSumIndexPatternColumn } from './cumulative_sum'; -export { derivativeOperation, DerivativeIndexPatternColumn } from './derivative'; +export { derivativeOperation, DerivativeIndexPatternColumn } from './differences'; export { movingAverageOperation, MovingAverageIndexPatternColumn } from './moving_average'; diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/cardinality.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/cardinality.tsx index 513bac14af7a3..fa1691ba9a78e 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/cardinality.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/cardinality.tsx @@ -25,7 +25,7 @@ const supportedTypes = new Set([ ]); const SCALE = 'ratio'; -const OPERATION_TYPE = 'cardinality'; +const OPERATION_TYPE = 'unique_count'; const IS_BUCKETED = false; function ofName(name: string) { @@ -40,7 +40,7 @@ function ofName(name: string) { export interface CardinalityIndexPatternColumn extends FormattedIndexPatternColumn, FieldBasedIndexPatternColumn { - operationType: 'cardinality'; + operationType: typeof OPERATION_TYPE; } export const cardinalityOperation: OperationDefinition = { diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/filters/filters.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/filters/filters.tsx index 03f8375409246..bf563b877ef5e 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/filters/filters.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/filters/filters.tsx @@ -26,6 +26,7 @@ import { buildExpressionFunction } from '../../../../../../../../src/plugins/exp import { NewBucketButton, DragDropBuckets, DraggableBucketContainer } from '../shared_components'; const generateId = htmlIdGenerator(); +const OPERATION_NAME = 'filters'; // references types from src/plugins/data/common/search/aggs/buckets/filters.ts export interface Filter { @@ -70,14 +71,14 @@ export const isQueryValid = (input: Query, indexPattern: IndexPattern) => { }; export interface FiltersIndexPatternColumn extends BaseIndexPatternColumn { - operationType: 'filters'; + operationType: typeof OPERATION_NAME; params: { filters: Filter[]; }; } export const filtersOperation: OperationDefinition = { - type: 'filters', + type: OPERATION_NAME, displayName: filtersLabel, priority: 3, // Higher than any metric input: 'none', @@ -86,7 +87,7 @@ export const filtersOperation: OperationDefinition filtersLabel, buildColumn({ previousColumn }) { let params = { filters: [defaultFilter] }; - if (previousColumn?.operationType === 'terms') { + if (previousColumn?.operationType === 'terms' && 'sourceField' in previousColumn) { params = { filters: [ { @@ -103,7 +104,7 @@ export const filtersOperation: OperationDefinition { dataType: 'number', isBucketed: false, label: 'Foo', - operationType: 'avg', // <= invalid + operationType: 'average', // <= invalid sourceField: 'timestamp', }, createMockedIndexPattern() @@ -46,7 +46,7 @@ describe('helpers', () => { dataType: 'number', isBucketed: false, label: 'Foo', - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, createMockedIndexPattern() diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/index.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/index.ts index cdb93048c9a58..b3aa93b062eb1 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/index.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/index.ts @@ -437,3 +437,11 @@ export const operationDefinitionMap: Record< (definitionMap, definition) => ({ ...definitionMap, [definition.type]: definition }), {} ); + +/** + * Cannot map the prev names, but can guarantee the new names are matching up using the type system + */ +export const renameOperationsMapping: Record = { + avg: 'average', + cardinality: 'unique_count', +}; diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/metrics.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/metrics.tsx index 1767b76e88202..20580634d12e6 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/metrics.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/metrics.tsx @@ -27,7 +27,7 @@ type MetricColumn = FormattedIndexPatternColumn & const typeToFn: Record = { min: 'aggMin', max: 'aggMax', - avg: 'aggAvg', + average: 'aggAvg', sum: 'aggSum', median: 'aggMedian', }; @@ -76,7 +76,6 @@ function buildMetricOperation>({ }, isTransferable: (column, newIndexPattern) => { const newField = newIndexPattern.getFieldByName(column.sourceField); - return Boolean( newField && supportedTypes.includes(newField.type) && @@ -124,7 +123,7 @@ function buildMetricOperation>({ } export type SumIndexPatternColumn = MetricColumn<'sum'>; -export type AvgIndexPatternColumn = MetricColumn<'avg'>; +export type AvgIndexPatternColumn = MetricColumn<'average'>; export type MinIndexPatternColumn = MetricColumn<'min'>; export type MaxIndexPatternColumn = MetricColumn<'max'>; export type MedianIndexPatternColumn = MetricColumn<'median'>; @@ -154,7 +153,7 @@ export const maxOperation = buildMetricOperation({ }); export const averageOperation = buildMetricOperation({ - type: 'avg', + type: 'average', priority: 2, displayName: i18n.translate('xpack.lens.indexPattern.avg', { defaultMessage: 'Average', diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.tsx index a31cf9f019480..639b9e3a95c47 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.tsx @@ -74,7 +74,10 @@ export const percentileOperation: OperationDefinition { const existingPercentileParam = - previousColumn?.operationType === 'percentile' && previousColumn?.params.percentile; + previousColumn?.operationType === 'percentile' && + previousColumn.params && + 'percentile' in previousColumn.params && + previousColumn.params.percentile; const newPercentileParam = existingPercentileParam || DEFAULT_PERCENTILE_VALUE; return { label: ofName(getSafeName(field.name, indexPattern), newPercentileParam), diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.test.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.test.ts index 4a05e6f372d30..62cce21ead636 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.test.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.test.ts @@ -120,7 +120,7 @@ describe('state_helpers', () => { isBucketed: false, // Private - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, }, @@ -147,7 +147,7 @@ describe('state_helpers', () => { isBucketed: false, // Private - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, }, @@ -365,7 +365,7 @@ describe('state_helpers', () => { isBucketed: false, // Private - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, col2: { @@ -533,7 +533,7 @@ describe('state_helpers', () => { isBucketed: false, // Private - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, col2: { @@ -840,7 +840,7 @@ describe('state_helpers', () => { }, indexPattern, columnId: 'col2', - op: 'avg', + op: 'average', field: indexPattern.fields[2], // bytes field visualizationGroups: [], }); @@ -856,7 +856,7 @@ describe('state_helpers', () => { dataType: 'number', isBucketed: false, sourceField: 'bytes', - operationType: 'avg', + operationType: 'average', }), }, incompleteColumns: {}, @@ -1027,7 +1027,7 @@ describe('state_helpers', () => { dataType: 'number' as const, isBucketed: false, sourceField: 'bytes', - operationType: 'avg' as const, + operationType: 'average' as const, }, }, }; @@ -1056,7 +1056,7 @@ describe('state_helpers', () => { { input: ['field'], validateMetadata: () => true, - specificOperations: ['cardinality', 'sum', 'avg'], // this order is ignored + specificOperations: ['unique_count', 'sum', 'average'], // this order is ignored }, ]; const layer: IndexPatternLayer = { @@ -1083,7 +1083,7 @@ describe('state_helpers', () => { expect(result.columnOrder).toEqual(['id1', 'col1']); expect(result.columns).toEqual({ id1: expect.objectContaining({ - operationType: 'avg', + operationType: 'average', }), col1: expect.objectContaining({ operationType: 'testReference', @@ -1097,7 +1097,7 @@ describe('state_helpers', () => { { input: ['field'], validateMetadata: () => true, - specificOperations: ['cardinality'], + specificOperations: ['unique_count'], }, ]; const layer: IndexPatternLayer = { @@ -1122,7 +1122,7 @@ describe('state_helpers', () => { }); expect(result.incompleteColumns).toEqual({ - id1: { operationType: 'cardinality' }, + id1: { operationType: 'unique_count' }, }); expect(result.columns).toEqual({ col1: expect.objectContaining({ @@ -1347,7 +1347,7 @@ describe('state_helpers', () => { columns: { id1: expect.objectContaining({ sourceField: 'timestamp', - operationType: 'cardinality', + operationType: 'unique_count', }), output: expect.objectContaining({ references: ['id1'] }), }, @@ -1463,7 +1463,7 @@ describe('state_helpers', () => { dataType: 'number' as const, isBucketed: false, sourceField: 'bytes', - operationType: 'avg' as const, + operationType: 'average' as const, }; const layer: IndexPatternLayer = { @@ -1475,7 +1475,7 @@ describe('state_helpers', () => { label: 'Reference', dataType: 'number', isBucketed: false, - operationType: 'derivative', + operationType: 'differences', references: ['metric'], }, }, @@ -1829,7 +1829,7 @@ describe('state_helpers', () => { dataType: 'number', isBucketed: false, // Private - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }; @@ -1915,7 +1915,7 @@ describe('state_helpers', () => { isBucketed: false, // Private - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, col3: { @@ -1963,7 +1963,7 @@ describe('state_helpers', () => { isBucketed: false, // Private - operationType: 'avg', + operationType: 'average', sourceField: 'bytes', }, ref2: { @@ -2006,7 +2006,7 @@ describe('state_helpers', () => { searchable: true, type: 'number', aggregationRestrictions: { - avg: { + average: { agg: 'avg', }, }, @@ -2076,7 +2076,7 @@ describe('state_helpers', () => { dataType: 'number', isBucketed: false, label: '', - operationType: 'avg', + operationType: 'average', sourceField: 'xxx', }, }, @@ -2107,7 +2107,7 @@ describe('state_helpers', () => { dataType: 'number', isBucketed: false, label: '', - operationType: 'avg', + operationType: 'average', sourceField: 'fieldB', }, }, @@ -2170,7 +2170,7 @@ describe('state_helpers', () => { dataType: 'number', isBucketed: false, label: '', - operationType: 'avg', + operationType: 'average', sourceField: 'fieldD', }, }, @@ -2220,14 +2220,14 @@ describe('state_helpers', () => { describe('getErrorMessages', () => { it('should collect errors from metric-type operation definitions', () => { const mock = jest.fn().mockReturnValue(['error 1']); - operationDefinitionMap.avg.getErrorMessage = mock; + operationDefinitionMap.average.getErrorMessage = mock; const errors = getErrorMessages( { indexPatternId: '1', columnOrder: [], columns: { // @ts-expect-error invalid column - col1: { operationType: 'avg' }, + col1: { operationType: 'average' }, }, }, indexPattern diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/operations.test.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/operations.test.ts index 8c5dee8bbb28f..4c54b777b66f3 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/operations.test.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/operations.test.ts @@ -56,7 +56,7 @@ describe('getOperationTypesForField', () => { aggregatable: true, searchable: true, }) - ).toEqual(['terms', 'cardinality', 'last_value']); + ).toEqual(['terms', 'unique_count', 'last_value']); }); it('should return only bucketed operations on strings when passed proper filterOperations function', () => { @@ -87,11 +87,11 @@ describe('getOperationTypesForField', () => { 'range', 'terms', 'median', - 'avg', + 'average', 'sum', 'min', 'max', - 'cardinality', + 'unique_count', 'percentile', 'last_value', ]); @@ -109,7 +109,16 @@ describe('getOperationTypesForField', () => { }, (op) => !op.isBucketed ) - ).toEqual(['median', 'avg', 'sum', 'min', 'max', 'cardinality', 'percentile', 'last_value']); + ).toEqual([ + 'median', + 'average', + 'sum', + 'min', + 'max', + 'unique_count', + 'percentile', + 'last_value', + ]); }); it('should return operations on dates', () => { @@ -286,7 +295,7 @@ describe('getOperationTypesForField', () => { }, Object { "field": "bytes", - "operationType": "avg", + "operationType": "average", "type": "field", }, Object { @@ -303,7 +312,7 @@ describe('getOperationTypesForField', () => { "type": "fullReference", }, Object { - "operationType": "derivative", + "operationType": "differences", "type": "fullReference", }, Object { @@ -322,17 +331,17 @@ describe('getOperationTypesForField', () => { }, Object { "field": "timestamp", - "operationType": "cardinality", + "operationType": "unique_count", "type": "field", }, Object { "field": "bytes", - "operationType": "cardinality", + "operationType": "unique_count", "type": "field", }, Object { "field": "source", - "operationType": "cardinality", + "operationType": "unique_count", "type": "field", }, Object { diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/operations.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/operations.ts index 140ebc813f6c1..a45650f9323f9 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/operations.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/operations.ts @@ -12,12 +12,19 @@ import { operationDefinitions, GenericOperationDefinition, OperationType, + renameOperationsMapping, } from './definitions'; import { IndexPattern, IndexPatternField } from '../types'; import { documentField } from '../document_field'; export { operationDefinitionMap } from './definitions'; - +/** + * Map aggregation names from Elasticsearch to Lens names. + * Used when loading indexpatterns to map metadata (i.e. restrictions) + */ +export function translateToOperationName(agg: string): OperationType { + return agg in renameOperationsMapping ? renameOperationsMapping[agg] : (agg as OperationType); +} /** * Returns all available operation types as a list at runtime. * This will be an array of each member of the union type `OperationType` diff --git a/x-pack/plugins/lens/server/migrations.test.ts b/x-pack/plugins/lens/server/migrations.test.ts index 01329d85baf00..bed19942e52bc 100644 --- a/x-pack/plugins/lens/server/migrations.test.ts +++ b/x-pack/plugins/lens/server/migrations.test.ts @@ -670,4 +670,168 @@ describe('Lens migrations', () => { expect(result.attributes.title).toEqual(example.attributes.title); }); }); + + describe('7.13.0 rename operations for Formula', () => { + const context = ({ log: { warning: () => {} } } as unknown) as SavedObjectMigrationContext; + const example = { + type: 'lens', + id: 'mocked-saved-object-id', + attributes: { + savedObjectId: '21c145c0-8667-11eb-b6a9-a5bf52bdf519', + title: 'MyRenamedOps', + description: '', + visualizationType: 'lnsXY', + state: { + datasourceStates: { + indexpattern: { + layers: { + '5ab74ddc-93ca-44e2-9857-ecf85c86b53e': { + columns: { + '2e57a41e-5a52-42d3-877f-bd211d903ef8': { + label: '@timestamp', + dataType: 'date', + operationType: 'date_histogram', + sourceField: '@timestamp', + isBucketed: true, + scale: 'interval', + params: { interval: 'auto' }, + }, + '5fea2a56-7b73-44b5-9a50-7f0c0c4f8fd0': { + label: 'Unique count of agent.keyword', + dataType: 'number', + operationType: 'cardinality', + scale: 'ratio', + sourceField: 'agent.keyword', + isBucketed: false, + }, + 'e5efca70-edb5-4d6d-a30a-79384066987e': { + label: 'Average of bytes', + dataType: 'number', + operationType: 'avg', + sourceField: 'bytes', + isBucketed: false, + scale: 'ratio', + }, + '7ffb7bde-4f42-47ab-b74d-1b4fd8393e0f': { + label: 'Differences of bytes', + dataType: 'number', + operationType: 'derivative', + isBucketed: false, + scale: 'ratio', + references: ['9ca33a9b-f2e6-46ef-a5e1-14bfbe262605'], + }, + '9ca33a9b-f2e6-46ef-a5e1-14bfbe262605': { + label: 'Average of bytes', + dataType: 'number', + operationType: 'avg', + sourceField: 'bytes', + isBucketed: false, + scale: 'ratio', + }, + }, + columnOrder: [ + '2e57a41e-5a52-42d3-877f-bd211d903ef8', + '5fea2a56-7b73-44b5-9a50-7f0c0c4f8fd0', + 'e5efca70-edb5-4d6d-a30a-79384066987e', + '9ca33a9b-f2e6-46ef-a5e1-14bfbe262605', + '7ffb7bde-4f42-47ab-b74d-1b4fd8393e0f', + ], + incompleteColumns: {}, + }, + }, + }, + }, + visualization: { + title: 'Empty XY chart', + legend: { isVisible: true, position: 'right' }, + valueLabels: 'hide', + preferredSeriesType: 'bar_stacked', + layers: [ + { + layerId: '5ab74ddc-93ca-44e2-9857-ecf85c86b53e', + accessors: [ + '5fea2a56-7b73-44b5-9a50-7f0c0c4f8fd0', + 'e5efca70-edb5-4d6d-a30a-79384066987e', + '7ffb7bde-4f42-47ab-b74d-1b4fd8393e0f', + ], + position: 'top', + seriesType: 'bar_stacked', + showGridlines: false, + xAccessor: '2e57a41e-5a52-42d3-877f-bd211d903ef8', + }, + ], + }, + query: { query: '', language: 'kuery' }, + filters: [], + }, + }, + }; + + it('should rename only specific operation types', () => { + const result = migrations['7.13.0'](example, context) as ReturnType< + SavedObjectMigrationFn + >; + const layers = result.attributes.state.datasourceStates.indexpattern.layers; + expect(layers).toEqual({ + '5ab74ddc-93ca-44e2-9857-ecf85c86b53e': { + columns: { + '2e57a41e-5a52-42d3-877f-bd211d903ef8': { + label: '@timestamp', + dataType: 'date', + operationType: 'date_histogram', + sourceField: '@timestamp', + isBucketed: true, + scale: 'interval', + params: { interval: 'auto' }, + }, + '5fea2a56-7b73-44b5-9a50-7f0c0c4f8fd0': { + label: 'Unique count of agent.keyword', + dataType: 'number', + operationType: 'unique_count', + scale: 'ratio', + sourceField: 'agent.keyword', + isBucketed: false, + }, + 'e5efca70-edb5-4d6d-a30a-79384066987e': { + label: 'Average of bytes', + dataType: 'number', + operationType: 'average', + sourceField: 'bytes', + isBucketed: false, + scale: 'ratio', + }, + '7ffb7bde-4f42-47ab-b74d-1b4fd8393e0f': { + label: 'Differences of bytes', + dataType: 'number', + operationType: 'differences', + isBucketed: false, + scale: 'ratio', + references: ['9ca33a9b-f2e6-46ef-a5e1-14bfbe262605'], + }, + '9ca33a9b-f2e6-46ef-a5e1-14bfbe262605': { + label: 'Average of bytes', + dataType: 'number', + operationType: 'average', + sourceField: 'bytes', + isBucketed: false, + scale: 'ratio', + }, + }, + columnOrder: [ + '2e57a41e-5a52-42d3-877f-bd211d903ef8', + '5fea2a56-7b73-44b5-9a50-7f0c0c4f8fd0', + 'e5efca70-edb5-4d6d-a30a-79384066987e', + '9ca33a9b-f2e6-46ef-a5e1-14bfbe262605', + '7ffb7bde-4f42-47ab-b74d-1b4fd8393e0f', + ], + incompleteColumns: {}, + }, + }); + // should leave other parts alone + expect(result.attributes.state.visualization).toEqual(example.attributes.state.visualization); + expect(result.attributes.state.query).toEqual(example.attributes.state.query); + expect(result.attributes.state.filters).toEqual(example.attributes.state.filters); + expect(result.attributes.title).toEqual(example.attributes.title); + }); + }); }); diff --git a/x-pack/plugins/lens/server/migrations.ts b/x-pack/plugins/lens/server/migrations.ts index 4c6dfcd7949be..430c1a6caa667 100644 --- a/x-pack/plugins/lens/server/migrations.ts +++ b/x-pack/plugins/lens/server/migrations.ts @@ -106,6 +106,86 @@ interface DatatableStatePost711 { }; } +type OperationTypePre712 = + | 'avg' + | 'cardinality' + | 'derivative' + | 'filters' + | 'terms' + | 'date_histogram' + | 'min' + | 'max' + | 'sum' + | 'median' + | 'percentile' + | 'last_value' + | 'count' + | 'range' + | 'cumulative_sum' + | 'counter_rate' + | 'moving_average'; +type OperationTypePost712 = Exclude< + OperationTypePre712 | 'average' | 'unique_count' | 'differences', + 'avg' | 'cardinality' | 'derivative' +>; +interface LensDocShapePre712 { + visualizationType: string | null; + title: string; + expression: string | null; + state: { + datasourceStates: { + // This is hardcoded as our only datasource + indexpattern: { + layers: Record< + string, + { + columns: Record< + string, + { + operationType: OperationTypePre712; + } + >; + } + >; + }; + }; + visualization: VisualizationState; + query: Query; + filters: Filter[]; + }; +} + +interface LensDocShapePost712 { + visualizationType: string | null; + title: string; + expression: string | null; + state: { + datasourceMetaData: { + filterableIndexPatterns: Array<{ id: string; title: string }>; + }; + datasourceStates: { + // This is hardcoded as our only datasource + indexpattern: { + currentIndexPatternId: string; + layers: Record< + string, + { + columns: Record< + string, + { + operationType: OperationTypePost712; + } + >; + } + >; + }; + }; + visualization: VisualizationState; + query: Query; + filters: Filter[]; + }; +} + /** * Removes the `lens_auto_date` subexpression from a stored expression * string. For example: aggConfigs={lens_auto_date aggConfigs="JSON string"} @@ -387,6 +467,44 @@ const transformTableState: SavedObjectMigrationFn< return newDoc; }; +const renameOperationsForFormula: SavedObjectMigrationFn< + LensDocShapePre712, + LensDocShapePost712 +> = (doc) => { + const renameMapping = { + avg: 'average', + cardinality: 'unique_count', + derivative: 'differences', + } as const; + function shouldBeRenamed(op: OperationTypePre712): op is keyof typeof renameMapping { + return op in renameMapping; + } + const newDoc = cloneDeep(doc); + const datasourceLayers = newDoc.attributes.state.datasourceStates.indexpattern.layers || {}; + (newDoc.attributes as LensDocShapePost712).state.datasourceStates.indexpattern.layers = Object.fromEntries( + Object.entries(datasourceLayers).map(([layerId, layer]) => { + return [ + layerId, + { + ...layer, + columns: Object.fromEntries( + Object.entries(layer.columns).map(([columnId, column]) => { + const copy = { + ...column, + operationType: shouldBeRenamed(column.operationType) + ? renameMapping[column.operationType] + : column.operationType, + }; + return [columnId, copy]; + }) + ), + }, + ]; + }) + ); + return newDoc as SavedObjectUnsanitizedDoc; +}; + export const migrations: SavedObjectMigrationMap = { '7.7.0': removeInvalidAccessors, // The order of these migrations matter, since the timefield migration relies on the aggConfigs @@ -395,4 +513,5 @@ export const migrations: SavedObjectMigrationMap = { '7.10.0': extractReferences, '7.11.0': removeSuggestedPriority, '7.12.0': transformTableState, + '7.13.0': renameOperationsForFormula, }; diff --git a/x-pack/plugins/lists/server/index.ts b/x-pack/plugins/lists/server/index.ts index 1ebdf9f04bf9d..250b5e79ed109 100644 --- a/x-pack/plugins/lists/server/index.ts +++ b/x-pack/plugins/lists/server/index.ts @@ -12,7 +12,10 @@ import { ListPlugin } from './plugin'; // exporting these since its required at top level in siem plugin export { ListClient } from './services/lists/list_client'; -export { CreateExceptionListItemOptions } from './services/exception_lists/exception_list_client_types'; +export { + CreateExceptionListItemOptions, + UpdateExceptionListItemOptions, +} from './services/exception_lists/exception_list_client_types'; export { ExceptionListClient } from './services/exception_lists/exception_list_client'; export type { ListPluginSetup, ListsApiRequestHandlerContext } from './types'; diff --git a/x-pack/plugins/maps/common/elasticsearch_util/index.ts b/x-pack/plugins/maps/common/elasticsearch_util/index.ts index 0b6eaa435264c..24dd56b217401 100644 --- a/x-pack/plugins/maps/common/elasticsearch_util/index.ts +++ b/x-pack/plugins/maps/common/elasticsearch_util/index.ts @@ -8,3 +8,4 @@ export * from './es_agg_utils'; export * from './convert_to_geojson'; export * from './elasticsearch_geo_utils'; +export { isTotalHitsGreaterThan, TotalHits } from './total_hits'; diff --git a/x-pack/plugins/maps/common/elasticsearch_util/total_hits.test.ts b/x-pack/plugins/maps/common/elasticsearch_util/total_hits.test.ts new file mode 100644 index 0000000000000..211cb2d302f2c --- /dev/null +++ b/x-pack/plugins/maps/common/elasticsearch_util/total_hits.test.ts @@ -0,0 +1,48 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { isTotalHitsGreaterThan, TotalHits } from './total_hits'; + +describe('total.relation: eq', () => { + const totalHits = { + value: 100, + relation: 'eq' as TotalHits['relation'], + }; + + test('total.value: 100 should be more than 90', () => { + expect(isTotalHitsGreaterThan(totalHits, 90)).toBe(true); + }); + + test('total.value: 100 should not be more than 100', () => { + expect(isTotalHitsGreaterThan(totalHits, 100)).toBe(false); + }); + + test('total.value: 100 should not be more than 110', () => { + expect(isTotalHitsGreaterThan(totalHits, 110)).toBe(false); + }); +}); + +describe('total.relation: gte', () => { + const totalHits = { + value: 100, + relation: 'gte' as TotalHits['relation'], + }; + + test('total.value: 100 should be more than 90', () => { + expect(isTotalHitsGreaterThan(totalHits, 90)).toBe(true); + }); + + test('total.value: 100 should be more than 100', () => { + expect(isTotalHitsGreaterThan(totalHits, 100)).toBe(true); + }); + + test('total.value: 100 should throw error when value is more than 100', () => { + expect(() => { + isTotalHitsGreaterThan(totalHits, 110); + }).toThrow(); + }); +}); diff --git a/x-pack/plugins/maps/common/elasticsearch_util/total_hits.ts b/x-pack/plugins/maps/common/elasticsearch_util/total_hits.ts new file mode 100644 index 0000000000000..5de38d3f28851 --- /dev/null +++ b/x-pack/plugins/maps/common/elasticsearch_util/total_hits.ts @@ -0,0 +1,33 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { i18n } from '@kbn/i18n'; + +export interface TotalHits { + value: number; + relation: 'eq' | 'gte'; +} + +export function isTotalHitsGreaterThan(totalHits: TotalHits, value: number) { + if (totalHits.relation === 'eq') { + return totalHits.value > value; + } + + if (value > totalHits.value) { + throw new Error( + i18n.translate('xpack.maps.totalHits.lowerBoundPrecisionExceeded', { + defaultMessage: `Unable to determine if total hits is greater than value. Total hits precision is lower then value. Total hits: {totalHitsString}, value: {value}. Ensure _search.body.track_total_hits is at least as large as value.`, + values: { + totalHitsString: JSON.stringify(totalHits, null, ''), + value, + }, + }) + ); + } + + return true; +} diff --git a/x-pack/plugins/maps/public/classes/layers/blended_vector_layer/blended_vector_layer.ts b/x-pack/plugins/maps/public/classes/layers/blended_vector_layer/blended_vector_layer.ts index d795315acbf50..6dd454137be7d 100644 --- a/x-pack/plugins/maps/public/classes/layers/blended_vector_layer/blended_vector_layer.ts +++ b/x-pack/plugins/maps/public/classes/layers/blended_vector_layer/blended_vector_layer.ts @@ -22,6 +22,7 @@ import { LAYER_STYLE_TYPE, FIELD_ORIGIN, } from '../../../../common/constants'; +import { isTotalHitsGreaterThan, TotalHits } from '../../../../common/elasticsearch_util'; import { ESGeoGridSource } from '../../sources/es_geo_grid_source/es_geo_grid_source'; import { canSkipSourceUpdate } from '../../util/can_skip_fetch'; import { IESSource } from '../../sources/es_source'; @@ -323,13 +324,18 @@ export class BlendedVectorLayer extends VectorLayer implements IVectorLayer { syncContext.startLoading(dataRequestId, requestToken, searchFilters); const abortController = new AbortController(); syncContext.registerCancelCallback(requestToken, () => abortController.abort()); + const maxResultWindow = await this._documentSource.getMaxResultWindow(); const searchSource = await this._documentSource.makeSearchSource(searchFilters, 0); + searchSource.setField('trackTotalHits', maxResultWindow + 1); const resp = await searchSource.fetch({ abortSignal: abortController.signal, sessionId: syncContext.dataFilters.searchSessionId, + legacyHitsTotal: false, }); - const maxResultWindow = await this._documentSource.getMaxResultWindow(); - isSyncClustered = resp.hits.total > maxResultWindow; + isSyncClustered = isTotalHitsGreaterThan( + (resp.hits.total as unknown) as TotalHits, + maxResultWindow + ); const countData = { isSyncClustered } as CountData; syncContext.stopLoading(dataRequestId, requestToken, countData, searchFilters); } catch (error) { diff --git a/x-pack/plugins/maps/public/classes/sources/es_geo_grid_source/es_geo_grid_source.tsx b/x-pack/plugins/maps/public/classes/sources/es_geo_grid_source/es_geo_grid_source.tsx index 4715398dab97b..7910e931e60e6 100644 --- a/x-pack/plugins/maps/public/classes/sources/es_geo_grid_source/es_geo_grid_source.tsx +++ b/x-pack/plugins/maps/public/classes/sources/es_geo_grid_source/es_geo_grid_source.tsx @@ -368,6 +368,7 @@ export class ESGeoGridSource extends AbstractESAggSource implements ITiledSingle ): Promise { const indexPattern: IndexPattern = await this.getIndexPattern(); const searchSource: ISearchSource = await this.makeSearchSource(searchFilters, 0); + searchSource.setField('trackTotalHits', false); let bucketsPerGrid = 1; this.getMetricFields().forEach((metricField) => { diff --git a/x-pack/plugins/maps/public/classes/sources/es_geo_line_source/es_geo_line_source.tsx b/x-pack/plugins/maps/public/classes/sources/es_geo_line_source/es_geo_line_source.tsx index c652935d7188a..9a1f23e055af1 100644 --- a/x-pack/plugins/maps/public/classes/sources/es_geo_line_source/es_geo_line_source.tsx +++ b/x-pack/plugins/maps/public/classes/sources/es_geo_line_source/es_geo_line_source.tsx @@ -190,6 +190,7 @@ export class ESGeoLineSource extends AbstractESAggSource { // Fetch entities // const entitySearchSource = await this.makeSearchSource(searchFilters, 0); + entitySearchSource.setField('trackTotalHits', false); const splitField = getField(indexPattern, this._descriptor.splitField); const cardinalityAgg = { precision_threshold: 1 }; const termsAgg = { size: MAX_TRACKS }; @@ -250,6 +251,7 @@ export class ESGeoLineSource extends AbstractESAggSource { const tracksSearchFilters = { ...searchFilters }; delete tracksSearchFilters.buffer; const tracksSearchSource = await this.makeSearchSource(tracksSearchFilters, 0); + tracksSearchSource.setField('trackTotalHits', false); tracksSearchSource.setField('aggs', { tracks: { filters: { diff --git a/x-pack/plugins/maps/public/classes/sources/es_pew_pew_source/es_pew_pew_source.js b/x-pack/plugins/maps/public/classes/sources/es_pew_pew_source/es_pew_pew_source.js index e3ee9599d86a9..781cc7f8c36b0 100644 --- a/x-pack/plugins/maps/public/classes/sources/es_pew_pew_source/es_pew_pew_source.js +++ b/x-pack/plugins/maps/public/classes/sources/es_pew_pew_source/es_pew_pew_source.js @@ -109,6 +109,7 @@ export class ESPewPewSource extends AbstractESAggSource { async getGeoJsonWithMeta(layerName, searchFilters, registerCancelCallback) { const indexPattern = await this.getIndexPattern(); const searchSource = await this.makeSearchSource(searchFilters, 0); + searchSource.setField('trackTotalHits', false); searchSource.setField('aggs', { destSplit: { terms: { @@ -168,6 +169,7 @@ export class ESPewPewSource extends AbstractESAggSource { async getBoundsForFilters(boundsFilters, registerCancelCallback) { const searchSource = await this.makeSearchSource(boundsFilters, 0); + searchSource.setField('trackTotalHits', false); searchSource.setField('aggs', { destFitToBounds: { geo_bounds: { @@ -185,7 +187,10 @@ export class ESPewPewSource extends AbstractESAggSource { try { const abortController = new AbortController(); registerCancelCallback(() => abortController.abort()); - const esResp = await searchSource.fetch({ abortSignal: abortController.signal }); + const esResp = await searchSource.fetch({ + abortSignal: abortController.signal, + legacyHitsTotal: false, + }); if (esResp.aggregations.destFitToBounds.bounds) { corners.push([ esResp.aggregations.destFitToBounds.bounds.top_left.lon, diff --git a/x-pack/plugins/maps/public/classes/sources/es_search_source/es_search_source.tsx b/x-pack/plugins/maps/public/classes/sources/es_search_source/es_search_source.tsx index 3b6a7202691b6..eae00710c4c25 100644 --- a/x-pack/plugins/maps/public/classes/sources/es_search_source/es_search_source.tsx +++ b/x-pack/plugins/maps/public/classes/sources/es_search_source/es_search_source.tsx @@ -18,6 +18,7 @@ import { addFieldToDSL, getField, hitsToGeoJson, + isTotalHitsGreaterThan, PreIndexedShape, } from '../../../../common/elasticsearch_util'; // @ts-expect-error @@ -313,6 +314,7 @@ export class ESSearchSource extends AbstractESSource implements ITiledSingleLaye }; const searchSource = await this.makeSearchSource(searchFilters, 0); + searchSource.setField('trackTotalHits', false); searchSource.setField('aggs', { totalEntities: { cardinality: addFieldToDSL(cardinalityAgg, topHitsSplitField), @@ -343,11 +345,10 @@ export class ESSearchSource extends AbstractESSource implements ITiledSingleLaye const areEntitiesTrimmed = entityBuckets.length >= DEFAULT_MAX_BUCKETS_LIMIT; let areTopHitsTrimmed = false; entityBuckets.forEach((entityBucket: any) => { - const total = _.get(entityBucket, 'entityHits.hits.total', 0); const hits = _.get(entityBucket, 'entityHits.hits.hits', []); // Reverse hits list so top documents by sort are drawn on top allHits.push(...hits.reverse()); - if (total > hits.length) { + if (isTotalHitsGreaterThan(entityBucket.entityHits.hits.total, hits.length)) { areTopHitsTrimmed = true; } }); @@ -385,6 +386,7 @@ export class ESSearchSource extends AbstractESSource implements ITiledSingleLaye maxResultWindow, initialSearchContext ); + searchSource.setField('trackTotalHits', maxResultWindow + 1); searchSource.setField('fieldsFromSource', searchFilters.fieldNames); // Setting "fields" filters out unused scripted fields if (sourceOnlyFields.length === 0) { searchSource.setField('source', false); // do not need anything from _source @@ -408,7 +410,7 @@ export class ESSearchSource extends AbstractESSource implements ITiledSingleLaye hits: resp.hits.hits.reverse(), // Reverse hits so top documents by sort are drawn on top meta: { resultsCount: resp.hits.hits.length, - areResultsTrimmed: resp.hits.total > resp.hits.hits.length, + areResultsTrimmed: isTotalHitsGreaterThan(resp.hits.total, resp.hits.hits.length), }, }; } @@ -508,6 +510,7 @@ export class ESSearchSource extends AbstractESSource implements ITiledSingleLaye const initialSearchContext = { docvalue_fields: docValueFields }; // Request fields in docvalue_fields insted of _source const searchService = getSearchService(); const searchSource = await searchService.searchSource.create(initialSearchContext as object); + searchSource.setField('trackTotalHits', false); searchSource.setField('index', indexPattern); searchSource.setField('size', 1); @@ -520,7 +523,7 @@ export class ESSearchSource extends AbstractESSource implements ITiledSingleLaye searchSource.setField('query', query); searchSource.setField('fieldsFromSource', this._getTooltipPropertyNames()); - const resp = await searchSource.fetch(); + const resp = await searchSource.fetch({ legacyHitsTotal: false }); const hit = _.get(resp, 'hits.hits[0]'); if (!hit) { diff --git a/x-pack/plugins/maps/public/classes/sources/es_source/es_source.ts b/x-pack/plugins/maps/public/classes/sources/es_source/es_source.ts index c55a564951c4e..222c49abfa16a 100644 --- a/x-pack/plugins/maps/public/classes/sources/es_source/es_source.ts +++ b/x-pack/plugins/maps/public/classes/sources/es_source/es_source.ts @@ -195,6 +195,7 @@ export class AbstractESSource extends AbstractVectorSource implements IESSource resp = await searchSource.fetch({ abortSignal: abortController.signal, sessionId: searchSessionId, + legacyHitsTotal: false, }); if (inspectorRequest) { const responseStats = search.getResponseInspectorStats(resp, searchSource); @@ -247,6 +248,7 @@ export class AbstractESSource extends AbstractVectorSource implements IESSource } } const searchService = getSearchService(); + const searchSource = await searchService.searchSource.create(initialSearchContext); searchSource.setField('index', indexPattern); @@ -272,6 +274,7 @@ export class AbstractESSource extends AbstractVectorSource implements IESSource registerCancelCallback: (callback: () => void) => void ): Promise { const searchSource = await this.makeSearchSource(boundsFilters, 0); + searchSource.setField('trackTotalHits', false); searchSource.setField('aggs', { fitToBounds: { geo_bounds: { @@ -284,7 +287,10 @@ export class AbstractESSource extends AbstractVectorSource implements IESSource try { const abortController = new AbortController(); registerCancelCallback(() => abortController.abort()); - const esResp = await searchSource.fetch({ abortSignal: abortController.signal }); + const esResp = await searchSource.fetch({ + abortSignal: abortController.signal, + legacyHitsTotal: false, + }); if (!esResp.aggregations) { return null; diff --git a/x-pack/plugins/maps/public/classes/sources/es_term_source/es_term_source.ts b/x-pack/plugins/maps/public/classes/sources/es_term_source/es_term_source.ts index 5c41971fb629c..caae4385aeec6 100644 --- a/x-pack/plugins/maps/public/classes/sources/es_term_source/es_term_source.ts +++ b/x-pack/plugins/maps/public/classes/sources/es_term_source/es_term_source.ts @@ -127,6 +127,7 @@ export class ESTermSource extends AbstractESAggSource implements ITermJoinSource const indexPattern = await this.getIndexPattern(); const searchSource: ISearchSource = await this.makeSearchSource(searchFilters, 0); + searchSource.setField('trackTotalHits', false); const termsField = getField(indexPattern, this._termField.getName()); const termsAgg = { size: this._descriptor.size !== undefined ? this._descriptor.size : DEFAULT_MAX_BUCKETS_LIMIT, diff --git a/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_control.tsx b/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_control.tsx index f68875dc81394..a1bea4a8e93dc 100644 --- a/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_control.tsx +++ b/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_control.tsx @@ -12,20 +12,10 @@ import MapboxDraw from '@mapbox/mapbox-gl-draw'; // @ts-expect-error import DrawRectangle from 'mapbox-gl-draw-rectangle-mode'; import { Map as MbMap } from 'mapbox-gl'; -import { i18n } from '@kbn/i18n'; -import { Filter } from 'src/plugins/data/public'; -import { Feature, Polygon } from 'geojson'; -import { DRAW_TYPE, ES_GEO_FIELD_TYPE, ES_SPATIAL_RELATIONS } from '../../../../common/constants'; -import { DrawState } from '../../../../common/descriptor_types'; -import { DrawCircle, DrawCircleProperties } from './draw_circle'; -import { - createDistanceFilterWithMeta, - createSpatialFilterWithGeometry, - getBoundingBoxGeometry, - roundCoordinates, -} from '../../../../common/elasticsearch_util'; +import { Feature } from 'geojson'; +import { DRAW_TYPE } from '../../../../common/constants'; +import { DrawCircle } from './draw_circle'; import { DrawTooltip } from './draw_tooltip'; -import { getToasts } from '../../../kibana_services'; const DRAW_RECTANGLE = 'draw_rectangle'; const DRAW_CIRCLE = 'draw_circle'; @@ -35,10 +25,8 @@ mbDrawModes[DRAW_RECTANGLE] = DrawRectangle; mbDrawModes[DRAW_CIRCLE] = DrawCircle; export interface Props { - addFilters: (filters: Filter[], actionId: string) => Promise; - disableDrawState: () => void; - drawState?: DrawState; - isDrawingFilter: boolean; + drawType?: DRAW_TYPE; + onDraw: (event: { features: Feature[] }) => void; mbMap: MbMap; } @@ -70,100 +58,26 @@ export class DrawControl extends Component { return; } - if (this.props.isDrawingFilter) { + if (this.props.drawType) { this._updateDrawControl(); } else { this._removeDrawControl(); } }, 0); - _onDraw = async (e: { features: Feature[] }) => { - if ( - !e.features.length || - !this.props.drawState || - !this.props.drawState.geoFieldName || - !this.props.drawState.indexPatternId - ) { - return; - } - - let filter: Filter | undefined; - if (this.props.drawState.drawType === DRAW_TYPE.DISTANCE) { - const circle = e.features[0] as Feature & { properties: DrawCircleProperties }; - const distanceKm = _.round( - circle.properties.radiusKm, - circle.properties.radiusKm > 10 ? 0 : 2 - ); - // Only include as much precision as needed for distance - let precision = 2; - if (distanceKm <= 1) { - precision = 5; - } else if (distanceKm <= 10) { - precision = 4; - } else if (distanceKm <= 100) { - precision = 3; - } - filter = createDistanceFilterWithMeta({ - alias: this.props.drawState.filterLabel ? this.props.drawState.filterLabel : '', - distanceKm, - geoFieldName: this.props.drawState.geoFieldName, - indexPatternId: this.props.drawState.indexPatternId, - point: [ - _.round(circle.properties.center[0], precision), - _.round(circle.properties.center[1], precision), - ], - }); - } else { - const geometry = e.features[0].geometry as Polygon; - // MapboxDraw returns coordinates with 12 decimals. Round to a more reasonable number - roundCoordinates(geometry.coordinates); - - filter = createSpatialFilterWithGeometry({ - geometry: - this.props.drawState.drawType === DRAW_TYPE.BOUNDS - ? getBoundingBoxGeometry(geometry) - : geometry, - indexPatternId: this.props.drawState.indexPatternId, - geoFieldName: this.props.drawState.geoFieldName, - geoFieldType: this.props.drawState.geoFieldType - ? this.props.drawState.geoFieldType - : ES_GEO_FIELD_TYPE.GEO_POINT, - geometryLabel: this.props.drawState.geometryLabel ? this.props.drawState.geometryLabel : '', - relation: this.props.drawState.relation - ? this.props.drawState.relation - : ES_SPATIAL_RELATIONS.INTERSECTS, - }); - } - - try { - await this.props.addFilters([filter!], this.props.drawState.actionId); - } catch (error) { - getToasts().addWarning( - i18n.translate('xpack.maps.drawControl.unableToCreatFilter', { - defaultMessage: `Unable to create filter, error: '{errorMsg}'.`, - values: { - errorMsg: error.message, - }, - }) - ); - } finally { - this.props.disableDrawState(); - } - }; - _removeDrawControl() { if (!this._mbDrawControlAdded) { return; } this.props.mbMap.getCanvas().style.cursor = ''; - this.props.mbMap.off('draw.create', this._onDraw); + this.props.mbMap.off('draw.create', this.props.onDraw); this.props.mbMap.removeControl(this._mbDrawControl); this._mbDrawControlAdded = false; } _updateDrawControl() { - if (!this.props.drawState) { + if (!this.props.drawType) { return; } @@ -171,27 +85,27 @@ export class DrawControl extends Component { this.props.mbMap.addControl(this._mbDrawControl); this._mbDrawControlAdded = true; this.props.mbMap.getCanvas().style.cursor = 'crosshair'; - this.props.mbMap.on('draw.create', this._onDraw); + this.props.mbMap.on('draw.create', this.props.onDraw); } const drawMode = this._mbDrawControl.getMode(); - if (drawMode !== DRAW_RECTANGLE && this.props.drawState.drawType === DRAW_TYPE.BOUNDS) { + if (drawMode !== DRAW_RECTANGLE && this.props.drawType === DRAW_TYPE.BOUNDS) { this._mbDrawControl.changeMode(DRAW_RECTANGLE); - } else if (drawMode !== DRAW_CIRCLE && this.props.drawState.drawType === DRAW_TYPE.DISTANCE) { + } else if (drawMode !== DRAW_CIRCLE && this.props.drawType === DRAW_TYPE.DISTANCE) { this._mbDrawControl.changeMode(DRAW_CIRCLE); } else if ( drawMode !== this._mbDrawControl.modes.DRAW_POLYGON && - this.props.drawState.drawType === DRAW_TYPE.POLYGON + this.props.drawType === DRAW_TYPE.POLYGON ) { this._mbDrawControl.changeMode(this._mbDrawControl.modes.DRAW_POLYGON); } } render() { - if (!this.props.isDrawingFilter || !this.props.drawState) { + if (!this.props.drawType) { return null; } - return ; + return ; } } diff --git a/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_filter_control/draw_filter_control.tsx b/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_filter_control/draw_filter_control.tsx new file mode 100644 index 0000000000000..c0cbd3566ca8f --- /dev/null +++ b/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_filter_control/draw_filter_control.tsx @@ -0,0 +1,126 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import _ from 'lodash'; +import React, { Component } from 'react'; +import { Map as MbMap } from 'mapbox-gl'; +import { i18n } from '@kbn/i18n'; +import { Filter } from 'src/plugins/data/public'; +import { Feature, Polygon } from 'geojson'; +import { + DRAW_TYPE, + ES_GEO_FIELD_TYPE, + ES_SPATIAL_RELATIONS, +} from '../../../../../common/constants'; +import { DrawState } from '../../../../../common/descriptor_types'; +import { + createDistanceFilterWithMeta, + createSpatialFilterWithGeometry, + getBoundingBoxGeometry, + roundCoordinates, +} from '../../../../../common/elasticsearch_util'; +import { getToasts } from '../../../../kibana_services'; +import { DrawControl } from '../draw_control'; +import { DrawCircleProperties } from '../draw_circle'; + +export interface Props { + addFilters: (filters: Filter[], actionId: string) => Promise; + disableDrawState: () => void; + drawState?: DrawState; + isDrawingFilter: boolean; + mbMap: MbMap; +} + +export class DrawFilterControl extends Component { + _onDraw = async (e: { features: Feature[] }) => { + if ( + !e.features.length || + !this.props.drawState || + !this.props.drawState.geoFieldName || + !this.props.drawState.indexPatternId + ) { + return; + } + + let filter: Filter | undefined; + if (this.props.drawState.drawType === DRAW_TYPE.DISTANCE) { + const circle = e.features[0] as Feature & { properties: DrawCircleProperties }; + const distanceKm = _.round( + circle.properties.radiusKm, + circle.properties.radiusKm > 10 ? 0 : 2 + ); + // Only include as much precision as needed for distance + let precision = 2; + if (distanceKm <= 1) { + precision = 5; + } else if (distanceKm <= 10) { + precision = 4; + } else if (distanceKm <= 100) { + precision = 3; + } + filter = createDistanceFilterWithMeta({ + alias: this.props.drawState.filterLabel ? this.props.drawState.filterLabel : '', + distanceKm, + geoFieldName: this.props.drawState.geoFieldName, + indexPatternId: this.props.drawState.indexPatternId, + point: [ + _.round(circle.properties.center[0], precision), + _.round(circle.properties.center[1], precision), + ], + }); + } else { + const geometry = e.features[0].geometry as Polygon; + // MapboxDraw returns coordinates with 12 decimals. Round to a more reasonable number + roundCoordinates(geometry.coordinates); + + filter = createSpatialFilterWithGeometry({ + geometry: + this.props.drawState.drawType === DRAW_TYPE.BOUNDS + ? getBoundingBoxGeometry(geometry) + : geometry, + indexPatternId: this.props.drawState.indexPatternId, + geoFieldName: this.props.drawState.geoFieldName, + geoFieldType: this.props.drawState.geoFieldType + ? this.props.drawState.geoFieldType + : ES_GEO_FIELD_TYPE.GEO_POINT, + geometryLabel: this.props.drawState.geometryLabel ? this.props.drawState.geometryLabel : '', + relation: this.props.drawState.relation + ? this.props.drawState.relation + : ES_SPATIAL_RELATIONS.INTERSECTS, + }); + } + + try { + await this.props.addFilters([filter!], this.props.drawState.actionId); + } catch (error) { + getToasts().addWarning( + i18n.translate('xpack.maps.drawFilterControl.unableToCreatFilter', { + defaultMessage: `Unable to create filter, error: '{errorMsg}'.`, + values: { + errorMsg: error.message, + }, + }) + ); + } finally { + this.props.disableDrawState(); + } + }; + + render() { + return ( + + ); + } +} diff --git a/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_filter_control/index.ts b/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_filter_control/index.ts new file mode 100644 index 0000000000000..17f4d919fb7e0 --- /dev/null +++ b/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_filter_control/index.ts @@ -0,0 +1,32 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { AnyAction } from 'redux'; +import { ThunkDispatch } from 'redux-thunk'; +import { connect } from 'react-redux'; +import { DrawFilterControl } from './draw_filter_control'; +import { updateDrawState } from '../../../../actions'; +import { getDrawState, isDrawingFilter } from '../../../../selectors/map_selectors'; +import { MapStoreState } from '../../../../reducers/store'; + +function mapStateToProps(state: MapStoreState) { + return { + isDrawingFilter: isDrawingFilter(state), + drawState: getDrawState(state), + }; +} + +function mapDispatchToProps(dispatch: ThunkDispatch) { + return { + disableDrawState() { + dispatch(updateDrawState(null)); + }, + }; +} + +const connected = connect(mapStateToProps, mapDispatchToProps)(DrawFilterControl); +export { connected as DrawFilterControl }; diff --git a/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_tooltip.tsx b/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_tooltip.tsx index 099f409c91c21..df650d5dfe410 100644 --- a/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_tooltip.tsx +++ b/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/draw_tooltip.tsx @@ -11,13 +11,12 @@ import { EuiPopover, EuiText } from '@elastic/eui'; import { i18n } from '@kbn/i18n'; import { Map as MbMap } from 'mapbox-gl'; import { DRAW_TYPE } from '../../../../common/constants'; -import { DrawState } from '../../../../common/descriptor_types'; const noop = () => {}; interface Props { mbMap: MbMap; - drawState: DrawState; + drawType: DRAW_TYPE; } interface State { @@ -58,16 +57,16 @@ export class DrawTooltip extends Component { } let instructions; - if (this.props.drawState.drawType === DRAW_TYPE.BOUNDS) { + if (this.props.drawType === DRAW_TYPE.BOUNDS) { instructions = i18n.translate('xpack.maps.drawTooltip.boundsInstructions', { defaultMessage: 'Click to start rectangle. Move mouse to adjust rectangle size. Click again to finish.', }); - } else if (this.props.drawState.drawType === DRAW_TYPE.DISTANCE) { + } else if (this.props.drawType === DRAW_TYPE.DISTANCE) { instructions = i18n.translate('xpack.maps.drawTooltip.distanceInstructions', { defaultMessage: 'Click to set point. Move mouse to adjust distance. Click to finish.', }); - } else if (this.props.drawState.drawType === DRAW_TYPE.POLYGON) { + } else if (this.props.drawType === DRAW_TYPE.POLYGON) { instructions = i18n.translate('xpack.maps.drawTooltip.polygonInstructions', { defaultMessage: 'Click to start shape. Click to add vertex. Double click to finish.', }); diff --git a/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/index.ts b/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/index.ts index cc2f560c63d24..63f91a03a5d01 100644 --- a/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/index.ts +++ b/x-pack/plugins/maps/public/connected_components/mb_map/draw_control/index.ts @@ -5,28 +5,4 @@ * 2.0. */ -import { AnyAction } from 'redux'; -import { ThunkDispatch } from 'redux-thunk'; -import { connect } from 'react-redux'; -import { DrawControl } from './draw_control'; -import { updateDrawState } from '../../../actions'; -import { getDrawState, isDrawingFilter } from '../../../selectors/map_selectors'; -import { MapStoreState } from '../../../reducers/store'; - -function mapStateToProps(state: MapStoreState) { - return { - isDrawingFilter: isDrawingFilter(state), - drawState: getDrawState(state), - }; -} - -function mapDispatchToProps(dispatch: ThunkDispatch) { - return { - disableDrawState() { - dispatch(updateDrawState(null)); - }, - }; -} - -const connected = connect(mapStateToProps, mapDispatchToProps)(DrawControl); -export { connected as DrawControl }; +export { DrawFilterControl } from './draw_filter_control'; diff --git a/x-pack/plugins/maps/public/connected_components/mb_map/mb_map.tsx b/x-pack/plugins/maps/public/connected_components/mb_map/mb_map.tsx index fae89a0484f11..5e4c3c9b1981f 100644 --- a/x-pack/plugins/maps/public/connected_components/mb_map/mb_map.tsx +++ b/x-pack/plugins/maps/public/connected_components/mb_map/mb_map.tsx @@ -17,7 +17,7 @@ import sprites2 from '@elastic/maki/dist/sprite@2.png'; import { Adapters } from 'src/plugins/inspector/public'; import { Filter } from 'src/plugins/data/public'; import { ActionExecutionContext, Action } from 'src/plugins/ui_actions/public'; -import { DrawControl } from './draw_control'; +import { DrawFilterControl } from './draw_control'; import { ScaleControl } from './scale_control'; // @ts-expect-error import { TooltipControl } from './tooltip_control'; @@ -418,7 +418,7 @@ export class MBMap extends Component { let scaleControl; if (this.state.mbMap) { drawControl = this.props.addFilters ? ( - + ) : null; tooltipControl = !this.props.settings.disableTooltipControl ? ( {}); +import { LAYER_STYLE_TYPE, LAYER_TYPE, SOURCE_TYPES } from '../../common'; + jest.mock('../classes/layers/tiled_vector_layer/tiled_vector_layer', () => {}); jest.mock('../classes/layers/blended_vector_layer/blended_vector_layer', () => {}); jest.mock('../classes/layers/heatmap_layer', () => {}); @@ -23,13 +24,23 @@ jest.mock('../kibana_services', () => ({ getMapsCapabilities() { return { save: true }; }, + getIsDarkMode() { + return false; + }, })); import { DEFAULT_MAP_STORE_STATE } from '../reducers/store'; -import { areLayersLoaded, getDataFilters, getTimeFilters } from './map_selectors'; -import { LayerDescriptor } from '../../common/descriptor_types'; +import { + areLayersLoaded, + getDataFilters, + getTimeFilters, + getQueryableUniqueIndexPatternIds, +} from './map_selectors'; + +import { LayerDescriptor, VectorLayerDescriptor } from '../../common/descriptor_types'; import { ILayer } from '../classes/layers/layer'; import { Filter } from '../../../../../src/plugins/data/public'; +import { ESSearchSource } from '../classes/sources/es_search_source'; describe('getDataFilters', () => { const mapExtent = { @@ -193,3 +204,76 @@ describe('areLayersLoaded', () => { expect(areLayersLoaded.resultFunc(layerList, waitingForMapReadyLayerList, zoom)).toBe(true); }); }); + +describe('getQueryableUniqueIndexPatternIds', () => { + function createLayerMock({ + isVisible = true, + indexPatterns = [], + }: { + isVisible?: boolean; + indexPatterns?: string[]; + }) { + return ({ + isVisible: () => { + return isVisible; + }, + getQueryableIndexPatternIds: () => { + return indexPatterns; + }, + } as unknown) as ILayer; + } + + function createWaitLayerDescriptorMock({ + indexPatternId, + visible = true, + }: { + visible?: boolean; + indexPatternId: string; + }) { + return { + type: LAYER_TYPE.VECTOR, + style: { + type: LAYER_STYLE_TYPE.VECTOR, + }, + visible, + sourceDescriptor: ESSearchSource.createDescriptor({ + type: SOURCE_TYPES.ES_SEARCH, + indexPatternId, + geoField: 'field', + }), + }; + } + + test('should only include visible', () => { + const layerList: ILayer[] = [ + createLayerMock({}), + createLayerMock({ indexPatterns: ['foo'] }), + createLayerMock({ indexPatterns: ['bar'] }), + createLayerMock({ indexPatterns: ['foobar'], isVisible: false }), + createLayerMock({ indexPatterns: ['bar'] }), + ]; + const waitingForMapReadyLayerList: VectorLayerDescriptor[] = ([] as unknown) as VectorLayerDescriptor[]; + expect( + getQueryableUniqueIndexPatternIds.resultFunc(layerList, waitingForMapReadyLayerList) + ).toEqual(['foo', 'bar']); + }); + + test('should only include visible and waitlist should take precedence', () => { + const layerList: ILayer[] = [ + createLayerMock({}), + createLayerMock({ indexPatterns: ['foo'] }), + createLayerMock({ indexPatterns: ['bar'] }), + createLayerMock({ indexPatterns: ['foobar'], isVisible: false }), + createLayerMock({ indexPatterns: ['bar'] }), + ]; + const waitingForMapReadyLayerList: VectorLayerDescriptor[] = ([ + createWaitLayerDescriptorMock({ indexPatternId: 'foo' }), + createWaitLayerDescriptorMock({ indexPatternId: 'barfoo', visible: false }), + createWaitLayerDescriptorMock({ indexPatternId: 'fbr' }), + createWaitLayerDescriptorMock({ indexPatternId: 'foo' }), + ] as unknown) as VectorLayerDescriptor[]; + expect( + getQueryableUniqueIndexPatternIds.resultFunc(layerList, waitingForMapReadyLayerList) + ).toEqual(['foo', 'fbr']); + }); +}); diff --git a/x-pack/plugins/maps/public/selectors/map_selectors.ts b/x-pack/plugins/maps/public/selectors/map_selectors.ts index 2f87aa7d11394..a818cdd2d00f9 100644 --- a/x-pack/plugins/maps/public/selectors/map_selectors.ts +++ b/x-pack/plugins/maps/public/selectors/map_selectors.ts @@ -377,16 +377,6 @@ export const getSelectedLayerJoinDescriptors = createSelector(getSelectedLayer, }); }); -// Get list of unique index patterns used by all layers -export const getUniqueIndexPatternIds = createSelector(getLayerList, (layerList) => { - const indexPatternIds: string[] = []; - layerList.forEach((layer) => { - indexPatternIds.push(...layer.getIndexPatternIds()); - }); - return _.uniq(indexPatternIds).sort(); -}); - -// Get list of unique index patterns, excluding index patterns from layers that disable applyGlobalQuery export const getQueryableUniqueIndexPatternIds = createSelector( getLayerList, getWaitingForMapReadyLayerListRaw, @@ -396,11 +386,15 @@ export const getQueryableUniqueIndexPatternIds = createSelector( if (waitingForMapReadyLayerList.length) { waitingForMapReadyLayerList.forEach((layerDescriptor) => { const layer = createLayerInstance(layerDescriptor); - indexPatternIds.push(...layer.getQueryableIndexPatternIds()); + if (layer.isVisible()) { + indexPatternIds.push(...layer.getQueryableIndexPatternIds()); + } }); } else { layerList.forEach((layer) => { - indexPatternIds.push(...layer.getQueryableIndexPatternIds()); + if (layer.isVisible()) { + indexPatternIds.push(...layer.getQueryableIndexPatternIds()); + } }); } return _.uniq(indexPatternIds); diff --git a/x-pack/plugins/maps/server/mvt/get_tile.ts b/x-pack/plugins/maps/server/mvt/get_tile.ts index d6ebf2fb216b2..95b8e043e0ce4 100644 --- a/x-pack/plugins/maps/server/mvt/get_tile.ts +++ b/x-pack/plugins/maps/server/mvt/get_tile.ts @@ -23,7 +23,12 @@ import { SUPER_FINE_ZOOM_DELTA, } from '../../common/constants'; -import { convertRegularRespToGeoJson, hitsToGeoJson } from '../../common/elasticsearch_util'; +import { + convertRegularRespToGeoJson, + hitsToGeoJson, + isTotalHitsGreaterThan, + TotalHits, +} from '../../common/elasticsearch_util'; import { flattenHit } from './util'; import { ESBounds, tileToESBbox } from '../../common/geo_tile_utils'; import { getCentroidFeatures } from '../../common/get_centroid_features'; @@ -67,6 +72,7 @@ export async function getGridTile({ MAX_ZOOM ); requestBody.aggs[GEOTILE_GRID_AGG_NAME].geotile_grid.bounds = tileBounds; + requestBody.track_total_hits = false; const response = await context .search!.search( @@ -78,6 +84,7 @@ export async function getGridTile({ }, { sessionId: searchSessionId, + legacyHitsTotal: false, abortSignal, } ) @@ -130,6 +137,7 @@ export async function getTile({ const searchOptions = { sessionId: searchSessionId, + legacyHitsTotal: false, abortSignal, }; @@ -141,6 +149,7 @@ export async function getTile({ body: { size: 0, query: requestBody.query, + track_total_hits: requestBody.size + 1, }, }, }, @@ -148,7 +157,12 @@ export async function getTile({ ) .toPromise(); - if (countResponse.rawResponse.hits.total > requestBody.size) { + if ( + isTotalHitsGreaterThan( + (countResponse.rawResponse.hits.total as unknown) as TotalHits, + requestBody.size + ) + ) { // Generate "too many features"-bounds const bboxResponse = await context .search!.search( @@ -165,6 +179,7 @@ export async function getTile({ }, }, }, + track_total_hits: false, }, }, }, @@ -191,7 +206,10 @@ export async function getTile({ { params: { index, - body: requestBody, + body: { + ...requestBody, + track_total_hits: false, + }, }, }, searchOptions diff --git a/x-pack/plugins/ml/public/application/datavisualizer/index_based/components/field_data_row/action_menu/lens_utils.ts b/x-pack/plugins/ml/public/application/datavisualizer/index_based/components/field_data_row/action_menu/lens_utils.ts index 8d078b59ad778..9690fd1b6c473 100644 --- a/x-pack/plugins/ml/public/application/datavisualizer/index_based/components/field_data_row/action_menu/lens_utils.ts +++ b/x-pack/plugins/ml/public/application/datavisualizer/index_based/components/field_data_row/action_menu/lens_utils.ts @@ -66,7 +66,7 @@ export function getNumberSettings(item: FieldVisConfig, defaultIndexPattern: IIn defaultMessage: 'Average of {fieldName}', values: { fieldName: item.fieldName }, }), - operationType: 'avg', + operationType: 'average', sourceField: item.fieldName!, }, col1: { diff --git a/x-pack/plugins/security/public/management/users/edit_user/create_user_page.test.tsx b/x-pack/plugins/security/public/management/users/edit_user/create_user_page.test.tsx index 2c40fec2ec31d..4c0bb6a67f2e4 100644 --- a/x-pack/plugins/security/public/management/users/edit_user/create_user_page.test.tsx +++ b/x-pack/plugins/security/public/management/users/edit_user/create_user_page.test.tsx @@ -53,7 +53,8 @@ describe('CreateUserPage', () => { }); }); - it('validates form', async () => { + // flaky https://github.com/elastic/kibana/issues/95345 + it.skip('validates form', async () => { const coreStart = coreMock.createStart(); const history = createMemoryHistory({ initialEntries: ['/create'] }); const authc = securityMock.createSetup().authc; diff --git a/x-pack/plugins/security_solution/common/endpoint/constants.ts b/x-pack/plugins/security_solution/common/endpoint/constants.ts index 90e025de1dcc8..d9f67e31196ca 100644 --- a/x-pack/plugins/security_solution/common/endpoint/constants.ts +++ b/x-pack/plugins/security_solution/common/endpoint/constants.ts @@ -15,8 +15,10 @@ export const telemetryIndexPattern = 'metrics-endpoint.telemetry-*'; export const LIMITED_CONCURRENCY_ENDPOINT_ROUTE_TAG = 'endpoint:limited-concurrency'; export const LIMITED_CONCURRENCY_ENDPOINT_COUNT = 100; +export const TRUSTED_APPS_GET_API = '/api/endpoint/trusted_apps/{id}'; export const TRUSTED_APPS_LIST_API = '/api/endpoint/trusted_apps'; export const TRUSTED_APPS_CREATE_API = '/api/endpoint/trusted_apps'; +export const TRUSTED_APPS_UPDATE_API = '/api/endpoint/trusted_apps/{id}'; export const TRUSTED_APPS_DELETE_API = '/api/endpoint/trusted_apps/{id}'; export const TRUSTED_APPS_SUMMARY_API = '/api/endpoint/trusted_apps/summary'; diff --git a/x-pack/plugins/security_solution/common/endpoint/schema/trusted_apps.test.ts b/x-pack/plugins/security_solution/common/endpoint/schema/trusted_apps.test.ts index e9ae439d0ac8c..326795ae55662 100644 --- a/x-pack/plugins/security_solution/common/endpoint/schema/trusted_apps.test.ts +++ b/x-pack/plugins/security_solution/common/endpoint/schema/trusted_apps.test.ts @@ -5,8 +5,18 @@ * 2.0. */ -import { GetTrustedAppsRequestSchema, PostTrustedAppCreateRequestSchema } from './trusted_apps'; -import { ConditionEntryField, OperatingSystem } from '../types'; +import { + GetTrustedAppsRequestSchema, + PostTrustedAppCreateRequestSchema, + PutTrustedAppUpdateRequestSchema, +} from './trusted_apps'; +import { + ConditionEntry, + ConditionEntryField, + NewTrustedApp, + OperatingSystem, + PutTrustedAppsRequestParams, +} from '../types'; describe('When invoking Trusted Apps Schema', () => { describe('for GET List', () => { @@ -72,17 +82,18 @@ describe('When invoking Trusted Apps Schema', () => { }); describe('for POST Create', () => { - const createConditionEntry = (data?: T) => ({ + const createConditionEntry = (data?: T): ConditionEntry => ({ field: ConditionEntryField.PATH, type: 'match', operator: 'included', value: 'c:/programs files/Anti-Virus', ...(data || {}), }); - const createNewTrustedApp = (data?: T) => ({ + const createNewTrustedApp = (data?: T): NewTrustedApp => ({ name: 'Some Anti-Virus App', description: 'this one is ok', - os: 'windows', + os: OperatingSystem.WINDOWS, + effectScope: { type: 'global' }, entries: [createConditionEntry()], ...(data || {}), }); @@ -329,4 +340,55 @@ describe('When invoking Trusted Apps Schema', () => { }); }); }); + + describe('for PUT Update', () => { + const createConditionEntry = (data?: T): ConditionEntry => ({ + field: ConditionEntryField.PATH, + type: 'match', + operator: 'included', + value: 'c:/programs files/Anti-Virus', + ...(data || {}), + }); + const createNewTrustedApp = (data?: T): NewTrustedApp => ({ + name: 'Some Anti-Virus App', + description: 'this one is ok', + os: OperatingSystem.WINDOWS, + effectScope: { type: 'global' }, + entries: [createConditionEntry()], + ...(data || {}), + }); + + const updateParams = (data?: T): PutTrustedAppsRequestParams => ({ + id: 'validId', + ...(data || {}), + }); + + const body = PutTrustedAppUpdateRequestSchema.body; + const params = PutTrustedAppUpdateRequestSchema.params; + + it('should not error on a valid message', () => { + const bodyMsg = createNewTrustedApp(); + const paramsMsg = updateParams(); + expect(body.validate(bodyMsg)).toStrictEqual(bodyMsg); + expect(params.validate(paramsMsg)).toStrictEqual(paramsMsg); + }); + + it('should validate `id` params is required', () => { + expect(() => params.validate(updateParams({ id: undefined }))).toThrow(); + }); + + it('should validate `id` params to be string', () => { + expect(() => params.validate(updateParams({ id: 1 }))).toThrow(); + }); + + it('should validate `version`', () => { + const bodyMsg = createNewTrustedApp({ version: 'v1' }); + expect(body.validate(bodyMsg)).toStrictEqual(bodyMsg); + }); + + it('should validate `version` must be string', () => { + const bodyMsg = createNewTrustedApp({ version: 1 }); + expect(() => body.validate(bodyMsg)).toThrow(); + }); + }); }); diff --git a/x-pack/plugins/security_solution/common/endpoint/schema/trusted_apps.ts b/x-pack/plugins/security_solution/common/endpoint/schema/trusted_apps.ts index 6d40dc75fd1c1..e582744e1a141 100644 --- a/x-pack/plugins/security_solution/common/endpoint/schema/trusted_apps.ts +++ b/x-pack/plugins/security_solution/common/endpoint/schema/trusted_apps.ts @@ -6,8 +6,8 @@ */ import { schema } from '@kbn/config-schema'; -import { ConditionEntryField, OperatingSystem } from '../types'; -import { getDuplicateFields, isValidHash } from '../validation/trusted_apps'; +import { ConditionEntry, ConditionEntryField, OperatingSystem } from '../types'; +import { getDuplicateFields, isValidHash } from '../service/trusted_apps/validations'; export const DeleteTrustedAppsRequestSchema = { params: schema.object({ @@ -15,10 +15,17 @@ export const DeleteTrustedAppsRequestSchema = { }), }; +export const GetOneTrustedAppRequestSchema = { + params: schema.object({ + id: schema.string(), + }), +}; + export const GetTrustedAppsRequestSchema = { query: schema.object({ page: schema.maybe(schema.number({ defaultValue: 1, min: 1 })), per_page: schema.maybe(schema.number({ defaultValue: 20, min: 1 })), + kuery: schema.maybe(schema.string()), }), }; @@ -40,18 +47,18 @@ const CommonEntrySchema = { schema.siblingRef('field'), ConditionEntryField.HASH, schema.string({ - validate: (hash) => + validate: (hash: string) => isValidHash(hash) ? undefined : `invalidField.${ConditionEntryField.HASH}`, }), schema.conditional( schema.siblingRef('field'), ConditionEntryField.PATH, schema.string({ - validate: (field) => + validate: (field: string) => field.length > 0 ? undefined : `invalidField.${ConditionEntryField.PATH}`, }), schema.string({ - validate: (field) => + validate: (field: string) => field.length > 0 ? undefined : `invalidField.${ConditionEntryField.SIGNER}`, }) ) @@ -99,7 +106,7 @@ const EntrySchemaDependingOnOS = schema.conditional( */ const EntriesSchema = schema.arrayOf(EntrySchemaDependingOnOS, { minSize: 1, - validate(entries) { + validate(entries: ConditionEntry[]) { return ( getDuplicateFields(entries) .map((field) => `duplicatedEntry.${field}`) @@ -108,8 +115,8 @@ const EntriesSchema = schema.arrayOf(EntrySchemaDependingOnOS, { }, }); -export const PostTrustedAppCreateRequestSchema = { - body: schema.object({ +const getTrustedAppForOsScheme = (forUpdateFlow: boolean = false) => + schema.object({ name: schema.string({ minLength: 1, maxLength: 256 }), description: schema.maybe(schema.string({ minLength: 0, maxLength: 256, defaultValue: '' })), os: schema.oneOf([ @@ -117,6 +124,26 @@ export const PostTrustedAppCreateRequestSchema = { schema.literal(OperatingSystem.LINUX), schema.literal(OperatingSystem.MAC), ]), + effectScope: schema.oneOf([ + schema.object({ + type: schema.literal('global'), + }), + schema.object({ + type: schema.literal('policy'), + policies: schema.arrayOf(schema.string({ minLength: 1 })), + }), + ]), entries: EntriesSchema, + ...(forUpdateFlow ? { version: schema.maybe(schema.string()) } : {}), + }); + +export const PostTrustedAppCreateRequestSchema = { + body: getTrustedAppForOsScheme(), +}; + +export const PutTrustedAppUpdateRequestSchema = { + params: schema.object({ + id: schema.string(), }), + body: getTrustedAppForOsScheme(true), }; diff --git a/x-pack/plugins/security_solution/common/endpoint/service/trusted_apps/to_update_trusted_app.ts b/x-pack/plugins/security_solution/common/endpoint/service/trusted_apps/to_update_trusted_app.ts new file mode 100644 index 0000000000000..fcde1d44b682d --- /dev/null +++ b/x-pack/plugins/security_solution/common/endpoint/service/trusted_apps/to_update_trusted_app.ts @@ -0,0 +1,30 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { MaybeImmutable, NewTrustedApp, UpdateTrustedApp } from '../../types'; + +const NEW_TRUSTED_APP_KEYS: Array = [ + 'name', + 'effectScope', + 'entries', + 'description', + 'os', + 'version', +]; + +export const toUpdateTrustedApp = ( + trustedApp: MaybeImmutable +): UpdateTrustedApp => { + const trustedAppForUpdate: UpdateTrustedApp = {} as UpdateTrustedApp; + + for (const key of NEW_TRUSTED_APP_KEYS) { + // This should be safe. Its needed due to the inter-dependency on property values (`os` <=> `entries`) + // @ts-expect-error + trustedAppForUpdate[key] = trustedApp[key]; + } + return trustedAppForUpdate; +}; diff --git a/x-pack/plugins/security_solution/common/endpoint/validation/trusted_apps.ts b/x-pack/plugins/security_solution/common/endpoint/service/trusted_apps/validations.ts similarity index 93% rename from x-pack/plugins/security_solution/common/endpoint/validation/trusted_apps.ts rename to x-pack/plugins/security_solution/common/endpoint/service/trusted_apps/validations.ts index faad639eeacb3..b0828be6af6c5 100644 --- a/x-pack/plugins/security_solution/common/endpoint/validation/trusted_apps.ts +++ b/x-pack/plugins/security_solution/common/endpoint/service/trusted_apps/validations.ts @@ -5,7 +5,7 @@ * 2.0. */ -import { ConditionEntry, ConditionEntryField } from '../types'; +import { ConditionEntry, ConditionEntryField } from '../../types'; const HASH_LENGTHS: readonly number[] = [ 32, // MD5 diff --git a/x-pack/plugins/security_solution/common/endpoint/types/index.ts b/x-pack/plugins/security_solution/common/endpoint/types/index.ts index 87268f02a16e1..0b41dc5608fe9 100644 --- a/x-pack/plugins/security_solution/common/endpoint/types/index.ts +++ b/x-pack/plugins/security_solution/common/endpoint/types/index.ts @@ -62,6 +62,11 @@ type ImmutableMap = ReadonlyMap, Immutable>; type ImmutableSet = ReadonlySet>; type ImmutableObject = { readonly [K in keyof T]: Immutable }; +/** + * Utility type that will return back a union of the given [T]ype and an Immutable version of it + */ +export type MaybeImmutable = T | Immutable; + /** * Stats for related events for a particular node in a resolver graph. */ diff --git a/x-pack/plugins/security_solution/common/endpoint/types/trusted_apps.ts b/x-pack/plugins/security_solution/common/endpoint/types/trusted_apps.ts index a5c3c1eab52b3..d36958c11d2a1 100644 --- a/x-pack/plugins/security_solution/common/endpoint/types/trusted_apps.ts +++ b/x-pack/plugins/security_solution/common/endpoint/types/trusted_apps.ts @@ -9,14 +9,22 @@ import { TypeOf } from '@kbn/config-schema'; import { ApplicationStart } from 'kibana/public'; import { DeleteTrustedAppsRequestSchema, + GetOneTrustedAppRequestSchema, GetTrustedAppsRequestSchema, PostTrustedAppCreateRequestSchema, + PutTrustedAppUpdateRequestSchema, } from '../schema/trusted_apps'; import { OperatingSystem } from './os'; /** API request params for deleting Trusted App entry */ export type DeleteTrustedAppsRequestParams = TypeOf; +export type GetOneTrustedAppRequestParams = TypeOf; + +export interface GetOneTrustedAppResponse { + data: TrustedApp; +} + /** API request params for retrieving a list of Trusted Apps */ export type GetTrustedAppsListRequest = TypeOf; @@ -39,6 +47,15 @@ export interface PostTrustedAppCreateResponse { data: TrustedApp; } +/** API request params for updating a Trusted App */ +export type PutTrustedAppsRequestParams = TypeOf; + +/** API Request body for Updating a new Trusted App entry */ +export type PutTrustedAppUpdateRequest = TypeOf & + (MacosLinuxConditionEntries | WindowsConditionEntries); + +export type PutTrustedAppUpdateResponse = PostTrustedAppCreateResponse; + export interface GetTrustedAppsSummaryResponse { total: number; windows: number; @@ -76,17 +93,38 @@ export interface WindowsConditionEntries { entries: WindowsConditionEntry[]; } +export interface GlobalEffectScope { + type: 'global'; +} + +export interface PolicyEffectScope { + type: 'policy'; + /** An array of Endpoint Integration Policy UUIDs */ + policies: string[]; +} + +export type EffectScope = GlobalEffectScope | PolicyEffectScope; + /** Type for a new Trusted App Entry */ export type NewTrustedApp = { name: string; description?: string; + effectScope: EffectScope; } & (MacosLinuxConditionEntries | WindowsConditionEntries); +/** An Update to a Trusted App Entry */ +export type UpdateTrustedApp = NewTrustedApp & { + version?: string; +}; + /** A trusted app entry */ export type TrustedApp = NewTrustedApp & { + version: string; id: string; created_at: string; created_by: string; + updated_at: string; + updated_by: string; }; /** diff --git a/x-pack/plugins/security_solution/common/experimental_features.ts b/x-pack/plugins/security_solution/common/experimental_features.ts index c764c31a2d781..19de81cb95c3f 100644 --- a/x-pack/plugins/security_solution/common/experimental_features.ts +++ b/x-pack/plugins/security_solution/common/experimental_features.ts @@ -13,6 +13,7 @@ export type ExperimentalFeatures = typeof allowedExperimentalValues; */ const allowedExperimentalValues = Object.freeze({ fleetServerEnabled: false, + trustedAppsByPolicyEnabled: false, }); type ExperimentalConfigKeys = Array; diff --git a/x-pack/plugins/security_solution/public/common/components/item_details_card/index.tsx b/x-pack/plugins/security_solution/public/common/components/item_details_card/index.tsx index 6fcf688fff7a7..c9fb502956053 100644 --- a/x-pack/plugins/security_solution/public/common/components/item_details_card/index.tsx +++ b/x-pack/plugins/security_solution/public/common/components/item_details_card/index.tsx @@ -5,7 +5,15 @@ * 2.0. */ -import React, { FC, isValidElement, memo, ReactElement, ReactNode, useMemo } from 'react'; +import React, { + FC, + isValidElement, + memo, + PropsWithChildren, + ReactElement, + ReactNode, + useMemo, +} from 'react'; import styled from 'styled-components'; import { EuiPanel, @@ -92,41 +100,46 @@ export const ItemDetailsAction: FC> = memo( ItemDetailsAction.displayName = 'ItemDetailsAction'; -export const ItemDetailsCard: FC = memo(({ children }) => { - const childElements = useMemo( - () => groupChildrenByType(children, [ItemDetailsPropertySummary, ItemDetailsAction]), - [children] - ); - - return ( - - - - - {childElements.get(ItemDetailsPropertySummary)} - - - - - -
{childElements.get(OTHER_NODES)}
-
- {childElements.has(ItemDetailsAction) && ( - - - {childElements.get(ItemDetailsAction)?.map((action, index) => ( - - {action} - - ))} - +export type ItemDetailsCardProps = PropsWithChildren<{ + 'data-test-subj'?: string; +}>; +export const ItemDetailsCard = memo( + ({ children, 'data-test-subj': dataTestSubj }) => { + const childElements = useMemo( + () => groupChildrenByType(children, [ItemDetailsPropertySummary, ItemDetailsAction]), + [children] + ); + + return ( + + + + + {childElements.get(ItemDetailsPropertySummary)} + + + + + +
{childElements.get(OTHER_NODES)}
- )} -
-
-
-
- ); -}); + {childElements.has(ItemDetailsAction) && ( + + + {childElements.get(ItemDetailsAction)?.map((action, index) => ( + + {action} + + ))} + + + )} +
+
+
+
+ ); + } +); ItemDetailsCard.displayName = 'ItemDetailsCard'; diff --git a/x-pack/plugins/security_solution/public/common/hooks/use_experimental_features.test.ts b/x-pack/plugins/security_solution/public/common/hooks/use_experimental_features.test.ts new file mode 100644 index 0000000000000..2ac5948641d7d --- /dev/null +++ b/x-pack/plugins/security_solution/public/common/hooks/use_experimental_features.test.ts @@ -0,0 +1,47 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { useSelector } from 'react-redux'; +import { ExperimentalFeatures } from '../../../common/experimental_features'; +import { useIsExperimentalFeatureEnabled } from './use_experimental_features'; + +jest.mock('react-redux'); +const useSelectorMock = useSelector as jest.Mock; +const mockAppState = { + app: { + enableExperimental: { + featureA: true, + featureB: false, + }, + }, +}; + +describe('useExperimentalFeatures', () => { + beforeEach(() => { + useSelectorMock.mockImplementation((cb) => { + return cb(mockAppState); + }); + }); + afterEach(() => { + useSelectorMock.mockClear(); + }); + it('throws an error when unexisting feature', async () => { + expect(() => + useIsExperimentalFeatureEnabled('unexistingFeature' as keyof ExperimentalFeatures) + ).toThrowError(); + }); + it('returns true when existing feature and is enabled', async () => { + const result = useIsExperimentalFeatureEnabled('featureA' as keyof ExperimentalFeatures); + + expect(result).toBeTruthy(); + }); + it('returns false when existing feature and is disabled', async () => { + const result = useIsExperimentalFeatureEnabled('featureB' as keyof ExperimentalFeatures); + + expect(result).toBeFalsy(); + }); +}); diff --git a/x-pack/plugins/security_solution/public/common/hooks/use_experimental_features.ts b/x-pack/plugins/security_solution/public/common/hooks/use_experimental_features.ts new file mode 100644 index 0000000000000..247b7624914cf --- /dev/null +++ b/x-pack/plugins/security_solution/public/common/hooks/use_experimental_features.ts @@ -0,0 +1,28 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { useSelector } from 'react-redux'; +import { State } from '../../common/store'; +import { + ExperimentalFeatures, + getExperimentalAllowedValues, +} from '../../../common/experimental_features'; + +const allowedExperimentalValues = getExperimentalAllowedValues(); + +export const useIsExperimentalFeatureEnabled = (feature: keyof ExperimentalFeatures): boolean => { + return useSelector(({ app: { enableExperimental } }: State) => { + if (!enableExperimental || !(feature in enableExperimental)) { + throw new Error( + `Invalid enable value ${feature}. Allowed values are: ${allowedExperimentalValues.join( + ', ' + )}` + ); + } + return enableExperimental[feature]; + }); +}; diff --git a/x-pack/plugins/security_solution/public/common/store/app/model.ts b/x-pack/plugins/security_solution/public/common/store/app/model.ts index 38ecedc0c7ba7..5a252e4aa48f2 100644 --- a/x-pack/plugins/security_solution/public/common/store/app/model.ts +++ b/x-pack/plugins/security_solution/public/common/store/app/model.ts @@ -5,6 +5,7 @@ * 2.0. */ +import { ExperimentalFeatures } from '../../../../common/experimental_features'; import { Note } from '../../lib/note'; export type ErrorState = ErrorModel; @@ -24,4 +25,5 @@ export type ErrorModel = Error[]; export interface AppModel { notesById: NotesById; errors: ErrorState; + enableExperimental?: ExperimentalFeatures; } diff --git a/x-pack/plugins/security_solution/public/common/store/reducer.test.ts b/x-pack/plugins/security_solution/public/common/store/reducer.test.ts index 9a2289765e85d..d2808a02c8621 100644 --- a/x-pack/plugins/security_solution/public/common/store/reducer.test.ts +++ b/x-pack/plugins/security_solution/public/common/store/reducer.test.ts @@ -5,6 +5,7 @@ * 2.0. */ +import { parseExperimentalConfigValue } from '../../..//common/experimental_features'; import { createInitialState } from './reducer'; jest.mock('../lib/kibana', () => ({ @@ -22,6 +23,7 @@ describe('createInitialState', () => { kibanaIndexPatterns: [{ id: '1234567890987654321', title: 'mock-kibana' }], configIndexPatterns: ['auditbeat-*', 'filebeat'], signalIndexName: 'siem-signals-default', + enableExperimental: parseExperimentalConfigValue([]), } ); @@ -35,6 +37,7 @@ describe('createInitialState', () => { kibanaIndexPatterns: [{ id: '1234567890987654321', title: 'mock-kibana' }], configIndexPatterns: [], signalIndexName: 'siem-signals-default', + enableExperimental: parseExperimentalConfigValue([]), } ); diff --git a/x-pack/plugins/security_solution/public/common/store/reducer.ts b/x-pack/plugins/security_solution/public/common/store/reducer.ts index 27fddafc3781f..c2ef2563fe63e 100644 --- a/x-pack/plugins/security_solution/public/common/store/reducer.ts +++ b/x-pack/plugins/security_solution/public/common/store/reducer.ts @@ -21,6 +21,7 @@ import { ManagementPluginReducer } from '../../management'; import { State } from './types'; import { AppAction } from './actions'; import { KibanaIndexPatterns } from './sourcerer/model'; +import { ExperimentalFeatures } from '../../../common/experimental_features'; export type SubPluginsInitReducer = HostsPluginReducer & NetworkPluginReducer & @@ -36,14 +37,16 @@ export const createInitialState = ( kibanaIndexPatterns, configIndexPatterns, signalIndexName, + enableExperimental, }: { kibanaIndexPatterns: KibanaIndexPatterns; configIndexPatterns: string[]; signalIndexName: string | null; + enableExperimental: ExperimentalFeatures; } ): PreloadedState => { const preloadedState: PreloadedState = { - app: initialAppState, + app: { ...initialAppState, enableExperimental }, dragAndDrop: initialDragAndDropState, ...pluginsInitState, inputs: createInitialInputsState(), diff --git a/x-pack/plugins/security_solution/public/common/store/test_utils.ts b/x-pack/plugins/security_solution/public/common/store/test_utils.ts index c1d54192c86b1..7616dfccddaff 100644 --- a/x-pack/plugins/security_solution/public/common/store/test_utils.ts +++ b/x-pack/plugins/security_solution/public/common/store/test_utils.ts @@ -9,6 +9,10 @@ import { Dispatch } from 'redux'; import { State, ImmutableMiddlewareFactory } from './types'; import { AppAction } from './actions'; +interface WaitForActionOptions { + validate?: (action: A extends { type: T } ? A : never) => boolean; +} + /** * Utilities for testing Redux middleware */ @@ -21,7 +25,10 @@ export interface MiddlewareActionSpyHelper(actionType: T) => Promise; + waitForAction: ( + actionType: T, + options?: WaitForActionOptions + ) => Promise; /** * A property holding the information around the calls that were processed by the internal * `actionSpyMiddelware`. This property holds the information typically found in Jets's mocked @@ -78,7 +85,7 @@ export const createSpyMiddleware = < let spyDispatch: jest.Mock>; return { - waitForAction: async (actionType) => { + waitForAction: async (actionType, options = {}) => { type ResolvedAction = A extends { type: typeof actionType } ? A : never; // Error is defined here so that we get a better stack trace that points to the test from where it was used @@ -87,6 +94,10 @@ export const createSpyMiddleware = < return new Promise((resolve, reject) => { const watch: ActionWatcher = (action) => { if (action.type === actionType) { + if (options.validate && !options.validate(action as ResolvedAction)) { + return; + } + watchers.delete(watch); clearTimeout(timeout); resolve(action as ResolvedAction); diff --git a/x-pack/plugins/security_solution/public/common/types.ts b/x-pack/plugins/security_solution/public/common/types.ts index 68346847eb8d1..f1a7cdc8abc60 100644 --- a/x-pack/plugins/security_solution/public/common/types.ts +++ b/x-pack/plugins/security_solution/public/common/types.ts @@ -10,3 +10,7 @@ export interface ServerApiError { error: string; message: string; } + +export interface SecuritySolutionUiConfigType { + enableExperimental: string[]; +} diff --git a/x-pack/plugins/security_solution/public/management/common/routing.ts b/x-pack/plugins/security_solution/public/management/common/routing.ts index cbcc054e7c6a9..bf754720f314b 100644 --- a/x-pack/plugins/security_solution/public/management/common/routing.ts +++ b/x-pack/plugins/security_solution/public/management/common/routing.ts @@ -108,6 +108,7 @@ const normalizeTrustedAppsPageLocation = ( : {}), ...(!isDefaultOrMissing(location.view_type, 'grid') ? { view_type: location.view_type } : {}), ...(!isDefaultOrMissing(location.show, undefined) ? { show: location.show } : {}), + ...(!isDefaultOrMissing(location.id, undefined) ? { id: location.id } : {}), }; } else { return {}; @@ -147,11 +148,20 @@ export const extractListPaginationParams = (query: querystring.ParsedUrlQuery) = export const extractTrustedAppsListPageLocation = ( query: querystring.ParsedUrlQuery -): TrustedAppsListPageLocation => ({ - ...extractListPaginationParams(query), - view_type: extractFirstParamValue(query, 'view_type') === 'list' ? 'list' : 'grid', - show: extractFirstParamValue(query, 'show') === 'create' ? 'create' : undefined, -}); +): TrustedAppsListPageLocation => { + const showParamValue = extractFirstParamValue( + query, + 'show' + ) as TrustedAppsListPageLocation['show']; + + return { + ...extractListPaginationParams(query), + view_type: extractFirstParamValue(query, 'view_type') === 'list' ? 'list' : 'grid', + show: + showParamValue && ['edit', 'create'].includes(showParamValue) ? showParamValue : undefined, + id: extractFirstParamValue(query, 'id'), + }; +}; export const getTrustedAppsListPath = (location?: Partial): string => { const path = generatePath(MANAGEMENT_ROUTING_TRUSTED_APPS_PATH, { diff --git a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/service/index.ts b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/service/index.ts index 578043f4321e9..5f572251daeda 100644 --- a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/service/index.ts +++ b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/service/index.ts @@ -10,7 +10,9 @@ import { HttpStart } from 'kibana/public'; import { TRUSTED_APPS_CREATE_API, TRUSTED_APPS_DELETE_API, + TRUSTED_APPS_GET_API, TRUSTED_APPS_LIST_API, + TRUSTED_APPS_UPDATE_API, TRUSTED_APPS_SUMMARY_API, } from '../../../../../common/endpoint/constants'; @@ -21,19 +23,39 @@ import { PostTrustedAppCreateRequest, PostTrustedAppCreateResponse, GetTrustedAppsSummaryResponse, + PutTrustedAppUpdateRequest, + PutTrustedAppUpdateResponse, + PutTrustedAppsRequestParams, + GetOneTrustedAppRequestParams, + GetOneTrustedAppResponse, } from '../../../../../common/endpoint/types/trusted_apps'; import { resolvePathVariables } from './utils'; +import { sendGetEndpointSpecificPackagePolicies } from '../../policy/store/services/ingest'; export interface TrustedAppsService { + getTrustedApp(params: GetOneTrustedAppRequestParams): Promise; getTrustedAppsList(request: GetTrustedAppsListRequest): Promise; deleteTrustedApp(request: DeleteTrustedAppsRequestParams): Promise; createTrustedApp(request: PostTrustedAppCreateRequest): Promise; + updateTrustedApp( + params: PutTrustedAppsRequestParams, + request: PutTrustedAppUpdateRequest + ): Promise; + getPolicyList( + options?: Parameters[1] + ): ReturnType; } export class TrustedAppsHttpService implements TrustedAppsService { constructor(private http: HttpStart) {} + async getTrustedApp(params: GetOneTrustedAppRequestParams) { + return this.http.get( + resolvePathVariables(TRUSTED_APPS_GET_API, params) + ); + } + async getTrustedAppsList(request: GetTrustedAppsListRequest) { return this.http.get(TRUSTED_APPS_LIST_API, { query: request, @@ -50,7 +72,21 @@ export class TrustedAppsHttpService implements TrustedAppsService { }); } + async updateTrustedApp( + params: PutTrustedAppsRequestParams, + updatedTrustedApp: PutTrustedAppUpdateRequest + ) { + return this.http.put( + resolvePathVariables(TRUSTED_APPS_UPDATE_API, params), + { body: JSON.stringify(updatedTrustedApp) } + ); + } + async getTrustedAppsSummary() { return this.http.get(TRUSTED_APPS_SUMMARY_API); } + + getPolicyList(options?: Parameters[1]) { + return sendGetEndpointSpecificPackagePolicies(this.http, options); + } } diff --git a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/state/trusted_apps_list_page_state.ts b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/state/trusted_apps_list_page_state.ts index ea934881f6220..1c1fca4b55abc 100644 --- a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/state/trusted_apps_list_page_state.ts +++ b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/state/trusted_apps_list_page_state.ts @@ -7,6 +7,7 @@ import { NewTrustedApp, TrustedApp } from '../../../../../common/endpoint/types/trusted_apps'; import { AsyncResourceState } from '.'; +import { GetPolicyListResponse } from '../../policy/types'; export interface Pagination { pageIndex: number; @@ -29,7 +30,9 @@ export interface TrustedAppsListPageLocation { page_index: number; page_size: number; view_type: ViewType; - show?: 'create'; + show?: 'create' | 'edit'; + /** Used for editing. The ID of the selected trusted app */ + id?: string; } export interface TrustedAppsListPageState { @@ -51,9 +54,13 @@ export interface TrustedAppsListPageState { entry: NewTrustedApp; isValid: boolean; }; + /** The trusted app to be edited (when in edit mode) */ + editItem?: AsyncResourceState; confirmed: boolean; submissionResourceState: AsyncResourceState; }; + /** A list of all available polices for use in associating TA to policies */ + policies: AsyncResourceState; location: TrustedAppsListPageLocation; active: boolean; } diff --git a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/state/type_guards.ts b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/state/type_guards.ts index 66f4eff81dbdd..3f9e9d53f69e4 100644 --- a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/state/type_guards.ts +++ b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/state/type_guards.ts @@ -8,7 +8,11 @@ import { ConditionEntry, ConditionEntryField, + EffectScope, + GlobalEffectScope, MacosLinuxConditionEntry, + MaybeImmutable, + PolicyEffectScope, WindowsConditionEntry, } from '../../../../../common/endpoint/types'; @@ -23,3 +27,15 @@ export const isMacosLinuxTrustedAppCondition = ( ): condition is MacosLinuxConditionEntry => { return condition.field !== ConditionEntryField.SIGNER; }; + +export const isGlobalEffectScope = ( + effectedScope: MaybeImmutable +): effectedScope is GlobalEffectScope => { + return effectedScope.type === 'global'; +}; + +export const isPolicyEffectScope = ( + effectedScope: MaybeImmutable +): effectedScope is PolicyEffectScope => { + return effectedScope.type === 'policy'; +}; diff --git a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/action.ts b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/action.ts index aaa05f550b208..34f48142c7032 100644 --- a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/action.ts +++ b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/action.ts @@ -9,6 +9,7 @@ import { Action } from 'redux'; import { NewTrustedApp, TrustedApp } from '../../../../../common/endpoint/types'; import { AsyncResourceState, TrustedAppsListData } from '../state'; +import { GetPolicyListResponse } from '../../policy/types'; export type TrustedAppsListDataOutdated = Action<'trustedAppsListDataOutdated'>; @@ -51,6 +52,10 @@ export type TrustedAppCreationDialogFormStateUpdated = Action<'trustedAppCreatio }; }; +export type TrustedAppCreationEditItemStateChanged = Action<'trustedAppCreationEditItemStateChanged'> & { + payload: AsyncResourceState; +}; + export type TrustedAppCreationDialogConfirmed = Action<'trustedAppCreationDialogConfirmed'>; export type TrustedAppCreationDialogClosed = Action<'trustedAppCreationDialogClosed'>; @@ -59,6 +64,10 @@ export type TrustedAppsExistResponse = Action<'trustedAppsExistStateChanged'> & payload: AsyncResourceState; }; +export type TrustedAppsPoliciesStateChanged = Action<'trustedAppsPoliciesStateChanged'> & { + payload: AsyncResourceState; +}; + export type TrustedAppsPageAction = | TrustedAppsListDataOutdated | TrustedAppsListResourceStateChanged @@ -67,8 +76,10 @@ export type TrustedAppsPageAction = | TrustedAppDeletionDialogConfirmed | TrustedAppDeletionDialogClosed | TrustedAppCreationSubmissionResourceStateChanged + | TrustedAppCreationEditItemStateChanged | TrustedAppCreationDialogStarted | TrustedAppCreationDialogFormStateUpdated | TrustedAppCreationDialogConfirmed | TrustedAppsExistResponse + | TrustedAppsPoliciesStateChanged | TrustedAppCreationDialogClosed; diff --git a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/builders.ts b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/builders.ts index 3acb55904d298..ece2c9e29750f 100644 --- a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/builders.ts +++ b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/builders.ts @@ -28,6 +28,7 @@ export const defaultNewTrustedApp = (): NewTrustedApp => ({ os: OperatingSystem.WINDOWS, entries: [defaultConditionEntry()], description: '', + effectScope: { type: 'global' }, }); export const initialDeletionDialogState = (): TrustedAppsListPageState['deletionDialog'] => ({ @@ -48,10 +49,12 @@ export const initialTrustedAppsPageState = (): TrustedAppsListPageState => ({ }, deletionDialog: initialDeletionDialogState(), creationDialog: initialCreationDialogState(), + policies: { type: 'UninitialisedResourceState' }, location: { page_index: MANAGEMENT_DEFAULT_PAGE, page_size: MANAGEMENT_DEFAULT_PAGE_SIZE, show: undefined, + id: undefined, view_type: 'grid', }, active: false, diff --git a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/middleware.test.ts b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/middleware.test.ts index 064b108848d2f..ed45d077dd0ca 100644 --- a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/middleware.test.ts +++ b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/middleware.test.ts @@ -21,10 +21,11 @@ import { } from '../test_utils'; import { TrustedAppsService } from '../service'; -import { Pagination, TrustedAppsListPageState } from '../state'; +import { Pagination, TrustedAppsListPageLocation, TrustedAppsListPageState } from '../state'; import { initialTrustedAppsPageState } from './builders'; import { trustedAppsPageReducer } from './reducer'; import { createTrustedAppsPageMiddleware } from './middleware'; +import { Immutable } from '../../../../../common/endpoint/types'; const initialNow = 111111; const dateNowMock = jest.fn(); @@ -32,7 +33,7 @@ dateNowMock.mockReturnValue(initialNow); Date.now = dateNowMock; -const initialState = initialTrustedAppsPageState(); +const initialState: Immutable = initialTrustedAppsPageState(); const createGetTrustedListAppsResponse = (pagination: Partial) => { const fullPagination = { ...createDefaultPagination(), ...pagination }; @@ -49,6 +50,9 @@ const createTrustedAppsServiceMock = (): jest.Mocked => ({ getTrustedAppsList: jest.fn(), deleteTrustedApp: jest.fn(), createTrustedApp: jest.fn(), + getPolicyList: jest.fn(), + updateTrustedApp: jest.fn(), + getTrustedApp: jest.fn(), }); const createStoreSetup = (trustedAppsService: TrustedAppsService) => { @@ -87,6 +91,15 @@ describe('middleware', () => { }; }; + const createLocationState = ( + params?: Partial + ): TrustedAppsListPageLocation => { + return { + ...initialState.location, + ...(params ?? {}), + }; + }; + beforeEach(() => { dateNowMock.mockReturnValue(initialNow); }); @@ -102,7 +115,10 @@ describe('middleware', () => { describe('refreshing list resource state', () => { it('refreshes the list when location changes and data gets outdated', async () => { const pagination = { pageIndex: 2, pageSize: 50 }; - const location = { page_index: 2, page_size: 50, show: undefined, view_type: 'grid' }; + const location = createLocationState({ + page_index: 2, + page_size: 50, + }); const service = createTrustedAppsServiceMock(); const { store, spyMiddleware } = createStoreSetup(service); @@ -136,7 +152,10 @@ describe('middleware', () => { it('does not refresh the list when location changes and data does not get outdated', async () => { const pagination = { pageIndex: 2, pageSize: 50 }; - const location = { page_index: 2, page_size: 50, show: undefined, view_type: 'grid' }; + const location = createLocationState({ + page_index: 2, + page_size: 50, + }); const service = createTrustedAppsServiceMock(); const { store, spyMiddleware } = createStoreSetup(service); @@ -161,7 +180,7 @@ describe('middleware', () => { it('refreshes the list when data gets outdated with and outdate action', async () => { const newNow = 222222; const pagination = { pageIndex: 0, pageSize: 10 }; - const location = { page_index: 0, page_size: 10, show: undefined, view_type: 'grid' }; + const location = createLocationState(); const service = createTrustedAppsServiceMock(); const { store, spyMiddleware } = createStoreSetup(service); @@ -224,7 +243,10 @@ describe('middleware', () => { freshDataTimestamp: initialNow, }, active: true, - location: { page_index: 2, page_size: 50, show: undefined, view_type: 'grid' }, + location: createLocationState({ + page_index: 2, + page_size: 50, + }), }); const infiniteLoopTest = async () => { @@ -240,7 +262,7 @@ describe('middleware', () => { const entry = createSampleTrustedApp(3); const notFoundError = createServerApiError('Not Found'); const pagination = { pageIndex: 0, pageSize: 10 }; - const location = { page_index: 0, page_size: 10, show: undefined, view_type: 'grid' }; + const location = createLocationState(); const getTrustedAppsListResponse = createGetTrustedListAppsResponse(pagination); const listView = createLoadedListViewWithPagination(initialNow, pagination); const listViewNew = createLoadedListViewWithPagination(newNow, pagination); diff --git a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/middleware.ts b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/middleware.ts index 3e83b213f0f7e..7f940f14f9c6c 100644 --- a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/middleware.ts +++ b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/middleware.ts @@ -5,6 +5,7 @@ * 2.0. */ +import { i18n } from '@kbn/i18n'; import { Immutable, PostTrustedAppCreateRequest, @@ -54,7 +55,15 @@ import { getListTotalItemsCount, trustedAppsListPageActive, entriesExistState, + policiesState, + isEdit, + isFetchingEditTrustedAppItem, + editItemId, + editingTrustedApp, + getListItems, + editItemState, } from './selectors'; +import { toUpdateTrustedApp } from '../../../../../common/endpoint/service/trusted_apps/to_update_trusted_app'; const createTrustedAppsListResourceStateChangedAction = ( newState: Immutable> @@ -139,9 +148,11 @@ const submitCreationIfNeeded = async ( store: ImmutableMiddlewareAPI, trustedAppsService: TrustedAppsService ) => { - const submissionResourceState = getCreationSubmissionResourceState(store.getState()); - const isValid = isCreationDialogFormValid(store.getState()); - const entry = getCreationDialogFormEntry(store.getState()); + const currentState = store.getState(); + const submissionResourceState = getCreationSubmissionResourceState(currentState); + const isValid = isCreationDialogFormValid(currentState); + const entry = getCreationDialogFormEntry(currentState); + const editMode = isEdit(currentState); if (isStaleResourceState(submissionResourceState) && entry !== undefined && isValid) { store.dispatch( @@ -152,12 +163,27 @@ const submitCreationIfNeeded = async ( ); try { + let responseTrustedApp: TrustedApp; + + if (editMode) { + responseTrustedApp = ( + await trustedAppsService.updateTrustedApp( + { id: editItemId(currentState)! }, + // TODO: try to remove the cast + entry as PostTrustedAppCreateRequest + ) + ).data; + } else { + // TODO: try to remove the cast + responseTrustedApp = ( + await trustedAppsService.createTrustedApp(entry as PostTrustedAppCreateRequest) + ).data; + } + store.dispatch( createTrustedAppCreationSubmissionResourceStateChanged({ type: 'LoadedResourceState', - // TODO: try to remove the cast - data: (await trustedAppsService.createTrustedApp(entry as PostTrustedAppCreateRequest)) - .data, + data: responseTrustedApp, }) ); store.dispatch({ @@ -268,6 +294,139 @@ const checkTrustedAppsExistIfNeeded = async ( } }; +export const retrieveListOfPoliciesIfNeeded = async ( + { getState, dispatch }: ImmutableMiddlewareAPI, + trustedAppsService: TrustedAppsService +) => { + const currentState = getState(); + const currentPoliciesState = policiesState(currentState); + const isLoading = isLoadingResourceState(currentPoliciesState); + const isPageActive = trustedAppsListPageActive(currentState); + const isCreateFlow = isCreationDialogLocation(currentState); + + if (isPageActive && isCreateFlow && !isLoading) { + dispatch({ + type: 'trustedAppsPoliciesStateChanged', + payload: { + type: 'LoadingResourceState', + previousState: currentPoliciesState, + } as TrustedAppsListPageState['policies'], + }); + + try { + const policyList = await trustedAppsService.getPolicyList({ + query: { + page: 1, + perPage: 1000, + }, + }); + + dispatch({ + type: 'trustedAppsPoliciesStateChanged', + payload: { + type: 'LoadedResourceState', + data: policyList, + }, + }); + } catch (error) { + dispatch({ + type: 'trustedAppsPoliciesStateChanged', + payload: { + type: 'FailedResourceState', + error: error.body || error, + lastLoadedState: getLastLoadedResourceState(policiesState(getState())), + }, + }); + } + } +}; + +const fetchEditTrustedAppIfNeeded = async ( + { getState, dispatch }: ImmutableMiddlewareAPI, + trustedAppsService: TrustedAppsService +) => { + const currentState = getState(); + const isPageActive = trustedAppsListPageActive(currentState); + const isEditFlow = isEdit(currentState); + const isAlreadyFetching = isFetchingEditTrustedAppItem(currentState); + const editTrustedAppId = editItemId(currentState); + + if (isPageActive && isEditFlow && !isAlreadyFetching) { + if (!editTrustedAppId) { + const errorMessage = i18n.translate( + 'xpack.securitySolution.trustedapps.middleware.editIdMissing', + { + defaultMessage: 'No id provided', + } + ); + + dispatch({ + type: 'trustedAppCreationEditItemStateChanged', + payload: { + type: 'FailedResourceState', + error: Object.assign(new Error(errorMessage), { statusCode: 404, error: errorMessage }), + }, + }); + return; + } + + let trustedAppForEdit = editingTrustedApp(currentState); + + // If Trusted App is already loaded, then do nothing + if (trustedAppForEdit && trustedAppForEdit.id === editTrustedAppId) { + return; + } + + // See if we can get the Trusted App record from the current list of Trusted Apps being displayed + trustedAppForEdit = getListItems(currentState).find((ta) => ta.id === editTrustedAppId); + + try { + // Retrieve Trusted App record via API if it was not in the list data. + // This would be the case when linking from another place or using an UUID for a Trusted App + // that is not currently displayed on the list view. + if (!trustedAppForEdit) { + dispatch({ + type: 'trustedAppCreationEditItemStateChanged', + payload: { + type: 'LoadingResourceState', + // No easy way to get around this that I can see. `previousState` does not + // seem to allow everything that `editItem` state can hold, so not even sure if using + // type guards would work here + // @ts-ignore + previousState: editItemState(currentState)!, + }, + }); + + trustedAppForEdit = (await trustedAppsService.getTrustedApp({ id: editTrustedAppId })).data; + } + + dispatch({ + type: 'trustedAppCreationEditItemStateChanged', + payload: { + type: 'LoadedResourceState', + data: trustedAppForEdit, + }, + }); + + dispatch({ + type: 'trustedAppCreationDialogFormStateUpdated', + payload: { + entry: toUpdateTrustedApp(trustedAppForEdit), + isValid: true, + }, + }); + } catch (e) { + dispatch({ + type: 'trustedAppCreationEditItemStateChanged', + payload: { + type: 'FailedResourceState', + error: e, + }, + }); + } + } +}; + export const createTrustedAppsPageMiddleware = ( trustedAppsService: TrustedAppsService ): ImmutableMiddleware => { @@ -282,6 +441,8 @@ export const createTrustedAppsPageMiddleware = ( if (action.type === 'userChangedUrl') { updateCreationDialogIfNeeded(store); + retrieveListOfPoliciesIfNeeded(store, trustedAppsService); + fetchEditTrustedAppIfNeeded(store, trustedAppsService); } if (action.type === 'trustedAppCreationDialogConfirmed') { diff --git a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/reducer.test.ts b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/reducer.test.ts index 5f37d0d674558..6965172ef773d 100644 --- a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/reducer.test.ts +++ b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/reducer.test.ts @@ -37,7 +37,13 @@ describe('reducer', () => { expect(result).toStrictEqual({ ...initialState, - location: { page_index: 5, page_size: 50, show: 'create', view_type: 'list' }, + location: { + page_index: 5, + page_size: 50, + show: 'create', + view_type: 'list', + id: undefined, + }, active: true, }); }); diff --git a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/reducer.ts b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/reducer.ts index aff5cacf081c6..ea7bbb44c9bf2 100644 --- a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/reducer.ts +++ b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/reducer.ts @@ -29,6 +29,8 @@ import { TrustedAppCreationDialogConfirmed, TrustedAppCreationDialogClosed, TrustedAppsExistResponse, + TrustedAppsPoliciesStateChanged, + TrustedAppCreationEditItemStateChanged, } from './action'; import { TrustedAppsListPageState } from '../state'; @@ -37,7 +39,7 @@ import { initialDeletionDialogState, initialTrustedAppsPageState, } from './builders'; -import { entriesExistState } from './selectors'; +import { entriesExistState, trustedAppsListPageActive } from './selectors'; type StateReducer = ImmutableReducer; type CaseReducer = ( @@ -110,7 +112,7 @@ const trustedAppCreationDialogStarted: CaseReducer = ( + state, + action +) => { + return { + ...state, + creationDialog: { ...state.creationDialog, editItem: action.payload }, + }; +}; + const trustedAppCreationDialogConfirmed: CaseReducer = ( state ) => { @@ -155,6 +167,16 @@ const updateEntriesExists: CaseReducer = (state, { pay return state; }; +const updatePolicies: CaseReducer = (state, { payload }) => { + if (trustedAppsListPageActive(state)) { + return { + ...state, + policies: payload, + }; + } + return state; +}; + export const trustedAppsPageReducer: StateReducer = ( state = initialTrustedAppsPageState(), action @@ -187,6 +209,9 @@ export const trustedAppsPageReducer: StateReducer = ( case 'trustedAppCreationDialogFormStateUpdated': return trustedAppCreationDialogFormStateUpdated(state, action); + case 'trustedAppCreationEditItemStateChanged': + return handleUpdateToEditItemState(state, action); + case 'trustedAppCreationDialogConfirmed': return trustedAppCreationDialogConfirmed(state, action); @@ -198,6 +223,9 @@ export const trustedAppsPageReducer: StateReducer = ( case 'trustedAppsExistStateChanged': return updateEntriesExists(state, action); + + case 'trustedAppsPoliciesStateChanged': + return updatePolicies(state, action); } return state; diff --git a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/selectors.ts b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/selectors.ts index baa68eb314140..7c131c3eaa7a9 100644 --- a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/selectors.ts +++ b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/store/selectors.ts @@ -24,6 +24,7 @@ import { TrustedAppsListPageLocation, TrustedAppsListPageState, } from '../state'; +import { GetPolicyListResponse } from '../../policy/types'; export const needsRefreshOfListData = (state: Immutable): boolean => { const freshDataTimestamp = state.listView.freshDataTimestamp; @@ -130,7 +131,7 @@ export const getDeletionDialogEntry = ( }; export const isCreationDialogLocation = (state: Immutable): boolean => { - return state.location.show === 'create'; + return !!state.location.show; }; export const getCreationSubmissionResourceState = ( @@ -185,3 +186,56 @@ export const entriesExist: (state: Immutable) => boole export const trustedAppsListPageActive: (state: Immutable) => boolean = ( state ) => state.active; + +export const policiesState = ( + state: Immutable +): Immutable => state.policies; + +export const loadingPolicies: ( + state: Immutable +) => boolean = createSelector(policiesState, (policies) => isLoadingResourceState(policies)); + +export const listOfPolicies: ( + state: Immutable +) => Immutable = createSelector(policiesState, (policies) => { + return isLoadedResourceState(policies) ? policies.data.items : []; +}); + +export const isEdit: (state: Immutable) => boolean = createSelector( + getCurrentLocation, + ({ show }) => { + return show === 'edit'; + } +); + +export const editItemId: ( + state: Immutable +) => string | undefined = createSelector(getCurrentLocation, ({ id }) => { + return id; +}); + +export const editItemState: ( + state: Immutable +) => Immutable['creationDialog']['editItem'] = (state) => { + return state.creationDialog.editItem; +}; + +export const isFetchingEditTrustedAppItem: ( + state: Immutable +) => boolean = createSelector(editItemState, (editTrustedAppState) => { + return editTrustedAppState ? isLoadingResourceState(editTrustedAppState) : false; +}); + +export const editTrustedAppFetchError: ( + state: Immutable +) => ServerApiError | undefined = createSelector(editItemState, (itemForEditState) => { + return itemForEditState && getCurrentResourceError(itemForEditState); +}); + +export const editingTrustedApp: ( + state: Immutable +) => undefined | Immutable = createSelector(editItemState, (editTrustedAppState) => { + if (editTrustedAppState && isLoadedResourceState(editTrustedAppState)) { + return editTrustedAppState.data; + } +}); diff --git a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/test_utils/index.ts b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/test_utils/index.ts index faf111b1a55d8..faffc6b04a0cd 100644 --- a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/test_utils/index.ts +++ b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/test_utils/index.ts @@ -44,12 +44,16 @@ const generate = (count: number, generator: (i: number) => T) => export const createSampleTrustedApp = (i: number, longTexts?: boolean): TrustedApp => { return { id: String(i), + version: 'abc123', name: generate(longTexts ? 10 : 1, () => `trusted app ${i}`).join(' '), description: generate(longTexts ? 10 : 1, () => `Trusted App ${i}`).join(' '), created_at: '1 minute ago', created_by: 'someone', + updated_at: '1 minute ago', + updated_by: 'someone', os: OPERATING_SYSTEMS[i % 3], entries: [], + effectScope: { type: 'global' }, }; }; diff --git a/x-pack/plugins/security_solution/public/management/pages/trusted_apps/view/__snapshots__/trusted_apps_page.test.tsx.snap b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/view/__snapshots__/trusted_apps_page.test.tsx.snap new file mode 100644 index 0000000000000..35fc520558d6a --- /dev/null +++ b/x-pack/plugins/security_solution/public/management/pages/trusted_apps/view/__snapshots__/trusted_apps_page.test.tsx.snap @@ -0,0 +1,5573 @@ +// Jest Snapshot v1, https://goo.gl/fbAQLP + +exports[`When on the Trusted Apps Page and the Add Trusted App button is clicked and there is a feature flag for agents policy should display agents policy if feature flag is enabled 1`] = ` +Object { + "asFragment": [Function], + "baseElement": + .c0 { + padding: 24px; +} + +.c0.siemWrapperPage--fullHeight { + height: 100%; + display: -webkit-box; + display: -webkit-flex; + display: -ms-flexbox; + display: flex; + -webkit-flex-direction: column; + -ms-flex-direction: column; + flex-direction: column; + -webkit-flex: 1 1 auto; + -ms-flex: 1 1 auto; + flex: 1 1 auto; +} + +.c0.siemWrapperPage--noPadding { + padding: 0; + display: -webkit-box; + display: -webkit-flex; + display: -ms-flexbox; + display: flex; + -webkit-flex-direction: column; + -ms-flex-direction: column; + flex-direction: column; + -webkit-flex: 1 1 auto; + -ms-flex: 1 1 auto; + flex: 1 1 auto; +} + +.c0.siemWrapperPage--withTimeline { + padding-bottom: 70px; +} + +.c3 { + margin-top: 8px; +} + +.c3 .siemSubtitle__item { + color: #6a717d; + font-size: 12px; + line-height: 1.5; +} + +.c1 { + margin-bottom: 24px; +} + +.c2 { + display: block; +} + +.c4 .euiFlyout { + z-index: 4001; +} + +.c5 .and-badge { + padding-top: 20px; + padding-bottom: calc(32px + (8px * 2) + 3px); +} + +.c5 .group-entries { + margin-bottom: 8px; +} + +.c5 .group-entries > * { + margin-bottom: 8px; +} + +.c5 .group-entries > *:last-child { + margin-bottom: 0; +} + +.c5 .and-button { + min-width: 95px; +} + +.c6 .policy-name .euiSelectableListItem__text { + -webkit-text-decoration: none !important; + text-decoration: none !important; + color: #343741 !important; +} + +.c7 { + background-color: #f5f7fa; + padding: 16px; +} + +.c10 { + padding: 16px; +} + +.c8.c8.c8 { + width: 40%; +} + +.c9.c9.c9 { + width: 60%; +} + +@media only screen and (min-width:575px) { + .c3 .siemSubtitle__item { + display: inline-block; + margin-right: 16px; + } + + .c3 .siemSubtitle__item:last-child { + margin-right: 0; + } +} + +
+
+
+
+
+

+ Trusted Applications +

+
+

+ Add a trusted application to improve performance or alleviate conflicts with other applications running on your hosts. Trusted applications will be applied to hosts running Endpoint Security. +

+
+
+
+ +
+
+
+
+ + +
+
+
+
+