Skip to content

Commit

Permalink
[Fleet] Support user overrides in composable templates (#101769) (#10…
Browse files Browse the repository at this point in the history 
…3126)

## Summary
Closes #90454
Closes #72959

 * Rename the component templates which are [installed for some packages](https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts#L197-L213) from `${templateName}-mappings` and `${templateName}-settings` to `${templateName}@mappings` and `${templateName}@settings`
 * When any package is installed, add a component template named `${templateName}@custom`
 * Any of above templates also include a `_meta` property with `{ package: { name: packageName } }`
 * On package installation, add any installed component templates to the `installed_es` property of the `epm-packages` saved object
 * On package removal, remove any installed component templates from the `installed_es` property of the `epm-packages` saved object

<details><summary>Kibana logs showing component templates added for package</summary>

```
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [.logs-endpoint.diagnostic.collection@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [.logs-endpoint.diagnostic.collection@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata@custom]
```

</details>

<details><summary>screenshot - component templates are editable in the Stack Management UI</summary>
<img width="1342" alt="Screen Shot 2021-06-17 at 4 06 24 PM" src="https://user-images.githubusercontent.com/57655/122465421-1502bb80-cf86-11eb-94f4-9880cb3ea844.png">
</details>


### Checklist

Delete any items that are not applicable to this PR.

- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: John Schulz <john.schulz@elastic.co>
  • Loading branch information
@kibanamachine
kibanamachine and John Schulz authored Jun 23, 2021
1 parent 6e6cdb0 commit f4b4d20
Show file tree
Hide file tree
Showing 17 changed files with 452 additions and 204 deletions.
7 changes: 4 additions & 3 deletions x-pack/plugins/fleet/common/types/models/epm.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
* 2.0.
*/

import type { estypes } from '@elastic/elasticsearch';
// Follow pattern from https://github.com/elastic/kibana/pull/52447
// TODO: Update when https://github.com/elastic/kibana/issues/53021 is closed
import type { SavedObject, SavedObjectAttributes, SavedObjectReference } from 'src/core/public';
Expand Down Expand Up @@ -299,8 +300,8 @@ export interface RegistryDataStream {
}

export interface RegistryElasticsearch {
'index_template.settings'?: object;
'index_template.mappings'?: object;
'index_template.settings'?: estypes.IndicesIndexSettings;
'index_template.mappings'?: estypes.MappingTypeMapping;
}

export interface RegistryDataStreamPermissions {
Expand Down Expand Up @@ -425,7 +426,7 @@ export interface IndexTemplate {
_meta: object;
}

export interface TemplateRef {
export interface IndexTemplateEntry {
templateName: string;
indexTemplate: IndexTemplate;
}
235 changes: 140 additions & 95 deletions x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@ import type { ElasticsearchClient, SavedObjectsClientContract } from 'src/core/s
import { ElasticsearchAssetType } from '../../../../types';
import type {
RegistryDataStream,
TemplateRef,
IndexTemplateEntry,
RegistryElasticsearch,
InstallablePackage,
} from '../../../../types';
import { loadFieldsFromYaml, processFields } from '../../fields/field';
import type { Field } from '../../fields/field';
import { getPipelineNameForInstallation } from '../ingest_pipeline/install';
import { getAsset, getPathParts } from '../../archive';
import { removeAssetsFromInstalledEsByType, saveInstalledEsRefs } from '../../packages/install';
import { removeAssetTypesFromInstalledEs, saveInstalledEsRefs } from '../../packages/install';

import {
generateMappings,
Expand All @@ -34,52 +34,44 @@ export const installTemplates = async (
esClient: ElasticsearchClient,
paths: string[],
savedObjectsClient: SavedObjectsClientContract
): Promise<TemplateRef[]> => {
): Promise<IndexTemplateEntry[]> => {
// install any pre-built index template assets,
// atm, this is only the base package's global index templates
// Install component templates first, as they are used by the index templates
await installPreBuiltComponentTemplates(paths, esClient);
await installPreBuiltTemplates(paths, esClient);

// remove package installation's references to index templates
await removeAssetsFromInstalledEsByType(
savedObjectsClient,
installablePackage.name,
ElasticsearchAssetType.indexTemplate
);
await removeAssetTypesFromInstalledEs(savedObjectsClient, installablePackage.name, [
ElasticsearchAssetType.indexTemplate,
ElasticsearchAssetType.componentTemplate,
]);
// build templates per data stream from yml files
const dataStreams = installablePackage.data_streams;
if (!dataStreams) return [];

const installedTemplatesNested = await Promise.all(
dataStreams.map((dataStream) =>
installTemplateForDataStream({
pkg: installablePackage,
esClient,
dataStream,
})
)
);
const installedTemplates = installedTemplatesNested.flat();

// get template refs to save
const installedTemplateRefs = dataStreams.map((dataStream) => ({
id: generateTemplateName(dataStream),
type: ElasticsearchAssetType.indexTemplate,
}));
const installedIndexTemplateRefs = getAllTemplateRefs(installedTemplates);

// add package installation's references to index templates
await saveInstalledEsRefs(savedObjectsClient, installablePackage.name, installedTemplateRefs);

if (dataStreams) {
const installTemplatePromises = dataStreams.reduce<Array<Promise<TemplateRef>>>(
(acc, dataStream) => {
acc.push(
installTemplateForDataStream({
pkg: installablePackage,
esClient,
dataStream,
})
);
return acc;
},
[]
);

const res = await Promise.all(installTemplatePromises);
const installedTemplates = res.flat();
await saveInstalledEsRefs(
savedObjectsClient,
installablePackage.name,
installedIndexTemplateRefs
);

return installedTemplates;
}
return [];
return installedTemplates;
};

const installPreBuiltTemplates = async (paths: string[], esClient: ElasticsearchClient) => {
Expand Down Expand Up @@ -160,7 +152,7 @@ export async function installTemplateForDataStream({
pkg: InstallablePackage;
esClient: ElasticsearchClient;
dataStream: RegistryDataStream;
}): Promise<TemplateRef> {
}): Promise<IndexTemplateEntry> {
const fields = await loadFieldsFromYaml(pkg, dataStream.path);
return installTemplate({
esClient,
Expand All @@ -171,84 +163,118 @@ export async function installTemplateForDataStream({
});
}

interface TemplateMapEntry {
_meta: { package: { name: string } };
template:
| {
mappings: NonNullable<RegistryElasticsearch['index_template.mappings']>;
}
| {
settings: NonNullable<RegistryElasticsearch['index_template.settings']> | object;
};
}
type TemplateMap = Record<string, TemplateMapEntry>;
function putComponentTemplate(
body: object | undefined,
name: string,
esClient: ElasticsearchClient
): { clusterPromise: Promise<any>; name: string } | undefined {
if (body) {
const esClientParams = {
name,
body,
};

return {
// @ts-expect-error body expected to be ClusterPutComponentTemplateRequest
clusterPromise: esClient.cluster.putComponentTemplate(esClientParams, { ignore: [404] }),
name,
};
esClient: ElasticsearchClient,
params: {
body: TemplateMapEntry;
name: string;
create?: boolean;
}
): { clusterPromise: Promise<any>; name: string } {
const { name, body, create = false } = params;
return {
clusterPromise: esClient.cluster.putComponentTemplate(
// @ts-expect-error body is missing required key `settings`. TemplateMapEntry has settings *or* mappings
{ name, body, create },
{ ignore: [404] }
),
name,
};
}

function buildComponentTemplates(registryElasticsearch: RegistryElasticsearch | undefined) {
let mappingsTemplate;
let settingsTemplate;
const mappingsSuffix = '@mappings';
const settingsSuffix = '@settings';
const userSettingsSuffix = '@custom';
type TemplateBaseName = string;
type UserSettingsTemplateName = `${TemplateBaseName}${typeof userSettingsSuffix}`;

const isUserSettingsTemplate = (name: string): name is UserSettingsTemplateName =>
name.endsWith(userSettingsSuffix);

function buildComponentTemplates(params: {
templateName: string;
registryElasticsearch: RegistryElasticsearch | undefined;
packageName: string;
}) {
const { templateName, registryElasticsearch, packageName } = params;
const mappingsTemplateName = `${templateName}${mappingsSuffix}`;
const settingsTemplateName = `${templateName}${settingsSuffix}`;
const userSettingsTemplateName = `${templateName}${userSettingsSuffix}`;

const templatesMap: TemplateMap = {};
const _meta = { package: { name: packageName } };

if (registryElasticsearch && registryElasticsearch['index_template.mappings']) {
mappingsTemplate = {
templatesMap[mappingsTemplateName] = {
template: {
mappings: {
...registryElasticsearch['index_template.mappings'],
},
mappings: registryElasticsearch['index_template.mappings'],
},
_meta,
};
}

if (registryElasticsearch && registryElasticsearch['index_template.settings']) {
settingsTemplate = {
templatesMap[settingsTemplateName] = {
template: {
settings: registryElasticsearch['index_template.settings'],
},
_meta,
};
}
return { settingsTemplate, mappingsTemplate };
}

async function installDataStreamComponentTemplates(
templateName: string,
registryElasticsearch: RegistryElasticsearch | undefined,
esClient: ElasticsearchClient
) {
const templates: string[] = [];
const componentPromises: Array<Promise<any>> = [];
// return empty/stub template
templatesMap[userSettingsTemplateName] = {
template: {
settings: {},
},
_meta,
};

const compTemplates = buildComponentTemplates(registryElasticsearch);
return templatesMap;
}

const mappings = putComponentTemplate(
compTemplates.mappingsTemplate,
`${templateName}-mappings`,
esClient
);
async function installDataStreamComponentTemplates(params: {
templateName: string;
registryElasticsearch: RegistryElasticsearch | undefined;
esClient: ElasticsearchClient;
packageName: string;
}) {
const { templateName, registryElasticsearch, esClient, packageName } = params;
const templates = buildComponentTemplates({ templateName, registryElasticsearch, packageName });
const templateNames = Object.keys(templates);
const templateEntries = Object.entries(templates);

const settings = putComponentTemplate(
compTemplates.settingsTemplate,
`${templateName}-settings`,
esClient
// TODO: Check return values for errors
await Promise.all(
templateEntries.map(async ([name, body]) => {
if (isUserSettingsTemplate(name)) {
// look for existing user_settings template
const result = await esClient.cluster.getComponentTemplate({ name }, { ignore: [404] });
const hasUserSettingsTemplate = result.body.component_templates?.length === 1;
if (!hasUserSettingsTemplate) {
// only add if one isn't already present
const { clusterPromise } = putComponentTemplate(esClient, { body, name, create: true });
return clusterPromise;
}
} else {
const { clusterPromise } = putComponentTemplate(esClient, { body, name });
return clusterPromise;
}
})
);

if (mappings) {
templates.push(mappings.name);
componentPromises.push(mappings.clusterPromise);
}

if (settings) {
templates.push(settings.name);
componentPromises.push(settings.clusterPromise);
}

// TODO: Check return values for errors
await Promise.all(componentPromises);
return templates;
return templateNames;
}

export async function installTemplate({
Expand All @@ -263,7 +289,7 @@ export async function installTemplate({
dataStream: RegistryDataStream;
packageVersion: string;
packageName: string;
}): Promise<TemplateRef> {
}): Promise<IndexTemplateEntry> {
const validFields = processFields(fields);
const mappings = generateMappings(validFields);
const templateName = generateTemplateName(dataStream);
Expand Down Expand Up @@ -310,11 +336,12 @@ export async function installTemplate({
await esClient.indices.putIndexTemplate(updateIndexTemplateParams, { ignore: [404] });
}

const composedOfTemplates = await installDataStreamComponentTemplates(
const composedOfTemplates = await installDataStreamComponentTemplates({
templateName,
dataStream.elasticsearch,
esClient
);
registryElasticsearch: dataStream.elasticsearch,
esClient,
packageName,
});

const template = getTemplate({
type: dataStream.type,
Expand Down Expand Up @@ -342,3 +369,21 @@ export async function installTemplate({
indexTemplate: template,
};
}

export function getAllTemplateRefs(installedTemplates: IndexTemplateEntry[]) {
return installedTemplates.flatMap((installedTemplate) => {
const indexTemplates = [
{
id: installedTemplate.templateName,
type: ElasticsearchAssetType.indexTemplate,
},
];
const componentTemplates = installedTemplate.indexTemplate.composed_of.map(
(componentTemplateId) => ({
id: componentTemplateId,
type: ElasticsearchAssetType.componentTemplate,
})
);
return indexTemplates.concat(componentTemplates);
});
}
8 changes: 4 additions & 4 deletions x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import type { ElasticsearchClient } from 'kibana/server';
import type { Field, Fields } from '../../fields/field';
import type {
RegistryDataStream,
TemplateRef,
IndexTemplateEntry,
IndexTemplate,
IndexTemplateMappings,
} from '../../../../types';
Expand Down Expand Up @@ -456,7 +456,7 @@ function getBaseTemplate(

export const updateCurrentWriteIndices = async (
esClient: ElasticsearchClient,
templates: TemplateRef[]
templates: IndexTemplateEntry[]
): Promise<void> => {
if (!templates.length) return;

Expand All @@ -471,7 +471,7 @@ function isCurrentDataStream(item: CurrentDataStream[] | undefined): item is Cur

const queryDataStreamsFromTemplates = async (
esClient: ElasticsearchClient,
templates: TemplateRef[]
templates: IndexTemplateEntry[]
): Promise<CurrentDataStream[]> => {
const dataStreamPromises = templates.map((template) => {
return getDataStreams(esClient, template);
Expand All @@ -482,7 +482,7 @@ const queryDataStreamsFromTemplates = async (

const getDataStreams = async (
esClient: ElasticsearchClient,
template: TemplateRef
template: IndexTemplateEntry
): Promise<CurrentDataStream[] | undefined> => {
const { templateName, indexTemplate } = template;
const { body } = await esClient.indices.getDataStream({ name: `${templateName}-*` });
Expand Down
Loading

0 comments on commit f4b4d20

Please sign in to comment.