-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Reporting] ILM policy for managing reporting indices #100130
Changes from 6 commits
944c541
13885c6
c94974f
cc739ba
e9a27f7
385e170
ccaa9a7
bcabc5c
a838735
4f2aa3f
c74cf0e
be1d195
15f2d34
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License | ||
* 2.0; you may not use this file except in compliance with the Elastic License | ||
* 2.0. | ||
*/ | ||
|
||
import { PutLifecycleRequest } from '@elastic/elasticsearch/api/types'; | ||
|
||
// TODO: Review the default/starting policy | ||
export const reportingIlmPolicy: PutLifecycleRequest['body'] = { | ||
policy: { | ||
phases: { | ||
hot: { | ||
actions: {}, | ||
min_age: '0ms', | ||
}, | ||
}, | ||
}, | ||
}; |
Original file line number | Diff line number | Diff line change | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
@@ -14,6 +14,7 @@ import { ReportTaskParams } from '../tasks'; | |||||||||
import { indexTimestamp } from './index_timestamp'; | ||||||||||
import { mapping } from './mapping'; | ||||||||||
import { Report, ReportDocument, ReportSource } from './report'; | ||||||||||
import { reportingIlmPolicy } from './report_ilm_policy'; | ||||||||||
|
||||||||||
/* | ||||||||||
* When searching for long-pending reports, we get a subset of fields | ||||||||||
|
@@ -71,19 +72,22 @@ export class ReportingStore { | |||||||||
return exists; | ||||||||||
} | ||||||||||
|
||||||||||
const indexSettings = { | ||||||||||
number_of_shards: 1, | ||||||||||
auto_expand_replicas: '0-1', | ||||||||||
}; | ||||||||||
const body = { | ||||||||||
settings: indexSettings, | ||||||||||
mappings: { | ||||||||||
properties: mapping, | ||||||||||
}, | ||||||||||
}; | ||||||||||
|
||||||||||
try { | ||||||||||
await client.indices.create({ index: indexName, body }); | ||||||||||
await client.indices.create({ | ||||||||||
index: indexName, | ||||||||||
body: { | ||||||||||
settings: { | ||||||||||
number_of_shards: 1, | ||||||||||
auto_expand_replicas: '0-1', | ||||||||||
lifecycle: { | ||||||||||
name: this.ilmPolicyName, | ||||||||||
}, | ||||||||||
}, | ||||||||||
mappings: { | ||||||||||
properties: mapping, | ||||||||||
}, | ||||||||||
Comment on lines
+78
to
+88
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Moved this in here to take advantage of the TS typed body. If we want to move the body back outside we can also import the payload type to still have TS checking the payload here. |
||||||||||
}, | ||||||||||
}); | ||||||||||
|
||||||||||
return true; | ||||||||||
} catch (error) { | ||||||||||
|
@@ -130,6 +134,46 @@ export class ReportingStore { | |||||||||
return client.indices.refresh({ index }); | ||||||||||
} | ||||||||||
|
||||||||||
private readonly ilmPolicyName = 'kibana-reporting'; | ||||||||||
|
||||||||||
private async doesIlmPolicyExist(): Promise<boolean> { | ||||||||||
const client = await this.getClient(); | ||||||||||
try { | ||||||||||
await client.ilm.getLifecycle({ policy: this.ilmPolicyName }); | ||||||||||
return true; | ||||||||||
} catch (e) { | ||||||||||
if (e.statusCode === 404) { | ||||||||||
return false; | ||||||||||
} | ||||||||||
throw e; | ||||||||||
} | ||||||||||
} | ||||||||||
|
||||||||||
/** | ||||||||||
* Function to be called during plugin start phase. This ensures the environment is correctly | ||||||||||
* configured for storage of reports. | ||||||||||
*/ | ||||||||||
public async start() { | ||||||||||
const client = await this.getClient(); | ||||||||||
try { | ||||||||||
if (await this.doesIlmPolicyExist()) { | ||||||||||
this.logger.debug(`Found ILM policy ${this.ilmPolicyName}; skipping creation.`); | ||||||||||
return; | ||||||||||
} | ||||||||||
this.logger.debug( | ||||||||||
`Creating ILM policy for managing reporting indices: ${this.ilmPolicyName}` | ||||||||||
); | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
await client.ilm.putLifecycle({ | ||||||||||
policy: this.ilmPolicyName, | ||||||||||
body: reportingIlmPolicy, | ||||||||||
}); | ||||||||||
} catch (e) { | ||||||||||
this.logger.error('Error in start phase'); | ||||||||||
this.logger.error(e.body.error); | ||||||||||
throw e; | ||||||||||
} | ||||||||||
} | ||||||||||
|
||||||||||
public async addReport(report: Report): Promise<Report> { | ||||||||||
let index = report._index; | ||||||||||
if (!index) { | ||||||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need the
min_age
parameter? https://www.elastic.co/guide/en/elasticsearch/reference/master/getting-started-index-lifecycle-management.html#CO461-1 does state thatmin_age: '0ms'
is the default. However, if we can omit this parameter and allow Elastisearch to configure the default, it'll be more future proof if ES changes the default in the future.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point! Thanks for the update!