Skip to content

Commit

Permalink
Merge branch 'master' into #60333_agg_config
Browse files Browse the repository at this point in the history
  • Loading branch information
elasticmachine authored Apr 14, 2020
2 parents d2adc78 + d015c24 commit 0f15ac3
Show file tree
Hide file tree
Showing 651 changed files with 11,506 additions and 11,000 deletions.
1 change: 1 addition & 0 deletions .i18nrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
"src/legacy/core_plugins/management",
"src/plugins/management"
],
"maps_legacy": "src/plugins/maps_legacy",
"indexPatternManagement": "src/plugins/index_pattern_management",
"advancedSettings": "src/plugins/advanced_settings",
"kibana_legacy": "src/plugins/kibana_legacy",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ export declare class Field implements IFieldType
| [format](./kibana-plugin-plugins-data-public.indexpatternfield.format.md) | | <code>any</code> | |
| [lang](./kibana-plugin-plugins-data-public.indexpatternfield.lang.md) | | <code>string</code> | |
| [name](./kibana-plugin-plugins-data-public.indexpatternfield.name.md) | | <code>string</code> | |
| [routes](./kibana-plugin-plugins-data-public.indexpatternfield.routes.md) | | <code>Record&lt;string, string&gt;</code> | |
| [script](./kibana-plugin-plugins-data-public.indexpatternfield.script.md) | | <code>string</code> | |
| [scripted](./kibana-plugin-plugins-data-public.indexpatternfield.scripted.md) | | <code>boolean</code> | |
| [searchable](./kibana-plugin-plugins-data-public.indexpatternfield.searchable.md) | | <code>boolean</code> | |
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,5 @@ export declare class FieldList extends Array<Field> implements IFieldList
| [getByName](./kibana-plugin-plugins-data-public.indexpatternfieldlist.getbyname.md) | | <code>(name: string) =&gt; Field &#124; undefined</code> | |
| [getByType](./kibana-plugin-plugins-data-public.indexpatternfieldlist.getbytype.md) | | <code>(type: string) =&gt; any[]</code> | |
| [remove](./kibana-plugin-plugins-data-public.indexpatternfieldlist.remove.md) | | <code>(field: IFieldType) =&gt; void</code> | |
| [update](./kibana-plugin-plugins-data-public.indexpatternfieldlist.update.md) | | <code>(field: Field) =&gt; void</code> | |
| [update](./kibana-plugin-plugins-data-public.indexpatternfieldlist.update.md) | | <code>(field: Record&lt;string, any&gt;) =&gt; void</code> | |
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,5 @@
<b>Signature:</b>

```typescript
update: (field: Field) => void;
update: (field: Record<string, any>) => void;
```
Binary file added docs/images/tutorial-ilm-custom-policy.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/tutorial-ilm-delete-rollover.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -1,23 +1,179 @@
[role="xpack"]

[[example-using-index-lifecycle-policy]]
=== Example of using an index lifecycle policy
=== Tutorial: Use {ilm-init} to manage {filebeat} time-based indices

With {ilm} ({ilm-init}), you can create policies that perform actions automatically
on indices as they age and grow. {ilm-init} policies help you to manage
performance, resilience, and retention of your data during its lifecycle. This tutorial shows
you how to use {kib}’s *Index Lifecycle Policies* to modify and create {ilm-init}
policies. You can learn more about all of the actions, benefits, and lifecycle
phases in the {ref}/overview-index-lifecycle-management.html[{ilm-init} overview].


[discrete]
[[example-using-index-lifecycle-policy-scenario]]
==== Scenario

You’re tasked with sending syslog files to an {es} cluster. This
log data has the following data retention guidelines:

* Keep logs on hot data nodes for 30 days
* Roll over to a new index if the size reaches 50GB
* After 30 days:
** Move the logs to warm data nodes
** Set {ref}/glossary.html#glossary-replica-shard[replica shards] to 1
** {ref}/indices-forcemerge.html[Force merge] multiple index segments to free up the space used by deleted documents
* Delete logs after 90 days


[discrete]
[[example-using-index-lifecycle-policy-prerequisites]]
==== Prerequisites

To complete this tutorial, you'll need:

* An {es} cluster with hot and warm nodes configured for shard allocation
awareness. If you’re using {cloud}/ec-getting-started-templates-hot-warm.html[{ess}],
choose the hot-warm architecture deployment template.

+
For a self-managed cluster, add node attributes as described for {ref}/shard-allocation-filtering.html[shard allocation filtering]
to label data nodes as hot or warm. This step is required to migrate shards between
nodes configured with specific hardware for the hot or warm phases.
+
For example, you can set this in your `elasticsearch.yml` for each data node:
+
[source,yaml]
--------------------------------------------------------------------------------
node.attr.data: "warm"
--------------------------------------------------------------------------------

* A server with {filebeat} installed and configured to send logs to the `elasticsearch`
output as described in {filebeat-ref}/filebeat-getting-started.html[Getting Started with {filebeat}].

[discrete]
[[example-using-index-lifecycle-policy-view-fb-ilm-policy]]
==== View the {filebeat} {ilm-init} policy

{filebeat} includes a default {ilm-init} policy that enables rollover. {ilm-init}
is enabled automatically if you’re using the default `filebeat.yml` and index template.

To view the default policy in {kib}, go to *Management > Index Lifecycle Policies*,
search for _filebeat_, and choose the _filebeat-version_ policy.

This policy initiates the rollover action when the index size reaches 50GB or
becomes 30 days old.

[role="screenshot"]
image::images/tutorial-ilm-hotphaserollover-default.png["Default policy"]


[float]
==== Modify the policy

The default policy is enough to prevent the creation of many tiny daily indices.
You can modify the policy to meet more complex requirements.

. Activate the warm phase.

+
. Set either of the following options to control when the index moves to the warm phase:

** Provide a value for *Timing for warm phase*. Setting this to *15* keeps the
indices on hot nodes for a range of 15-45 days, depending on when the initial
rollover occurred.

** Enable *Move to warm phase on rollover*. The index might move to the warm phase
more quickly than intended if it reaches the *Maximum index size* before the
the *Maximum age*.

. In the *Select a node attribute to control shard allocation* dropdown, select
*data:warm(2)* to migrate shards to warm data nodes.

. Change *Number of replicas* to *1*.

. Enable *Force merge data* and set *Number of segments* to *1*.
+
NOTE: When rollover is enabled in the hot phase, action timing in the other phases
is based on the rollover date.

+
[role="screenshot"]
image::images/tutorial-ilm-modify-default-warm-phase-rollover.png["Modify to add warm phase"]

. Activate the delete phase and set *Timing for delete phase* to *90* days.
+
[role="screenshot"]
image::images/tutorial-ilm-delete-rollover.png["Add a delete phase"]

[float]
==== Create a custom policy

If meeting a specific retention time period is most important, you can create a
custom policy. For this option, you will use {filebeat} daily indices without
rollover.

. Create a custom policy in {kib}, go to *Management > Index Lifecycle Policies >
Create Policy*.

. Activate the warm phase and configure it as follows:
+
|===
|*Setting* |*Value*

|Timing for warm phase
|30 days from index creation

|Node attribute
|`data:warm`

|Number of replicas
|1

|Force merge data
|enable

|Number of segments
|1
|===

+
[role="screenshot"]
image::images/tutorial-ilm-custom-policy.png["Modify the custom policy to add a warm phase"]


A common use case for managing index lifecycle policies is when you’re using
{beats-ref}/beats-reference.html[Beats] to continually send time-series data,
such as metrics and log data, to {es}. When you create the Beats packages, an
index template is installed. The template includes a default policy to apply
when new indices are created.
+
. Activate the delete phase and set the timing.
+
|===
|*Setting* |*Value*
|Timing for delete phase
|90
|===

You can edit the policy in {kib}'s *Index Lifecycle Policies*. For example, you might:
+
[role="screenshot"]
image::images/tutorial-ilm-delete-phase-creation.png["Delete phase"]

* Rollover the index when it reaches 50 GB in size or is 30 days old. These
settings are the default for the Beats lifecycle policy. This avoids
having 1000s of tiny indices. When a rollover occurs, a new “hot” index is
created and added to the index alias.
. Configure the index to use the new policy in *{kib} > Management > Index Lifecycle
Policies*

* Move the index into the warm phase, shrink the index down to a single shard,
and force merge to a single segment.
.. Find your {ilm-init} policy.
.. Click the *Actions* link next to your policy name.
.. Choose *Add policy to index template*.
.. Select your {filebeat} index template name from the *Index template* list. For example, `filebeat-7.5.x`.
.. Click *Add Policy* to save the changes.

* After 60 days, move the index into the cold phase and onto less expensive hardware.
+
NOTE: If you initially used the default {filebeat} {ilm-init} policy, you will
see a notice that the template already has a policy associated with it. Confirm
that you want to overwrite that configuration.

* Delete the index after 90 days.
+
+
TIP: When you change the policy associated with the index template, the active
index will continue to use the policy it was associated with at index creation
unless you manually update it. The next new index will use the updated policy.
For more reasons that your {ilm-init} policy changes might be delayed, see
{ref}/update-lifecycle-policy.html#update-lifecycle-policy[Update Lifecycle Policy].
5 changes: 2 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -376,7 +376,7 @@
"@types/recompose": "^0.30.6",
"@types/redux-actions": "^2.6.1",
"@types/request": "^2.48.2",
"@types/selenium-webdriver": "4.0.9",
"@types/selenium-webdriver": "^4.0.5",
"@types/semver": "^5.5.0",
"@types/sinon": "^7.0.13",
"@types/strip-ansi": "^3.0.0",
Expand Down Expand Up @@ -462,7 +462,6 @@
"load-grunt-config": "^3.0.1",
"mocha": "^7.1.1",
"mock-http-server": "1.3.0",
"ms-chromium-edge-driver": "^0.2.0",
"multistream": "^2.1.1",
"murmurhash3js": "3.0.1",
"mutation-observer": "^1.0.3",
Expand All @@ -481,7 +480,7 @@
"react-textarea-autosize": "^7.1.2",
"regenerate": "^1.4.0",
"sass-lint": "^1.12.1",
"selenium-webdriver": "^4.0.0-alpha.7",
"selenium-webdriver": "^4.0.0-alpha.5",
"simple-git": "1.116.0",
"simplebar-react": "^2.1.0",
"sinon": "^7.4.2",
Expand Down
Loading

0 comments on commit 0f15ac3

Please sign in to comment.