Rename input_type to type in config and input_type to prospector.type…

… in event Document type was removed recently because _type does not exist anymore in Elasticsearch. As input_type is actually the type of the prospector it makes more sense to have it as type config options instead of type. Renaming it in the config means it should also be renamed in the event. In the event the field was renamed to `prospector.type`. This change is on the config side backward compatible as input_type was only deprecated. On the event side the old field does not exist anymore. * Cleanup expected test json files. Indentation was standardised and `input_type` replaced by `prospector.type`. * Update changelog * Add system tests for deprecated message
elastic · May 15, 2017 · ac35a91 · ac35a91
1 parent 9ff2c65
commit ac35a91
Show file tree

Hide file tree

Showing 52 changed files with 1,135 additions and 1,043 deletions.
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
@@ -15,6 +15,7 @@ https://github.com/elastic/beats/compare/v6.0.0-alpha1...master[Check the HEAD d
 *Affecting all Beats*
 
 *Filebeat*
+- Rename `input_type` field to `prospector.type` {pull}4294[4294]
 
 *Heartbeat*
 
@@ -66,6 +67,8 @@ https://github.com/elastic/beats/compare/v6.0.0-alpha1...master[Check the HEAD d
 *Affecting all Beats*
 
 *Filebeat*
+- Deprecate `input_type` prospector config. Use `type` config option instead. {pull}4294[4294]
+
 
 *Heartbeat*
 

diff --git a/filebeat/_meta/common.full.p2.yml b/filebeat/_meta/common.full.p2.yml
@@ -14,7 +14,7 @@ filebeat.prospectors:
 # * stdin: Reads the standard in
 
 #------------------------------ Log prospector --------------------------------
-- input_type: log
+- type: log
 
   # Change to true to enable this prospector configuration.
   enabled: false
@@ -208,7 +208,7 @@ filebeat.prospectors:
 
 #----------------------------- Stdin prospector -------------------------------
 # Configuration to use stdin input
-#- input_type: stdin
+#- type: stdin
 
 #========================= Filebeat global options ============================
 

diff --git a/filebeat/_meta/common.p2.yml b/filebeat/_meta/common.p2.yml
@@ -9,7 +9,7 @@ filebeat.prospectors:
 # you can use different prospectors for various configurations.
 # Below are the prospector specific configurations.
 
-- input_type: log
+- type: log
 
   # Change to true to enable this prospector configuration.
   enabled: false

diff --git a/filebeat/_meta/fields.common.yml b/filebeat/_meta/fields.common.yml
@@ -23,10 +23,10 @@
       description: >
         The content of the line read from the log file.
 
-    - name: input_type
+    - name: prospector.type
       required: true
       description: >
-        The input type from which the event was generated. This field is set to the value specified for the `input_type` option in the prospector section of the Filebeat config file.
+        The prospector type from which the event was generated. This field is set to the value specified for the `type` option in the prospector section of the Filebeat config file.
 
     - name: read_timestamp
       description: >

diff --git a/filebeat/config/config.go b/filebeat/config/config.go
@@ -15,7 +15,7 @@ import (
 
 // Defaults for config variables which are not set
 const (
-	DefaultInputType = "log"
+	DefaultType = "log"
 )
 
 type Config struct {
@@ -39,15 +39,16 @@ var (
 	}
 )
 
+// Contains available prospector types
 const (
-	LogInputType   = "log"
-	StdinInputType = "stdin"
+	LogType   = "log"
+	StdinType = "stdin"
 )
 
 // List of valid input types
-var ValidInputType = map[string]struct{}{
-	StdinInputType: {},
-	LogInputType:   {},
+var ValidType = map[string]struct{}{
+	StdinType: {},
+	LogType:   {},
 }
 
 // getConfigFiles returns list of config files.

diff --git a/filebeat/docs/fields.asciidoc b/filebeat/docs/fields.asciidoc
@@ -768,11 +768,11 @@ The content of the line read from the log file.
 
 
 [float]
-=== input_type
+=== prospector.type
 
 required: True
 
-The input type from which the event was generated. This field is set to the value specified for the `input_type` option in the prospector section of the Filebeat config file.
+The prospector type from which the event was generated. This field is set to the value specified for the `type` option in the prospector section of the Filebeat config file.
 
 
 [float]

diff --git a/filebeat/docs/filebeat-filtering.asciidoc b/filebeat/docs/filebeat-filtering.asciidoc
@@ -93,7 +93,9 @@ The resulting output looks something like this:
   "inner": {
     "data": "value"
   },
-  "input_type": "log",
+  "prospector": {
+    "type": "log",
+  },
   "offset": 55,
   "outer": "value",
   "source": "input.json",

diff --git a/filebeat/docs/getting-started.asciidoc b/filebeat/docs/getting-started.asciidoc
@@ -137,7 +137,7 @@ started experience for common log formats. See <<filebeat-modules-quickstart>>
 to learn how to get started with modules. If you use Filebeat modules to get
 started, you can skip the content in this section, including the remaining
 getting started steps, and go directly to the <<filebeat-modules-quickstart>>
-page. 
+page.
 
 To configure Filebeat manually, you edit the configuration file. For rpm and deb,
 you'll find the configuration file at `/etc/filebeat/filebeat.yml`. For mac and
@@ -155,7 +155,7 @@ default values for most configuration options.
 [source,yaml]
 -------------------------------------------------------------------------------------
 filebeat.prospectors:
-- input_type: log
+- type: log
   paths:
     - /var/log/*.log
     #- c:\programdata\elasticsearch\logs\*
@@ -170,7 +170,7 @@ For the most basic Filebeat configuration, you can define a single prospector wi
 [source,yaml]
 -------------------------------------------------------------------------------------
 filebeat.prospectors:
-- input_type: log
+- type: log
   paths:
     - /var/log/*.log
 -------------------------------------------------------------------------------------

diff --git a/filebeat/docs/how-filebeat-works.asciidoc b/filebeat/docs/how-filebeat-works.asciidoc
@@ -1,9 +1,9 @@
 [[how-filebeat-works]]
 == How Filebeat Works
 
-In this topic, you learn about the key building blocks of Filebeat and how they work together. Understanding these concepts will help you make informed decisions about configuring Filebeat for specific use cases.  
+In this topic, you learn about the key building blocks of Filebeat and how they work together. Understanding these concepts will help you make informed decisions about configuring Filebeat for specific use cases.
 
-Filebeat consists of two main components: <<prospector,prospectors>> and <<harvester,harvesters>>. These components work together to tail files and send event data to the output that you specify. 
+Filebeat consists of two main components: <<prospector,prospectors>> and <<harvester,harvesters>>. These components work together to tail files and send event data to the output that you specify.
 
 
 [float]
@@ -26,14 +26,14 @@ To control when a harvester is closed, use the <<close-options,`close_*`>> confi
 
 A prospector is responsible for managing the harvesters and finding all sources to read from.
 
-If the input type is `log`, the prospector finds all files on the drive that match the defined glob paths and starts a harvester for each file. Each prospector runs in its own Go routine. 
+If the input type is `log`, the prospector finds all files on the drive that match the defined glob paths and starts a harvester for each file. Each prospector runs in its own Go routine.
 
 The following example configures Filebeat to harvest lines from all log files that match the specified glob patterns:
 
 [source,yaml]
 -------------------------------------------------------------------------------------
 filebeat.prospectors:
-- input_type: log
+- type: log
   paths:
     - /var/log/*.log
     - /var/path2/*.log
@@ -50,19 +50,19 @@ Filebeat keeps the state of each file and frequently flushes the state to disk i
 
 Each prospector keeps a state for each file it finds. Because files can be renamed or moved, the filename and path are not enough to identify a file. For each file, Filebeat stores unique identifiers to detect whether a file was harvested previously.
 
-If your use case involves creating a large number of new files every day, you might find that the registry file grows to be too large. See <<reduce-registry-size>> for details about configuration options that you can set to resolve this issue. 
+If your use case involves creating a large number of new files every day, you might find that the registry file grows to be too large. See <<reduce-registry-size>> for details about configuration options that you can set to resolve this issue.
 
 [float]
 [[at-least-once-delivery]]
 === How Does Filebeat Ensure At-Least-Once Delivery?
 
 Filebeat guarantees that events will be delivered to the configured output at
 least once and with no data loss. Filebeat is able to achieve this behavior
-because it stores the delivery state of each event in the registry file. 
+because it stores the delivery state of each event in the registry file.
 
 In situations where the defined output is blocked and has not confirmed all
 events, Filebeat will keep trying to send events until the output acknowledges
-that it has received the events. 
+that it has received the events.
 
 If Filebeat shuts down while it's in the process of sending events, it does not
 wait for the output to acknowledge all events before shutting down. Any events

diff --git a/filebeat/docs/load-balancing.asciidoc b/filebeat/docs/load-balancing.asciidoc
@@ -37,7 +37,7 @@ Example:
 [source,yaml]
 -------------------------------------------------------------------------------
 filebeat.prospectors:
-- input_type: log
+- type: log
   paths:
     - /var/log/*.log
 output.logstash:
@@ -63,7 +63,7 @@ Example:
 [source,yaml]
 -------------------------------------------------------------------------------
 filebeat.prospectors:
-- input_type: log
+- type: log
   paths:
     - /var/log/*.log
 filebeat.spool_size: 4096

diff --git a/filebeat/docs/migration.asciidoc b/filebeat/docs/migration.asciidoc
@@ -135,24 +135,24 @@ The equivalent `prospectors` section would look like this:
 [source,yaml]
 -------------------------------------------------------------------------------------
 filebeat.prospectors:
-- input_type: log
+- type: log
   paths:
     - /var/log/messages
     - /var/log/*.log
   fields:
     service: apache
     zone: us-east-1
   fields_under_root: true
-- input_type: stdin <1>
-- input_type: log
+- type: stdin <1>
+- type: log
   paths:
     - /var/log/apache2/httpd-*.log
 -------------------------------------------------------------------------------------
 
-<1> The explicit `input_type` option was introduced to differentiate between normal files and
+<1> The explicit `type` option was introduced to differentiate between normal files and
     stdin. In the future, additional types might be supported.
 
-As you can see, apart from the new `input_type` options,
+As you can see, apart from the new `type` options,
 which were before implicitly defined via the `type` custom field, the remaining
 options can be migrated mechanically.
 
@@ -279,7 +279,7 @@ Filebeat configuration:
 [source,yaml]
 -------------------------------------------------------------------------------------
 filebeat.prospectors:
-- input_type: log
+- type: log
   paths:
     - /var/log/*.log
   fields:
@@ -366,7 +366,7 @@ you can use the <<fields-under-root>> configuration option:
 [source,yaml]
 -------------------------------------------------------------------------------------
 filebeat.prospectors:
-- input_type: log
+- type: log
   paths:
     - /var/log/*.log
   fields:

diff --git a/filebeat/docs/multiple-prospectors.asciidoc b/filebeat/docs/multiple-prospectors.asciidoc
@@ -4,22 +4,22 @@
 When you need to collect lines from multiple files, you can simply configure a single prospector and specify multiple
 paths to start a harvester for each file. However, if you want to apply additional prospector-specific
 <<configuration-filebeat-options,configuration settings>> (such as `fields`, `include_lines`, `exclude_lines`, `multiline`, and so on)
-to the lines harvested from specific files, you need to define multiple prospectors in the Filebeat config file. 
+to the lines harvested from specific files, you need to define multiple prospectors in the Filebeat config file.
 
 Within the config file, you can specify multiple prospectors, and each prospector can define multiple paths to crawl, as
-shown in the following example. 
+shown in the following example.
 
 NOTE: Make sure a file is not defined more than once across all prospectors because this can lead
 to unexpected behaviour.
 
 [source,yaml]
 -------------------------------------------------------------------------------------
 filebeat.prospectors:
-- input_type: log
+- type: log
   paths:
     - /var/log/system.log
     - /var/log/wifi.log
-- input_type: log
+- type: log
   paths:
     - "/var/log/apache2/*"
   fields:

diff --git a/filebeat/docs/reference/configuration/filebeat-options.asciidoc b/filebeat/docs/reference/configuration/filebeat-options.asciidoc
@@ -11,26 +11,26 @@ Here is a sample configuration:
 [source,yaml]
 -------------------------------------------------------------------------------------
 filebeat.prospectors:
-- input_type: log
+- type: log
   paths:
     - /var/log/apache/httpd-*.log
 
-- input_type: log
+- type: log
   paths:
     - /var/log/messages
     - /var/log/*.log
 -------------------------------------------------------------------------------------
 
 ==== Options
 
-===== input_type
+===== type
 
 One of the following input types:
 
     * log: Reads every line of the log file (default)
     * stdin: Reads the standard in
 
-The value that you specify here is used as the `input_type` for each event published to Logstash and Elasticsearch.
+The value that you specify here is used as the `type` for each event published to Logstash and Elasticsearch.
 
 [[prospector-paths]]
 ===== paths

diff --git a/filebeat/docs/reference/configuration/reload-configuration.asciidoc b/filebeat/docs/reference/configuration/reload-configuration.asciidoc
@@ -4,7 +4,7 @@
 beta[]
 
 You can configure Filebeat to dynamically reload prospector configuration files
-when there are changes. To do this, you specify a path 
+when there are changes. To do this, you specify a path
 (https://golang.org/pkg/path/filepath/#Glob[Glob]) to watch for prospector
 configuration changes. When the files found by the Glob change, new prospectors
 are started/stopped according to changes in the configuration files.
@@ -13,7 +13,7 @@ This feature is especially useful in container environments where one container
 is used to tail logs for services running in other containers on the same host.
 
 To enable dynamic config reloading, you specify the `path` and `reload` options
-in the main `filebeat.yml` config file. For example: 
+in the main `filebeat.yml` config file. For example:
 
 [source,yaml]
 ------------------------------------------------------------------------------
@@ -23,24 +23,24 @@ filebeat.config.prospectors:
   reload.period: 10s
 ------------------------------------------------------------------------------
 
-`path`:: A Glob that defines the files to check for changes. 
-`reload.enabled`:: When set to `true`, enables dynamic config reload. 
+`path`:: A Glob that defines the files to check for changes.
+`reload.enabled`:: When set to `true`, enables dynamic config reload.
 `reload.period`:: Specifies how often the files are checked for changes. Do not
 set the `period` to less than 1s because the modification time of files is often
 stored in seconds. Setting the `period` to less than 1s will result in
 unnecessary overhead.
 
 Each file found by the Glob must contain a list of one or more prospector
-definitions. For example: 
+definitions. For example:
 
 [source,yaml]
 ------------------------------------------------------------------------------
-- input_type: log
+- type: log
   paths:
     - /var/log/mysql.log
   scan_frequency: 10s
 
-- input_type: log
+- type: log
   paths:
     - /var/log/apache.log
   scan_frequency: 5s

diff --git a/filebeat/docs/system-module-note.asciidoc b/filebeat/docs/system-module-note.asciidoc
@@ -7,7 +7,7 @@ you plan to run the `system` module, edit the Filebeat configuration file,
 
 [source,yaml]
 ----------------------------------------------------------------------
-#- input_type: log
+#- type: log
   #paths:
     #- /var/log/*.log
 ----------------------------------------------------------------------
@@ -16,4 +16,4 @@ For rpm and deb, you'll find the configuration file at
 `/etc/filebeat/filebeat.yml`. For mac and win, look in the archive that you
 extracted when you installed Filebeat.
 
-===============================================================================
+===============================================================================
diff --git a/filebeat/filebeat.full.yml b/filebeat/filebeat.full.yml
@@ -183,7 +183,7 @@ filebeat.prospectors:
 # * stdin: Reads the standard in
 
 #------------------------------ Log prospector --------------------------------
-- input_type: log
+- type: log
 
   # Change to true to enable this prospector configuration.
   enabled: false
@@ -377,7 +377,7 @@ filebeat.prospectors:
 
 #----------------------------- Stdin prospector -------------------------------
 # Configuration to use stdin input
-#- input_type: stdin
+#- type: stdin
 
 #========================= Filebeat global options ============================