[WIP] Rename prospector to input #5944

ph · 2017-12-21T21:07:35Z

Rename usage of prospector to input inside the code base to align with the wording or other product like Logstash and also its better name for UDP and TCP.

This PR use type aliasing to make the old types works with the new names, this mean it should be backward compatible with the community beats without requiring them to changes their code. They will need golang 1.9 to leverage that feature, we have guards in place to fails the build in other version.

Also the usage of prospectors in the configuration and in the module definition will work until 7.0.

TODO

Rename prospector usage in python scripts.
Real world testing
Create a PR for the documentation. related to Beats doc updates in 6.x #5632
squash better commit message
split into multiple PR.

Uses type aliases and variable aliases to move all the execution into the input package

Since we are using type aliasing to make the backward compatibility works and this feature is only available in golang 1.9, we should fails hard if you try to use it in a previous golang release.

skip the file directory for now

structure

ph · 2017-12-22T14:58:06Z

@ruflin After changing the name from prospector to input, I believe we should also change of harvester, It sound weird in the context of TCP/UDP and also redis. I don't have a proposal yet for that.

harvester | ˈhärvəstər |
noun
a person or machine that gathers crops as a harvest: back-breaking labor as hired sugar beet harvesters | the world's first mechanical harvester of grapes.
• a person who catches or kills animals for human consumption: coastal fish harvesters are always the first to notice the decline of fish stocks.
• a person or organization that collects or obtains a resource for future use: one of the world's biggest harvesters of the sun's energy.

ruflin · 2017-12-26T22:21:57Z

Agree. I think harvesters together with prospectors made kind of sense but removing one the other one has to follow too. But I would do this in separate steps.

I had in the past discussions with @urso about this. There are at least 2 cases here. In same cases what we have as harvester is not needed anymore. For example for stdin there is only 1 prospector, 1 harvester and we have a 1-1 relation ship. Potentially harvester can become obsolete here. For the log case we have 1-n. In the log harvester big parts are abtracted as readers. The readers can be reused across different prospector/input types like multiline etc. They are chained together. We discussed in the past to rename harvester to reader but I think the way we have reader at the moment is pretty nice and it would conflict here.

An option could be to call it listeners. It only fits partially with the file case. The reason I liked harvester is it describes filebeat going "actively" to the file and check for new data.

One additional note for the prospector / input. For the file case (and probably some others) there is a specific scanner phase which we discussed to extract in the past which is responsible to scan the file system. It potentially could be exchanged with a different scanner (inotify). Just to add some more names to the equation.

Let's keep it "simple" for now and do only the prospector -> input renaming and I'm pretty sure if we add more inputs some of the names become more obvious.

ph · 2018-01-08T14:12:28Z

Agree lets do it in multiple steps, naming things are hard :)

urso · 2018-01-09T14:17:45Z

about harvester: From my point of view the prospectors are the inputs. If an input/prospector can handle the input directly, it should not be required to create another go-routine doing the actual work. The harvesters to date are basically just the workers of the log-input type -> therefore I'd just rename them to worker and even hide the fact we have workers from the user (it's an implementation detail, one could use select on FDs as well). If you really need a name collector or reader might do.

dedemorton

I was looking at this because I wanted to see where you are at with the renaming. I've added a few minor comments about stuff I noticed while reading through the message and config-related changes.

dedemorton · 2018-01-09T19:31:10Z

filebeat/_meta/common.reference.p2.yml

-# Each - is a prospector. Most options can be set at the prospector level, so
-# you can use different prospectors for various configurations.
-# Below are the prospector specific configurations.
+# List of input to fetch data.


I'd say inputs (plural)

dedemorton · 2018-01-09T19:31:58Z

filebeat/_meta/common.reference.p2.yml

 #
 # Possible options are:
 # * log: Reads every line of the log file (default)
 # * stdin: Reads the standard in

-#------------------------------ Log prospector --------------------------------
+#------------------------------ Log input --------------------------------


Should we change this to say "Input" instead of "Log input" given the various types we now support?

dedemorton · 2018-01-09T19:35:35Z

filebeat/tests/system/test_registrar.py

@@ -980,13 +980,13 @@ def test_restart_state(self):
        # Make sure all 4 states are persisted
        self.wait_until(
            lambda: self.log_contains(
-                "Prospector states cleaned up. Before: 4, After: 4", logfile="filebeat2.log"),
+                "input states cleaned up. Before: 4, After: 4", logfile="filebeat2.log"),


Shouldn't this be capitalized?

dedemorton · 2018-01-09T19:35:52Z

filebeat/tests/system/test_registrar.py

            max_timeout=10)

        # Wait until registry file is cleaned
        self.wait_until(
            lambda: self.log_contains(
-                "Prospector states cleaned up. Before: 0, After: 0", logfile="filebeat2.log"),
+                "input states cleaned up. Before: 0, After: 0", logfile="filebeat2.log"),


Same as previous comment

ph · 2018-01-09T21:09:30Z

@dedemorton Thanks for the check :)

ruflin · 2018-01-12T02:44:04Z

@ph Ping me when you think this needs a review.

ph · 2018-01-15T17:57:09Z

closing opening followup PRs.

ph added Filebeat Filebeat in progress Pull request is currently in progress. refactoring labels Dec 21, 2017

ph added 27 commits December 21, 2017 16:25

First step of refactoring move file into an input package

78107fc

Uses type aliases and variable aliases to move all the execution into the input package

Rename Prospector interface to input

fea2be3

Rename Prospectorer struct to InputRunner

9cd712f

Rename prospectorConfig to inputConfig

8d8719e

rename variables

9d9b63f

change variables

7d5748a

inputrunner -> runner

87360a8

Retricts go version to build the backward compatibility layer

a403fe3

Since we are using type aliasing to make the backward compatibility works and this feature is only available in golang 1.9, we should fails hard if you try to use it in a previous golang release.

rename prospector -> input

7978674

prospector -> input

d3b7b84

more rename prospector -> input

3728c00

Renaming prospector -> input

53d4026

rename prospector -> input

be47d56

prospector -> input

d8ec7e5

prospector -> input

1f10990

prospector -> input

6bf250c

prospector -> input

e4e0db1

prospector -> input

fb470ef

rename prospector -> input

aa66a55

prospector -> input

c04bb35

prospector -> input

4834d76

make it configurable

ef85195

Fileset correctly migrate from prospector to input

8d2d4b1

pluginKey from prospector to input

32d179e

Allow module manifest to define either an input or a prospector

743acbc

Backward compatibility last fixes

7e703a8

Add the input to the event data

aa09e39

ph added 16 commits December 21, 2017 16:25

Migrate: nginx prospector -> input (json)

791a5d2

Migrate: mysql prospector -> input (yaml)

250daf9

Migrate: logstash prospector -> input (yaml)

c65a132

Migrate: Logstash module prospector -> input (json)

c5731d3

Migrate: Kafka module prospector -> input (yaml)

1478e5d

Migrate: Kafka module prospector -> input (json)

7e87955

Migrate: icinga module prospector -> input (yaml)

a6ac2e8

Migrate: icinga module prospector -> input (json)

7d68769

Migrate: Auditd module prospector -> input (yaml)

7fa797b

Migrate: Auditd module prospector -> input (json)

bd3ff74

Migrate: apache2 module prospector -> input (yaml)

0b80320

Migrate: apache2 module prospector -> input (json)

edfe266

Update filebeat yml files and template for inputs

52682a5

Add input.type field and keep prospector.type for backward compatibility

4c8fdd9

Adjust the import generator to check into the input folder instead and

6501bed

skip the file directory for now

missing changing packages

0af0e45

ph force-pushed the refactor/rename-prospector-to-input branch from 0b923cd to 0af0e45 Compare December 21, 2017 21:26

Adjust docker compose to find the docker file into the new directory

825831e

structure

dedemorton reviewed Jan 9, 2018

View reviewed changes

dedemorton mentioned this pull request Jan 9, 2018

Beats doc updates in 6.x #5632

Closed

37 tasks

ph closed this Jan 15, 2018

ph mentioned this pull request Jan 15, 2018

Refactor: Renaming prospector to input #6078

Merged

dedemorton mentioned this pull request Mar 13, 2018

Refactor input docs #6537

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Rename prospector to input #5944

[WIP] Rename prospector to input #5944

ph commented Dec 21, 2017 •

edited

Loading

ph commented Dec 22, 2017

ruflin commented Dec 26, 2017

ph commented Jan 8, 2018

urso commented Jan 9, 2018

dedemorton left a comment

dedemorton Jan 9, 2018

dedemorton Jan 9, 2018

dedemorton Jan 9, 2018

dedemorton Jan 9, 2018

ph commented Jan 9, 2018

ruflin commented Jan 12, 2018

ph commented Jan 15, 2018

[WIP] Rename prospector to input #5944

[WIP] Rename prospector to input #5944

Conversation

ph commented Dec 21, 2017 • edited Loading

TODO

ph commented Dec 22, 2017

ruflin commented Dec 26, 2017

ph commented Jan 8, 2018

urso commented Jan 9, 2018

dedemorton left a comment

Choose a reason for hiding this comment

dedemorton Jan 9, 2018

Choose a reason for hiding this comment

dedemorton Jan 9, 2018

Choose a reason for hiding this comment

dedemorton Jan 9, 2018

Choose a reason for hiding this comment

dedemorton Jan 9, 2018

Choose a reason for hiding this comment

ph commented Jan 9, 2018

ruflin commented Jan 12, 2018

ph commented Jan 15, 2018

ph commented Dec 21, 2017 •

edited

Loading