Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor FileWatchSensor to remove logshipper #5096

Draft
wants to merge 44 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
6a8dc6d
Refactor FileWatchSensor to use watchdog instead of our logshipper fork
blag Nov 29, 2020
0a2f6ae
Remove logshipper and pyinotify from requirements files
blag Nov 29, 2020
2f71b2f
Import eventlet since we use it
blag Dec 12, 2020
3f0abb2
Use self.logger instead of self._logger
blag Dec 12, 2020
d175507
Use self.trigger instead of self._trigger
blag Dec 12, 2020
ce6e8ef
Merge branch 'master' into remove-logshipper
blag Dec 15, 2020
6657a25
Merge branch 'master' into remove-logshipper
blag Dec 18, 2020
8add2cc
Refactor for testability and test coverage
blag Dec 22, 2020
169cc5a
Add tests for FileWatchSensor components
blag Dec 22, 2020
fa68bef
Linting
blag Dec 23, 2020
1493656
Merge branch 'master' into remove-logshipper
blag Dec 23, 2020
e932ffe
Add class docstrings
blag Jan 17, 2021
5a22c27
Add logging
blag Jan 17, 2021
490595d
No need for eventlet.sleep() when time.sleep() should do just fine
blag Jan 17, 2021
05603f6
Refactor TailManageer.run() to use time.sleep()
blag Jan 20, 2021
8a05c36
Refactor calculating parent_dir
blag Jan 20, 2021
847d830
Refactor SingleFileTail.close() to keep parent watch open
blag Jan 20, 2021
914b3f5
Refactor follow logic
blag Jan 20, 2021
5caa459
Refactor TailManager.start() to make it idempotent
blag Jan 20, 2021
e6d3911
Refactor FileWatchSensor.run() to use TailManager.run()
blag Jan 20, 2021
456daba
Break out tests into multiple modules
blag Jan 20, 2021
a19eac6
Merge branch 'master' into remove-logshipper
blag Apr 8, 2021
4279986
Add .vagrant/ to .gitignore
blag Apr 8, 2021
d915752
Merge branch 'master' into remove-logshipper
blag Apr 12, 2021
5601743
Ignore a few more virtualenv directories
blag Apr 12, 2021
0c10559
Make pylint happy
blag Apr 12, 2021
f8a3056
Run all tests when run directly
blag Apr 12, 2021
5f3cf9a
Make black happy
blag Apr 12, 2021
1e5e389
Make flake8 happy
blag Apr 12, 2021
5e0b7e6
Make pylint happy
blag Apr 12, 2021
d90be9c
Make black happy
blag Apr 12, 2021
87d32aa
Don't use threading.Thread() after all
blag Apr 12, 2021
e64e547
Do whatever it takes to make all of the linters happy
blag Apr 12, 2021
ec856da
Merge branch 'master' into remove-logshipper
cognifloyd Oct 5, 2021
dca3e4f
Merge branch 'master' into remove-logshipper
cognifloyd Oct 6, 2021
3055413
add changelog entry
cognifloyd Oct 7, 2021
9631524
Merge branch 'master' into remove-logshipper
cognifloyd Jan 20, 2024
83fa1ec
Fix borked merge
cognifloyd Jan 20, 2024
bab2952
pin transitive dep in test-requirements.txt
cognifloyd Jan 20, 2024
f8d1f50
Merge branch 'master' into remove-logshipper
cognifloyd Jan 29, 2024
5b38106
regen st2 lockfile to switch logshipper -> watchdog
cognifloyd Jan 29, 2024
ed32348
linux pack: Drop out-of-date README notes about pack config
cognifloyd Jan 30, 2024
465a8f4
update the linux pack README to explain how to use FileWatchSensor
cognifloyd Jan 30, 2024
36bacae
linux pack: add LinuxFileWatchSensor.update_trigger
cognifloyd Jan 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ conf/st2.travis.conf
# generated GitHub Actions conf
conf/st2.githubactions.conf

# Vagrant
.vagrant/

# Installer logs
pip-log.txt

Expand Down
9 changes: 9 additions & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,15 @@ Changed
~~~~~~~
* Bumped `jsonschema` 2.6.0 -> 3.2.0 now that python3.6 is not supported. #6118

* The FileWatchSensor in the linux pack uses `watchdog` now instead of `logshipper` (`logshipper` and its requirement `pyinotify` are unmaintained).
The `watchdog` project is actively maintained, and has wide support for Linux, MacOS, BSD, Windows, and a polling fallback implementation.
Dropping `pyinotify` has a side benefit of allowing MacOS users to more easily hack on StackStorm's code, since `pyinotify` cannot install on MacOS.

The FileWatchSensor is now an excellent example of how to write a sensor following this refactor. It breaks up the code into well structured classes
that all have a single focus. You can also run the sensor Python script itself, which makes development and testing much easier. #5096

Contributed by @blag

Added
~~~~~
* Continue introducing `pants <https://www.pantsbuild.org/docs>`_ to improve DX (Developer Experience)
Expand Down
42 changes: 26 additions & 16 deletions contrib/linux/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,32 +2,42 @@

This pack contains actions for commonly used Linux commands and tools.

## Configuration
## Sensors

### FileWatchSensor

This sensor monitors files specified in rules (under trigger parameters) for new lines.
Once a new line is detected, a trigger instance is emitted.

#### Adding a file path to the file watch sensor

* ``file_watch_sensor.file_paths`` - A list of paths to the files to monitor.
Note: Those need to be full paths to the files (e.g. ``/var/log/auth.log``)
and not directories (files don't need to exist yet when the sensor is ran
though).
To tell the FileWatchSensor to start watching a new file, define a rule that
- uses the `linux.file_watch.line` trigger type
- pass the `file_path` to watch under trigger parameters.

Example:
For example, this rule would cause the sensor to start watching `/tmp/st2_test`:

```yaml
---
file_watch_sensor:
file_paths:
- /opt/data/absolute_path_to_file.log
```
name: sample_rule_file_watch
description: Run echo on changes to /tmp/st2_test
enabled: false

## Sensors
trigger:
type: linux.file_watch.line
parameters:
file_path: /tmp/st2_test

### FileWatchSensor
action:
ref: core.local
parameters:
cmd: echo "{{trigger}}"
```

This sensor monitors specified files for new new lines. Once a new line is
detected, a trigger is emitted.

### linux.file_watch.line trigger
#### linux.file_watch.line trigger

Example trigger payload:
Example trigger instance payload:

```json
{
Expand Down
4 changes: 1 addition & 3 deletions contrib/linux/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1 @@
# used by file watcher sensor
pyinotify>=0.9.5,<=0.10 ; platform_system=="Linux"
logshipper@ git+https://github.com/StackStorm/logshipper.git@stackstorm_patched ; platform_system=="Linux"
watchdog
2 changes: 1 addition & 1 deletion contrib/linux/sensors/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## NOTICE

File watch sensor has been updated to use trigger with parameters supplied via a rule approach. Tailing a file path supplied via a config file is now deprecated.
File watch sensor has been updated to use trigger with parameters supplied via a rule approach.

An example rule to supply a file path is as follows:

Expand Down
Loading
Loading