[meta] defaults plugins migration to new Event interface and plugin-api 2.0 #5227

colinsurprenant · 2016-04-29T21:11:45Z

This is a meta issue to track progress for refactoring the defaults plugins. Core plugins migration has been completed per #5170. Related #5140.

This migration recipe goals are:

avoid publishing any core plugins for the refactor
be able to test refactored plugins within their refactor branch in Travis
be able to test/build logstash using the refactored plugins within their branches

Plugin Migration Recipe

create branch feature/plugin-api-2_0
this branch should be published upstream in the logstash-plugins repo.

.travis.yml should be updated to

sudo: false
jdk:
 - oraclejdk8
language: ruby
cache: bundler
rvm:
 - jruby-1.7.25
before_install:
 - git clone -b feature/event_interface https://github.com/elastic/logstash
script:
 - bundle exec rspec spec

minimally these gems need to be added to the plugin Gemfile

# this is temporary for the feature/plugin-api-2_0 branch and is meant for travis testing
gem "logstash-core", :path => "./logstash/logstash-core"
gem "logstash-core-plugin-api", :path => "./logstash/logstash-core-plugin-api"
gem "logstash-core-event-java", :path => "./logstash/logstash-core-event-java"
gem "logstash-devutils", :github => "elastic/logstash-devutils", :branch => "feature/plugin-api-2_0"

add any other plugins in the gemspec required by this plugin, for example, if logstash-codec-plain is a required plugin add this:
```
gem "logstash-codec-plain", :github => "logstash-plugins/logstash-codec-plain", :branch => "feature/plugin-api-2_0"
```
modify the gemspec to bump the plugin-api version constrain to 2.0:
```
s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
```
and finally, refactor code and specs for new Event interface

Running local specs

if you want to test locally you will have to create a symbolic link to the logstash code base with the feature/event_interface shared branch
```
ln -s /you/path/to/logstash .
bundle
bundle exec rspec spec
```

Helpful Scripts

@ph has added this ruby script to automate much of the initial work: https://gist.github.com/e28dddccfbe32c3985ab51d6f9b9ec41

Plugins for core specs

The text was updated successfully, but these errors were encountered:

ph · 2016-04-29T21:22:21Z

We can automate all of the heavy work minus the actual code/spec?
Are we fine doing that?

suyograo · 2016-04-29T21:31:30Z

@ph +1 for automating..

colinsurprenant · 2016-04-29T21:39:46Z

it might be tricky to automate the addition of dependent plugins from the gemspec but the basic block

# this is temporary for the feature/plugin-api-2_0 branch and is meant for travis testing
gem "logstash-core", :path => "./logstash/logstash-core"
gem "logstash-core-plugin-api", :path => "./logstash/logstash-core-plugin-api"
gem "logstash-core-event-java", :path => "./logstash/logstash-core-event-java"
gem "logstash-devutils", :github => "elastic/logstash-devutils", :branch => "feature/plugin-api-2_0"

can be easily automated

colinsurprenant · 2016-04-29T21:40:40Z

also I've seen a bit different .travis.yml, with the integration tests flag?

suyograo · 2016-04-29T21:44:15Z

Btw, all plugins should have the integration flag set if its not already there..

colinsurprenant · 2016-04-29T22:10:27Z

I added the list of plugins done so far for passing core specs.

ph · 2016-05-04T02:25:17Z

@suyograo logstash-codec-oldlogstashjson is marked as completed but I can't find the PR.

suyograo · 2016-05-04T20:44:43Z

colinsurprenant · 2016-05-04T20:53:39Z

@suyograo

the plugins branches off master will be 2.x ?
we need to revert both .travis.yml and Gemfile in mass update
you mean new 5.0.0-beta1.snapshot2 ?

suyograo · 2016-05-04T20:55:44Z

@colinsurprenant updated.

colinsurprenant · 2016-05-04T21:05:11Z

@suyograo @ph also, .travis.yml should not be updated to old state, we certainly want to keep the Java 8 selection?

colinsurprenant · 2016-05-04T21:45:57Z

@suyograo @ph core-api-v1 (not v2) branch for LS 2.x development.

ph · 2016-05-04T21:50:10Z

@colinsurprenant We already have v1, it was before the shutdown change, and used to target LS 1.5?

colinsurprenant · 2016-05-04T21:54:51Z

@ph how is that possible? we only recently introduced the plugin-core-api gem recently
ah yes, ok, I think I remember - it's core-api, we need to create plugin-api-v1

suyograo · 2016-05-04T22:17:43Z

@colinsurprenant core-api-v1 branch already exists and was created to continue development of plugins under v1 core API (pre shutdown changes, < LS 2.0). core-api-v2 once branched will match v2 of plugin APIs and continue development for them (pre Java Event, < LS 2.0).

Also, here is a .travis.yml we can mass update with: https://gist.github.com/suyograo/331f6384e143b147c09db51a854540c4

Includes Java 8 and --integration tags

ph · 2016-05-05T13:37:50Z

Also, here is a .travis.yml we can mass update with: https://gist.github.com/suyograo/331f6384e143b147c09db51a854540c4

I am not sure we should mass update with the bundle exec rspec --tag integration, since integration usually mean that we need to add some steps in the travis file to make sure the environment is ready to be used for integration testing. Like installing redis, rabbitmq or supporting libraries.

But this could be a good way to find which plugins are failling and change their travis.yml manually.

@colinsurprenant @suyograo core-api-v2 I agree is a weird name, another option would be to make it explicit and rename them logstash-1.5.x and logstash-2.x ;)

ph · 2016-05-05T15:07:55Z

@suyograo After discussing on slack ~~zoom~~ I am ok to move ahead with the core-api-v2, its only a branch name, if we change our mind in the future we can always rename it/them.

@colinsurprenant Can we move forward?

colinsurprenant · 2016-05-05T18:23:25Z

@ph @suyograo it is still confused.

core-api-v1 branch was introduced for supporting the pre-shutdown changes for the 1.5 series. At this point we had no notion of plugin-api, we came up with that ad-hoc branch name at the time.
we need a branch to be able to continue to support le LS versions that are using the plugin-api v1 which are all pre-5.0 releases. if we use the name "v2" in that branch name it will be extremely confusing.
since we now have a concept of plugin-api formal versioning I think we need to use that concept to name branches and use corresponding names and versions numbers.
in that respect, the problem here is the original "core-api-v1" branch name which does not make sense with the new plugin-api concept.

For this I suggest we simply leave the core-api-v1 branch name as-is, for legacy reasons, which anyway will not be needed anymore, and introduce branches with the plugin-api name and versions.

suyograo · 2016-05-05T20:27:27Z

For this I suggest we simply leave the core-api-v1 branch name as-is, for legacy reasons, which anyway will not be needed anymore, and introduce branches with the plugin-api name and versions.

@colinsurprenant @ph I am fine with naming the branch based on plugin API version. Good with plugin-api-v1

colinsurprenant · 2016-05-06T14:56:43Z

@suyograo, @ph and myself just reviewed the tasks sequence and Houston, all engines go!

ph added the v5.0.0-beta1 label Apr 29, 2016

colinsurprenant closed this as completed Apr 29, 2016

colinsurprenant reopened this Apr 29, 2016

talevy added java event labels Apr 29, 2016

ph mentioned this issue May 3, 2016

Update plugin to the v2.0 of the api logstash-plugins/logstash-mixin-http_client#21

Merged

colinsurprenant mentioned this issue May 3, 2016

[meta] Event API refactor #5140

Closed

4 tasks

ph mentioned this issue May 4, 2016

Update with new Event APIs logstash-plugins/logstash-codec-netflow#36

Merged

This was referenced May 4, 2016

update to new event api logstash-plugins/logstash-output-sqs#14

Merged

update to new event api logstash-plugins/logstash-output-cloudwatch#10

Merged

suyograo closed this as completed May 13, 2016

suyograo mentioned this issue May 17, 2016

Update remaining plugins with Event API #5309

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[meta] defaults plugins migration to new Event interface and plugin-api 2.0 #5227

[meta] defaults plugins migration to new Event interface and plugin-api 2.0 #5227

colinsurprenant commented Apr 29, 2016 •

edited by ph

Loading

ph commented Apr 29, 2016

suyograo commented Apr 29, 2016

colinsurprenant commented Apr 29, 2016

colinsurprenant commented Apr 29, 2016

suyograo commented Apr 29, 2016

colinsurprenant commented Apr 29, 2016

ph commented May 4, 2016

suyograo commented May 4, 2016 •

edited by ph

Loading

colinsurprenant commented May 4, 2016

suyograo commented May 4, 2016

colinsurprenant commented May 4, 2016

colinsurprenant commented May 4, 2016

ph commented May 4, 2016 •

edited

Loading

colinsurprenant commented May 4, 2016

suyograo commented May 4, 2016

ph commented May 5, 2016

ph commented May 5, 2016 •

edited

Loading

colinsurprenant commented May 5, 2016 •

edited

Loading

suyograo commented May 5, 2016

colinsurprenant commented May 6, 2016

[meta] defaults plugins migration to new Event interface and plugin-api 2.0 #5227

[meta] defaults plugins migration to new Event interface and plugin-api 2.0 #5227

Comments

colinsurprenant commented Apr 29, 2016 • edited by ph Loading

Plugin Migration Recipe

Running local specs

Helpful Scripts

Plugins for core specs

ph commented Apr 29, 2016

suyograo commented Apr 29, 2016

colinsurprenant commented Apr 29, 2016

colinsurprenant commented Apr 29, 2016

suyograo commented Apr 29, 2016

colinsurprenant commented Apr 29, 2016

ph commented May 4, 2016

suyograo commented May 4, 2016 • edited by ph Loading

colinsurprenant commented May 4, 2016

suyograo commented May 4, 2016

colinsurprenant commented May 4, 2016

colinsurprenant commented May 4, 2016

ph commented May 4, 2016 • edited Loading

colinsurprenant commented May 4, 2016

suyograo commented May 4, 2016

ph commented May 5, 2016

ph commented May 5, 2016 • edited Loading

colinsurprenant commented May 5, 2016 • edited Loading

suyograo commented May 5, 2016

colinsurprenant commented May 6, 2016

colinsurprenant commented Apr 29, 2016 •

edited by ph

Loading

suyograo commented May 4, 2016 •

edited by ph

Loading

ph commented May 4, 2016 •

edited

Loading

ph commented May 5, 2016 •

edited

Loading

colinsurprenant commented May 5, 2016 •

edited

Loading