Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wip] event-protocol: Add events & schemas for test runs #172

Closed
wants to merge 9 commits into from
Closed
Prev Previous commit
Next Next commit
event-protocol: Add Ruby formatter for publishing events
  • Loading branch information
mattwynne committed May 15, 2017

Unverified

This user has not yet uploaded their public signing key.
commit c987e422556383d16cb9ac59b7a94bf9a0645a4a
13 changes: 13 additions & 0 deletions event-protocol/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
MAKEFILES=validator/Makefile ruby/Makefile

default: $(patsubst %/Makefile,default-%,$(MAKEFILES))
.PHONY: default

default-%: %
cd $< && make default

clean: $(patsubst %/Makefile,clean-%,$(MAKEFILES))
.PHONY: clean

clean-%: %
cd $< && make clean
10 changes: 10 additions & 0 deletions event-protocol/ruby/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
/.bundle/
/.yardoc
/Gemfile.lock
/_yardoc/
/coverage/
/doc/
/pkg/
/spec/reports/
/tmp/
/Gemfile.local
2 changes: 2 additions & 0 deletions event-protocol/ruby/.rspec
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
--format documentation
--color
5 changes: 5 additions & 0 deletions event-protocol/ruby/.travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
sudo: false
language: ruby
rvm:
- 2.1.5
before_install: gem install bundler -v 1.13.1
4 changes: 4 additions & 0 deletions event-protocol/ruby/Gemfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
source 'https://rubygems.org'
gemspec

gem 'cucumber', git: 'https://github.com/cucumber/cucumber-ruby.git', branch: 'event-stream-3'
21 changes: 21 additions & 0 deletions event-protocol/ruby/LICENSE.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
The MIT License (MIT)

Copyright (c) Cucumber Limited and contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
11 changes: 11 additions & 0 deletions event-protocol/ruby/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
default: ../validator/yarn.lock Gemfile.lock test

Gemfile.lock: Gemfile
bundle install

test:
bundle exec rake
.PHONY: test

clean:
rm Gemfile.lock
25 changes: 25 additions & 0 deletions event-protocol/ruby/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Cucumber EventStream Formatter

This is a plugin for Cucumber-Ruby that emits the Cucumber [event protocol](https://github.com/cucumber/cucumber/blob/master/event-protocol).

## Installation

This gem is not designed to be installed stand-alone. It is a component used within the `cucumber` gem.

## Development

After checking out the repo, run `bin/setup` to install dependencies. Then, run `bundle exec rake` to run the tests.

To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).

## Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/cucumber/cucumber

## Code of Conduct

Everyone interacting in this codebase and issue tracker is expected to follow the Cucumber [code of conduct](https://github.com/cucumber/cucumber/blob/master/CODE_OF_CONDUCT.md).

## License

The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
6 changes: 6 additions & 0 deletions event-protocol/ruby/Rakefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
require "bundler/gem_tasks"
require "rspec/core/rake_task"

RSpec::Core::RakeTask.new(:spec)

task :default => :spec
8 changes: 8 additions & 0 deletions event-protocol/ruby/bin/setup
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
set -vx

bundle install

# Do any other automated setup that you need to do here
30 changes: 30 additions & 0 deletions event-protocol/ruby/cucumber-formatter-event_stream.gemspec
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# coding: utf-8
lib = File.expand_path('../lib', __FILE__)
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
require 'cucumber/formatter/event_stream/version'

Gem::Specification.new do |spec|
spec.name = "cucumber-formatter-event_stream"
spec.version = Cucumber::Formatter::EventStream::VERSION
spec.authors = ["Matt Wynne"]
spec.email = ["matt@cucumber.io"]

spec.summary = %q{NDJSON event stream output of Cucumber events}
spec.description = %q{Streams events that describe your test run, as they happen, live, in real-time.}
spec.homepage = "https://cucumber.io"
spec.license = "MIT"

spec.files = `git ls-files -z`.split("\x0").reject do |f|
f.match(%r{^(test|spec|features)/})
end
spec.bindir = "exe"
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
spec.require_paths = ["lib"]

spec.add_development_dependency "bundler", "~> 1.13"
spec.add_development_dependency "rake", "~> 10.0"
spec.add_development_dependency "rspec", "~> 3.0"
spec.add_development_dependency "cucumber", "~> 3.0.0.pre.2"
spec.add_development_dependency "json", "~> 1.8"
spec.add_development_dependency "io-console", "~> 0.4.2"
end
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{"type":"source","uri":"features/failing.feature","data":"Feature: Failing feature\n Scenario: failling\n Given this step fails\n Then this step will be skipped\n","media":{"encoding":"utf-8","type":"text/vnd.cucumber.gherkin+plain"}}
{"type":"test-run-started","workingDirectory":"{{Dir.pwd}}/examples/a-failing-scenario","timestamp":1490450321}
{"type":"test-case-prepared","sourceLocation":{"uri":"features/failing.feature","line":2},"steps":[{"actionLocation":{"uri":"{{Cucumber::LIBDIR}}/cucumber/filters/prepare_world.rb","line":28}},{"actionLocation":{"uri":"step_definitions/steps.rb","line":1},"sourceLocation":{"uri":"features/failing.feature","line":3}},{"actionLocation":{"uri":"step_definitions/steps.rb","line":5},"sourceLocation":{"uri":"features/failing.feature","line":4}}]}
{"type":"test-case-started","sourceLocation":{"uri":"features/failing.feature","line":2}}
{"type":"test-step-started","testCase":{"sourceLocation":{"uri":"features/failing.feature","line":2}},"index":0}
{"type":"test-step-finished","testCase":{"sourceLocation":{"uri":"features/failing.feature","line":2}},"index":0,"result":{"status":"passed","duration":3000}}
{"type":"test-step-started","testCase":{"sourceLocation":{"uri":"features/failing.feature","line":2}},"index":1}
{"type":"test-step-finished","testCase":{"sourceLocation":{"uri":"features/failing.feature","line":2}},"index":1,"result":{"status":"failed","duration":140000,"exception":{"message":"Failing step","type":"RuntimeError","stackTrace":["{{Dir.pwd}}/examples/a-failing-scenario/step_definitions/steps.rb:2:in `/^this step fails$/'","features/failing.feature:3:in `Given this step fails'"]}}}
{"type":"test-step-started","testCase":{"sourceLocation":{"uri":"features/failing.feature","line":2}},"index":2}
{"type":"test-step-finished","testCase":{"sourceLocation":{"uri":"features/failing.feature","line":2}},"index":2,"result":{"status":"skipped","exception":{"message":"","type":"Cucumber::Core::Test::Result::Skipped","stackTrace":[]}}}
{"type":"test-case-finished","sourceLocation":{"uri":"features/failing.feature","line":2},"result":{"status":"failed","duration":12344000,"exception":{"message":"Failing step","type":"RuntimeError","stackTrace":["{{Dir.pwd}}/examples/a-failing-scenario/step_definitions/steps.rb:2:in `/^this step fails$/'","features/failing.feature:3:in `Given this step fails'"]}}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Feature: Failing feature
Scenario: failling
Given this step fails
Then this step will be skipped
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Given(/^this step fails$/) do
raise "Failing step"
end

Then(/^this step will be skipped$/) do
# noop
end
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{"type":"source","uri":"features/passing.feature","data":"Feature: Passing feature\n Scenario: Passing scenario\n Given this step passes\n","media":{"encoding":"utf-8","type":"text/vnd.cucumber.gherkin+plain"}}
{"type":"test-run-started","workingDirectory":"{{Dir.pwd}}/examples/a-single-passing-scenario","timestamp":1490444711}
{"type":"test-case-prepared","sourceLocation":{"uri":"features/passing.feature","line":2},"steps":[{"actionLocation":{"uri":"{{Cucumber::LIBDIR}}/cucumber/filters/prepare_world.rb","line":28}},{"actionLocation":{"uri":"step_definitions/steps.rb","line":1},"sourceLocation":{"uri":"features/passing.feature","line":3}}]}
{"type":"test-case-started","sourceLocation":{"uri":"features/passing.feature","line":2}}
{"type":"test-step-started","testCase":{"sourceLocation":{"uri":"features/passing.feature","line":2}},"index":0}
{"type":"test-step-finished","testCase":{"sourceLocation":{"uri":"features/passing.feature","line":2}},"index":0,"result":{"status":"passed","duration":3000}}
{"type":"test-step-started","testCase":{"sourceLocation":{"uri":"features/passing.feature","line":2}},"index":1}
{"type":"test-step-finished","testCase":{"sourceLocation":{"uri":"features/passing.feature","line":2}},"index":1,"result":{"status":"passed","duration":381000}}
{"type":"test-case-finished","sourceLocation":{"uri":"features/passing.feature","line":2},"result":{"status":"passed","duration":24008000}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Feature: Passing feature
Scenario: Passing scenario
Given this step passes
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Given(/^this step passes$/) do
# noop
end
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{"type":"source","uri":"features/undefined.feature","data":"Feature: Undefined feature\n Scenario: Undefined scenario\n Given this step is undefined\n","media":{"encoding":"utf-8","type":"text/vnd.cucumber.gherkin+plain"}}
{"type":"test-run-started","workingDirectory":"{{Dir.pwd}}/examples/an-undefined-step","timestamp":1490444711}
{"type":"test-case-prepared","sourceLocation":{"uri":"features/undefined.feature","line":2},"steps":[{"actionLocation":{"uri":"{{Cucumber::LIBDIR}}/cucumber/filters/prepare_world.rb","line":28}},{"sourceLocation":{"uri":"features/undefined.feature","line":3}}]}
{"type":"test-case-started","sourceLocation":{"uri":"features/undefined.feature","line":2}}
{"type":"test-step-started","testCase":{"sourceLocation":{"uri":"features/undefined.feature","line":2}},"index":0}
{"type":"test-step-finished","testCase":{"sourceLocation":{"uri":"features/undefined.feature","line":2}},"index":0,"result":{"status":"passed","duration":3000}}
{"type":"test-step-started","testCase":{"sourceLocation":{"uri":"features/undefined.feature","line":2}},"index":1}
{"type":"test-step-finished","testCase":{"sourceLocation":{"uri":"features/undefined.feature","line":2}},"index":1,"result":{"status":"undefined","exception":{"message":"Undefined step: \"this step is undefined\"","type":"Cucumber::Core::Test::Result::Undefined","stackTrace":["features/undefined.feature:3:in `Given this step is undefined'"]}}}
{"type":"test-case-finished","sourceLocation":{"uri":"features/undefined.feature","line":2},"result":{"status":"undefined","duration":12811000,"exception":{"message":"Undefined step: \"this step is undefined\"","type":"Cucumber::Core::Test::Result::Undefined","stackTrace":["features/undefined.feature:3:in `Given this step is undefined'"]}}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Feature: Undefined feature
Scenario: Undefined scenario
Given this step is undefined
14 changes: 14 additions & 0 deletions event-protocol/ruby/lib/cucumber/formatter/event_stream.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
require "cucumber/formatter/event_stream/version"
require "cucumber/formatter/event_stream/plugin"

module Cucumber
module Formatter
module EventStream

def self.new(*args)
Plugin.new(*args)
end

end
end
end
147 changes: 147 additions & 0 deletions event-protocol/ruby/lib/cucumber/formatter/event_stream/plugin.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
require 'socket'

module Cucumber
module Formatter
module EventStream

class Plugin
def initialize(config, options)
io = create_io(config, options)

EventEmitter.new(config).call do |event|
io.puts event.to_json
end
end

def create_io(config, options)
return config.out_stream if !options.key?('port')

io = TCPSocket.new(options['host'] || 'localhost', options['port'].to_i)
config.on_event :test_run_finished, -> (event) { io.close }
io
end
end

class EventEmitter
attr_reader :config

def initialize(config)
@config = config
@working_dir = Pathname.new(Dir.pwd)
end

def call
current_test_case = nil

config.on_event :gherkin_source_read, -> (event) {
yield \
type: "source",
uri: uri(event.path),
data: event.body,
media: {
encoding: 'utf-8',
type: 'text/vnd.cucumber.gherkin+plain'
}
}

config.on_event :test_run_starting, -> (event) {
yield \
type: "test-run-started",
workingDirectory: working_dir,
timestamp: Time.now.to_i

event.test_cases.each { |test_case|
yield \
type: "test-case-prepared",
sourceLocation: location_to_json(test_case.location),
steps: test_case.test_steps.map { |test_step|
test_step_to_json(test_case, test_step)
}
}
}

config.on_event :test_case_starting, -> (event) {
current_test_case = event.test_case # TODO: add this to the core step events so we don't have to cache it here
yield \
type: "test-case-started",
sourceLocation: location_to_json(event.test_case.location)
}

config.on_event :test_step_starting, -> (event) {
yield \
type: "test-step-started",
testCase: { sourceLocation: location_to_json(current_test_case.location) },
index: current_test_case.test_steps.index(event.test_step)
}

config.on_event :test_step_finished, -> (event) {
yield \
type: "test-step-finished",
testCase: { sourceLocation: location_to_json(current_test_case.location) },
index: current_test_case.test_steps.index(event.test_step),
result: result_to_json(event.result)
}

config.on_event :test_case_finished, -> (event) {
yield \
type: "test-case-finished",
sourceLocation: location_to_json(event.test_case.location),
result: result_to_json(event.result)
}

end

private

attr_reader :working_dir

def result_to_json(result)
data = { status: result.to_sym.to_s }
result.duration.tap do |duration|
data["duration"] = duration.nanoseconds
end
if result.respond_to?(:exception)
data[:exception] = {
message: result.exception.message,
type: result.exception.class,
stackTrace: result.exception.backtrace || []
}
end
data
end

def test_step_to_json(test_case, test_step)
result = {}
unless undefined?(test_step)
result['actionLocation'] = location_to_json(test_step.action_location)
end
unless hook?(test_step)
result['sourceLocation'] = location_to_json(test_step.source.last.location)
end
result
end

def undefined?(test_step)
test_step.action_location.file.match(/\.feature$/)
end

def hook?(test_step)
test_step.action_location.file.match(/\.rb$/) &&
test_step.source.last.location == test_step.action_location
end

def location_to_json(location)
{
uri: uri(location.file),
line: location.line
}
end

def uri(path)
Pathname.new(File.expand_path(path)).relative_path_from(working_dir)
end
end
end
end

end
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
module Cucumber
module Formatter
module EventStream
VERSION = "0.1.0"
end
end
end
Loading