Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for TeamCity Test history via Teamcity Service Messages #44813

Closed

Conversation

gilescope
Copy link
Contributor

This PR enables Teamcity Service Messages ( https://confluence.jetbrains.com/display/TCD10/Build+Script+Interaction+with+TeamCity ) to be output when running tests or benchmarks. This enables easy integration of Rust projects into TeamCity and hopefully enables easier test runner integration into Intelli Rust ( #43381 ). This will help Rust be easily integrated into existing enterprise environments.

One neat touch is that the rust benchmarks are turned into TeamCity Stats and can then be easily graphed over time through the TeamCity UI.

To turn on having a '--teamcity' option like NUnit3 or by having an environment variable set (RUST_TEST_TEAMCITY_SERVICE_MESSAGES=1) should turn these on. They should not be on by default.
(I believe no corresponding changes to the Cargo project need be made.)

All feedback welcome!

@rust-highfive
Copy link
Collaborator

r? @aturon

(rust_highfive has picked a reviewer for you, use r? to override)

@carols10cents carols10cents added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 25, 2017
@aturon
Copy link
Member

aturon commented Sep 25, 2017

r? @alexcrichton

@rust-highfive rust-highfive assigned alexcrichton and unassigned aturon Sep 25, 2017
@alexcrichton
Copy link
Member

Thanks for the PR! Integration with existing test frameworks has long been a desire for us and historically hasn't seen much action unfortunately.

At this time though I think we're not quite ready to accept a PR like this. We'd ideally not be adding one-off support for all sorts of test suites and frameworks to libtest, but rather we'd have a more fleshed out design of where we want to go to start out with. We've thought about this in the past and it tends to boil down to "bring your own libtest" rather than adding support to libtest directly itself.

I realize though this probably isn't the response you wanted :(. There's some more info at rust-lang/rfcs#816 as well I believe. I'm curious about what you think about this?

@gilescope
Copy link
Contributor Author

Totally understand - hence wanted an early review.

Ideally we want to split test runners from test listeners. We'd like to be agnostic of the test-runner for team city service messages. The testing framework should have a logging/reporting interface and the teamcity service messages could be one such listener (the console and crate cargo-test-junit being other implementations of test listeners).

Is it currently possible to switch out the test runner for another? It feels a little like bench is a separate test runner.

@gilescope
Copy link
Contributor Author

(Btw, on a separate note, just want to say how much I love that when you compile tests the compiler complains about unused code that isn't referenced from the tests. It's effectively a compile time coverage tool - super cool.)

@alexcrichton
Copy link
Member

I indeed share many of the same desires! Unfortunately right now though it's not possible to switch out the test runner, that's mostly what rust-lang/rfcs#816 is targeted at.

@carols10cents
Copy link
Member

@alexcrichton so should this PR be closed, pending resolution to rust-lang/rfcs#816?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-review Status: Awaiting review from the assignee but also interested parties.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants