-
Notifications
You must be signed in to change notification settings - Fork 2.5k
[ST-NNNN] Targeted Interoperability between Swift Testing and XCTest #2982
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
empowered to choose Swift Testing when writing new tests or test helpers, as | ||
it will work properly in both types of tests. | ||
|
||
We don't propose supporting interoperability for APIs without risk of data loss, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't propose supporting interoperability for APIs without risk of data loss, | |
We don't propose supporting interoperability for APIs which are not lossy without interop, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't not hate the double negative here, but can't think of better wording at the moment :)
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
a8c3a7e
to
b275e55
Compare
For the remainder of this proposal, we’ll describe test APIs which exhibit this | ||
problem as **lossy without interop**. | ||
|
||
This problem risks regressing test coverage for projects which migrate to Swift |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"This problem" is a bit vague in this context since the problem would actually be the regressed test coverage. Consider rephrasing so that it's clear that the lack of interop is what leads to symptoms/problems X, Y, and Z.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. I reworded this whole section for better clarity
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Show resolved
Hide resolved
from XCTest to Swift Testing. For issue details unique to XCTest, we will | ||
include them as a comment when constructing the Swift Testing issue. | ||
|
||
Note that no changes are proposed for the `XCTSkip` API, because they already |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we think of other XCTest features that are explicitly not supported and should be called out here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
XCTestAssertion
and XCTWaiter
, I guess!
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
We propose supporting the following Swift Testing APIs in XCTest: | ||
|
||
- `#expect` and `#require` | ||
- Includes [exit testing][] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Includes [exit testing][] | |
- Includes `#expect(throws:)` | |
- Includes [exit testing][] |
Assuming #expect(throws:)
will work! If it won't work, we need to call that out too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Based on my understanding of that API, we're catching thrown errors and recording issues if not thrown. Since we'd need to add interop support for Swift Testing Issue as part of the default #expect
, it seems to me the check throws variant should work too
usage occurs when running Swift Testing tests by turning those warnings into a | ||
`fatalError`. | ||
|
||
Configure the interoperability mode when running tests using the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This environment variable affects the behaviour of XCTest API, right? Then I think we want to put it in the XCTEST_
namespace. 🤔
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
ff379e8
to
0545e6c
Compare
0545e6c
to
f119329
Compare
Many projects want to migrate from XCTest to Swift Testing, and may be in an | ||
intermediate state where test helpers written using XCTest API are called from | ||
Swift Testing. Today, the Swift Testing and XCTest libraries stand mostly | ||
independently, which means an [`XCTAssert`][XCTest assertions] failure in a |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
independently, which means an [`XCTAssert`][XCTest assertions] failure in a | |
independently, which means an [`XCTAssert()`][XCTest assertions] failure in a |
This and the next suggestion are completely nitpickish.
intermediate state where test helpers written using XCTest API are called from | ||
Swift Testing. Today, the Swift Testing and XCTest libraries stand mostly | ||
independently, which means an [`XCTAssert`][XCTest assertions] failure in a | ||
Swift Testing test or an [`#expect`][Swift Testing expectations] failure in an |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Swift Testing test or an [`#expect`][Swift Testing expectations] failure in an | |
Swift Testing test or an [`#expect()`][Swift Testing expectations] failure in an |
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md
Outdated
Show resolved
Hide resolved
- [Expected failures][], such as `XCTExpectFailure`: marking a Swift Testing | ||
issue in this way will generate a runtime warning issue. | ||
- [`XCTAttachment`][XCTest attachments] | ||
- [Issue handling traits][]: we will make our best effort to translate issues |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this point about traits or about issue handling in general? (Since we're not translating traits.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, we're not proposing that you could attach a Swift Testing trait to an XCTest test function. This is about the issue handling trait in a Swift Testing test, and what happens when it's asked to handle a XCTAssert
failure.
Draft proposal for interop between Swift Testing and XCTest.
Preview it here: https://github.com/jerryjrchen/swift-evolution/blob/swt-xct-interop/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md