Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for use of prebuilt test Daphne container image. #435

Merged
merged 4 commits into from
Aug 30, 2022

Conversation

branlwyd
Copy link
Contributor

This is controlled by a new compile-time DAPHNE_INTEROP_CONTAINER
environment variable (matching the JANUS_INTEROP_CONTAINER variable used
to control building of the Janus interop test containers). See the
updates to README.md for more details.

Also:

@branlwyd branlwyd requested a review from a team as a code owner August 26, 2022 02:43
@branlwyd
Copy link
Contributor Author

Notes:

  • As part of this I pushed a daphne_test image to our private infra, and tested that it works as a prebuilt target. I'd like to set up CI to use it too, but before I do: the name of the image includes the name of our private Docker repo; should this be stored in a secret? (I think the answer is yes, but I want to poll folks before I add the secret.)

This is controlled by a new compile-time DAPHNE_INTEROP_CONTAINER
environment variable (matching the JANUS_INTEROP_CONTAINER variable used
to control building of the Janus interop test containers). See the
updates to README.md for more details.

Also:
* Update Daphne to the latest commit. This is important as it picks up
  cloudflare/daphne#99 which de-flakes
  Janus/Daphne integration tests again after they were accidentally
  re-flaked by switching to DAP_DEPLOYMENT=prod.
* Add ability to skip building the Daphne container, and do so for our
  CI lints.
@branlwyd branlwyd force-pushed the bran/prebuilt-daphne branch from 5ed0c60 to 7c96da6 Compare August 26, 2022 02:50
@divergentdave
Copy link
Collaborator

Putting the image name in a GHA secret makes sense to me, as attempting to use it outside a workflow under our control would result in an authentication error.

@branlwyd branlwyd force-pushed the bran/prebuilt-daphne branch 4 times, most recently from 23cb6aa to 47df577 Compare August 26, 2022 23:54
@branlwyd branlwyd force-pushed the bran/prebuilt-daphne branch from 47df577 to 620b53e Compare August 29, 2022 16:33
@branlwyd branlwyd force-pushed the bran/prebuilt-daphne branch from 620b53e to a13af74 Compare August 29, 2022 16:41
Copy link
Contributor

@tgeoghegan tgeoghegan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm uncomfortable with us being Daphne's build system, and with the amount of surprising side effects that running cargo test or cargo build is sprouting, like putting images into your local Docker image repository. I don't want to block forward progress but I hope we have a plan to stop building Daphne in this project, and I also think we should consider moving what are plainly expensive and complicated integration tests out of the cargo test path, which I think people generally expect to be pretty cheap and safe to run.

Comment on lines +50 to +52
"build",
"--file=daphne_worker_test/docker/miniflare.Dockerfile",
"--quiet",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is going to break if the layout of the Daphne repository changes, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we will have to update these instructions to account for backwards-incompatible changes in Daphne's build system when we bump Daphne versions, similarly to the human-targeted instructions. (note that we fix a version of Daphne under test, so changes to the Daphne repo won't trigger the need to make these changes automatically)

@branlwyd
Copy link
Contributor Author

I'm uncomfortable with us being Daphne's build system

We aren't Daphne's build system? We just build Daphne.

and with the amount of surprising side effects that running cargo test or cargo build is sprouting, like putting images into your local Docker image repository.

cargo {build,test} also puts binary artifacts into your /target/ directory -- think of it like that.

I don't want to block forward progress but I hope we have a plan to stop building Daphne in this project,

This PR is the plan: it allows you to use a prebuilt Daphne container rather than building the image yourself.

and I also think we should consider moving what are plainly expensive and complicated integration tests out of the cargo test path, which I think people generally expect to be pretty cheap and safe to run.

Disagree: cargo test should run all tests, without the need for folks to look up complicated one-off invocations to run the "special" tests that we decided are too complicated.

We could hide these behind a feature, if we needed to. But TBH, the tests run quite quickly, and with prebuilt support the build-time hit is completely gone. (Even before we had support for prebuilt containers, the build was effectively a one-off since it would be the same every time, unless you cleared your Docker build cache.)

Copy link
Contributor

@tgeoghegan tgeoghegan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough -- I don't have the depth of understanding on this to critique it substantially or more importantly to suggest other strategies, so I think we should move forward, see what if anything is painful and iterate on those points.

@branlwyd branlwyd force-pushed the bran/prebuilt-daphne branch 6 times, most recently from cdc08d0 to 4e7145a Compare August 30, 2022 01:15
@branlwyd branlwyd force-pushed the bran/prebuilt-daphne branch from 4e7145a to b9b87be Compare August 30, 2022 01:31
@branlwyd branlwyd merged commit d971ce0 into main Aug 30, 2022
@branlwyd branlwyd deleted the bran/prebuilt-daphne branch August 30, 2022 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants