integration_tests::daphne is flaky (maybe)

[tgeoghegan]

I think there is still some flakiness in integration tests. Here's two failures I saw in recent ci-build runs:

https://github.com/divviup/janus/runs/8122772496?check_suite_focus=true
https://github.com/divviup/janus/runs/8142726381?check_suite_focus=true

Problem is, the failure just looks like:

test daphne_janus ... ok
test janus_daphne has been running for over 60 seconds
test janus_daphne ... FAILED

failures:

---- janus_daphne stdout ----
thread 'janus_daphne' panicked at 'assertion failed: Instant::now() < collect_job_poll_timeout', integration_tests/tests/common/mod.rs:209:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    janus_daphne

So not a ton of information there. I think we will want to augment the tests so that we scrape logs out of all the interop containers, and then attach them to the test run as an artifact, like we do over in janus-ops (https://github.com/divviup/janus-ops/blob/723f8a3be226e3e1bad4f969a1378bb3da9c17ba/.github/workflows/ci-build.yml#L147). Once we learn more about what is going wrong, we can dig in and debug.

Besides grabbing container logs, we should have the test harness log some useful information like the ports it's connecting to, which might give us a leg up detecting wacky things like IPv4/IPv6 port collisions on localhost.

[branlwyd]

Can this be closed? I think it was fixed with cloudflare/daphne@80b53c4 (& related work on Janus to use that commit in integration tests).

[tgeoghegan]

Yeah, we can write new issues if we continue to see test failures.