roachtest: c2c/kv0 failed

[cockroach-teamcity]

roachtest.c2c/kv0 failed with artifacts on release-23.1 @ 023468c57cfabd42d811906cabaf0ccfa4788151:

(cluster.go:2042).Start: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/c2c/kv0/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=aws
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

• roachtest: c2c/kv0 failed [flakes in require.NoError not detected] #125674 roachtest: c2c/kv0 failed [flakes in require.NoError not detected] [C-test-failure E-quick-win O-roachtest P-2 T-testeng branch-release-24.1]

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-41859

[msbutler]

looks like roachprod start failed, according to failure_1.log, on node 2

Node 2's logs suggest it fataled due to disk stall. Passing this over to test-eng, which may know if we can close this as a one time flake.

I240904 05:53:28.295500 72 1@cli/start.go:1528 ⋮ [T1,n?] 19 +Node will now attempt to join a running cluster, or wait for `cockroach init`.
I240904 05:53:28.295500 72 1@cli/start.go:1528 ⋮ [T1,n?] 19 +Client connections will be accepted after this completes successfully.
I240904 05:53:28.295500 72 1@cli/start.go:1528 ⋮ [T1,n?] 19 +Check the log file(s) for progress.
W240904 05:53:38.974805 171 1@util/log/file.go:274 ⋮ [-] 20  disk slowness detected: unable to sync log files within 10s

[blathers-crl]

cc @cockroachdb/test-eng

[renatolabs]

n2 definitely had a disk stall, it crashed shortly after that log line was printed. Given that there was nothing happening in this test (workload wasn't running yet), it's unlikely this is an issue related to CRDB, so I'm closing.