ci(sync): increase the height of blocks for some full sync jobs

[arya2]

Motivation

This is the standard fix for when a full sync job runs too long (logs-checkpoint needs to sync ~45k blocks as of the latest checkpoint update).

Solution

• Replace full sync job for 1790k blocks with one for 1800k blocks
• Add full sync job for 1820k blocks

See fixing-ci-sync-timeouts

Review

Anyone can review.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
• How do you know it works? Does it have tests?

[arya2]

The checkpoint step took 4-5 hours to get to 1810k blocks (timed out after ~1817k). Adding just one extra step this time.

[arya2]

ssh failure in logs-checkpoint:

Timeout, server 34.123.232.71 not responding.

..

ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

https://github.com/ZcashFoundation/zebra/actions/runs/3236318547/jobs/5302131447#step:6:777

[teor2345]

ssh failure in logs-checkpoint: https://github.com/ZcashFoundation/zebra/actions/runs/3236318547/jobs/5302131447#step:6:777

There are lots of different ways ssh can fail, can you please quote the actual failure log?

Timeout, server 34.123.232.71 not responding.

[teor2345]

I also saw these sync warnings in the logs:

2022-10-11T03:59:58.369043Z WARN {net="Main"}: zebrad::components::sync::progress: chain updates have stalled, state height has not increased for 30 minutes. Hint: check your network connection, and your computer clock and time zone sync_percent=31.016% current_height=Height(572486) network_upgrade=Sapling time_since_last_state_block=30m target_block_spacing=PT75S max_block_spacing=None is_syncer_stopped=false

https://github.com/ZcashFoundation/zebra/actions/runs/3223410431/jobs/5295854595#step:6:2844

Every time they happen, the sync takes half an hour longer waiting for a single block. So maybe we could drop the timeout back down to 10 minutes first?

(We might still need this PR.)

[teor2345]

Every time they happen, the sync takes half an hour longer waiting for a single block. So maybe we could drop the timeout back down to 10 minutes first?

Let's see how #5397 goes, then try this one?

There is also a timeout in the Rust test we might need to fix.

[arya2]

I also saw these sync warnings in the logs:

2022-10-11T03:59:58.369043Z WARN {net="Main"}: zebrad::components::sync::progress: chain updates have stalled, state height has not increased for 30 minutes. Hint: check your network connection, and your computer clock and time zone sync_percent=31.016% current_height=Height(572486) network_upgrade=Sapling time_since_last_state_block=30m target_block_spacing=PT75S max_block_spacing=None is_syncer_stopped=false

https://github.com/ZcashFoundation/zebra/actions/runs/3223410431/jobs/5295854595#step:6:2844

Every time they happen, the sync takes half an hour longer waiting for a single block. So maybe we could drop the timeout back down to 10 minutes first?

We should definitely fix that first.

(We might still need this PR.)

It's doubtful it'll make it from 1790k to ~1834k, though we might be able to replace logs-1790k (~2h 40m) with logs-1800k (seems to be another ~2h 20m) instead of adding a new step.

[gustavovalverde]

@arya2 do we still need this PR?

[gustavovalverde]

Seems like we are still having timeouts in the main branch, but I'll left @teor2345 the last call here even though is a standard change, as I'm not sure if there are other works in progress to fix it another way

[arya2]

do we still need this PR?

I believe we do, but I'll just quickly check the logs for the timeouts and perhaps replace logs-1790k with logs-1800k.

[teor2345]

Seems like we are still having timeouts in the main branch, but I'll left @teor2345 the last call here even though is a standard change, as I'm not sure if there are other works in progress to fix it another way

We're done with the fixes due to PR #5257, it should all be standard fixes from now on.

[gustavovalverde]

LGTM

[teor2345]

I think the regexes might be wrong here?

[mpguerra]

is there a github issue that this is related to that I can link in?

[arya2]

is there a github issue that this is related to that I can link in?

No, this was ad-hoc and trivial.

[teor2345]

Thanks!

[teor2345]

@Mergifyio update

[mergify]

update

✅ Branch has been successfully updated