Stop worker when ctx is canceled during recovered round execution

[poszu]

Closes #400

Root cause

When the poet was shut down during the recovered round execution, it would not immediately quit but proceed to select from the closed rounds channel and ctx.Done(). It would have a 50% chance to pick up the same round ID again and start it from the start because the registration also recovers a round in-progress and sends it over the closed channel. In a normal use case (when the recovered round finishes) the round recovered from registration is skipped.

The fix

The fix is to quit the Service::loop() immediately if the recovered round either fails or is canceled.

[codecov]

Codecov Report

Merging #401 (5c143be) into develop (91c73ef) will decrease coverage by 0.2%.
Report is 3 commits behind head on develop.
The diff coverage is 58.3%.

@@            Coverage Diff            @@
##           develop    #401     +/-   ##
=========================================
- Coverage     74.8%   74.6%   -0.2%     
=========================================
  Files           23      23             
  Lines         1898    1904      +6     
=========================================
+ Hits          1421    1422      +1     
- Misses         359     363      +4     
- Partials       118     119      +1     

Files Changed	Coverage Δ
service/service.go	79.8% <58.3%> (-0.5%)	⬇️

... and 1 file with indirect coverage changes

[fasmat]

I don't understand what the issue is and how this PR fixes it. Could you elaborate?

[poszu]

I don't understand what the issue is and how this PR fixes it. Could you elaborate?

Sorry for this. Updated the PR description.

[fasmat]

Would it make sense to make this a method? Especially since further below the same code is called again (without the cleanupRound shell?

[poszu]

I'm not sure, the method would need to return:

func() error {
				if err := round.Teardown(ctx, err == nil); err != nil {
					logger.Warn("round teardown failed", zap.Error(err), zap.Uint("epoch", round.epoch))
				}
				return nil
			}
}

to be scheduled on the local eg.