Session crashes (without errors in log) right after the O sends a tx

[0xVires]

Seeing some strange behavior with the latest livepeer version:
All my sessions crash without anything showing up in the logs. All the transcoding just seems to stop:

Dec 26 23:53:24 livepeer[129136]: I1226 23:53:24.710594  129136 orchestrator.go:766] manifestID=19782171-0479-4561-be44-2c3417c1c94d seqNo=370 orchSessionID=304da1aa Successfully received results from remote transcoder=x:59400 segments=2 taskId=737779 fname=https://x:8935/stream/304da1aa/370.tempfile dur=463.42726ms
Dec 26 23:53:26 livepeer[129136]: I1226 23:53:26.322271  129136 orchestrator.go:766] manifestID=09dcsceke8mmnotk seqNo=135 orchSessionID=66d7a6b4 Successfully received results from remote transcoder=x:44522 segments=4 taskId=737781 fname=https://x:8935/stream/66d7a6b4/135.tempfile dur=109.798038ms
Dec 27 00:29:26 livepeer[129136]: 2021/12/27 00:29:26 http: TLS handshake error from 143.244.33.78:59807: EOF

Only in the T logs I see a few Transcode loop timed out for key=ebf7dc72_0 and Deleted transcode session for key=ebf7dc72_0, but also no errors. Some more info here: https://discord.com/channels/423160867534929930/426114749370204170/924479708605861920

The weird thing is that both of those crashes happened just a minute or two after the O sent (or was supposed to send) a tx. The first time right after the start of a new round, the second time right after the O claimed a winning ticket.

Crash 1 (new round started 4min before 19:00):

Crash 2 (winning ticket tx was mined at 23:50):

Any guesses why my sessions crash without anything showing up in the logs?

[Strykar]

I see similar behavior approximately 6 minutes after a winning ticket was claimed.
Dec 26 16:32:21 i5 livepeer[3604]: Invoking transaction: "redeemWinningTicket". Inputs: "_ticket: { Recipient: 0x1a...

[ChuckChain]

Just to add that we're seeing the same behavior. O/T "crashed" just after redeeming a winning ticket, but no errors in the log. Restarted the O/T this morning (about 2 hours ago), and it's having trouble keeping sessions since.

[0xVires]

Happened for the third time - again right after redeeming a winning ticket

@ChuckChain the sessions will eventually come back, but it might take a few hours to recover

[ChuckChain]

@0xVires You're right, it took a couple of hours, but it went back to normal.

[yondonfu]

Just a heads up that this issue will be investigated this week, but progress may be a bit slow given that we're in a holiday period.

[Franck-UltimaRatio]

I can confirm that, our 0 crashed just after sending a TX for redeem our last ticket (T / O split, ubuntu)

[0xVires]

adding to all the cases where it happens after redeeming a winning ticket: It also seems to happen in some cases when a new round starts and the O is supposed to send the reward claim tx.

I've already described this in my first post and it just happened again in round 2419. All my sessions were gone approx. a minute after the new round started (so when the O was supposed to send the reward claim tx - but it never did):

I don't know under which circumstances it happens - I've also rounds where the O successfully sent the reward tx without session crashes. And rounds where I had to manually send it due to my maxGasPrice setting, but also without crashing the sessions.

But this is the second time now so I don't think it's a coincidence...

[criticaltv]

I experienced a complete loss of traffic today, which coincided with the start of the new round.

I also recently had problems restarting my O, only to find that the nvidia drivers had somehow uninstalled themselves. It also proceeded the first time my node did not automatically call reward in a long while. I reinstalled latest drivers, then re-keylase and everything is fine again now (aside from the above complete loss of traffic).

[leszko]

I started working on the issue. The good news is that I managed to reproduce it locally with the following steps:

• Set up local Livepeer Geth
• Start Orchestrator
• Update max gas price to 1000000010
• After a few minutes there is deadlock in TimeWatcher

The deadlock prevents the lastSeenBlock to get refreshed and as a result, there is an error TicketParams expired.

I'm still not sure how the change in maxGasPrice is related to the deadlock, but I think I'll manage to nail it soon.

[yondonfu]

This should be fixed in the v0.5.24 release. If anyone continues to encounter this issue please re-open the issue and provide a report.

[0xVires]

Had another session crash right after redeeming a winning ticket (https://etherscan.io/tx/0x59fd2abf5ec03aa1623d506e20a1bd8a5e2c358a8946333a8e1f72f9f27c3886).

Again, no errors in the logs. But with the newest version, the O recovered itself without needing a restart. Still odd, especially since another winning ticket a few minutes earlier (https://etherscan.io/tx/0xc943a4a30f76e1ab369949124587dbb8ad3581fdbf823ea0f5635962ac6c39ce) didn't cause any problems.

[yondonfu]

Hm might be related to #2168 cc @leszko

[leszko]

Yeah, my rough guess is that it's related. Reopening the issue. The weird thing is that the version v0.5.24 reverted all the tx related changes, so it should behave exactly the same as the version v0.5.22.

@0xVires do you experience more this issue in the version v0.5.22 than you did in v0.5.24?

[leszko]

Closing, since I believe it is fixed by #2208. Reopen if you still encounter the issue after the next go-livepeer release.