River on k8s?

[mesquita]

Hey hey,
what your experience running river on k8s?
My team has been experimenting using it in our service, the only problem that we seem to have is that the jobs are running on more than 1 pod at the same time. I don't know if it's a configuration problem on our side, but we thought we'd be fine due to the way river works.

Thank you.
btw, we love river

[bgentry]

Are you saying that an individual job is running on more than one pod at a time? That should not be possible, and we would need a lot more info about what you’re seeing to be able to look into it.

Or are you saying that each of your pods is running different jobs? Because that’s how it’s supposed to work—each client is supposed to work jobs for whichever queues are specified in its config. You are supposed to run as many workers on the same queue as you want in order to distribute many jobs across them.

I don’t think there’s anything k8s specific to pay attention to with config. Each client is essentially stateless except for actively running jobs. As long as you give the Client time to shut down cleanly before terminating your process and respect context cancellation in workers you should be fine. You can utilize the more aggressive shutdown mode as needed if you really want to interrupt existing jobs immediately.

[gabrielclimb]

Hey @bgentry , I work together with @mesquita.
So, following this article, I was able to monitor the river_leader table, it's changing a lot.
logs -> leader_election_monitoring.csv
Not sure if suppose to work like this but i think this might be the root of our problems.
might also related with #197

[bgentry]

@gabrielclimb I think you've hit a bug here, @brandur and I are confirming it now and working on a fix.

[gabrielclimb]

Let me know if there is a way I can help you guys.

[bgentry]

This should do it: #217

In the run up to release, we overlooked adding comprehensive test coverage on leader election, but it's pretty clear we need that. We'll get that added soon, sorry for the trouble!

[bgentry]

@gabrielclimb We will have that out in a release shortly. However this issue should only have resulted in duplicated work for the responsibilities of the leader such as pruning old jobs, inserting scheduled jobs, etc. If you have any reason to believe the same job was being worked twice simultaneously, please do share more details as that absolutely should not happen 🙏 Thanks for all the info to help us fix this fast.

[gabrielclimb]

Thank you @bgentry , really appreciated

[brandur]

@gabrielclimb We just cut https://github.com/riverqueue/river/releases/tag/v0.0.21 containing a bug fix for this. Please give it a shot and let us know how it goes.

[gabrielclimb]

just updated, looks like it is fixed.

thank you very much guys
@bgentry @brandur

[mesquita]

Thank you so much!! @bgentry @brandur @gabrielclimb