UnknownJobKindError restricts client to only insert jobs for configured workers

[magaldima]

I'm experiencing an issue where I am trying to configure and start two river clients that belong to two separate applications that are each configured with separate worker bundles (and separate queues) that point to the same database/schema where each application CAN (as part of being the elected leader), insert jobs to be worked by the OTHER application/client.

• application A embeds client A which contains workers A and C
• application B embeds client B which contains workers, B and C

Where C represents a shared, periodic job that both clients can work in which C's Work() method inserts jobs for both A and B. Because of river's election system, only one client is the leader at any given point in time. This means client A will be inserting B jobs and client B will be inserting A jobs.

I would expect the A and B clients to be notified of the respective jobs that they are configured to work. How is this that much different from an insert-only client able to Insert jobs that are not configured as part of its own worker bundle?

Is this potentially something that could be changed where the validateJobArgs function invocation is removed? Does this have any side-effects/implications with River's internals?

EDIT: I just tried to have each client's C Work() method use an insert-only River client. This works and allows the insertion of jobs which are not configured to be worked by the application's "main" river client. However, the "main" river client A still picks up jobs for worker B and the worker then fails with the same error message: "job kind is not registered in the client's Workers bundle". I guess, more generally, am I thinking about this wrong? Is it possible to do what I'm trying to do?

If I run both clients simultaneous, it seems that each client evenly split up the jobs when they become available which results in ½ of the jobs being worked by the correct client/worker while the other ½ are worked by the "wrong" client and continue to fail for the reason above.

[bgentry]

Hi @magaldima, you mentioned "separate queues" at the beginning, and this is indeed the way to achieve the separation you're describing. If you want to prevent apps A and B from working each other's jobs, they need to be on separate queues—or at least the jobs you want to be segregated need to be on a queue that's only being worked by the clients you want to work it.

On the specific issue with inserting jobs that are meant to be worked by the other application, the UnknownJobKindError will indeed be triggered if you attempt to insert a job from a Client that has a Workers bundle configured but whose kind is not part of that bundle. If you're trying to insert a job B from within a client that only has A and C configured, you're going to run into an issue there. Depending on your setup, you could add those job types to all clients to facilitate insertion, but that may not be an option for you. Indeed an insert-only client with no Workers configured can insert any job.

A point on leader election: there is one leader elected per database and postgres schema combination. The maintenance processes that run on the elected leader generally run tasks that apply across the entire db/schema combination without regard to configured queues or worker bundles. The exception as you've encountered is the periodic job scheduler, which is not going to be able to insert jobs for workers that aren't configured on that Client. Of course the client doesn't participate in leader election if it isn't started & working jobs, so that does present an issue if you want the kind of separation you're attempting to get.

I'd consider that a bit of a bug or shortcoming that we could hopefully address, perhaps by skipping that validation when inserting periodic jobs or by giving the user the ability to do so via an InsertOpts addition. Maybe @brandur has some thoughts on what we might do here?

[brandur]

Well, this definitely feels a bit like an odd edge that I wouldn't think too many other people are going to run into, but I suppose an extra configuration option like SkipUnknownJobCheck or something of that nature wouldn't hurt. Maybe it's just a client-level option? I would imagine having to add it to InsertOpts on every insertion would be kind of annoying.

@magaldima Just to be definitive: if something like SkipUnknownJobCheck, that would be enough to shore up what you're trying to do here?

[magaldima]

Yes, I think that would be sufficient, but I can verify this change satisfies my use case locally too since I'm not intimately familiar with the river internals and/or if there are any other potential implications of this change.

[magaldima]

Validated this simple config option works locally! raised a PR here: #731

[magaldima]

I just thought of a potential workaround (as you were typing your reply): I could have each client configured with dynamic job-specific queues such that application A defines queue A while application B defines queue B and each job is enqueued on the respective queues regardless of which client inserts. I'm pretty sure this might solve the second problem, however it's still not ideal that I need to instantiate the insert-only clients to begin with which just seems like unnecessary overhead.