bodhi randomly throws "NoSuchColumnError testcases.id" server errors for various queries

[decathorpe]

This looks like a continuation of the #4287 saga but with a different database table / column involved:

{
    errors: [
        {
            "name": "NoSuchColumnError",
            "location": "body",
            "description": "Could not locate column in row for column 'testcases.id'"
        },
    ],
    status: "error"
}

This seems to happen at about the same frequency as the "bugs.id" problem that happened before bodhi 5.7.2 was deployed.

[AdamSaleh]

Damn it. I never see problems like this when I test on staging. Looking into it.

[mattiaverga]

@AdamSaleh can you please deploy this commit as a hotfix in prod:

diff --git a/bodhi/server/models.py b/bodhi/server/models.py
index c2506cc2..7f669904 100644
--- a/bodhi/server/models.py
+++ b/bodhi/server/models.py
@@ -1918,8 +1918,8 @@ class Update(Base):
 
     # One-to-many relationships
     comments = relationship('Comment', backref='update', cascade="all,delete,delete-orphan",
-                            order_by='Comment.timestamp')
-    builds = relationship('Build', backref='update', order_by='Build.nvr')
+                            order_by='Comment.timestamp', lazy='joined')
+    builds = relationship('Build', backref='update', order_by='Build.nvr', lazy='joined')
 
     # Many-to-many relationships
     bugs = relationship('Bug', secondary=update_bug_table, backref='updates', order_by='Bug.bug_id')

It partially reverts what I've done in #4048 , let's see if it can help... otherwise we'll need someone with a lot of experience in sqlalchemy to review our relationship joins...

[AdamSaleh]

ok, and I definitely need to add something like this to my pre-release checklist. Because while I will try to deploy the fix asap, I am very annoyed that I can't reliably reproduce the damn thing.

[AdamSaleh]

@mattiaverga PR here: #4303

[mattiaverga]

ok, and I definitely need to add something like this to my pre-release checklist. Because while I will try to deploy the fix asap, I am very annoyed that I can't reliably reproduce the damn thing.

I assume this problem shows only when the db is under load or after some cache is full... I checked the logs yesterday 10 hours later you deployed 5.7.2 and there was no sign of the issue

[mattiaverga]

@mattiaverga PR here: #4303

Can't we just deploy the patch through ansible to see if it's effective, rather than committing to git branch?

[AdamSaleh]

Ok, will try to deploy from my branch

[AdamSaleh]

Ok, hotfix is running in prod

[mattiaverga]

Thanks. I'm attending an online course now, so I'll check logs from time to time. Everything seems fine till now.

The PR will need some changes to the tests to get the pass (not sure why in some cases the build nvr order doesn't seem to be honored, I had to change that also in my old PR).

[AdamSaleh]

Yes, had to disable those checks when building the image. Should be able to look into that/copy it from your old PR.