Autopush seems to be broken

[bowlofeggs]

Usually when an update reaches the stable karma threshold, there will be a comment from the bodhi user stating that the update has reached this status and that it will be pushed to stable. For example, this update from bodhi-2.0-212 exibited that behavior:

https://bodhi.fedoraproject.org/updates/FEDORA-2016-ffc86f2b30

Since deploying 2.2.0 to production, it has been noted that updates no longer seem to get that comment from Bodhi when they reach stable karma. For example, see this update in stg:

https://bodhi.stg.fedoraproject.org/updates/FEDORA-2016-4d0198119a

My comment should have caused bodhi to submit it for stable automatically, but it did not. Here is one more example of a non-critical path update that should be autopushed:

https://bodhi.stg.fedoraproject.org/updates/FEDORA-2016-93c58fa1b3

[trishnaguha]

@bowlofeggs Not sure why. We have test for this.

[trishnaguha]

What about this one? https://bodhi.fedoraproject.org/updates/quantum-espresso-5.4.0-10.fc25
This seems to be autopushed.

[bowlofeggs]

@trishnaguha I think that update may have happened before we finished updating production to 2.2.0. The outage started at 19:00:00 UTC that day, and that update comment came it at ~19:25, but it took us longer than that to get everything done. I don't know the exact time we started/finished, but I could probably find out in the logs if we want to know for sure. I could be wrong.

Furthermore, there is some suspicion that the way we resumed a failed push might have put some strange data into the database that might be causing this issue for some updates. It's possible also that on some updates, the client submitted a form with the old version of bodhi running in the browser that was in some way incompatible with the new server. We really don't know what is causing this, but there are definitely updates in production that have some issue here. I was even able to reproduce the problem in stg by adding a comment that should have triggered the stability.

[bowlofeggs]

This may be related to #949 in some way, but that's just a guess.

[bowlofeggs]

@puiterwijk has come to the rescue again! It turns out that there was a commit a while ago that added an autokarma column to the Update model. A later commit added a schema migration, but the schema migration did not include a data migration to fill out this column for existing commits. When the code that detected autokarma was changed to use this column, all existing Updates were skipped in the autokarma logic since they had their autokarma columns set to None.

I will fix this by writing a new migration that sets all Updates with autokarma set to NULL to True.