Improve the efficiency of DbLog migration for Django

[sphuber]

Fixes #2539

The old implementation was using Django's ORM to get the count and rows
of certain rows in the DbLog table. However, this was generating
extremely inefficient queries that made the migration run for days on
big databases. A similar problem was encountered with the original
SqlAlchemy implementation. That was solved in an earlier commit
80a0aa9117f399a5385a30ba0ce3e2afc7016a4e, which created a more
efficient query. We now replace the Django ORM solution with the same
custom SQL query as used for SqlAlchemy. This should significantly
improve the efficiency of this migration.

[sphuber]

I tested this on the mat2D database and the DbLog migrations now runs in roughly 10 minutes. Good enough I would say.

[giovannipizzi]

@sphuber just double checking that you also verified that the DbLogs are still there and correct in the mat2D, before merging

[sphuber]

I did, but it is hard to say. The unit tests should cover this. I forgot to note the number of rows in the original database for the DbLog table (did do this for a bunch of other tables). After migration there are still 220.000 logs remaining and some have been written to the two JSON dumps. But cannot say for certain that everything is there. Open to suggestions

[giovannipizzi]

I think it's ok then (maybe just run a few verdi commands to get logs and check the they work). I agree that tests should ideally already cover this

[coveralls]

Coverage decreased (-0.02%) to 72.713% when pulling 01b2eba on sphuber:fix_2539_efficiency_dblog_migration_django into c4eb8a0 on aiidateam:develop.

[coveralls]

Coverage decreased (-0.02%) to 72.713% when pulling 01b2eba on sphuber:fix_2539_efficiency_dblog_migration_django into c4eb8a0 on aiidateam:develop.

[coveralls]

Coverage decreased (-0.02%) to 72.713% when pulling 01b2eba on sphuber:fix_2539_efficiency_dblog_migration_django into c4eb8a0 on aiidateam:develop.

[szoupanos]

Thanks Seb! I also believe that if the tests pass and you did a basic check that everything is as it should, it should be OK