Ensure MySQL data supports timezones

[lazyfrosch]

This migrates all DATETIME to TIMESTAMP in MySQL, so we support timezones and every timestamp can be correctly converted between UTC (storage) and local timezone of the user.

Only affects MySQL

• Offer DB migration
• Test performance against large DB
• Disable max_execution_time for migrations
• Add note here how to apply manually

Affects the following tables:

• director_activity_log (largest update set)
• director_deployment_log
• director_schema_migration
• director_job
• import_source
• sync_rule
• sync_run

fixes #1332
fixes #1813

Manual installation

This fix will be released with 1.7.0 and not backported to an earlier version.

If you currently experience problems with timezones, you can apply the schema migration manually like this:

Make a mysqldump backup before starting with the changes*

$ mysql icinga_director

// Set your preferred timezone if you like (important for converting from local to UTC)
// By default: client system timezone
mysql> SET time_zone = '+02:00'; -- for CEST

After this apply the ALTER TABLE statements manually:

SET sql_mode = 'STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,NO_ENGINE_SUBSTITUTION,PIPES_AS_CONCAT,ANSI_QUOTES,ERROR_FOR_DIVISION_BY_ZERO';

ALTER TABLE director_activity_log
  MODIFY change_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP;

ALTER TABLE director_deployment_log
  MODIFY start_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  MODIFY end_time TIMESTAMP NULL DEFAULT NULL,
  MODIFY abort_time TIMESTAMP NULL DEFAULT NULL;

ALTER TABLE director_schema_migration
  MODIFY migration_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP;

ALTER TABLE director_job
  MODIFY ts_last_attempt TIMESTAMP NULL DEFAULT NULL,
  MODIFY ts_last_error TIMESTAMP NULL DEFAULT NULL;

ALTER TABLE import_source
  MODIFY last_attempt TIMESTAMP NULL DEFAULT NULL;

ALTER TABLE import_run
  MODIFY start_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  MODIFY end_time TIMESTAMP NULL DEFAULT NULL;

ALTER TABLE sync_rule
  MODIFY last_attempt TIMESTAMP NULL DEFAULT NULL;

ALTER TABLE sync_run
  MODIFY start_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP;

Depending on your database size, this can take up to a few minutes.

WARNING: Do not change anything in the director_schema_migration table.

When you upgrade to >= 1.7 at a later date the normal schema migration will be run again, but since the table columns are already in the correct format, it won't change anything and should not fail.

---

ref/IP/13765
ref/IP/13087

[lazyfrosch]

Tests with the dataset of a large user: table-sizes.txt
(Import data and config files have been omitted from dump)

Schema was upgraded to 162 before testing:

Server version: 10.2.22-MariaDB-1:10.2.22+maria~bionic mariadb.org binary distribution

MariaDB [director]> select count(*) from director_activity_log;
+----------+
| count(*) |
+----------+
|   597108 |
+----------+
1 row in set (0.20 sec)

MariaDB [director]> select count(*) from director_deployment_log;
+----------+
| count(*) |
+----------+
|     7314 |
+----------+
1 row in set (0.02 sec)


$ time icingacli director migration run

real	0m43,630s

Same test with an older MySQL version:

Server version: 5.5.62 MySQL Community Server (GPL)

$ time icingacli director migration run

real	1m49,527s

All tests ran on my notebook.

[lazyfrosch]

@Thomas-Gelf Opinions?

[Thomas-Gelf]

@lazyfrosch: thanks for tackling this, but please go for BIGINT/msec's as proposed in #58 - that way we are completely independent from DB specific issues

[Thomas-Gelf]

@lazyfrosch: please see related mail, afterwards feel free to finalize/merge this. Thanks for the good work!

Ensure MySQL data supports timezones (#1840)Ensure MySQL data supports timezones (#1840)

[lazyfrosch]

Added a commit for disabling max_execution_time and amended the top comment with details.

Ready for merge after CI.

[friesoft]

@lazyfrosch the mysql migration doesn't work for us using MySQL 5.6 Enterprise Edition:

Migration 163 failed (SQLSTATE[42000]: Syntax error or access violation: 1067 Invalid default value for 'change_time') while running ALTER TABLE director_activity_log MODIFY change_time TIMESTAMP NOT NULL

Using a dump of the same database in the icinga standalone vagrant box with MariaDB 5.5.60 works.

[lazyfrosch]

I heard about some corner case with MySQL 5.6.

Will look into it, a small adjustment to the migration should be all needed.

[lazyfrosch]

Interesting, a fresh schema in MySQL 5.6 with migration works fine...

@friesoft just to be sure, could you get me this for your current schema:

DESC director_activity_log;
SHOW CREATE TABLE director_activity_log \G

You could also try this migration: (note the extra default values with NOT NULL)

REMOVED

[lazyfrosch]

Ah I found the problem, ignore the suggestion above

[lazyfrosch]

The correct migration is:

SET sql_mode = 'STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,NO_ENGINE_SUBSTITUTION,PIPES_AS_CONCAT,ANSI_QUOTES,ERROR_FOR_DIVISION_BY_ZERO';

ALTER TABLE director_activity_log
  MODIFY change_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP;

ALTER TABLE director_deployment_log
  MODIFY start_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  MODIFY end_time TIMESTAMP NULL DEFAULT NULL,
  MODIFY abort_time TIMESTAMP NULL DEFAULT NULL;

ALTER TABLE director_schema_migration
  MODIFY migration_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP;

ALTER TABLE director_job
  MODIFY ts_last_attempt TIMESTAMP NULL DEFAULT NULL,
  MODIFY ts_last_error TIMESTAMP NULL DEFAULT NULL;

ALTER TABLE import_source
  MODIFY last_attempt TIMESTAMP NULL DEFAULT NULL;

ALTER TABLE import_run
  MODIFY start_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  MODIFY end_time TIMESTAMP NULL DEFAULT NULL;

ALTER TABLE sync_rule
  MODIFY last_attempt TIMESTAMP NULL DEFAULT NULL;

ALTER TABLE sync_run
  MODIFY start_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP;

@friesoft appreciate your feedback!

[friesoft]

@lazyfrosch works 😃 Thanks for the fast fix 👍