Check mysql charset, if it's utf8, display an error log

[lunny]

This PR will check MySQL database charset when Gitea start. If it's utf8, an error log will be displayed.

[wolfogre]

The CI failure is related.

Hmm, so we used an incorrect database to test for years? 😂

[lunny]

It's wired, we never test emoji with our CI mysql services?

[wxiaoguang]

What about adding a "self check" page in site admin?

It could do a full self check, including:

1. database versions (pgsql, mysql, etc)
2. database charsets (mysql), not only just checking default, but check all columns.
3. reverse proxy rewriting and sub path (eg: is %2F kept?)
4. system time (to resolve incorrect OTP problems)
5. system health like free disk, etc

[lunny]

What about adding a "self check" page in site admin?

It could do a full self check, including:

1. database versions (pgsql, mysql, etc)

2. database charsets (mysql), not only just checking default, but check all columns.

3. reverse proxy rewriting and sub path (eg: is `%2F` kept?)

4. system time (to resolve incorrect OTP problems)

5. system health like free disk, etc

sounds like web-based gitea doctor

[wxiaoguang]

Yes, some checks could only be done by web UI (eg: 3, 4)

[silverwind]

UI self check I'd say is out of scope for this PR, but I agree it might be useful.

[silverwind]

Also match for utf8mb3 which is an alias to utf8 and interpolate charset into the message.

Reference: https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8.html

[wxiaoguang]

UI self check I'd say is out of scope for this PR, but I agree it might be useful.

But checking default charset is not enough. Every column should be checked one by one.

[lunny]

UI self check I'd say is out of scope for this PR, but I agree it might be useful.

But checking default charset is not enough. Every column should be checked one by one.

I think you are right, but we should not check them when start. It should be a backend task.

[wxiaoguang]

UI self check I'd say is out of scope for this PR, but I agree it might be useful.

But checking default charset is not enough. Every column should be checked one by one.

I think you are right, but we should not check them when start. It should be a backend task.

That why I suggest to make an admin UI

[silverwind]

Still not sure about UI.

How about actually refusing to start on wrong charset? Yes it's strict, but it's the only way to really ensure all users get it.

[wxiaoguang]

IMO an admin UI is the only solution. See these 5 points (actually not only these)

We shouldn't stop Gitea from starting, and we should do full check (check every column) when checking. So we still need an admin UI.

[silverwind]

How about doing a full table check and display a warning on frontpage to every admin user if issues were found? That should be a simple UI to add and would be better than something hidden away in the admin section. I assume a full table scan won't take so long.

For people concerned about startup performance, we could add a opt-out to these checks in config.

[techknowlogick]

How about doing a full table check and display a warning on frontpage to every admin user if issues were found? That should be a simple UI to add and would be better than something hidden away in the admin section. I assume a full table scan won't take so long.

For people concerned about startup performance, we could add a opt-out to these checks in config.

Atlassian does exactly this (a full table check, and then reports in UI) for the datacentre version of their products, they aren't blocking on the boot of the server (IIRC) and so we could probably do something similar.

I agree re: having it block the startup for the check is bad. At the very least a one-off "on-boot" cron might be sufficient.

[lunny]

Close as this will break users' instances.