Update embedded SQLite to 3.41.1

[DL6ER]

What does this implement/fix?

Update the embedded SQLite3 engine. Pi-hole will benefit from improvements to the PRAGMA integrity_check command and miscellaneous performance enhancements.

We do not add SQLITE_ENABLE_STMT_SCANSTATUS, the SQLite3 help says:

Those interfaces are normally omitted from the build because they imposes a performance penalty, even on statements that do not use the feature.

2023-02-21 (3.41.0)

1. Query planner improvements:
   1. Make use of indexed expressions within an aggregate query that includes a GROUP BY clause.
   2. The query planner has improved awareness of when an index is a covering index and adjusts predicted runtimes accordingly.
   3. The query planner is more aggressive about using co-routines rather than materializing subqueries and views.
   4. Queries against the built-in table-valued functions json_tree() and json_each() will now usually treat "ORDER BY rowid" as a no-op.
   5. Enhance the ability of the query planner to use indexed expressions even if the expression has been modified by the constant-propagation optimization. (See forum thread 0a539c7.)
2. Add the built-in unhex() SQL function.
3. Add the base64 and base85 application-defined functions as an extension and include that extension in the CLI.
4. Add the sqlite3_stmt_scanstatus_v2() interface. (This interface is only available if SQLite is compiled using SQLITE_ENABLE_STMT_SCANSTATUS.)
5. In-memory databases created using sqlite3_deserialize() now report their filename as an empty string, not as 'x'.
6. Changes to the CLI:
   1. Add the new base64() and base85() SQL functions
   2. Enhanced EXPLAIN QUERY PLAN output using the new sqlite3_stmt_scanstatus_v2() interface when compiled using SQLITE_ENABLE_STMT_SCANSTATUS.
   3. The ".scanstats est" command provides query planner estimates in profiles.
   4. The continuation prompt indicates if the input is currently inside of a string literal, identifier literal, comment, trigger definition, etc.
   5. Enhance the --safe command-line option to disallow dangerous SQL functions.
   6. The double-quoted string misfeature is now disabled by default for CLI builds. Legacy use cases can reenable the misfeature at run-time using the ".dbconfig dqs_dml on" and ".dbconfig dqs_ddl on" commands.
7. Enhance the PRAGMA integrity_check command so that it detects when text strings in a table are equivalent to but not byte-for-byte identical to the same strings in the index.
8. Enhance the carray table-valued function so that it is able to bind an array of BLOB objects.
9. Added the sqlite3_is_interrupted() interface.
10. Long-running calls to sqlite3_prepare() and similar now invoke the progress handler callback and react to sqlite3_interrupt().
11. The sqlite3_vtab_in_first() and sqlite3_vtab_in_next() functions are enhanced so that they reliably detect if they are invoked on a parameter that was not selected for multi-value IN processing using sqlite3_vtab_in(). They return SQLITE_ERROR instead of SQLITE_MISUSE in this case.
12. The parser now ignores excess parentheses around a subquery on the right-hand side of an IN operator, so that SQLite now works the same as PostgreSQL in this regard. Formerly, SQLite treated the subquery as an expression with an implied "LIMIT 1".
13. Added the SQLITE_FCNTL_RESET_CACHE option to the sqlite3_file_control() API.
14. Makefile improvements:
    1. The new makefile target "sqlite3r.c" builds an amalgamation that includes the recovery extension.
    2. New makefile targets "devtest" and "releasetest" for running a quick developmental test prior to doing a check-in and for doing a full release test, respectively.
15. Miscellaneous performance enhancements.

2023-03-10 (3.41.1)

1. Provide compile-time options -DHAVE_LOG2=0 and -DHAVE_LOG10=0 to enable SQLite to be compiled on systems that omit the standard library functions log2() and log10(), repectively.
2. Ensure that the datatype for column t1.x in "CREATE TABLE t1 AS SELECT CAST(7 AS INT) AS x;" continues to be INT and is not NUM, for historical compatibility.
3. Enhance PRAGMA integrity_check to detect when extra bytes appear at the end of an index record.
4. Fix various obscure bugs reported by the user community. See the timeline of changes for details.

Related issue or feature (if applicable): N/A

Pull request in docs with documentation (if applicable): N/A

---

By submitting this pull request, I confirm the following:

1. I have read and understood the contributors guide, as well as this entire template. I understand which branch to base my commits and Pull Requests against.
2. I have commented my proposed changes within the code.
3. I am willing to help maintain this change if there are issues with it later.
4. It is compatible with the EUPL 1.2 license
5. I have squashed any insignificant commits. (git rebase)

Checklist:

• The code change is tested and works locally.
• I based my code and PRs against the repositories developmental branch.
• I signed off all commits. Pi-hole enforces the DCO for all contributions
• I signed all my commits. Pi-hole requires signatures to verify authorship
• I have read the above and my PR is ready for review.

[DL6ER]

Codespell needs to be overwritten, this is no error and not even related to this PR:

[yubiuser]

Added the sqlite3_is_interrupted() interface.

Could this be something for v6 - if users select a very long time range and query runs longer than 5 seconds display a modal and offering cancleation?

The double-quoted string misfeature is now disabled by default for CLI builds.

Are we affected by this? Did you check that we don't use double-quotes string as string literals?

[DL6ER]

Added the sqlite3_is_interrupted() interface.

Could this be something for v6 - if users select a very long time range and query runs longer than 5 seconds display a modal and offering cancleation?

It is actually not trivial at all to do this. The thread that does the long computation is blocking. The interrupt routines are meant for the case when exactly this thread gets interrupted, e.g., by Ctrl + C. This is not something we could do (or even emulate) over a HTTP connection.

So, the only thing we could do is sending such a cancellation request via another HTTP connection but how does it know which process to cancel (there may be several long-running queries at the same time)? We'd need to somehow maintain an array with database pointers in global memory with some smart IDs usable to cancel requests. This would be ... let's say ... not a very trivial endeavour. I also see at least one possible race condition with a delayed cancellation request for some query that is already finished so garbage collection on this global array is another complex thing.

The double-quoted string misfeature is now disabled by default for CLI builds.

Are we affected by this? Did you check that we don't use double-quotes string as string literals?

We are not affected. We disabled this misfeature already back in 2019 intentionally. It just now became the default for the CLI sqlite3 command compilation. This doesn't change anything on your side as our compile-time options already enforced this also in the CLI application (and not a single user ever complained about this).

[pralor-bot]

This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/pi-hole-ftl-v5-22-web-v5-19-and-core-v5-16-1-released/61999/1