Embedded lua scripts into FTL

[DL6ER]

What does this implement/fix?

Add facility to embed lua scripts into FTL. This reduces dependency on externally provided infrastructure and is easily extensible for to an arbitrary number of scripts whenever we deem this appropriate.

How to add new scripts? Just add them into the src/lua/scripts/ directory and amend CMakeLists.txt as well as the code parts of FTL which are responsible for reading the scripts.

Related issue or feature (if applicable): N/A

Pull request in docs with documentation (if applicable): N/A

---

By submitting this pull request, I confirm the following:

1. I have read and understood the contributors guide, as well as this entire template. I understand which branch to base my commits and Pull Requests against.
2. I have commented my proposed changes within the code.
3. I am willing to help maintain this change if there are issues with it later.
4. It is compatible with the EUPL 1.2 license
5. I have squashed any insignificant commits. (git rebase)

Checklist:

• The code change is tested and works locally.
• I based my code and PRs against the repositories developmental branch.
• I signed off all commits. Pi-hole enforces the DCO for all contributions
• I signed all my commits. Pi-hole requires signatures to verify authorship
• I have read the above and my PR is ready for review.

[yubiuser]

rockpi@rockpi-4b:~$ pihole-FTL -vv
****************************** FTL **********************************
Version:         vDev-395e079
Branch:          new/embedded_lua_scripts
Commit:          395e0792 (2022-11-20 13:53:41 +0100)
Architecture:    aarch64 (compiled on CI)
Compiler:        aarch64-linux-gnu-gcc (Debian 8.3.0-2) 8.3.0

****************************** dnsmasq ******************************
Version:         pi-hole-v2.88rc3
Compile options: IPv6 GNU-getopt no-DBus no-UBus no-i18n IDN DHCP DHCPv6 Lua TFTP no-conntrack ipset no-nftset auth cryptohash DNSSEC loop-detect inotify dumpfile

****************************** SQLite3 ******************************
Version:         3.40.0
Compile options: ATOMIC_INTRINSICS=1 COMPILER=gcc-8.3.0 DEFAULT_AUTOVACUUM DEFAULT_CACHE_SIZE=-16384 DEFAULT_FILE_FORMAT=4 DEFAULT_FOREIGN_KEYS DEFAULT_JOURNAL_SIZE_LIMIT=-1 DEFAULT_MEMSTATUS=0 DEFAULT_MMAP_SIZE=0 DEFAULT_PAGE_SIZE=4096 DEFAULT_PCACHE_INITSZ=20 DEFAULT_RECURSIVE_TRIGGERS DEFAULT_SECTOR_SIZE=4096 DEFAULT_SYNCHRONOUS=2 DEFAULT_WAL_AUTOCHECKPOINT=1000 DEFAULT_WAL_SYNCHRONOUS=2 DEFAULT_WORKER_THREADS=0 DQS=0 ENABLE_DBPAGE_VTAB MALLOC_SOFT_LIMIT=1024 MAX_ATTACHED=10 MAX_COLUMN=2000 MAX_COMPOUND_SELECT=500 MAX_DEFAULT_PAGE_SIZE=8192 MAX_EXPR_DEPTH=1000 MAX_FUNCTION_ARG=127 MAX_LENGTH=1000000000 MAX_LIKE_PATTERN_LENGTH=50000 MAX_MMAP_SIZE=0x7fff0000 MAX_PAGE_COUNT=1073741823 MAX_PAGE_SIZE=65536 MAX_SQL_LENGTH=1000000000 MAX_TRIGGER_DEPTH=1000 MAX_VARIABLE_NUMBER=32766 MAX_VDBE_OP=250000000 MAX_WORKER_THREADS=8 MUTEX_PTHREADS OMIT_DEPRECATED OMIT_DESERIALIZE OMIT_LOAD_EXTENSION OMIT_PROGRESS_CALLBACK SYSTEM_MALLOC TEMP_STORE=1 THREADSAFE=1
******************************** LUA ********************************
Lua 5.4.1  Copyright (C) 1994-2020 Lua.org, PUC-Rio

***************************** LIBNETTLE *****************************
Version: 3.8
GMP: Full

Just a thought: can we add which scripts are added (e.g. inspect.lua) to the LUA section in the version output?

[DL6ER]

New pihole-FTL -vv output:

[...]

******************************** LUA ********************************
Lua 5.4.1  Copyright (C) 1994-2020 Lua.org, PUC-Rio
Embedded libraries: inspect.lua (8.46 KB) 

[...]

[yubiuser]

Should we add a test to check if all *.lua files in src/lua/scripts end up in the final binary?

[yubiuser]

This code can go now:

FTL/test/run.sh

Lines 63 to 65 in 844c8b9

	# Prepare LUA scripts
	mkdir -p /opt/pihole/libs
	wget -O /opt/pihole/libs/inspect.lua https://ftl.pi-hole.net/libraries/inspect.lua

[DL6ER]

@yubiuser This code is already removed in this PR: 6b6c069

I'm not sure a test will be necessary. What the user tries to do simply won't work if you don't include the new scripts. I'd rather like to ask for adding individual library tests very much like we have a test dedicated to test if the inspect library does what it should do. We can also very conveniently use this very same library to check the properties of any other future libraries we load and use by default.

[yubiuser]

@yubiuser This code is already removed in this PR: 6b6c069

How did I miss this?

---

I'd rather like to ask for adding individual library tests very much like we have a test dedicated to test if the inspect library does what it should do.

I'm fine with this approach, we only need to make sure we/others don't forget to add a test

[dschaper]

FTL already checks for .lua files in the lua path, correct? So this takes specific .lua files and bakes them in to the binary, no need for the file on the path.

What happens when there is a .lua file in the path that conflicts or duplicates what is in the binary? Say a version conflict.

[pralor-bot]

This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/pi-hole-ftl-v5-20-and-web-v5-18-released/59959/1