feat(lua): port of NodeMCU Lua53 read-only tables

[raphaelcoeffic]

Port of #2986 to main, so that it can be tested separately.

Summary of changes:

• removed duplicated light C functions (introduced by eLua LTR patches, but already integrated in Lua 5.2)
• removed read-only table implementation introduced in OpenTx 2.3.0 (see here)
• added read-only table implementation from NodeMCU (see here) including lookup cache (512 bytes)
• changed LUA_NUMBER default type from double to float

After some tweaks, it seems that changing from double to float has a nice side effect: it removes some huge floating point functions and uses the smaller float version. This allows to regain around 8 KB of Flash memory!

Please note: Changing from double to float has been postponed as it introduces issue with LcdFlags using the full 32bits, whereby a float can only represent 24 bits faithfully.

TODOs:

• test / fix byte-code compatibility

[stronnag]

TX16S, INAV Lua script.
On selecting any model that uses the INAV lua, the screen freezes, requiring hard shutdown (power button 10s).

Models not using the script load normally.

[gagarinlg]

And if you delete the luac files and just leave the lua files on the SD card?

[stronnag]

And if you delete the luac files and just leave the lua files on the SD card?

Same. Freeze

[raphaelcoeffic]

@stronnag it would be highly beneficial if you could plug a on-board debugger ;-) That would at least give us a hint on what is happening.

When you say "model that use the INAV script", you mean that have a widget with that script, correct?

[stronnag]

I don't have a on-board debugger, alas.
The INAV lua script comprises WIDGET & SCRIPTS/TELEMETRY

In companion29, having deleted every luac, the default model happens to use the INAV script.

0.02s: luaDumpState(/WIDGETS/GPS/main.luac): Saved bytecode to file.
0.02s: luaLoadWidgetCallback()
0.02s: register widget GPS
0.02s: Loaded Lua widget GPS
0.02s: 	not found
0.02s: f_stat(/home/jrh/Projects/EdgeTX28/sdcard/WIDGETS/iNav) = OK
0.02s: f_stat(/home/jrh/Projects/EdgeTX28/sdcard/WIDGETS/iNav/main.lua) = OK
0.02s: luaLoadFile(/WIDGETS/iNav/main.lua)
0.02s: 	not found
0.02s: f_stat(/home/jrh/Projects/EdgeTX28/sdcard/WIDGETS/iNav/main.luac) = error 2 (No such file or directory)
0.02s: f_stat(/home/jrh/Projects/EdgeTX28/sdcard/WIDGETS/iNav/main.lua) = OK
0.02s: luaLoadScriptFileToState(/WIDGETS/iNav/main.lua, T): loading /WIDGETS/iNav/main.lua
0.02s: f_open(/home/jrh/Projects/EdgeTX28/sdcard/WIDGETS/iNav/main.lua, 1) = 0x7f02f0002f30 (FIL 0x7f02fb7fcf38)
0.02s: f_close(0x7f02f0002f30) (FIL:0x7f02fb7fcf38)
0.02s: 	not found
0.02s: f_open(/home/jrh/Projects/EdgeTX28/sdcard/WIDGETS/iNav/main.luac, a) = 0x7f02f0002f30 (FIL 0x7f02fb7fd620)
0.02s: f_close(0x7f02f0002f30) (FIL:0x7f02fb7fd620)
0.02s: f_utime(/home/jrh/Projects/EdgeTX28/sdcard/WIDGETS/iNav/main.luac) set mtime = Tue Jan 25 09:07:26 2022

0.02s: luaDumpState(/WIDGETS/iNav/main.luac): Saved bytecode to file.
0.02s: 	not found
0.02s: f_stat(/home/jrh/Projects/EdgeTX28/sdcard/SCRIPTS/TELEMETRY/iNav.luac) = error 2 (No such file or directory)
0.02s: f_stat(/home/jrh/Projects/EdgeTX28/sdcard/SCRIPTS/TELEMETRY/iNav.lua) = OK
0.02s: luaLoadScriptFileToState(/SCRIPTS/TELEMETRY/iNav, bt): loading /SCRIPTS/TELEMETRY/iNav.lua
0.02s: f_open(/home/jrh/Projects/EdgeTX28/sdcard/SCRIPTS/TELEMETRY/iNav.lua, 1) = 0x7f02f0002f30 (FIL 0x7f02fb7fcc78)

Freeze. In the splash screen.

[raphaelcoeffic]

@stronnag it should work better now: I had a nasty bug left from clean-up of some strange OpenTx change to computesizes(). It seems I had not fully reverted the code.

[stronnag]

For the record and as reported on Discord:

• 3b04042 et al fix the freezes
• There are undesirable visual artefacts that are NOT present in contemporary main

[raphaelcoeffic]

@stronnag regarding the visual artefacts: the issue is with the flags....

Lua uses in v5.2 only floating point numbers for storing any number. Now that we move to single precision floating point numbers, we have only 24 significant bits, which is not enough to encode the flags in full.

Using just RIGHT (8) is ok, as it makes a number that can be represented this way.

When, however, a color like WHITE is added to the mix, the flags become 0xFFFF0008, which cannot be represented with a 32 bit float...

So what happens is that the addition overflows, and this is what we see in the C code:

There are different possible solution to this issue:

• make flags a custom type (using "light user data" and a meta-table overloading the necessary operations like +),
• jump the gun and move to Lua 5.3, which has support for 32 integers, which would be exactly what we need.

[raphaelcoeffic]

@stronnag I reverted the change to float. It should behave normally now.

[stronnag]

@stronnag I reverted the change to float. It should behave normally now.

OK again in the simulator and radio (TX16S).

[pfeerick]

When jumping between v2.8.0 and this PR build, most Lua widgets and scripts I tried seem to 'just work' fine... (i.e. most of the standard widgets, ExpressLRS lua, inav, few other random ones on the radio)... the exception being whatever build of Yappu telemetry I have on the TX16S (and also latest 2.0.0 dev build) ... works fine on 2.8.0 but with this PR I get one of those lovely cryptic ERROR in create(): ?:0: attempt to index field '?' (a table value errors... for both the yappu widget and also standalone lua config tool... I'm guessing something has changed? 😭

[raphaelcoeffic]

@pfeerick This errors almost looks like some binary loading error. Did you try for the fun to remove the .luac files and see if that compiles fine from source on radio? If this is not the case, we have a real API change, which was not intended.

[pfeerick]

Nope. I'll give that a try and report back.

[raphaelcoeffic]

@pfeerick at least on Boxer and Color simu I can load ELRS LUA seamlessly regardless of using main, 2.8 or this PR. So as of now I cannot reproduce this. I had this issue earlier (basically before applying 670a515).

[raphaelcoeffic]

I just found something that might affect lua_typename(). This could have affected this part of Yaapu script: https://github.com/yaapu/FrskyTelemetryScript/blob/master/OTX_ETX/c480x272/SD/WIDGETS/yaapu/main.lua#L654

-----------------------------
-- clears the loaded table
-- and recovers memory
-----------------------------
utils.clearTable = function(t)
  if type(t)=="table" then
    for i,v in pairs(t) do
      if type(v) == "table" then
        utils.clearTable(v)
      end
      t[i] = nil
    end
  end
  t = nil
  collectgarbage()
  collectgarbage()
end

The type() function uses lua_typename() underneath, and I believe it could have caused some troubles with the exact type returned being different (shift by 2 in the string table....).

This has been fixed with d7bc59e.

[pfeerick]

As mentioned above, yappu telemetry was the exception, and I've not gotten back to this PR yet (or anything EdgeTX today yet 😭 ) but will try deleting the .luac shortly, and then with updated PR. As well as checking out the latest state of the module-api PR wrt EL18 and XJT :)

[raphaelcoeffic]

@pfeerick actually it should work without deleting the .luac files. That was just a suggestion to be able to better check the issue.

[raphaelcoeffic]

To enable tests on Companion simulation, I started these 2 manual builds:

• Windows Companion 64-bit
• MacOSX Companion

[pfeerick]

I've just retried this with the latest commit, and TX16S now starts yappu without any issue :)