For a very long time now I have been looking at being able to execute programs on the esp32 without having to load the code via any IDE(Arduino, VSCode ...). I rapidly forgot that topic and focus my energy on writing leds driver and artnet libraries for the esp32. Once that done, I start thinking again about being able to write animations 'live'. Of course in my searches for existing previous implementations, I fell upon PixelBlaze and later Arti-Fx (part of the WLED MoonModules). In both cases the scripting lanquage is a subset of JavaScript and to my knowkledge interpreted. A number of impressive animations have already been written for both these environments and PixelBlaze has a full web based accessible examples library to which one can contribute. WLED Arti-Fx and PixelBlaze both have a web interface to program the animations. PixelBlaze is a closed/licensed product so no way to 'look' into it and I haven't installed Arti-Fx I have to admit :). Hence I do not have any real clue about the performance of these two implementations.
Several interpreted scripting lanquages are available (Lua, JavaScript, Gravity, ...) for the esp32, I have tried Lua and Gravity on my panel (48 16x16 ws2812 panels or 12288 leds) the refresh rate of the panel is 128fps thanks to my VirtualLedDriver. The results were quite a let down in term of performance. For a simple rainbow animation the global frame rate comparison is:
- 85 fps: for the C++ program uplaod via arduino
- 10 fps: for Gravity script
- 3 fps: for Lua script
This is indeed due do the large number of pixels needed to be calculated for my panel. Hence none of these language will make the cut for me.
In the sc_examples directory you will find examples of complexe scripts.
So I have decided also give it a go. Can I also conceive a 'language' to program led animations.
Being old school, I know that assembly language is the way to get the most of performance out of a CPU (given that your skills at writing ML are quite good of course). So I've tried to duplicate the same program directly assembly (still using arduino) and it worked great.
So I wrote an assembly language compiler as well as a small terminal interface to edit/save/compile/execute these programs (It will later become ledOS )
Of course proposing an assembly parer is not a really something usable :) so I took my chance at writing a compiler which will produce assembly languages for the esp32 xtensa CPU.
I have chosen to go for a C like syntax which is closed to JavaScript with stronger typing. I have made a loose adaptation of the language. But you can write thing like this:
void main()
{
setTime();
resetStat();
int h = 1 ;
while (h > 0)
{
beforeRender();
for (int i = 0; i < width; i++)
{
for (int j = 0; j < height; j++)
{
render2D(i, j);
}
}
show();
}
}I am not a compiler specialist and I wrote 'by hand' the parser and compiler using no existing grammar. Please do not throw me into the fire for this. The entire exercice has been a learning experience also and the result is not half cooked (according to me).
Later in this documentation you will see what is possible and not.
This libray doesn't provide an environment to write scripts. Nevertheless it has been included in:
- LedOS : an environment in a console terminal to edit, save and execute programs
- MoonModules StarLight : Web enabled ESP32 application for LED manipulation based on WLED which contains several modules (artnet, DMX, powerful mapping tool ....) Live Scripts doc
Even if the language is made to be of general application, the target of this language is targeted at creating led animations. As a consequence the scripting language has some limitations that
- To create a new instance of a parser :
Parser _parser; - To parse a script and create an executable :
Executable exec=_parser.parseScript(&script); - To execute the code assuming there is function called main in you code:
exec.execute("main");
If you execute the following program firstlight.ino:
#include "ESPLiveScript.h"
string script="\
void main()\n\
{\n\
for(int i=0;i<20;i++)\n\
{\n\
printfln(\"i:%2d 3xi:%2d\",i,3*i);\n\
}\n\
}";
void setup() {
// put your setup code here, to run once:
Serial.begin(115200);
Parser _parser;
Executable exec=_parser.parseScript(&script);
if(exec.isExeExists())
{
exec.execute("main");
}
}
void loop() {
// put your main code here, to run repeatedly:
}the output will be:
***********PARSING DONE*********
***********COMPILING DONE*********
max used memory: 7064 maxstack:2156 started 265960 free mem:258796 consumed 7164 time:30ms
max used memory: 7064 maxstack:2156 started 265960 free mem:262940 consumed 3020 time:41ms
***********AFTER CLEAN*********
***********CREATE EXECUTABLE*********
Creation of an 208 bytes binary and 70 bytes data
Parsing 75 assembly lines ...
max used memory: 7064 maxstack:2156 started 265960 free mem:265000 consumed 960 time:104ms
i: 0 3*i: 0
i: 1 3*i: 3
i: 2 3*i: 6
i: 3 3*i: 9
i: 4 3*i:12
i: 5 3*i:15
i: 6 3*i:18
i: 7 3*i:21
i: 8 3*i:24
i: 9 3*i:27
i:10 3*i:30
i:11 3*i:33
i:12 3*i:36
i:13 3*i:39
i:14 3*i:42
i:15 3*i:45
i:16 3*i:48
i:17 3*i:51
i:18 3*i:54
i:19 3*i:57
NB: if you have several functions it the same script you can call any of the function
To delete an executable hence freeing the used memory by the binary use: exec.free()
To add parameter to the exection call
Arguments args;
args.add(1);
args.add(2.3);
...
exec.execute("main", args);NB: For the moment you can only have interger and float entry parameters
In this example we do calculate the factorial:
#include "ESPLiveScript.h"
string script="\
int fact(int h)\n\
{\n\
if(h==1)\n\
{ \n\
return 1;\n\
} \n\
return h*fact(h-1);\n\
}\n\
\n\
void main(int g)\n\
{ \n\
printfln(\"factorial of %d is %d\",g,fact(g));\n\
}";
void setup() {
// put your setup code here, to run once:
Serial.begin(115200);
Parser p;
Executable exec=p.parseScript(&script);
if(exec.isExeExists())
{
Arguments args;
args.add(5);
exec.execute("main",args);
args.clear();
args.add(6);
exec.execute("main",args);
args.clear();
args.add(7);
exec.execute("main",args);
}
}
void loop() {
// put your main code here, to run repeatedly:
}result:
***********PARSING DONE*********
***********COMPILING DONE*********
max used memory: 8468 maxstack:2304 started 265896 free mem:257328 consumed 8568 time:33ms
max used memory: 8468 maxstack:2304 started 265896 free mem:262196 consumed 3700 time:44ms
***********AFTER CLEAN*********
***********CREATE EXECUTABLE*********
Creation of an 272 bytes binary and 90 bytes data
Parsing 98 assembly lines ...
max used memory: 8468 maxstack:2304 started 265896 free mem:264808 consumed 1088 time:121ms
factorial of 5 is 120
factorial of 6 is 720
factorial of 7 is 5040
With the ESPScript is not able to code everything with the same efficiency as the espressif compiler plus it doesn't gfive you accès to WiFi, bluetooth, SPI, I2C, ... Futhermore, it will not be concievable to rewrite functions like the one the the FastLED library or any other library. Hence the ESPScript can call pre-compiled functions. In other case you can need to access a 'precompile' variable which is changed by another process for instance.
You need in your sketch that your variable needs to be accessible from the scripts:
addExternal("name_of_the_variable_int_the_script", externalType::value, (void *)&address_to_the_variable);
In your script you need to declare your variable as external:
external type name_of_the_variable_int_the_script;
You read and write the variables. NB: a variable can also be an array.
Example:
#include "ESPLiveScript.h"
string script="\
external int value;\n\
external uint16_t *array;\n\
//external uint16_t array[10]; is the same \n\
void fillArray()\n\
{\n\
for(int i=0;i<10;i++)\n\
{\n\
array[i]=i*3;\n\
}\n\
}\n\
void change()\n\
{\n\
value=value+2;\n\
}\n\
void main()\n\
{ \n\
printfln(\"value: %d \",value);\n\
}";
int variable=0;
uint16_t _array[10];
void setup() {
// put your setup code here, to run once:
Serial.begin(115200);
Parser p;
addExternal("value", externalType::value, (void *)&variable);
addExternal("array", externalType::value, (void *)_array);
Executable exec=p.parseScript(&script);
if(exec.isExeExists())
{
variable=5;
exec.execute("main");
variable=240;
exec.execute("main");
variable=15;
printf("old value:%d ",variable);
exec.execute("change");
printf("new value:%d\n",variable);
exec.execute("fillArray");
for (int i=0;i<10;i++)
{
printf("%d:%d\n",i,_array[i]);
}
}
}
void loop() {
// put your main code here, to run repeatedly:
}NB: here we did call three different functions all defined in the script.
Result:
***********PARSING DONE*********
***********COMPILING DONE*********
max used memory: 9172 maxstack:2112 started 265376 free mem:256100 consumed 9276 time:40ms
max used memory: 9172 maxstack:2112 started 265376 free mem:261868 consumed 3508 time:52ms
***********AFTER CLEAN*********
***********CREATE EXECUTABLE*********
Creation of an 264 bytes binary and 75 bytes data
Parsing 99 assembly lines ...
max used memory: 9172 maxstack:2112 started 265376 free mem:264168 consumed 1208 time:129ms
value: 5
value: 240
old value:15 new value:17
0:0
1:3
2:6
3:9
4:12
5:15
6:18
7:21
8:24
9:27
You can call 'core' functions which would be to complicated to reproduced in scripting (like fft , showing leds ...)
addExternal("funtion name", externalType::function, (void *)function);
In your script you need to declare your variable as external:
external type function_name;
Example
#include "ESPLiveScript.h"
string script="\
external float calc(int h);\n\
external void displayfloat(float nb);\n\
void main()\n\
{\n\
float h=calc(52);\n\
displayfloat(h);\n\
}";
void displayfloat(float nb)
{
printf("from pre-compiled %f\n",nb);
}
float calcul(int pos)
{
return (float)(pos/34.0);
}
void setup() {
// put your setup code here, to run once:
Serial.begin(115200);
addExternal("calc", externalType::function, (void *)calcul);
addExternal("displayfloat", externalType::function, (void *)displayfloat);
Parser p;
Executable exec=p.parseScript(&script);
if(exec.isExeExists())
{
exec.execute("main");
}
}
void loop() {
// put your main code here, to run repeatedly:
}Result:
***********PARSING DONE*********
***********COMPILING DONE*********
max used memory: 7572 maxstack:1964 started 265984 free mem:258484 consumed 7500 time:28ms
max used memory: 7572 maxstack:1964 started 265984 free mem:263600 consumed 2384 time:40ms
***********AFTER CLEAN*********
***********CREATE EXECUTABLE*********
Creation of an 160 bytes binary and 56 bytes data
Parsing 59 assembly lines ...
max used memory: 7572 maxstack:1964 started 265984 free mem:265124 consumed 860 time:92ms
from pre-compiled 1.529412
Let's consider the following Use case:
In your c++ code
uint16_t _array[10];
addExternal("array", externalType::value, (void *)_array);and the script
external uint16_t *array; //or external uint16_t array[10];
void main()
{
for(int i=0;i<200;i++)
{
array[i]=200;
}
}the script will write out of bounds of the array with unintended consequences.
To avoid this you can use the safemode:
safe_mode
external uint16_t array[10];
void main()
{
for(int i=0;i<200;i++)
{
array[i]=200;
}
}you will get
***********PARSING DONE*********
***********COMPILING DONE*********
max used memory: 7988 maxstack:2112 started 265528 free mem:257492 consumed 8036 time:32ms
max used memory: 7988 maxstack:2112 started 265528 free mem:262424 consumed 3104 time:44ms
***********AFTER CLEAN*********
***********CREATE EXECUTABLE*********
Creation of an 232 bytes binary and 60 bytes data
Parsing 82 assembly lines ...
max used memory: 7988 maxstack:2112 started 265528 free mem:264784 consumed 744 time:111ms
Overflow error line 0 max size: 10 got 11
NB: As the check will be done everytime a write is done then it will slow the script down.
Here are the default types:
uint8_tcharint: be careful it's int over 2 bytesuint16_tuint32_tfloatCRGBCRGBW
You can define new types call struct
example:
struct new_type
{
float k;
int l;
}The structures can have methods
struct new_type
{
float h;
int l;
void display()
{
printf("l :%d\n",l);
}
}NB: not like in a class you do not have constructor or destructor (at least not yet :) ).
Example:
#include "ESPLiveScript.h"
string script="\
struct new_type\n\
{\n\
float f;\n\
int index;\n\
void display(int multi)\n\
{\n\
printfln(\"from structure:%d\",multi);\n\
}\n\
void func2() \n\
{\n\
display(f*index);\n\
}\n\
}\n\
\n\
new_type var;\n\
void main()\n\
{\n\
var.f=0.8;\n\
var.index=12;\n\
var.func2();\n\
var.display(23);\n\
}";
void setup() {
// put your setup code here, to run once:
Serial.begin(115200);
Parser p;
Executable exec=p.parseScript(&script);
if(exec.isExeExists())
{
exec.execute("main");
}
}
void loop() {
// put your main code here, to run repeatedly:
}Result:
***********PARSING DONE*********
***********COMPILING DONE*********
max used memory: 11060 maxstack:1836 started 265840 free mem:254916 consumed 10924 time:40ms
max used memory: 11060 maxstack:1836 started 265840 free mem:260924 consumed 4916 time:52ms
***********AFTER CLEAN*********
***********CREATE EXECUTABLE*********
Creation of an 312 bytes binary and 104 bytes data
Parsing 114 assembly lines ...
max used memory: 11060 maxstack:1836 started 265840 free mem:263380 consumed 2460 time:147ms
from structure:9
from structure:23
NB 1: you can have arrays of structure : new_type arr[10]
NB 2: the functions needs to be in order i.e you cannot all a function which has not be defined previously. (for the moment)
NB 3: Due to some memory alignment concern, for the moment you need to order the variable by order of size. Here is the order of size:
float, uint32_tint, uint16_tuint8_t, CRGB, CRGBW
i.e :
struct varname
{
char c;
uint16_t k;
float h;
}
will not work. you need to arrange your data as such
struct varname
{
float h;
uint16_t k;
char c;
}
Like any normal language you can have:
- loops (
while,for) break,continue- testing :
if,else ++: for integers and pointers- pointers
^for power>>and<<- type convertion
(float),(int)NB: the convertion float <-> int is automatic - arrays (multidimensional arrays are work in progress)
define: see below- predefined functions:
printf,pritnflnonly for intergers for now.fabs,absother to come
More or less like in C you can have pre-process instructions with a limited scope for now
define TOKEN 25
if(i<TOKEN)
{
...
}
vill be compiled as
if(i<25)
{
...
}
Of course you can have this
define p printf
...
p("here %d\n",25);
NB: there is no macro yet
For the moment you cannot write combination of test in the same if:
if( a<b and c>d) is not possible yet. You will have to have the tests in two separate if
You have the possibility of running scripts as task in the background (interesting when you want several tasks at once):
exec.executeAsTask("function_name")exec.executeAsTask("function_name",arguments)exec.executeAsTask("function_name", __RUN_CORE, args)
When using this you can also do the following
exec.suspend()bool exec.isRunning()exec.estart()exec.kill(): will not delete the binary hence will not free the memory useexec.free()
When you deal with several binaries, it could get complicated to follow a lot of different variables. You can 'register' your executable in the Script runtime:
scriptRuntime.addExe(executable exec,string exename)
string script1="...";
string scrip2="...";
Executable exec=p.parseScript(&script1);
if(exec.isExeExists())
{
scriptRuntime.addExe(exec,"script1");
}
exec=p.parseScript(&script2);
if(exec.isExeExists())
{
scriptRuntime.addExe(exec,"script2");
}
...
scriptRuntime.execute("script1");
scriptRuntime.execute("script2");Here are all the functions of the scriptRuntime:
scriptRuntime.addExe(executable exec,string exename)scriptRuntime.excute(string execname): the runtime will execute the "main" function by defaultscriptRuntime.excute(string execname,string function_name)scriptRuntime.excute(string execname,Arguments args):: the runtime will execute the "main" function by defaultscriptRuntime.excute(string execname,Arguments args,string function_name)scriptRuntime.excuteAsTask(string execname): : the runtime will execute the "main" function by defaultscriptRuntime.excuteAsTask(string execname,string function_name)scriptRuntime.excuteAsTask(string execname,Arguments args): the runtime will execute the "main" function by defaultscriptRuntime.excuteAsTask(string execname,Arguments args,string function_name)scriptRuntime.excuteAsTask(string execname,Arguments args): the runtime will execute the "main" function by defaultscriptRuntime.excuteAsTask(string execname,int core,Arguments args): the runtime will execute the "main" function by defaultscriptRuntime.kill(string execname): this will not free the memoryscriptRuntime.deleteExe(string execname): will free the memory of the binaryExecutable *findExecutable(string execname)vector<exe_info> scriptRuntime.getListExecutables(): will list all the executables:exe_info getExecutableInfo(string execname)- exe_info:
uint16_t data_sizeuint16_t binary_sizeuint16_t total_sizebool isRunningstring name
- exe_info:
As discussed earlier this scripting language aims at being used primarily for leds animations. when you have several scripts running at the same time it is important to synchronize the script to avoid artifacts:
Let's say that you need to synchronize the show function which is an 'pre-compiled' function. in your C/C++ program add this:
void show()
{
...
driver.showPixel();
...
}
string script1="
...
while(2>1)
{
rainbow();
sync(); //put this instead of show();
}";
string script2="
...
while(2>1)
{
gameoflife();
sync(); //put this instead of show();
}";
void setup()
{
runningPrograms.setFunctionToSync(show);
Parser p;
scriptRuntime.addExe(p.parseScript(&script1,"exe1"));
scriptRuntime.addExe(p.parseScript(&script2,"exe2"));
scriptRuntime.executeAsTask("exe1");
scriptRuntime.executeAsTask("exe2");
}When you kill a task you cannot control where exactly in the program it will be stopped. As a consequence it could be stopped when calling an external function with interupts or calling another background task.
For instance in the case of the I2SClockLessDriver and the I2SVirtualClocklessDriver it is necessary to stop the driver from displaying anything before killing the script.
this is done as such :
void pre() {
driver.__enableDriver = false;
while (driver.isDisplaying) {};
}
void post() {
driver.__enableDriver = true;
}
..
void setup()
{
...
runningPrograms.setPrekill(pre, post);
...
}This is my first try at creating such a thing. You can see a video of me talking about it live on youtube where I use the compiler with ledOS.
Please have a look at Starlight
I will add an issue for all the functionalities you want to see added in the; compiler.
As always enjoy and have fun.