Add recursive call limit

[hackaugusto]

Currently the VM uses recursion to interpret procedure calls, which means an attacker can easily write a program that will exhaust the host stack, killing it.

POC: A dyncall instruction in MASM causes the Rust program to recurse. The following program can be used to create infinite recursion:

proc.rec
  dynexec
end

begin
  procref.rec
  exec.rec
end

Running the above causes a crash:

thread 'main' has overflowed its stack
fatal runtime error: stack overflow
[1]    99618 abort      cargo run --features="executable" -- run --assembly t.masm

[hackaugusto]

Another approach is to change the way the bytecode is interpreted, and rely exclusively on the max cycle counter to prevent running out of main memory. This is related to #1217

[cyberbono3]

Can I start working on it?

[hackaugusto]

Can I start working on it?

@bobbinth @bitwalker ping. I'm not sure what are the plans with the interpreter rewrite to remove the recursion calls.

[bobbinth]

@cyberbono3 - thank you for wanting to help with this! I would probably hold off on this issue for now as we will be refactoring how the processor runs though the program in the next couple of weeks (the Program struct will be moved to table-based description of MAST - see #1226). So, the mechanism that may apply to the current implementation may need to be different after the refactor.