Cannot capture continuation from JavaScript code not called directly by executeScriptWithContinuations or callFunctionWithContinuations

[821938089]

I've used executeScriptWithContinuations to call the js code, but it still throws this error.
In testing, I found that continuation cannot be captured in some cases.

function test1 () {
// call a function that capture continuation
}
test1() // success
test1.call() // success
test1.apply(this) // success
(function test1 () {
// call a function that capture continuation
})(); // success

test1.bind(this)() // fail
test1.bind(this).call() // fail

test2 = () => {
// call a function that capture continuation
}
test2() // fail
test2.call() // fail
(() => {
// call a function that capture continuation
})(); // fail


eval('// call a function that capture continuation') // fail

eval(`
function test1 () {
// call a function that capture continuation
}
test1()
`) // fail

eval(`
function test1 () {
// call a function that capture continuation
}
`)() // success

eval(`
function test1 () {
// call a function that capture continuation
}
`).call() // success

eval(`
function test1 () {
// call a function that capture continuation
}
`).apply(this) // success

eval(`
function test1 () {
// call a function that capture continuation
}
`).bind(this)() // fail

eval(`
function test1 () {
// call a function that capture continuation
}
`).bind(this).call() // fail

example code that list fail case:

import org.mozilla.javascript.*;

public class Main {

  public static class MyObj {
    public void capture() {
      try (var cx = Context.enter()) {
        throw cx.captureContinuation();
      }
    }
  }

  static Scriptable globalScope;

  public static void main(String[] args) {
    try (var cx = Context.enter()) {
      globalScope = cx.initStandardObjects();
      globalScope.put("myObj", globalScope, Context.javaToJS(new MyObj(), globalScope));
      cx.setOptimizationLevel(-1);

      System.out.println("capture.bind(this)()");
      executeScript(cx, "function capture(){myObj.capture()};capture.bind(this)()");

      System.out.println("capture.bind(this).call()");
      executeScript(cx, "function capture(){myObj.capture()};capture.bind(this).call()");

      System.out.println("[arrow function] capture()");
      executeScript(cx, "capture=()=>{myObj.capture()};capture()");

      System.out.println("[arrow function] capture.call()");
      executeScript(cx, "capture=()=>{myObj.capture()};capture.call()");

      System.out.println("[arrow function] immediate call");
      executeScript(cx, "(()=>{myObj.capture()})();");

      System.out.println("[eval] capture()");
      executeScript(cx, "function capture(){myObj.capture()};eval('capture()')");

    }
  }

  public static void executeScript(Context cx, String script) {
    Script s = cx.compileString(script, "unknown source", 0, null);
    try {
      cx.executeScriptWithContinuations(s, globalScope);
    } catch (ContinuationPending pending) {
      System.out.println("captured!");
    } catch (WrappedException e) {
      System.out.println("capture failed!");
    }
  }

}

The reason for this failure is that the call method of the InterpretedFunction class does not inherit from the previous CallFrame, causing outermost.isContinuationsTopFrame to be false.

rhino/rhino/src/main/java/org/mozilla/javascript/InterpretedFunction.java

Lines 83 to 88 in f48a40d

	public Object call(Context cx, Scriptable scope, Scriptable thisObj, Object[] args) {
	if (!ScriptRuntime.hasTopCall(cx)) {
	return ScriptRuntime.doTopCall(this, cx, scope, thisObj, args, idata.isStrict);
	}
	return Interpreter.interpret(this, cx, scope, thisObj, args);
	}

rhino/rhino/src/main/java/org/mozilla/javascript/Interpreter.java

Lines 1074 to 1076 in f48a40d

	CallFrame frame = initFrame(cx, scope, thisObj, args, null, 0, args.length, ifun, null);
	frame.isContinuationsTopFrame = cx.isContinuationsTopCall;
	cx.isContinuationsTopCall = false;

rhino/rhino/src/main/java/org/mozilla/javascript/Interpreter.java

Lines 3477 to 3487 in f48a40d

	if (requireContinuationsTopFrame) {
	while (outermost.parentFrame != null) outermost = outermost.parentFrame;
	if (!outermost.isContinuationsTopFrame) {
	throw new IllegalStateException(
	"Cannot capture continuation "
	+ "from JavaScript code not called directly by "
	+ "executeScriptWithContinuations or "
	+ "callFunctionWithContinuations");
	}
	}

[p-bakker]

Don't think this is actionable as it currently stand.

In order to make it actionable, this issue would have to include a runnable (java) sample, that demonstrates the issue and some analysis of the likely reason of the issue and whether the reason it fails in several scenarios is likely the same or not

[821938089]

@p-bakker Hello, I have updated the comment as per your requirements and I hope this information is useful to you.

[p-bakker]

Thnx, that indeed provides a better starting point to understand the issue!

I doubt there are many devs around that have intricate knowledge of the inner workings of continuations, why they fail in these scenarios, if they can be supported in these scenarios and what is needed to do so... 😐

But who knows... @szegedi: any thoughts maybe?

Otherwise if you have ideas on how to make this work, a PR is always welcome

There's also case #1475, which is a different scenario in which capturing a Continuation fails

[szegedi]

👋 Continuations work as long as all of the execution is contained within a single invocation of Interpreter.interpretLoop(). The loop maintains its internal stack for functions, and as long as everything in the call chain is an InterpretedFunction, it is supposed to work. Reading the code shows that there's already special handling for .call() and .apply() so that's why those work.

Bound functions aren't handled specially by the interpreter loop – they could be, though. Just as the special handling for .call() and .apply() will check if the function they're called on is an interpreted function, the interpreter loop could also check for whether the bound function's underlying function is an interpreted function and install it on its stack directly. This of course leads to a bit of extra complexity.

Same for arrow functions – I was actually surprised these didn't work out of the box, but I just looked at the code now as I didn't really work much on Rhino since they were introduced and I can see that lambdas are implemented as ArrowFunction wrappers around another Callable, and not directly as an interpreted function. We'd need few more "if"s in the call logic in the interpreter and most likely new initFrameForBoundFunction, initFrameForArrowFunction methods too.

As these things can of course get combined (e.g. you can call apply() on a bound arrow function) to implement this properly, we'd need a generic mechanism for peeling the invoked function in the interpreter back to the interpreted function it ultimately reduces to, and if it does, then walk the chain of functions one more time, invoke all requisite initFrameForXxx methods and finally install the function on the stack.

It might actually be simpler to write the code to mark the stack position, then opportunistically start executing all this with the assumption the ultimate function will be an interpreted one, and when it isn't, pop the stack back to the marked position and bail out of the interpreter by invoking fun.call() as it is now on this line.

[p-bakker]

@szegedi Thx a lot for the analysis & input!

@821938089 Think you can roll with all this info and maybe put together a PR?

[821938089]

@p-bakker No, I don't know is what "opportunistically start executing all this" means in the last paragraph.

And I don't know how the interpreter works and I can't be sure if the code I wrote is correct.

[szegedi]

I might take a stab at it, it looks like a rather fun task.

[p-bakker]

Great @szegedi !

Maybe I'm pushing the envelope a bit, but while at it, could you then also have a look if you see a way to support the usecase described in #1475, which tries to capture a Continuation from Java, but not a Java method called from scripting?

[szegedi]

PR ready for review!

[821938089]

Does this pr work in eval? I see the test doesn't have anything about this.

[szegedi]

• eval('function () {c.run()}').bind(this).call() will work as you'll call the capturing function within a continuation-capturing execution (outside of eval).
• var f = function() {c.run()}; eval('f()') will not work as you'd call the capturing function within a non-capturing execution (inside of eval)

(I used the c.run() to represent the capturing invocation, as in https://github.com/mozilla/rhino/blob/master/tests/src/test/java/org/mozilla/javascript/tests/InterpreterFunctionPeelingTest.java)

[821938089]

Is it possible to make continuations work in eval?
What it ends up executing is also an InterpreterFunction.

I looked at the source code and found that specific function names can also cause continuations to fail.
This is so surprising.

      System.out.println("eval()");
      executeScript(cx, "function eval(){myObj.capture()};eval()");

      System.out.println("With()");
      executeScript(cx, "function With(){myObj.capture()};With()");

[szegedi]

Having some names work differently is curious – where in the source code did you look?

And while the functions invoked in eval are interpreted functions, eval itself is not an interpreted function, so when you call eval, you'll step out of the current interpreter loop, hence functions won't work. You'll have on stack:

interpreter loop
eval
another interpreter loop

I don't expect this will change. At least I'm definitely not inclined to put in the work to make this additional corner case work.

[821938089]

I think it's possible to handle eval in the same interpreter loop for all cases:

---

What I've found here is that a function with the name eval or With is assumed to be a special call, and there is a special branch in the interpreter that takes care of it, so it won't jump out of the interpreter loop in that case.

I would expect the continuation to work in this case.

In doCallSpecial the call object is checked to see if it is expected, otherwise it falls back to a normal call, but this normal call will not run in the same interpreter.

rhino/rhino/src/main/java/org/mozilla/javascript/IRFactory.java

Lines 2069 to 2091 in d3ebdc5

	private Node createCallOrNew(int nodeType, Node child) {
	int type = Node.NON_SPECIALCALL;
	if (child.getType() == Token.NAME) {
	String name = child.getString();
	if (name.equals("eval")) {
	type = Node.SPECIALCALL_EVAL;
	} else if (name.equals("With")) {
	type = Node.SPECIALCALL_WITH;
	}
	} else if (child.getType() == Token.GETPROP) {
	String name = child.getLastChild().getString();
	if (name.equals("eval")) {
	type = Node.SPECIALCALL_EVAL;
	}
	}
	Node node = new Node(nodeType, child);
	if (type != Node.NON_SPECIALCALL) {
	// Calls to these functions require activation objects.
	parser.setRequiresActivation();
	node.putIntProp(Node.SPECIALCALL_PROP, type);
	}
	return node;
	}

rhino/rhino/src/main/java/org/mozilla/javascript/CodeGenerator.java

Lines 603 to 649 in 3581b02

	case Token.REF_CALL:
	case Token.CALL:
	case Token.NEW:
	{
	if (type == Token.NEW) {
	visitExpression(child, 0);
	} else {
	generateCallFunAndThis(child);
	}
	int argCount = 0;
	while ((child = child.getNext()) != null) {
	visitExpression(child, 0);
	++argCount;
	}
	int callType = node.getIntProp(Node.SPECIALCALL_PROP, Node.NON_SPECIALCALL);
	if (type != Token.REF_CALL && callType != Node.NON_SPECIALCALL) {
	// embed line number and source filename
	addIndexOp(Icode_CALLSPECIAL, argCount);
	addUint8(callType);
	addUint8(type == Token.NEW ? 1 : 0);
	addUint16(lineNumber & 0xFFFF);
	} else {
	// Only use the tail call optimization if we're not in a try
	// or we're not generating debug info (since the
	// optimization will confuse the debugger)
	if (type == Token.CALL
	&& (contextFlags & ECF_TAIL) != 0
	&& !compilerEnv.isGenerateDebugInfo()
	&& !itsInTryFlag) {
	type = Icode_TAIL_CALL;
	}
	addIndexOp(type, argCount);
	}
	// adjust stack
	if (type == Token.NEW) {
	// new: f, args -> result
	stackChange(-argCount);
	} else {
	// call: f, thisObj, args -> result
	// ref_call: f, thisObj, args -> ref
	stackChange(-1 - argCount);
	}
	if (argCount > itsData.itsMaxCalleeArgs) {
	itsData.itsMaxCalleeArgs = argCount;
	}
	}
	break;

rhino/rhino/src/main/java/org/mozilla/javascript/Interpreter.java

Lines 1737 to 1747 in 3581b02

	case Icode_CALLSPECIAL:
	{
	if (instructionCounting) {
	cx.instructionCount += INVOCATION_COST;
	}
	stackTop =
	doCallSpecial(
	cx, frame, stack, sDbl, stackTop, iCode,
	indexReg);
	continue Loop;
	}

rhino/rhino/src/main/java/org/mozilla/javascript/Interpreter.java

Lines 2919 to 2964 in 3581b02

	private static int doCallSpecial(
	Context cx,
	CallFrame frame,
	Object[] stack,
	double[] sDbl,
	int stackTop,
	byte[] iCode,
	int indexReg) {
	int callType = iCode[frame.pc] & 0xFF;
	boolean isNew = (iCode[frame.pc + 1] != 0);
	int sourceLine = getIndex(iCode, frame.pc + 2);
	// indexReg: number of arguments
	if (isNew) {
	// stack change: function arg0 .. argN -> newResult
	stackTop -= indexReg;
	Object function = stack[stackTop];
	if (function == DOUBLE_MARK) function = ScriptRuntime.wrapNumber(sDbl[stackTop]);
	Object[] outArgs = getArgsArray(stack, sDbl, stackTop + 1, indexReg);
	stack[stackTop] =
	ScriptRuntime.newSpecial(cx, function, outArgs, frame.scope, callType);
	} else {
	// stack change: function thisObj arg0 .. argN -> result
	stackTop -= 1 + indexReg;
	// Call code generation ensure that stack here
	// is ... Callable Scriptable
	Scriptable functionThis = (Scriptable) stack[stackTop + 1];
	Callable function = (Callable) stack[stackTop];
	Object[] outArgs = getArgsArray(stack, sDbl, stackTop + 2, indexReg);
	stack[stackTop] =
	ScriptRuntime.callSpecial(
	cx,
	function,
	functionThis,
	outArgs,
	frame.scope,
	frame.thisObj,
	callType,
	frame.idata.itsSourceFile,
	sourceLine);
	}
	frame.pc += 4;
	return stackTop;
	}

rhino/rhino/src/main/java/org/mozilla/javascript/ScriptRuntime.java

Lines 2757 to 2780 in 3581b02

	public static Object callSpecial(
	Context cx,
	Callable fun,
	Scriptable thisObj,
	Object[] args,
	Scriptable scope,
	Scriptable callerThis,
	int callType,
	String filename,
	int lineNumber) {
	if (callType == Node.SPECIALCALL_EVAL) {
	if (thisObj.getParentScope() == null && NativeGlobal.isEvalFunction(fun)) {
	return evalSpecial(cx, scope, callerThis, args, filename, lineNumber);
	}
	} else if (callType == Node.SPECIALCALL_WITH) {
	if (NativeWith.isWithFunction(fun)) {
	throw Context.reportRuntimeErrorById("msg.only.from.new", "With");
	}
	} else {
	throw Kit.codeBug();
	}
	return fun.call(cx, scope, thisObj, args);
	}