Executor thread not set after <some value> ms - intermittently

[pradeepKaatnam]

Hey

I am getting into this error Executor thread not set after ms intermittently. I have gone through the code and I came up with a theory where this can happen. This happens when threadToMonitor variable is null in run() method of ThreadMonitor. The reason can be below:

final JsEvaluator evaluator = getEvaluator(op);
executor.execute(evaluator);
evaluator.runMonitor();

Lets say the executor is created with a max pool size of 5 and there are some 100 incoming requests. Here is there is a chance for some requests to be waiting in the queue of executor. Since it is asynchronous, evaluator's runMonitor() method will be called irrespective of whether its corresponding thread is initiated or not. There is wait time of maxCPUTime / MILI_TO_NANO for the threadToMonitor to be set by the executor api. It might be because previously long running threads will get terminated after their maxCPUTime because of threadToMonitor.interrupt(). Here if the thread doesn't goes down after interrupt then it is waiting for 50 more milli seconds and doing a hard shutdown().

So in the case of hard shutdown there is chance of this exception as it is waiting for only maxCPUTime and not for maxCPUTime + 50(the extra time taken for hard shutdown)

synchronized (monitor) {
    if (threadToMonitor == null) {
	monitor.wait(maxCPUTime / MILI_TO_NANO);
    }
}
if (threadToMonitor == null) {
	throw new IllegalStateException("Executor thread not set after " + maxCPUTime / MILI_TO_NANO + " ms");
}

To conclude, in the run() method of ThreadMonitor class, there should be some additional buffer time i.e.

void run() {
    try {
	// wait, for threadToMonitor to be set in JS evaluator thread
	synchronized (monitor) {
		if (threadToMonitor == null) {
			monitor.wait((maxCPUTime + 50) / MILI_TO_NANO);
		}
	}
	if (threadToMonitor == null) {
		throw new IllegalStateException("Executor thread not set after " + maxCPUTime / MILI_TO_NANO + " ms");
	}

Let me know your thoughts.

[srinivasarajuch]

@mxro This is possible issue, I can raise PR for this. Please let me know you thoughts

[mxro]

Thank you for the PR! Code is merged in released with version 0.1.21 which should be available via Maven Central shortly.