Getting SharedWorkers to work

[tve]

I'd appreciate some help getting a SharedWorker to do something for me... I put together a basic example and am getting an error when creating the SharedWorker:

No Wi-Fi SSID
# Chunk allocation: 20 bytes failed in fixed size heap
XS abort: memory full
/home/sw/moddable/moddable/xs/platforms/esp/xsPlatform.c (237) # Break: C: xsDebugger!

main.ts:

import { SharedWorker } from "worker"

const minimumOptions = {
  allocation: 8192,
  stackCount: 64,
  slotCount: 64,
  keyCount: 7,
}
const imu = new SharedWorker("imuworker") // , minimumOptions)
if (!imu) {
  throw new Error("Failed to create SharedWorker")
}
;(imu as any).port.onmessage = function (e: any) {
  trace(`IMU: ${JSON.stringify(e.data)}\n`)
}
;(imu as any).port.postMessage("Hello from main.ts")

imuworker.ts:

;(globalThis as any).onconnect = function (e: any) {
  var port = e.ports[0]
  port.onmessage = function (e: any) {
    port.postMessage("You said: " + e.data)
  }
}

Full project:
repro-worker.tar.gz
Run on a moddable two: mcconfig -v -d -m -p esp32/moddable_two_io

Adding the minimumOptions makes no difference. I saw those in https://github.com/Moddable-OpenSource/moddable/blob/public/tests/modules/base/sharedworker/messages.js#L19 but they don't really make sense to me: doesn't a sharedworker use the same VM as main? I thought that was the point of a shared worker? (I guess there needs to be a separate stack?)

I would appreciate some help, both in getting this to work and in understanding a bit better what a shared worker in xs is. (I have never used a worker in JS, so the whole topic is new to me.)

As background, I'm trying to run a 100Hz interrupt handler type task in a shared worker to update IMU information and hope that will run time-sliced with poco.end. Dunno whether that's actually doable...

[phoddie]

I'd suggest starting with a regular worker. They are almost always enough and a bit simpler. From what you describe, that seems to be the case here.

There is an example of workers in this repository. That's a good place to start exploring.

Workers (and Shared Workers) in the Moddable SDK are documented.

...doesn't a sharedworker use the same VM as main?

No.

As background, I'm trying to run a 100Hz interrupt handler type task in a shared worker to update IMU information and hope that will run time-sliced with poco.end

Yes, it will. On an ESP32 those will run independently on separate cores. They can only run independently if they are separate VMs, naturally.

[tve]

Thanks for the pointer.
I had misunderstood what 'shared' refers to in SharedWorker, LOL!

[tve]

Can you clarify the memory allocation a little for me?
I tried a simple example, like the one you linked to (examples/base/worker), but without the creation spec in the manifest and it failed with # Slot allocation: failed in fixed size heap.
Which allocation where failed? Did the worker VM get created and it failed to allocate a slot when starting to run? Or did it not get created because xyz?
I believe the creation spec in the example manifest changes the "static" memory allocation from 0 to some value. The worker constructor options don't specify a "static" field: does the "static" field include all worker VMs as well? Or was the problem that perhaps on the moddable_two_io the default creation chunk+heap use up all memory? (I've reread the docs several times and looked at the default manifest and I find it not obvious to reverse engineer what the defaults mean in plain english).
Is the "allocation" in the worker options in bytes and an upper bound on chunks+stack+slots similar to "static" but contained within the main VM's "static" allocation?

[tve]

You took a working example and broke it? That doesn't look like a fruitful path.

Of course, wouldn't be me otherwise! :-) :-)

I wasn't clear in my question. I understand why the creation parameters for the worker VM are necessary. What I don't understand is why special creation parameters for the main VM are needed and how the two interact.
A couple of simple questions:

• is the "allocation" field in the worker options in bytes and a max for the worker's chunks+slots+stack? I.e. similar to the main VM's "static"?
• is the worker's VM allocated within the main VM's "static" or is it allocated out of non-VM system memory?
• do the worker's "chunk" and "slot" options support "initial"/"incremental" or only a fixed number?

[phoddie]

I understand why the creation parameters for the worker VM are necessary. What I don't understand is why special creation parameters for the main VM are needed and how the two interact.

"Special creation" is a mischaracterization. Each VM takes memory. These devices have limited memory. To make efficient use of memory each VM should allocate no more than it needs. Each VM performs a different set of functions, so each likely has different memory needs.

There is a single creation in the manifest. It is used to create the main VM. Worker VMs are created by the main VM. While they can inherit their creation from the main VM, this is generally suboptimal to fatal. So, don't do this unless you have tons of RAM. But even then, why waste it? So don't. Why is it allowed at all?.Because that's how the Worker API behaves on the web. The web has effectively infinite memory. A microcontroller does not.

When creating a worker, fully specify at least the allocation creation parameter. Here's the details:

• allocation is the static allocation size of the VM (like static in the manifest).
• slotCount is the initial number of slots in the slot heap. The number of bytes allocated is sizeof(xsSlot) * slotCount. Slots are 16 bytes on a 32-but MCU. This is allocated within the static allocation.
• stackCount is the number of slots in the stack. This is allocated within the static allocation.
• keyCount is less important these days as it can grow, so a small number is fine
• There is no chunk parameter because it is more-or-less equal to (allocation - slots - stack).

Because the initial slots and stack size on main may be pretty big and a worker may need something much smaller, always specify them rather than relying on default which can change out from under you. Then there are no surprises. So, specify allocation, slotCount, and stackCount. Like the example shows. And the documentation recommends.

[tve]

I'm not arguing over the need to specify the amt of memory. By "special creation" I simply meant non-default.

My worker ran out of stack. I increased the stack. Now XS resets in the Worker() constructor without any diagnostic printed anywhere. I'm trying to understand whether I need to increase or decrease the main VM's "static" to make room for the worker... Hence my questions...

Given the following statement in the manifest doc:

The static property also imposes a strict limit on the memory allocated by the language runtime to guarantee that scripts cannot exceed their memory budget (if they could, a script could take memory required by the host OS leading to failures and instabilities).

I assume that the worker's allocation must come out of the main VM's "static" allocation. If it doesn't, then a script can easily exceed the limit set by "static" by launching a worker with a large allocation. So to increase the worker's stack I probably also need to increase its "allocation" and then also the "static" of the main VM. But so far all I'm getting is silent resets.... Repro:
repro-worker.tar.gz

[tve]

After a bit more futzing around, I got the worker to cooperate :-). It does indeed time-slice nicely! Now I better optimize the driver 'cause reading a bunch of i2c registers at 100Hz isn't going to work very well...

I did discover two properties of the worker's allocation:

• the "allocation" bytes are allocated as a chunk when the work starts, it's not a "high water mark"
• the "allocation" bytes come out of the system heap, not the main VM's "static"

[phoddie]

Congratulation on getting it work.

the "allocation" bytes are allocated as a chunk when the work starts, it's not a "high water mark"

Correct. That's why it is also called the "static" allocation.

the "allocation" bytes come out of the system heap, not the main VM's "static"

Correct.