Brython riddle : how to create a REAL js array?

[ed2050]

This one has me stuck, wondering if anyone can help.

I need to create a real js array in my brython code (reasons below). Let's say blist is my brython list object. Here's what I tried :

array = window.Array.new ()          # then populate array with blist
array = window.Array.from (blist)    # syntax error
array = getattr (window.Array, 'from') (blist) 
array = window.Object.new (blist)

None of these work. The array object is always subtly different from a native js array. In particular, it has an attribute __class__ that native js arrays lack. This __class__ attribute causes my deserialization problems.

Does anyone have ideas for another way to make a js-native array in brython?? One without the __class__ attribute.

Demo

Here's the best way to see the difference. Go to brython console and enter:

from browser import window, console
x = window.Array.new ()
console.log (x)

Then open the js console and enter:

y = new Array ();
console.log (y)

You'll immediately see the difference in javascript console:

[__class__: {…}]      <-- js array from brython 
> y = new Array ()
> console.log (y)
[]      <-- real js array 

Why do this?

It's not important but if you're curious...

I'm passing messages between a web page and an extension. The extension returns data via callbacks. Data gets serialized with JSON.stringify when it's passed across the extension boundary. Brython lists don't deserialize correctly.

I tried creating a real js array instead. Native js arrays do deserialize correctly. I have a brython script sending messages with callbacks, and a brython listener in the extension sending responses. Response data (array of tabs) is provided by callback from chrome.tabs.query.

As long I don't alter the tabs array and just pass it back verbatim, everything works fine (more or less). However, I need to filter the array a bit and only return matching items. This is where I run into problems.

If I do tabs = [ t for t in tabs if condition (t) ] then tabs is a brython list and deserialization fails. So instead I tried making a real js array as above. But the resulting array is not quite the same as a real js array (see class element) and doesn't deserialize correctly.

[ed2050]

I have more observations, but first a question for @PierreQuentel:

Question

Is there an explicit way to convert a brython object to a js object? We have 'to_dict' for converting js objects to python objects. Can we manually do the reverse? The only way mentioned in the docs is automatic conversion when calling a js function.

I think my problem comes down to this: apparently brython doesn't evaluate nested objects when calling javascript functions. When a brython object is nested inside a js object, and js object is passed to a js function, brython doesn't do any type conversions. Is that accurate Pierre?

If so, something like this would not be converted:

win = window.Object ({ "foo": "bar" })   <-- js object
win.tabs = list ([ 1,2,3,4 ])            <-- brython object
console.log (win)  <-- js func and js object, no conversions done

Perhaps above example is too simple; it seems to produce correct results. But more complex objects in observations below seem to suggest this happens.

Read on if you want details...

Observations

In line with my post above, it seems the best way to make a js-compatible array is the obvious one: python list. A simple x = [] above produces the same result as y = new Array (); in the console log.

However that doesn't solve my original issue. There's hidden magic here. Per the docs, Brython converts python objects to js objects when calling js functions. Brython knows to convert a list to an Array when calling console.log. Docs say this:

If the Javascript object is a function, the arguments passed to the Python function are converted into Javascript objects, using the reverse of the above table.

Problem revised

Returning to my root problem (see original post), here's my current working theory. The problem is not the array of tabs itself. The problem is assigning any non-primitive object to a property of a js object in brython. My evidence is this:

• I'm given a js object win that has a property tabs as an array of tabs. The win object and its contents are provided by chrome.window.get. Caller provides a callback, now I have to return win to them like this: callback (win).
• callback (win) works fine if win and tabs are unaltered.
• callback (win) works fine if win is altered with a primitive (boolean, int, string) and an existing field. Setting win.tabs = 'foo' is ok.
• callback (win) does not work if I replace tabs array with a brython list. win.tabs = [ t for t in tabs if x ] causes callback to fail [1].
• callback (win) does not work if I replace tabs array with a js Array. win.tabs = browser.window.Array.new () causes callback to fail in same way.
• callback (win) does not work even if I modify win.tabs in place without assigning a new object. win.tabs.clear (); win.tabs.extend (newtabs) causes callback to fail in same way.
• the js method callback.win.tabs.filter does not work, even though win is a js object created outside brython. not sure why the brython methods in previous bullet work, but js methods are not available on win.tabs object. perhaps the dot access triggers brython to implicitly convert tabs? i don't know.
• when callback fails, rest of win object is fine. other values of win like id, height, state, etc are as expected. only tabs is mangled.
• in different calls, my extension returns a tabs array with no enclosing win object. those calls work fine, even if I modify tabs with a brython list. Eg 'newtabs = [ t for t in tabs if x ] ; callback (newtabs)` works fine. Callback gets correct data.

My Guess

Altering the win.tabs array in brython makes win.tabs into a brython list object. The structure is then a js object (win) containing a brython object (tabs).

When callback is called, the underlying function is a js function. Because chrome has to serialize the callback arg and send it to a different execution context (from extension background to unprivileged page script) to resume callback in original script. So from Brython's perspective, callback is a js function.

This explains why replacing tabs works when tabs array/list is returned directly, and fails when returning win which contains tabs:

• In first case, Brython sees a brython object (tabs list) and implicitly converts it to a js object (Array). Callback receives normal js Array.
• In second case, Brython sees a js object win and says: js function, js argument, oh no conversion needed. So Brython doesn't try to convert win, which means it doesn't convert tabs either. Then javascript serializer gets a win object containing a brython list object (tabs) and just serializes what it sees, which is the unholy mess in footnotes below.

Conclusion

My conclusion is: apparently brython doesn't evaluate nested objects when calling javascript functions. When a brython object is nested inside a js object, and js object is passed to a js function, brython doesn't do any type conversions.

Maybe brython's behavior is the right approach, I don't know. All I can say is that it's hidden. Because brython-to-js type conversions are handled implicitly, it's hard to figure out what's going on. Would be nice if this behavior was documented somewhere, and if manual conversions to js were possible (hence question to Pierre at the top).

Footnotes

1. Technically the callback succeeds, but win.tabs is mangled on receiving end. Instead of a simple list / array, tabs contains a deeply nested structure like this:

{
    ...
    "height": 800,
    "id": 12345678,
    "tabs": [
        {
            "__class__": {
                "__class__": {
                    "__mro__": [
                        {
                            "$infos": {
                                "__name__": "object",
                                "__module__": "builtins",
                                "__qualname__": "object"
                            },
                            "$is_class": true,
                            "$native": true,
                            "__mro__": [],
                            "__module__": "builtins",
                            "__bases__": [],
                            "$subclasses": [
                                {
                                    "__bases__": [
                                        null
                                    ],
                                    "__dict__": {
                                        "__class__": {
                                            "__mro__": [
                                                null
                                            ],
                                            "$infos": {
                                                "__module__": "builtins",
                                                "__name__": "dict",
                                                "__qualname__": "dict"
                                            },
                                            "$is_class": true,
                                            "$native": true,
                                            "__hash__": {
                                                "__class__": {
                                                    "$infos": {
                                                        "__name__": "NoneType",
                                                        "__module__": "builtins",
                                                        "__qualname__": "NoneType"
...