Define the resources.json format from @ResourceIdentifiers and @pragma('dart2js:resource-identifier')

[dcharkes]

Last year the concept of resource identifiers was introduced.

Currently, there are some inconsistencies between the format output by the VM and by dart2js.

• dart2js does not output an id (The ResourceIdentifier.metadata field in the package:meta annotation.)
• nonconstant vs nonConstant spelling. (Shouldn't a bool just be constant? Why use the negation?)

The current serialization is centered around loading units, but not all use cases might require loading units.

The current serialization contains source locations and loading units unconditionally, but this is fragile if the format is used for caching. So we should consider making these optional, depending on use case.

The format should be versioned.

A sketch of what the serialization could look like when addressing these issues:

{
  "_comment": "Resources referenced by annotated resource identifers",
  "_version": [
    1,
    0,
    0
  ],
  "identifiers": [
    {
      "uri": "org-dartlang-sdk:///lib/_internal/js_runtime/lib/js_helper.dart",
      "containerType": "class",
      "container": "MyClass",
      "name": "loadDeferredLibrary",
      "constant": true,
      "references": [
        {
          "arguments": {
            "1": "lib_SHA1",
            "2": 0
          },
          "loadingUnit": "o.js",
          "@": {
            "uri": "file://benchmarks/OmnibusDeferred/dart/OmnibusDeferred.dart",
            "line": 14,
            "column": 48
          }
        }
      ]
    }
  ]
}

@rakudrama I'd like to understand what the requirements are from dart2js. Then we can figure out if the requirements are compatible with the on for link hooks, so that we can settle on a single format. (Hopefully yes, if we make some things optional. I believe dart2js explicitly wants to use loading units for example.)

I have done a long write-up of the requirements for the resources.json in link hooks here:

• [native_assets_cli] link hook resources.json format native#1093

cc @mosuem @mkustermann @natebosch Feel free to chime in!
cc @johnniwinther Having a concept of fully qualified (unique) names would help us to avoid uri-containerType-container-name, but I believe we have no such thing in Dart?

[johnniwinther]

We don't have fully qualified name but be somewhat created using the import uri (aka canonical uri) of the library and the qualified name from within the library (taking getter/setter into account) similar to how we create canonical names for serialization.

[dcharkes]

Thanks @rakudrama for the good chat!

Some notes:

• We have one user of this format via dart2js, Including the loading units: ACX. The format was implemented to serve that use case and wasn't implemented further. We can do breaking changes to the format as long as we also migrate this use.
• The source URI/line/column is to be able to give useful error messages when consuming the file. E.g. from hook/link.dart in our new use case.
  • For caching, we could consider using a projection from the format that hides the source information to avoid running link hooks again.
• notconstant is an optimization so that not all references have to be read.
• Adding a version to the format is a good idea.
• Adding container (class/extension) and container type would be non breaking changes.
• So would adding support for get:xxx and set:xxx (where getters would have no parameters, so they could only be used for saying a resource is used at all yes or no).

[mosuem]

After a discussion offline with @dcharkes, we came up with:

A recordedReference consists of

• the annotation(s) causing the reference to be recorded, together with their fields, to make them extensible Make it possible to implement ResourceIdentifier #55568.
• the definitions, uniquely identified by URI, parent, and name.
  • The name can be get:xxx or set:xxx.
• the references with their arguments.
• The arguments have explicit positional and named arguments, and explicit const and non const argument values.
• Now that optional non-passed parameters are distinguishable from non const parameters, a nonconstant field is no longer needed.
• The loading units are a property of definitions and references now.
• The definitions now also have source positions.
• The hashes can be used for more efficient caching. (Source positions and loading units might not warrant reruns for some use cases.)

{
    "metadata": {
        "comment": "Recorded references at compile time and their argument values, as far as known, to definitions annotated with @RecordReference",
        "version": "1.6.2-wip+5.-.2.z",
        "timestamp": 321432153,
        "hashes": {
            "noPositions": "dasdsadfasfwagwraf",
            "noPositionsNoLoadingUnits": "fdsfdsafdsagh"
        }
    },
    "staticFunctionReferences": [
        {
            "annotations": [
                {
                    "identifier": {
                        "uri": "file://lib/_internal/js_runtime/lib/js_helper.dart",
                        "name": "RecordReference"
                    },
                    "fields": {
                        "_metadata": {
                            "key": true
                        }
                    }
                }
            ],
            "definition": {
                "identifier": {
                    "uri": "file://lib/_internal/js_runtime/lib/js_helper.dart",
                    "parent": "MyClass",
                    "name": "get:loadDeferredLibrary"
                },
                "@": {
                    "uri": "file://lib/_internal/js_runtime/lib/js_helper.dart",
                    "line": 12,
                    "column": 67
                },
                "loadingUnit": "part_15.js"
            },
            "references": [
                {
                    "arguments": {
                        "const": {
                            "positional": {
                                "0": "lib_SHA1",
                                "1": false,
                                "2": 1
                            },
                            "named": {
                                "leroy": "jenkins",
                                "freddy": "mercury"
                            }
                        },
                        "nonConst": {
                            "positional": [],
                            "named": []
                        }
                    },
                    "loadingUnit": "o.js",
                    "@": {
                        "uri": "file://benchmarks/OmnibusDeferred/dart/OmnibusDeferred.dart",
                        "line": 14,
                        "column": 48
                    }
                },
                {
                    "arguments": {
                        "const": {
                            "positional": {
                                "0": "lib_SHA1",
                                "2": 0
                            },
                            "named": {
                                "leroy": "jenkins"
                            }
                        },
                        "nonConst": {
                            "positional": [
                                "1"
                            ],
                            "named": [
                                "freddy"
                            ]
                        }
                    },
                    "loadingUnit": "o.js",
                    "@": {
                        "uri": "file://benchmarks/OmnibusDeferred/dart/OmnibusDeferred.dart",
                        "line": 14,
                        "column": 48
                    }
                }
            ]
        }
    ]
}

Any thoughts or input @rakudrama?

[dcharkes]

@mkustermann Asked if there is something we could do to make this format smaller.

• We could for example add a URI-map to avoid repeating the same URI multiple times.
• We could add an identifier-map to avoid repeating the same URI+identifier string multiple times.

The question is whether we should optimize for size. This file will only live on host machines, it's not part of a compiled app bundle. Also, since the contents of this format are filtered on the annotations, maybe they won't grow so big.

(If we decide we want to tackle size later, we'd have to go with uri_v2 a la dart-lang/native#93 (comment), and afterwards remove uri.)

@mkustermann You mentioned some other AOT treeshaking format in the SDK that is size optimized. Do you have a reference to it?

[dcharkes]

RE: hashes:

We are looking into more sophisticated caching schemes for depending on part of the resources.json (dart-lang/native#1178). So it could make sense to either

1. remove the hashes and have the consumer (native_assets_builder) rehash for what it wants to do caching on, or
2. have hashes per annotation class (which is what the native_assets_builder would use most likely).