The Challenges of Sub-Elements ( [el , [tr , [td , etc.)

[tajmone]

There are some nodes that pose various challenges in the implementation of a PML syntax for editors, i.e. those nodes which are designed to be sub-nodes of another node — e.g. list element [el, which should occur inside a [list; and similarly all the [table sub-nodes ([tr, [tc, etc.).

Instinctively, it's tempting to implement them in such a way that [el nodes will only be recognized as valid elements when occurring inside a [list, and to scope as invalid any [el nodes occurring in the root context.

But the question begs: are lists nestable? What type of block nodes can a [list contain?

The documentation doesn't delve much into these details and, without a BNF grammar of the PML syntax, devs can only attempt trial and error cases to shed light on the issue. But whether lists are (or will eventually be) nestable can make a huge difference on how [list and [el should be implemented in an editor syntax.

Also, while it might at first seem reasonable to scope as invalid any element occurring out of its natural context, we need to remember that the use of [insert could result in PML snippet-files containing only lists elements ([el) or table inner element ([tr, [tc, etc.), which would then be scoped as invalid (i.g. invalid.out-of-context, or something like that). Is that what we really want?

The availability of the [insert node is one of the main reasons that make me reluctant in enforcing strict nodes-context validation in Sublime PML, because I can easily envision snippet files containing partial sub-elements of all kinds.

The lack of a BNF, and the fact that many elements could change in the future, are also factors which are holding me back from implementing some syntax elements which can be challenging — I want to avoid at all costs that future changes could break the ST syntax, because all these RegEx-based Stack states can too easily crumble to pieces if any element radically changes behaviour.

But a full knowledge of which specific node (block and inline alike) are permitted inside each node (especially block nodes) is critical for the syntax implementation. So far, I've been relying on trial and error testing using pmlc, but the possible edge cases to cover are really huge in number.

E.g. if we want to ensure that an [el node is marked as invalid when it occurs outside a list, then we need to capture it as invalid in the root context, and as valid within a [list context — which in turn means that the [list context needs to support (include) all the nodes that are eligible as it sub-elements, and mark as invalid those which aren't.

As already mentioned elsewhere, there's a major syntax rework awaiting here, because once all the nodes are implemented I'll need to revise the whole syntax to ensure that valid sub-nodes are matched via their explicit inclusion in each parent node, so that all other nodes can be matched as invalid.

Needless to say, it's always a bit scary to face a similar syntax rewrite, because things could break badly at any stage. Now, Sublime Text offers a powerful system to test syntax scopes, so the included PML test suite is a safety anchor to prevent total breakage. But many editors that adopt a similar scoping approach (a la TextMate, to be clear) don't provide similar tools — e.g. VSCode offers syntax debugging, but it's not the same thing, and doesn't allow for fine grain testing with a simple keystroke to auto-detect any breaking changes. So I'm no quite sure how the same syntax granularity could be achieved in other editors without loosing sight of the great picture and incurring in bugs.

[pml-lang]

are lists nestable?

Yes. And there is no limit as to the depth of nesting.

What type of block nodes can a [list contain?

There are currently no rules applied by the parser. If elements are nested incorrectly then the behavior is undefined.
Example: [el nodes without a [list as parent
The converter should emit an error, but doesn't (yet). It will create HTML li elements without embedding them in ul.
Hence the outcome depends on the browser. In Firefox, for example, the elements are displayed, but not indented.

scope as invalid any element occurring out of its natural context

That would be really great, but I think it's only doable in the context of using LSP.

the use of [insert could result in PML snippet-files containing only lists elements ([el) or table inner element ([tr, [tc, etc.), which would then be scoped as invalid

Yes, that's the main challenge.

The availability of the [insert node is one of the main reasons that make me reluctant in enforcing strict nodes-context validation in Sublime PML

Exactly! That's why it would be very difficult (if not impossible) to reliably add this kind of validation in an editor without using LSP.

many elements could change in the future, are also factors which are holding me back from implementing some syntax elements which can be challenging

True. Once these rules will be defined and implemented in the new parser, they risk to change often (especially at the beginning, and when new nodes are added). Each change would require the editor plugin to be updated. Maintenance nightmare!
One could of course define the rules in a standard format (e.g. XML Schema). But then the editor plugin must be able to read the schema, and apply the rules.
'Tree structure validation' must surely be implemented in a future version of the converter, but just now no decision has been taken yet as to how to do it and which format to use.

it's always a bit scary to face a similar syntax rewrite, because things could break badly at any stage.

I agree, and therefore IMHO I think that tree structure validation should currently not be implemented in any editor plugin.

[celtic-coder]

are lists nestable?

Yes. And there is no limit as to the depth of nesting.

What type of block nodes can a [list contain?

There are currently no rules applied by the parser. If elements are nested incorrectly then the behavior is undefined.
Example: [el nodes without a [list as parent
The converter should emit an error, but doesn't (yet). It will create HTML li elements without embedding them in ul.
Hence the outcome depends on the browser. In Firefox, for example, the elements are displayed, but not indented.

I tried the following experiment:

However, I am getting the following error:

Further, if I nest an [el element within another [el element, I just get a list of items all at the same level. Is the syntax that I am using for nestable lists incorrect, perhaps?

[celtic-coder]

Hmm... I have resolved the issue. It turns out that the [list element must be enclosed in a parent [el element. It's always worth experimenting in order to find a solution!

[tajmone]

I have resolved the issue. It turns out that the [list element must be enclosed in a parent [el element.

Do you mean like this?

[list
    [el Element]
    [el
        [list
            [el Sub-Element]
        ]
    ]
]

That would be the way HTML handles lists — which has always been a bit confusing to beginners, but correct when you come to think about.

It's always worth experimenting in order to find a solution!

Indeed, when I have to add a new syntax element to Sublime PML I always end up having to do lot's of tests to try and work out how nodes behave in edge-cases, in combinations with other nodes, etc. You often discover unexpected things, which are not mentioned in the documentation.

Also, when it comes to creating a syntax highlighter things are more complex, because you have to deal with subtle aspects of nodes and attributes (e.g. see tajmone/Sublime-PML#26 on the [code).

But I still think it's worth documenting edge case behaviours, at least in asides or footnotes. Also, it might be useful to add such examples somewhere in the PML Playground repository, mainly to ensure that future updates to the language don't change these edge cases.

If you look a the syntax test of Sublime PML, you'll find many nodes' edge cases and quirks being covered. Without these tests, it would be hard to detect if changes to the language are breaking how the package parses nodes.

[tajmone]

@pml-lang, is there an ETA for when a PML BNF document will be available?

It doesn't have to be parser-proof, just a representation of how the syntax is structured. Having a BNF of some sort would be very useful for developers working on highlighters, converters, etc.

[pml-lang]

is there an ETA for when a PML BNF document will be available?

Please have a look at the syntax of the upcoming PML version 2.0 here.

Note that this is a draft. It is not official yet (you can't access it by a link from PML's website). Moreover, the rules of lenient parsing are not yet described.

I will review this EBNF in the coming days, and add information about lenient parsing.

I hope this helps.