Smart Scoping in View of Auto-Completion

[tajmone]

I just wanted to share some thoughts and experience from working on the Sublime PML package for Sublime Text 3, hoping that they might be useful for anyone working on a PML editor syntax.

Like many other modern editors (e.g. VSCode and Atom), Sublime Text syntaxes are inspired by TextMate grammars and their scope naming conventions, i.e. syntax definitions (often referred to as "grammars") consist of JSON (or YAML) files where the various syntax elements are defined via regular expressions (RegExs) that capture them

Sublime Text 3 (ST3) introduced a newer and better syntax definition format (.sublime-syntax) which uses YAML instead of JSON (more human friendly) and adds some advanced features to control syntaxes, via a context stack.

When working on a new syntax definition, it might be tempting to begin by simply capturing as many keywords as possible under some generic keyword category, in order to obtain quick syntax highlighting, and then plan to achieve finer-grain scoping later on. I personally think that it's better to plan syntax scoping carefully, with a clear goal on how scoping should support advanced features (e.g. autocompletion suggestions) and start building the syntax one element at the time.

ST3 also offers another powerful feature: syntax tests. These tests allow developers to ensure that the syntax doesn't break as new elements are added (which can easily happen, when a syntax grows in complexity). Syntax tests are just sample files in the target syntax/language in which comment lines are used to check (assert) that certain elements are matching expected scopes.

When planning the PML syntax, I realized that an important goal of an editor package/plug-in would be to offer a rich editing experience, especially through smart auto-completions and snippets. In order to achieve this, you need to ensure that auto-completion will only suggest tags and attributes which are relevant to the current editing context.

To make a long story short, in the case of PML I came to realize that the best approach would be to focus on detailed semantic scoping of tags, in order to enable completion to only suggest attributes which are supported by any given tag. This meant being able to track each node's tag definition, and where the legitimate place for inserting attributes end.

I also realized that to keep this task simple, the syntax should syntax highlight (match) any attribute which occurs in a valid attribute place, regardless of whether that tag supports it. It's not a syntax's job to validate the source file, on the other hand it should error tolerant and provide visual cues even in partial or incorrect constructs — because editing is not always a linear process, and often users just copy and paste snippets around.

So, my approach was to determine the valid place of insertion for an attribute as follows:

1. The text immediately following a node's tag,
2. or following another attribute definition,
3. up to the end of the line, unless a \ line continuation is encountered,

So my solution was to ensure that each node tag creates a node-specific scope for attributes definitions, e.g. meta.annotation.node-attributes.chapter for the [chapter tag, as shown in this test file:

    [chapter title = H1 title
!   ^^^^^^^^^^^^^^^^^^^^^^^^^  meta.block.chapter
!            ^^^^^^^^^^^^^^^^  meta.annotation.node-attributes.chapter

The context of each node would setup that special node-attributes scope, which might or might not contain any attributes at all. Then, having set the scope to contain the specific node's name (e.g. .chapter in this case) it would simply include: a commonly shared context which scans for attributes, and which is able to determine when the scope should end (pop) taking into account the possibility of multiple attributes in a row, or on separate lines in case of a \ line continuation:

  attributes:
    - include: line_continuation   # <- mark `\` and consume `\n` to keep scope
    - match: (?=\])  # <- detect tag end and pop (without consuming)
      pop: true
    - include: pop-at-EOL  # <- end of line (\n) ends attributes definitions
    - include: attr_date
      # [...] include remaining attributes...
    - include: attr_unknown   # <- Temp fall-back for unimplemented attributes!

This looping mechanism is a simple way to manage attributes and lines continuations with a shareable context for all tags.

Since this system allows each node to define a tag-specific scope for areas of text in which attributes can defined, it also allows completion settings to target each node tag with supported attributes only. E.g. the Chapter_attributes.sublime-completions file defines which attributes suggestion are valid for [chapter:

{
  "scope": "text.html.pml meta.annotation.node-attribues.chapter.pml",
  "completions":
  [
    { "trigger": "title\tChapter title", "contents": "title = ${1:Chapter Title} $0" },
    { "trigger": "id\tChapter ID", "contents": "id = ${1:identifier} $0" },
  ]
}

thanks to the tag-specific scopes, it's possible to avoid auto-completion pollution (i.e. the annoying experience where the editor keeps suggesting elements which are out of context).

So, I hope that this brief excursus might have shed some light on the benefits of planning scope names in view of their potential use, rather than focusing on obtaining syntax highlighting only. Sometimes, you'll need to find ways to introduce an artificially created scope in order to obtain this, like in the above example, where the meta.annotation.node-attributes.XXX scopes had to be forcefully set-off, since they aren't necessarily going to actually match any attributes — e.g. the chapter_attributes scope below:

  chapter:
    - match: '\[(ch(?:apter)?\b|=(?!=))'
      scope: punctuation.definition.section.block.chapter.begin.pml
      push: [chapter_block, chapter_attributes]

  chapter_block:
    - meta_scope: meta.block.chapter.pml
    - include: pml_nodes
    - match: ']'
      scope: punctuation.definition.section.block.chapter.end.pml
      pop: true

  chapter_attributes:
    - meta_scope: meta.annotation.node-attributes.chapter.pml
    - include: chapter_title  # <- 1st, handle node specific attributes
    - include: attributes     # <- Shared attributes context

As a final note, for tags that don't support any attributes at all, it might be reasonable to simply skip including the shared attributes context; whereas for those which support only a single attribute, it might be worth including that attribute only; and, as the above [chapter definition show, some tags might have to handle special attributes which are not part of the shared context (in this case, Chapter's title being headings which need to be indexed as Symbols which can be navigated through).

Of course, this can only be done with editor syntaxes that support defining isolated context which can be included, pushed or set, and when a context stack is available for manipulation.

But I'm confident that the lesson learned here might apply to different editors plug-in, and in some measure to syntax highlighter too.