RFC: Semantic Highlighting

[s0kil]

The VS Code docs state:

Highlighting based on semantic tokens is considered an addition to the TextMate-based syntax highlighting. Semantic highlighting goes on top of the syntax highlighting. And as language servers can take a while to load and analyze a project, semantic token highlighting may appear after a short delay.

To be honest, I would not mind the tiny delay, Since our language server is built in Crystal, it's fast, and it should not be a problem if we skip the tmLanguage definition.
The benefits would be correct highlighting, better maintainability.

For Reference:

• Semantic Tokens API microsoft/vscode#86415
• https://github.com/microsoft/vscode/wiki/Semantic-Highlighting-Overview
  The links on the bottom of the wiki page point to the API which is necessary.
  It's also important to point out that this Semantic Tokens Spec is not specific to VS Code, It will be part of the next LSP version 3.16.0, so other editors will begin to adopt it.
• https://github.com/microsoft/vscode-extension-samples/tree/master/semantic-tokens-sample

Copied from conversation at Mint Lobby

[lukepighetti]

Do we have a proper language server in mint to enable building this feature today?

[s0kil]

@lukepighetti From what I understand, here is what features Semantic Tokens allow.
Since the Mint parser fully understands what mint code does, all we have to do is specify what each token does: Here is an example from here

	const tokenTypesLegend = [
		'comment', 'string', 'keyword', 'number', 'regexp', 'operator', 'namespace',
		'type', 'struct', 'class', 'interface', 'enum', 'typeParameter', 'function',
		'member', 'macro', 'variable', 'parameter', 'property', 'label'
	];

We could configure the existing Mint language server to tell VS Code what each token means, and it will take care of the highlighting, this is much simpler and more maintainable over writing tmLanaguage regex's.

Another example:

/* This Is Comment In Mint */

The language server will tell VS Code that on line 0, starting from character 0 to character 29 that is a comment.

[s0kil]

This also fixes the issues we currently have where the render function could be defined multiple ways:

fun render {}
fun render: Html {}

fun render {
      if (clicked) {
        "I have been clicked!"
      }
}

We currently only support highlighting for fun render: Html {}
Also the Mint specific conditional statements are obviously not part of HTML, so the highlighting does not work.

[lukepighetti]

Can you point me towards any documentation or source that will help me understand what is currently available from the Mint language server? I was under the impression that there is no language server available to us yet.

[s0kil]

We do not have any documentation, but the existing work is being done the language-server branch

[s0kil]

@lukepighetti You probably already know about this, but I realized a simple way to views changes specific to the language server, using GitHub compare feature: https://github.com/mint-lang/mint/compare/language-server

[lukepighetti]

Shall we create a branch to start implementing these hover features? It looks like it's about halfway done. Whats the recommended way to run a development version of mint?

[s0kil]

It would probably be easier if a binary was built from the language-server branch, and we could simply download it in the extension.

[lukepighetti]

I'll have to claim ignorance on the best practices there, do you have any examples or docs I can refer to so I can educate myself further?

Alternative: what if we ask @gdotdesign to put the lang server in the production build as a technical preview?

[gdotdesign]

I'll have to claim ignorance on the best practices there, do you have any examples or docs I can refer to so I can educate myself further?

If it's about compiling Mint just follow the steps in the contribution guide: https://github.com/mint-lang/mint/blob/master/CONTRIBUTING.md#development , to have the language server just compile it on the language-server branch (the language server will be available as mint-dev ls)

[gdotdesign]

Also it might be more productive to do this in a chat format 🙂 We are on Discord and Gitter 😉

[s0kil]

I'll have to claim ignorance on the best practices there, do you have any examples or docs I can refer to so I can educate myself further?

Alternative: what if we ask @gdotdesign to put the lang server in the production build as a technical preview?

Building from scratch requires the user to have Crystal installed, with all the necessary packages that Crystal requires, this creates an unnecessary barrier.

But we could still continue this path for a development preview, meaning the extension would not be widly released until the language server is merged into master.

[s0kil]

Additional Reference, Upcoming LSP Version 3.16 Specification with Semantic Tokens Support:
https://microsoft.github.io/language-server-protocol/specifications/specification-3-16/