Pandoc to PML Custom Lua Writer

[tajmone]

Since the pandoc PML custom writer is quite different from the PML reader filter, I've decided to create a dedicated Discussion for it, separate from the general discussion on pandoc filter at #26.

So, I've finally managed to set-up a dedicated repository folder for our custom PML writer:

• pandoc/filters-lua/pml-writer/

HTML Writer as Boilerplate

Currently pml-writer.lua is just a copy of sample.lua, the sample HTML writer that ships with pandoc, and default.pml.lua is just a copy of the default pandoc HTML template.

But since PML and HTML share a common document structure, this is a good starting point. In many cases, it should be sufficient to simply replace the HTML tags in output string with PML notes, so we could start working on these.
Other AST nodes will require ad hoc adaptation to PML needs, and we'll address them according to each node's priority.

What's important, is that now we finally have a proper scaffolding for the project, with a fully functional build script and sample test docs to work with.

Next Steps

The next steps will be to covert the default.pml.lua template to a real PML template, trying to map as many attributes as possible; and then start to edit pml-writer.lua to gradually implement proper PML nodes and attributes.

It's definitely going to be a rather long journey, but every project needs to start from somewhere.

Support and Collaboration

This thread should serve for further discussion on the implementation details, to prevent polluting Issues — we can always create a concise Issue once we have a clear task to add to our TODOs list.

Please, share any thoughts, doubts and tips — I have no specific knowledge on how to create a pandoc Lua writer, beside a quick brush with Lua filters in the past. Any contributions to the repository are most welcome.

[pml-lang]

Well done! Excellent start!

[tajmone]

Stuck in Conversion of Headings to Chapter Nodes

I knew this was going to be a problem, because in the pandoc AST a Heading element is a standalone block (with its level stored) whereas in PML it maps to a [ch node, which actually wraps its contents (including sub-Chapters).

The solution I came up with it this:

currChLev = 0
openChNodes = 0

-- lev is an integer, the header level.
function Header(lev, s, attr)
  local closeNodes = ""

  if currChLev > lev then
    for i=0,currChLev-lev do
      closeNodes = closeNodes .. "]"
      openChNodes = openChNodes -1
    end
  elseif currChLev < lev then
    for i=0,lev-currChLev do
      openChNodes = openChNodes +1
    end
  else
    closeNodes = closeNodes .. "]"
  end

  currChLev = lev
  return closeNodes .. "[ch " .. s
end

which works correctly ([ch nodes are closed as expected when a same- or lower-level Heading is encountered), except that I couldn't find a way to close all the open [ch nodes at the end of the document.

I originally hoped that I could do this via PML's default.pml.lua pandoc template, by exploiting the openChNodes variable:

$body$
$for(openChNodes)$]$$endif$
]

If openChNodes was visible to template, it would simply add as many closing ]s as there are open Chapter nodes, but the variable is not accessible to the template in any form (tried different approaches too).

So, for the above solution to work we'd need a way to ensure that either the writer or the template can take care of closing any pending open Chapter nodes at the end of the document.

I couldn't find any references on how to expose a custom writer/filter variable to the pandoc template, nor any way of explicitly emitting some text after the document has been processed.

Any ideas on this?

The only alternative method that comes to my mind would be to make the Header function walk through the nodes that follow it, but the problem is that these are not children nodes, just siblings — whereas in PML anything inside a Chapter would be its children, including sub-Chapters.

Pandoc does expose some functions to walk or manipulate AST nodes:

• walk_block() — Apply a filter inside a block element, walking its contents.
• walk_inline() — Apply a filter inside an inline element, walking its contents.
• blocks_to_inlines() — Squash a list of blocks into a list of inlines.

Maybe this might help:

• make_sections() — Converts list of Blocks into sections. Divs will be created beginning at each Header and containing following content until the next Header of comparable level.
• run_json_filter() — Filter the given doc by passing it through the a JSON filter.

The make_sections() seems to do something very close to what we need, although I'm not sure if/how the Headings are transformed in the process.

The run_json_filter() function could be used to pre-process the document with a custom binary tool that will manipulate the JSON AST to convert Headers into blocks first — but then, we might just as well handle the whole conversion via that app, instead of having to use both.

---

I'm quite lost at this point ... I was betting on using the template to handle closing the open Chapter tags, and now that I came to discover it doesn't work, I have no clue where to go from here.

[pml-lang]

I never used the Pandoc AST nor Lua, so I'm not sure about what's the best solution.

However, consider the following HTML snippet (code indented for better readability):
HTML_1:

<h1>Article Title</h1>
    <h2>Chapter 1</h2>
        <p>Text</p>
    <h2>Chapter 2</h2>
        <p>Text</p>
        <h3>Chapter 2.1</h3>
            <p>Text</p>
        <h3>Chapter 2.2</h3>
            <p>Text</p>

A better and more semantic version would be HTML_2:

<article><h1>Article Title</h1>
    <section><h2>Chapter 1</h2>
        <p>Text</p>
    </section>
    <section><h2>Chapter 2</h2>
        <p>Text</p>
        <section><h3>Chapter 2.1</h3>
            <p>Text</p>
        </section>
        <section><h3>Chapter 2.2</h3>
            <p>Text</p>
        </section>
    </section>
</article>

The above code is structurally similar to the code created by the PML converter for the following PML code:

[doc title="Article Title"
    [ch Chapter 1
        Text
    ]
    [ch Chapter 2
        Text
        [ch Chapter 2.1
            Text
        ]
        [ch Chapter 2.2
            Text
        ]
    ]
]

It would be interesting to see the ASTs created by Pandoc for examples HTML_1 and HTML_2.
Pandoc could create structurally equal ASTs (e.g. with chapters embedded in elements (like section or div)), but that's not the case according to your description.

The solution I came up with it this:

Yes, that's how I would do it too.

I couldn't find a way to close all the open [ch nodes at the end of the document

That surprises me a lot. It would be very useful if Pandoc provided a callback function called at the end of the document.

the variable is not accessible to the template in any form (tried different approaches too).

That's a pity too.

Possible Solution 1

For code like HTML_1, maybe headers can be embedded in elements (like section above) with the make_sections function you mentioned. The doc. states:

Divs will be created beginning at each Header and containing following content until the next Header of comparable level.

That would solve the problem, because HTML_1 would be converted in a structure like HTML_2.

Possible Solution 2

I was thinking at code similar to the one in your comment. But for this solution to work there must also be a way to close chapters at the end of the document.

Possible Solution 3

Docbook uses XML chapter and section elements to define the hierarchy of chapters. I think it's similar to the ch node in PML.
Therefore it might be useful to do some tests with a Docbook document as input for Pandoc, and have a look at the AST created as output.
If that AST is more convenient then one could possibly use the following Pandoc chain:
Format X -> Docbook -> Pandoc AST -> PML
Liam (@celtic-coder) hinted in a comment that he uses Docbook as an intermediary format too.
Maybe this (more bulky) solution would also simplify other aspects of converting from Format X to PML

Solution 4

There is always the solution of converting a Pandoc AST output to PML with an external tool written in any language. That solution would probably be less restrictive, but more time-consuming to implement.

If all this doesn't help it might be useful to ask in a forum dedicated to Pandoc/Lua programming, or to have a look at Lua filters and writers written by other people.

[tajmone]

Solved!

After hours of digging through the Lua filters examples, I've finally found a way to inject in the final document all the closing node brackets for [ch nodes.

Basically, the solution was to leverage function Doc(body, metadata, variables), which is called once for each document, and to concatenate the missing brackets at the end of body buffer:

function Doc(body, metadata, variables)
  local buffer = {}
  local function add(s)
    table.insert(buffer, s)
  end
  add(body)
  for i=1,openChNodes do
    add(']')
  end
  return table.concat(buffer,'\n') .. '\n'
end

... where openChNodes is my global variable that tracks how many [ch nodes are waiting to be closed (see my code in previous posts).

So, in the end, I didn't expose any variables to the template, but handled the problem internally to the Lua filter (which is safer, since templates can be tweaked).

I'm really glad that I solved this, because it was a blocker in the way of the pandoc filter. Now I can carry on transforming the pandoc document AST into PML tags, which is quite straight forward — at least, for supported features.

As for the unsupported features (footnotes, etc.) I'll either try to emulate them, or either suppress them or render the text raw (maybe with some comments to guide manual fixes).

Having the pandoc 2 PML converter is very important if we'd like to help people migrate from other doc formats to PML.

[pml-lang]

Great!

[tajmone]

Pandoc could create structurally equal ASTs (e.g. with chapters embedded in elements (like section or div)), but that's not the case according to your description.

Pandoc already distinguished between HTML4 and HTML5, but this doesn't really affect the AST which is a universal document representation, not tied to any specific syntax, so these types of info is not to be found in the AST, but needs to be deduced from the context.

I'm against enforcing this type of semantics, partly because the generated PML document uses different headings (i.e. in the AST they are a one-line block, whereas in PML they are nodes wrapping the whole sections within them, which is what made it hard to start with), and partly because the converter might be used to recover snippets intended for inclusion, so there are no guarantees that the first [ch node is actually a Level 1 heading (it could be any level, depending on where it's being included). Of course, the generated doc is always fully standalone ([doc, etc.) but in case of includable snippets the end user will strip away the header part.

That surprises me a lot. It would be very useful if Pandoc provided a callback function called at the end of the document.

That's what I though too. Then I discovered the main entry-point callback function which handles the entire doc, and simply added my code at the end of it (but to find this function I had to read through all the sample filters, since it's not mentioned in the docs). It's all a bit complicated, because there are different ways to configure how pandoc filters are applied to the AST (i.e. how it's traversed), and on top of that you can also change the order of the applied filters, but I didn't find the examples of much help in understanding the difference.

That's a pity too.

The documentation clearly mentions that the meta element can be used to store custom variable which can then be used in the custom template (which is very useful indeed, since templates are powerful), but it doesn't provide an example on how to create a custom variable. I looked at the sample filters but couldn't find one that uses this feature. I'm sure it's an easy thing to do, once you know how. Again, the issue is complicated by the custom pandoc types, which are defined in Lua via tables, and probably need to be manipulated via the exposed pandoc Lua API (unpacking tables, re-packing them, etc.). I'm sure that once you get the hang of it it's pretty straight forward, but right now the documentation seems to be for "initiates only" (the Haskellites Order of the Grand Orient).

For code like HTML_1, maybe headers can be embedded in elements (like section above) with the make_sections function you mentioned. The doc. states:

I'm not sure I follow you. I'm not dealing with HTML anywhere in the filter, just the pandoc AST and its elements types (string, heading, bold, quotation block, etc.). I'm reusing the HTML sample filter, just because PML is similar to HTML in many respects, but I don't get to handle any actual HTML, since I'm replacing the generated code with PML nodes. Also, many of your suggested features are usually handled by pandoc's native converter, not the filter, and leveraging templates (filters can access templates too, though).

The AST doesn't contain any "section" element, that's usually handled by the pandoc engine (for formats that support it) and controlled via the native extensions (sort of like options). In order to access such features in PML, a Lua filter is not enough, it would require creating a native pandoc extension for PML, in Haskell, and include it in the pandoc official package.

Filters are lossy in this respect, since the AST representation is format agnostic. Still, it contains all the essential semantics for controlling typographic elements, including quotes (double, single, typographic, straight, etc.) and special punctuation, via attributes.

Docbook uses XML chapter and section elements to define the hierarchy of chapters. I think it's similar to the ch node in PML.
Therefore it might be useful to do some tests with a Docbook document as input for Pandoc, and have a look at the AST created as output.

In the AST these heading would be flattened to single-line blocks (i.e. just a Level 1 Heading node, with attributes). Headings are always treated as standalone nodes, they don't wrap their contents in the AST. The difference with PML is that [ch nodes don't carry their level information, which is instead calculated at conversion time by taking into account their nesting level. DocBook heading tags do carry their level info, IRC. That's the main difference on how they affect the AST. But yes, if I was creating a filter to convert to DocBook, I'll be facing the same problem of having to close the XML tags of these headers, like I did for PML.

If that AST is more convenient then one could possibly use the following Pandoc chain:
Format X -> Docbook -> Pandoc AST -> PML

This wouldn't result in a different AST (the AST is only one) and these intermediate passages would most likely result in structural data loss.

Liam (@celtic-coder) hinted in a comment that he uses Docbook as an intermediary format too.

That's for converting AsciiDoc to PDF, or from AsciiDoc to pandoc (pandoc only supports AsciiDoc as an output format, not an input). The Asciidoctor PDF backend is still unfinished, so in order to use some advanced features (e.g. footnotes inside tables) you have to rely on the DocBook FOP toolchain, so you need to convert from AsciiDoc to DocBook first, then from DocBook to PDF — which is already a lossy conversion, since some AsciiDoc specific formatting is lost in the process, but until the official PDF backend is completed there are no choices really.

The same with pandoc, by first converting to DocBook you can then use pandoc to convert it to any other supported format. But this only applies to native conversion really, not filters, so the AST we're speaking about is not really an issue there. AsciiDoc is too complex to parse, so pandoc will most likely never support it as an input format, and only supports it as output (and only a subset, in any case, where features like callouts are dropped, etc.).

I'm not sure whether pandoc internally uses the same AST exposed to the filters (probably yes) or whether it has a richer AST that stores extra details. But the internal conversions benefit from ad hoc parsers that can do smart things (via extensions) that are not readily available via Lua filters. Obviously, Haskell is a powerful language for parsing, unlike Lua, so you can do things in Haskell which in Lua are hard to achieve.

Custom Converter

There is always the solution of converting a Pandoc AST output to PML with an external tool written in any language. That solution would probably be less restrictive, but more time-consuming to implement.

Yes, almost any language nowadays can parse the JSON AST. I'm not sure exactly how much work it would take, but definitely the Lua filter is quite practical since you can build it one step at the time, and also benefit from the pandoc API which is ready made (many useful helpers, especially for Unicode). The advantage of a custom filter would be that it would be more procedurally defined, instead of depending on all those hook callbacks. My guess is that if you know Lua well enough, and understand the extra options of the pandoc API, you can do almost anything via the Lua filters, since you can control the parsing process in many ways, including forcing the filter to parse the remaining sub-nodes from within any AST callback hook — but this probably needs some playing around and getting acquainted with the whole system.

I did consider creating a custom converter leveraging the JSON AST, but the main reason I didn't proceed in that direction is that I still can't work out from the pandoc documentation the full list of the AST nodes and their attributes — it's all intermingled with the Haskell module that pandoc uses for its AST, so there's no clear-cut list of the various elements and their attributes. I find the autogenerated Haskell API docs hard to read, with all those cross-reference links which end up in dead-ends.

Once I manage to extract the full list of the AST elements and their attributes, I'll consider creating a binary tool, if the Lua filter doesn't serve the purpose well. But right now, the Lua filter is a good way to learn how the AST works, and getting to know the various elements at hand (at least the essential ones).

Also, the Lua filter will always be faster than using the JSON AST, since it adds the overhead of converting to and from JSON, and the Lua filter being embedded in pandoc is very fast in execution.

let's see how it goes with the Lua filter. If it works fine, I'd stick with it, since whenever pandoc is updated the filter API is also updated, whereas a custom converter might require significant work to embrace new changes. Furthermore, it would require compiling a binary for each platform, which can be a pain (especially for macOS, since I don't have a Mac).

[pml-lang]

let's see how it goes with the Lua filter. If it works fine, I'd stick with it, since whenever pandoc is updated the filter API is also updated, whereas a custom converter might require significant work to embrace new changes.

Yes. Good approach!

[celtic-coder]

Hi Tristano (@tajmone),

I'm only starting to get my head around the Lua language so, I'm afraid that I'm not going to be much help right now. I did notice your 2bbcode from about 4 years ago. So, it is good that you have that experience from the past. I will add my 2c whenever I have some ideas.

Kind Regards,
Liam

[tajmone]

I can't say that I know Lua all that well either, but I've managed to use it on and off. Unfortunately, by looking at the pandoc Lua filters examples I couldn't find any that does what we need for this task — but then, few of the examples are true writers, most deal with tweaks on pre-existing formats.

I have the impression that to expose a Lua variable to the template you either need to use the meta node, or need to creates it as a Lua table or list, or some other type known to the pandoc AST — the documentation mentions custom types, but only for version number.

Probably I should ask this on the pandoc mailing list.

[tajmone]

PML Writer Updates

Just a quick update on the Lua filter progress...

Implementing escape rules turned out to be easier than I though, luckily the pandoc AST process one token (word, symbol, etc.) at the time, which turns out to allow fine-grain control on when and how to enforce escaping rules.

Also, since we're writing to PML is quite easy to just stick to the mandatory escaping rules for text; as for attributes, the general approach is to always quote them, and the escaping function has an optional boolean parameter to enforce the extra escaping of " inside the quoted string. When generating ad hoc PML nodes, on the other hand, attributes are manually injected, so there's rarely the need to pass them to the escape() function, unless we're passing on attributes provided by pandoc.

I've been testing all the inline elements, but still need to cover some block elements, just to check that the escape() function in called only when needed (e.g. not in raw blocks).

There's still a lot of work to be done, merely because pandoc covers lot's of style and formatting elements which are not natively supported by PML, which need to be either passed as raw html or manipulated in various ways.

HTML/SGML entities are probably going to require considerable work in order to be converted to Unicode code-point escape, especially named entities (I'm not sure if pandoc emits them, or whether it converts them to decimal or hex values, need to test that). If named entities are passed, it might require creating a dictionary/hash table to convert them to Unicode escapes.

Other elements requiring HTML hacks are probably images with internet URLs, since PML only supports filepath URIs for images.

Then there are those elements which require translation to nodes + html attributes (e.g. ordered lists and their various numeral types like romans, alphanumeric, etc.), and so on.

But so far, so good! the writer is already usable, although not fully complete, and the pros of using it for bulk documents-conversion already overweight the cons of having to manually fix the few elements which are not handled well.

[pml-lang]

Just a quick update on the Lua filter progress...

Thanks.

as for attributes, the general approach is to always quote them

Good.

If named entities are passed, it might require creating a dictionary/hash table to convert them to Unicode escapes.

Can't you just embed HTML entities in a verbatim node, to avoid the need for a hash table? E.g. € would be written as [verbatim €] in the PML document.

PML only supports filepath URIs for images.

This will be changed in the next PML version, as confirmed here.

the writer is already usable

Great!!!

manually fix the few elements which are not handled well.

Just an idea: Maybe a PML comment could be inserted for these element, so that the user can easily find all the places that might need manual corrections.

[tajmone]

Can't you just embed HTML entities in a verbatim node, to avoid the need for a hash table? E.g. € would be written as [verbatim €] in the PML document.

Indeed I could (and that's what I'm currently doing as a quick workaround for similar elements), but I think that it's better to put in a bit more effort and produce PML native nodes whenever possible. Numbered entities should not be hard to convert to Unicode point escapes, and for named entities the "hard" part is more the creation of a full dictionary of the named entities than the code itself.

E.g. HTML comments are currently being passed as Raw HTML, but in the future (once all elements are implemented) I'd like to catch comments via a RegEx and convert them to [- .. -] nodes.

PML only supports filepath URIs for images.

This will be changed in the next PML version, as confirmed here.

Excellent! Then I can postpone the various URL/path checks and workarounds in the writer until the next version, which will save me a lot of work (which would have to be undone when the new feature is ready).

Just an idea: Maybe a PML comment could be inserted for these element, so that the user can easily find all the places that might need manual corrections.

That's an excellent idea! I didn't think of it. Especially if we adopt some typographical convention (e.g. [- @FIXME ..) which allows end users to quickly find all occurrences of such elements in the converted source, via editor "Find" functionality, instead of having to manually spot them. Even if it's just a brief message, or an error code that can be looked up in the documentation (e.g. @FIXME 31, where 31 stands for "unsupported feature X") it would be very useful.