Make punctuation collapsing localisable

[georgd]

Punctuation in input data may collide with punctuation in style definitions. The citeprocs handle these punctuation clusters by applying hard-coded rules. While these rules have been chosen on a good basis and probably suffice for all English language documents and most others in Latin script, some style guides or non-English standards might require divergent collapsing.

I brought this up on the citeproc-js issue tracker for a German language style which requires suppression of subsequent , after titles ending in ! or ?, with the intention of a CSLm-extension: Juris-M/citeproc-js#154. @denismaier suggested to discuss this here and also came up with possible solutions to make punctuation collapsing localisable on a per-style basis:

<punct-handling>
  <punct>
    <input value="?,"/>
    <output value="?"/>
  </punct>
</punct-handling>

Perhaps this is closer to how it looks in citeproc-js:

<punct-handling>
  <punct value="?">
    <next value="!" result="?!"/>
    <next value="." result="?"/>
    <next value=":" result="?"/>
    <next value="," result="?,"/>
    <next value=";" result="?;"/>
  </punct>
</punct-handling>

We can take the combinations from citeproc-js as a basis for here if we can agree upon a syntax. I think CSL 1.1 is a good target for this. But perhaps we can even add it to 1.0.2 as this just makes the current behaviour explicit and configurable.

[cormacrelf]

https://github.com/cormacrelf/citeproc-rs/blob/b724318fe90c4c401ffbd4f34e8916f4e07435c9/crates/io/src/output/markup/move_punctuation.rs#L853

This is closer to how it's implemented now, rather than nested punc/next blocks. There is no need for the hierarchy. But as you can see, implementation would not be terribly difficult, just swap out the lookup tables.

[denismaier]

@cormacrelf so what do you suggest concerning style syntax?

[georgd]

And @fbennett do you have a preference for any of the two models?

[bdarcus]

Maybe should step back from syntax and settle what the logic is?

…

[denismaier]

Maybe should step back from syntax and settle what the logic is?

@bdarcus What do you mean with "logic" in that case? Right now, we have an established set of replacements, and the question is if we want to make this customizable. Is yes, we just need to define a syntax how this those replacement pairs should be defined. Maybe I'm just not understanding what you mean?

[bdarcus]

I guess aka "model."

[georgd]

I’m not sure, this is what you’re missing, but it’s what comes to my mind:

• Most of the hard-coded rules would never be touched so it makes sense to provide a syntax that allows redefinition of collapsing rules for single punctuation pairs.
• As a consequence, the processors can’t just overwrite the whole ruleset.
• Now, I see two possible ways to achieve the localisation:
  1. On a pair of punctuation first apply the localised punctuation rules, then run through the unchanged hard-coded set.
  2. Run through a modified copy of the hard-coded set.

Does this make sense? I think, the last point is getting a bit too deep into implementation details.

A thing, I was considering are punctuation clusters that upon consolidation produce a new punctuation cluster. But I don’t think these should be handled differently from how they are normally, except for each time the hard-coded collapsing rules are applied, the localised set should be considered.

[fbennett]

As @cormacrelf says, it's just a mapping table, and making changes isn't much of a problem in implementations. As @georgd points out, adjustments demanded by a style or language domain will affect only a small number of entries, so expressing the entire mapping table in XML shouldn't be necessary.

I wonder how much variation there is in punctuation merge and transformation conventions? If there are only a handful of patterns, they could be called by name, masking the details from CSL entirely. That would align with settings for quote-swapping of punctuation, where we have two modes and don't declare in CSL code exactly which punctuation marks are affected in which context.

[cormacrelf]

WRT the logic of how these additional rules work, the word "override" would suffice, we can take it from there. You can specify "go back to no collapsing for !." with rule that maps "!." to "!.".

So we're all on the same page, this is roughly how it's done in citeproc-rs. I'll get to my point in a moment.

  "String!" + ". etc"
= "String" + "!." + " etc"
= "String" + lookup("!.") + " etc"
= "String" + "!" + " etc"
= "String! etc"

A similar thing operates over quotation boundaries, with different lookup table and inside/outside location depending on your punctuation in quote settings. Hopefully it will be clear that a "flat" syntax ie a list of <punct pair="!." replace="!" /> would be easier to use, and would expose less implementationy stuff than a punct/next hierarchy, that IIRC is a detail of citeproc-js optimising the routine, whether that even helps or not I can't guess.

One optimisation I will not give up lightly is that each side contributes exactly one Unicode scalar value. AFAIK no language has punctuation represented with more than one scalar value. Punctuation smashing is already an expensive operation. But swapping out "lookup" for a different hash table is absolutely fine.

WRT adding the implementation detail to CSL, I'm not aware that the spec has any reference to punctuation smashing beyond punctuation-in-quote. So perhaps it at least deserves a mention, especially since it's already in the CSL test suite as something you have to have to be a real implementation.

The punctuation pairs are not really an implementation detail. It's just that the simplest user-facing conceptualisation happens to be very similar to the implementation. So I don't really see any downsides to exposing it. Because you still have complete flexibility to implement however you wish. And adding complete customisation of this does not realistically add any additional complexity that punctuation-smash-style="german" would not.

Compare this to defining in the spec where precisely punctuation smashing occurs. For example, does it happen inside "My title has !. in it"? As a processor dev, I need the flexibility to say no, if only because this saves a large number of CPU cycles scanning linearly through all text in all citations everywhere, for something that I don't believe anyone actually needs, because users already control what's in the title. Also, defining it would require acknowledging/cementing precise rich text structure in the spec, which is also undesirable. We get better at smashing punctuation as time goes on, and I don't want a plain description of the current best effort to become a limitation.

Where does that leave the punctuation pairs? I don't see that as a limitation, just an input parameter to a larger problem. Would you, for example, want multiple punctuation marks in a row to be customised? No, because users already control this by not writing dumb styles like delimiter=", " + <text value=".;/:!?"/> and expecting magic.

So final opinion is that sure, put some <punct pair="!." replace="!"/> syntax or a bikeshed thereof in the locales with normal term-like locale override semantics. Pair must be exactly two Unicode scalar values. Replace can be anything you like, including "SMASH_SUCCESS" in a test suite. Another thing to bikeshed is whether you can also indicate whether the pairing operates on a quotation border and where each mark goes in each case, which would ultimately build the mappings FULL_MONTY_QUOTES_IN/OUT in that file I linked above. You could use an attribute like in[/out]-quote="split|inside|outside". I'll yield my time there.

[fbennett]

Still wondering whether a limited set of named punctuation-merger patterns might be viable. Thoughts about that notion, @bdarcus, @georgd, @denismaier, @bwiernik, @adam3smith, @rmzelle?

[cormacrelf]

If you can really pin it down to one or two variations on the whole mapping, then yeah, that would be good probably. But that does sound like more work for us compiling various punctuation conventions than letting people figure it out over time. Custom is set and forget and we never have to worry about it again.

[georgd]

I think the big advantage of a limited set of predefined patterns is that it can be thoroughly tested.

I really can't tell how many patterns would appear but for German I can name two already that are not mutually exclusive: s/([!?]),/$1/ and (between title-main and title-sub) s/([!?]):/$1/ (or s/([!?]):/$1:/, depending on the actual default).

I don't expect lots of additional sets for Latin or Cyrillic script but every script with its own set of punctuation marks may add to this (Greek, Arabic, Hebrew,...).

[cormacrelf]

You can thoroughly test the operation of the customisable rules, if you build them. If you're saying "the built-in modes would be part of the CSL test suite", there are other ways to test your style and locale combo. I built one called jest-csl and Frank built another one independently at the same time (haha). But ultimately if you test that the rules engine works, you shouldn't really need a second test suite? It's a set of declarative rules that you would simply be replicating in test case form.

I accept the point that we could be doing all this for exactly one conflicting rule, and technically we could do some research and squish all the world's punctuation rules into one hashmap, simply special-casing German. But I would also welcome letting locale maintainers figure that out themselves and not have to worry about it here.

[georgd]

I accept the point that we could be doing all this for exactly one conflicting rule, and technically we could do some research and squish all the world's punctuation rules into one hashmap, simply special-casing German.

Just not to be mistaken: the two rules I posted above are not universally applicable for German but have to be set on a per style basis, either one of them alone, or both combined, or neither of them.

But I would also welcome letting locale maintainers figure that out themselves and not have to worry about it here.

I prefer this, too.