new in-code docs parser

[alex-s168]

this does not work currently, and I do not know why. It parses documentation correctly, but after some time (during typechecking and maybe lowering), it just exits with Killed, and no error signal or anything. I think it might be related to the newly added AST nodes.
Please help

doc syntax

this should probably be moved to the lsts lang ref

## some text
## same paragraph, but new line
##
## new paragraph. **bold text**
## __same paragraph, but underlined__ .
##
## see [print]
##
## see [type(Vector<t>)]. type() is required because inside the square brackets is an LSTS expression
##
## [[attribute]]
## [[another-attribute: lsts-expression]]
## [[deprecated: "don't use this anymore lol"]]
some-code

the expected AST for the above:

Seq { some-code, Meta { DocMeta {
  Seq { Seq {
    DocParagraph { Seq { Seq { DocWord("some"), DocWord("text") }, DocWord("same") } .... },
    DocPagraph { ..., DocUnderline("same paragraph, but underlined") }
  },DocParagraph { Seq { DocWord("see"), AIdent("print") } }
   ...
  },
  { "attribute": None,
    "another-attribute": Some(lsts-expression),
    "deprecated": str-node }
} } }

TODO

• a way to reference functions with specific args. example: <print(U64)>
• doc generator for this
• use new syntax in stdlib collection improvements #1250

Checklist before requesting a review

• I have performed a self-review of my code
• If it is a new feature, I have added thorough tests.
• I agree to release these changes under the terms of the permissive MIT license (1).

1. https://github.com/andrew-johnson-4/lambda-mountain/blob/main/LICENSE

[andrew-johnson-4]

"Killed" sounds like an infinite loop or infinite recursion.

[alex-s168]

"Killed" sounds like an infinite loop or infinite recursion.

I think infinite recursion would have caused a signal that is captirable by the debugger

[alex-s168]

actually maybe it was OOM?

[andrew-johnson-4]

If an infinite loop happens, then often times there will be allocations with no GC, leading to OOM.

[alex-s168]

If an infinite loop happens, then often times there will be allocations with no GC, leading to OOM.

I don't think the infinite loop is in my code tho

[andrew-johnson-4]

Yes, possibly. Let me read code for a bit to see what is happening.

[andrew-johnson-4]

It isn't making it past parse. Idk what is going on yet, but it stops there.

[andrew-johnson-4]

(set program (Seq(
   (close program)
   (close (Meta( (close m) )))
)))
(set tokens r1)
# ignored. just use lsts

I understand the desire to get rid of this. But we first need to get rid of code that references this. This is an infinite loop because it won't consume these tokens.

[andrew-johnson-4]

I rearranged the DocMeta ast nodes into normal ast nodes.

Benefits of using normal AST App/Var/Lit nodes:

• interesting backends are then possible (using a Blob render to generate a documentation website etc.)
• term rewriting can be made much simpler and used even for exotic code objects such as Meta nodes
• type inference is easy (no changes needed)
• maximizes chance that existing code will work without modification (the LM codebase has no dead or near-dead code)
• minimizes manual changes necessary to change stuff (we don't have a staff of 100 interns)