Rewrite

[nobodywasishere]

The current implementation is hacky, buggy, doesn't provide as many features as I want it to, insecure, and has no introspection into what happens when a build fails. It also has some design flaws that make it a lot more difficult to use.

Some ideas:

• Different "providers" handle different endpoints
  • GitHub would be at /github, with it's own class that would handle everything from updating the docs (probably using a base Git class), endpoints (/github/username/project), etc
  • There would be a base Git provider that would allow generic git URLs at /git/uri-encoded-url, for generic access
  • Chisel / Fossil support with its own provider(s) at /chiselapp/username/project
• List all projects from a given user (will be limited to specific providers)
  • /github/username
  • /chiselapp/username
• With this, I do not want to break existing links, but at the same time, I want URLs to be as clean as possible
• Switch from using firejail to a custom version of the docs generator
  • This would allow escaping HTML in the docs and providing other unique documentation features (maybe linking between docs)
  • This custom generator would also disable run macros and commands at the language level, preventing them from being used at compile time
  • Wouldn't need to "post process" the docs when generating via an html parser, can just add the required stuff to the generator itself
  • Can also set it up to always show docs for private / protected / lib objects by default, though this may be unpopular
• When documentation fails to build, it would be really useful for people to be able to get information about why the docs failed. I originally didn't include this functionality as I viewed it as a security risk, but the benefits outweigh the drawbacks
• Improvements to markd such as table support are required
• Loading screen if documentation generation is still in progress
• Independent search page w/ query parameter support
• Custom repo metadata handled by each provider (stars, fork, etc)
• Syntax highlighting for languages other than just Crystal
• Admin dashboard for rebuilding certain docs, more fine-tuned control
• Use sqlite instead of postgres for simplicity (how big could it be committed?)
• I should probably use something better than Kemal. While it's good, it's very limited

Things I should investigate:

• Docs for different platforms can technically be different (but are more often than not the same); it may be useful to allow people to generate different versions for different platforms, i.e. 1.4.0 (Linux), 1.4.0 (Windows), etc.
• How easy would it be to make this self-hostable, so that projects such as ameba could host their own documentation with all the benefits of crystaldoc? Maybe outside the scope.
• If someone uses Int32 in their documentation, it should allow linking to the latest crystal-lang/crystal directly, even if it's in a different repo. This could also be expanded to other dependencies, allowing easier time of accessing the documentation

Steps:

• Pull out docs generator into its own program from stdlib
  • Use newer markd
  • Pull in changes to sanitize the docs when generating them
  • Make necessary changes to make it more secure (disallowing run commands, macro file access, etc), may require parser changes
• Get basic Git / GitHub provider working
• Draw the rest of the horse

[straight-shoota]

• In general, I'd suggest to treat all upstream sources just as URLs. That's how shards handles things internally as well. The github, gitlab etc. keys are just aliases for the provider's URL schemes. Of course crystaldocs could support such aliases as well, but it's probably best to build everything on fully qualified, normalized URLs. That's universal and unambiguous.
• Grouping projects can then be trivially achieved based on URL-prefixes. No provider awareness necessary.

• Admin dashboard for rebuilding certain docs, more fine-tuned control
  I'd be very cautious about this to avoid complexity escalation. IMO a doc building service should need no control access whatsoever. The most manual intervention required should be busting corrupted data to force a rebuild. But that shouldn't need much admin dashboard.

• Docs for different platforms can technically be different (but are more often than not the same); it may be useful to allow people to generate different versions for different platforms, i.e. 1.4.0 (Linux), 1.4.0 (Windows), etc.
  I suppose that's a topic to discuss for the entire community because it affects any kind of API doc generation, not just for crystaldoc.info.

Not sure if you're aware of this prior discussion about doc hosting: https://forum.crystal-lang.org/t/hosted-documentation-site/1896
It discusses how such a service could be structured, and I'm sure it can provide some good ideas for the rewrite. At the time it was only theoretical discussion, and wasn't directly implemented. But the operating history of crystaldocs.info should help align the considerations with practical experience.

[watzon]

test