Provide llms.txt file

[cpfiffer]

Related to the v1 documentation refactoring #1528.

llms.txt is an increasingly popular standard for making websites easily accessible to machine reading. Website owners provide a llms.txt file at the root of the docs page (docs.dottxt.co/outlines/llms.txt, in our case) containing a condensed version of the documentation or website in Markdown format.

You should think of the HTML main page as being for humans, and llms.txt for language models or other agents.

A few examples:

• Anthropic's full doc and their abridged doc
• ElevenLabs
• Pinecone
• Full directory of existing llms.txt users

We would provide an abridged version of our docs, describing:

1. An overview of structured generation and outlines
2. Links to markdown content on the repo, such as guides or tutorials.
3. A cheatsheet of available methods, syntax, and expected output.
4. What good practices are.
5. Supported JSON schema features.
6. Simple troubleshooting and debugging advice.
7. How to use different inference backends.

We would also provide an llms-full.txt file containing all markdown from the docs composited into a single document. llms-full.txt is used when you want the model to understand the entirety of the documentation at once, whereas llms.txt is primarily for providing a directory, cheatsheet, and general overview.

llms.txt would be more focused, and would likely require focused maintenance updated whenever the documentation is updated.

llms-full.txt is relatively easy to do -- we would have a simple script as part of the site rendering GitHub action to recursively stack all documentation markdown files on top of each other and place them into llms-full.txt