add docs for DocMuncher (#529)

Author: poornimaramesh

docs/blog/posts/docmuncher.md

@@ -0,0 +1,23 @@
+---
+category:
+  - Admin App
+  - API
+date: 2025-04-15
+---
+
+# Eating documents for breakfast -- introducing DocMuncher
+
+Content cards are great, but what if all your information lives in documents? No problem -- now you can just upload the documents and turn them into cards!
+
+<!-- more -->
+
+## What does DocMuncher do?
+
+On the Admin App, you can now upload either .zip files or .pdfs, in addition to creating content cards directly.
+
+The backend then converts the document into bite-sized chunks that get turned into content cards.
+
+This is the first version that we have for ingesting documents, but we're planning to build clustering for semantically similar cards and paraphrasing them so they're more readable.
+
+## Doc references
+- [DocMuncher](../../components/docmuncherce/index.md)

docs/components/docmuncher/docmuncher_api.png

@@ -0,0 +1,26 @@
+# Document Ingestion aka DocMuncher
+
+`/docmuncher` is the endpoint that allows document uploads. We then chunk each uploaded document to create content cards tagged with each document name.
+This content is then searchable using the `/search` endpoints.
+
+<img src="./docmuncher_api.png">
+
+There are specifically two endpoints: the `POST` endpoint accepts document uploads (.pdf or .zip) and creates FastAPI background tasks for each document uploaded. The `GET` endpoints return the status of the created jobs.
+
+## Process flow for document ingestion
+```mermaid
+sequenceDiagram
+  autonumber
+  Admin App->>Backend: Document upload
+  Backend->>MistralAI: Convert document test to Markdown
+  MistralAI->>LangChain: Chunk Markdown text by headers
+  LangChain->>Backend: <Markdown chunks>
+  Backend->>Backend: Post-process chunks for continuity, formatting
+  Backend->>Admin App: <Final content cards + tags>
+```
+
+
+## Upcoming
+
+- [ ] Semantiic clustering: cluster cards with similar content
+- [ ] Paraphrasing cards for brevity

docs/components/docmuncher/index.md

@@ -83,4 +83,13 @@ There are 5 main components in Ask-A-Question.
 
     [:octicons-arrow-right-24: More info](./voice-service/index.md)
 
+
+- :material-api:{ .lg .middle .red} __Document Ingestion__
+
+    ---
+
+    _(Optional)_ Supports .pdf or .zip file ingestion to create content cards
+
+    [:octicons-arrow-right-24: More info](./docmuncher/index.md)
+
 </div>

docs/components/index.md

@@ -10,7 +10,7 @@
 |             | Content Tags                                         | :white_check_mark: | Add tags to your content for easy browsing (and more to come!)                                                                                    |
 | **Q3 2024** | Analytics for Feedback and Content                   | :white_check_mark: |  See content use, questions that receive poor feedback, missing content, and more                                                                 |
 |             | Voice notes support                                  | :white_check_mark: | Automatic Speech Recognition for audio message to content matching                                                                                |
-|             | Multi-turn chat                                      | :pencil:           | Refine or clarify user question through conversation.                                                                                             |
+|             | Multi-turn chat                                      | :white_check_mark:           | Refine or clarify user question through conversation.                                                                                             |
 |             | Engineering Dashboard                                | :pencil:           | Monitor uptime, response rates, throughput HTTP response codes                                                                                    |
 | Q4 2024     | Personalization and contextualization                | :pencil:           | Use contextual information to improve responses                                                                                                   |
 |             | Multimedia content                                   | :pencil:           | Respond with not just text but images and audio as well.                                                                                          |

docs/roadmap.md

@@ -44,16 +44,22 @@ nav:
               - Content Feedback: components/qa-service/content-feedback.md
           - Urgency Detection Service:
               - components/urgency-detection/index.md
+          - Workspaces:
+              - components/workspaces/index.md
       - Internal components:
           - LLM Proxy Server:
               - components/litellm-proxy/index.md
           - Hugging Face Embeddings:
               - components/huggingface-embeddings/index.md
               - How to use: components/huggingface-embeddings/how-to-use.md
+          - Multi-turn Chat:
+              - components/multi-turn-chat/index.md
           - Voice Service:
               - components/voice-service/index.md
               - In-house models: components/voice-service/in-house-models.md
               - External APIs: components/voice-service/external-apis.md
+          - DocMuncher:
+              - components/docmuncher/index.md
 
   - Integrations:
       - integrations/index.md