feat: remove prompt table #5348

Weves · 2025-09-06T06:36:30Z

Description

Fixes https://linear.app/danswer/issue/DAN-2419/merge-prompt-and-persona-table

How Has This Been Tested?

Locally + new tests

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

This PR should be backported (make sure to check that the backport attempt succeeds)
[Optional] Override Linear Check

Summary by cubic

Merges Prompt into Persona and removes the prompt table. Prompt configuration is now embedded in personas; APIs, models, seeding, and tests updated to drop prompt_id and use persona fields.

Refactors
- Database: add system_prompt, task_prompt, include_citations, datetime_aware, is_default_prompt to persona; migrate data; drop prompt and persona__prompt; remove chat_message.prompt_id and backfill persona where possible.
- Models/API: remove Prompt model and relationships; ChatMessage no longer references a prompt; CreateChatMessageRequest and PersonaUpsertRequest drop prompt_id(s); prompt utils and builders now read from Persona.
- Seeding: prompts.yml now creates personas with embedded prompt fields; personas.yml accepts prompt_config for these fields.
- Server: persona create/update endpoints embed prompt fields directly; Slack channel config and assistants API updated.
- Compatibility: helpers now live in persona.py (get_default_prompt, get_prompts_by_ids, get_prompt_by_name, upsert_prompt) and return/use Persona.
Migration
- Run the Alembic migration to merge prompts into personas and drop legacy tables/columns.
- Update clients:
  - Stop sending prompt_id in chat and persona APIs; use persona_id and provide system_prompt, task_prompt, include_citations, datetime_aware via persona.
  - Replace any prompt ID references with persona IDs.
- Ensure a default persona exists; migration marks prior default prompts as is_default_prompt.
- Review YAML seeds: prompts are now personas; personas.yml can set prompt_config.

vercel · 2025-09-06T06:36:36Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
internal-search	Ready	Preview	Comment	Sep 10, 2025 5:21pm

greptile-apps

Greptile Summary

This PR implements a major database schema refactoring that merges the separate Prompt and Persona tables into a single unified Persona table. The change eliminates the complexity of managing separate prompt entities by embedding prompt configuration directly within personas.

Key Changes:

Database Schema: Adds new columns (system_prompt, task_prompt, datetime_aware, is_default_prompt) to the persona table and removes the prompt table and persona__prompt junction table
Migration: Comprehensive Alembic migration (abbfec3a5ac5_merge_prompt_into_persona.py) handles data transfer from prompt/persona__prompt tables to embedded persona fields, updates chat message references, and drops legacy tables
API Simplification: Removes prompt_id parameters throughout the codebase - from CreateChatMessageRequest, PersonaUpsertRequest, chat APIs, and assistant endpoints
Model Updates: PromptConfig.from_model() now accepts Persona objects instead of Prompt objects; prompt-related functions moved from prompts.py to persona.py with compatibility shims
Seeding Changes: Replaces YAML-based prompt loading with programmatic persona definitions in prebuilt_personas.py, eliminating separate prompt entity management
Code Cleanup: Updates all prompt access patterns from persona.prompts[0].system_prompt to direct field access persona.system_prompt or "" across agent search, chat utilities, and Slack handlers

The refactoring consolidates what was previously a many-to-many relationship (which in practice was mostly one-to-one) into embedded fields, simplifying queries, reducing joins, and making the API more intuitive. All prompt functionality is preserved but now accessed through the persona context rather than separate entities.

Confidence score: 2/5

This PR has significant architectural changes that could cause data corruption or system failures if the migration logic is flawed
Critical issues exist in the migration script including incorrect column existence checks and potentially wrong alternate_assistant_id usage for chat message migration
Database migration complexity introduces substantial risk during deployment, with potential for data loss if rollback is needed

_{52 files reviewed, 16 comments}

_{Edit Code Review Bot Settings | Greptile}

backend/onyx/db/pydantic_type.py

backend/ee/onyx/server/seeding.py

backend/tests/integration/common_utils/managers/persona.py

backend/alembic/versions/abbfec3a5ac5_merge_prompt_into_persona.py

backend/onyx/db/persona.py

backend/onyx/server/openai_assistants_api/asssistants_api.py

backend/onyx/chat/models.py

backend/onyx/agents/agent_search/dr/nodes/dr_a0_clarification.py

cubic-dev-ai

16 issues found across 54 files

_{React with 👍 or 👎 to teach cubic. You can also tag @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

cubic-dev-ai · 2025-09-06T22:49:32Z

backend/onyx/chat/models.py

-            datetime_aware=model.datetime_aware,
+            system_prompt=override_system_prompt or model.system_prompt or "",
+            task_prompt=override_task_prompt or model.task_prompt or "",
+            datetime_aware=model.datetime_aware or True,


Using or True forces datetime_aware to True even when the persona explicitly sets False; honor False by using the model value (or only default if None).

Prompt for AI agents

Address the following comment on backend/onyx/chat/models.py at line 283: <comment>Using `or True` forces datetime_aware to True even when the persona explicitly sets False; honor False by using the model value (or only default if None).</comment> <file context> @@ -270,17 +270,17 @@ class PromptConfig(BaseModel): - datetime_aware=model.datetime_aware, + system_prompt=override_system_prompt or model.system_prompt or "", + task_prompt=override_task_prompt or model.task_prompt or "", + datetime_aware=model.datetime_aware or True, ) </file context>

cubic-dev-ai · 2025-09-06T22:49:33Z

backend/onyx/server/features/persona/models.py

+    def from_model(cls, persona: Persona) -> "PromptSnapshot":
+        """Create PromptSnapshot from persona's embedded prompt fields"""
+        if persona.deleted:
+            raise ValueError("Persona has been deleted")


Raising here breaks allow_deleted behavior: FullPersonaSnapshot.from_model may proceed when allow_deleted=True but this call still raises, causing an unexpected error. Remove the hard raise or thread allow_deleted through.

Prompt for AI agents

Address the following comment on backend/onyx/server/features/persona/models.py at line 34: <comment>Raising here breaks allow_deleted behavior: FullPersonaSnapshot.from_model may proceed when allow_deleted=True but this call still raises, causing an unexpected error. Remove the hard raise or thread allow_deleted through.</comment> <file context> @@ -29,18 +28,19 @@ class PromptSnapshot(BaseModel): + def from_model(cls, persona: Persona) -> "PromptSnapshot": + """Create PromptSnapshot from persona's embedded prompt fields""" + if persona.deleted: + raise ValueError("Persona has been deleted") return PromptSnapshot( </file context>

cubic-dev-ai · 2025-09-06T22:49:33Z

backend/onyx/db/pydantic_type.py

        return None
+
+
+class PydanticListType(TypeDecorator):


Missing cache_ok on custom TypeDecorator may disable SQLAlchemy type caching and emit warnings; set cache_ok = True on the class.

Prompt for AI agents

Address the following comment on backend/onyx/db/pydantic_type.py at line 35: <comment>Missing cache_ok on custom TypeDecorator may disable SQLAlchemy type caching and emit warnings; set cache_ok = True on the class.</comment> <file context> @@ -30,3 +30,27 @@ def process_result_value( return None + + +class PydanticListType(TypeDecorator): + impl = JSONB + </file context>

backend/onyx/db/persona.py

backend/ee/onyx/server/seeding.py

backend/tests/external_dependency_unit/startup/test_persona_seeding.py

cubic-dev-ai · 2025-09-06T22:49:34Z

backend/onyx/server/openai_assistants_api/asssistants_api.py

-    if "instructions" in update_data and persona.prompts:
-        persona.prompts[0].system_prompt = update_data["instructions"]
+    if "instructions" in update_data:
+        persona.system_prompt = update_data["instructions"]


modify_assistant assigns list[dict] to persona.tools relationship, causing SQLAlchemy error on commit.

Prompt for AI agents

Address the following comment on backend/onyx/server/openai_assistants_api/asssistants_api.py at line 201: <comment>modify_assistant assigns list[dict] to persona.tools relationship, causing SQLAlchemy error on commit.</comment> <file context> @@ -208,8 +197,8 @@ def modify_assistant( - if "instructions" in update_data and persona.prompts: - persona.prompts[0].system_prompt = update_data["instructions"] + if "instructions" in update_data: + persona.system_prompt = update_data["instructions"] db_session.commit() </file context>

evan-onyx

some migration issues and minor code cleanups

evan-onyx · 2025-09-09T22:18:47Z

backend/alembic/versions/abbfec3a5ac5_merge_prompt_into_persona.py

+
+# revision identifiers, used by Alembic.
+revision = "abbfec3a5ac5"
+down_revision = "8818cf73fa1a"


confirm before merge

backend/alembic/versions/abbfec3a5ac5_merge_prompt_into_persona.py

backend/ee/onyx/server/seeding.py

backend/onyx/chat/models.py

backend/onyx/db/seeding/chat_history_seeding.py

evan-onyx · 2025-09-09T22:40:09Z

backend/tests/integration/common_utils/managers/persona.py

-                        ]
+                mismatches: list[tuple[str, object, object]] = []
+
+                if fetched_persona.name != persona.name:


pls make a list of (tag, expected, actual) tuples and iterate over it

vercel bot deployed to Preview September 6, 2025 06:41 View deployment

Weves force-pushed the merge-prompt-and-persona branch from 7df9d87 to a8e90bb Compare September 6, 2025 21:40

Weves changed the title ~~Initial go at remove prompt table~~ feat: remove prompt table Sep 6, 2025

vercel bot deployed to Preview September 6, 2025 21:45 View deployment

vercel bot deployed to Preview September 6, 2025 21:51 View deployment

vercel bot deployed to Preview September 6, 2025 22:04 View deployment

Weves marked this pull request as ready for review September 6, 2025 22:36

Weves requested a review from a team as a code owner September 6, 2025 22:36

vercel bot deployed to Preview September 6, 2025 22:36 View deployment

greptile-apps bot reviewed Sep 6, 2025

View reviewed changes

vercel bot deployed to Preview September 6, 2025 22:40 View deployment

cubic-dev-ai bot reviewed Sep 6, 2025

View reviewed changes

vercel bot deployed to Preview September 6, 2025 22:55 View deployment

vercel bot deployed to Preview September 7, 2025 00:25 View deployment

vercel bot deployed to Preview September 7, 2025 00:33 View deployment

vercel bot deployed to Preview September 7, 2025 00:37 View deployment

vercel bot deployed to Preview September 7, 2025 01:34 View deployment

vercel bot deployed to Preview September 7, 2025 01:53 View deployment

vercel bot deployed to Preview September 7, 2025 02:08 View deployment

Weves force-pushed the merge-prompt-and-persona branch from a5a8300 to 5dfcb4b Compare September 7, 2025 18:58

vercel bot deployed to Preview September 7, 2025 19:02 View deployment

evan-onyx approved these changes Sep 9, 2025

View reviewed changes

Weves added 7 commits September 9, 2025 17:14

Squashed

a5391bf

Fixes

15c8f53

Remove yamls

bd5a1e6

Fix

f92c668

Fix mypy

0db1ccc

Add test

f53580f

Remove unused column

4f8c681

Weves added 9 commits September 9, 2025 17:14

Fix migrations

1e9f55c

Fix + add tests

2d26395

Adjust frontend models

3233f4c

Address all Greptile comments

27fc306

Fix mypy

66d3375

Fix mypy r2

0b59d9c

Fix mypy r3

096a22a

Fix mypy r4

ac44a44

Address EL comments

d345095

Weves force-pushed the merge-prompt-and-persona branch from ca01644 to d345095 Compare September 10, 2025 00:14

vercel bot deployed to Preview September 10, 2025 00:20 View deployment

Fix playwright test

2300904

vercel bot deployed to Preview September 10, 2025 17:21 View deployment

Weves merged commit 458ed93 into main Sep 10, 2025
16 of 17 checks passed

Weves deleted the merge-prompt-and-persona branch September 10, 2025 17:21

AnkitTukatek pushed a commit to TukaTek/onyx that referenced this pull request Sep 23, 2025

feat: remove prompt table (onyx-dot-app#5348)

4af375b

feat: remove prompt table #5348

feat: remove prompt table #5348

Uh oh!

Conversation

Weves commented Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How Has This Been Tested?

Backporting (check the box to trigger backport action)

Summary by cubic

Uh oh!

vercel bot commented Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Greptile Summary

Confidence score: 2/5

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai bot Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

evan-onyx left a comment

Choose a reason for hiding this comment

Uh oh!

evan-onyx Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

evan-onyx Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Weves commented Sep 6, 2025 •

edited

Loading

vercel bot commented Sep 6, 2025 •

edited

Loading

cubic-dev-ai bot Sep 6, 2025 •

edited

Loading

cubic-dev-ai bot Sep 6, 2025 •

edited

Loading

cubic-dev-ai bot Sep 6, 2025 •

edited

Loading

cubic-dev-ai bot Sep 6, 2025 •

edited

Loading