Skip to content

Incorrect Citation Rendering in "Ask Your Data" Mode Due to Missing Prompt Instruction #2473

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fabod opened this issue Apr 2, 2025 · 3 comments
Assignees
Labels
bug Something isn't working rag Related to RAG flow/logic/approaches

Comments

@fabod
Copy link

fabod commented Apr 2, 2025

Description:
When using the "Ask your data" mode, the citations are not rendered correctly. Unlike Chat mode, which renders citations using footnotes and a list of linked citations at the end , the "Ask your data" mode only appends the source names in round brackets within the text. This appears to be caused by a missing instruction in the prompt file.

Steps to Reproduce:

  1. Use the "Ask your data" mode in the application.
  2. Observe that citations are rendered with round brackets instead of the expected square bracket format and thus not being rendered.
  3. Compare pre-rendered answers from Chat mode and "Ask the data" mode by adding console.log(answer.message.content); to app/frontend/src/components/Answer/Answer.tsx as first command in export const Answer (e.g., before const followupQuestions = answer.context?.followup_questions;

Expected Behavior:
Answers are rendered with a footnote at the respective position and a linked list of citations at the end of the answer.

Actual Behavior:
Citations are displayes as plain text with round brackets, which does not follow the intended format.

Proposed Fix:
Update the file app/backend/approaches/prompts/ask_answer_question.prompty to include the following instruction, similar to the one in chat_answer_question.prompty:

Each source has a name followed by the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, for example [info1.txt]. Don't combine sources, list each source separately, for example [info1.txt][info2.pdf].

By adding this instruction, the "Ask your data" mode should render citations correctly.

Additional Context:
The chat_answer_question.prompty file already includes the correct instruction.
The discrepancy in prompt instructions between the two modes leads to inconsistent citation rendering.

@pamelafox
Copy link
Collaborator

You're right, we somehow lost that part of the instruction. I can make a PR for it, although you're welcome to make it as well.

Unfortunately, I currently only run quality evaluations on the /chat endpoint when I make changes, so I didn't notice that issue for the /ask endpoint.

@pamelafox pamelafox self-assigned this Apr 3, 2025
@pamelafox pamelafox added bug Something isn't working rag Related to RAG flow/logic/approaches labels Apr 3, 2025
@pamelafox
Copy link
Collaborator

Curiously, it appears we never had that instruction for the Ask tab, even before the Prompty change:
https://github.com/Azure-Samples/azure-search-openai-demo/pull/2178/files#diff-8457664fc3e0fe6195bebe311a95cd6d1b59418ed7448327d871078b076867e9L20
I agree that it's helpful for the LLM however, so I will add it.

@fabod
Copy link
Author

fabod commented Apr 3, 2025

I'm glad to hear I could help and I will happily make a PR tomorrow morning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working rag Related to RAG flow/logic/approaches
Projects
None yet
Development

No branches or pull requests

2 participants