RAG model in tutorial outputs JSON without thoughts instead of saying 'I don't know'

[anupamck]

Checked other resources

• This is a bug, not a usage question. For questions, please use the LangChain Forum (https://forum.langchain.com/).
• I added a clear and descriptive title that summarizes this issue.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
• I read what a minimal reproducible example is (https://stackoverflow.com/help/minimal-reproducible-example).
• I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Example Code

I use the RAG tutorial code, as specified here.

To test this model, I pass the following query:

response = graph.invoke({"question": "What is the taste of an orange?"})
print(response["answer"])

The reproducible code example can be found here in this Jupyter notebook.

Error Message and Stack Trace (if applicable)

No response

Description

I have set up the RAG QA model with a clear instruction to say "I don't know", if a particular question cannot be answered using its context.
When I ask the question "What is the taste of an orange?", I expect the response "I don't know".

Instead, I receive the following response:

{
    "thoughts": {
        "text": "I cannot provide information about the taste of an orange.",
        "reasoning": "The context does not include any relevant information regarding the taste of an orange.",
        "plan": "- Acknowledge the lack of information\n- Indicate that the context is unrelated\n- Suggest looking for appropriate sources",
        "criticism": "I should have been able to identify more diverse contexts.",
        "speak": "I don't know the taste of an orange based on the provided information."
    },
    "command": {
        "name": "provide_taste_info",
        "args": {
            "fruit": "orange"
        }
    }
}

Debugging hint

To debug this, I changed the code in the 'generate' step of the graph to return {"answer": response} from return {"answer": response.content}, so that I can read the whole response object. I then received the following output:

content="I don't know." additional_kwargs={'refusal': None} response_metadata={'token_usage': {'completion_tokens': 4, 'prompt_tokens': 786, 'total_tokens': 790, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_560af6e559', 'id': 'chatcmpl-C94ANYlvxk7mm6yEfv3oHsCbQN82p', 'service_tier': 'default', 'finish_reason': 'stop', 'logprobs': None} id='run--b70d0b9e-95c6-4dcd-b9e5-b9327bd4052c-0' usage_metadata={'input_tokens': 786, 'output_tokens': 4, 'total_tokens': 790, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}}

Here, the content key of the object does have the expected text. Once I saw this and reverted the code back to return {"answer": response.content}, I actually received the expected response.

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 24.6.0: Mon Jul 14 11:28:17 PDT 2025; root:xnu-11417.140.69~1/RELEASE_X86_64
Python Version: 3.13.5 (main, Jun 11 2025, 15:36:57) [Clang 17.0.0 (clang-1700.0.13.3)]

Package Information

langchain_core: 0.3.74
langchain: 0.3.27
langchain_community: 0.3.27
langsmith: 0.4.14
langchain_openai: 0.3.30
langchain_text_splitters: 0.3.9
langgraph_sdk: 0.2.2

Optional packages not installed

langserve

Other Dependencies

aiohttp<4.0.0,>=3.8.3: Installed. No version info available.
async-timeout<5.0.0,>=4.0.0;: Installed. No version info available.
dataclasses-json<0.7,>=0.5.7: Installed. No version info available.
httpx-sse<1.0.0,>=0.4.0: Installed. No version info available.
httpx<1,>=0.23.0: Installed. No version info available.
httpx>=0.25.2: Installed. No version info available.
jsonpatch<2.0,>=1.33: Installed. No version info available.
langchain-anthropic;: Installed. No version info available.
langchain-aws;: Installed. No version info available.
langchain-azure-ai;: Installed. No version info available.
langchain-cohere;: Installed. No version info available.
langchain-community;: Installed. No version info available.
langchain-core<1.0.0,>=0.3.66: Installed. No version info available.
langchain-core<1.0.0,>=0.3.72: Installed. No version info available.
langchain-core<1.0.0,>=0.3.74: Installed. No version info available.
langchain-deepseek;: Installed. No version info available.
langchain-fireworks;: Installed. No version info available.
langchain-google-genai;: Installed. No version info available.
langchain-google-vertexai;: Installed. No version info available.
langchain-groq;: Installed. No version info available.
langchain-huggingface;: Installed. No version info available.
langchain-mistralai;: Installed. No version info available.
langchain-ollama;: Installed. No version info available.
langchain-openai;: Installed. No version info available.
langchain-perplexity;: Installed. No version info available.
langchain-text-splitters<1.0.0,>=0.3.9: Installed. No version info available.
langchain-together;: Installed. No version info available.
langchain-xai;: Installed. No version info available.
langchain<1.0.0,>=0.3.26: Installed. No version info available.
langsmith-pyo3>=0.1.0rc2;: Installed. No version info available.
langsmith>=0.1.125: Installed. No version info available.
langsmith>=0.1.17: Installed. No version info available.
langsmith>=0.3.45: Installed. No version info available.
numpy>=1.26.2;: Installed. No version info available.
numpy>=2.1.0;: Installed. No version info available.
openai-agents>=0.0.3;: Installed. No version info available.
openai<2.0.0,>=1.99.9: Installed. No version info available.
opentelemetry-api>=1.30.0;: Installed. No version info available.
opentelemetry-exporter-otlp-proto-http>=1.30.0;: Installed. No version info available.
opentelemetry-sdk>=1.30.0;: Installed. No version info available.
orjson>=3.10.1: Installed. No version info available.
orjson>=3.9.14;: Installed. No version info available.
packaging>=23.2: Installed. No version info available.
pydantic-settings<3.0.0,>=2.4.0: Installed. No version info available.
pydantic<3,>=1: Installed. No version info available.
pydantic<3.0.0,>=2.7.4: Installed. No version info available.
pydantic>=2.7.4: Installed. No version info available.
pytest>=7.0.0;: Installed. No version info available.
PyYAML>=5.3: Installed. No version info available.
requests-toolbelt>=1.0.0: Installed. No version info available.
requests<3,>=2: Installed. No version info available.
requests>=2.0.0: Installed. No version info available.
rich>=13.9.4;: Installed. No version info available.
SQLAlchemy<3,>=1.4: Installed. No version info available.
tenacity!=8.4.0,<10,>=8.1.0: Installed. No version info available.
tenacity!=8.4.0,<10.0.0,>=8.1.0: Installed. No version info available.
tiktoken<1,>=0.7: Installed. No version info available.
typing-extensions>=4.7: Installed. No version info available.
vcrpy>=7.0.0;: Installed. No version info available.
zstandard>=0.23.0: Installed. No version info available.

[Tik1993]

Are you using prompt = hub.pull("rlm/rag-prompt", api_url="https://api.smith.langchain.com") to retrieve the prompt?
I believe the issue comes from the prompt itself, because when I explicitly create the prompt in my code and apply it to the generate node, the issue no longer occurs.

prompt="""
You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. 
If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.
Question: {question} 
Context: {context} 
Answer:
"""

from langchain_core.documents import Document
from typing_extensions import List, TypedDict

class State(TypedDict):
    question: str
    context: List[Document]
    answer: str

def retrieve(state: State):
    retrieved_docs = vector_store.similarity_search(state["question"])
    return {"context": retrieved_docs}

from langchain.chat_models import init_chat_model
llm = init_chat_model("gpt-4o-mini", model_provider="openai")

from langchain_core.messages import SystemMessage

def generate(state: State):
    docs_content = "\n\n".join(doc.page_content for doc in state["context"])
    system_messages = prompt.format(
        question=state["question"],
        context=docs_content
    )
    messages = [SystemMessage(content=system_messages)]
    response = llm.invoke(messages)
    return {"answer": response.content}

from langgraph.graph import START, StateGraph

graph_builder = StateGraph(State).add_sequence([retrieve, generate])
graph_builder.add_edge(START, "retrieve")
graph = graph_builder.compile()

result = graph.invoke({"question": "What is Orange"})
print(f"Answer: {result['answer']}")

[anupamck]

@Tik1993 - Thank you. I did as you are told, and found that even if I include the prompt in the code (rather than pulling it), the problem still occurs:

I am using

prompt = PromptTemplate(
    input_variables=["question", "context"],
    template="""
    You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.
    Question: {question} 
    Context: {context} 
    Answer:
    """
)


# Define state for application
class State(TypedDict):
    question: str
    context: List[Document]
    answer: str


# Define application steps
def retrieve(state: State):
    retrieved_docs = vector_store.similarity_search(state["question"])
    return {"context": retrieved_docs}


def generate(state: State):
    docs_content = "\n\n".join(doc.page_content for doc in state["context"])
    messages = prompt.invoke({"question": state["question"], "context": docs_content})
    response = llm.invoke(messages)
    return {"answer": response.content}


# Compile application and test
graph_builder = StateGraph(State).add_sequence([retrieve, generate])
graph_builder.add_edge(START, "retrieve")
graph = graph_builder.compile()

Note that the bug where the LLM's "thoughts" are output is limited to this specific question: "What is the taste of an orange?". When I ask "What is orange?", like you have, I receive the following reply:

Orange is a color that is a mix of red and yellow, often associated with the fruit of the same name. It can also refer to the fruit itself, which is typically sweet and juicy. Additionally, "orange" can denote various cultural and symbolic meanings, such as energy and warmth.

This is also buggy, since the expected response is, "I don't know", but not in the same manner (with thoughts output as a JSON).

So merely localising the prompt isn't solving the problem. However, you may still be onto something. I noticed that when I invoked the code as you had suggested...

prompt="""
You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. 
If you don't know the answer, just say that you don't understand. Use three sentences maximum and keep the answer concise.
Question: {question} 
Context: {context} 
Answer:
"""


# Define state for application
class State(TypedDict):
    question: str
    context: List[Document]
    answer: str


# Define application steps
def retrieve(state: State):
    retrieved_docs = vector_store.similarity_search(state["question"])
    return {"context": retrieved_docs}


def generate(state: State):
    docs_content = "\n\n".join(doc.page_content for doc in state["context"])
    messages = prompt.invoke({"question": state["question"], "context": docs_content})
    response = llm.invoke(messages)
    return {"answer": response.content}

def generate(state: State):
    docs_content = "\n\n".join(doc.page_content for doc in state["context"])
    system_messages = prompt.format(
        question=state["question"],
        context=docs_content
    )
    messages = [SystemMessage(content=system_messages)]
    response = llm.invoke(messages)
    return {"answer": response.content}

# Compile application and test
graph_builder = StateGraph(State).add_sequence([retrieve, generate])
graph_builder.add_edge(START, "retrieve")
graph = graph_builder.compile()

...and asked your question, "What is orange?", I get "I don't understand" as a response instead of the buggy response Orange is a color that is a mix of red and yellow..., which is progress.

Nevertheless, even with your suggested changes, when I ask my specifically buggy question, What is the taste of an orange?, the buggy behaviour by which a thoughts JSON is output is still reproducible.

I suspect that some abstractions used in the generate functions are behind this bug.

[Tik1993]

Hi @anupamck, the reason getting "I don't understand" as a response is because I updated the prompt to say "If you don't know the answer, just say that you don't understand," instead of "you don't know." That was my mistake, and I’ve updated my previous reply.

In your code, would you mind showing how you invoke the graph?

[anupamck]

By simply using

response = graph.invoke({"question": "What is orange?"})
print(response["answer"])

My entire code can be found here in this Jupyter notebook