Using Claude Sonnet 3.7+ in BedrockChatGenerator in multi-turn setting with tools + thinking causes error

[sjrl]

Describe the bug
cc @deep-rloebbert

When using Claude Sonnet 3.7 (or 4.0) with tools and thinking enabled inside an Agent we get the following error message

haystack_integrations.common.amazon_bedrock.errors.AmazonBedrockInferenceError: Could not perform inference for Amazon Bedrock model arn:aws:bedrock:us-east-1::inference-profile/us.anthropic.claude-3-7-sonnet-20250219-v1:0 due to:
An error occurred (ValidationException) when calling the Converse operation: The model returned the following errors: messages.1.content.0.type: Expected thinking or redacted_thinking, but found text. When thinking is enabled, a final assistant message must start with a thinking block (preceeding the lastmost set of tool_use and tool_result blocks). We recommend you include thinking blocks from previous turns. To avoid this requirement, disable thinking. Please consult our documentation at https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking

This is due to us not properly converting our Haystack ChatMessage objects back into the expected format by Bedrock's Converse API. We are unable to preform the conversion because we are not storing a new field called reasoningContent in our ChatMessage which contains the thinking content from Claude.

We should update our conversion of Haystack ChatMessage to Bedrock's format taking into account the thinking tokens that can be present when any assistant message is returned.

For example, here is a response from Bedrock and the corresponding Haystack ChatMessage which is missing the thinking content

Bedrock Response

response_body = {
    "ResponseMetadata": {
        "RequestId": "d7be81a1-5d37-40fe-936a-7c96e850cdda",
        "HTTPStatusCode": 200,
        "HTTPHeaders": {
            "date": "Tue, 15 Jul 2025 12:49:56 GMT",
            "content-type": "application/json",
            "content-length": "1107",
            "connection": "keep-alive",
            "x-amzn-requestid": "d7be81a1-5d37-40fe-936a-7c96e850cdda",
        },
        "RetryAttempts": 0,
    },
    "output": {
        "message": {
            "role": "assistant",
            "content": [
                {
                    "reasoningContent": {
                        "reasoningText": {
                            "text": 'The user wants to know the weather in Paris. I have a `weather` function '
                                    'available that can provide this information. \n\nRequired parameters for '
                                    'the weather function:\n- city: The city to get the weather for\n\nIn this '
                                    'case, the user has clearly specified "Paris" as the city, so I have all '
                                    'the required information to make the function call.',
                            "signature": "..."
                        }
                    }
                },
                {"text": "I'll check the current weather in Paris for you."},
                {
                    "toolUse": {
                        "toolUseId": "tooluse_iUqy8-ypSByLK5zFkka8uA",
                        "name": "weather",
                        "input": {"city": "Paris"},
                    }
                },
            ],
        }
    },
    "stopReason": "tool_use",
    "usage": {
        "inputTokens": 412,
        "outputTokens": 146,
        "totalTokens": 558,
        "cacheReadInputTokens": 0,
        "cacheWriteInputTokens": 0,
    },
    "metrics": {"latencyMs": 4811},
}

Haystack ChatMessage

chat_message = {
    "replies": [
        ChatMessage(
            _role=ChatRole.ASSISTANT,
            _content=[
                TextContent(text="I'll check the current weather in Paris for you."),
                ToolCall(tool_name="weather", arguments={"city": "Paris"}, id="tooluse_iUqy8-ypSByLK5zFkka8uA"),
            ],
            _name=None,
            _meta={
                "model": "arn:aws:bedrock:us-east-1::inference-profile/us.anthropic.claude-3-7-sonnet-20250219-v1:0",
                "index": 0,
                "finish_reason": "tool_use",
                "usage": {"prompt_tokens": 412, "completion_tokens": 146, "total_tokens": 558},
            },
        )
    ]
}

To Reproduce
We are actually able to reproduce this issue without needing to use Agent, but this could also be done using Agent.

import os

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_SESSION_TOKEN"] = ""
os.environ["AWS_DEFAULT_REGION"] = "us-east-1"

from haystack.dataclasses import ChatMessage
from haystack.tools.tool import Tool

from haystack_integrations.components.generators.amazon_bedrock.chat.chat_generator import AmazonBedrockChatGenerator


def weather(city: str):
    """Get weather for a given city."""
    return f"The weather in {city} is sunny and 32°C"


tool_parameters = {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
tool = Tool(
    name="weather",
    description="useful to determine the weather in a given location",
    parameters=tool_parameters,
    function=weather,
)

initial_messages = [ChatMessage.from_user("What's the weather like in Paris?")]
component = AmazonBedrockChatGenerator(
    model="arn:aws:bedrock:us-east-1::inference-profile/us.anthropic.claude-3-7-sonnet-20250219-v1:0",
    tools=[tool],
    generation_kwargs={
        "maxTokens": 8192,
        "thinking": {
            "type": "enabled",
            # "budget_tokens": 8192,
            "budget_tokens": 1024,
        }
    },
)
results = component.run(messages=initial_messages)

tool_call_message = results["replies"][0]

tool_calls = tool_call_message.tool_calls

# Mock the response we'd get from ToolInvoker
tool_result_messages = [
    ChatMessage.from_tool(tool_result="22° C and sunny", origin=tool_call) for tool_call in tool_calls
]

new_messages = [*initial_messages, tool_call_message, *tool_result_messages]
results = component.run(new_messages)  # <-- This returns the error

print(results)

Describe your environment (please complete the following information):

• OS: MacOS
• Haystack version: Haystack 2.15.1
• Integration version: 3.8.1

[sjrl]

@anakin87 would like your opinion on two ideas for fixes:

1. We add reasoningContent into the metadata of ChatMessage so we keep the thinking content to be able to convert the ChatMessage back into the correct format expected by Bedrock.
2. We add something like a ThinkingContent content type to ChatMessage so we can directly store it in _content.

[sjrl]

Opened a draft PR here #2094 that includes tests to replicate the issue described above.

[sjrl]

We should also check we have a similar issue in our Anthropic integration.

[anakin87]

Just my two cents.

• For the moment, I would add the thinking/reasoning content to meta
• I would also open an issue on Haystack to investigate a general way to store this information in ChatMessage (for providers which return the thinking content)

[sjrl]

@anakin87 sounds good and I agree! I opened an issue in Haystack for the investigation here