[bug] missing llm input messages in smolagents

[andstor]

Describe the bug
"Input Messages" are no longer available in traces from smolagents.

As can be seen from the following image, only "Input" details in JSON are available.

This is hard to inspect and follow. The LLM Input Messages should therefore also be available, showing the chat messages used in one turn. Following image show desired output.

Environment

• openai==1.101.0
• smolagents==1.21.2
• opentelemetry-sdk==1.36.0
• opentelemetry-exporter-otlp==1.36.0
• openinference-instrumentation-smolagents==0.1.16

[dosubot]

This was a known issue with smolagents instrumentation: LLM input messages were missing from traces due to double instrumentation of LLM spans, which caused input message attributes to be lost. This was fixed in openinference-instrumentation-smolagents version 0.1.16 (PR #2118), which prevents duplicate LLM spans and restores proper handling of input messages.

With 0.1.16, LLM input messages should appear in spans with attributes like llm.input_messages.0.message.role and llm.input_messages.0.message.content as long as the input payload includes a messages key (the standard for chat LLMs) and the span kind is set to LLM (source). If you still only see raw JSON input, make sure you are looking at the LLM-level spans (not higher-level agent/tool spans), as input messages are only captured for LLM spans, not for parent agent/tool spans (source).

If you are on 0.1.16 and still missing input messages, double-check that your LLM calls use the messages key in their input and that you are inspecting the correct spans in your trace viewer.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[axiomofjoy]

Do you have a code snippet to reproduce the issue @andstor? I have not been able to reproduce.

[axiomofjoy]

It looks like you're using Qwen. Wondering if the issue also appears for OpenAI?

[andstor]

@axiomofjoy, I was running Qwen from vLLM.

Here is a bare-bones example using the ConsoleSpanExporter.

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
from openinference.instrumentation.smolagents import SmolagentsInstrumentor
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
SmolagentsInstrumentor().instrument(tracer_provider=tracer_provider)

from smolagents import OpenAIServerModel
model = OpenAIServerModel(
    model_id="gemini-2.5-flash",
    api_base="https://generativelanguage.googleapis.com/v1beta/openai/",
    api_key="API_KEY"
)

from smolagents import CodeAgent
agent = CodeAgent(tools=[], model=model)

agent.run("What is 1 + 1?")

This outputs:

╭──────────────────────────────────────────────────── New run ────────────────────────────────────────────────────╮
│                                                                                                                 │
│ What is 1 + 1?                                                                                                  │
│                                                                                                                 │
╰─ OpenAIServerModel - gemini-2.5-flash ──────────────────────────────────────────────────────────────────────────╯
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Step 1 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
{
    "name": "OpenAIServerModel.generate",
    "context": {
        "trace_id": "0x889384ee2ae8d609cdd4427e4051cb87",
        "span_id": "0x4ea6cea87aba8763",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0xc7e6003b8f353853",
    "start_time": "2025-08-25T17:43:39.276403Z",
    "end_time": "2025-08-25T17:43:40.698856Z",
    "status": {
        "status_code": "OK"
    },
    "attributes": {
        "input.mime_type": "application/json",
        "input.value": "{\"messages\": [\"ChatMessage(role=<MessageRole.SYSTEM: 'system'>, content=[{'type': 'text', 'text': 'You are an expert assistant ...'}], tool_calls=None, raw=None, token_usage=None)\", \"ChatMessage(role=<MessageRole.USER: 'user'>, content=[{'type': 'text', 'text': 'New task:\\\\nWhat is 1 + 1?'}], tool_calls=None, raw=None, token_usage=None)\"], \"stop_sequences\": [\"Observation:\", \"Calling tools:\", \"</code>\"], \"response_format\": null, \"tools_to_call_from\": null, \"kwargs\": {}}",
        "llm.invocation_parameters": "{}",
        "llm.token_count.prompt": 2064,
        "llm.token_count.completion": 16,
        "llm.token_count.total": 2080,
        "llm.model_name": "gemini-2.5-flash",
        "llm.output_messages.0.message.role": "assistant",
        "llm.output_messages.0.message.contents.0.message_content.type": "text",
        "llm.output_messages.0.message.contents.0.message_content.text": "<code>\nresult = 1 + 1\nfinal_answer(result)\n",
        "output.mime_type": "application/json",
        "output.value": "{\"role\": \"assistant\", \"content\": \"<code>\\nresult = 1 + 1\\nfinal_answer(result)\\n\", \"tool_calls\": null, \"token_usage\": {\"input_tokens\": 2064, \"output_tokens\": 16, \"total_tokens\": 2080}}",
        "openinference.span.kind": "LLM"
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.36.0",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}
 ─ Executing parsed code: ──────────────────────────────────────────────────────────────────────────────────────── 
  result = 1 + 1                                                                                                   
  final_answer(result)                                                                                             
 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
{
    "name": "FinalAnswerTool",
    "context": {
        "trace_id": "0x889384ee2ae8d609cdd4427e4051cb87",
        "span_id": "0x3bd80dde05c76410",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0xc7e6003b8f353853",
    "start_time": "2025-08-25T17:43:40.862481Z",
    "end_time": "2025-08-25T17:43:40.862576Z",
    "status": {
        "status_code": "OK"
    },
    "attributes": {
        "input.value": "{\"args\": [2], \"sanitize_inputs_outputs\": false, \"kwargs\": {}}",
        "tool.name": "final_answer",
        "tool.description": "Provides a final answer to the given problem.",
        "tool.parameters": "{\"answer\": {\"type\": \"any\", \"description\": \"The final answer to the problem\"}}",
        "openinference.span.kind": "TOOL"
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.36.0",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}
Final answer: 2
[Step 1: Duration 1.60 seconds| Input tokens: 2,064 | Output tokens: 16]
{
    "name": "CodeAgent.run",
    "context": {
        "trace_id": "0x889384ee2ae8d609cdd4427e4051cb87",
        "span_id": "0xc7e6003b8f353853",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": null,
    "start_time": "2025-08-25T17:43:39.264461Z",
    "end_time": "2025-08-25T17:43:46.463074Z",
    "status": {
        "status_code": "OK"
    },
    "attributes": {
        "input.value": "{\"task\": \"What is 1 + 1?\", \"stream\": false, \"reset\": true, \"images\": null, \"additional_args\": null, \"max_steps\": null}",
        "smolagents.max_steps": 20,
        "smolagents.tools_names": [
            "final_answer"
        ],
        "llm.token_count.prompt": 2064,
        "llm.token_count.completion": 16,
        "llm.token_count.total": 2080,
        "output.value": "2",
        "openinference.span.kind": "AGENT"
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.36.0",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}

As you see the llm.input_messages are missing.

Here is the same log using #2059:

╭──────────────────────────────────────────────────── New run ────────────────────────────────────────────────────╮
│                                                                                                                 │
│ What is 1 + 1?                                                                                                  │
│                                                                                                                 │
╰─ OpenAIServerModel - gemini-2.5-flash ──────────────────────────────────────────────────────────────────────────╯
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Step 1 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
{
    "name": "OpenAIServerModel.generate",
    "context": {
        "trace_id": "0x41415405b09bae551269f989bfe71bbf",
        "span_id": "0xf1778c5e8969cf4b",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0x6f39d36ebee71ded",
    "start_time": "2025-08-25T17:45:19.509522Z",
    "end_time": "2025-08-25T17:45:21.995611Z",
    "status": {
        "status_code": "OK"
    },
    "attributes": {
        "input.mime_type": "application/json",
        "input.value": "{\"messages\": [\"ChatMessage(role=<MessageRole.SYSTEM: 'system'>, content=[{'type': 'text', 'text': 'You are an expert assistant ...'}], tool_calls=None, raw=None, token_usage=None)\", \"ChatMessage(role=<MessageRole.USER: 'user'>, content=[{'type': 'text', 'text': 'New task:\\\\nWhat is 1 + 1?'}], tool_calls=None, raw=None, token_usage=None)\"], \"stop_sequences\": [\"Observation:\", \"Calling tools:\", \"</code>\"], \"response_format\": null, \"tools_to_call_from\": null, \"kwargs\": {}}",
        "llm.invocation_parameters": "{}",
        "llm.input_messages.0.message.role": "system",
        "llm.input_messages.0.message.contents.0.message_content.type": "text",
        "llm.input_messages.0.message.contents.0.message_content.text": "You are an expert assistant ...",
        "llm.input_messages.1.message.role": "user",
        "llm.input_messages.1.message.contents.0.message_content.type": "text",
        "llm.input_messages.1.message.contents.0.message_content.text": "New task:\nWhat is 1 + 1?",
        "llm.token_count.prompt": 2064,
        "llm.token_count.completion": 16,
        "llm.token_count.total": 2080,
        "llm.model_name": "gemini-2.5-flash",
        "llm.output_messages.0.message.role": "assistant",
        "llm.output_messages.0.message.contents.0.message_content.type": "text",
        "llm.output_messages.0.message.contents.0.message_content.text": "<code>\nresult = 1 + 1\nfinal_answer(result)\n",
        "output.mime_type": "application/json",
        "output.value": "{\"role\": \"assistant\", \"content\": \"<code>\\nresult = 1 + 1\\nfinal_answer(result)\\n\", \"tool_calls\": null, \"token_usage\": {\"input_tokens\": 2064, \"output_tokens\": 16, \"total_tokens\": 2080}}",
        "openinference.span.kind": "LLM"
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.36.0",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}
 ─ Executing parsed code: ──────────────────────────────────────────────────────────────────────────────────────── 
  result = 1 + 1                                                                                                   
  final_answer(result)                                                                                             
 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
{
    "name": "FinalAnswerTool",
    "context": {
        "trace_id": "0x41415405b09bae551269f989bfe71bbf",
        "span_id": "0x10704dc6f954b856",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0x6f39d36ebee71ded",
    "start_time": "2025-08-25T17:45:22.302464Z",
    "end_time": "2025-08-25T17:45:22.302611Z",
    "status": {
        "status_code": "OK"
    },
    "attributes": {
        "input.value": "{\"args\": [2], \"sanitize_inputs_outputs\": false, \"kwargs\": {}}",
        "tool.name": "final_answer",
        "tool.description": "Provides a final answer to the given problem.",
        "tool.parameters": "{\"answer\": {\"type\": \"any\", \"description\": \"The final answer to the problem\"}}",
        "openinference.span.kind": "TOOL"
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.36.0",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}
Final answer: 2
[Step 1: Duration 2.83 seconds| Input tokens: 2,064 | Output tokens: 16]
{
    "name": "CodeAgent.run",
    "context": {
        "trace_id": "0x41415405b09bae551269f989bfe71bbf",
        "span_id": "0x6f39d36ebee71ded",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": null,
    "start_time": "2025-08-25T17:45:19.428922Z",
    "end_time": "2025-08-25T17:45:27.642325Z",
    "status": {
        "status_code": "OK"
    },
    "attributes": {
        "input.value": "{\"task\": \"What is 1 + 1?\", \"stream\": false, \"reset\": true, \"images\": null, \"additional_args\": null, \"max_steps\": null}",
        "smolagents.max_steps": 20,
        "smolagents.tools_names": [
            "final_answer"
        ],
        "llm.token_count.prompt": 2064,
        "llm.token_count.completion": 16,
        "llm.token_count.total": 2080,
        "output.value": "2",
        "openinference.span.kind": "AGENT"
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.36.0",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}

[axiomofjoy]

Thanks @andstor. I suspect this issue is specific to vLLM.

[andstor]

Well, not vLLM directly, but it might be specific to OpenAI-compatible API servers, yes. In practice, this would mean the OpenAIServerModel from smolagents is incorrectly instrumented.