[BUG]: Tool calling not detected in Phoenix evals when using Gemini or Ollama (works correctly with OpenAI)

[vigneshrajaswarise]

Where do you use Phoenix

Self-hosted

What version of Phoenix are you using?

11.21.0

What happened?

When running Phoenix locally with Flowise local setup and connecting Phoenix to evaluate pre-build tool-calling behavior, the evaluation works as expected with OpenAI models. Tool calls are detected and scored correctly.

However, when switching to Gemini or Ollama models:

 1.The models successfully produce tool calls (verified in logs).

 2.The tools are executed correctly.

 3.Phoenix eval reports that no tool was called, leading to inaccurate evaluation results.

This issue seems specific to Phoenix’s tool-call detection logic for non-OpenAI models.

Additional information

Steps to Reproduce

    1.Run Phoenix locally with Flowise setup.

    2.Connect Phoenix to a local LLM (Gemini or Ollama).

    3.Configure a tool (e.g., search, calculator).

    4.Run evaluation for tool calling.

    5.Observe:

             OpenAI → tool calls detected correctly.

             Gemini/Ollama → tool calls executed but Phoenix eval shows tool not called.

[mikeldking]

@vigneshrajaswarise - Thanks for the issue. Given that it's flowise related I'm going to move it to our instrumentation project. However in general the phoenix integration sits in the flowise codebase and not in phoenix so let us know if there's anything we can do to coordinate.

[mikeldking]

cc @ialisaleh any ideas here why tool calling is not instrumented for ollama?

[ialisaleh]

Hi @vigneshrajaswarise - Thanks for raising this issue.

Let me clarify what is happening with Flowise & Phoenix instrumentation across different models.

In Flowise, we use both manual & auto instrumentation for Arize & Phoenix.

Manual Instrumentation

We have an AnalyticHandler class for the OpenAI Assistant component in Flowise. This is where we explicitly capture traces for CHAIN, LLM & TOOL calls from OpenAI models.

Auto Instrumentation

For other cases, Flowise uses the additionalCallbacks method for tracing. Specifically, we use the LangChainInstrumentation class from the Arize NPM package named @arizeai/openinference-instrumentation-langchain for Arize & Phoenix setup, which currently only implements tracing with the OpenAI module. This is likely the reason Phoenix is missing the tool call traces when directly using Gemini or Ollama.

That said, I tested Gemini tool calls with the Tool Agent component setup in Flowise, and in that project, I do get the proper tool call traces. I have attached the screenshots for that.

The difference comes down to how the tools are implemented. If the tools are built on top of LangChain components, then traces are captured correctly. If they bypass LangChain’s standard callbacks, like using non-OpenAI LLMs like Gemini or Ollama directly, then Phoenix may not receive the tool call spans unless additional instrumentation is added.

Workaround

If you run Gemini with the Tool Agent component setup where tool calls flow through LangChain APIs, you should see the tool traces as expected. This means the evaluation is not fundamentally broken in Phoenix. It is just a gap in the callback handler coverage for non-OpenAI models in the Arize NPM package.

If you can export & share your Chatflow project, then I can debug this issue properly. Thanks!

Flowise Phoenix / Gemini Tool Trace

Flowise Phoenix / Gemini LLM Trace

cc: @mikeldking @caroger