Skip to content

[FEATURE] Add all tool details under gen_ai.agent.tools attribute #1083

@kiranramnath007

Description

@kiranramnath007

Problem Statement

Requesting Strands team to emit the following tool details in the invoke_agent.metadata. attributes.gen_ai.agent.tools.

Currently, this information is only available in the tool_invocation span for the single tool that is invoked, but we need some extra details for AgentCore Evaluation to support the following evaluation needs:

  • For ToolSelectionAccuracy evaluator, we need to provide the list of all available tools to help the LLM-as-a-Judge decide if the correct tool was chosen.

  • For ToolParamaterAccuracy evaluator, we need to provide the list of all required parameters (and not just the ones sent in the request) to help the LLMJ decide if the parameters were inputted correctly

Proposed Solution

Add the following details under invoke_agent.metadata.attributes.gen_ai.agent.tools

List[ToolConfig]

where each ToolConfig consists of

  • Tool Name
  • Tool Input parameters + types
  • Tool output parameter + type
  • Tool description / docstring

Use Case

This information is required to input into the LLM-as-a-Judge prompt for AgentCoreEvaluation, and also Strands Evaluation feature.

Alternatives Solutions

No response

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions