-
Notifications
You must be signed in to change notification settings - Fork 451
Description
Problem Statement
Requesting Strands team to emit the following tool details in the invoke_agent.metadata. attributes.gen_ai.agent.tools.
Currently, this information is only available in the tool_invocation span for the single tool that is invoked, but we need some extra details for AgentCore Evaluation to support the following evaluation needs:
-
For ToolSelectionAccuracy evaluator, we need to provide the list of all available tools to help the LLM-as-a-Judge decide if the correct tool was chosen.
-
For ToolParamaterAccuracy evaluator, we need to provide the list of all required parameters (and not just the ones sent in the request) to help the LLMJ decide if the parameters were inputted correctly
Proposed Solution
Add the following details under invoke_agent.metadata.attributes.gen_ai.agent.tools
List[ToolConfig]
where each ToolConfig consists of
- Tool Name
- Tool Input parameters + types
- Tool output parameter + type
- Tool description / docstring
Use Case
This information is required to input into the LLM-as-a-Judge prompt for AgentCoreEvaluation, and also Strands Evaluation feature.
Alternatives Solutions
No response
Additional Context
No response