feature: Agent Retrieved Context #157
Description
Request
The ability to return data from a frontend tool using useRegisterFrontendTool or similar back to the Mastra Agent so that the agent can retrieve context on-demand.
In my view this would be akin to a "reverse mention" in the Cedar state system since mentions are mechanisms for the User to subscribe the discrete context that they need for the task.
On the other hand, the Agent can use a similar system to retrieve discrete context from the frontend when it needs it. For example chart data, useRef data or even HTML tags that should not be a constant subscription.
An Agent equipped with these capabilities can behave similar to a terminal or coding agent with access to the filesystem. Where instead of adding the entire codebase into context or vectorizing it, the agent instead calls tools to read the data it finds necessary.
Use Case
Slow data and large data. Potentially so large it could pollute the context if always used and should only be available when needed by mention or agent request.
Slow data may come in the form of network calls to other systems or serial operations that can only be performed from an embedding frontend. If you subscribe to this data directly you need to pause the chat for long periods or risk race conditions between messages and async data.
In either scenario, it's best to let Users or Agents decide when this data is needed.
User & Agent Flow
- User asks Agent to filter dashboard and change parameters. Then inspect a few charts and answer a question
- The Agent calls state setting agentic actions to filter and parameterize the view
- The Agent then calls tools to inspect each chart to get data that is correctly filtered
- The Agent gets the data back from each chart it inspected and generates an answer