API to manage debugging info (steps, full request/responses)

[apostolisCodpal]

"convex": "^1.23.0"
"@convex-dev/agent": "0.0.15-alpha.2"

This is not urgent, but would become a problem over time.
I couldn't find an API for deleting the steps in the steps table.
Calling components.agent.messages.deleteAllForThreadIdAsync or the Sync version deletes all messages and embeddings. It makes sense to have the steps separately but there should still be an API to delete them. Perhaps by deleted threads.

[ianmacartney]

Agreed! I actually think we should get rid of steps, so long as there's sufficient data available on the messages table. I'm not sure if there are any remaining fields of value.

I avoided surfacing a way to get the steps or manage them b/c I wanted them to be an implementation detail I could change. But not deleting the steps when deleting the thread is a bug, and exposing vacuuming steps would be nice since they're somewhat heavy-weight.

Do you get anything special out of steps? Any objection to phasing them out?

[apostolisCodpal]

Steps show the real conversation thread sent to the model, together will exact tool specs etc. It is helping me tremendously for investigation purposes. I don't think I would ever have a need for it other than for investigations though.

[ianmacartney]

What if those were logged extensively so you could capture and investigate and even copy the json and turn the logging on/off when you aren’t investigating? Or have a flag to dump all requests and responses in full to a dedicated table?

…

On Sat, May 10, 2025 at 1:19 PM apostolisCodpal ***@***.***> wrote: *apostolisCodpal* left a comment (get-convex/agent#42) <#42 (comment)> Steps show the real conversation thread sent to the model, together will exact tool specs etc. It is helping me tremendously for investigation purposes. I don't think I would ever have a need for it other than for investigations though. — Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACZQW65Q4NDVHIPVRSBARD25ZGLXAVCNFSM6AAAAAB4ZCTPFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNRZGEYTSNZQGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[apostolisCodpal]

Those are great suggestions. Database might be easier for a "quick" lookup but logs are lighter and we have integrations that can stream them outside of convex so that direction might make more sense.

LLMs can be quite unpredictable at times as-it-is. Having the ability to log/dump the real JSON and know that you see what they see is crucial to understanding seemingly "unexpected" behaviour.

[ianmacartney]

I just had a thought that puts a bit more work on the user, but gives more control over the behavior without the library needing to be overly opinionated / configurable:

const agent = new Agent(components.agent, {
  ...
  onLLMResponse: async (ctx, { request, response, agentName, userId, threadId, ...? }) => {
    // log it in your preferred format to introspect yourself
    console.log({ request, response });
    // and/or dump to a table conditionally
    if (process.env.CAPTURE_RAW_REQUESTS) {
      await ctx.db.insert("rawRequests", { request, response });
    }
  }
});

Not sure about the naming for onLLMResponse.. open to suggestions

[apostolisCodpal]

That's a great thought @ianmacartney.
Exposing hooks like that is far more powerful than exposing configuration.
The name is good. The only other name I think of is onAfterGenerate.

If you are doing with this direction I would urge to include also onBeforeGenerate or onBeforeLLMResponse to allow the user to process the request before sending it to the provider.
What I would specifically like to achieve with this (maybe it fits better somewhere else) is the ability to strip text from messages, or messages as a whole, before they are sent to the LLM.

[ianmacartney]

Pre-processing messages is possible at the start of the {generate,stream}{Text,Object} calls, but if you provide maxSteps then we currently only have as much control as the AI SDK exposes - and they don't currently have a callback per-step that modifies the in-progress messages.
However, it sounds like you'd be happy with a callback that did common filtering of the context messages at the very start (or messages in general), maybe even as a parameter in contextOptions. Each subsequent step would have the new messages and not per-step adjustments, however.
wdyt?

[apostolisCodpal]

I need to work with it a bit more to formulate a stronger opinion. Let's leave this for now, if I see any limitations that can be resolved I'll open a separate ticket.