The point about paying to store traces nobody reads hit home. We had the same problem with customer interviews. We did 8 of them and the notes just sat in a doc until we tagged each problem by how many customers it affected. Curious how you decide which flagged errors matter most. Is it how often they show up across traces, or do you weight them by user impact?
dmos62
How do you reuse traces as benchmarks (eval datasets)? Is it about being able to synthesise an agent session from the trace, and having another agent watch the first agent's behavior? Having trouble wrapping my mind around all the little details.
Edit: I initially asked also about general tips for observing agents, but later, noticing that there were no interactions with my comment yet, edited my comment to remove that question, because I felt that it was too broad and would detract from my main question. To my surprise, both were answered by op after my edit; kudos!
show comments
ghm2180
Very cool. I liked the high level overview where you can view it like do a funnel analysis for failures. Good for fast decision making.
I wonder if you have thought of using this for the voice AI space for creating traces from many test calls to speed up testing of voice bots? It's true there are a lot of challenges _before_ one gets to the stage of running actual calls like creating good test data etc, but I wonder if you see a definite role for this framework in that space?
show comments
danielmarkbruce
To answer your question - we use cheaper models to just flag if something looks obviously flawed. That covers the vast majority of things because it's the propagation of obviously stupid stuff that causes a problem for our system. To the extent we are doing something sophisticated, we use an appropriate architecture, like it's built into the system.
show comments
SantiDev
Congrats on the product! I love the design of the website and the app. Curious whether it was made in-house, by an external design agency, with AI models, or something else
show comments
BoorishBears
You should try the bullet points then earning someone reading the rest, that's where I bounced.
The point about paying to store traces nobody reads hit home. We had the same problem with customer interviews. We did 8 of them and the notes just sat in a doc until we tagged each problem by how many customers it affected. Curious how you decide which flagged errors matter most. Is it how often they show up across traces, or do you weight them by user impact?
How do you reuse traces as benchmarks (eval datasets)? Is it about being able to synthesise an agent session from the trace, and having another agent watch the first agent's behavior? Having trouble wrapping my mind around all the little details.
Edit: I initially asked also about general tips for observing agents, but later, noticing that there were no interactions with my comment yet, edited my comment to remove that question, because I felt that it was too broad and would detract from my main question. To my surprise, both were answered by op after my edit; kudos!
Very cool. I liked the high level overview where you can view it like do a funnel analysis for failures. Good for fast decision making.
I wonder if you have thought of using this for the voice AI space for creating traces from many test calls to speed up testing of voice bots? It's true there are a lot of challenges _before_ one gets to the stage of running actual calls like creating good test data etc, but I wonder if you see a definite role for this framework in that space?
To answer your question - we use cheaper models to just flag if something looks obviously flawed. That covers the vast majority of things because it's the propagation of obviously stupid stuff that causes a problem for our system. To the extent we are doing something sophisticated, we use an appropriate architecture, like it's built into the system.
Congrats on the product! I love the design of the website and the app. Curious whether it was made in-house, by an external design agency, with AI models, or something else
You should try the bullet points then earning someone reading the rest, that's where I bounced.
[dead]
[dead]