E2E tests are now quick to write due to LLMs, and are then deterministic AND cheap to run. How would this compare to the token costs of running an agent the whole time for each test? How do you make sure results stay stable regardless of the nondeterministic nature? Do customers still need to create test cases - any way to import from test case management system - based on which they could have already generate e2e tests locally?
show comments
dbbk
"Traditional E2E tests are slow to set up and expensive to maintain." I don't really understand this. If I'm already using Opus to write the code, surely it would know best what E2E tests to write to be able to verify its own output? This seems like an unnecessary external step.
show comments
RayFitzgerald
Love your approach to product.
It feels like TesterArmy will become the "Vercel for testing".
Refreshing stuff!
show comments
msencenb
Have you been able to nail down a loop where your tool can take an open pr, guess the code path and do some testing?
We use cypress heavily for our core flows which has a similar ai prompt thing but it’s not quite ad hoc enough for smaller fixes which is where the bottleneck still comes in for us.
I've been experimenting with Revyl and it's really nice. I think this agent-driven testing is the future.
show comments
Laurel1234
Seems interesting, but I wonder about this
> Traditional E2E tests are slow to set up and expensive to maintain.
Isn't this just using agents to create e2e tests or is there some better new approach I'm missing?
show comments
j0sip
I wonder how does it compare to mobileboost.io, which has been used by some companies like Duolingo?
show comments
yohguy
Does it work of mobile native applications or expo apps that have native modules?
Pricing question, the usage on the plans seems low considering in the demo you said that you have 25 tests per pr which would mean you get only 10 PRs per month on the hobby plan?
show comments
zuzululu
not sure the pain point you mentioned resonate. with LLMs its very easy to do E2E testing. also I feel uneasy about outsourcing this part with all the security issues these days.
show comments
rpunkfu
Congratulations on launch, I’ve been tracking your progress since you’ve been accepted for spring batch.
E2E tests are now quick to write due to LLMs, and are then deterministic AND cheap to run. How would this compare to the token costs of running an agent the whole time for each test? How do you make sure results stay stable regardless of the nondeterministic nature? Do customers still need to create test cases - any way to import from test case management system - based on which they could have already generate e2e tests locally?
"Traditional E2E tests are slow to set up and expensive to maintain." I don't really understand this. If I'm already using Opus to write the code, surely it would know best what E2E tests to write to be able to verify its own output? This seems like an unnecessary external step.
Love your approach to product. It feels like TesterArmy will become the "Vercel for testing". Refreshing stuff!
Have you been able to nail down a loop where your tool can take an open pr, guess the code path and do some testing?
We use cypress heavily for our core flows which has a similar ai prompt thing but it’s not quite ad hoc enough for smaller fixes which is where the bottleneck still comes in for us.
I'm curious how your mobile testing compares to https://revyl.com
I've been experimenting with Revyl and it's really nice. I think this agent-driven testing is the future.
Seems interesting, but I wonder about this
> Traditional E2E tests are slow to set up and expensive to maintain.
Isn't this just using agents to create e2e tests or is there some better new approach I'm missing?
I wonder how does it compare to mobileboost.io, which has been used by some companies like Duolingo?
Does it work of mobile native applications or expo apps that have native modules?
Pricing question, the usage on the plans seems low considering in the demo you said that you have 25 tests per pr which would mean you get only 10 PRs per month on the hobby plan?
not sure the pain point you mentioned resonate. with LLMs its very easy to do E2E testing. also I feel uneasy about outsourcing this part with all the security issues these days.
Congratulations on launch, I’ve been tracking your progress since you’ve been accepted for spring batch.
Always happy to see cool products from Poland! :)
.army?