> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming.
In theory. The only difference between today and "the aughts" is that we have machines that can spit out a ton of code very quickly.
Nothing has changed about the discipline or honesty around testing (you can skip automated tests even faster now if you wish). You can and should work with AI to write tests, but you have to know the difference between a good test and a "looks good on paper" test in order for it to truly be effective and raise the quality of what you're building.
mlmonkey
Writing unit tests used to be the bane of my existence. I used to hate them. Often times, the LoC for unit tests was 3X the LoC of the actual code.
But not any more! Now I point the LLM to the code and order it to write unit tests, covering all edge cases, etc. I'd rather spend 3 hours arguing with the LLM than writing unit tests! :-D
simianwords
Scenario testing is the new word for it and I think this is a game changer.
Two of the reasons I never liked writing tests is
- they didn’t seem to usually assert much internal logic
- they would have to be maintained along with the original code
I think scenario testing is much better instead because the actual way a person uses a feature hardly changes but the internals might change a lot.
So imagine I’m making an e-commerce website. There are lots of internal mechanisms. I’ll have an agent testing all the functionalities as if it were a customer. This gives me much much more confidence while writing code because it is more uncorellated with the code.
Tomorrow I can change a lot of internals but the testing agent stays the same.
There’s something to note though: not all code is possible to be scenario tested. Like data engineering and other things where the feedback time is huge.
show comments
wrxd
I believe this can work if done on top of traditional testing. I would feel very uneasy to replace deterministic (ok, not always but mostly) test suites with something that is not deterministic at all
> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming.
In theory. The only difference between today and "the aughts" is that we have machines that can spit out a ton of code very quickly.
Nothing has changed about the discipline or honesty around testing (you can skip automated tests even faster now if you wish). You can and should work with AI to write tests, but you have to know the difference between a good test and a "looks good on paper" test in order for it to truly be effective and raise the quality of what you're building.
Writing unit tests used to be the bane of my existence. I used to hate them. Often times, the LoC for unit tests was 3X the LoC of the actual code.
But not any more! Now I point the LLM to the code and order it to write unit tests, covering all edge cases, etc. I'd rather spend 3 hours arguing with the LLM than writing unit tests! :-D
Scenario testing is the new word for it and I think this is a game changer.
Two of the reasons I never liked writing tests is
- they didn’t seem to usually assert much internal logic
- they would have to be maintained along with the original code
I think scenario testing is much better instead because the actual way a person uses a feature hardly changes but the internals might change a lot.
So imagine I’m making an e-commerce website. There are lots of internal mechanisms. I’ll have an agent testing all the functionalities as if it were a customer. This gives me much much more confidence while writing code because it is more uncorellated with the code.
Tomorrow I can change a lot of internals but the testing agent stays the same.
There’s something to note though: not all code is possible to be scenario tested. Like data engineering and other things where the feedback time is huge.
I believe this can work if done on top of traditional testing. I would feel very uneasy to replace deterministic (ok, not always but mostly) test suites with something that is not deterministic at all