Our version has its own TUI and was intended for semi-adversarial scenarios like OSS contribution and enterprise. Thus a lot of anti-cheat tech under the covers (including invalid questions).
My guess is that a lot of folks have built something in this vein. It really made sense earlier in the coding agents cycle. These days, I dunno. Seems like we need to more fundamentally rethink the relationship between coding agents and human understanding of the implementation.
theParadox42
I certainly feel like certain AI coding practices I have leave me with nothing learned in a frustrating way. Sometimes the most effective way to getting what I want is just describing a goal really clearly, and then bashing the AI until it gets there. That bashing process can take a lot of energy and yet I don’t gain useful skills or a “better way” that I would if I was still coding. I generate more training data for BigAI corp, but that’s a slow return cycle.
Not sure if this is the right solution, but I would like to learn/understand/grow more as I pursue reckless efficiency.
show comments
fwlr
It’s a neat idea, i think it’ll really take off if you change “before pushing to prod” to “before requesting another human’s time or attention”. If something like this was gatekeeping the “submit PR” and “update PR” buttons it would prevent a lot of waste.
show comments
pjmlp
The gimmicks we come around to justify existing practices being pushed into the industry never stops surprising me.
One would expect the actual answer would be to write the code ourselves.
rDr4g0n
my technique to get a similar benefit is to write a quality PR description.
a good PR description should be compact and helpful. describing something complex in a compact and approachable way requires understanding.
the process of writing the description often leads me to understand hidden decisions the LLM made and occasionally uncovers bugs.
my PR descriptions separately answer the following questions:
Why is this change being made? a sentence or two about the problem, objective, etc, and a link to more detail (tickets, docs, etc) if available
What does this change accomplish? typically describes observable changes, requirements, etc. crucially this section avoids describing how the change is implemented. the act of separating what from how isn't new, but it's crucial in the LLM era
How does this change accomplish its goal? the answer to this question is "read the code", but the job of a PR description is to provide a quick hand drawn map before dumping the reviewer into the jungle. drawing a good "how" map requires understanding the jungle and the new path through it.
the risk with LLMs (or any outsource) doing the work is the loss of understanding. this exercise encourages understanding.
Crowberry
Where is that quote from, dis you just make it up?
> You can outsource your thinking,
but not your understanding.
— Sun Tzu
show comments
chacha-bong
Lol I built this and immediately got frustrated with it and stopped using it. I think I don't like being questioned by someone who I expect to work like magic.
bluerooibos
What's with Claude preferring cursive and italic font in landing page hero sections?
Looks absolutely awful.
alexwrich
Nice idea. I like that it tests whether the person understands the change instead of adding another AI reviewer. We added the tests before we switch to fully agentic AI development using automo - ie we tested every member of staff and now trust them to be the eyes.
Disclosure: I’m building automo.build fully to fix this.
lowkeyokay
I actually made this as a skill last week. It is helpful. Especially when reviewing a PR with 1500 LOC from a non-dev. It keeps me focused an helps me do a structured review. But it does not catch poor decisions, like calling the wring endpoint a such.
jamienk
I can kinda understand the motivation here: you push a bunch of slop (or is it deep, genius code?) but you have a kind of nagging suspicion that you should get to know it more familiarly, like back in the good old days a few months back when you sort of understood what was going on. I... don't... think this is going to work...? Remember Confirmation Fatigue? Soon we will have agents that interact with Sphinx for us.
gandhiandy69
We went from micromanaging code to micromanaging tools and markdowns.
maximg68
After noticing a drop in my understanding of the code I ship, I built and started using a similar skill ("Quiz me on the PR or the current branch"). I didn't tie it to a specific commit, though - to me it makes more sense to look at the PR as a whole.
So far I was positively surprised with the questions it could come up with.
freshnode
Have you ever heard the idiom "ambulance at the bottom of the cliff"?
shigosag
Love this idea, it’s way too easy to get lazy with AI, so having a quick reality check like this is actually a super healthy habit to build
sshah03
I considered doing this with an open source app I have that let's you get a visual summary on a PR + get quizzed on it, but ultimately decided it's better to keep it an optional thing for people that want to understand AI PRs rather than force it as a gate, the volume is slowly becoming too high for even that. I agree with the sentiment behind this though as I felt this was needed as well.
show comments
SDeniz
I think this could be really useful. Any chance you could add native Windows support?
aavisangle
I ask the question to agent from separate session without history to know more about my code
bodash
Why stop there? Combine this with other knowledge testing methods:
pre-code: planning mode
coding: humantodo.dev
after-code: sphinx hook
man_luke
So who is the master of whom?
nkmnz
It’s ironic that all negative comments on here are dismissive because “slop’s baaaaad”. Yes, slop is bad. Carelessness is bad. Lack of understanding is bad. It always has been. Code has been copied from stackoverflow. Built-in methods or std libraries have been used in a wrong way. Edge cases haven’t been covered. Rate-limiting and other hardening tasks haven’t been taken care of until the system broke. The code wasn’t DRY.
Now, people are building tools that not only hide the solutions to these problems in some obscure coding tool, markdown file, or agent harness, but they build gates that help developers to elevate themselves from the darkness of not knowing better. The number of developers roughly doubles every ~3 years, so at any given time, half of the people building software have less than 3 years of experience. Most of them didn’t even study CS before starting their careers. Tools like this will be the new warning voice that told people like us 5 to 10 years ago to never copy code from stackoverflow that you didn’t understand. It’s great.
show comments
thomask1995
I think this would be killer for OSS maintainers.
show comments
reddmachine
I think this should kick in before any code is written not before any code is pushed. It's an odd thing to check before pushing new code.
Also, from my experience implementing something similar, an MCQ style makes people guess without really understanding.
sailorganymede
Honestly I love to see people working on this problem ! I actually built a similar project myself (https://github.com/bhurghundii/vibecheck.dev) except it prevents you from merging on the PR level. I’d love to get in touch to talk about it sometimes.
Edit: I see there are a lot of people questioning the value of this. The thing is that there are teams where people are just pumping tons of AI code without understanding their work (literally take the ticket and dump it) and shipping it. culture change is the right answer but v hard but this at least slows things down and is quick to implement
show comments
ericand
Humans are now test subjects in the ci pipeline
hypfer
I mean I see the idea, but at least I use LLMs for code generation in two distinct ways.
The first is when I care about the code. That's when I actually read what it produces and force it to use my mental model of how things should be.
The second is when I just need a solution that sorta fits the requirements provided. Then I don't read the code because I really do not care.
So this hook would either ask me things I know, or it would ask me things I do not care about.
But I guess this is very workflow-dependent.
show comments
cjimesh
Its the right problem statement, but we normally let the code review process take care of this.
compounding-err
This is cute and novel, but practically useless. For those who use AI efficiently and effectively, this is useless. For those who don't, this becomes friction they don't want. Who really needs this?
rglover
All of this to just avoid knowing what you're doing.
Folks, this is the wrong direction. Use AI, but ffs take the time to properly learn what you're doing. We don't need to speed run civilizational collapse because we can't understand the concept of patience leading to better outcomes.
You can tell people are increasingly full of shit because they're doing this under the pretense that the problem is creating software faster, when really, a lot of people need to quit optimizing process and develop some skill and rigor. Most of the things that slow down software reaching production are human-created problems like "meetings," counting lines of code, or sending everyone to yet-another-conference that teaches them nothing (but boy does it look good on social media). It's all time waste and theater. And for what? So you feel included?
Pathetic. Learn to do this properly or get out of this industry.
Your slop generator should check if a name exist before claiming it.
show comments
jan_m_savage
a tool written by AI (mostly) to check code written by (possibly other) AI.
however, it still feels like a lot of work. what if AI hallucinates during the check?
show comments
jdthedisciple
Whenever I stop for a moment to think about it:
It is absolutely batsh*t crazy insane what SWE has come to in the last cple years.
And yet part of me is loving it.
altmanaltman
People who care about the problem will definitely not be using this as a method to "understand" their code. People who don't care about this problem will continue to not care.
So, who is this for?
princejain756
The quiz axis is interesting because it tests the person, not the diff. The complementary failure mode is the stuff you can pass a quiz on while still missing: a leftover debug print, an assertion weakened to match the bug it was meant to catch, an import of a package that never existed. Those live in the diff, not in comprehension. I hit this auditing my own patches last week - I could explain every line and still shipped a stdin read that broke on non-blocking pipes. Different catches, probably need both.
We built something quite similar 6 months ago called PEAK (Proof of Effort And Knowledge). Blog post and YT video here: https://substack.com/@jeffksmithjr/p-195763168
Our version has its own TUI and was intended for semi-adversarial scenarios like OSS contribution and enterprise. Thus a lot of anti-cheat tech under the covers (including invalid questions).
My guess is that a lot of folks have built something in this vein. It really made sense earlier in the coding agents cycle. These days, I dunno. Seems like we need to more fundamentally rethink the relationship between coding agents and human understanding of the implementation.
I certainly feel like certain AI coding practices I have leave me with nothing learned in a frustrating way. Sometimes the most effective way to getting what I want is just describing a goal really clearly, and then bashing the AI until it gets there. That bashing process can take a lot of energy and yet I don’t gain useful skills or a “better way” that I would if I was still coding. I generate more training data for BigAI corp, but that’s a slow return cycle.
Not sure if this is the right solution, but I would like to learn/understand/grow more as I pursue reckless efficiency.
It’s a neat idea, i think it’ll really take off if you change “before pushing to prod” to “before requesting another human’s time or attention”. If something like this was gatekeeping the “submit PR” and “update PR” buttons it would prevent a lot of waste.
The gimmicks we come around to justify existing practices being pushed into the industry never stops surprising me.
One would expect the actual answer would be to write the code ourselves.
my technique to get a similar benefit is to write a quality PR description.
a good PR description should be compact and helpful. describing something complex in a compact and approachable way requires understanding.
the process of writing the description often leads me to understand hidden decisions the LLM made and occasionally uncovers bugs.
my PR descriptions separately answer the following questions:
Why is this change being made? a sentence or two about the problem, objective, etc, and a link to more detail (tickets, docs, etc) if available
What does this change accomplish? typically describes observable changes, requirements, etc. crucially this section avoids describing how the change is implemented. the act of separating what from how isn't new, but it's crucial in the LLM era
How does this change accomplish its goal? the answer to this question is "read the code", but the job of a PR description is to provide a quick hand drawn map before dumping the reviewer into the jungle. drawing a good "how" map requires understanding the jungle and the new path through it.
the risk with LLMs (or any outsource) doing the work is the loss of understanding. this exercise encourages understanding.
Where is that quote from, dis you just make it up?
> You can outsource your thinking, but not your understanding. — Sun Tzu
Lol I built this and immediately got frustrated with it and stopped using it. I think I don't like being questioned by someone who I expect to work like magic.
What's with Claude preferring cursive and italic font in landing page hero sections?
Looks absolutely awful.
Nice idea. I like that it tests whether the person understands the change instead of adding another AI reviewer. We added the tests before we switch to fully agentic AI development using automo - ie we tested every member of staff and now trust them to be the eyes.
Disclosure: I’m building automo.build fully to fix this.
I actually made this as a skill last week. It is helpful. Especially when reviewing a PR with 1500 LOC from a non-dev. It keeps me focused an helps me do a structured review. But it does not catch poor decisions, like calling the wring endpoint a such.
I can kinda understand the motivation here: you push a bunch of slop (or is it deep, genius code?) but you have a kind of nagging suspicion that you should get to know it more familiarly, like back in the good old days a few months back when you sort of understood what was going on. I... don't... think this is going to work...? Remember Confirmation Fatigue? Soon we will have agents that interact with Sphinx for us.
We went from micromanaging code to micromanaging tools and markdowns.
After noticing a drop in my understanding of the code I ship, I built and started using a similar skill ("Quiz me on the PR or the current branch"). I didn't tie it to a specific commit, though - to me it makes more sense to look at the PR as a whole.
So far I was positively surprised with the questions it could come up with.
Have you ever heard the idiom "ambulance at the bottom of the cliff"?
Love this idea, it’s way too easy to get lazy with AI, so having a quick reality check like this is actually a super healthy habit to build
I considered doing this with an open source app I have that let's you get a visual summary on a PR + get quizzed on it, but ultimately decided it's better to keep it an optional thing for people that want to understand AI PRs rather than force it as a gate, the volume is slowly becoming too high for even that. I agree with the sentiment behind this though as I felt this was needed as well.
I think this could be really useful. Any chance you could add native Windows support?
I ask the question to agent from separate session without history to know more about my code
Why stop there? Combine this with other knowledge testing methods:
pre-code: planning mode
coding: humantodo.dev
after-code: sphinx hook
So who is the master of whom?
It’s ironic that all negative comments on here are dismissive because “slop’s baaaaad”. Yes, slop is bad. Carelessness is bad. Lack of understanding is bad. It always has been. Code has been copied from stackoverflow. Built-in methods or std libraries have been used in a wrong way. Edge cases haven’t been covered. Rate-limiting and other hardening tasks haven’t been taken care of until the system broke. The code wasn’t DRY.
Now, people are building tools that not only hide the solutions to these problems in some obscure coding tool, markdown file, or agent harness, but they build gates that help developers to elevate themselves from the darkness of not knowing better. The number of developers roughly doubles every ~3 years, so at any given time, half of the people building software have less than 3 years of experience. Most of them didn’t even study CS before starting their careers. Tools like this will be the new warning voice that told people like us 5 to 10 years ago to never copy code from stackoverflow that you didn’t understand. It’s great.
I think this would be killer for OSS maintainers.
I think this should kick in before any code is written not before any code is pushed. It's an odd thing to check before pushing new code.
Also, from my experience implementing something similar, an MCQ style makes people guess without really understanding.
Honestly I love to see people working on this problem ! I actually built a similar project myself (https://github.com/bhurghundii/vibecheck.dev) except it prevents you from merging on the PR level. I’d love to get in touch to talk about it sometimes.
Edit: I see there are a lot of people questioning the value of this. The thing is that there are teams where people are just pumping tons of AI code without understanding their work (literally take the ticket and dump it) and shipping it. culture change is the right answer but v hard but this at least slows things down and is quick to implement
Humans are now test subjects in the ci pipeline
I mean I see the idea, but at least I use LLMs for code generation in two distinct ways.
The first is when I care about the code. That's when I actually read what it produces and force it to use my mental model of how things should be.
The second is when I just need a solution that sorta fits the requirements provided. Then I don't read the code because I really do not care.
So this hook would either ask me things I know, or it would ask me things I do not care about.
But I guess this is very workflow-dependent.
Its the right problem statement, but we normally let the code review process take care of this.
This is cute and novel, but practically useless. For those who use AI efficiently and effectively, this is useless. For those who don't, this becomes friction they don't want. Who really needs this?
All of this to just avoid knowing what you're doing.
Folks, this is the wrong direction. Use AI, but ffs take the time to properly learn what you're doing. We don't need to speed run civilizational collapse because we can't understand the concept of patience leading to better outcomes.
You can tell people are increasingly full of shit because they're doing this under the pretense that the problem is creating software faster, when really, a lot of people need to quit optimizing process and develop some skill and rigor. Most of the things that slow down software reaching production are human-created problems like "meetings," counting lines of code, or sending everyone to yet-another-conference that teaches them nothing (but boy does it look good on social media). It's all time waste and theater. And for what? So you feel included?
Pathetic. Learn to do this properly or get out of this industry.
"Outsource your thinking"
I'm sorry, what?
https://www.sphinx-doc.org/
Your slop generator should check if a name exist before claiming it.
a tool written by AI (mostly) to check code written by (possibly other) AI.
however, it still feels like a lot of work. what if AI hallucinates during the check?
Whenever I stop for a moment to think about it:
It is absolutely batsh*t crazy insane what SWE has come to in the last cple years.
And yet part of me is loving it.
People who care about the problem will definitely not be using this as a method to "understand" their code. People who don't care about this problem will continue to not care.
So, who is this for?
The quiz axis is interesting because it tests the person, not the diff. The complementary failure mode is the stuff you can pass a quiz on while still missing: a leftover debug print, an assertion weakened to match the bug it was meant to catch, an import of a package that never existed. Those live in the diff, not in comprehension. I hit this auditing my own patches last week - I could explain every line and still shipped a stdin read that broke on non-blocking pipes. Different catches, probably need both.