This looks cool and the mechanism looks plausible. I found the experience of trying to understand whether the claims here are legit to be aggravating.
First of all, the whole readme section about benchmarks appears to be Claude/Codex written. What a slog to read this.
Second, they claim success on SWE-Bench Verified, but it's only on 50 tasks, not making clear how these tasks are chosen. I know from experience that you can keep selecting sets of tasks until you get a result. Also, this was run 1x, and the lift they show actually only has a p val of 0.22.
There's a famous book "how to lie with statistics". I don't think the continual posts about benchmark results here are purposeful lies, but I think it's just so easy to fool yourself (and I've been burned, most recently building http://pellmell.ai ). Also I don't think this post is the most egregious example.
anotherhue
I'm glad Claude is so recognisable, it lets me bounce right off the empty calorie language very efficiently.
Maybe this thing is great, but it cannot be determined with this presentation.
show comments
icodestuff
Love the idea, I was thinking about this problem a few weeks ago, but never got anywhere with it. I'm intrigued by the latency savings in particular, that sounds great.
One concern I have is that right now each session gets fresh "eyes" on the problem. Right now I find I get a lot of mileage out of a combination of long-running sessions and fresh ones. I worry with a single generated concept graph that gets only incremental refreshes will become stale slowly, and in subtle ways that are hard to detect. That could lead to semantic drift in the graph from reality, and every new session will take the drifted form as gospel. Have you run any long tests (weeks or longer) on this to make sure that this doesn't happen? My understanding is SWE Bench is only a point-in-time evaluation.
Also the graph is stored in the repo, right? How mergeable is it? I know I wouldn't want to do conflict resolution on that myself, and even Opus struggles to keep all the references correct (especially when comments are involved) when there's an B->C, A->B symbol rename.
show comments
xhrpost
I intuitively and perhaps naively thought that Claude using the LSP server would negate a lot of grep use. Is this tool solving the same problem or something else?
show comments
skerit
Does Claude's grep still prepend the relative path of the file before _every_ single line? Because that is nasty, especially in java projects.
show comments
Fidelix
Why these animations in the github README? why? It just made understanding anything more difficult
show comments
gabosarmiento
What's the benchmark against graphify?
show comments
gavmor
Is this still cheaper when stale?
show comments
peter_d_sherman
>"The problem:
Every task, your coding agent starts blind. Before it changes anything, it re-explores the repo: grep a term, open a file, follow an import, back out, try again. It is rebuilding a picture of a codebase it mapped an hour ago and threw away.
That rediscovery burns most of a run's tool calls, tokens, and latency, and it is pure overhead"
The author of this article brings up a very interesting problem -- that, at least as far as using LLM's as coders/coding assistants go, eventually context runs out and context related to the underlying codebase does too. This in turn burns tokens and in turn, wastes energy resources.
Historically (well, in the past couple of years!), a bunch of solutions have been proposed to address this problem (i.e., take abstracts/subsets/maps of code, write them to different databases and persistent storage methods, bring them back in when the LLM requires it, etc., etc.)...
But there's no really good solution to this problem (although, arguably Graft goes a lot farther than past tools and should be commended for that!) because the problem seems to lie in separate parts, across several problem domains:
1) LLM context window size -- limited. Anything that future LLM's do to make context windows larger will help ameliorate this problem.
2) Lack of a good way to represent a codebase to an LLM for training other than text.
In other words, first we need some kind of way to map codebases into Tensors rather than text (i.e., a higher-level "map" of the code) then train future LLM's on those code-specific Tensors.
3) Arguably, programming languages themselves share some of the blame...
Programming languages have historically been written so that an arbitrary corpus of text represents and can be interpreted and/or compiled into a computer program.
That is, while tools for mapping codebases exist, tools for directly training LLM's on those specific created "code maps" as Tensors, do not, do not seem to, or at least I'm currently unaware of any!
(Anyway, just thinking aloud...)
Graft looks good, and looks like it has made some serious inroads to solving the problem...
This looks cool and the mechanism looks plausible. I found the experience of trying to understand whether the claims here are legit to be aggravating.
First of all, the whole readme section about benchmarks appears to be Claude/Codex written. What a slog to read this.
Second, they claim success on SWE-Bench Verified, but it's only on 50 tasks, not making clear how these tasks are chosen. I know from experience that you can keep selecting sets of tasks until you get a result. Also, this was run 1x, and the lift they show actually only has a p val of 0.22.
There's a famous book "how to lie with statistics". I don't think the continual posts about benchmark results here are purposeful lies, but I think it's just so easy to fool yourself (and I've been burned, most recently building http://pellmell.ai ). Also I don't think this post is the most egregious example.
I'm glad Claude is so recognisable, it lets me bounce right off the empty calorie language very efficiently.
Maybe this thing is great, but it cannot be determined with this presentation.
Love the idea, I was thinking about this problem a few weeks ago, but never got anywhere with it. I'm intrigued by the latency savings in particular, that sounds great.
One concern I have is that right now each session gets fresh "eyes" on the problem. Right now I find I get a lot of mileage out of a combination of long-running sessions and fresh ones. I worry with a single generated concept graph that gets only incremental refreshes will become stale slowly, and in subtle ways that are hard to detect. That could lead to semantic drift in the graph from reality, and every new session will take the drifted form as gospel. Have you run any long tests (weeks or longer) on this to make sure that this doesn't happen? My understanding is SWE Bench is only a point-in-time evaluation.
Also the graph is stored in the repo, right? How mergeable is it? I know I wouldn't want to do conflict resolution on that myself, and even Opus struggles to keep all the references correct (especially when comments are involved) when there's an B->C, A->B symbol rename.
I intuitively and perhaps naively thought that Claude using the LSP server would negate a lot of grep use. Is this tool solving the same problem or something else?
Does Claude's grep still prepend the relative path of the file before _every_ single line? Because that is nasty, especially in java projects.
Why these animations in the github README? why? It just made understanding anything more difficult
What's the benchmark against graphify?
Is this still cheaper when stale?
>"The problem:
Every task, your coding agent starts blind. Before it changes anything, it re-explores the repo: grep a term, open a file, follow an import, back out, try again. It is rebuilding a picture of a codebase it mapped an hour ago and threw away.
That rediscovery burns most of a run's tool calls, tokens, and latency, and it is pure overhead"
The author of this article brings up a very interesting problem -- that, at least as far as using LLM's as coders/coding assistants go, eventually context runs out and context related to the underlying codebase does too. This in turn burns tokens and in turn, wastes energy resources.
Historically (well, in the past couple of years!), a bunch of solutions have been proposed to address this problem (i.e., take abstracts/subsets/maps of code, write them to different databases and persistent storage methods, bring them back in when the LLM requires it, etc., etc.)...
But there's no really good solution to this problem (although, arguably Graft goes a lot farther than past tools and should be commended for that!) because the problem seems to lie in separate parts, across several problem domains:
1) LLM context window size -- limited. Anything that future LLM's do to make context windows larger will help ameliorate this problem.
2) Lack of a good way to represent a codebase to an LLM for training other than text.
In other words, first we need some kind of way to map codebases into Tensors rather than text (i.e., a higher-level "map" of the code) then train future LLM's on those code-specific Tensors.
3) Arguably, programming languages themselves share some of the blame...
Programming languages have historically been written so that an arbitrary corpus of text represents and can be interpreted and/or compiled into a computer program.
That is, while tools for mapping codebases exist, tools for directly training LLM's on those specific created "code maps" as Tensors, do not, do not seem to, or at least I'm currently unaware of any!
(Anyway, just thinking aloud...)
Graft looks good, and looks like it has made some serious inroads to solving the problem...