Interesting. I too have been working in this space, though I took a different approach. Rather than building an index, I worked on making a "smarter grep" by offering search over codebases (and any text content really) with ranking and some structural awareness of the code. Most of my time was spend dealing with performance, and as a result it runs extremely quickly.
I will have to add this as a comparison to https://github.com/boyter/cs and see what my LLMs prefer for the sort of questions I ask. It too ships with MCP, but does NOT build an index for its search. I am very curious to see how it would rank seeing as it does not do basic BM25 but a code semantic variant of it.
This seems to work better for the "how does auth work" style of queries, while cs does "authenticate --only-declarations" and then weighs results based on content of the files, IE where matches are, in code, comments and the overall complexity of the file.
Have starred and will be watching.
ind-igo
I've been skeptical of these semantic search tools. Not only are agents already great with grep, the problem imo is these search tools treat your specific code like a destination, but your codebase is actually a graph, and your agent needs more context around your search term in order to make changes.
Luckily, graph traversal of your code has been solved for a long time, by LSP. But LSP is so extremely memory inefficient.
I created cx[0] to strip away the bloat from LSP into a lightweight navigation tool for agents, using only tree-sitter. I never got around to sharing on HN but might be time for a post.
I did some evals with pi and GPT 5.5. I tested RTK on / headroom on / both on / both off (all with the standard pi system instructions and no AGENTS.md).
I forget the exact tests I used (a couple of the standard agent evals that people use, one python and one typescript because those are what I use).
I don't claim it was an exhaustive test, or even a good one. It's possible I could have spent a day or so tuning my AGENTS.md and the pi system prompt/tool instructions and gotten better results, because if there's one thing running evals taught me it's that subtle differences there can change the results a lot.
However, I got clearly better results with both off, enough to convince me to stop the tests immediately after 3 rounds.
The problem was that while context use did go down (sometimes), the number of turns to complete went up so the overall cost of the conversation was higher.
It's made me very aware of one thing: so many people are sharing these kind of tools, but either with zero evals (or suspiciously hard to reproduce), or in the case of this one, extensive benchmarks testing the wrong thing.
I'm sure this tool does use fewer tokens than grep, and the benchmarks prove it, but that's not what matters here. What matters is, does an agent using it get the same quality of work done more quickly and for lower cost?
show comments
jerezzprime
I'd be interested in seeing actual agent benchmarks (eg CC or Copilot CLI with grep removed and this tool instead).
For example, I have explored RTK and various LSP implementations and find that the models are so heavily RL'd with grep that they do not trust results in other forms and will continually retry or reread, and all token savings are lost because the model does not trust the results of the other tools.
show comments
andai
Nice, this sounds great. I want to mention a related issue here, which is that on small codebases, Claude spends a lot of time looking for stuff when it could have just dumped the whole codebase into the context in one go and used very little tokens.
I found a nice workaround which is that you can just dump the whole directory into context, as a startup hook. So then Claude skips the "fumble around blindly in the dark" portion of every task. (I've also seen a great project that worked on bigger repos where it'll give the model an outline with stubs, though I forget what it was called.)
show comments
aadishv
Seems like a cool idea so I decided to play with it a bit. The test I ran was in the browsercode (https://github.com/browser-use/browsercode) repo with the following prompt:
"Answer this question by only using the `semble` CLI (docs below):
> What tools does Browsercode provide to the agent other than the base OpenCode tools? Provide the exact schema for tool input and tool output and briefly summarize what they do and how they work
"Answer this question by only using the `rg` and `fd` CLIs:
> What tools does Browsercode provide to the agent other than the base OpenCode tools? Provide the exact schema for tool input and tool output and briefly summarize what they do and how they work"
In both cases, I used Pi with gpt-5.4 medium and a very minimal setup otherwise. (And yes, I did verify that either instance only used rg & fd, or only used semble.)
Without Semble, it used 10.9% of the model context and used $0.144 of API credits (or, at least, that's what Pi reported - I used this with a Codex sub so cannot be sure). With Semble, it used 9.8% of the model context and $0.172 of API credits. The resulting responses were also about the same. Very close!
I tried one more test in the OpenCode repo. The question was
> Trace the path from 1) the OPENCODE_EXPERIMENTAL_EXA env var being set to to 1 to 2) the resulting effects in the system prompt or tool provided to the OpenCode agent.
And I included the same instructions/docs as above. The non-Semble version was a bit more detailed -- it went into whether the tool call path invoked Exa based on whether Exa or Parallel was enabled for the web search provider -- but w.r.t. actually answering the question, both versions were accurate. The Semble version used 14.7% context / $0.282 API cost, while the non-Semble version used 19.0% / $0.352. Clearly a win for Semble for context efficiency, but note that the non-Semble version finished about twice as fast as the Semble version.
Of course this is just me messing around. ymmv.
AussieWog93
Better than grep obviously, but how does this compare to existing LSPs?
show comments
singpolyma3
Semantic code search seems like a useful tool for a human too. Not just for agents.
Shouldn’t it be a part of the harness at least for local codebase? I wonder how many harnesses are doing that already.
show comments
wrxd
I also like the index feature form https://maki.sh
Source code has a lot of structure, using a real parser instead of grepping and reading files can potentially save a lot of tokens
_ink_
Would this replace something like codebase-memory-mcp[1] or improve when both is being used?
Does this support any language or is it limited to a specific set of languages?
jahala
This looks great! I built a tool in the same space- and I found that the biggest challenge was often to get the agent to prefer to use the tool over bash tools. What’s your experience with that?
show comments
smcleod
How does it compare to context-mode or serina that are both well established now?
onoesworkacct
fantastic token savings and performance... but unlike grep it's probabilistic search on search terms.
is that an issue? the tiny model might not surface something important
esafranchik
Is the benchmark measuring one-shot retrieval accuracy, or Coding agent response accuracy?
show comments
porker
Congratulations on the release!
Could you add fff to the benchmarks?
show comments
mrpf1ster
Does this work well for non-coding documents as well? Say api docs or AI memory files?
show comments
ramsono
Very useful thanks for sharing!
jasonli0226
thanks for sharing!
vemulasukrit
Nice!
ludicrousdispla
grep doesn't need tokens, so what is 98% fewer than zero?
show comments
vikeri
very curious to give it a spin but why write a cli in python? would surely be faster and more portable with go or rust?
Interesting. I too have been working in this space, though I took a different approach. Rather than building an index, I worked on making a "smarter grep" by offering search over codebases (and any text content really) with ranking and some structural awareness of the code. Most of my time was spend dealing with performance, and as a result it runs extremely quickly.
I will have to add this as a comparison to https://github.com/boyter/cs and see what my LLMs prefer for the sort of questions I ask. It too ships with MCP, but does NOT build an index for its search. I am very curious to see how it would rank seeing as it does not do basic BM25 but a code semantic variant of it.
This seems to work better for the "how does auth work" style of queries, while cs does "authenticate --only-declarations" and then weighs results based on content of the files, IE where matches are, in code, comments and the overall complexity of the file.
Have starred and will be watching.
I've been skeptical of these semantic search tools. Not only are agents already great with grep, the problem imo is these search tools treat your specific code like a destination, but your codebase is actually a graph, and your agent needs more context around your search term in order to make changes.
Luckily, graph traversal of your code has been solved for a long time, by LSP. But LSP is so extremely memory inefficient.
I created cx[0] to strip away the bloat from LSP into a lightweight navigation tool for agents, using only tree-sitter. I never got around to sharing on HN but might be time for a post.
[0] https://github.com/ind-igo/cx
I did some evals with pi and GPT 5.5. I tested RTK on / headroom on / both on / both off (all with the standard pi system instructions and no AGENTS.md).
I forget the exact tests I used (a couple of the standard agent evals that people use, one python and one typescript because those are what I use).
I don't claim it was an exhaustive test, or even a good one. It's possible I could have spent a day or so tuning my AGENTS.md and the pi system prompt/tool instructions and gotten better results, because if there's one thing running evals taught me it's that subtle differences there can change the results a lot.
However, I got clearly better results with both off, enough to convince me to stop the tests immediately after 3 rounds.
The problem was that while context use did go down (sometimes), the number of turns to complete went up so the overall cost of the conversation was higher.
It's made me very aware of one thing: so many people are sharing these kind of tools, but either with zero evals (or suspiciously hard to reproduce), or in the case of this one, extensive benchmarks testing the wrong thing.
I'm sure this tool does use fewer tokens than grep, and the benchmarks prove it, but that's not what matters here. What matters is, does an agent using it get the same quality of work done more quickly and for lower cost?
I'd be interested in seeing actual agent benchmarks (eg CC or Copilot CLI with grep removed and this tool instead).
For example, I have explored RTK and various LSP implementations and find that the models are so heavily RL'd with grep that they do not trust results in other forms and will continually retry or reread, and all token savings are lost because the model does not trust the results of the other tools.
Nice, this sounds great. I want to mention a related issue here, which is that on small codebases, Claude spends a lot of time looking for stuff when it could have just dumped the whole codebase into the context in one go and used very little tokens.
I found a nice workaround which is that you can just dump the whole directory into context, as a startup hook. So then Claude skips the "fumble around blindly in the dark" portion of every task. (I've also seen a great project that worked on bigger repos where it'll give the model an outline with stubs, though I forget what it was called.)
Seems like a cool idea so I decided to play with it a bit. The test I ran was in the browsercode (https://github.com/browser-use/browsercode) repo with the following prompt:
"Answer this question by only using the `semble` CLI (docs below):
> What tools does Browsercode provide to the agent other than the base OpenCode tools? Provide the exact schema for tool input and tool output and briefly summarize what they do and how they work
---
[the AGENTS.md snippet provided from https://github.com/MinishLab/semble#bash-integration]"
And the equivalent for the non-Semble test:
"Answer this question by only using the `rg` and `fd` CLIs:
> What tools does Browsercode provide to the agent other than the base OpenCode tools? Provide the exact schema for tool input and tool output and briefly summarize what they do and how they work"
In both cases, I used Pi with gpt-5.4 medium and a very minimal setup otherwise. (And yes, I did verify that either instance only used rg & fd, or only used semble.)
Without Semble, it used 10.9% of the model context and used $0.144 of API credits (or, at least, that's what Pi reported - I used this with a Codex sub so cannot be sure). With Semble, it used 9.8% of the model context and $0.172 of API credits. The resulting responses were also about the same. Very close!
I tried one more test in the OpenCode repo. The question was > Trace the path from 1) the OPENCODE_EXPERIMENTAL_EXA env var being set to to 1 to 2) the resulting effects in the system prompt or tool provided to the OpenCode agent.
And I included the same instructions/docs as above. The non-Semble version was a bit more detailed -- it went into whether the tool call path invoked Exa based on whether Exa or Parallel was enabled for the web search provider -- but w.r.t. actually answering the question, both versions were accurate. The Semble version used 14.7% context / $0.282 API cost, while the non-Semble version used 19.0% / $0.352. Clearly a win for Semble for context efficiency, but note that the non-Semble version finished about twice as fast as the Semble version.
Of course this is just me messing around. ymmv.
Better than grep obviously, but how does this compare to existing LSPs?
Semantic code search seems like a useful tool for a human too. Not just for agents.
How does this compare with colgrep?
https://github.com/lightonai/next-plaid/tree/main/colgrep
Shouldn’t it be a part of the harness at least for local codebase? I wonder how many harnesses are doing that already.
I also like the index feature form https://maki.sh Source code has a lot of structure, using a real parser instead of grepping and reading files can potentially save a lot of tokens
Would this replace something like codebase-memory-mcp[1] or improve when both is being used?
[1] - https://github.com/DeusData/codebase-memory-mcp
Does this support any language or is it limited to a specific set of languages?
This looks great! I built a tool in the same space- and I found that the biggest challenge was often to get the agent to prefer to use the tool over bash tools. What’s your experience with that?
How does it compare to context-mode or serina that are both well established now?
fantastic token savings and performance... but unlike grep it's probabilistic search on search terms.
is that an issue? the tiny model might not surface something important
Is the benchmark measuring one-shot retrieval accuracy, or Coding agent response accuracy?
Congratulations on the release!
Could you add fff to the benchmarks?
Does this work well for non-coding documents as well? Say api docs or AI memory files?
Very useful thanks for sharing!
thanks for sharing!
Nice!
grep doesn't need tokens, so what is 98% fewer than zero?
very curious to give it a spin but why write a cli in python? would surely be faster and more portable with go or rust?