So if I understand correctly, one of the agents is creating the workflow as a DAG dynamically for each new job it's given? That's the most interesting part to me. The rest I was already pretty familiar with.
So maybe that's a big chunk of what you need for an 'AI Company': an agent that manages the goals and hierarchies. Although of course the DAG and agent hierarchy is not quite the same thing. But maybe the workflows and subworkflows are what matter.
show comments
hanneshdc
Any benchmarks showing if this actually improves problem solving? Or reduces errors?
The idea is cool, but from own experience in harness engineering, lots of cool sounding ideas can have a negative impact on performance due to emergent and confounding effects.
So I'm a bit skeptical!
show comments
DerrickDevo1
A good tutorial. Generally speaking, the harness is the environment layer between a language model and its task, such as the action set it can call, the state, the context it can see and the memory etc.
However, currently the bigger question comes to my experience during harness is actually not where we use LLM in the system, but where we do NOT use LLM in the system. And the validation of the results becomes more and more important. Any thoughts on this?
bryan0
I (like presumably many others) have built something similar. My main difference though is that the critics operate on each stage of development before it can move onto the next. The stages are defined by deliverable artifacts: issue, plan, pull request. Critics must approve each artifact before you can move onto the next. So the process is defined by a DAG which defines how to transition successfully from one artifact to the next. It's been fun to work on and I would like to open source it soon, but I assume many others are working on similar systems.
abdullahkhalids
A harness [1] was developed by Terrence Tao and some collaborators to prove mathematical results. It has since then been used by others with positive effect. Can someone critique the structure of this harness? I don't know anything about this stuff.
I much prefer giving the LLM a REPL loop, and injecting all the tools as functions inside the REPL loop.
That means that the LLM isn't constrained to writing a DAG, it can write code that loops, exits early, etc.
show comments
Axsuul
Anyone else have related reading that touches on this? I'm building my own custom harness and want to start implementing loop support, etc. But I also want to build some sort of framework so that it's dynamic (e.g. this needs to run x number of iterations, while planning needs to run y number of iterations).
show comments
jumploops
Contrary to the title and intro, this appears to be an agentic _workflow_ builder/runner, not an advanced “agent harness”
A few things:
- they note: “nothing in this post proves it actually works in most cases”
- the DAG sounds good, but LLMs often split tasks into smaller pieces than they need to, which can cause them to lose the forest for the trees
- the forced JSON interplay, in my experience, causes even gpt-5.6-sol to lose a few “IQ points”
For anyone reading this, this tutorial is much more reminiscent of how folks were building “agents” pre-Claude Code.
tl;dr the “orchestrator” here is just a software loop, and the LLM prompts restrict flexibility of the planner/workers
why do i hate skillks, harnesses , memory systems whatever. such ideas that everyone thinks they've discovered but are totally useless in practice.
show comments
tosh
I love reading about orchestration concepts.
But pulling orchestration off is very very tricky.
Even if it is just a small, simple orchestrator.
Ideas like planner, memory, log, subagents, graphs (each on their own) sound great and very promising.
So promising that one would think they must work, how could they not?
I've been there as well!
The challenge is that all these parts of the orchestrator are intertwined with each other
and they are all causing overhead in the main context window in some form or at least overall complexity that is difficult to grasp and predict/engineer for
(even though the idea is to help exactly with the fact that the context window is limited)
To save context window there is also more communication that has 'stille post' ('chinese whispers') like dynamics
Turns out it is very difficult to find out the right context to bubble up and down.
It's very similar to human org communication challenges (think large org stucture vs small teams vs one person that can keep it all in their head)
Yeah, what do you do if one person can't keep it all in their head?
But how great is it when it's possible?
Companies must have figured out how that works right? Maybe we can adopt and implement these ideas?
And yet … easy it is not, especially when you're not dealing with run-of-the-mill well-defined tasks.
But more like with open-ended software development?
I'm not saying it's not possible or that it should not be tried.
On the contrary, I think this is worth pursuing and a bit like the search for the holy grail.
But I also think the other direction of the search space is under-explored.
The holy grail is glamorous.
With 'smol' I'm spelunking on this other extreme (non-orchestration?)
(welcome, join us, we have cookies, and context windows with a lot of room for work items!)
smol is a minimalist agent harness that protects the context window
- no system prompt
- no tool spamming (just 1 tool: sh)
- no agents.md
- no mcp
- no planning, todos, graphs, beads, …
and figuring out how that looks like and performs
it is a worthwhile thread to pull I think
at least from the dozens of benches I'm looking at I see that less stuff in the context window does help a lot
- cheaper per task
- finishing faster
- better tool composition (sh and pipes are great!)
but also for more complicated longer-term tasks the model gets less confused when the context window is not getting spammed
So if I understand correctly, one of the agents is creating the workflow as a DAG dynamically for each new job it's given? That's the most interesting part to me. The rest I was already pretty familiar with.
So maybe that's a big chunk of what you need for an 'AI Company': an agent that manages the goals and hierarchies. Although of course the DAG and agent hierarchy is not quite the same thing. But maybe the workflows and subworkflows are what matter.
Any benchmarks showing if this actually improves problem solving? Or reduces errors?
The idea is cool, but from own experience in harness engineering, lots of cool sounding ideas can have a negative impact on performance due to emergent and confounding effects.
So I'm a bit skeptical!
A good tutorial. Generally speaking, the harness is the environment layer between a language model and its task, such as the action set it can call, the state, the context it can see and the memory etc.
However, currently the bigger question comes to my experience during harness is actually not where we use LLM in the system, but where we do NOT use LLM in the system. And the validation of the results becomes more and more important. Any thoughts on this?
I (like presumably many others) have built something similar. My main difference though is that the critics operate on each stage of development before it can move onto the next. The stages are defined by deliverable artifacts: issue, plan, pull request. Critics must approve each artifact before you can move onto the next. So the process is defined by a DAG which defines how to transition successfully from one artifact to the next. It's been fun to work on and I would like to open source it soon, but I assume many others are working on similar systems.
A harness [1] was developed by Terrence Tao and some collaborators to prove mathematical results. It has since then been used by others with positive effect. Can someone critique the structure of this harness? I don't know anything about this stuff.
[1] https://github.com/1stproof/batch-2/tree/main/batch-2-submis...
> The plan is a graph
I much prefer giving the LLM a REPL loop, and injecting all the tools as functions inside the REPL loop.
That means that the LLM isn't constrained to writing a DAG, it can write code that loops, exits early, etc.
Anyone else have related reading that touches on this? I'm building my own custom harness and want to start implementing loop support, etc. But I also want to build some sort of framework so that it's dynamic (e.g. this needs to run x number of iterations, while planning needs to run y number of iterations).
Contrary to the title and intro, this appears to be an agentic _workflow_ builder/runner, not an advanced “agent harness”
A few things:
- they note: “nothing in this post proves it actually works in most cases”
- the DAG sounds good, but LLMs often split tasks into smaller pieces than they need to, which can cause them to lose the forest for the trees
- the forced JSON interplay, in my experience, causes even gpt-5.6-sol to lose a few “IQ points”
For anyone reading this, this tutorial is much more reminiscent of how folks were building “agents” pre-Claude Code.
tl;dr the “orchestrator” here is just a software loop, and the LLM prompts restrict flexibility of the planner/workers
And the associated GitHub repo: https://github.com/DataForScience/LLMs
Good stuff, keeping it
why do i hate skillks, harnesses , memory systems whatever. such ideas that everyone thinks they've discovered but are totally useless in practice.
I love reading about orchestration concepts.
But pulling orchestration off is very very tricky.
Even if it is just a small, simple orchestrator.
Ideas like planner, memory, log, subagents, graphs (each on their own) sound great and very promising.
So promising that one would think they must work, how could they not?
I've been there as well!
The challenge is that all these parts of the orchestrator are intertwined with each other
and they are all causing overhead in the main context window in some form or at least overall complexity that is difficult to grasp and predict/engineer for
(even though the idea is to help exactly with the fact that the context window is limited)
To save context window there is also more communication that has 'stille post' ('chinese whispers') like dynamics
Turns out it is very difficult to find out the right context to bubble up and down.
It's very similar to human org communication challenges (think large org stucture vs small teams vs one person that can keep it all in their head)
Yeah, what do you do if one person can't keep it all in their head?
But how great is it when it's possible?
Companies must have figured out how that works right? Maybe we can adopt and implement these ideas?
And yet … easy it is not, especially when you're not dealing with run-of-the-mill well-defined tasks.
But more like with open-ended software development?
I'm not saying it's not possible or that it should not be tried.
On the contrary, I think this is worth pursuing and a bit like the search for the holy grail.
But I also think the other direction of the search space is under-explored.
The holy grail is glamorous.
With 'smol' I'm spelunking on this other extreme (non-orchestration?)
(welcome, join us, we have cookies, and context windows with a lot of room for work items!)
smol is a minimalist agent harness that protects the context window
and figuring out how that looks like and performsit is a worthwhile thread to pull I think
at least from the dozens of benches I'm looking at I see that less stuff in the context window does help a lot
but also for more complicated longer-term tasks the model gets less confused when the context window is not getting spammedthe context window is precious
https://x.com/__tosh/status/2084985580144722369
https://github.com/smol-env/smol