You give it a problem, you then refine that problem where a fast, cheaper model asks you questions which you answer to get a better input prompt. You then choose a MA strategy for example take problem break up to sections then final judge concludes or you do multi turn where agents debate then judge summarises debate.
The best approach is what I call 'all angles' where all these strategies run in parallel the final meta-judge synthesise the response - the most useful part of this which I recently added is a view to see the variance in each strategy.
Been using this for life stuff - housing search, schools, family challenges!
Perhaps I should make a video of it in action if people in HN community interested let me know.
show comments
sedatk
One month I could use Github Copilot fully with no disruptions. The next month, after pricing changes, I’ve run out of tokens in two days.
Such drastic changes tell me that pricing of tokens is arbitrary, and AI business is running out of money fast.
show comments
bob1029
> Furthermore, we observe that input tokens consistently constitute the largest share of consumption for an average of 53.9%
I'm seeing a ratio of around 10:1 in my usage. A vast majority of the tokens consumed are on the input side. The agent will often read a million tokens just to patch one line of code.
I think if you are seeing something closer to 1:1 or more on the output side, there is either a problem with the agent or the codebase is new / empty.
It’s just like Airline reward miles and offers no benefit to companies over just renting bare metal GPU time
show comments
senectus1
amusing side note:
Was in a meeting reviewing a potential new product, it was going well until they showed us that they had added AI to it (of course they have). It was pretty obviously just shoehorned in, and one part of that obviousness was that they had a column that showed how many tokens it took to make each query.
I asked who is paying for the tokens, they said its included in the license. I said, so is there a budget or is it all you can eat.
they said good question they didnt know and would get back to me. I said the reason i asked was just one query there had a 250k token burn on it. and it was a fairly simple query about one device.
then, one of the execs on their side was heard saying out loud "Why are we even showing this to the customers?"
it have us quite a chuckle.
But lesson learned... the cost of adding AI to anything isnt really being accounted for let alone the true cost of actually running the AI.
all things AI are going to get more expensive. even if you dont want the AI aspect.
show comments
becomevocal
First thought was "only 30 tasks" however the findings map to what I've seen personally: code review consumes majority of tokens
show comments
drivebyhooting
In the past Google et al would hire engineers based on how well they could optimize the infrastructure.
Maybe soon companies will look at how engineers can optimize the token efficiency of AI.
show comments
satvikpendem
Tokenomics is already a word used to describe cryptocurrency economics, not sure why they'd try to redefine it for AI even if a different sort of token is used.
show comments
emsign
At its current iteration the AI tech market is not economically sustainable, not for the other markets outside the AI economy, and most deadly not even for the main target customers or AI tech companies themselves. There have been several news of companies having overspent their token budget month after month. The hardware monopolist and his network of buddy companies can determine the token price as freely as they want, there are no competitors, their only "competitor" is when people stop using AI alltogether.
I have a MA system setup for personal use.
You give it a problem, you then refine that problem where a fast, cheaper model asks you questions which you answer to get a better input prompt. You then choose a MA strategy for example take problem break up to sections then final judge concludes or you do multi turn where agents debate then judge summarises debate.
The best approach is what I call 'all angles' where all these strategies run in parallel the final meta-judge synthesise the response - the most useful part of this which I recently added is a view to see the variance in each strategy.
Been using this for life stuff - housing search, schools, family challenges!
Perhaps I should make a video of it in action if people in HN community interested let me know.
One month I could use Github Copilot fully with no disruptions. The next month, after pricing changes, I’ve run out of tokens in two days.
Such drastic changes tell me that pricing of tokens is arbitrary, and AI business is running out of money fast.
> Furthermore, we observe that input tokens consistently constitute the largest share of consumption for an average of 53.9%
I'm seeing a ratio of around 10:1 in my usage. A vast majority of the tokens consumed are on the input side. The agent will often read a million tokens just to patch one line of code.
I think if you are seeing something closer to 1:1 or more on the output side, there is either a problem with the agent or the codebase is new / empty.
I wrote a Subsack post on this topic back in December https://open.substack.com/pub/zacharywhitley/p/the-coming-ag...
One thing I've noticed using agents for coding is that they really like to write thousands of unit tests but not dynamically test.
Reminded me of this paper from last year trying to optimize efficient token usage providing budget guidance information. [1]
[1] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=Stee...
It’s just like Airline reward miles and offers no benefit to companies over just renting bare metal GPU time
amusing side note:
Was in a meeting reviewing a potential new product, it was going well until they showed us that they had added AI to it (of course they have). It was pretty obviously just shoehorned in, and one part of that obviousness was that they had a column that showed how many tokens it took to make each query.
I asked who is paying for the tokens, they said its included in the license. I said, so is there a budget or is it all you can eat. they said good question they didnt know and would get back to me. I said the reason i asked was just one query there had a 250k token burn on it. and it was a fairly simple query about one device.
then, one of the execs on their side was heard saying out loud "Why are we even showing this to the customers?"
it have us quite a chuckle. But lesson learned... the cost of adding AI to anything isnt really being accounted for let alone the true cost of actually running the AI.
all things AI are going to get more expensive. even if you dont want the AI aspect.
First thought was "only 30 tasks" however the findings map to what I've seen personally: code review consumes majority of tokens
In the past Google et al would hire engineers based on how well they could optimize the infrastructure.
Maybe soon companies will look at how engineers can optimize the token efficiency of AI.
Tokenomics is already a word used to describe cryptocurrency economics, not sure why they'd try to redefine it for AI even if a different sort of token is used.
At its current iteration the AI tech market is not economically sustainable, not for the other markets outside the AI economy, and most deadly not even for the main target customers or AI tech companies themselves. There have been several news of companies having overspent their token budget month after month. The hardware monopolist and his network of buddy companies can determine the token price as freely as they want, there are no competitors, their only "competitor" is when people stop using AI alltogether.