> Our findings suggest that 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks. This makes it very hard to know how good AI models are at the work Americans actually do.
> ...
> But even after controlling for AI-exposure, O*NET tasks from occupations with larger wage bills are significantly overrepresented among benchmark prompts:
So, in short: they want AI to take the good jobs. AI is not coming for jobs with tasks like "gather and empty trash" or "service, clean, or supply restrooms," or "mix water and detergents or acids in containers to prepare cleaning solutions according to specifications." We software engineers and lawyers would be smart to start retraining (and adjusting our living standards) now!
And this points to one of the ways AI will help improve society and living standards for all: we'll have cleaner, better-maintained bathrooms, as labor one wasted on engaging intellectual tasks is redirected towards menial, low-pay tasks that still need a human touch.
Reason to be optimistic, amirite?
show comments
qurren
About 80% of my software engineering job is fighting disagreements between "stakeholders".
show comments
armcat
Reinforces what Karpathy tweeted [1], that AI is effectively tackling 2-5% of work and the gap between those fully exploiting AI and barely touching it is growing. The question remains whether this AI focus on the small % of workers is actually intentional (by tech/labs), or a natural limit of where AI is right now.
Perhaps a version of the Doorman Fallacy, ie. replacing a doorman with an automatic door seems like an obvious win, but it turns out the doorman was performing a number of other vital functions (customer experience, for example) which weren't considered.
show comments
OtherShrezzing
I think “covered well” is being used here as a synonym for “covered at all”.
For example, the “social workers & care -> counsel parents with child rearing problems” category is considered “well covered” because there are 13 prompts.
It’s self-evident that the spectrum of work carried out by a social worker undertaking that work cannot be well represented by thirteen prompts.
I suspect if you really strictly defined “covered well”, to the extent that any normal person would use that phrase, the headline would be above 99.9%.
mikert89
The big secret is most people actually arent producing actual useful output. And so there is nothing to automate... they dont actually do anything.
show comments
mrandish
Glad to see more data and discussion of this. Lately the gap between ambient belief that "LLMs can now replace ~90% of all 'knowledge work'" and actual reality seems like it's growing.
I've found discussions on HN tend to quickly get pulled toward polar extremes between those who are justifiably impressed at the increasing effectiveness of LLMs and those who highlight the brittle jaggedness and counter-intuitive gaps of LLM 'intelligence' in daily practice. One side keeps citing more examples of 'uniquely human traits' LLMs don't seem close to replicating while the other complains of eternally moving goalposts.
Today I think both extremes are partly right but also partly wrong, and end up largely talking past each other because so much remains ill-defined, uncertain or currently unknowable. IMHO, the most accurate position is a carefully nuanced middle-ground built on well-defined scopes, terms, time frames and probabilities with frustratingly wide error bars. Partisans at either extreme seem uninterested in more than drive-by hot takes.
It can be simultaneously true that "Someday AI will be able to functionally replace all human knowledge work" and "That day does not appear to be soon, and even when it arrives, there will still be uniquely human traits which are valuable and that LLMs can't satisfactorily replicate."
legitster
One thing I've been thinking about is Baumol's Cost Disease. It's usually applied in at a macro scale across an economy: when a job somewhere becomes much more efficient, it makes all other jobs comparatively more expensive.
I've started to notice this on a micro scale as well. I am 20x more productive at coding and scripting, but I don't spend more of my time coding. My time is now filled with things like presentations, Jira, QA, etc. Ironically, despite all of the automation, the things that are painstaking and time consuming disproportionately require more time now.
show comments
pcthrowaway
Curious how food service is at 0 tasks covered by AI. A few places have been switching to using those robots on wheels to run food to tables. And Doordash and Uber Eats have some robot and drone delivery in select markets now.
show comments
bunderbunder
Based on my own experience working on AI-based solutions for one of the better-covered job families according to this map, I think 90% might be overstating the case. It's true that we've got a lot of published benchmarks, but they tend to cover the lowest value add tasks.
It's really easy to fall into the McNamara fallacy when rolling out these kinds of solutions. Focusing on the low value stuff is attractive. It tends to be easier to do, and also easier to quantify outcomes. But it also tends to direct attention away from important but difficult to quantify considerations, which may mean you don't notice negative externalities. So we often found ourselves in a verschilmmbessern situation where automating what we could automate made things worse overall.
Animats
"90% of American working hours are spent on tasks with little or no coverage from existing benchmarks" just says there are a lot more different tasks than benchmarks. This is a comment on the measuring tool, not AI.
dclowd9901
I feel like the big takeaway here isn't what's covered or not covered, but the percentages.
We have 3 companies claiming something like a $35T market for their services. The world's entire GDP is something like $126T. So 10% of the productive work that is done on this planet, that AI can actually perform makes up fully 1/4 of the entire world's economic output? best guesses for output solely of tech companies is something like $12T. Where are they getting these figures?
austin-cheney
Worse still most of the cost savings from AI in software development is primarily due to labor inefficiency, such as tasks more efficiently accomplished in other ways. Most of that is due to development reliance on inefficient conventions to cover for poor training and emotional insecurity. No matter how you slice it it’s always human capital problem more than a technology problem.
show comments
alexandre_m
Of course. Like being dragged in endless Teams meetings.
iugtmkbdfil834
I don't know if it if 90%, but it is interesting how many things I do are not covered by standard tracking, but become visible once I am not there to keep an eye on them. Naturally, I only had one manager who kinda knew what was going on around him -- in case you are wondering: not the current one.
zackmorris
*yet
I understand the resistance to having nearly all labor devalued because it can be automated. We're still in the denial stage of grief.
We only tax capital gains, not the destruction of wealth or property. So someone can steal the livelihoods of millions of people with no recourse. Just like they can tear down a 100 year old house and build 3 townhouses in its place and make hundreds of thousands of dollars, despite driving up housing costs in the neighborhood to the point that they're unaffordable. Large swaths of profit in the economy are based on externalities and captured markets now.
To think we'll make it through this without tax reform is frankly naive. Tax reform as in: human labor should not be taxed, and non-human labor and capital gains should be taxed more to cover the difference - including unemployment checks and UBI when it comes to it.
Expect howls of derision from moneyed interests who will never let common sense reasoning or deduction enter the law books for the public good.
Not to defeat my own argument, but I predict we will experience a roughly 3x gain in productivity by using AI, so the output of every job would need to triple to maintain current employment levels. Will that happen anytime soon? Of course not. Which is why layoffs in the short term are inevitable. I wouldn't be surprised if unemployment hits 15+% in 2027 or 2029 just after the elections, with 25+% unemployment for the young in entry-level roles being highly likely.
show comments
drnick1
Yes Claude is not spending any time around the water cooler.
CoolestBeans
Following is my experience and constantly pestering my friends across the industry for anecdotes. It isn't even that AI moves the bottleneck, the thing it solves was never the bottleneck in the first place (except for certain narrow verticals). AI both can't and doesn't do your thinking for you and worse yet, it robs you of the environment where you do your best thinking about data structures, algorithms, organization, etc. Code. Of course we can do that in natural language, but it's hardly the right tool for the job.
All this being said, I don't think AI usage will meaningfully go down in the long run cause it's a very powerful dopamine loop and tech companies are very good at making those. How many people report social media making them unhappy and how many people do you know who have actually quit? Or to paraphrase Jim Gaffigan, nobody says they eat at McDonalds, but the sign outside says "billions served". Somebody's lying (and it's not McDonalds).
PinkaDunka
100% of US working hours are spent on tasks not covered well by pen and pencil benchmarks
blmarket
Just counting by hours is not reflecting the impact on the economy. Treemap weighted by GDP will make more sense.
guybedo
drinking coffee?
mdemare
Video game designers: 0/24 tasks.
Apparently Epic thinks otherwise.
show comments
jerf
It's time for the rest of the world to learn about Amdahl's Law. Even if we flipped a switch and tomorrow everything that could reasonably done by AI was suddenly free and instantaneous, the world does not instantly transform into an AI paradise wonderland of free goodies and lollipops because there's still plenty of other things that need to be done for goodie and lollipop production and distribution.
I guess if I wanted to put my skepticism about the AI stock bubble into one little phrase, it would be that I don't think the bubble has accounted for Amdahl's Law. Costs don't actually go to zero with AI, and there's a huge difference between costs falling even 90% and costs falling a full 100%. And the difference between "30% improvement this year, trending to 85% in five years" and "90% this year" is pretty significant too.
show comments
qrush
Flips from Claude tab to Hacker News tab
Yep, makes sense...
lufxen
Due to non-performance.
inerte
Used to be 100%...
avereveard
Don't give them ideas?
verdverm
A lot of what humans do does not have a tight feedback loop needed for training and agentic workflows. This is why we are not seeing the jobpocolypse and productivity gains look a lot more like the very long-term 2% per year.
show comments
zkmon
What? But we already spent our family silver for the tokens and AI stocks.
You know what? The truth doesn't matter much. If people believe in the resale value of tulips, each tulip would sell for a million.
It's the beliefs and perception that create the reality. Reality and truth don't exist on their own.
rglover
This was obvious from the beginning but greed has deluded people to the point of utter absurdity. The comeuppance on this whole thing is going to make me run out of napkins and resort to finger lickin'.
zzzeek
which is it folks, is AI going to take all the jobs because it's so effective? or is it worthless? every other thread says the opposite.
hint: it's neither
[deleted]
insane_dreamer
yesterday I needed to create a spreadsheet, so in Google docs (paid Workplace account), I fire up Gemini to help with moving around text, creating columns, etc. Absolutely horrid experience -- totally useless. I ended up doing everything manually.
> Our findings suggest that 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks. This makes it very hard to know how good AI models are at the work Americans actually do.
> ...
> But even after controlling for AI-exposure, O*NET tasks from occupations with larger wage bills are significantly overrepresented among benchmark prompts:
So, in short: they want AI to take the good jobs. AI is not coming for jobs with tasks like "gather and empty trash" or "service, clean, or supply restrooms," or "mix water and detergents or acids in containers to prepare cleaning solutions according to specifications." We software engineers and lawyers would be smart to start retraining (and adjusting our living standards) now!
And this points to one of the ways AI will help improve society and living standards for all: we'll have cleaner, better-maintained bathrooms, as labor one wasted on engaging intellectual tasks is redirected towards menial, low-pay tasks that still need a human touch.
Reason to be optimistic, amirite?
About 80% of my software engineering job is fighting disagreements between "stakeholders".
Reinforces what Karpathy tweeted [1], that AI is effectively tackling 2-5% of work and the gap between those fully exploiting AI and barely touching it is growing. The question remains whether this AI focus on the small % of workers is actually intentional (by tech/labs), or a natural limit of where AI is right now.
[1] https://x.com/karpathy/status/2109361546505966046?s=20
Perhaps a version of the Doorman Fallacy, ie. replacing a doorman with an automatic door seems like an obvious win, but it turns out the doorman was performing a number of other vital functions (customer experience, for example) which weren't considered.
I think “covered well” is being used here as a synonym for “covered at all”.
For example, the “social workers & care -> counsel parents with child rearing problems” category is considered “well covered” because there are 13 prompts.
It’s self-evident that the spectrum of work carried out by a social worker undertaking that work cannot be well represented by thirteen prompts.
I suspect if you really strictly defined “covered well”, to the extent that any normal person would use that phrase, the headline would be above 99.9%.
The big secret is most people actually arent producing actual useful output. And so there is nothing to automate... they dont actually do anything.
Glad to see more data and discussion of this. Lately the gap between ambient belief that "LLMs can now replace ~90% of all 'knowledge work'" and actual reality seems like it's growing.
I've found discussions on HN tend to quickly get pulled toward polar extremes between those who are justifiably impressed at the increasing effectiveness of LLMs and those who highlight the brittle jaggedness and counter-intuitive gaps of LLM 'intelligence' in daily practice. One side keeps citing more examples of 'uniquely human traits' LLMs don't seem close to replicating while the other complains of eternally moving goalposts.
Today I think both extremes are partly right but also partly wrong, and end up largely talking past each other because so much remains ill-defined, uncertain or currently unknowable. IMHO, the most accurate position is a carefully nuanced middle-ground built on well-defined scopes, terms, time frames and probabilities with frustratingly wide error bars. Partisans at either extreme seem uninterested in more than drive-by hot takes.
It can be simultaneously true that "Someday AI will be able to functionally replace all human knowledge work" and "That day does not appear to be soon, and even when it arrives, there will still be uniquely human traits which are valuable and that LLMs can't satisfactorily replicate."
One thing I've been thinking about is Baumol's Cost Disease. It's usually applied in at a macro scale across an economy: when a job somewhere becomes much more efficient, it makes all other jobs comparatively more expensive.
I've started to notice this on a micro scale as well. I am 20x more productive at coding and scripting, but I don't spend more of my time coding. My time is now filled with things like presentations, Jira, QA, etc. Ironically, despite all of the automation, the things that are painstaking and time consuming disproportionately require more time now.
Curious how food service is at 0 tasks covered by AI. A few places have been switching to using those robots on wheels to run food to tables. And Doordash and Uber Eats have some robot and drone delivery in select markets now.
Based on my own experience working on AI-based solutions for one of the better-covered job families according to this map, I think 90% might be overstating the case. It's true that we've got a lot of published benchmarks, but they tend to cover the lowest value add tasks.
It's really easy to fall into the McNamara fallacy when rolling out these kinds of solutions. Focusing on the low value stuff is attractive. It tends to be easier to do, and also easier to quantify outcomes. But it also tends to direct attention away from important but difficult to quantify considerations, which may mean you don't notice negative externalities. So we often found ourselves in a verschilmmbessern situation where automating what we could automate made things worse overall.
"90% of American working hours are spent on tasks with little or no coverage from existing benchmarks" just says there are a lot more different tasks than benchmarks. This is a comment on the measuring tool, not AI.
I feel like the big takeaway here isn't what's covered or not covered, but the percentages.
We have 3 companies claiming something like a $35T market for their services. The world's entire GDP is something like $126T. So 10% of the productive work that is done on this planet, that AI can actually perform makes up fully 1/4 of the entire world's economic output? best guesses for output solely of tech companies is something like $12T. Where are they getting these figures?
Worse still most of the cost savings from AI in software development is primarily due to labor inefficiency, such as tasks more efficiently accomplished in other ways. Most of that is due to development reliance on inefficient conventions to cover for poor training and emotional insecurity. No matter how you slice it it’s always human capital problem more than a technology problem.
Of course. Like being dragged in endless Teams meetings.
I don't know if it if 90%, but it is interesting how many things I do are not covered by standard tracking, but become visible once I am not there to keep an eye on them. Naturally, I only had one manager who kinda knew what was going on around him -- in case you are wondering: not the current one.
*yet
I understand the resistance to having nearly all labor devalued because it can be automated. We're still in the denial stage of grief.
We only tax capital gains, not the destruction of wealth or property. So someone can steal the livelihoods of millions of people with no recourse. Just like they can tear down a 100 year old house and build 3 townhouses in its place and make hundreds of thousands of dollars, despite driving up housing costs in the neighborhood to the point that they're unaffordable. Large swaths of profit in the economy are based on externalities and captured markets now.
To think we'll make it through this without tax reform is frankly naive. Tax reform as in: human labor should not be taxed, and non-human labor and capital gains should be taxed more to cover the difference - including unemployment checks and UBI when it comes to it.
Expect howls of derision from moneyed interests who will never let common sense reasoning or deduction enter the law books for the public good.
Not to defeat my own argument, but I predict we will experience a roughly 3x gain in productivity by using AI, so the output of every job would need to triple to maintain current employment levels. Will that happen anytime soon? Of course not. Which is why layoffs in the short term are inevitable. I wouldn't be surprised if unemployment hits 15+% in 2027 or 2029 just after the elections, with 25+% unemployment for the young in entry-level roles being highly likely.
Yes Claude is not spending any time around the water cooler.
Following is my experience and constantly pestering my friends across the industry for anecdotes. It isn't even that AI moves the bottleneck, the thing it solves was never the bottleneck in the first place (except for certain narrow verticals). AI both can't and doesn't do your thinking for you and worse yet, it robs you of the environment where you do your best thinking about data structures, algorithms, organization, etc. Code. Of course we can do that in natural language, but it's hardly the right tool for the job.
All this being said, I don't think AI usage will meaningfully go down in the long run cause it's a very powerful dopamine loop and tech companies are very good at making those. How many people report social media making them unhappy and how many people do you know who have actually quit? Or to paraphrase Jim Gaffigan, nobody says they eat at McDonalds, but the sign outside says "billions served". Somebody's lying (and it's not McDonalds).
100% of US working hours are spent on tasks not covered well by pen and pencil benchmarks
Just counting by hours is not reflecting the impact on the economy. Treemap weighted by GDP will make more sense.
drinking coffee?
Video game designers: 0/24 tasks.
Apparently Epic thinks otherwise.
It's time for the rest of the world to learn about Amdahl's Law. Even if we flipped a switch and tomorrow everything that could reasonably done by AI was suddenly free and instantaneous, the world does not instantly transform into an AI paradise wonderland of free goodies and lollipops because there's still plenty of other things that need to be done for goodie and lollipop production and distribution.
I guess if I wanted to put my skepticism about the AI stock bubble into one little phrase, it would be that I don't think the bubble has accounted for Amdahl's Law. Costs don't actually go to zero with AI, and there's a huge difference between costs falling even 90% and costs falling a full 100%. And the difference between "30% improvement this year, trending to 85% in five years" and "90% this year" is pretty significant too.
Flips from Claude tab to Hacker News tab
Yep, makes sense...
Due to non-performance.
Used to be 100%...
Don't give them ideas?
A lot of what humans do does not have a tight feedback loop needed for training and agentic workflows. This is why we are not seeing the jobpocolypse and productivity gains look a lot more like the very long-term 2% per year.
What? But we already spent our family silver for the tokens and AI stocks.
You know what? The truth doesn't matter much. If people believe in the resale value of tulips, each tulip would sell for a million.
It's the beliefs and perception that create the reality. Reality and truth don't exist on their own.
This was obvious from the beginning but greed has deluded people to the point of utter absurdity. The comeuppance on this whole thing is going to make me run out of napkins and resort to finger lickin'.
which is it folks, is AI going to take all the jobs because it's so effective? or is it worthless? every other thread says the opposite.
hint: it's neither
yesterday I needed to create a spreadsheet, so in Google docs (paid Workplace account), I fire up Gemini to help with moving around text, creating columns, etc. Absolutely horrid experience -- totally useless. I ended up doing everything manually.
[flagged]
[flagged]
[flagged]
[flagged]
No shit?