Every week, get the latest curated signals about how AI is impacting society & culture. Paid subscribers get full access to all newsletters, deep-dives, essays, field guides, and curated podcast Q&A notes. Also, please complete our anonymous survey so we can better gauge your content interests and better cater deals & discounts for you. Thank you!
Last week, a founder friend sent me a late-night DM that perfectly captures the current state of the current AI vibe…if you want to call it that.
He and his team had spent the weekend setting up an autonomous agentic workflow to tackle their backlog of pull requests and bugs, including a good chunk of their technical debt. They had roughly ~100 minor bugs and documentation updates that had been sitting in Jira for months. Instead of assigning them to junior developers, they let a fleet of coding agents loose on the codebase.
A week later, he messaged me again. This time, the tone was very different.
The first issue was the API bill.
The agents had run in autonomous loops, repeatedly fetching context, running test suites, failing, and then retrying. By the time they finished, they had burned through millions of tokens, racking up a few thousand dollars in bills on reasoning models - a tad out of their budget. Note: it wasn’t all that bad; they did factor it in, but it was just a tough pill to swallow.
The second issue was the code itself.
When their senior engineers sat down to review the pull requests, they found that while the code passed basic unit tests, about 40% of it introduced architectural anti-patterns, duplicate functions, or security vulnerabilities. It took two senior engineers a few days of refactoring to untangle the mess — I know most will say that this is normal and the models will get better & smarter, which I agree with, but even so, I think it’s foolish to not vet production level code using a human.
If you do the math, the unit economics show that:
Raw API compute cost: $4,500
Senior developer review/repair time: 48 hours at a fully loaded rate of $120/hour = $5,760
Total cost to resolve 100 minor bugs: $10,260
Cost per resolved issue: $102
For context, a junior human developer could have handled those same issues in about a week. At a fully loaded salary of $100,000, that week of human labor costs the company roughly $2,000.
In their rush to automate, my friend’s company paid a 400% premium for the privilege of saying their code was written by AI, and I think this is the reality of the Compute Wall — it’s the point at which the cost of running AI to do a particular job reaches, or even surpasses, the cost of hiring a human.
I—The Myth of Zero Marginal Cost
For the past three years, the tech industry has operated under a single assumption: software has zero marginal cost.
In the traditional software era, once you wrote the code, serving it to the 1,000th user cost virtually the same as serving it to the 10th. The margins were software-rich, and scalability was infinite. When generative AI first burst onto the scene, we applied the same mental model, assuming running these models would follow the same downward cost curve.
But AI is not designed to be traditional software. It’s actually a very capital-intensive infrastructure play.
In the early days of LLMs called the System-1 Era—this distinction was kind of masked. System-1 thinking, to borrow Daniel Kahneman’s terminology, is fast, instinctive, and cheap. For e.g., when you ask ChatGPT to summarize a PDF, the model performs a single forward pass and stops, costing fractions of a cent. For these simple tasks, the marginal cost of AI is indeed racing toward zero, which is why basic copywriting and support tasks were automated instantly.
But we have exhausted the low-hanging fruit of System-1 automation. The frontier of AI is now in the System-2 Era.
System-2 thinking is slow, deliberate, and logical. It is the reasoning we use to solve hard math problems, write complex code, or design business strategies. To achieve this, we are shifting from single-turn prompts to inference-time compute scaling (exemplified by models like OpenAI’s o1 and o3, DeepSeek-R1, and Claude) and autonomous agentic workflows.
As a quick refresher:
Inference-time compute scaling is about using more compute at inference (not training) to make a fixed model smarter, more reliable, or more capable on the fly.
The core idea is that for a fixed model:
Training-time scaling: bigger model, more data, longer training → better capability.
Inference-time scaling: same model and weights, but more thinking per question (more tokens, more passes, more search) → better capability.
So inference-time compute scaling is the infrastructure knob that enables Systems 2 behaviour.
Instead of answering immediately, the model is allowed to “think” before it speaks. It generates hidden chain-of-thought tokens, tests assumptions, runs code in a sandbox, evaluates output, and corrects its own path.
Unlike training compute, which is a one-time capital expenditure, inference compute is a variable, per-use cost. Every time an agent loops, runs a test, or writes a hidden reasoning token, a meter is running in the background.
As we scale the complexity of the tasks we assign to these agents, we’re running headfirst into the Compute Wall.
II—Inside the Unit Economics of “Tokenmaxxing”
To understand why AI costs can surpass human labor, we should look at the unit economics of agentic orchestration.
A quick aside: I wrote a piece on this growing trend a while back, and it’s now becoming the norm.
Here’s a common use case:
Let’s break down what happens when you ask an autonomous agent to migrate a legacy API endpoint. A human engineer does this by reading the documentation, writing the code, running a local server, debugging errors, and committing the change. It might take three hours. At a fully loaded rate of $100/hour, the human cost is $300.
Now, look at the AI workflow.
The agent loads the codebase repository (100k input tokens), does some planning via Claude Opus, writes code, compiles, runs tests, and debugs. If this loop takes 50 turns, the token consumption compounds. At typical model pricing (~$15/million input, $60/million output), the raw API cost alone is $90 ($75 input + $15 output/reasoning).
On paper, $90 looks like a bargain compared to the human’s $300. But this assumes a 100% success rate on the first run.
In reality, coding agents have a resolve rate of roughly 30% to 40% on benchmarks like SWE-bench. To get a high-confidence solution, you must run the agent in parallel multiple times—often using “best-of-N” sampling. If you run the agent five times to find the best patch, the raw compute cost is $450.00— which is 50% higher than the human developer.
This is where the Jevons Paradox enters.
As LLM providers lower the price per million tokens, we don’t spend less money. Instead, we use the cost savings to build more complex, multi-agent architectures that debate each other, audit code, and run continuous validation loops. We basically “tokenmax.” The efficiency gains are instantly consumed by our appetite for task complexity…sigh!
III—The “Senior Tax” and the Last-Mile Problem
The raw compute cost is only half the battle. The true, hidden economic killer of agentic workflows is the Senior Tax.
When an AI agent finishes a task, it outputs a result. But because AI is probabilistic, that result can’t be trusted. It has to be verified. If an agent writes code, a senior engineer must review the PR. This review process is not cheap. In fact, it is often more expensive than the original labor.
Consider the software engineering pipeline.
If an agent generates a pull request, a senior developer must spend time understanding the agent’s logic, checking for edge cases, and testing the code.
If the code is perfect, the review takes 10 minutes.
If the code has a subtle bug, such as a memory leak that only occurs under high traffic, the senior engineer would probably spend hours debugging it.
Because the agent doesn’t understand the broader system architecture on an intuitive level, its fixes are often local optimizations that can cause global breaks.
A GitClear study analyzed over 150M–200M lines of code and found increased churn/duplication and reduced refactoring since widespread AI assistant adoption, suggesting downward pressure on code maintainability.
If a senior developer earning $150/hour spends two hours fixing a buggy AI-generated pull request, the all-in cost of that task is $390 ($90 compute + $300 human review). BUT…if they had assigned it to a human junior developer earning $40/hour who took four hours to write it clean, the cost would be just $160.
This is the Last-Mile Problem of AI.
Generating 80% of the work is cheap and fast. But resolving the remaining 20% to make it production-ready requires human context and accountability. As long as that last mile requires senior human oversight, the economic advantage of AI remains highly fragile.
IV—The Physical Constraints: Power, Grids, and Capital
To understand why the Compute Wall is likely to persist, we should look at physical compute.
The software industry has spent decades operating in the cloud, detached from physical constraints. But the inference-heavy System-2 AI era is bringing us back down to earth.
Modern AI data centers consume electricity at a scale that strains national grids. An NVIDIA H100 GPU consumes 700 watts at peak, while Blackwell B200 chips draw up to ~1,200 watts. Clustering 100,000 of these chips creates a city’s worth of power demand.
According to the International Energy Agency (IEA), data center electricity consumption could double by 2030+, reaching over 1,000 terawatt-hours which is equivalent to the entire electricity consumption of Japan!!!
This has created a physical power bottleneck.
The wait times to connect a new data center to the utility grid in Northern Virginia now stretch up to five years. There is a severe shortage of high-voltage transformers, with lead times of up to five years for critical grid hardware.
This mismatch between the exponential demand for AI compute and the linear supply of energy infrastructure is driving up the marginal cost of electricity. Hyperscalers are bypassing local grids entirely, buying up dedicated power plants. Microsoft’s deal to resurrect the Three Mile Island nuclear plant to power its AI data centers is not a PR stunt; it is a desperate attempt to secure stable, baseload electricity.
When compute is constrained by physical power grids and multi-billion-dollar nuclear deals, it can’t really be priced at zero. The cost of every token you consume is anchored to the price of copper, uranium, and grid transformers. As long as these physical constraints exist, the cost of compute will remain high, protecting human labor from total economic displacement.
The Human Moat
For the last three years, the narrative has been that human intelligence is somewhat of a rounding error, designed to be automated away by a software curve that marches to zero cost.
But as we move from System-1 to System-2 AI, we’re finding that the curve is bending back. The physics of the grid, the architecture of agentic loops, and the realities of verification are building a wall.
Because in the reasoning era, the ultimate competitive advantage isn’t having the most compute. It’s having the taste, the judgment, and the human oversight to know when compute is actually worth the price.
ICYMI:
Thanks for reading so far 🙏🏼
If you liked the content so far, you can check out our full TWS archive, plus more in-depth content: under the hood and field guides.
I’d love to get some feedback. If you have some time, fill out the following survey.
Want to get in front of our audience? You can partner with us.
Upgrade to the premium version for full access to all content.
Follow me on LinkedIn or X for more musings.
Check out Workstak to get the latest in-depth honest reviews and insights into AI workflow tools to boost your productivity.
If you liked this, please consider sharing it with your network. Thanks for your support!