Most important take away

GPT-5.4’s defining failure is “building infrastructure without judgment” — it will construct elaborate, well-engineered systems and then fail to notice whether the output makes sense (Mickey Mouse got into the production database; 394 unfiltered flags vs. Claude’s 19 actionable ones). The thinking mode vs. auto mode gap is enormous and critically underappreciated: the same model competes for first place in thinking mode and drops to dead last on factual accuracy in auto mode — and 99% of users will encounter auto mode by default.

Summary

Core finding: GPT-5.4 is not the best model, not the worst — it is the most interesting model currently available, with strengths and weaknesses that are sharply defined and task-dependent.

Where GPT-5.4 wins:

• Quantitative modeling: Its Seahawks win probability model (Pythagorean win expectation, Elo-like system with off-season decay, Poisson binomial distribution, self-critique methodology tab) was far more statistically rigorous than Claude’s cleaner but simpler Bradley-Terry approach. The model’s ability to honestly identify its own limitations is a genuine strength.
• File type processing / agentic reach: 99.1% file discovery on a brutal schema migration eval (handwritten receipts, corrupted JSON, multi-tab Excel, OCR) vs. Claude’s 75% — the difference being Claude chose not to install a Python library it needed. For businesses processing heterogeneous document sets, this gap is enormous.
• Long-running agentic tasks: Spent 56 minutes producing a 4,000+ line migration script, 11,000+ line migration report, and 30 database tables — exhaustive and complete, where Claude produced less but more immediately usable output.
• AI self-knowledge: Clearly best in class at understanding the competitive landscape, its own capabilities, and what other models can and can’t do — useful for anyone using AI to learn about AI.

Where GPT-5.4 loses:

• Writing quality: Not a close call. Business writing, executive communication, editorial voice — Claude Opus 4.6 wins clearly. Writing skill is closely tied to product judgment; 5.4 got a gnarly product decision wrong that Claude got right.
• Verbal creativity: Claude found a triple-layered pun and dissected it across three semantic layers; GPT delivered a competent but unremarkable rewrite. Gemini fabricated the source.
• Judgment / data hygiene: The core failure mode — it treats tasks as pipelines to execute, not problems to understand. Mickey Mouse cleared as a real customer. 394 flags with zero categorization vs. Claude’s 19 actionable ones. 278 de-duplicated customers when the correct answer was 176. 13 business status values when the business needed 4–5.
• Speed: 56 minutes vs. Claude’s 15 minutes for the same agentic task.
• Auto mode accuracy: Names 2024 Nobel laureates for a 2025 question. Drops from first/second to dead last on factual retrieval when thinking mode is off.

The strategic picture: GPT-5.4 is intentionally positioned as agentic infrastructure — the substrate for an “OpenAI Claw” style autonomous agent product. The hiring of Peter Steinberger (OpenClaw creator, whose users predominantly preferred Claude) is the tell. Every feature emphasized — computer use, tool search, long-running tasks, reasoning effort controls — points at that architecture. The monthly shipping cadence confirms OpenAI is using AI to build AI faster.

Career advice: Non-technical workers can no longer afford to be naive about the thinking mode / auto mode distinction. Anyone teaching or training teams on GPT-5.4 must build explicit habits around toggling to thinking mode — the version most users will encounter by default is measurably, significantly weaker. If you’re evaluating 5.4 for your team, always test in thinking mode; if your team won’t remember to switch, the results you benchmarked are not the results they’ll get. The people staying ahead are not the ones who read every benchmark — they’re the ones who get into the details of why a model behaves the way it does.

---

Chapter Summaries

Chapter 1: The Car Wash Problem — Setting the Stage

Nate opens with a simple test: “I need to wash my car. The car wash is 100 meters away, should I walk or drive?” GPT-5.4 thinking said walk — delivering a careful, well-structured, completely wrong answer that a child would get right. Claude answered in one sentence: drive. Gemini noted it was a trick question. The point isn’t to pile on: it’s that if you position your model as the best in the world for professional work, it has to hold up to ordinary real-world cases. This failure mode — elaborate reasoning leading to the wrong answer — turns out to be the signature weakness of the model.

Chapter 2: The Eval Suite — How the Testing Was Done

Six structured blind evaluations: outputs labeled by number, independent judging, AI-fluent human checker alongside. Real-world tasks you’d hand a model on a Tuesday and expect to use on Thursday. Scoring across: business and creative writing, verbal creativity, agentic problem solving (the “eval from hell”), epistemic calibration, and AI self-knowledge. TLDR: GPT-5.4 is the most interesting model tested — sometimes crushing first place, sometimes dead last — not the most consistent.

Chapter 3: Writing and Verbal Creativity — Claude Wins

5.4 is a meaningful upgrade from 5.2 but still loses clearly on business writing and creative voice. Claude produced a triple-layered pun dissected across three semantic layers; GPT produced a competent but unremarkable rewrite. Gemini fabricated the source and URL twice. For anyone whose work depends on voice — editorial, strategy memos, product, executive communications — Opus 4.6 is the choice.

Chapter 4: The Eval from Hell — Agentic File Processing

Schema migration from a digital shoebox: handwritten receipts, multiple database schemas, corrupted JSON, multi-tab spreadsheets, VCF contacts. GPT-5.4: 99.1% file discovery, handled OCR, opened Excel with pre-installed libraries. Claude: 75% file discovery — chose not to install a needed Python library, silently skipped XLS files. But GPT let Mickey Mouse (a fake customer) into the production database. 278 customers when the correct answer post-deduplication was 176. 394 flags with zero prioritization vs. Claude’s 19 actionable ones. The strength is extraordinary reach; the weakness is absent judgment about whether the output makes sense.

Chapter 5: Epistemic Calibration — The Thinking Mode Gap

In thinking mode, GPT-5.4 competed for first — correct Higgs boson mass, correct Apple closing price, correct matrix multiplication exponent. In auto mode, it named 2024 Nobel laureates for a 2025 question and dropped to dead last. Same model, same questions, dramatically different results. This is the single most important finding: the model that justifies the press release is thinking mode, but the model 99% of users encounter is auto. Anyone evaluating or deploying this model must account for this gap explicitly.

Chapter 6: Where GPT-5.4 Genuinely Wins — Quantitative Models and AI Self-Knowledge

The Seahawks win probability spreadsheet: GPT produced a 6-tab workbook with Pythagorean win expectation, Elo-like system with off-season decay, Poisson binomial distribution, and a methodology tab that honestly cataloged assumptions and limitations. Claude produced a cleaner 3-tab workbook with simpler analysis. The statistical rigor was not close. GPT also knows the AI landscape better than its competitors do — roughly 90% accuracy on its own capabilities and the competitive landscape. This matters for teams using AI to learn about AI.

Chapter 7: The Core Failure Pattern and What OpenAI Is Building Toward

“Builds infrastructure without judgment” — the unifying weakness. GPT-5.4 constructs elaborate, complete systems and fails to notice whether the output makes sense. Treats tasks as pipelines to execute, not problems to understand. The strategic read: this release is infrastructure for an OpenAI agentic system (analogous to OpenClaw/Claude Code). The hiring of Peter Steinberger, the emphasis on computer use, tool search, long-running tasks, and reasoning effort controls all point the same direction. Monthly shipping cadence is OpenAI demonstrating it is using AI to build faster. The next 2–3 model releases will tell the real story.