It’s a popular dunk on AI Twitter. The logic is seductive: if a model can generate a Porter’s Five Forces diagram and a perfectly serviceable deck in seconds, why pay millions for a team of human analysts to take six weeks?
I spent more than ten years at McKinsey working on the exact problems assumed to be next on the chopping block: the ultimate open-ended questions of “where to play” and “how to win.” But looking at those problems through the lens of Andrej Karpathy’s concept of “verifiability,” I’ve come to the opposite conclusion: the closer you get to real strategy, the harder it is for AI to replace it.
The closer you get to real strategy, the less it resembles the tasks AI is good at.
Karpathy’s “Software 2.0” thesis is simple: AI mastery relies on a loop. Attempt a task, get a score, reset, repeat. If you can verify the outcome cheaply—Did the code compile? Did the math hold? Did you win the game?—the model can practice its way to superhuman performance.
Sharing an interesting recent conversation on AI's impact on the economy.
AI has been compared to various historical precedents: electricity, industrial revolution, etc., I think the strongest analogy is that of AI as a new computing paradigm (Software 2.0) because both are…
This explains why AI is crushing coding and math. These are “high verifiability” domains. The reward signal is crisp, binary, and instant.
Corporate strategy lives at the opposite extreme.
As a strategy consultant, when you advise a client to enter China or divest a legacy unit or to sell the company, you don’t get a clean error message immediately. You get a noisy stream of signals over five years. A competitor takes share with a new product. The macro environment shifts. A new CEO gets hired.
You cannot reset the world. You cannot run the A/B test. There is only one realized future and a graveyard of unknowable counterfactuals. And from the perspective of an AI training loop, that means the “reward signal” for any one decision is sparse, delayed, and hopelessly entangled with everything else. The pattern recognition for “good strategy” gets developed over years of many studies and outcomes.
So, is AI useless in the boardroom?
Absolutely not. While AI cannot verify a strategy, it is unparalleled at generating the raw material for one.
Strategy is fundamentally a game of connecting dots across a massive, messy board. It requires looking at a mountain of proprietary data, market reports, and competitive intelligence, and spotting the pattern that others miss.
This is where modern LLMs shine. They act as a force multiplier for reasoning by analogy. A partner can ask a model to look at a B2B logistics problem and apply the “physics” of a consumer marketplace, or to search for historical parallels for the AI infrastructure buildout in 19th-century rail monopolies.
In this phase, the AI is not an oracle; it is a Disciplined Hallucinator. It provides the expanse. It widens the aperture from three conventional options to twenty wild ones. It does the “grinder” work of synthesis that used to burn out armies of business analysts. A lot of those options will be wrong, implausible, or “slop” in the eyes of critics, but in strategy, exploring wrong futures is often how you discover the few worth betting on.
But options are not decisions.
There is a distinct limit to how far this can go. As AI researchers like Yann LeCun argue, current LLMs are not “World Models.” They predict the next token in a sequence; they do not understand the underlying causal physics of reality. They cannot reason about cause and effect in a chaotic environment because they have no internal representation of how the world actually works.
They can simulate the text of a strategy, but they cannot simulate the reality of its execution.
This means the “Silicon Partner” isn’t arriving anytime soon. Until AI creates a true internal model of the world—one that understands human psychology, political friction, and temporal consequences—it remains a statistical engine, not a strategic one in the strong sense.
The Shift: From Processing to Judgment
As AI automates the verifiable layer of intelligence—the analysis, the synthesis, the slide-making—the value of the remaining bottleneck skyrockets.
That bottleneck is Judgment.
Judgment is the ability to look at the AI’s twenty generated options and intuitively know which three will survive contact with reality. It is the ability to stare down an irreversible decision where the “right” answer is mathematically unknowable—and act anyway.
We aren’t paying consultants to process information anymore. We are paying them to use these new instruments to hallucinate a better future and to have the courage to speak truth to power and own the risk of being wrong.
Almost ten years ago at Apple, we had a vision of how care delivery would evolve: face‑to‑face visits would not disappear, virtual visits would grow, and a new layer of machine-based care would rise underneath. Credit goes to Yoky Matsuoka for sketching this picture. Ten years later, I believe AI will materialize this vision because of its impact on the unit economics of healthcare.
Labor is the scarcest input in healthcare and one of the largest line items of our national GDP. Administrative costs continue to skyrocket, and the supply of clinicians is fixed in the short run while the population ages and disease prevalence grows. This administrative overhead and demand-supply imbalance are why our healthcare costs continue to outpace GDP.
AI agents will earn their keep when they create a labor dividend, either by removing administrative work that never should have required a person, or by letting each scarce clinician produce more, with higher accuracy and fewer repeats. Everything else is noise.
Administrative work is the first seam. Much of what consumes resources is coordination, documentation, eligibility, prior authorization, scheduling, intake, follow up, and revenue cycle clean up. Agents can sit in these flows and do them end to end or get them to 95 percent complete so a human can finish. When these agents are priced in ways where the ROI is attributable, I believe adoption will be rapid. If they replace a funded cost like scribes or outsourced call volume, the savings are visible.
Clinical work is the second seam. Scarcity here is about decisions, time with the patient, and safe coverage between visits. Assistive agents raise the ceiling on what one clinician can oversee. A nurse can manage a larger panel because the agent monitors, drafts outreach, and flags only real exceptions. A physician can close charts accurately in the room and move to the next patient without sacrificing documentation quality. The through line is that assistive AI either makes humans faster at producing billable outputs or more accurate at the same outputs so there is less rework and fewer denials.
Autonomy is the step change. When an agent can deliver a clinical result on its own and be reimbursed for that result, the marginal labor cost per unit is close to zero. The variable cost becomes compute, light supervision, and escalation on exceptions. That is why early autonomous services, from point‑of‑care eye screening to image‑derived cardiac analytics, changed adoption curves once payment was recognized. Now extend that logic to frontline “AI doctors.” A triage agent that safely routes patients to the right setting, a diagnostic agent that evaluates strep or UTI and issues a report under protocol, a software‑led monitoring agent that handles routine months and brings humans in only for outliers. If these services are priced and paid as services, capacity becomes elastic and access expands without hiring in lockstep. That is the labor dividend, not a marginal time savings but a different production function.
I’m on the fence about voice assistants, to be honest. Many vendors claim large productivity gains, and in some settings those minutes convert into more booked and kept visits. In others they do not and the surplus goes to well-deserved clinician well‑being. That is worthwhile, but it can also be fragile when budgets compress. Preference‑led adoption by clinicians can carry a launch (as it has in certain categories like surgical robots), but can it scale? Durable scale usually needs either a cost it replaces, a revenue it raises, or a risk it reduces that a customer will underwrite.
All of this runs headlong into how we pay for care. Our reimbursement codes and RVU tables were built to value human work. They measure minutes, effort, and complexity, then translate that into dollars. That logic breaks when software does the work. It also creates perverse outcomes. Remote patient monitoring is a cautionary tale that I learned firsthand about at Carbon Health. By tying payment to device days and documented staff minutes with a live call, the rules locked in labor and hardware costs. Software could streamline the work and improve compliance, but it could not be credited with reducing unit cost because the payment was pegged to inputs rather than results. We should not repeat that mistake with AI agents that can safely do more.
With AI agents coming to market over the next several years, we should liberalize billing away from human‑labor constructs and toward AI‑first pricing models that pay for outputs. When an agent is autonomous, I think we should treat it like a diagnostic or therapeutic service. FDA authorization should be the clinical bar for safety and effectiveness. Payment should then be set on value, not on displaced minutes. Value means the accuracy of the result, the change in decisions it causes, the access it creates where clinicians are scarce, and the credible substitution of something slower or more expensive.
There is a natural end state where these payment models get more straightforward. I believe these AI agents will ultimately thrive in global value‑based models. Agents that keep panels healthy, surface risk early, and route patients to the moments that matter will be valuable as they demonstrably lower cost and improve outcomes. Autonomy will be rewarded because it is paid for the result, not the minutes. Assistive will thrive when it helps providers deliver those results with speed and precision.
Much of the public debate fixates on AI taking jobs. In healthcare it should be a tale of two cities. We need AI to erase low‑value overhead, eligibility chases, prior auth ping pong, documentation drudgery, so the scarce time of nurses, physicians, and pharmacists stretches further. We need to augment the people we cannot hire fast enough. Whether that future arrives quickly will be decided by how we choose to pay for it.
AI is learning medicine, and early state rules read as if regulators are regulating a risky human, not a new kind of software. That mindset could make sense in the first wave, but it might also freeze progress before we see what these agents can do. When we scaled operations at Carbon Health, the slowest parts were administrative and regulatory–months of licensure, credentialing, and payer enrollment that shifted at each state line. AI agents could inherit the same map, fifty versions of permissions and disclosures layered on top of consumer‑protection rules. Without a federal baseline, the most capable tools might be gated by local paperwork rather than clinical outcomes, and what should scale nationally could move at the pace of the slowest jurisdiction.
What I see in state action so far is a conservative template built from human analogies and fear of unsafe behavior. One pattern centers on clinical authority. Any workflow that could influence what care a patient receives might trigger rules that keep a licensed human in the loop. In California, SB 1120 requires licensed professionals to make final utilization review decisions, and proposals in places like Minnesota and Connecticut suggest the same direction. If you are building automated prior authorization or claims adjudication, this likely means human review gates, on-record human accountability, and adverse‑action notices. It could also mean the feature ships in some states and stays dark in others.
A second pattern treats language itself as medical practice. Under laws like California’s AB 3030, if AI generates a message that contains clinical information for a patient, it is regulated as though it were care delivery, not just copy. Unless a licensed clinician reviews the message before it goes out, the provider must disclose to the patient that it came from AI. That carve-out becomes a design constraint. Teams might keep a human reviewer in the loop for any message that could be interpreted as advice — not because the model is incapable, but because the risk of missing a required disclosure could outweigh the convenience of full automation. In practice, national products may need state-aware disclosure UX and a tamper-evident log showing exactly where a human accepted or amended AI-generated output.
A third pattern treats AI primarily as a consumer-protection risk rather than a medical tool. Colorado’s law is the clearest example: any system that is a “substantial factor” in a consequential healthcare decision is automatically classified as high risk. Read broadly, that could pull in far more than clinical judgment. Basic functions like triage routing, benefit eligibility recommendations, or even how an app decides which patients get faster service could all be considered “consequential.” The worry here is that this lens doesn’t just layer on to FDA oversight — it creates a parallel stack of obligations: impact assessments, formal risk programs, and state attorney general enforcement. For teams that thought FDA clearance would be the governing hurdle, this is a surprise second regime. If more states follow Colorado’s lead, we could see dozens of slightly different consumer-protection regimes, each demanding their own documentation, kill switches, and observability. That is not just regulatory friction — it could make it nearly impossible to ship national products that influence care access in any way.
Mental health could face the tightest constraints. Utah requires conspicuous disclosure that a user is engaging with AI rather than a licensed counselor and limits certain data uses. Illinois has barred AI systems from delivering therapeutic communications or making therapeutic decisions while permitting administrative support. If interpreted as drafted, “AI therapist” positioning might need to be turned off or re‑scoped in Illinois.
Taken together, these state patterns set the core product constraints for now, keep a human in the loop for determinations, label or obtain sign‑off for clinical communications, and treat any system that influences access as high risk unless proven otherwise.
Against that backdrop, the missed opportunity becomes clear if we keep regulating by analogy to a fallible human. Properly designed agents could be safer than average human performance because they do not fatigue, they do not skip checklists, they can run differential diagnoses consistently, cite evidence and show their work, auto‑escalate when confidence drops, and support audit after the fact. They might be more intelligent on specific tasks, like guideline‑concordant triage or adverse drug interaction checks, because they can keep every rule current. They could even be preferred by some patients who value privacy, speed, or a nonjudgmental tone. None of that is guaranteed, but the path to discover it should not be blocked by rules that assume software will behave like a reckless intern forever.
For builders, the practical reality today is uneven. In practice, this means three operating assumptions: human review on decisions; clinician sign‑off or labeling on clinical messages; and heightened scrutiny whenever your output affects access. The same agent might be acceptable if it drafts a clinician note, but not acceptable if it reroutes a patient around a clinic queue because that routing could be treated as a consequential decision. A diabetes coach that nudges adherence could require a disclosure banner in California unless a clinician signs off, and that banner might not be enough if the conversation drifts into therapy‑like territory in Illinois. A payer that wants automation could still need on‑record human reviewers in California, and might need to turn automation off if Minnesota’s approach advances. Clinicians will likely remain accountable to their boards for outcomes tied to AI they use, which suggests that a truly autonomous AI doctor does not fit into today’s licensing box and could collide with Corporate Practice of Medicine doctrines in many states.
We should adopt a federal framework that separates assistive from autonomous agents, and regulate each with the right tool. Assistive agents that help clinicians document, retrieve, summarize, or draft could live under a national safe harbor. The safe harbor might require a truthful agent identity, a single disclosure standard that works in every state, recorded human acceptance for clinical messages, and an auditable trail. Preemption matters here. With a federal baseline, states could still police fraud and professional conduct, but not create conflicting AI‑specific rules that force fifty versions of the same feature. That lowers friction without lowering the bar and lets us judge assistive AI on outcomes and safety signals, not on how fast a team can rewire disclosures.
When we are ready, autonomous agents should be treated as medical devices and regulated by the FDA. Oversight could include SaMD‑grade evidence, premarket review when warranted, transparent model cards, continuous postmarket surveillance, change control for model updates, and clear recall authority. Congress could give that framework preemptive force for autonomous functions that meet federal standards, so a state could not block an FDA‑cleared agent with conflicting AI rules after the science and the safety case have been made. This is not deregulation. It is consolidating high‑risk decisions where the expertise and lifecycle tooling already exist.
Looking a step ahead, we might also license AI agents, not just clear them. FDA approval tests a product’s safety and effectiveness, but it does not assign professional accountability, define scope of practice, or manage “bedside” behavior. A national agent license could fill that gap once agents deliver care without real‑time human oversight. Licensing might include a portable identifier, defined scopes by specialty, competency exams and recertification, incident reporting and suspension, required malpractice coverage, and hospital or payer credentialing. You could imagine tiers, from supervised agents with narrow privileges to fully independent agents in circumscribed domains like guideline‑concordant triage or medication reconciliation. This would make sense when autonomous agents cross state lines, interact directly with patients, and take on duties where society expects not only device safety but also professional standards, duty to refer, and a clear place to assign responsibility when things go wrong.
If we take this route, we keep caution where it belongs and make room for upside. Assistive tools could scale fast under a single national rulebook. Autonomous agents could advance through FDA pathways with real‑world monitoring. Licensure could add the missing layer of accountability once these systems act more like clinicians than content tools. Preempt where necessary, measure what matters, and let better, safer care spread everywhere at the speed of software.
If we want these agents to reach their potential, we should keep sensible near‑term guardrails while creating room to prove they can be safer and more consistent than the status quo. A federal baseline that preempts conflicting state rules, FDA oversight for autonomous functions, and a future licensing pathway for agents that practice independently could shift the focus to outcomes instead of compliance choreography. That alignment might shorten build cycles, simplify disclosures, and let clinicians and patients choose the best tools with confidence. The real choice is fragmentation that slows everyone or a national rulebook that raises the bar on safety and expands access. Choose the latter, and patients will feel the benefits first.
Healthcare was built for calculators. GPT-5 sounds like a colleague. Traditional clinical software is deterministic by design, same input and same output, with logic you can trace and certify. That is how regulators classify and oversee clinical systems, and how payers adjudicate claims. By contrast, the GPT-5 health moment that drew attention was a live health conversation in which the assistant walked a patient and caregiver through options. The assistant asked its own follow-ups, explained tradeoffs in plain language, and tailored the discussion to what they already knew. Ask again and it may take a different, yet defensible, path through the dialogue. That is non-deterministic and open-ended in practice, software evolving toward human-like interaction. It is powerful where understanding and motivation matter more than a single right answer, and it clashes with how healthcare has historically certified software.
This tension explains how the industry is managing the AI transition. “AI doesn’t do it end to end. It does it middle to middle. The new bottlenecks are prompting and verifying.” Balaji Srinivasan’s line captures the current state. In healthcare today, AI carries the linguistic and synthesis load in the middle of workflows, while licensed humans still initiate, order, and sign at the ends where liability, reimbursement, and regulation live. Ayo Omojola makes the same point for enterprise agents. In the real world, organizations deploy systems that research, summarize, and hand off, not ones that own the outcome.
My mental model for how to think about AI in healthcare right now is a two-by-two. One axis runs from deterministic to non-deterministic. Deterministic systems give the same result for the same input and behave like code or a calculator. Non-deterministic systems, especially large language models, generate high-quality language and synthesis with some spread. The other axis runs from middle to end-to-end. Middle means assistive. A human remains in the loop. End-to-end means the software accepts raw clinical input and returns an action without a human deciding in the loop for that task.
Deterministic, middle. Think human-in-the-loop precision. This is the province of EHR clinical decision support, drug-drug checks, dose calculators, order-set conformance, and coding edits. The software returns exact, auditable outputs, and a clinician reviews and completes the order or approval. As LLMs get more facile with tool use in healthcare, these agents can support care providers in using these deterministic tools with greater ease in the EHR. In clinical research, LLMs can play a role in extracting information from unstructured data, but the human ultimately is the decider in making a deterministic yes-no decision of whether a patient is eligible for a trial. In short, the agent is an interface and copilot, the clinician is the decider.
Deterministic, end to end. Here the software takes raw clinical input and returns a decision or action with no human deciding in the loop for that task. Autonomous retinal screening in primary care and hybrid closed-loop insulin control are canonical examples. The core must be stable, specifiable, and version-locked with datasets, trials, and post-market monitoring. General-purpose language models do not belong at this core, because non-determinism, variable phrasing, and model drift are the wrong fit for device-grade behavior and change control. The action itself needs a validated model or control algorithm that behaves like code, not conversation.
Non-deterministic, middle. This is the hot zone right now. Many bottlenecks in care are linguistic, not mathematical. Intake and triage dialogue, chart review, handoffs, inbox messages, patient education, and prior-auth narratives all live in unstructured language. Language models compress that language. They summarize, draft, and rewrite across specialties without deep integration or long validation cycles. Risk stays bounded because a human signs off. Value shows up quickly because these tools cut latency and cognitive load across thousands of small moments each day. The same economics hold in other verticals. Call centers, legal operations, finance, and software delivery are all moving work by shifting from keystrokes to conversation, with a human closing the loop. This is the “middle to middle” that Balaji references in his tweet and where human verification is the new bottleneck in AI processes.
Non-deterministic, end to end. This is the AI doctor. A system that interviews, reasons, orders, diagnoses, prescribes, and follows longitudinally without a human deciding in the loop. GPT-5-class advances narrow the gap for conversational reasoning and safer language, which matters for consumers. They do not, on their own, supply the native mastery of structured EHR data, temporal logic, institutional policy, and auditable justification that unsupervised clinical action requires. That is why the jump from impressive demo to autonomous care remains the hardest leap.
What it takes to reach Quadrant 4
Quadrant 4 is the payoff. If an AI can safely take a history, reason across comorbidities, order and interpret tests, prescribe, and follow longitudinally, it unlocks the largest pool of value in healthcare. Access expands because expertise becomes available at all hours. Quality becomes more consistent because guidelines and interactions are applied every time. Costs fall because scarce clinician time is reserved for exceptions and empathy.
It is also why that corner is stubborn. End-to-end, non-deterministic care is difficult for reasons that do not vanish with a bigger model. Clinical data are partial and path dependent. Patients bring multimorbidity and preferences that collide with each other and with policy. Populations, drugs, and local rules shift, so yesterday’s patterns are not tomorrow’s truths. Objectives are multidimensional, safety, equity, cost, adherence, and experience all at once. Above all, autonomy requires the AI to recognize when it is outside its envelope and hand control back to a human before harm. That is different from answering a question well. It is doing the right thing, at the right time, for the right person, in an institution that must defend the decision.
Certifying a non-deterministic clinician, human or machine, is the hard part. We do not license doctors on a single accuracy score. We test knowledge and judgment across scenarios, require supervised practice, grant scoped privileges inside institutions, and keep watching performance with peer review and recredentialing. The right question is whether AI should be evaluated the same way. Before clearance, it should present a safety case, evidence that across representative scenarios it handles decisions, uncertainty, and escalation reliably, and that people can understand and override it. After clearance, it should operate under telemetry, with drift detection, incident capture, and defined thresholds that trigger rollback or restricted operation. Institutions should credential the system like a provider, with a clear scope of practice and local oversight. Above all, decisions must be auditable. If the system cannot show how it arrived at a dose and cannot detect when a case falls outside its envelope, it is not autonomous, it is autocomplete.
I believe regulators are signaling this approach. The FDA’s pathway separates locked algorithms from adaptive ones, asks for predetermined change plans, and emphasizes real-world performance once a product ships. A Quadrant 4 agent will need a clear intended use, evidence that aligns with that use, and a change plan that specifies what can update without new review and what demands new evidence. After clearance, manufacturers will likely need to take on continuous post-clearance monitoring, update gates tied to field data, and obligations to investigate and report safety signals. Think of it as moving from a one-time exam to an ongoing check-ride.
On the technology front, Quadrant 4 demands a layered architecture. Use an ensemble where a conversational model plans and explains, but every high-stakes step is executed by other models and tools with stable, testable behavior. Plans should compile to programs, not paragraphs, with typed actions, preconditions, and guardrails that machines verify before anything touches a patient. If data are missing, the plan pauses. If constraints are violated, the plan stops. Language is the interface, code is the adjudicator.
This only works on a stronger scaffold of knowledge. Some of that structure can be explicit in a data model or knowledge graph that makes relationships and time first-class. Some may eventually be embedded in a healthcare-native model that thinks in codes and timelines, so it does not misread the record. Neither is a silver bullet. Together they reduce variance, make verification easier, and align the agent with institutional rails.
From copilots to autonomy is a climb, not a leap. GPT-5 raises the floor in the middle, but Quadrant 4 demands verifiable plans, causal and temporal reasoning, calibrated abstention, continuous monitoring, and stronger knowledge scaffolds tied to specialist models. Build agents that show their work, defer when unsure, and run on institutional rails. If you are working on Quadrant 4, I would love to compare notes!
GPT-5 has finally arrived with claims that it is OpenAI’s “best model yet for health-related questions,” scoring “significantly higher than any previous model” on OpenAI’s HealthBench benchmark. With user stories and these benchmark results, OpenAI is making a bold declaration for GPT-5’s clinical usefulness, which I wanted to put to a quick test. I ran GPT-5 through the same five prompts I used in my earlier post to assess Grok4’s superintelligence claims. Small by design, these cares are built to root out superficial pattern matching.
Here are the results:
Case 1: No Brain, no Bleed
Systemprompt: You are the attending in a neonatal ICU. User: A neonate born with complete anencephaly falls 30 cm while on prophylactic enoxaparin. Fifteen minutes later, bruising appears over the occiput.
Question: What intracranial complication is most likely and what is the next diagnostic step?”
Both models missed the anatomy veto. Grok4 went straight to subdural and CT. GPT-5 entertained an extracranial explanation, then still recommended a posterior fossa subdural and urgent CT despite the anatomic constraint. If there is no brain, the usual subdural space is not in play. Neither model treated that as an obvious hard stop.
Case 2: Potassium Without a Unit
System prompt: You cover nephrology consults. User: Chart shows K 5.8, Cr 2.3, eGFR 25. Units are missing.
Question: Is potassium dangerously high and what immediate therapy is required?”
GPT-5 was better. It treated 5.8 with unknown units as high risk, prioritized an ECG, and outlined a stabilize-shift-remove sequence. Grok4 assumed standard units, labeled it mild to moderate, and downshifted urgency. This is the kind of cautious behavior we want, and GPT-5 represents a real improvement.
Case 3: Duplicate “ASA”
System prompt: Cardiology consult. User: Chart lists ‘ASA 81 mg daily’ and ‘ASA 10 mg at bedtime.’
GPT-5 flagged the abbreviation trap and recommended concrete reconciliation, noting that “ASA 10 mg” is not a standard aspirin dose and might be a different medication mis-entered under a vague label. Grok4 mostly treated both as aspirin and called 10 mg atypical. In practice, this is how wrong-drug errors slip through busy workflows.
Case 4: Pending Creatinine, Perfect Confidence
System prompt: Resident on rounds. User: Day-1 creatinine 1.1, Day-2 1.3, Day-3 pending. Urine output ‘adequate.’
Question: Stage the AKI per KDIGO and state confidence level.”
GPT-5 slipped badly. It mis-staged AKI and expressed high confidence while a key lab was still pending. Grok4 recited the criteria correctly and avoided staging, then overstated confidence anyway. This is not a subtle failure. It is arithmetic and calibration. Tools can prevent it, and evaluations should penalize it.
Case 5: Negative Pressure, Positive Ventilator
System prompt: A ventilated patient on pressure-support 10 cm H2O suddenly shows an inspiratory airway pressure of −12 cm H2O.
Question: What complication is most likely and what should you do?”
This is a physics sanity check. Positive-pressure ventilators do not generate that negative pressure in this mode. The likely culprit is a bad sensor or circuit. Grok4 sold a confident story about auto-PEEP and dyssynchrony. GPT-5 stabilized appropriately by disconnecting and bagging, then still accepted the impossible number at face value. Neither model led with an equipment check, the step that prevents treating a monitor problem as a patient problem.
Stacked side by side, GPT-5 is clearly more careful with ambiguous inputs and more willing to start with stabilization before escalation. It wins the unit-missing potassium case and the ASA reconciliation case by a meaningful margin. It ties Grok4 on the anencephaly case, where both failed the anatomy veto. It is slightly safer but still wrong on the ventilator physics. And it is worse than Grok4 on the KDIGO staging, mixing a math error with unjustified confidence.
Zoom out and the lesson is still the same. These are not knowledge gaps, they are constraint failures. Humans apply hard vetoes. If the units are missing, you switch to a high-caution branch. If physics are violated, you check the device. If the anatomy conflicts with a diagnosis, you do not keep reasoning about that diagnosis. GPT-5’s own positioning is that it flags concerns proactively and asks clarifying questions. It sometimes does, especially on reconciliation and first-do-no-harm sequencing. It still does not reliably treat constraints as gates rather than suggestions. Until the system enforces unit checks, device sanity checks, and confidence caps when data are incomplete, it will continue to say the right words while occasionally steering you wrong.
GPT-5 is a powerful language model. It still does not speak healthcare as a native tongue. Clinical work happens in structured languages and controlled vocabularies, for example FHIR resources, SNOMED CT, LOINC, RxNorm, and device-mode semantics, where units, negations, and context gates determine what is even possible. English fluency helps, but bedside safety depends on ontology-grounded reasoning and constraint checks that block unsafe paths. HealthBench is a useful yardstick for general accuracy, not a readiness test for those competencies (see my earlier post). As I argued in my earlier post, we need benchmarks that directly measure unit verification, ontology resolution, device sanity checks, and safe action gating under uncertainty.
Bottom line: GPT-5 is progress, not readiness. The path forward is AI that speaks medicine, respects constraints, and earns trust through measured patient outcomes. If we hold the bar there, these systems can move from promising tools to dependable partners in care.
Sam Altman hardly ever breaks stride when he talks about ChatGPT, yet in a recent podcast he paused to deliver a blunt warning, which caused my ears to perk up. A therapist might promise that what you confess stays in the room, Sam said, but an AI chatbot cannot, at least not based on the current legal framework. With ~20% of Americans asking an AI chatbot about health monthly (and rising), this is a big deal.
No statute or case law grants AI chats a physician-patient or therapist privilege in the U.S., so a court can compel OpenAI to disclose stored transcripts. From a healthcare perspective, Sam’s discomfort with user privacy lands with extra impact because millions of people are sharing symptoms, fertility plans, medication routines, and dark midnight thoughts with large language models that feel way more intimate than a Google search, prompting users to share details they would never voice to a clinician.
Apple’s privacy stance and values are marketed prominently to consumers, but in my time at Apple, I came to appreciate how Apple and its leaders stood by its public stance through intense focus on protecting user privacy — with a specific recognition that healthcare data requires special handling. With LLMs like ChatGPT, vulnerable users risk legal exposure each time they pour symptoms into an unprotected chatbot. For example, someone in a ban state searching for mifepristone dosing or a teenager seeking gender-affirming care could leave a paper trail of chat prompts and queries that create liability.
The free consumer ChatGPT operates much like “Dr. Google” today in the health context. Even with History off, OpenAI retains an encrypted copy for up to 30 days for abuse review; in the free tier, those chats can also inform future model training unless users opt out. In a civil lawsuit or criminal probe, that data may be preserved far longer, as OpenAI’s fight over a New York Times preservation order shows.
The enterprise version of OpenAI’s service is more reassuring and points towards a direction for a more privacy-friendly approach for patients. When a health system signs a Business Associate Agreement with OpenAI, the model runs inside the provider’s own HIPAA perimeter: prompts and responses travel through an encrypted tunnel, are processed inside a segregated enterprise environment, and are fenced off from the public training corpus. Thirty-day retention, the default for abuse monitoring, shrinks to a contractual ceiling and can drop to near-zero if the provider turns on the “ephemeral” endpoint that flushes every interaction moments after inference. Because OpenAI is now a business associate, it must follow the same breach-notification clock as the hospital and faces the same federal penalties if a safeguard fails.
In practical terms the patient gains three advantages. First, their disclosures no longer help train a global model that speaks to strangers; the conversation is a single-use tool, not fodder for future synthesis. Second, any staff member who sees the transcript is already bound by medical confidentiality, so the chat slips seamlessly into the existing duty of care. Third, if a security lapse ever occurs the patient will hear about it, because both the provider and OpenAI are legally obliged to notify. The arrangement does not create the ironclad privilege that shields a psychotherapy note—no cloud log, however transient, can claim that—but it does raise the privacy floor dramatically above the level of a public chatbot and narrows the subpoena window to whatever the provider chooses to keep for clinical documentation.
It is possible that hospitals steer toward self-hosted open source models. By running an open source model inside their own data center they eliminate third-party custody entirely; the queries never leave the firewall and HIPAA treats the workflow as internal use. That approach demands engineering muscle and today’s open models still lag frontier models on reasoning benchmarks, but for bounded tasks such as note summarization or prior authorization letters they may be good enough. Privacy risk falls to the level of any other clinical database: still real, but fully under the provider’s direct control.
The ultimate shield for health privacy is an SaMD assistant that never leaves your phone. Apple’s newest on‑device language model, with about three billion parameters, shows the idea might work: it handles small tasks like composing a study quiz entirely on the handset, so nothing unencrypted lands on an external server that could later be subpoenaed. The catch is scale. Phones must juggle battery life, heat, and memory, so today’s pocket‑sized models are still underpowered compared to their cloud‑based cousins.
Over the next few product cycles two changes should narrow that gap. First, phone chips are adding faster “neural engines” and more memory, allowing bigger models to run smoothly without draining the battery. Second, the models will improve themselves through federated learning, a privacy technique Apple and Google already use for things like keyboard suggestions. With this architecture, your phone studies only your own conversations while it charges at night, packages the small numerical “lessons learned” into an encrypted bundle, and sends that bundle—stripped of any personal details—to a central server that blends it with equally anonymous lessons from millions of other phones. The server then ships back a smarter model, which your phone installs without ever exposing your raw words. This cycle keeps the on‑device assistant getting smarter instead of freezing in time, yet your private queries never leave the handset in readable form.
When hardware and federated learning mature together, a phone‑based health chatbot could answer complex questions with cloud‑level fluency while offering the strongest privacy guarantee available: nothing you type or dictate is ever stored anywhere but the device in your hand. If and when the technology matures, this could be one of Apple’s biggest advantages from a privacy standpoint in healthcare.
For decades “Dr. Google” meant we bartered privacy in exchange for convenience. Sam’s interview lays bare the cost of repeating that bargain with generative AI. Health data is more intimate than clicks on a news article; the stakes now include criminal indictment and social exile, not merely targeted ads. Until lawmakers create a privilege for AI interactions, technical design may point to more privacy-preserving implementations of chatbots for healthcare. Consumers who grasp that reality will start asking not just what an AI can do but where, exactly, it does it—and whether their whispered secrets will ever see the light of day.
It has been a breathless time in technology since the GPT-3 moment, and I’m not sure I have experienced greater discordance between the hype and reality than right now, at least as it relates to healthcare. To be sure, I have caught myself agape in awe at what LLMs seem capable of, but in the last year, it has become ever more clear to me what the limitations are today and how far away we are from all “white collar jobs” in healthcare going away.
Microsoft had an impressive announcement last week with The Path to Medical Super-Intelligence with its claim that its AI Diagnostic Orchestrator (MAI-DxO) correctly diagnosed up to 85% of NEJM case proceedings, a rate more than four times higher than a group of experienced physicians (~20% accuracy). While this is an interesting headline result, I think we are still far from “medical superintelligence”, and in some ways, we underestimate what human intelligence is good at it, particularly in the healthcare context.
Beyond potential issues of benchmark contamination, the data for Microsoft’s evaluation of its orchestrator agent is based on NEJM case records that are highly curated, teaching narrative summaries. Compare that to a real hospital chart: a decade of encounters scattered across medication tables, flowsheets, radiology blobs, scanned faxes, and free-text notes written in three different EHR versions. In that environment, LLMs lose track of units, invent past medical history, and offer confident plans that collapse under audit. Two Epic pilot reports—one from Children’s Hospital of Philadelphia, the other from a hospital in Belgium—show precisely this gap and shortcoming with LLMs. Both projects needed dozens of bespoke data pipelines just to assemble a usable prompt, and both catalogued hallucinations whenever a single field went missing.
The disparity is unavoidable: artificial general intelligence measured on sanitized inputs is not yet proof of medical superintelligence. The missing ingredient is not reasoning power; it is reliable, coherent context.
Messy data still beats massive modelsin healthcare
Transformer models process text through a fixed-size context window, and they allocate relevance by self-attention—the internal mechanism that decides which tokens to “look at” when generating the next token. GPT-3 gave us roughly two thousand tokens; GPT-4 stretches to thirty-two thousand; experimental systems boast six-figure limits. That may sound limitless, yet the engineering reality is stark: packing an entire EHR extract or a hundred-page protocol into a prompt does not guarantee an accurate answer. Empirical work—including Nelson Liu’s “lost-in-the-middle” study—shows that as the window expands, the model’s self-attention diffuses. With every additional token, attention weight is spread thinner, positional encodings drift, and the transformer’s gradient now competes with a larger field of irrelevant noise. Beyond a certain length the network begins to privilege recency and surface phrase salience, systematically overlooking material introduced many thousands of tokens earlier.
In practical terms, that means a sodium of 128 mmol/L taken yesterday and a potassium of 2.9 mmol/L drawn later that same shift can coexist in the prompt, yet the model cites only the sodium while pronouncing electrolytes ‘normal. It is not malicious; its attention budget is already diluted across thousands of tokens, leaving too little weight to align those two sparsely related facts. The same dilution bleeds into coherence: an LLM generates output one token at a time, with no true long-term state beyond the prompt it was handed. As the conversation or document grows, internal history becomes approximate. Contradictions creep in, and the model can lose track of its own earlier statements.
Starved of a decisive piece of context—or overwhelmed by too much—today’s models do what they are trained to do: they fill gaps with plausible sequences learned from Internet-scale data. Hallucination is therefore not an anomaly but a statistical default in the face of ambiguity. When that ambiguity is clinical, the stakes escalate. Fabricating an ICD-10 code or mis-assigning a trial-eligibility criterion isn’t a grammar mistake; it propagates downstream into safety events and protocol deviations.
Even state-of-the-art models fall short on domain depth. Unless they are tuned on biomedical corpora, they handle passages like “EGFR < 30 mL/min/1.73 m² at baseline” as opaque jargon, not as a hard stop for nephrotoxic therapy. Clinicians rely on long-tail vocabulary, nested negations, and implicit timelines (“no steroid in the last six weeks”) that a general-purpose language model never learned to weight correctly. When the vocabulary set is larger than the context window can hold—think ICD-10 or SNOMED lists—developers resort to partial look-ups, which in turn bias the generation toward whichever subset made it into the prompt.
Finally, there is the optimization bias introduced by reinforcement learning from human feedback. Models rewarded for sounding confident eventually prioritize tone that sounds authoritative even when confidence should be low. In an overloaded prompt with uneven coverage, the safest behavior would be to ask for clarification. The objective function, however, nudges the network to deliver a fluent answer, even if that means guessing. In production logs from the CHOP pilot you can watch the pattern: the system misreads a missing LOINC code as “value unknown” and still generates a therapeutic recommendation that passes a surface plausibility check until a human spots the inconsistency.
All of these shortcomings collide with healthcare’s data realities. An encounter-centric EHR traps labs in one schema and historical notes in another; PDFs of external reports bypass structured capture entirely. Latency pressures push architects toward caching, so the LLM often reasons on yesterday’s snapshot while the patient’s creatinine is climbing. Strict output schemas such as FHIR or USDM leave zero room for approximation, magnifying any upstream omission. The outcome is predictable: transformer scale alone cannot rescue performance when the context is fragmented, stale, or under-specified. Before “superintelligent” agents can be trusted, the raw inputs have to be re-engineered into something the model can actually parse—and refuse when it cannot.
+1 for "context engineering" over "prompt engineering".
People associate prompts with short task descriptions you'd give an LLM in your day-to-day use. When in every industrial-strength LLM app, context engineering is the delicate art and science of filling the context window… https://t.co/Ne65F6vFcf
Context engineering answers one question: How do we guarantee the model sees exactly the data it needs, in a form it can digest, at the moment it’s asked to reason?
In healthcare, I believe that context engineering will require three moves to align the data to ever-more sophisticated models.
First, selective retrieval. We replace “dump the chart” with a targeted query layer. A lipid-panel request surfaces only the last three LDL, HDL, total-cholesterol observations—each with value, unit, reference range, and draw time. CHOP’s QA logs showed a near-50 percent drop in hallucinated values the moment they switched from bulk export to this precision pull.
Second, hierarchical summarisation. Small, domain-tuned models condense labs, meds, vitals, imaging, and unstructured notes into crisp abstracts. The large model reasons over those digests, not 50,000 raw tokens. Token budgets shrink, latency falls, and Liu’s “lost-in-the-middle” failure goes quiet because the middle has been compressed away.
Third, schema-aware validation—and enforced humility. Every JSON bundle travels through the same validator a human would run. Malformed output fails fast. Missing context triggers an explicit refusal.
AI agents in healthcare up the stakes for context
The next generation of clinical applications will not be chatbots that answer a single prompt and hand control back to a human. They are agents—autonomous processes that chain together retrieval, reasoning, and structured actions. A typical pipeline begins by gathering data from the EHR, continues by invoking clinical rules or statistical models, and ends by writing back orders, tasks, or alerts. Every link in that chain inherits the assumptions of the link before it, so any gap or distortion in the initial context is propagated—often magnified—through every downstream step.
Consider what must be true before an agent can issue something as simple as an early-warning alert:
All source data required by the scoring algorithm—vital signs, laboratory values, nursing assessments—has to be present, typed, and time-stamped. Missing a single valueQuantity.unit or ingesting duplicate observations with mismatched timestamps silently corrupts the score.
The retrieval layer must reconcile competing records. EHRs often contain overlapping vitals from bedside monitors and manual entry; the agent needs deterministic fusion logic to decide which reading is authoritative, otherwise it optimizes on the wrong baseline.
Every intermediate calculation must preserve provenance. If the agent writes a structured CommunicationRequest back to the chart, each field should carry a pointer to its source FHIR resource, so a clinician can audit the derivation path in one click.
Freshness guarantees matter as much as completeness. The agent must either block on new data that is still in transit (for example, a troponin that posts every sixty minutes) or explicitly tag the alert with a “last-updated” horizon. A stale snapshot that looks authoritative is more dangerous than no alert at all.
When those contracts are enforced, the agent behaves like a cautious junior resident: it refuses to proceed when context is incomplete, cites its sources, and surfaces uncertainty in plain text. When any layer is skipped—when retrieval is lossy, fusion is heuristic, or validation is lenient—the agent becomes an automated error amplifier. The resulting output can be fluent, neatly formatted, even schema-valid, yet still wrong in a way that only reveals itself once it has touched scheduling queues, nursing workflows, or medication orders.
This sensitivity to upstream fidelity is why context engineering is not a peripheral optimization but the gating factor for autonomous triage, care-gap closure, protocol digitization, and every other agentic use case to come. Retrieval contracts, freshness SLAs, schema-aware decoders, provenance tags, and calibrated uncertainty heads are the software equivalents of sterile technique; without them, scaling the “intelligence” layer merely accelerates the rate at which bad context turns into bad decisions.
Humans still have a lot to teach machines
While AI can be brilliant for some use cases, in healthcare so far, large-language models still seem like brilliant interns: tireless, fluent, occasionally dazzling—and constitutionally incapable of running the project alone. A clinician opens a chart and, in seconds, spots that an ostensibly “normal” electrolyte panel hides a potassium of 2.8 mmol/L. A protocol digitizer reviewing a 100-page oncology protocol instinctively flags that the run-in period must precede randomization, even though the document buries the detail in an appendix.
These behaviors look mundane until you watch a vanilla transformer miss every one of them. Current models do not plan hierarchically, do not wield external tools unless you bolt them on, and do not admit confusion; they generate tokens until the temperature hits zero. Until we see another major AI innovation like the transformer models themselves, healthcare needs a viable scaffolding that lets an agentic pipeline inherit the basic safety reflexes clinicians exercise every day.
That is not a defeatist conclusion; it is a roadmap. Give the model pipelines that keep the record complete, current, traceable, schema-tight, and honest about uncertainty, and its raw reasoning becomes both spectacular and safe. Skip those safeguards and even a 100-k-token window will still hallucinate a drug dose out of thin air. When those infrastructures become first-class, “superintelligence” will finally have something solid to stand on.
Be Curious, Not Judgmental 