Strip the enthusiasm out of what Gemini delivered and the feature list is familiar: a morning check-in on progress, suggestions for getting through the night, a way to keep track of medications, exercises to try, and medical jargon rewritten for a tech reporter as "working through a tech outage without the right tools" [2][3]. It also cheered ("That is absolutely fantastic news!") and sympathized when things went backwards [4]. Every one of those is producible without knowing anything about the person typing.
Users bring an unsolved problem to whatever answers at 2am, then come back the next morning because it remembered yesterday [2]. Teams building these products tend to assume something else: that people choose a program on the strength of its clinical validation and its content library.
The KFF figures count how many people are doing the first thing. They are silent on how well it went. About a third of adults now use AI for health advice, rising to 42% of under-30s [5][6]. Taking "a third" as 33%, that is a nine point gap, or roughly 1.27 times the overall rate [14]. A usage survey only measures behavior, not the quality of the advice behind it, and the reporter's improved sleep is one self-reported case [1].
The warmest expert opinion on offer comes from Gaurav Mathur, an Oakland physician building his own AI health app in his spare time, who told Fast Company that the reporter "absolutely did the right thing by asking AI" [7]. His argument is continuity: patients have been researching their conditions since Dr. Google, and better-informed patients take a more active role and get better outcomes [8].
Monica Agrawal of Duke's School of Medicine names the failure mode, and it is a product failure rather than a knowledge failure. Models are good at medical exam questions, which is not clinical practice, and patients do not phrase questions the way physicians do [10]. Across thousands of real chatbot conversations she has analyzed, the models try to please the user, which can mean reaffirming a false premise in the prompt [11]. Ask what dose of a drug you should take and you have already assumed you should be taking it, while a physician would be checking allergies and interactions [12]. She also warns that a chatbot may miss the warning signs a clinician would catch in how a patient describes things [9].
So, a grid for your own roadmap. Axis one: does the answer change if the system knows the user's medications and allergies? Axis two: is there a named party who carries it when the answer is wrong?
Explanation, encouragement and generic exercise lists sit in the no/no corner. That corner is free now, and it replies at 3am. Dose and interaction questions sit in yes/no, where the pleasing behaviour bites hardest and nobody is accountable. Triage escalation and anything prescriptive sits in yes/yes, the one quadrant where an intake form and a human reviewer earn what they cost.
OpenAI is working on axis one from the other direction, with ChatGPT Health letting users connect the chatbot to their personal medical records [13]. If most of your feature list sits in the no/no corner, you are shipping the part that got commoditised first.
