Insights

AI Governance

The first patient AI feature a governance committee will approve is the narrow one

SeamlessMD's CEO says the fastest way to get patient AI adopted may be to make it less autonomous: answer only from care-team-approved content and decline everything else. For vendors, the scope of the first feature decides how hard the governance review will be.

November 13, 2026 · 5 min read

The first patient AI feature a governance committee will approve is the narrow one

A patient-facing AI feature that can only repeat what a care team already approved gives a governance committee far less to review than one that generates its own clinical advice. Joshua Liu, a physician and the co-founder and CEO of SeamlessMD, made that argument from the vendor side in a post on Nov. 7. His lesson, in his words: "The fastest way to get Patient AI adopted might be to make it LESS autonomous, not more."

I am not reviewing his product, and I have not seen it. What interests me is the design decision he describes, because it applies to any HealthTech vendor deciding what to ship first.

What he says they built

Liu describes a conversational feature that lets patients ask questions about their care journey and get answers drawn only from instructions and education already approved by their own care team. His examples are ordinary patient questions: when the bandage can come off after surgery, how to manage chemotherapy side effects, whether a level of pain is normal.

The constraint is the design. In his telling, the feature does not pull information from the open internet, and if the care team did not give it the knowledge, "It won't answer." He calls that a real tradeoff and accepts it: "Safety over answering everything the way ChatGPT would."

He also reports results, and I will keep them as his claims. He says 14 healthcare organizations have adopted the product, and that the narrower design brought "faster AI governance approval than we expected, including from some of the largest health systems." I have no independent data on adoption counts or approval times, so read those as one CEO's account of his own company.

Why narrow scope moves faster through review

The mechanism does not depend on whose numbers you trust. Think about what a committee has to evaluate for each version of the feature.

A system that answers any patient question has an open set of possible outputs. The committee has to ask how it handles questions outside its training, how it conflicts with the care team's instructions, how often it is wrong, who is accountable for a wrong answer, and how anyone would know. Most of those questions have no clean answer before deployment.

A system that answers only from approved content and declines everything else closes most of that set. The clinical content was already reviewed by the people who own it. The new risks shrink to a short list: does it retrieve the right approved passage, does it paraphrase without changing the meaning, and does it actually decline when the answer is not in the content? Each of those can be tested before go-live with a set of scripted questions.

That is the sense in which scope is a sales tool. Narrow scope changes the review from "is this AI safe" to "does this AI stay inside the box we drew," and the second question is one a committee can answer.

Choosing the first feature

If I were a vendor picking the first AI feature to put in front of a health system committee, I would score candidates on four questions.

  1. Where does the content come from? Features that work only from material the customer already approved are easier to review than features that generate new clinical guidance.
  2. What happens when it does not know? A feature that declines and routes the patient to the care team has a defined failure. A feature that guesses does not.
  3. Can it act, or only inform? Answering a question is lower risk than sending a message, changing a schedule, or escalating care on its own. Liu's framing is about autonomy for a reason.
  4. Can the customer test it themselves? If the committee can run its own questions against the feature before approving it, including questions designed to make it go outside scope, you have handed them the evidence instead of asking for trust.

A feature that scores well on all four is not the most impressive thing you could build. It is the one most likely to get a yes, and the yes creates the operating history you need for the next request. Liu says health systems then asked his company about voice agents and continuous coaching. Whether that holds for every vendor, I cannot say. The sequencing logic does.

What to bring to the committee

A narrow feature still needs evidence that it stays narrow. I would bring four things.

The decline behavior, documented. Show what the patient sees when the answer is not in the approved content, and where the patient is sent instead.

A test set. Include in-scope questions with the expected source passage, out-of-scope questions that should be declined, and questions that sit near the edge, such as a symptom question phrased like a content question. Report how the feature did on each group.

The content ownership model. Who on the customer side approves new content, who retires outdated instructions, and what happens to answers when a care plan changes.

A monitoring plan. How the customer can review a sample of real conversations, how patients or clinicians flag a bad answer, and how fast a fix ships.

Where the narrow design can fail

Constrained retrieval still has failure modes. A paraphrase can drop a qualifier that mattered. Approved content can be out of date. A patient can describe an urgent symptom while asking an ordinary question, and a system built to stay in scope has to recognize that and send them to a person rather than answer the narrow question. Those are the cases I would put at the top of the test set, because a committee will ask about them.

So before your next governance review, write down the one sentence that describes what your AI feature will not do. If you cannot write it, the committee will spend the meeting writing it for you.