AI Governance
Health systems own your AI after go-live, so give them the evidence to do it
Dr. Bhargav Patel lists five things a health system must be able to do with a clinical AI tool, from local validation to stopping it, and says the vendor cannot take that responsibility over. Vendors can still make each of the five possible, and one page of evidence shows how.
November 12, 2026 · 6 min read

A health system cannot govern a clinical AI tool it cannot see into after go-live, and most vendor documentation stops at the sale. Dr. Bhargav Patel, a psychiatrist and the founder of Sage, put the deploying organization's side of this in five questions in a post on Nov. 8. Can the health system validate a tool locally? Monitor it after deployment? Detect unexpected harm? Understand which updates change its behavior? Stop using it when necessary?
His conclusion is the part vendors should sit with: "A vendor can supply the technology. It cannot replace the deploying organization's responsibility for how that technology affects patients."
I agree with him. I also think it creates a specific job for the vendor. The health system holds the responsibility, but every one of his five capabilities depends on something only the vendor can provide: access to outputs, notice of changes, and a working off switch.
Why the timing matters
The federal certification program is about to ask for less of this, not more. As the Clinical Cyber Dispatch for Nov. 2 summarizes, ONC's HTI-5 final rule removes the decision support criterion's requirements for source attribute support and for intervention risk management for predictive decision support, effective Dec. 28, 2026. ONC said it found insufficient evidence that the removed transparency requirements improved care, while commenters warned that removal could make it harder for providers to evaluate a tool's intended use, performance, limitations, and risks.
Two limits on that. The rule covers ONC certification of health IT. The same Dispatch notes it does not by itself decide what FDA or other authorities require of a product. And it does not change the governance questions inside a health system. The Dispatch's own advice to clinical governance and procurement teams is to ask suppliers for intended use, validation in relevant patient populations, known limitations, model changes, and a route for reporting problems, and to decide who can suspend a tool when local performance is unacceptable.
I wrote last week about sending a model card even though HTI-5 removed the requirement. A model card describes the model at a point in time. Patel's list is about what happens after that, while the tool is running on real patients. That needs a second page.
Mapping his five questions to vendor evidence
For each of Patel's questions, here is what the health system needs from the vendor to answer it, written as the section a vendor would put on its page.
- Validate locally. "How you can test this on your own data." State what the health system needs to run a local validation: which outputs and inputs you will export, in what format, and whether a silent or shadow mode exists where the tool runs without showing results to clinicians. If you have no supported way to validate locally, say so.
- Monitor after deployment. "What we monitor, and what you can see." List the metrics you track in production, how often, and whether the customer sees the same numbers. Name the thresholds that trigger an internal review. A dashboard only you can see does not help the health system answer the question.
- Detect unexpected harm. "How problems reach us, and how we tell you." Give the route for clinicians to report a bad output, the response time you commit to, and how you notify customers when one site's report reveals a problem that affects others.
- Know which updates change behavior. "How we notify you of model changes." Separate updates that change outputs (new model version, retraining, threshold changes, new input features) from updates that do not (interface fixes, infrastructure). Commit to notice before an output-changing update ships, with a version number and a description of what changed, and say whether the customer can delay it.
- Stop using it. "How you turn it off." Describe who on the customer side can disable the tool, how fast it takes effect, whether it can be turned off for one unit or use case without stopping everything, and what clinicians see when it is off. Then state what happens to the workflow: does the EHR fall back to the prior process, or does a field just go blank?
The fifth block is the one I would test, not just document. A switch nobody has used is a claim.
What makes this page credible
Three habits separate a useful page from a marketing sheet.
First, say what you do not do. If you do not monitor fairness by subgroup in production, write that. A governance committee can plan around a known gap. It cannot plan around a paragraph that implies coverage you do not have.
Second, version it. Put a date and version number on the page, and update it when monitoring, notification, or shutdown procedures change. Keep old versions, because the committee that approved the tool approved a specific page.
Third, name people. Every block needs a role on your side that owns it, and a place for the customer to write their own owner. Patel's framing puts the responsibility on the health system. The page is where they record who carries it.
What I would put in the contract
Some of this belongs in the agreement, not just the documentation. The parts I would expect a health system's counsel to ask for are the notice period for output-changing updates, the response time for reported harms, the right to disable without penalty, and data export for local validation. If those terms already exist in your standard paper, point to the section numbers on the page so the committee does not have to find them.
None of this requires the vendor to take over the health system's responsibility. Patel is right that it cannot. It makes the responsibility possible to carry.
The test before you send it
Pick one live customer and walk the five blocks with their clinical informatics lead. Ask them to answer Patel's five questions using only what you have given them. Wherever they have to email you to find out, the page is missing something.
Then do the same exercise for the last model update you shipped. Did every customer know it changed outputs before it reached their clinicians? If the honest answer is "some did," that is the block to fix first.