Artificial intelligence is reshaping most industries, and healthtech is no exception. The sector faces enormous volumes of data, relentless cost pressures from insurers and regulators, and patients who increasingly expect fast, accurate care. AI can help address all three.
For health-focused companies, generative AI opens real opportunities, including supporting clinical workflows, EHR integrations, claims and coding, telehealth platforms, and digital health products. Here are some thoughts to help frame the issues that should be top of mind.
Generative AI is different from earlier machine learning
Machine learning is not new to healthtech and has been used for years in claims adjudication, image analysis, and clinical decision support. Generative AI, however, represents a major leap forward. Systems can now synthesize patient histories from multiple sources, support prior authorization reviews, draft initial responses to payer inquiries, analyze provider contracts, and identify themes across large bodies of clinical or operational documentation, just to name a few emerging capabilities.
The importance of keeping humans in control
A core principle of responsible AI (at least among those who see it as a tool to support humans rather than replace them) is that generative AI should augment professionals, not substitute for them. In healthcare, this means clinicians retain ultimate responsibility for patient care and compliance officers and other decision-makers in the organization retain their authority over other aspects of regulatory, contractual, and ethical requirements. AI tools should surface recommendations with supporting evidence, allow reviewers to validate and override outputs, and preserve a clear audit trail. Products that present AI conclusions as final decisions without meaningful human review create risks (for both patients and providers) that are difficult to quantify and even harder to justify.
Hallucinations are real and must be managed
By now, it should be widely understood that generative AI tools, despite producing confident, authoritative-sounding output, will sometimes return results that are incorrect or entirely made up. In healthcare, these so-called hallucinations can have profound consequences. A fabricated medication interaction, an invented clinical guideline, or a bogus billing code can cause direct patient harm, distort clinical decisions, or trigger regulatory and contractual exposure.
HealthTech companies developing or deploying generative AI can learn from industries that have been grappling with these issues longer, including legal services, financial services, and insurance.
First, start with lower-risk applications. Tasks like classification, summarization, and triage (where a qualified human reviews the output before any action is taken) carry far less risk than autonomous generation of clinical recommendations, patient-facing advice, or billing content. Companies that begin with human-in-the-loop use cases build institutional experience, testing protocols, and quality metrics before extending AI into higher-stakes workflows. These uses also are unlikely to trigger regulation under the myriad of new and proposed laws designed to prevent or otherwise restrict the use of generative AI in healthcare.
Second, how you prompt and ground the model matters. Well-designed systems require the model to identify supporting citations from source material, explain its reasoning, and consider counterarguments before reaching a conclusion. Independent verification of citations, whether by a second model or deterministic software, can dramatically reduce hallucination rates.
Third, these controls need to be built into the product from the start and not merely bolted on later. Retrieval pipelines, citation validation, confidence scoring, audit logging, and human review workflows should be core features, not afterthoughts. Documenting these controls, along with the testing performed against them, is increasingly expected by hospital customers, payer partners, and regulators. Expect this to become a standard part of vendor diligence, procurement questionnaires, and business associate agreements.
Data security, privacy, and HIPAA considerations
HIPAA compliance has already trained healthtech companies to follow strict protocols for securing sensitive health data. Consumer-grade AI tools that retain submitted content or use it for model training may not be suitable for working with protected health information (PHI). Companies evaluating or building AI capabilities should ensure:
- Business associate agreements appropriately cover any AI processing of PHI, including subcontractor relationships with model providers and cloud infrastructure vendors.
- Data flows are documented and auditable, supporting both HIPAA compliance and the growing patchwork of state consumer health data laws.
- Vendor diligence extends to the underlying model providers and infrastructure, not just the immediate AI vendor.
For companies negotiating SaaS agreements, EHR integration contracts, or API licensing arrangements that involve AI, these issues should be addressed head-on in the contract through specific representations, use restrictions, audit rights, and indemnities tailored to AI-related risks.
Intellectual property implications
Generative AI raises some distinctive intellectual property questions that healthtech companies should think through carefully. Where does the training data come from? Who owns the outputs? What is the infringement risk when AI generates content? Are AI-assisted inventions even patentable? These are all active areas of legal development. Companies building AI-enabled health technology should document the provenance of their training data, address output ownership and licensing rights in customer contracts, and consider how their IP strategy accounts for AI-assisted inventions. In many cases, trade secret protection will matter more than patents, especially for proprietary models, training datasets, and prompt engineering know-how.
Practical takeaways for healthtech leaders
Here are a few points to keep in mind as your organization develops, deploys, or invests in AI-enabled health technology:
- Start with use cases where humans review AI output before any action is taken. Move toward higher-autonomy applications only as validation supports it and applicable regulations permit.
- Treat data segregation, encryption, and, if model training is not in furtherance of the services provided, exclusion from model training as non-negotiable whenever PHI or other sensitive data is involved and confirm these protections extend to subcontractors.
- Invest in transparency and explainability as product features from day one.
- Build validation, monitoring, and audit capabilities into the product architecture from the start.
- Address AI-specific issues directly in your contracts, including representations about training data, output ownership, hallucination risk, and use restrictions.
Companies that internalize these lessons will be better positioned to win customers, satisfy regulators, and manage risk as the technology matures.
Please contact a member of our HealthTech team if you would like to discuss how these considerations apply to your products, contracts, or investments.