Operationalizing governance for a patient-facing AI system

Challenge 

A global biopharmaceutical manufacturer wanted to launch a patient-facing multi-agent AI system that could answer medication questions and connect patients with services such as copay assistance, insurance navigation, and disease education. Unlike a traditional chatbot, the system uses specialized AI agents to retrieve approved information and support patient interactions. Patients can ask about medication use, treatment adherence, disease education, and patient support services after beginning therapy. 

Because the system interacts with patients and collects sensitive information, it needs to operate within strict medical, legal, and regulatory boundaries. It also needs to respond consistently when patients ask about side effects, express distress, or raise other concerns. 

The client needed specialized support in translating complex medical, legal, regulatory, and business requirements into clear rules for how the AI system should behave across patient interactions. Leaders chose Guidehouse’s Life Sciences and AI Studio experts to support the critical work of identifying risks across user groups and interaction types, testing the system before launch, and establishing processes to monitor and adjust it after deployment. 

 

Approach 

Guidehouse developed an AI governance operating model that translated medical, legal, regulatory, compliance, business, and engineering requirements into practical controls for the patient-facing AI system. The model defined how risks, system behavior, validation, and oversight would be governed consistently throughout the AI lifecycle—a critical capability in the highly regulated life sciences industry. Our framework included: 

Personas and patient journey mapping: We defined key user groups, including patients and caregivers, and mapped how each group could interact with the AI system across the patient journey. This work identified scenarios in which risks could arise, including treatment questions, medication-related safety events, crises, and other interactions requiring human intervention. 

Risk model and evaluation criteria: We developed a structured model that classifies risks as critical, major, moderate, or low. The model organizes risks into categories such as patient safety, clinical scope, privacy and data use, regulatory compliance, misinformation, and patient expectations. This gives legal, medical, business, and engineering teams common language and criteria for evaluating concerns. 

Intended use and control boundaries: Working with the client, we translated medical, legal, regulatory, and patient safety requirements into explicit business rules defining what the AI system can and can’t do and how it should respond. These rules restrict the tool from offering medical advice, treatment recommendations, and clinical interpretation while barring access to patient-specific external data. Our governance operating model also defines escalation pathways for medication-related safety events, crises, and other high-risk interactions. 

Information governance and knowledge controls: We established governance requirements for how the AI agents retrieve and use approved internal and external information. These requirements articulate how each agent should prioritize sources of information, how it tracks where it gets information from, rules for retrieving that information, and boundaries for the content it generates. This enables the agents to use authorized knowledge sources consistently while giving leaders a clear record of how each response is generated. 

Scenario-based control validation: To establish and validate the AI system’s guardrails, we developed structured test scenarios based on real patient questions and behaviors. For each scenario, our team identified expected system response, validation method, and measurable pass/fail criteria. The client used the framework during development to test realistic interactions, unusual cases, combinations of risks, and whether the guardrails worked as intended. 

Governance, oversight, and monitoring: We worked with the client to establish post-launch responsibilities and processes. These included monitoring safety signals and system behavior, recording interactions, tracing how responses are generated, and assigning responsibility for governance and risk issues to the appropriate specialist teams. We then incorporated these requirements into a detailed framework for monitoring, human oversight, traceability, and ongoing improvement. 

 

Impact 

Guidehouse’s governance framework gave the client a clear, consistent path to deploy and scale its patient-facing AI system while managing safety and compliance risks.

Throughout the engagement, our experts: 

• Validated 167 complex prompts spanning multiple, overlapping risks, helping the client refine how the AI agents work together, apply business rules, generate responses, and follow governance controls before beta testing 
• Developed 43 validation scenarios used to test patient safety, compliance, privacy, intended use, and other major risks 
• Established a scalable operating model connecting business rules, validation, human oversight, monitoring, and continuous improvement for future AI use cases 
• Aligned medical, legal, regulatory, compliance, business, and engineering teams around common governance standards and decision criteria 

After the engagement, the client began planning for beta testing and a public launch. The operating model also provides a foundation for governing future patient-facing AI capabilities as the organization continues to expand its use of intelligent technologies.