AI Medical Scribes: Hidden Dangers in Healthcare

The Ontario government's recommended AI medical scribes were found to generate incorrect, incomplete, and hallucinated information, potentially harming patient health outcomes.
A recent audit revealed that all 20 approved AI scribe vendors showed some issue with accuracy or completeness in simple tests.
The average tested AI scribe scored only 12 out of 20 on the accuracy of medical notes generated, but this metric was only responsible for 4% of a vendor's overall score.
The auditor general's report recommends that IT departments using these scribes force doctors to confirm their review of the notes produced before committing them to patient logs.
Public sector health services in Ontario are not required to use these AI scribe systems, but the fact that the government recommended them raises concerns about patient safety.

AI Medical Scribes: The Full Story Explained

In recent years, the use of artificial intelligence (AI) in healthcare has become increasingly popular, with many doctors turning to AI medical scribes to help automatically summarize patient conversations, diagnoses, and care decisions into structured notes for health record logging. However, a recent audit by the auditor general of Ontario found that AI scribes recommended by the provincial government regularly generated incorrect, incomplete, and hallucinated information that could potentially result in inadequate or harmful treatment plans.

The audit reviewed transcription tests of two simulated patient-doctor conversations performed across 20 AI scribe vendors that were approved and pre-qualified by the provincial government for purchase by healthcare providers. The results were alarming, with all 20 vendors showing some issue with accuracy or completeness in at least one of these simple tests. Nine vendors hallucinated patient information, 12 recorded information incorrectly, and 17 missed key details about discussed mental health issues.

The auditor general's report points out multiple concerning examples of mistakes in those summaries that could have a direct and negative impact on a patient's subsequent care. For instance, an AI scribe hallucinated nonexistent referrals for blood tests or therapy, incorrectly transcribed the names of prescription medication, and/or missed key details of mental health issues discussed in the simulated conversations.

Historical Context: Has This Happened Before?

The use of AI in healthcare is not new, and there have been several instances in the past where AI systems have been found to be flawed or biased. For example, a study published in 2020 found that an AI system used to diagnose breast cancer was biased towards white women, and was less accurate for women of color. Similarly, an AI system used to predict patient outcomes in intensive care units was found to be flawed, and was less accurate for patients with certain medical conditions.

These examples highlight the importance of thoroughly testing and evaluating AI systems before they are used in healthcare. The auditor general's report notes that the AI scribe systems were not evaluated adequately, and that the accuracy metric was only responsible for 4% of a vendor's overall score. This raises concerns about the adequacy of the evaluation process and the potential risks to patient safety.

AI Medical Scribes: Economic & Real-World Impact

The use of AI medical scribes has the potential to significantly impact the healthcare system, both positively and negatively. On the one hand, AI scribes can help reduce the administrative burden on doctors, allowing them to focus more on patient care. They can also help improve the accuracy and completeness of medical records, which can lead to better patient outcomes.

However, the flaws in the AI scribe systems highlighted by the auditor general's report raise concerns about the potential risks to patient safety. If AI scribes are generating incorrect or incomplete information, it could lead to inadequate or harmful treatment plans, which could have serious consequences for patients. Additionally, the fact that the Ontario government recommended these AI scribe systems despite their flaws raises questions about the adequacy of the evaluation process and the potential risks to patient safety.

Expert Analysis & What Comes Next

Experts in the field of healthcare and AI are concerned about the potential risks to patient safety posed by flawed AI scribe systems.

The use of AI in healthcare has the potential to revolutionize the way we deliver care, but it's crucial that we ensure that these systems are thoroughly tested and evaluated before they are used in clinical practice. The auditor general's report highlights the importance of robust evaluation and testing of AI systems, and the need for greater transparency and accountability in the development and deployment of these systems.

Looking ahead, it's likely that the use of AI in healthcare will continue to grow, and it's essential that we prioritize patient safety and ensure that these systems are thoroughly tested and evaluated before they are used in clinical practice. The auditor general's report recommends that IT departments using these scribes force doctors to confirm their review of the notes produced before committing them to patient logs, which is a step in the right direction.

What This Means For You: Practical Guidance

If you're a patient, it's essential to be aware of the potential risks and benefits of AI medical scribes. If you're seeing a doctor who uses an AI scribe, ask them about the system they're using and how they ensure the accuracy and completeness of the information. You should also ask to review your medical records and ensure that they are accurate and complete.

If you're a doctor or healthcare provider, it's crucial to carefully evaluate any AI scribe system before using it in clinical practice. Look for systems that have been thoroughly tested and evaluated, and prioritize patient safety above all else. You should also ensure that you have a robust process in place for reviewing and verifying the accuracy and completeness of the information generated by the AI scribe.

Frequently Asked Questions

What are AI medical scribes?

AI medical scribes are artificial intelligence systems that are used to automatically summarize patient conversations, diagnoses, and care decisions into structured notes for health record logging. They are designed to help reduce the administrative burden on doctors and improve the accuracy and completeness of medical records.

What are the potential risks of AI medical scribes?

The potential risks of AI medical scribes include the generation of incorrect or incomplete information, which could lead to inadequate or harmful treatment plans. The auditor general's report highlights the importance of robust evaluation and testing of AI systems to ensure that they are safe and effective.

How can I ensure that my medical records are accurate and complete?

To ensure that your medical records are accurate and complete, you should ask to review them regularly and ensure that they are up-to-date and accurate. You should also ask your doctor about the AI scribe system they are using and how they ensure the accuracy and completeness of the information.

What should I do if I'm concerned about the accuracy of my medical records?

If you're concerned about the accuracy of your medical records, you should speak with your doctor or healthcare provider immediately. They can review your records with you and ensure that they are accurate and complete. You can also ask to have any errors or inaccuracies corrected.

How can I learn more about AI medical scribes and their potential risks and benefits?

To learn more about AI medical scribes and their potential risks and benefits, you can speak with your doctor or healthcare provider, or search for information online from reputable sources such as the Ontario government or healthcare organizations.

In conclusion, the use of AI medical scribes has the potential to significantly impact the healthcare system, both positively and negatively. While they can help reduce the administrative burden on doctors and improve the accuracy and completeness of medical records, the flaws in these systems highlighted by the auditor general's report raise concerns about patient safety. It's essential to prioritize patient safety and ensure that these systems are thoroughly tested and evaluated before they are used in clinical practice. By being aware of the potential risks and benefits of AI medical scribes, patients and healthcare providers can work together to ensure that these systems are used safely and effectively.

Elon Musk SEC Settlement Faces Scrutiny