Skip to main content

Main menu

  • Home
  • Content
    • Current issue
    • Past issues
    • Early releases
    • Collections
    • Sections
    • Blog
    • Infographics & illustrations
    • Podcasts
    • COVID-19 articles
    • Obituary notices
  • Authors & Reviewers
    • Overview for authors
    • Submission guidelines
    • Submit a manuscript
    • Forms
    • Editorial process
    • Editorial policies
    • Peer review process
    • Publication fees
    • Reprint requests
    • Open access
    • Patient engagement
  • Physicians & Subscribers
    • Benefits for Canadian physicians
    • CPD Credits for CMA Members
    • Subscribe to CMAJ digital
    • Subscribe to CMAJ print
    • Subscription prices
    • Obituary notices
  • Alerts
    • Email alerts
    • RSS
  • JAMC
    • À propos
    • Numéro en cours
    • Archives
    • Sections
    • Abonnement
    • Alertes
    • Trousse média 2023
    • Avis de décès
  • CMAJ JOURNALS
    • CMAJ Open
    • CJS
    • JAMC
    • JPN

User menu

Search

  • Advanced search
CMAJ
  • CMAJ JOURNALS
    • CMAJ Open
    • CJS
    • JAMC
    • JPN
CMAJ

Advanced Search

  • Home
  • Content
    • Current issue
    • Past issues
    • Early releases
    • Collections
    • Sections
    • Blog
    • Infographics & illustrations
    • Podcasts
    • COVID-19 articles
    • Obituary notices
  • Authors & Reviewers
    • Overview for authors
    • Submission guidelines
    • Submit a manuscript
    • Forms
    • Editorial process
    • Editorial policies
    • Peer review process
    • Publication fees
    • Reprint requests
    • Open access
    • Patient engagement
  • Physicians & Subscribers
    • Benefits for Canadian physicians
    • CPD Credits for CMA Members
    • Subscribe to CMAJ digital
    • Subscribe to CMAJ print
    • Subscription prices
    • Obituary notices
  • Alerts
    • Email alerts
    • RSS
  • JAMC
    • À propos
    • Numéro en cours
    • Archives
    • Sections
    • Abonnement
    • Alertes
    • Trousse média 2023
    • Avis de décès
  • Visit CMAJ on Facebook
  • Follow CMAJ on Twitter
  • Follow CMAJ on Instagram
  • Listen to CMAJ podcasts
News

AI in health care: Improving outcomes or threatening equity?

Wendy Glauser
CMAJ January 06, 2020 192 (1) E21-E22; DOI: https://doi.org/10.1503/cmaj.1095838
Wendy Glauser
Toronto, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Tables
  • Responses
  • Metrics
  • PDF
Loading

Scientists warn that the unexamined use of artificial intelligence (AI) in health care could result in worse health outcomes for marginalized people.

Recently, a panel of experts gathered at Women’s College Hospital in Toronto to discuss the risks that AI poses to health equity. Dr. Ruha Benjamin, an associate professor of African American Studies at Princeton University, explained that computer algorithms often produce bias because “human decisions comprise the data and shape the design of these systems.”

Built-in biases “hide, speed, and deepen racial discrimination behind a veneer of technical neutrality” Benjamin wrote in a recent commentary in Science. In her presentation, she cited several examples, from an Amazon-led AI recruiting tool that favoured men, to software used in the American criminal justice system that was reportedly biased against black people.

A paper published in Science revealed racial bias in the formula for one of the most commonly used health risk prediction tools in the United States. Ziad Obermeyer and coauthors found that the tool relied heavily on a patient’s health care costs in the previous year to predict future need. However, black patients access health care less frequently than white patients due to work, caregiving, transportation and other barriers.

The researchers found that if the algorithm was instead based on the number and severity of chronic illnesses, the software would have flagged twice as many black patients for enrollment in targeted interventions, including closer monitoring and additional primary care visits.

Figure

Embracing artificial intelligence may deepen discrimination in health care, experts warn.

Image courtesy of iStock.com/metamorworks

Dr. Jay Shaw, research director of AI ethics and health at the Joint Centre for Bioethics at University of Toronto, told CMAJ that AI technology is already being used in small, isolated ways in Canadian health care. “We’re in this phase of experimentation,” he says, where researchers are piloting AI software at single institutions. For example, a research project at St. Michael’s Hospital in Toronto is testing AI to identify patients who are eligible for blood thinners.

“AI is already here, especially in radiology and even cancer treatment,” says Dr. Andrew Pinto, a family physician and a scientist at the Centre for Urban Health Solutions at St. Michael’s. “The problem is we don’t know if it’s creating bias because we don’t often have data on things like race, gender, identity, education and income,” he explains. “We may inadvertently be replicating biases.”

A program trained on lung scans may seem neutral, but if the training data sets include only images from patients from one sex or racial group, it may miss health conditions in diverse populations. Experts have raised similar concerns about AI programs that diagnose skin cancer, given that decades of clinical research that might be used to train the programs focused mostly on people with light skin.

Over the next year, Pinto will survey health providers and patients, asking providers about the problems they want AI to solve, and asking patients questions like, “How do you feel about the computer creating a risk score for you?” One of Pinto’s concerns with algorithm-based care is that doctors will spend less time listening to patients, trying to understand the complex social determinants that factor into health, and more time looking at screens.

Shaw says that AI can be used to enhance equity if it’s designed in the right way, but he thinks this will happen only when the AI is developed by “an agency that is not driven solely by a profit motive, so a hospital or a university.” Development teams must also “elicit a diversity of views in a way that’s meaningful and not tokenistic” throughout the process of creating a program, he says.

Shaw works with the Women’s College Hospital Institute for Health System Solutions and Virtual Care to help teams behind AI initiatives understand the four pillars of equity, governance, public trust and how jobs will be affected. “The trend of big tech companies taking health system data in order to build algorithms, I think that’s really problematic,” he says.

In a paper published in the AMA Journal of Ethics, computer science researchers at Cambridge University called for robust “algorithmic scrutiny” to ensure equitable outcomes — a task that Benjamin points out is rarely possible with the algorithms used in US health care, due to their proprietary nature. The researchers also suggest that developers test the prediction accuracy for various demographic groups.

The myriad processes and decisions required to produce ethical AI won’t be easy to regulate; instead, they’ll require a high degree of public engagement and scrutiny, says Shaw. “The things that are needed to enable the equitable development and deployment of AI are actually outside the scope of regulation.”

Footnotes

  • Posted on cmajnews.com on December 05, 2019

PreviousNext
Back to top

In this issue

Canadian Medical Association Journal: 192 (1)
CMAJ
Vol. 192, Issue 1
6 Jan 2020
  • Table of Contents
  • Index by author

Article tools

Respond to this article
Print
Download PDF
Article Alerts
To sign up for email alerts or to access your current email alerts, enter your email address below:
Email Article

Thank you for your interest in spreading the word on CMAJ.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
AI in health care: Improving outcomes or threatening equity?
(Your Name) has sent you a message from CMAJ
(Your Name) thought you would like to see the CMAJ web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
AI in health care: Improving outcomes or threatening equity?
Wendy Glauser
CMAJ Jan 2020, 192 (1) E21-E22; DOI: 10.1503/cmaj.1095838

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
‍ Request Permissions
Share
AI in health care: Improving outcomes or threatening equity?
Wendy Glauser
CMAJ Jan 2020, 192 (1) E21-E22; DOI: 10.1503/cmaj.1095838
Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like

Jump to section

  • Article
    • Footnotes
  • Figures & Tables
  • Responses
  • Metrics
  • PDF

Related Articles

  • No related articles found.
  • PubMed
  • Google Scholar

Cited By...

  • No citing articles found.
  • Google Scholar

More in this TOC Section

  • Saying goodbye to CMAJ News
  • National survey highlights worsening primary care access
  • How Canadian hospitals are decreasing carbon emissions
Show more News

Similar Articles

Collections

  • Topics
    • Black health
    • Health technology
    • Vulnerable populations

 

View Latest Classified Ads

Content

  • Current issue
  • Past issues
  • Collections
  • Sections
  • Blog
  • Podcasts
  • Alerts
  • RSS
  • Early releases

Information for

  • Advertisers
  • Authors
  • Reviewers
  • CMA Members
  • CPD credits
  • Media
  • Reprint requests
  • Subscribers

About

  • General Information
  • Journal staff
  • Editorial Board
  • Advisory Panels
  • Governance Council
  • Journal Oversight
  • Careers
  • Contact
  • Copyright and Permissions
CMAJ Group

Copyright 2023, CMA Impact Inc. or its licensors. All rights reserved. ISSN 1488-2329 (e) 0820-3946 (p)

All editorial matter in CMAJ represents the opinions of the authors and not necessarily those of the Canadian Medical Association or its subsidiaries.

To receive any of these resources in an accessible format, please contact us at CMAJ Group, 500-1410 Blair Towers Place, Ottawa ON, K1J 9B9; p: 1-888-855-2555; e: [email protected]

CMA Civility, Accessibility, Privacy

 

Powered by HighWire