AI in health care: Improving outcomes or threatening equity?
============================================================

* Wendy Glauser

Scientists warn that the unexamined use of artificial intelligence (AI) in health care could result in worse health outcomes for marginalized people.

Recently, a panel of experts gathered at Women’s College Hospital in Toronto to discuss the risks that AI poses to health equity. Dr. Ruha Benjamin, an associate professor of African American Studies at Princeton University, explained that computer algorithms often produce bias because “human decisions comprise the data and shape the design of these systems.”

Built-in biases “hide, speed, and deepen racial discrimination behind a veneer of technical neutrality” Benjamin wrote in a recent commentary in *Science*. In her presentation, she cited several examples, from an Amazon-led AI recruiting tool that favoured men, to software used in the American criminal justice system that was reportedly biased against black people.

A paper published in *Science* revealed racial bias in the formula for one of the most commonly used health risk prediction tools in the United States. Ziad Obermeyer and coauthors found that the tool relied heavily on a patient’s health care costs in the previous year to predict future need. However, black patients access health care less frequently than white patients due to work, caregiving, transportation and other barriers.

The researchers found that if the algorithm was instead based on the number and severity of chronic illnesses, the software would have flagged twice as many black patients for enrollment in targeted interventions, including closer monitoring and additional primary care visits.

![Figure1](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/192/1/E21/F1.medium.gif)

[Figure1](http://www.cmaj.ca/content/192/1/E21/F1)

Embracing artificial intelligence may deepen discrimination in health care, experts warn.

Image courtesy of [iStock.com/metamorworks](http://iStock.com/metamorworks)

Dr. Jay Shaw, research director of AI ethics and health at the Joint Centre for Bioethics at University of Toronto, told *CMAJ* that AI technology is already being used in small, isolated ways in Canadian health care. “We’re in this phase of experimentation,” he says, where researchers are piloting AI software at single institutions. For example, a research project at St. Michael’s Hospital in Toronto is testing AI to identify patients who are eligible for blood thinners.

“AI is already here, especially in radiology and even cancer treatment,” says Dr. Andrew Pinto, a family physician and a scientist at the Centre for Urban Health Solutions at St. Michael’s. “The problem is we don’t know if it’s creating bias because we don’t often have data on things like race, gender, identity, education and income,” he explains. “We may inadvertently be replicating biases.”

A program trained on lung scans may seem neutral, but if the training data sets include only images from patients from one sex or racial group, it may miss health conditions in diverse populations. Experts have raised similar concerns about AI programs that diagnose skin cancer, given that decades of clinical research that might be used to train the programs focused mostly on people with light skin.

Over the next year, Pinto will survey health providers and patients, asking providers about the problems they want AI to solve, and asking patients questions like, “How do you feel about the computer creating a risk score for you?” One of Pinto’s concerns with algorithm-based care is that doctors will spend less time listening to patients, trying to understand the complex social determinants that factor into health, and more time looking at screens.

Shaw says that AI can be used to enhance equity if it’s designed in the right way, but he thinks this will happen only when the AI is developed by “an agency that is not driven solely by a profit motive, so a hospital or a university.” Development teams must also “elicit a diversity of views in a way that’s meaningful and not tokenistic” throughout the process of creating a program, he says.

Shaw works with the Women’s College Hospital Institute for Health System Solutions and Virtual Care to help teams behind AI initiatives understand the four pillars of equity, governance, public trust and how jobs will be affected. “The trend of big tech companies taking health system data in order to build algorithms, I think that’s really problematic,” he says.

In a paper published in the *AMA Journal of Ethics*, computer science researchers at Cambridge University called for robust “algorithmic scrutiny” to ensure equitable outcomes — a task that Benjamin points out is rarely possible with the algorithms used in US health care, due to their proprietary nature. The researchers also suggest that developers test the prediction accuracy for various demographic groups.

The myriad processes and decisions required to produce ethical AI won’t be easy to regulate; instead, they’ll require a high degree of public engagement and scrutiny, says Shaw. “The things that are needed to enable the equitable development and deployment of AI are actually outside the scope of regulation.”

## Footnotes

*   Posted on [cmajnews.com](http://cmajnews.com) on December 05, 2019