US Senators Ron Wyden (D-OR) and Cory Booker (D-NJ) are examining how federal agencies and healthcare companies are tackling algorithmic biases, after a recent study found that black patients were less likely to be referred to care programs by AI systems than white patients, despite being sicker. On Tuesday, Wyden (D-OR) and Booker (D-NJ) wrote
US Senators Ron Wyden (D-OR) and Cory Booker (D-NJ) are examining how federal agencies and healthcare companies are tackling algorithmic biases, after a recent study found that black patients were less likely to be referred to care programs by AI systems than white patients, despite being sicker.
On Tuesday, Wyden (D-OR) and Booker (D-NJ) wrote a series of letters directed at the Federal Trade Commission, the Centers for Medicare and Medicaid Services, as well as top five healthcare insurers including UnitedHealth Group and Blue Cross Blue Shield. Each letter lists specific requests for information, asking if the agencies or companies have any policies, investigations, or internal tools to help them audit the impact of biases in data.
Algorithms are increasingly being used to calculate an individual’s risk to certain diseases, before automatically deciding what level of care patients should be given. Such software is often based on historical medical data that may contain implicit racial or gender biases, these are then carried forward in the decision making process to affect marginalized groups.
Wyden and Booker’s efforts are spearheaded by a paper published in Science last month. Researchers from the University of California, Berkeley, the University of Chicago, the Brigham and Women’s Hospital and Massachusetts General Hospital, Boston, studied a commercial algorithm widely used by the US healthcare system.
They found that black patients were less likely to receive additional care than white patients despite scoring higher in risk. Poor performance was due to the algorithm predicting health care costs rather than potential illnesses, the study claimed.
“Because of the structural inequalities in our health care system, blacks at a given level of health end up generating lower costs than whites,” Ziad Obermeyer, first author of the paper and a researcher at UC Berkeley, previously said. “As a result, black patients were much sicker at a given level of the algorithm’s predicted risk.”
Oh dear… AI models used to flag hate speech online are, er, racist against black people
After the researchers altered the algorithm to take into account other variables such as extra costs that could be cut by having access to other forms of preventative care, the percentage of black patients flagged for further medical help increased from 17.7 per cent to 46.5 per cent.
“In using algorithms, organizations often attempt to remove human flaws and biases from the process,” Wyden and Booker said in a statement.
“Unfortunately, both the people who design these complex systems, and the massive sets of data that are used, have many historical and human biases built in. Without very careful consideration, the algorithms they subsequently create can further perpetuate those very biases.”
The FTC, the Centers for Medicare and Medicaid Services, and the five healthcare insurers addressed in the Senators’ letters have until 31 December to respond. ®