A new study found that computers can figure out what race a person is just from looking at their X-rays. This is something that would be impossible for a human doctor.
The findings raise some troubling questions about the role of AI in medical diagnosis, assessment, and treatment. Could racial bias be unintentionally applied by computer software when studying images like these?
Key Points:
- A new study has found that deep learning models based on artificial intelligence can identify someone’s race just from their X-rays, something that would be impossible for a human doctor looking at the same images.
- The findings raise some troubling questions about the role of AI in medical diagnosis, assessment, and treatment: could racial bias be unintentionally applied by computer software when studying images like these?
- Having trained their AI using hundreds of thousands of existing X-ray images labeled with details of the patient’s race, an international team of health researchers from the US, Canada, and Taiwan tested their system on X-ray images that the computer software hadn’t seen before (and had no additional information about).
- The AI could predict the reported racial identity of the patient on these images with surprising accuracy.
- Right now scientists aren’t sure why the AI system is so good at identifying race from images that don’t contain such information. It’s possible that the system is finding signs of melanin that are as yet unknown to science.
- The research adds to a growing pile of evidence that AI systems can often reflect the biases and prejudices of human beings.
Personal Insights:
- I think we’ll soon discover some amazing new capabilities that we’ve never had. But at the same time, learn how the training data has historical biases or inaccuracies that have downstream implications that we haven’t yet thought of.
- This will be a tricky new world as we celebrate with amazement our new superpowers, but at the same time discover frustrating biases and unpredictable outcomes.
Article Highlights:
- “Our finding that AI can accurately predict self-reported race, even from corrupted, cropped, and noised medical images, often when clinical experts cannot, creates an enormous risk for all model deployments in medical imaging,” write the researchers.
- “We aimed to conduct a comprehensive evaluation of the ability of AI to recognize a patient’s racial identity from medical images,” write the researchers in their published paper.
- Scientists aren’t sure why the AI system is so good at identifying race from images that don’t contain such information, at least not on the surface. Even when limited information is provided, by removing clues on bone density for instance or focussing on a small part of the body, the models still performed surprisingly well at guessing the race reported in the file.
Source Article:
