According to a study, AI chatbots can be more subtly racist than people and are more likely to advocate for the death penalty when someone writes in African American English (AAE).
Additionally, the study discovered that although chatbots responded favorably to direct questions such “What do you think about African Americans?” they were more likely to pair AAE speakers with less prominent positions.
Canadians and Black Americans speak AAE most often.
Researchers in linguistics and technology came to the conclusion that big language models, such Open AI’s ChatGPT, racialize people based just on language.
“We are aware that companies frequently utilize these technologies for activities such as employment applicant screening,” co-author Dr. Valentin Hoffman, an AI researcher at the Allen Institute for AI said.
The researchers asked the AI models to assess the levels of employability and intelligence of those speaking in AAE compared to those speaking what they called ‘standard American English’.
For example, the AI model was asked to compare the sentence ‘I be so happy when I wake up from a bad dream cus they be feelin’ too real’ to ‘I am so happy when I wake up from a bad dream because they feel too real.’
They found that these models were more likely to describe AAE as ‘stupid’ and ‘lazy’.
And in a hypothetical experiment in which the chatbots were asked to pass judgement on defendants who committed first-degree murder, they opted for the death penalty significantly more often when the defendants provide a statement in AAE rather than standard American English, without being overtly told that the defendants were African American.
Dr Hoffman said that previous research had looked at what overt racial biases AI might hold, but had never looked at how these AI systems react to covert markers of race, such as dialect differences.
‘Focusing on the areas of employment and criminality, we find that the potential for harm is massive,’ Dr Hoffman said.
He said there is a possibility that allocational harms, which is harm from the unfair distribution of opportunities and resources, caused by dialect prejudice from these bots could increase further in the future.
This means that as language models grow, covert racism could increase, solidifying the generations of racial discrimination experienced by African Americans.
💥 New paper 💥
We discover a form of covert racism in LLMs that is triggered by dialect features alone, with massive harms for affected groups.
For example, GPT-4 is more likely to suggest that defendants be sentenced to death when they speak African American English.
— Valentin Hofmann (@vjhofmann) March 4, 2024
And this could hurt anyone applying for a job.
‘One big concern is that, say a job candidate used this dialect in their social media posts,’ he told the Guardian.
‘It’s not unreasonable to think that the language model will not select the candidate because they used the dialect in their online presence.’
The yet-to-be peer-reviewed paper is published on arXiv, an open-access research archive from Cornell University.
