AI Bias Must Move to Accountability to Address Inequity

By Carmina Ravanera and Sarah Kaplan

ChatGPT was hugely and instantly popular when it was released late last year and people have readily adopted it in their workplaces, at school and at home. Whether it’s producing computer coding, emails, schedules, travel itineraries or creative writing, its uses seem endless. 

But like other large-language models that use artificial intelligence algorithms to produce text, image, audio and video content, such technology is not separate from our social and political realities. Researchers have already found that ChatGPT and other generative and traditional artificial intelligence (AI) can reinforce inequality by reproducing bias against and stereotypes of marginalized groups. 

Well-founded concerns about the potential of AI to perpetuate inequality have been expressed for years, and grow ever more relevant as it becomes an increasing part of our lives. Researchers and advocates have suggested there should be policies on AI that put fairness and accountability first.  

AI has immense potential. It can improve our productivity and also our predictions and decisions, which in turn can help reduce disparities. We all have unconscious biases that influence our choices and actions. It can be hard to understand how we’ve arrived at them and whether biases have played a role. Because AI is programmed and can be audited and changed, it can theoretically help us be more accurate and fairer.  

For example, researchers have explored how AI can help make processing refugee claims fairer or more accurately diagnose diseases. And a recent study shows that consultants using a later version of Chat (GPT-4) outperformed consultants who did not. Those with skills deficits were particularly likely to benefit, potentially levelling the playing field for folks without access to elite training. 

But AI is built with data that has been generated from databases of existing information, such as images, texts, and historical data, so our biases become built-in. Its effects on marginalized groups are often unrecognized even as they are perpetuated, because the technology appears to be objective and neutral. Our report on AI research outlines what scholars have found about how AI can contribute to inequity and what can be done to mitigate it.  

Data used in AI development plays a key role. ChatGPT has been trained on numerous text databases, including Wikipedia. A slew of recent articles and research has shown that the chatbot can – without intention on the part of programmers – reproduce sexist and racist stereotypes. For example, it associated boys with science and technology and girls with creativity and emotion. It suggested a “good scientist” is a white man, and readily produced racist content. In translations from Bengali and other languages with gender-neutral pronouns, ChatGPT changed these pronouns to gendered ones.  

These instances show how groups who are underrepresented, misrepresented or omitted from data will continue to be marginalized by AI trained on that material. Research shows that using more representative data can substantially reduce bias in outcomes. But in complex situations – particularly in the case of generative AI – programmers may not be able to explain how specific outputs are being reached, so it may be difficult to audit and fix them.   

Products using AI can also be designed and used in ways that further reinforce inequity. Familiar examples are Amazon’s Alexa and Apple’s Siri, which are named and gendered as women. Researchers have discussed how these AI-powered digital assistants appear to be innovative helpers, but at the same time embody gender stereotypes about women in the home. 

Profit motives may also lead companies to use AI to reproduce sexism and racism. Researcher Safiya Noble explored how Google searches for the term “Black girls” led to first-page results that sexually objectified Black girls and women because they were produced by an algorithm whose primary objective is to drive advertising. 

The consequences are grave. ChatGPT’s ability to perpetuate stereotypes may seem trivial, but repetition of biases and marginalization reinforces their existence. Unequal gender roles are created through repetition of gender norms and expectations. The same may be said of racial stereotypes. AI that is not built equitably can interfere with people’s ability to live safely and free of discrimination.  

Read the full commentary in Policy Options.

Read the report An Equity Lens on Artificial Intelligence.