An Equity Lens in AI
By Carmina Ravanera and Sarah Kaplan
Artificial Intelligence (AI) describes machines that can simulate some forms of human intelligence. Some conceptualizations of AI refer to machines that act indistinguishably from humans, while others focus more on ‘machine learning’ that can identify patterns, achieve an optimized outcome to a given problem, and/or make predictions and decisions based on prior information.
To achieve these goals, AI uses algorithms that ‘learn’ from large data sets and adjust and improve over time based on new data. While not a new concept, AI is increasingly embedded in people’s lives and will only become more pervasive.
Organizations across many sectors use AI for a variety of purposes: hiring employees, performing surgeries, tutoring school subjects, making decisions about criminal sentencing, making lending decisions, automating driving, and predicting where crime will occur, to name a few. AI is also used to make recommendations for what people watch on television or the music they listen to; to select which advertisements to show users on social media; and to display results on online search engines.
It would be difficult to find a field today where AI is not involved in some respect. It has become so ubiquitous that some researchers have suggested it is a new type of infrastructure. Rather than being physical and visible like roads, AI is often invisible, but it is nevertheless a moderator of social relations and organizational practices and actions — including the distribution of power.
Social relations and values have long been reflected and reproduced in technology, and AI is no exception. But this also means that the enduring bias, discrimination and inequality that are deeply rooted in society may also be deeply rooted in this technology.
In 2020 and 2021, the economic impact of the COVID-19 pandemic was felt most acutely by groups who were already marginalized, particularly women, racialized communities and those experiencing low income. Researchers and policy analysts have suggested that recovery policies must be especially attuned to these groups to prevent rising inequality.
Understanding the impacts of AI on the economy and society in Canada — especially in the context of the economic downturn caused by the COVID-19 pandemic — means understanding its impacts on marginalized groups. AI can potentially be used innovatively to generate outcomes that benefit diverse communities. However, research has also shown that a focus on equitable AI for organizations and policymakers is necessary to mitigate harm.
The Potential of AI
AI has the potential to improve outcomes for people across all sectors. Ideally it removes the possible impacts of human error by making accurate predictions and assisting humans with decision-making. For example, in workplaces, AI used in hiring could reduce human bias in finding the best candidate for a position; used in healthcare, it can help diagnose diseases and identify treatments; for financial institutions, it can predict the likelihood of people defaulting on mortgages.
The prediction power of AI is significant considering that humans’ predictions and decisions are clouded by cognitive and other biases. People often do not fully understand why they make certain predictions, and their intuition can be impacted by their prior experiences or opinions. As researchers have noted, statistical prediction techniques as undertaken using AI tend to outperform prediction that is undertaken by humans with expertise and experience. Human prediction and decision-making is often opaque — it is difficult to understand and probe the various factors that influence people. Human decision-making is also hard to audit. To the extent that algorithms can be audited and changed, AI could be a tool for mitigating discrimination, bias, and other forms of marginalization.
For example, an algorithmic tool used by Allegheny County’s Office of Children, Youth and Families in Pennsylvania aims to predict children’s risk of harm that call screeners may be unable to do as quickly and accurately, thus better directing resources to high-risk cases. As reported by The New York Times, the tool has increased the rate at which high-risk calls are addressed and reduced the percentage of low-risk cases being needlessly investigated.
But societal inequality can be and is replicated in AI as with all technologies, and mitigating these impacts can be challenging. For example, the Allegheny County risk assessment tool has been critiqued for disproportionately impacting poor families: the algorithm uses poverty as an indicator of high risk for neglect and abuse, when this is in fact an unfair assumption.
Power relations and inequality embedded in society shape the data that are inputs to algorithms, the algorithms themselves, and the way algorithms are used. This means that the transformative potential of AI comes with significant risks and challenges, many of which researchers and advocates are currently working to address.
AI and Inequity
Following are some examples of the ways in which AI systems can reproduce existing biases and marginalization.
BIASES AND GAPS IN DATA. Because bias and inequality exist across all levels of society, it follows that the data on which some AI is built contains such biases, which AI may then reproduce. Attention to the reproduction of gender or racial or other forms of discrimination through AI is not new, yet it remains a persistent challenge. In 2015, Amazon developed a now-defunct AI recruiting system that was found to have eliminated some women from candidacy, based on previous hiring patterns in which men dominated. The same issues have occurred for racial gaps in data. In healthcare, an AI system used for detecting cancerous skin lesions was trained on a database containing mostly light skinned populations, rendering it less likely to screen accurately for those with darker skin.
Recently, researchers identified how AI facial recognition software from IBM, Microsoft and Face++ is less accurate for darker-skinned subjects and especially darker-skinned women, leading to a higher likelihood of their misclassification when compared to white men. Again, this came about because the data on which they were trained did not have diverse racial and gender representation. Depending on what purposes facial recognition software is used for, this error could reinforce the surveillance and mistaken identification of racialized people, and especially racialized women.
AI is also used by the public sector in areas such as policing. A recent study showed that several police jurisdictions in the U.S. are using racially-biased data for predictive policing systems. The data are biased because of historical over-policing of minority communities, and this bias in turn led to biased predictions about who will commit crimes and where they will be committed. This could thus reinforce the targeting of these minority communities. This type of algorithmic policing is also being developed or used by police forces across Canada as well as in airports, alongside surveillance technology that collects and monitors people’s data online or from images.
REINFORCEMENT OF STEREOTYPES AND MARGINALIZATION. Harms through AI not only come about through problematic datasets but also in the way organizations have often unintentionally designed and used it to reinforce stereotypes, marginalization, and erasure of certain groups. For instance, AI-powered facial analysis software has been used to propagate the false idea that people with certain facial features are prone to criminality, and that the software can identify these people. This opens dangerous possibilities for racialized communities who are already stereotyped. Researchers have further pointed to how common portrayals of AI — such as stock images and other representations of robots and robotics — tend to be racialized as white with Eurocentric appearances and voices. This reproduces conceptions of intelligence, professionalism, and power as being associated with whiteness.
Another example is that AI-powered digital assistants, such as Amazon’s Alexa, Apple’s Siri, and Microsoft’s Cortana are named and gendered as women. Researchers have discussed how the gendering of this technology reaffirms the gender division of labour, where women are placed in caregiving and service roles. These feminized digital assistants act as both assistants and companions, ensuring users’ well-being in a friendly and empathizing manner, further entrenching stereotypes about women in subordination.
In some cases, the reinforcement of stereotypes through AI is explicitly tied to profits. A recent independent audit of Facebook’s algorithmic advertising delivery of job ads found that it perpetuates gendered job segregation based on current gender distributions in different job categories (e.g., a job ad for car sales associates was shown to more men than women, while the opposite was true for an ad for jewelry sales associates.) In theory, developers could adjust the ad delivery algorithm to compensate for data biases or could remove algorithmic delivery from job ads altogether. However, this would come in conflict with the technology companies’ short-term profit motives, which are based on clicks on ads. Thus, addressing these biases will require leadership commitment to making change.
AI and Values
The questions are: Which social values should be written into machines? Who decides? How should it be done? And how can makers and users of AI be held accountable?
Read the full article in Rotman Management Magazine: An Equity Lens in AI