Gender Bias in AI Decision-Making Systems

Rose Broccolo

Abstract

The rapid development and use of artificial intelligence is causing ethical issues that our legal system is not equipped to address. Gender and other biases are being “baked in” to algorithms created by men. This research paper reviews the implications of gender bias against women in AI automated decision-making (ADM) systems. It also explores several suggestions for how to mitigate gender bias in these systems.

Keywords

ADM Systems, Gender Bias, Ethical AI, Women in AI, Data Bias

Introduction

Gender bias has existed in many different forms. Today, women and non-binary persons continue to struggle for equality in society. Underlying factors, however, like the boom of artificial intelligence and its bias against women, are making that struggle more difficult.

One of the most critical applications of AI technology is in Automatic Decision-Making (ADM) systems. From recommending content on streaming platforms to assisting in hiring decisions, these systems hold the potential to enhance efficiency, accuracy, and objectivity in decision-making processes. However, as these systems become more pervasive, concerns about their ethical implications have come to the forefront. Gender bias in AI ADM systems has become a pressing issue, reflecting broader societal concerns about gender equality and discrimination. The consequences of these bias can be profound, as it can perpetuate harmful stereotypes, limit opportunities, and reinforce disparities between genders.

This research paper explores the types of gender bias in AI ADM systems and what causes them. I also will discuss the implications these biases have created, and how we can address this issue.

The Problem

There are many kinds of bias in ADM systems, each of which have different causes and different impacts. One of these is “prejudicial bias,” which has a large range, but can refer to bias against gender and sexuality [2]. Similarly, machines learn from the data they store and the previous decisions they are required to make.

One of the causes of bias in ADM systems is data bias. The data they are provided to be trained with are often biased, which causes the system itself to form its own discriminations. This can happen for several reasons, including the underrepresentation of women and overrepresentation of men in the data [10]. This is called exclusion bias. When the population represented in the training data is unbalanced, the biases are baked into the algorithms and continue to progress.

It can also occur because of historically biased data, which manifests itself by reinforcing stereotypes. Our society has a history of sexism and discrimination against women. Because data comes from the past and not the future, it may be tainted with old-fashioned or offensive perspectives on women. Certain health tracking apps, for example, act in ways that promote the assumption that all women are seeking to become pregnant [16]. ChatGPT is trained with data through 2021, so does not apply the most current trends to its responses. When asked by researcher Nicole Gross what an economics professor looks like, it responded with clear male features such as a “salt-and-pepper-beard,” and “tailored suits” [5]. We can’t know if this response would change if the system current information, but it is a clear example of data bias. It may be hard to train a system with a fair dataset, which is why algorithmic audits are becoming more common.

Implications

Recruiting and Hiring

Teams of men creating ADM systems has begun a vicious cycle of not hiring women and creating more systems without the intention of diversity, leading to job inequality. While efforts have been made to combat this issue, AI teams continue to fall behind. Algorithms used for recruiting and hiring are hurting the labor market. Certain professions that have a history of being male or female-dominated are being gendered by these algorithms. For example, doctors and engineers are gendered as male while secretaries and nurses are gendered as female [10]. AI-driven systems can scan job postings for certain keywords or phrases. If these ADM systems are not carefully designed, they may discourage female applicants. Research has shown that gendered language or certain masculine-coded terms can deter women from applying for jobs. Amazon stopped using a hiring algorithm altogether after discovering it was consistently rejecting women’s resumes and favored applicants based on words like “executed” or “captured” that were more commonly found on men’s resumes [12,9]. This shows that even when specific pronouns or gender indicators are removed from a job description, it is still flawed.

Financial services & lending

Fintech companies are using AI to assist with credit and loan decisions and there is debate about the fairness of these systems. ADM systems are assigning lower credit scores and providing fewer loans to women [6,8] The calculation systems have also assigned lower credit limits, enforcing stereotypes that women can’t handle money as well as men [16]. Steve Wozniak tweeted about his disappointment when Apple Card offered him 10 times the credit limit of his wife, who has the exact same assets and accounts [3].

Healthcare

AI-driven clinical decision support systems may provide treatment recommendations that are gender-biased. This could result in different treatment options or dosages being suggested for male and female patients, which may not be based on individual health needs, but rather on historical data patterns. For example, cardiovascular disease was long considered a men’s illness. A medical app may use that historical data to diagnose the symptoms of a heart attack differently in men and women [11]. Because of the exclusion criteria and regulations on pregnant women or women on birth control in research studies, a large amount of data are catered toward the bodies of men. Fitness and health apps have fallen behind as well because of their homogenous teams. Apple Health did not add period tracking until 2015. Apps related to sexual health often focus on satisfaction for men, while providing information about safety, risk, and medical issues for women [16].

It should be noted that AI gender bias in healthcare is a complicated issue, because women and men do have different needs. The solution in this case cannot simply be to remove gender or gender indicators from the algorithms. However, it is important that all parties are represented and included in the data when training these systems.

Self-Worth

Axel Honneth’s theory of recognition is that people seek recognition to develop a sense of self-worth. Feeling “seen” as an individual that contributes to society is important, and gender bias in ADM systems has psychological implications. Whether it is reinforcing stereotypes or exclusion bias, the “misrecognition” of women in ADM systems can take a toll on women’s mental health and self-esteem [16].

“The failure to reflect women’s needs in technology products suggests that women are not users worth designing for and that their needs, perspectives, and values do not warrant the inclusion in the design process of AI systems.” [16]

Solutions

Data Collection

Data collection and source selection are critical steps in mitigating gender bias in AI. Teams must ensure the data they provide the ADM system are fully representative of the population they seek as users. Incorporating a series of metrics to measure the “fairness” of the data should be a standard practice amongst AI teams. Pre-processing the algorithms through these fairness metrics is an important piece of the puzzle in solving gender bias during the AI design process [9].

Algorithm Audits and Data Cleaning

Once the ADM systems are released, it is essential to continue to check and audit the system. Companies like Google and IBM have created systems to test their ADM machine learning models. IBM created “IBM AI Fairness 360,” an open source toolkit to help examine datasets and determine its biases using a set of fairness metrics. FairML is a python library used to audit black-box machine learning models [7]. Other recommendations include hiring internal “red teams” or third party companies to audit the system [9].

Human in the loop (HITL) models are growing in popularity as well as an approach to apply human context where the ADM systems are missing [4,13]

Regulatory Approach

Unfortunately, there are very few official regulations in place to ensure the responsible use of AI. Researchers agree that there is an urgent need to create policies for transparency and accountability in these systems [10]. When it falls on the companies to determine the fairness of their systems, the metrics may all be different. How can one know if a system is fair if the definition of fair is different for everyone? Companies may also overlook the issue completely if they don’t have the resources or the motivation [13].

The Algorithmic Justice League is one of the lead organizations lobbying for responsible AI. It started as a research project for a PhD student at MIT and focused on facial recognition software, but has expanded to all forms of AI [12].

Diversify Teams

One of the root causes of gender bias in these systems is the lack of diversity in STEM. At a Fortune conference, the CEO of Pymetrics, Frida Polli said “Can you imagine if all the toddlers in the world were raised by 20-year-old men? That’s what our A.I. looks like today. It’s being built by a very homogenous group” [1] In a study released in March 2022, it was found that 15% of AI researchers at Facebook are women. At Google, the number is 10% [15]. Only 22% of AI professionals identify as female [8,4]. With teams of men creating and training these ADM systems, their unconscious bias is bound to find its way into the ADM systems. Bias will happen unless the systems are created with intention about diversity [2]. AI is not inherently biased, but a reflection of those that create it. Diversity is important in any workplace, and when creating something that reflects society it is necessary to be representative of the population. Efforts are being made for this initiative: Pymetrics Inc. is a company founded in 2011 that utilizes gamification and AI to assist with diversifying teams. AI4ALL is a nonprofit with a mission devoted to creating a more diverse AI workforce through education and mentorship [9].

“Can you imagine if all the toddlers in the world were raised by 20-year-old men? That’s what our A.I. looks like today. It’s being built by a very homogenous group," - Friday Polli [8]

Conclusion

My research revealed that this gender bias is a systemic issue rooted in the data and algorithms that support these systems. This bias can manifest in various ways, from reinforcing gender stereotypes to disproportionately impacting women. The consequences of gender bias in AI ADM systems are far-reaching, touching everything from employment opportunities and financial decisions to healthcare. This bias not only inhibits the advancement of gender equality but also undermines the credibility of these systems.

The journey toward eliminating gender bias in AI ADM systems is complex. The system is simply feeding into a society that already carries implicit bias and needs to somehow be trained to think differently and more progressively than our current state. This will require a multi-faceted approach. We must prioritize data collection and curation that is diverse, representative, and devoid of historical biases. Algorithms must be carefully designed and continually audited to ensure they do not perpetuate gender biases. Transparency in AI decision-making processes and increased accountability are essential as well. Furthermore, it is crucial to engage a wide range of stakeholders, including technologists, policymakers, and civil society, to collaboratively craft solutions that reflect the values of fairness, inclusivity, and equity [4,8].

AI professionals and researchers understand that the machines we create are not inherently good or bad, but they hold magnifying mirror up to society. The potential benefits great, but they can only be realized when we prioritize ethical considerations and recognize that the advancement of technology should not come at the cost of human dignity. By working together to confront and mitigate gender bias in AI ADM systems, we can ensure that these technologies are harnessed for the betterment of all members of society, regardless of their gender. The task at hand is daunting, but it is necessary to create a fairer and more inclusive future.

References

Danielle Abril. 2019. A.I. might be the reason you didn’t get the job. (December 2019). https://fortune.com/2019/12/11/mpw-nextgen-ai-hr-hiring-retention/
Joyce Chou, Oscar Murillo, and Roger Ibars. 2017. How to recognize exclusion in Ai. (September 2017). https://medium.com/microsoft-design/how-to-recognize-exclusion-in-ai-ec2d6d89f850
Clare Duffy. 2019. Apple co-founder Steve Wozniak says Apple Card discriminated against his wife | CNN business. (November 2019). https://www.cnn.com/2019/11/10/business/goldman-sachs-apple-card-discrimination/index.html
Josh Feast. 2020. 4 ways to address gender bias in AI. (October 2020). https://hbr.org/2019/11/4-ways-to-address-gender-bias-in-ai
Nicole Gross. 2023. What CHATGPT tells us about gender: A cautionary tale about performativity and gender biases in ai. (August 2023). https://doi.org/10.3390/socsci12080435
Aaron Klein, Fred Dews Nicol Turner Lee, Joseph B. Keller Nicol Turner Lee, and Chinasa T. Okolo. 2022. Reducing bias in AI-based financial services. (March 2022). https://www.brookings.edu/articles/reducing-bias-in-ai-based-financial-services/
Lightly. 2021. Bias in machine learning. https://www.lightly.ai/post/bias-in-machine-learning#:~:text=Exclusion%20bias%20results%20from%20exclusion,data%20thought%20to%20be%20unimportan
Anu Madgavkar. 2021. A conversation on Artificial Intelligence and gender bias. (April 2021).https://www.mckinsey.com/featured-insights/asia-pacific/a-conversation-on-artificial-intelligence-and-gender-bias
James Manyika, Jake Silberg, and Brittany Presten. 2019. What do we do about the biases in ai? (October 2019). https://hbr.org/2019/10/what-do-we-do-about-the-biases-in-ai
Ayesha Nadeem, Olivera Marjanovic, and Babak Abedin. 2022. Gender bias in AI-based decision-making systems: A systematic literature review. (2022). https://doi.org/10.3127/ajis.v26i0.3835
Carmen Niethammer. 2023. Ai bias could put women’s lives at risk - a challenge for Regulators. (October 2023). https://www.forbes.com/sites/carmenniethammer/2020/03/02/ai-bias-could-put-womens-lives-at-riska-challenge-for-regulators/?sh=5f410c5534f2
Christopher Seward, Paul Rachman, and Kurt Engfehr. 2020. Coded Bias, 7th Empire Media.
Sunny Shrestha and Sanchari Das. 2022. Exploring gender biases in ML and AI academic research through Systematic Literature Review. (September 2022). https://doi.org/10.3389/frai.2022.976838
Genevieve Smith. Ishita Rustagi, Genevieve Smith, Ishita Rustagi, and Genevieve Smith is the associate director at the Center for Equity. 2021. When good algorithms go sexist: Why and how to advance AI Gender Equity (SSIR). (March 2021). https://ssir.org/articles/entry/when_good_algorithms_go_sexist_why_and_how_to_advance_ai_gender_equity
Shawn Tan. 2022. Why diversity in AI remains a challenge and how to fix it: Computer Weekly. (March 2022). https://www.computerweekly.com/opinion/Why-diversity-in-AI-remains-a-challenge-and-how-to-fix-it
Rosalie Waelen and Michał Wieczorek. 2022. The Struggle for AI’s Recognition: Understanding the Normative Implications of Gender Bias in AI with Honneth’s Theory of Recognition (June 2022). https://link.springer.com/content/pdf/10.1007/s13347-022-00548-w.pdf