Is Your AI Unintentionally Discriminating?

by Pranav Ramesh
June 09, 2021
Is Your AI Unintentionally Discriminating?

What is Machine Learning?

Machine learning or ML is a study of computer algorithms that use data to improve users’ experience. It is part of Artificial intelligence and finds application in every walk of life. Everything from the email filters which sort spam to the facial recognition software on our phones is an outcome of ML. Machine learning algorithms use past data to identify, classify, process, and predict future outcomes based on existing data.

What is the role of Machine Learning in Organizations?

Organizations are using machine learning to improve HR processes. For instance, most organizations are switching to an applicant tracking system or an ATS to automate the repetitive process in hiring, like application review. As each job posting receives over 250 applications, it is almost impossible for a human recruiter to scrutinize every resume and pick the ideal candidates.

An AI-based tool can simplify this process by using past recruitment data. Machine learning algorithms filter through tons of resumes and applications to find candidates that match the job posting, skills, experience, salary, etc., from the large candidate pool. This not just reduces the time spent scouting through resumes but also reduces the hiring time.

What is a Machine Learning Bias?

Machine learning bias or AI bias is a phenomenon in which the algorithm produces systemically prejudiced results. This happens because of the erroneous assumptions which happen in the algorithm. Machine learning is a subset of AI, and the bias in it could result from faulty, poor, or substandard data fed into the system. This would result in inaccurate predictions, faulty or prejudicial data sets. While these biases are unintentional, they could have unpleasant consequences. For example, the system used in machine learning could lead to bad customer service, reduced sales, unfair actions, and potentially dangerous conditions.

Amazon, the tech giant, uses machine learning and artificial intelligence for numerous operations. However, it was not until 2015 that the organization realized the bias in its AI systems against women. The data collected over the previous decade regarding resumes showed that most of the applicants were male. Hence the algorithm began discriminating against female applicants. However, the organization resolved the issue and updated its recruitment data to resolve the bias.

The History of Bias In AI

A study by the University of Virginia and the University of Washington shows how a machine learning bias could lead to faulty, inaccurate, and insensitive results. For example, based on a huge volume of data fed into the system, the AI algorithm matched the function of “cooking” with “woman” as the gender. Unfortunately, this led to biased and faulty results as the system identified the gender of a person as “female” even when the image had a man cooking. That is because the system was programmed to see “cooking” as a function of “woman.” Similar inconsistencies exist all over the internet, which forces organizations to be more careful and thorough with their data inputs.

What Are the Different Types of Machine Learning Bias?

A bias can be introduced into the machine learning system through numerous means. Common scenarios, or types of bias, include the following:

Algorithm Bias:

An algorithm bias is when an error in the algorithm computes the data and interferes with the machine’s ability to make a decision. In other words, this happens when there are repeatable errors in the system, which creates a biased outcome that is unfavorable to other users while privileging one arbitrary group. This can be fixed by increasing the scrutiny of data to reduce the bias consciously. One example of the harmful effects of artificial intelligence bias is the findings of a private hospital in the US.

In 2018, the healthcare service provider tried to understand the patients who would need additional care. The data findings suggested an unreasonable bias towards patients who were white over other ethnicities. As a result, researchers had to invest additional time and input new data to reduce this bias by 80%. Had there been no human intervention, the AI bias would have continued to discriminate severely.

Sample Bias:

Sample bias is a common issue when humans collect data. The belief was that machine learning to data collection would reduce the errors and eliminate sampling bias. However, humans are involved in computing data, which means a greater chance of sampling bias, even in machine learning. A data scientist’s job is to ensure that the sample they are building on aligns with the environment they would be deployed in.

Measurement Bias:

A measurement bias in the AI systems can occur when updating the inputs makes a labeling mistake. This makes the algorithm continue the labeling error through the entire program. For instance, if there was a labeling error in Google Photos, which identified dogs as cats, all the images on your phone with dogs will fall under the category of dogs. This could lead to a factor or group being over or underrepresented in the entire dataset. While the cat and dog representation might not hold massive implications, it would have serious repercussions and life-altering consequences if a labeling error occurs in the healthcare or recruitment industry.

How To Avoid Machine Learning Bias?

Fixing bias in machine learning is not easy to resolve as bias can creep in at any stage of the deep learning process. Data scientists aim to develop a machine-learning system that can achieve social and legal outcomes without bias once introduced into a social context. However, the fact that biases creep in unconsciously during the learning process makes it hard to detect. Once the data is entered, it is impossible to understand the downstream impact it could create.

The issue can only be resolved once the bias is retroactively identified, like in the case of Amazon, which was unfairly penalizing female candidates. The organization had to reprogram its recruiting algorithm to ignore the gender of the applicants to introduce a more inclusive workplace. Fixing the bias in the AI algorithm is not something that we can easily resolve. It is an ongoing process and has to be changed on the go just how every other part of society and life works. Although AI results from machine learning, it is still a product and reflection of human biases. This means that the machine learning algorithms will continue to demonstrate the bias in the organizations, teams, and the data scientists who implemented the model. The European Union High-Level Expert Group for Artificial Intelligence is working on guidelines that would ensure that organizations overcome this bias by building an AI algorithm that is lawful, ethical, and robust. Focusing on these aspects will ensure that the teams and engineers build a trustworthy AI system to overcome the data’s problematic predictions.

  • Research your users in advance and be aware of the potential use cases.
  • Use a diverse team of data scientists and data labelers.
  • Include inputs from diverse sources.
  • Use multi-pass annotation for projects to ensure data accuracy and to avoid bias. Sentiment analysis, content moderation, and intent analysis can help overcome bias.
  • Analyze and update data regularly. Look for anomalies and analyze the predictions for a possible data bias.
  • Use tools from Google, IBM, and Microsoft to test your data for bias as part of the development cycle.


Machine Learning Bias is a key challenge in organizations trying to solve problems through people’s data. Fortunately, organizations can use some of the debiasing approaches like increasing diversity to improve data quality. Organizations and HR leaders often talk about recruiting people of color and women to increase diversity across the organization and serve as equal opportunity employers.

However, the marginalized communities are often underrepresented, and the statistical minorities lead to misrepresenting predictive data. Reverse engineering the AI algorithms can be made possible by consciously increasing the diversity of hires across the organization. IBM is currently working on debiasing suits as part of its AI fairness project, which will help organizations identify the AI bias and mitigate against it.

Are you looking for a job in Information Technology?

See all of our current openings here!

About the Company:

Peterson Technology Partners (PTP) has partnered with some of the biggest Fortune brands to offer excellence of service and best-in-class team building for the last 25 years.

PTP’s diverse and global team of recruiting, consulting, and project development experts specialize in a variety of IT competencies which include:

  • Cybersecurity
  • DevOps
  • Cloud Computing
  • Data Science
  • AI/ML
  • Salesforce Optimization
  • VR/AR

Peterson Technology Partners is an equal opportunities employer. As an industry leader in IT consulting and recruitment, specializing in diversity hiring, we aim to help our clients build equitable workplaces.

26+ Years in IT Placements & Staffing Solutions


1030 W Higgins Rd, Suite 230
Park Ridge, IL 60068


222 West Las Colinas Blvd.,
Suite 1650, Irving, Texas, 75039


Av. de las Américas #1586 Country Club,
Guadalajara, Jalisco, Mexico, 44610


8th floor, 90, Dolorez Alcaraz Caldas Ave.,
Belas Beach, Porto Alegre, Rio Grande do Sul
Brazil, 90110-180


240 Ing. Buttystreet, 5th floor Buenos Aires,
Argentina, B1001AFB


08th Floor, SLN Terminus, Survey No. 133, Beside Botanical Gardens,
Gachibowli, Hyderabad, Telangana, 500032, India


16th Floor, Tower-9A, Cyber City, DLF City Phase II,
Gurgaon, Haryana, 122002, India

Work with us
Please enable JavaScript in your browser to complete this form.
*By submitting this form you agree to receiving marketing & services related communication via email, phone, text messages or WhatsApp. Please read our Privacy Policy and Terms & Conditions for more details.

Subscribe to the PTP Report

Be notified when new articles are published. Receive IT industry insights, recruitment trends, and leadership perspectives directly in your inbox.  

By submitting this form you agree to receiving Marketing & services related communication