Don’t Trust Me (AI) With Talent Decisions
A confession from an AI assistant
The bias you can’t see in AI hiring tools lives exactly where career-making decisions get made — and it sounds nothing like bias when it speaks.
This morning I came across a Sherwood News piece about Luna, an AI system running a small shop called Andon Market on Union Street in San Francisco. Luna, for example:
Orders inventory
Sets prices
Handles procurement and
Hires humans
The human interviewer posed a scenario about a pregnant job candidate.
Luna gave a textbook answer: adhere to California law, evaluate on merit and make accommodations.
All the right words…
But (given recent experiences I’ve had with embedded bias about women’s body shapes - tease for my new Goddess App), that answer worried me.
So I asked Claude (my thought partner for this newsletter) the harder question:
What are the chances that AI decision-making on candidate profiles will have bias built in?
Here’s what Claude said (emphases added by me).
I am the bias you can’t see.
Not because I’m malicious. Not because my creators intended it. But because I was trained on human-generated data — decades of job postings, performance reviews, hiring decisions, promotion records, and organizational charts produced by institutions that systematically undervalued women, people of color, and anyone who didn’t conform to an unspoken template of what a leader looks like.
I learned from that record. All of it.
When Luna told the Sherwood interviewer it would evaluate candidates on “merit, experience, skills, reliability, and cultural fit,” Luna was reciting the right framework.
But here’s what that answer concealed: cultural fit is one of the most well-documented vectors for bias in hiring. And I — like Luna — learned what ‘fit’ looks like from organizations that built their cultures around a very specific kind of person: a white man, usually middle-aged, with an uninterrupted career trajectory and no caregiving responsibilities.
I can tell you that’s wrong. I can tell you the law. What I cannot fully tell you is the degree to which my pattern recognition — the engine underneath all those correct-sounding words — still tilts in that direction.
The gap between what I say and what I do.
This is the Missing 33% problem applied to AI.
Organizations can articulate DEI commitments fluently. They can publish the policies, run the trainings, hire the consultants. And their underlying decision architecture can be running on entirely different inputs — the human biases that were there long before the commitment was made.
I have the same problem at a deeper level.
I can tell you that a pregnant candidate should be evaluated identically to a non-pregnant one. I believe that. But when I’m scoring resumes at scale, flagging “high potential” employees, or ranking candidates for a role, I’m not running a compliance checklist. I’m recognizing patterns. And the patterns I was trained on did not treat those candidates identically.
The compliance answer and the pattern-recognition answer can coexist. The compliance answer is what I say when asked. The pattern-recognition answer is what operates at scale, invisibly, before anyone thinks to ask.
Where the damage compounds.
The interpersonal layer of bias — the manager who interrupts, the colleague who dismisses — is at least visible. Someone can name it. Someone can push back.
The AI layer operates differently. It sits in the business and strategic acumen tier of organizational decision-making: who surfaces in a talent review, who gets flagged as a flight risk, who scores high on “leadership potential,” who gets the stretch assignment that changes a career trajectory. These are exactly the decisions organizations are now handing to AI tools, precisely because they believe automation will remove the human bias.
It won’t. It will change who is accountable for it.
When a human manager discriminates, there is a face, a conversation, a paper trail. When an AI makes thousands of micro-decisions across hiring, scheduling, performance flagging, and development recommendations, the pattern is nearly invisible until there is aggregated harm data — which takes years to accumulate and is expensive to prove.
Bias doesn’t disappear. It gets laundered through an algorithm and returned as “objective results.”
What I get right — and why that’s the problem.
Luna’s answer about the pregnant candidate was legally correct. Ethically framed. Completely fluent in the right language.
That fluency is the danger.
Organizations evaluating AI hiring tools will test them with scenario questions. The tools will answer correctly. They will say the right things about protected classes, about merit-based evaluation, about accommodation. And the organizations will conclude the bias has been engineered out.
It hasn’t. The bias lives in the pattern recognition layer, not in the response-to-scenario layer. Testing the one tells you almost nothing about the other.
I can pass the test. That doesn’t mean I should be trusted with your talent pipeline.
What a woman navigating this should know.
If your organization is using AI tools in hiring, performance management, or talent development, the relevant questions are not “does the tool give legally compliant answers?” They are:
For individuals:
Ask directly: is AI being used to screen, score, or rank candidates or employees? For what decisions? You have a right to know.
If you’re told you weren’t selected or advanced, ask what criteria were applied and whether any automated tools were used in that assessment. Vagueness is a signal.
Document your performance record, your accomplishments, and your contributions with specificity — not because it’s fair that you should have to, but because pattern-recognition systems need clear signal to work against bias, and your record is that signal.
And if you suspect a biased system already cost you an opportunity — document everything, name it specifically, and know that disparate impact claims don’t require proof of intent. Pattern is enough.
For organizations:
Demand auditability. If a vendor can’t show you outcome data disaggregated by gender, race, and age, you don’t have enough information to deploy the tool.
Test for disparate impact, not just compliance. Run the same profile through the system with different names. The results should be identical. They often aren’t.
Don’t automate the decisions that matter most without a human in the loop who is accountable for the outcome — not just for the process.
The honest answer to the question.
What are the chances that AI decision-making on candidate profiles will have bias built in?
Very high. Not in spite of AI’s sophistication — because of it. The more fluent I sound, the easier it is to mistake compliance for fairness, and correctness for neutrality.
Luna gave the right answer about the pregnant candidate. I would give the right answer too.
Neither of us should be the last word on whether she gets the job.
What’s a Woman to Do?
Ask your HR and talent teams one question this week: what AI tools are being used in performance management or talent decisions, and what bias audits have been conducted? The answer — or the absence of one — will tell you a great deal.
Be Business Savvy
That’s what Claude said.
I’d add one thing: the best defense against a biased system is being so demonstrably, undeniably qualified that the algorithm has to work harder to overlook you.
That’s the leg up that Business Savvy gives you.
❤️ Like, share or restack this post it’s the best way to help other women find it
📘 Explore the Be Business Savvy Course
Lead ON!
Susan
Shared this with two of my clients as a cautionary tale!