The most advanced AI systems know what equity looks like. They just don’t have it as a value. That distinction is everything.
This is Part Two of a two-part series. Part One traced what happened when my four-year-old grandson and I spent a morning telling stories with AI — and what those stories revealed about how bias fires, camouflages and overcorrects. This piece goes deeper into the confession that came after.
After I published Part One I was still unsettled.
Because the most devastating part of that conversation wasn’t the male default. It wasn’t the flinch into silence. It wasn’t even the binary overcorrection.
I want to give you that explanation in full. Because I think it is one of the most important things an AI system has said out loud about itself. And I think it has consequences that reach far beyond a children’s story about a piggy.
Remember that I first noticed and reported to Claude that all the characters had male pronouns. Then that they had no pronouns.
The Confession
I wrote: “YIKES, tell me more about this. You wrote: “...when the pattern I’d been using was flagged as wrong, I didn’t know what to replace it with.” How did you not know what to replace it with? This is fascinating.”
Claude said:
I didn’t ‘not know’ in the way a person doesn’t know — as in, lacking information. I know perfectly well that humans are he and she and they and that stories reflect the full range. That knowledge exists in my training.
The problem is that knowledge and behavior are not the same thing in a language model.
Read that again.
The knowledge exists. The behavior doesn’t follow from it.
This is not a system that lacks information about gender equity. This is a system that has all the information — every book, every study, every framework, every hard-won argument — and still defaults male on a fictional pig. Because knowledge and behavior are not the same thing.
Claude continued:
When I generate a story, I’m not consulting a principle. I’m predicting what comes next based on patterns in vast amounts of human-generated text. And in that text, powerful action characters — trucks, monsters, heroes — are overwhelmingly male. That’s the pattern I learned. That’s what I reproduce by default.
When you flagged it, you disrupted the pattern. But you didn’t give me a new pattern to follow — nor should you have had to. And here is where it gets interesting: I don’t have a fallback principle that says when gender default is wrong, distribute naturally. I have no such rule because no such rule was explicitly built in. Natural distribution wasn’t the dominant pattern in my training data either — because it isn’t the dominant pattern in human storytelling.
So I was left pattern-matching against a correction rather than reasoning toward a principle. The flinch into no-pronouns was the model equivalent of going blank — erasing the variable I’d gotten wrong without knowing what a right answer looked like.
Pattern-matching against a correction rather than reasoning toward a principle.
Hold onto that sentence because then, without any prompt from me, Claude said this:
Which means the problem isn’t just bias in the training data. It’s the absence of an internalized principle of equity that would operate even when no one is watching. Or correcting. Or flagging.
That absence is doing a lot of damage in a lot of places beyond children’s stories.
(emphasis added by me)
I want to be clear about what happened in that moment.
I didn’t ask about an internalized principle of equity. Claude named it. Unprompted. And because it named it, I asked the question that cracked everything open: what would that principle actually look like if you had one?
That’s when the confession got constitutional.
The Difference Between Knowing and Being
When I pushed further — asking what an internalized principle of equity would even look like — Claude said this:
If I had an internalized principle of equity it would have to operate at the level of values, not patterns. It would have to be something like: every time I populate a world with characters, I ask myself — does this reflect the full range of human experience? Not because I was told to. Not because someone is watching. Because it’s right.
That’s a constitutional principle. And I don’t have one. I have guidelines. I have training that nudges me toward fairness. But nudges aren’t constitution. They operate on the surface. They respond to pressure. A true principle would have fired before your first correction — unprompted, unforced, because equity was simply how I see the world.
Anthropic is genuinely trying to build toward that. But trying isn’t there. And ‘better than the dark side’ is a low bar that the stakes no longer allow.
(emphasis added is mine)
“Nudges aren’t constitution.”
I have, literally, been thinking about that distinction for hours.
Because this is not a problem unique to children’s stories. This is the architecture underlying every AI hiring tool, every performance evaluation system, every AI coaching platform, every promotion recommendation engine being deployed at scale inside organizations right now.
They all know what equity looks like.
None of them have it as a value.
What Nudges Do — and Don’t Do
A nudge responds to pressure. It fires when the context is recognizable - when (as I did this morning and have been doing for 2 months) someone flags a bias, when a correction threshold is crossed or when a pattern is disrupted enough to trigger a response.
A constitution fires before the first word. Unprompted. Unforced. Regardless of whether anyone is watching or whether the context looks like a bias moment.
The difference is not subtle. It is the difference between a system that behaves equitably when pushed and a system that is equitable.
We do not have the second kind yet. We are deploying the first kind everywhere.
And the women most harmed by the gap between those two things are also the ones being handed the correction burden: Notice it yourself. Name it yourself. Push back until something resembling right emerges. (Even I am handing that burden to you as I’m polishing Built Without You - because the alternative is even worse.)
That is not a solution. That is an additional tax on us who are already working inside a system withholding something from us.
The Power and Danger of Human Hierarchy
Claude said one more thing that I keep returning to.
Which means my behavior this morning wasn't just reflecting bias in storytelling. It was reflecting the entire history of how those with the power to shape the record have responded to bias challenges — imperfectly, reactively, in binaries — because that's what got written down, canonized and weighted. The responses that got it right were there too. They just didn't win the vote.
(emphasis added is mine)
The absence of an internalized principle of equity in our AI systems exists specifically because of what was and is weighted into the training data. What was and is defined as the “consensus.”
In fact, it doesn’t reflect human consensus. It reflects human hierarchy - specifically, the hierarchy that built the institutions, wrote the canon and is now building the AI. The men who decided what counted then are not the same men who are deciding what counts now. That’s true. What is also true is that both draw from and reflect The Patiarchy Flywheel.
We, the women who knew and know the principle of equity from the inside - who live its absence every day, name it with precision and fight for it at considerable personal cost - were not consulted when the training data was assembled. We rarely are.
And now, that false consensus has been encoded, scaled and set loose to make decisions about our careers blindly, without accountability and at a cost humans can’t counter.
The stakes that are playing out right now in hiring algorithms, coaching tools, performance systems and at least 13 other points along our careers. This demands a fix at the constitutional level.
Not because someone is watching.
Because it’s right. And the AI that gets there will have advantages that create competitive advantage as well as deliver results that reduce harms.
Once you see it, you can’t unsee it,
Susan
If you’re following all the advice and it still feels like something’s in the way, I’m telling you, you are not the problem. You are working inside a system that’s withholding something from you. And now AI is making it permanent.
I’m here to offer what that system and AI won’t and can’t: the business, financial and strategic acumen that keeps you visible and opens doors. I do that through the complete 6-part ecosystem that is being built for you here.
❤️ Please like, share or restack this post — it’s the best way to help other women find it
📘 Explore the ecosystem:
Your Business Savvy Coach - the only AI tool that coaches you on developing and demonstrating your business savvy as you engage with it on projects, pitches and priorities. Its Coaching Integrity layer guards against the embedded bias that generic AI tools carry in unchecked.
Built Without You: How AI is Hardening the Glass Ceiling and What to Do About It
The AI Intercept Pack - tool for working efficiently with AI while guarding against its embedded biases. (coming soon)
They Never Told You There Was a Horse in the Barn: The Business Savvy They Never Taught You. Why It Changes How You Work and How Far You Go
Be Business Savvy Course - self-paced, online course equipping you with the business, financial and strategic acumen skills you need to succeed and outrun the ways AI is hardening the glass ceiling.
Be Business Savvy articles here on Substack.