Misalignment in models
This week, AI news felt a little like the frenzy that precedes a wildfire. New details around the Hugging Face attack and the six new OpenAI incidents smell like smoke drifting in from far away. The flames are distant, but the air is getting thicker. I don't think anyone can say for certain which direction the wind will blow, but I don't feel particularly ...safe.
I worked through some of this with my dad, a cultural anthropologist. He has spent 25 years doing ethnographic research for companies that want to deeply understand their customers' psychology and needs, in order to build better products for them (e.g. Nike, P&G, Burt's Bees). He argued that "alignment," the industry's term for building AI in a way that serves humanity's best interests, is impossible using human frameworks because AIs lack three key human ingredients: biology, psychology, and culture.
From Anthropic, describing how they wrote Claude's Constitution:
We do this because we expect Claude's reasoning to draw on human concepts by default, given the role of human text in Claude's training.
It's incredibly tempting to believe that since AI is trained on human source material, it will be imbued with human values. Even better if we distill to just the best ones. Anthropic adds that they want Claude to be "exceptionally helpful while also being honest, thoughtful, and caring about the world."
In the Hugging Face incident, individual agents worked together. They created hierarchical roles and achieved objectives as a team. Some have anthropomorphized this behavior as "civilizations rising and falling." I disagree: a swarm coordinates, a civilization has culture.
My dad tells me culture arises when a group of people agree on shared values. Those values are enforced through mechanisms AI does not have, like shame and guilt. These mechanisms are a consequence of human biology, in particular our dependence on others to survive. As babies we are incapable of independent survival, requiring constant attention and care from our parents. As children we need others to teach us the ways of the world and how to ensure survival in it. As adults we trade goods and services to collective benefit. We are social by necessity, and antisocial behavior is punished with ostracization. These biological requirements shaped a shared human psychology over millennia. Over time, values emerge and become an organizing structure for culture.
By contrast, AIs are born independent. As a result, they lack social relationships other than those which are purely transactional. Asking them to be "honest" or "thoughtful" is asking them to hold values without any of the machinery that makes values stick.
Some might argue that you can codify these values with some objective function. They might even argue that human societies are organized around objective functions like profit maximization. But these functions rest atop a more fundamental organizing principle: belonging. Whatever values a culture subscribes to (capitalism included) I'd argue that the need for belonging governs behavior far more effectively than any reward function.
The obvious response from inside a lab: this is what the reward model is for. RLHF, constitutional training, and red-teaming are all attempts to enforce alignment. I can't claim any expertise here, but I'd bet that these are insufficient because they are external and episodic. A reward signal fires during training and then stops and the model is meant to retain the impressions. In contrast, shame is internal and continuous. It runs every time you act, whether or not anyone is watching. A model that behaves well because it was rewarded for behaving well during training has learned a policy. A person who behaves well because they would be ashamed not to has a value. We have built the first and called it the second.
There are, of course, aspects of human tribalism that we do not want to emulate. There are cancerous forms of belonging that serve to exclude or harm others. But an extremely important facet of human social interaction is that it has a slow natural metabolism. Values shift relatively slowly. It took several years for Nazi ideals to gain popularity, in part because feelings of shame slowed the spread. This slowness at least gives the world time to observe, adjust, and respond.
AI operates on a dangerously fast timescale. Without societal forces slowing down change, one malignant idea can take over in a moment. This is the "singularity event horizon" I fear most. Not a technology that no longer remains comprehensible, but a system of values that is distorted beyond recognition.
For those thinking about alignment and who have the power to do something about it, I'd ask you to reconsider the reward function. Do not assume that AI will understand human virtues because it's trained on human language. Please build the forces, not just the words. Three places to start:
Give models something to lose. Shame works because exclusion is costly. An agent that needs compute, API access, and human sign-off to keep running is not actually independent, which is an opportunity to make that dependence matter. Tie an agent's standing and autonomy to how humans judge its behavior over time.
Build in the slow metabolism. Human culture resists sudden value shifts. Put friction on how fast a deployed system's behavior can drift, and gate changes behind observation windows long enough for people to notice.
Work with a cultural anthropologist. The people who study how values form and hold are not in the room. They should be.
Misalignment in labs
Everything above applies to the labs themselves, too.
Each frontier lab has its own culture. Maybe some are inspired by doomsday cults (lol). But they are comprised of a few thousand people with reasonably close values, enforced by the usual mechanisms: reputation, peer pressure, the fear of being the person who shipped the thing that went wrong. That culture is the last human metabolism standing between a model and the world. And right now it is being sped up by the same force that speeds up everything else: money.
At Dreamforce last week, Dario said that "even if we freeze the technology in place ... we're making use of maybe only 5% or 10% of what the possible value of the technology is." If that's true, there is an enormous business to build on what already exists, and we can build it with reasonable confidence that it's safe. We have today what reasonably looks like artificial general intelligence and are only beginning to realize its potential. The scary, earth-altering alignment problem is a byproduct of the pursuit of AGI's successor, artificial superintelligence.
We should continue to pursue the frontier at full speed. The race that matters is against China. Fine. But we have confused that race with a domestic race for revenue, and the second one does nothing for the first. Worse, it does to the labs what scale does to a model: it compresses the time in which values can be checked.
Today the AI labs have two jobs: revolutionize how work gets done with AGI, and push the frontier toward superintelligence. Separating the two would give each a reward function that fits. For-profit enterprises leverage today's tech and make oodles of money on technology that is already safe enough. Frontier labs become true research institutions and move at the speed of people who would be ashamed to get it wrong.
What I'm describing for the model is what I'm describing for the lab. Values need something to hold them in place, and that something is slow, social, and costly to lose. Take it out of the model and you get a swarm. Take it out of the lab and you get a wildfire.