\n
<\/p>\n
As writer Ted Chiang has observed, \u201cMost fears about A.I. are best understood as fears about capitalism.\u201d When we imagine rogue AI systems optimizing the world to death, we\u2019re actually describing what many corporations already do. If an organization\u2019s ethos is misaligned, any technology platform it creates will amplify that misalignment.<\/p>\n
Take a moment to reflect on the organization you hold most dear: Does it have an ethos, or is it drifting without one? Does it fight for human flourishing or against it? Is it creating value or extracting it? Are you shaping its ethos, or is it shaping yours? No matter your formal title, you have influence over this organization. Yet if you cannot answer these questions with confidence, you are not in control. You are a passenger in a vehicle you think you\u2019re driving.<\/p>\n
But if nobody is consciously driving, where is the organization going? If the behavior of superorganisms were random, we would see as many evolving toward flourishing as toward corruption. Instead, the vast majority wind up in the same place, with indistinguishable values.<\/p>\n
I recently had a conversation with the CEO of a multibillion\u2011dollar company who\u2019d come to see me for help with a baffling problem. He had made a bold new bet on the promise of AI to revolutionize his business. He\u2019d personally overseen the development of a new AI product and tested it with customers. They loved it. Satisfied that he had created something amazing, he turned it over to the rest of his team to commercialize and scale.<\/p>\n
Six months later, this new product was going nowhere. The sales team wasn\u2019t selling it. Developers were loath to work on it. Marketing would not promote it. When confronted, the executives would insist that the product was part of their quarterly priorities. Each of them was telling their teams to focus on it. Yet, somehow, line employees and middle managers were consistently avoiding the AI product. It was a pattern throughout the organization.<\/p>\n
When the CEO personally intervened, he could easily get individual customers to adopt the new product. But he could not get his team to do the same, even after investing extensively in training, performance bonuses, and top\u2011down directives. He couldn\u2019t identify any one person or group actively blocking the AI initiative. He even replaced several key executives with new ones explicitly chosen for their enthusiasm for this new direction. And yet, within months, the uneasy status quo prevailed.<\/p>\n
He shook his head. \u201cYou know what\u2019s driving me crazy? I founded this company fifteen years ago. When I first proposed the core idea, everyone thought I was nuts. Investors, potential employees, even my family.<\/p>\n
\u201cNow that we\u2019re successful, when I suggest we need to pivot, many of the same people, including my own leadership team, resist because \u2018we can\u2019t risk what we\u2019ve built.\u2019 \u201d<\/p>\n
He threw up his hands. \u201cThey\u2019re protecting something that exists only because I took the \u2018crazy\u2019 risk they all thought was impossible. Now they think I\u2019m reckless and can\u2019t seem to appreciate how many times they\u2019ve benefited from my pushing through the impossible to get us to new success.\u201d<\/p>\n
He could feel the stress his insistence was putting on his employees, he told me, as if they were being torn between two authorities. But who, exactly, was this other authority? Like so many founders, he had assumed that being in charge meant the organization would reflect his values. How could it be that he had all the control on paper but so little in daily practice? And if he wasn\u2019t in control, who was?<\/p>\n
John Steinbeck saw this predicament clearly in The Grapes of Wrath. Tenant farmers are being evicted from their land during the Dust Bowl, but when they confront the men carrying out the evictions, they can\u2019t get a clear answer as to who is making the decision to devastate them. The men sent to do the dirty work answer to the bank manager, who answers to distant owners, and those owners answer to shareholders. Each of them has a role in the organization, but somehow none of them is in charge.<\/p>\n
This leads to the farmers\u2019 bewildered question: Who is the bank?<\/p>\n
Steinbeck portrays it as a faceless system in which no individual can be held accountable, even though real people are losing their homes and livelihoods. He calls the bank \u201cthe monster. Men made it, but they can\u2019t control it.\u201d My friend the CEO, grappling with resistance he couldn\u2019t locate, experienced the same bewilderment. If the person at the top is not in charge, who is?<\/p>\n
The CEO I was counseling wasn\u2019t fighting his own people on the AI initiative. He was wrestling with a different kind of living being\u2014one that had developed its own character, its own preferences, and its own will. His personal ethos of innovation was at war with the organization\u2019s emergent character, which craved safety, certainty, and predictability. The fact that he had created it mattered very little.<\/p>\n
Life is defined not by flesh and blood but by specific properties. Living things maintain boundaries between themselves and their environment. They metabolize resources into energy. They grow, adapt, and reproduce. They exhibit behaviors that emerge from their parts but can\u2019t be predicted by studying those parts in isolation. Most importantly, they display a will to survive that shapes every action they take.<\/p>\n
Organizations exhibit every one of these properties. They maintain legal and cultural boundaries that define who\u2019s inside and who\u2019s out. They metabolize capital, talent, and raw materials into products and services. They grow through hiring, adapt to market pressures, and spawn subsidiaries or merge with others. A company\u2019s \u201cculture\u201d emerges from thousands of daily interactions but can\u2019t be found in any employee handbook or org chart. And when threatened, organizations exhibit a tenacity that can override the preferences of any individual within them\u2014even their nominal leader.<\/p>\n
Each one is a superorganism unto itself, operating according to its own rules of existence.<\/p>\n
Now we can answer Steinbeck\u2019s question, Who is the bank? An emergent intelligence, every bit as alive as you or me. A superorganism.<\/p>\n
This is who the bank is.<\/p>\n
When we tell horror stories about AI\u2014systems that optimize ruthlessly toward a narrow goal, blind to human costs\u2014we are, in many ways, describing the bank. Men made it, but they can\u2019t control it. Our fears about AI are, as Chiang suggests, often fears that we will pour unprecedented technical power into institutions whose ethos is already misaligned with human flourishing.<\/p>\n
If we want different outcomes, we cannot just ask what our machines are optimizing for. We have to ask the same question of our organizations.<\/p>\n