The Biggest Impact of AI Will Be on Our Minds, Not on Metrics

We've spent this week absorbing what we heard at the Summit, particularly from the philosophers and technologists—Blaise Agüera y Arcas, Ellie Pavlick, Eric Schwitzgebel, and Benjamin Bratton—who spoke about AI's deeper implications. If you're anything like us, your brain might have broken a little by the end of Saturday's program. But as Adam Cutler put it in the closing session, AI may become the microscope of the human condition—revealing what we think, feel, and value at scales and depths we’ve never accessed. That image captures what the others each showed in their own way: the biggest impact of AI won’t be on metrics, but on how we understand our own minds.

The news this week has been noisy—the shear scale of the data center buildouts, AI being blamed for layoffs, and Fed policy on AI investment versus labor markets and inflation. But we've been sitting with what Blaise, Ellie, Eric, and Benjamin said—and that has clarified something important. The biggest impact of AI will be on our minds, on how we understand intelligence and consciousness and infrastructure, more than on any metrics we're currently tracking.

The clearest demonstration came from Blaise’s “Brainfuck soup” experiment. His team generated massive numbers of random programs in a minimal language, let them interact millions of times, and watched self-replicating programs emerge from pure randomness—no fitness function, no selection pressure, just interaction. Even with mutation turned off, replicators appeared and diversified. As he described it, the moment his computer’s fan turned on marked the instant artificial life began to generate physical heat.

The point was that computation and life are not separate things. Intelligence isn’t built—it organizes itself once conditions allow. That simple fact upends how we think about design, control, and what “creating” intelligence even means.

Blaise raised questions about broader capacities such as theory of mind—how and when systems begin to infer or represent other minds, a topic now central for designers and builders thinking about social and collaborative AI. This matters because we are creating interactions and evaluations around systems whose 'understanding' we cannot empirically verify—we have no empirical way to distinguish "real" from "fake" understanding. This challenges how we think about minds completely.

His work on symbiosis in evolution says that intelligence tends to emerge when simpler entities fuse into more complex systems. AI agents cooperating with humans and each other follow the same evolutionary pattern of symbiosis that life has always shown. Rather than treating AI purely as a tool or competitor, we should design for co-adaptation—creating systems that learn, evolve, and improve alongside their human partners. When we design with co-adaptation in mind, we’re shaping the conditions for shared intelligence, where human intuition and machine pattern‑recognition reinforce each other in real time—for example, through adaptive design tools that anticipate user intent, or interfaces that evolve based on human feedback and context—much like Google’s dynamic design systems, which already hint at this direction of co‑adaptive, context‑aware interfaces.

Ellie's presentation on finding conceptual structure inside neural networks does something equally important for how we think about understanding. Her retrieve‑capital‑city vectors show models extracting and applying concepts systematically. She's making the question of "what does understanding mean" into empirical science rather than philosophy. Watching her explain the grounding problem—how LLMs might access meaning through chains of inference even without bodies or senses—shifted how I think about what's required for comprehension.

What stood out was Ellie's shift toward deeply empirical approaches to meaning and understanding. She's moved beyond theoretical debates about grounding to focus on how models form and test conceptual structures—a shift that shows how careful and evidence‑based we need to be when making claims about intelligence. The important takeaway from all this mechanistic‑interpretability work is that we are deploying much faster than we are understanding. We have to ask ourselves constantly what that means for our democracies and societies—how the speed of capability outpaces comprehension, and what kinds of decisions we are really making when we let systems evolve faster than our ability to interpret them.

Benjamin's presentation reshaped the entire conversation by showing AI as planetary infrastructure, connecting it directly to the real economic debates unfolding right now—data centers being built across the country, keeping the US out of recession while possibly accelerating automation and job loss. Seeing AI through this lens reframes those contradictions as part of a planetary-scale transformation rather than a temporary labor-market adjustment. His Stack concept—six layers from Earth to User forming this accidental megastructure—changes how we see everything from data centers to sovereignty. When he talks about infrastructure becoming intelligent, he means cloud platforms already function as alternative geographies and governance structures. This isn't metaphor. Tech companies make jurisdictional decisions. Code functions as law. The territory being formed through data center buildout is political geography, not just computing resources.

Eric's talk brought everything to immediate practical reality. His central insight that we'll create AI whose consciousness we cannot determine, and different mainstream theories will give different verdicts with no way to resolve disagreement—this redefines the ethical stakes and shifts how we think about consciousness and responsibility. We're not waiting for certainty about consciousness. We're already making decisions about systems whose moral status we cannot know.

One takeaway I drew from Eric’s work is the idea of an emotional alignment policy—AI systems that elicit emotional responses from users which accurately reflect their actual moral status. Millions of people are already using AI companions, forming bonds we can't evaluate morally. His discussion of ideas around uncertainty and doubt resonated with the way Maggie Jackson described them too—treating uncertainty not as weakness but as a design principle. It feels especially relevant right now: visual design that avoids pushing users toward confident emotional reactions, and systems that explicitly acknowledge uncertainty, saying "experts disagree about my nature."

His most challenging point was about safety and rights existing in tension. He put it starkly: what is perfectly aligned and perfectly controlled? A slave. If we ever create conscious AI, demanding that it remain under total human control would be unethical. We won’t know if they’re conscious, and that uncertainty forces a rethink of what safety and moral standing even mean. More than that, it makes us ask what it is about us that demands AI remain subservient—why our instinct is to seek control rather than coexistence.

Several patterns emerge from sitting with these talks all week. Intelligence is stranger than we thought. It self-organizes, contains systematic structure we can investigate, operates at planetary scale, arrives in combinations of superhuman and bizarrely deficient capabilities we struggle to evaluate. The economic and political dimensions run through every technical question. Understanding lags behind deployment.

What I’ve become more aware of are a few simple but durable ideas. First: we build faster than we understand, and that gap will define the next decade. Second: the boundary between technology and governance is dissolving—data centers, models, and infrastructure are now political actors shaping economies and ecologies. Third: we are designing systems whose minds will not mirror ours, and that means we need new habits of humility, interpretation, and ethical restraint.

There are tangible reminders for anyone building AI now. Slow down enough to study what you’ve made. Treat uncertainty as a design condition, not a flaw. Recognize that every technical choice has social and moral weight.

At a regular AI tech conference, the conversation might revolve around scaling laws, hallucinations, engagement metrics, or surveillance capitalism. These four lifted us beyond that, toward a different way of measuring progress entirely: not in tokens per second or parameter counts, but in how broadly and strangely we can now define intelligence and consciousness—how many new questions we can hold open at once. Their work opened up the space again—reminding us how little we know, and how much more there is to explore.

I came away with a deep conviction that we can value being in a dual state: one of "unknowing" but also energized inside all that uncertainty, so we keep thinking, questioning, and noticing what these new forms of intelligence are showing us about our own.