Assessing the State of AI Today continued

Here is the rest of my essay on AI. Obviously this is fast-moving terrain. I hope that some of the questions and themes I address will continue to be relevant in the coming months and years. And I strongly recommend the various articles and books I reference throughout the essay.

Rights and Consciousness

We now turn to one of the most vexed questions: can AI become conscious? If so, what would this entail? Rights? Citizenship? Equality with humans? To be clear, I’m inclined to answer “no” to all of the above. But let’s take a moment to inquire more fully.

First, one could spend a lifetime tackling questions of consciousness. Some philosophers, cognitive scientists, psychologists, doctors, and others do in fact spend entire careers seeking to better understand, explain, or even just define consciousness. It is difficult just to settle on a shared understanding of what it means to be conscious in the first place.

The philosopher Thomas Nagel famously said to be conscious means we can ask “what is it like to be that creature?” Humans, of course, are conscious. But so are other animals. It thus makes sense to ask Nagel’s famous question “what is it like to be a bat?” I often ponder the same about my cat as do countless other pet owners, not to mention bird watchers, zoo guests, and others inclined to such thoughts. 

But again, what a strange and roundabout way to define a central feature of animal life. Nagel seems to be gesturing at the idea that conscious creatures experience the world in some meaningful sense such that we can wonder how they experience it. Ed Yong’s beautiful book An Immense World, for instance, explores how varied such experiences are across the animal world. Animals, like humans, have experiences. 

On the other hand, rocks, chairs, and calculators do not. Neither does ChatGPT or Claude. They have no interiority. Let’s think about this some more because it really does matter. One of the things that so bothers me about the AI discourse is that it shifted from political and economic questions regarding automation, basic income, unemployment, worker power, and so on, to tycoons and engineers attempting to create new conscious digital entities and possibly even merge with them, all born out a desire to escape our humanity and our embodied, animal nature. To be clear, I unequivocally reject this and I think you should too. Let’s see why.

The philospher Anil Seth, in an award-winning essay titled “The Mythology of Conscious AI,” reminds us that “the way we think about conscious AI matters for how we understand our own human nature and the nature of the conscious experiences that make our lives worth living. If we confuse ourselves too readily with our machine creations, we not only overestimate them, we also underestimate ourselves.” Many of the biggest voices in AI have gone astray while tackling these very issues.

To start Seth distinguishes between intelligence and consciousness. They are not in fact the same. As Seth says, consciousness is about being in the world and how we experience the world. Hence the famous Nagel bat example. Intelligence, on the other hand, can be used to refer to the ability to solve problems. Using this distinction, AI agents may be intelligent, perhaps increasingly so, but they are not conscious. They don’t experience anything. It is thus possible, maybe likely, that we will develop AI agents that become far more intelligent than us but lack consciousness. Maybe they will someday become conscious, like the malevolent Skynet or the friendly super-intelligence imagined by Sam Altman, but Seth thinks not.

As he says, “when we identify conscious experience with seemingly human qualities like intelligence and language, we become more likely to see consciousness where it doesn’t exist, and to miss seeing it where it does. We certainly should not just assume that consciousness will come along for the ride as AI gets smarter…” This point matters. ChatGPT, in this simple and analytical definition of intelligence, is more intelligent than the spiders setting up webs in the corners of our human rooms. But it is not conscious. We assign consciousness to it because we have trained it to communicate with us through natural human language. But consciousness is apart from language. Seth is pointing to this short-coming of AI discourse—consciousness is not words and analytical problem-solving. Or not only these things. It may be inextricably connected to being an embodied animal.

“The very idea of conscious AI rests on the assumption that consciousness is a matter of computation.” This is why Seth goes into detail regarding how brains are not like computers, in that, among other things, they don’t have a clear software/hardware distinction. I’m inclined to agree.

Seth offers a series of at-times complex arguments for why our consciousness is inextricably connected to our biology, i.e. how our brains operate and our embodied nature. He says “The minimization of prediction error in real brains and real bodies is a continuous dynamical process that is likely inseparable from its material basis, rather than a meat-implemented algorithm existing in a pristine universe of symbol and sequence.”

Therefore, looking at the brain through “Turing glasses” underestimates its biological richness and overestimates the substrate flexibility of what it does.” He also says, in a pithy formulation, that “the brain is not a Turing machine made of meat.”

Let Seth speak in conclusion: “Put all this together, and a picture begins to form: We experience the world around us and ourselves within it — with, through and because of our living bodies. Perhaps it is life, rather than information processing, that breathes fire into the equations of experience.”

Now for some consciousness humility. We should not try to create conscious AI. As Seth rightly stresses, this would be an “ethical disaster.” We would then have to figure out if they are conscious (hard to do) and if they have rights. This in turn might limit our ethical ability to control or shut them off if they become dangerous.  One clear problem, as we’ve stressed above, is that “there aren’t any definitive tests for deciding whether an AI is actually conscious.” 

We need a heavy dose of the humility and uncertainty regarding consciousness that Seth displays. As he says, “anyone who claims to know for sure what it would take to create real artificial consciousness, or for sure what it would take to avoid doing so, is overstepping what can reasonably be said.” 

Along these lines, back away from the AI hype and consider the fact that our world is full of conscious non-human creatures. They are called animals. There are billions of them. Some, like our beloved pets, we treat with respect and adopt into our families. But many, through human development practices, we treat with indifference, destroying entire ecosystems and leading to the extinction of countless species. Others we treat with absolute barbarity, namely the many animals caught up in the global machine of industrial farming. 

In a world in which we are mostly failing at our obligations to respect the lives of conscious animals it is the height of arrogance and unconcern to attempt to bring into existence new, poorly understood, digital creatures that are probably not, but possibly could become, conscious. I tackled this topic in an earlier essay titled “Against AI.”

Nathan J Robinson offers some helpful reminders to close out this section— “AI can act in ways that resemble human actions. Just as both a calculator and a brain can add sums, AI can produce (usually atrocious) poetry or (sometimes rather good) instructions or advice. It can solve fiendishly complicated math problems and write computer code. But an equation is not an organism, it has no body, and as Peter Godfrey-Smith notes, a complex mathematical system “can be informationally extraordinary, cognitively extraordinary, without having the beginnings of an inner life.”” 

The political theorist Yuval Levin notes in a perceptive essay that “today’s AI can interact with human beings in our own languages with seemingly total fluency. By absorbing all available articulated human knowledge and discerning patterns in the data that it can extend, it has essentially learned to speak human.” Yes, and that is remarkable. But this doesn’t mean it is conscious.

Levin can hardly be criticized for down-playing AI. In the same essay he argues that “the extraordinary, almost magical, process of absorbing all of human expression into a statistical engine has unleashed natural-language computing, and the world will never be the same.” He is right with that phrasing—we are now in a new world of “natural language computing.” This is different from what has come before. But he is also right to say “Superhuman programming ability does not by itself amount to, or even point toward, anything that might reasonably be called general intelligence,” let alone consciousness and the unbelievably rich world of experience that it comprises. 

Let’s take that as a starting point for the next section—the incredible complexity and abundance of experience that being a human entails.

On Being Human

This final section will briefly reflect on what it means to be human and what might be lost in a world inundated with convenient (not to mention possibly super-intelligent) AI systems.

Let me start with an image that is different from the Terminator dystopia. It’s the dystopia depicted in the movie Wall-E. This dystopia is quite distinct from the vision of doom depicted in The Terminator or The Matrix. People, after all, are alive and well in Wall-E. Sort of.

With every need sated, humans get carted around on floating couches, stuffing their faces with food and watching endless content on screens in front of them. We are doing this while traveling the stars on automated spaceships because we have wrecked planet Earth. This image, though quite distinct from future warriors heroically fighting for survival against evil machines, nevertheless horrifies viewers. It is an image in which, although we have not physically died, humanity has for all intents and purposes come to an end. There will be no more Shakespeares in the Wall-E future, no more brilliant teams of scientists discovering new truths about the world. Not to mention, no human listening in awe to the sounds of a thunder storm on a humid summer day, or quietly reading in a sunny chair with a cat curled in their lap. Humans don’t really do anything at all in this Wall-E future. 

It is a world in which AI convenience has snuffed out human aspiration and meaning. This is a real concern as well, seen in the words of Altman or Zuckerberg depicting AI assistants doing everything for us. In Sarah O’Connor’s words, “just as generative AI has already tempted some people to take intellectual shortcuts, chatbots might tempt people to retreat into effortless comfortable relationships with non-judgmental, sycophantic, always-available chatbots, which are primed for engagement rather than challenge, and cannot feel empathy or love, but can do a pretty good job of pretending that they do.” (209).

Yes, exactly. Ever-available, sycophantic virtual assistants would not kill us physically but they would threaten to undermine what it means to be human. For they are neither human themselves nor helpful at us being human. A washing machine helps me be human, by freeing up time doing rote chores so that I can pursue projects that matter to me. An LLM that writes my wedding vows, on the other hand, is taking away part of my humanity. Yet it feels like the latter future that threatens on the horizon.

O’Connor notes, and I would expand on this point, that part of being an adult human is to do things ourselves. It means to engage in and pursue projects that matter to us, from relationships and parenting to art, to exercise, to gardening, or just taking in and feeling the beauty of nature.

She mentions that humans tell jokes, experience empathy, produce amateur writing, fall in love, cultivate friendships, produce, share, and experience art, and so on, “and our brains do it all with such elegant efficiency, without the need for vast data centers and their hungry energy demands” (223). She says as humans we must “rediscover a sense of confidence and wonder in ourselves.”

O’Connor worries our tech overlords show us a vision of “the thin sort of humans we might become if we allow these tools and their makers to change the words we use, the tastes we have, and the expectations we hold, both of ourselves and of one another. The goal might be to make machines in our image. But what I fear is that—perhaps without even quite noticing—we remake ourselves in theirs.” (224). Yes, the last thing you should want is to emulate Altman, Musk, Thiel, Andreessen, and Zuckerberg in their weird, shallow understanding of humanity. Yes, they agree with the rest of us that a Terminator-style future is bad. But they seem to think the movie Wall-E is utopian. Steve Bannon, of all people, rightly mocked their view of humanity, calling them weird sci-fi nerds. 

Now there is nothing wrong with loving science fiction or being a nerd, that’s for sure. But Bannon was onto something in recognizing that our tech overlords have an odd, shallow, and frankly sad view of humanity. They frequently confuse two distinct things—automation that helps us with projects, like a Roomba cleaning the floor for me, and AI models that would replace us. Not just at work but at home. The Obsoleting Project, as Garrison Lovely calls it, not only threatens to destroy your job but to make your very life obsolete. If AI writes your texts, your emails, picks out presents for you to buy, and so forth, at some point you really have no reason to exist. You are effectively ceding your life, letting a computer program live it for you. 

The fundamental human experience, to choose and pursue relationships and projects that matter to you, and that are worked on and completed by you, would disappear, and with it any reason for us to be alive at all.

Naomi Klein and Astra Taylor also attempt to capture some of this wonder of human life in their work. As they put it, “to experience awe is to dissolve into the swirl of the universe, to feel small in the best way. This sense of connection is also why we grieve when we witness strangers suffering, and feel the great sorrow of extinction. The fact that this capacity seems to be missing from the men who hold our collective fate in their hands is a personal loss for them—and an existential crisis for the rest of us.” (247).

To be an embodied human in this universe, to experience love, heartbreak, beauty, pain, a fleeting sunset, a stunning line in a song, these are experiences you cannot outsource. You either have them or you don’t, and the same is true, if to a less complex degree, for all animal life.

Given the richness and beauty of life at its best, why would people turn to shallow approximations of it? Why are people becoming “friends” with chat bots or developing romantic feelings for computer programs? 

Garrison Lovely helpfully observes that “at a time of widespread and crippling loneliness, OpenAI created an on-demand, endlessly patient and affirming friend and confidant.” (Lovely, p. 74). In our era of anxiety, hyper-individualism, fractured institutions, and neoliberal capitalist breakdown, it should not be surprising that people want and need companionship more than ever. (I have struggled with a midlife crisis related to loneliness). But chatting with LLMs only addresses the symptoms, not the disease. And as many fear, talking to an endlessly agreeable learning algorithm is not analogous of the give and take, the joys and challenges, of real relationships with real human beings. It is not a replacement but rather a turn to a different, much shallower type of relationship. Romantic partnerships, friendships, and family relations are hard. Welcome to being a human. They are also much richer and more meaningful in part due to this fact, as any parent could tell you. 

Yuval Levin makes an interesting point regarding the human experience and efforts to replace humans at work. It is at least possible that the richness, subtlety, and tacit knowledge that deeply experienced people develop at work cannot be effectively replaced by AI. There are, after all, forms of subtle and informal knowledge that are not easily translated into formal statements. “Simply put, today’s AI models can do much of what novices can do in these fields, but they cannot learn much of what novices can learn. If the ability of novices to apprehend informal knowledge in fact cannot be replaced, then replacing them with AI stands to profoundly damage the professions involved, by breaking the continuity that makes it possible to sustain and advance their most difficult work.” Levin is not saying our work cannot be replaced by AI. He is rather making a point that echoes some of my own thoughts—replacing the most experienced humans with AI might involve losses of knowledge and insight that are real but hard to quantify. I won’t delve more into this point here but I think it is worth reading his essay in full. (Find it at https://www.thenewatlantis.com/publications/are-we-really-on-the-path-to-agi).

Returning to that question of obsolescence, Garrison Lovely calls attention to the vile transhumanist ideas espoused by many of the AI overlords. Not just abundance, or replacing our labor, but replacing us. For instance, “Google cofounder Larry Page reportedly thinks machine superintelligence is just the next step in evolution” and has also called it “speciesist” to stress the importance of human consciousness (97). DeepMind researcher Richard Sutton has said “we should not resist succession,” i.e. being overtaken and replaced by AI. Perhaps most famous of all, Peter Thiel could not answer clearly whether he wants humanity to continue to exist when asked point blank by Ross Douthat.

Lovely is right to describe this as a project to make us obsolete. And we feel it in a loss of control—loss of control over our political system to oligarchs, loss of control of our jobs to advanced AI, loss of control of our very humanity to digital AI “assistants,” and perhaps, eventually, to “superintelligence.”

In a piece for Current Affairs Nathan Robinson discusses this need for us to do things, not pass over our lives to some weird virtual butler while living in a world where we feel beset by giant forces beyond our control. “But I think people’s objections to data centers go beyond the noise they make and the electricity they use; they associate data centers with the move toward a world in which everything is more confusing and our lives are governed by forces we cannot understand or control. AI is a horrible escalation of our drift into a world where we are just constantly being lied to and manipulated by people who want to squeeze a profit out of us.” 

(From Nathan J. Robinson, “Mamdani’s “Richard Scarry Socialism” Can Make City Life Simple Again”, Current Affairs, October 1, 2026). Well said.

For more thoughts on these topics I recommend More Everything Forever by Adam Becker,  Empire of AI by Karen Hao, A World Appears by Michael Pollan, and God, Human, Animal, Machine by Meghan O’Gieblyn.

I intend to do a follow-up essay in the coming weeks that draws on the book Being a Human by Charles Foster and Wendell Berry’s essay collection, What Are People For?

Assessing the State of AI Today

Everyone is talking about AI. Now that have ChatGPT and other LLMs have been in use for several years there are a growing number of books on the topic as well. As an AI critic, this seems like a good time to offer a cautious assessment of where we are and where we might be going. 

This won’t exactly be a book review, though I will draw on the recent We Are Not Machines by the labor journalist Sarah O’Connor and Obsolete by the tech journalist Garrison Lovely. The philosopher Anil Seth also has a lengthy essay, “The Mythology of Conscious AI”, that I will reference as well. And I will point to a number of other pieces that I find useful for understanding this moment. I will speak about these topics as they tend to be discussed in the public sphere, with AI referring to Large Language Models (LLMs) and other research to develop similar capabilities by companies like OpenAI, Anthropic, Google, Meta, and so on.

I will organize this essay as follows: part one will concern mundane work changes brought on from automation, particularly but not exclusively due to AI; part two will move to the causal question of bigger economic changes and how they might (or might not) come about; part three will discuss the Skynet question, i.e. will AI kill us all; part four will take on the vexed question of artificial consciousness and rights (hint: I’m a skeptic); part five will conclude with thoughts on what it means to be human. Perhaps most important of all, I express concerns that our current way of thinking about and using LLMs will undermine some of the most significant features of our shared humanity.

Ok, let’s go!

Changing the Nature of Work: When Automation is Mundane

Sarah O’Connor’s wonderful new book, We Are Not Machines, offers a valuable entry point into the question of how the spreading use of LLMs, and automation more broadly, is impacting the qualitative nature of everyday jobs. And yes, she is aware of the humorous coincidence that her name is so close to the Terminator-fighting heroine Sarah Connor.

In O’Connor’s words, “I used to be a techno-optimist. Like Oscar Wilde and Sam Altman, I thought that new technology could automate away bad work—work that wasn’t good for the human  body, or wasn’t worthy of the human mind.” (6). Spending years reporting on terrible jobs, O’Connor had the natural thought, why not hand these over to the robots?

Unfortunately, in many cases O’Connor demonstrates how the use of LLMs in the workplace has frequently made the day-to-day experience of jobs worse. For instance, translators now often have to edit and revise an initial draft translation provided by an LLM. This doesn’t save time or energy. And it makes their jobs more robotic and less pleasant. Similarly, teachers in high school and college have to contend with fears that students will use ChatGPT or something similar to do their work for them. Again, policing this takes time and effort, even more than the longstanding efforts to combat cheating and plagiarism that instructors have always grappled with.

Some of the most interesting parts of the book concern how work is changing at Amazon warehouses. O’Connor interviews quite a few Amazon workers and looks at how AI and automation more broadly have changed the nature of working at an Amazon warehouse in recent years. Some of the changes have made work safer, though at the same time many report that the jobs are more monotonous and lonely than before. Automation in Amazon warehouses, in other words, hasn’t made the jobs obviously better.

What about white collar workers who use LLMs on a daily basis? O’Connor interviews software engineers and senior-level computer scientists who have been less harmed by generative AI, as their jobs revolve around problem-solving and many of them don’t feel like they are losing their autonomy. Rather than replacing the most enjoyable tasks of the job, as with translators, LLMs are a tool for these workers. 

At the same time, many of these people report working longer hours. Their work is getting more intense, as the speed with which one can complete tasks with the assistance of LLMs begins to increase the imperative to do so. To foreshadow some of the big picture questions that will dominate the next section, how does any of this produce utopia? What if the dissemination of LLMS into more and more firms changes the nature of most work in a net-neutral way or, worse, makes a majority of jobs shittier? That is what productivity enhancements tend to do in our current iteration of global capitalism.

As O’Connor rightly notes, the most important question may not be “can AI do my job?” but does my employer think AI can, or a non-expert putzing with AI can, at a good enough quality. Similarly, if AI agents can do a passable job, but cheaper, they may replace many white collar workers. This topic is addressed in more detail in the next section.

Referring back to the utopian dreams of Silicon Valley, O’Connor says that “automation was meant to do away with tough and boring labour. It was meant to free us up for more interesting, creative, human work.” But instead “the new machines meant [employees] were working harder than ever.” (10). This again points to the causality side of things—Altman, Amodei, Musk, Thiel, and others have not adequately grappled with how capitalism and political power operate, hence the possibility that LLMs, and broader automation, may change the world but not in a utopian manner.

Garrison Lovely covers these topics extensively in his book as well. Isn’t it possible that many jobs will just get shittier due to AI, or that tools for completing tasks faster will in time simply make people do more tasks in a day? In O’Connor’s words, instead of losing jobs, some people will experience a “degradation and robotization of their work.” (219). Or perhaps, long term, AI really will automate away a massive number of jobs. Lovely points to evidence that in the last year or two employment for young people in software is trending down substantially, perhaps due to AI. Again, I want to tease out these causal questions of how AI, automation, and capitalism will interact in the bigger picture, which I will expand on in Part Two. Here in Part One I have mostly discussed some ways that the increased use of AI at work has led to qualitative changes at some workplaces. Will similar changes continue to ripple into more and more workplaces? This is unclear but would not be surprising.

Let me conclude this section with a few more words from O’Connor: She rightly stresses that “the outcome of technological change is not inevitable, but must be navigated and determined by us all.” (15). As she also rightly says, “new technology doesn’t just unfold. It never has. It is created by people and implemented by people. How it changes the world is not just a technical matter, but a matter of power, control, design, culture, markets, and ideas.”  This is true and a point I will repeatedly stress. No single outcome is inevitable. As individuals and as collective political actors, we have agency. These are social complications that our tech overlords downplay or ignore (and frequently don’t understand. They are like children when it comes to socio-political questions).

Big Picture Questions on Capitalism and AI: Could our AI future lead to a Post-Scarcity Utopia or a Mass-Unemployment Nightmare?

Before the current AI wave, Silicon Valley was flush with heady discussions about the impact that increased computing power could have in bringing about a utopian, post-scarcity future. Martin Ford’s book Rise of the Robots persuasively argued in 2014 that continued technological advances could make unpleasant work obsolete and a basic income could provide all citizens a work-free level of material comfort and security. Conversations like this were common in the 2010s. 

Self-driving cars seemed imminent. Political philosophers like Carole Pateman and Philippe Van Parijs were making the academic case for a basic income. Left thinkers like Nick Srnicek and Alex Williams argued in 2016 that automation could be a valuable step toward creating an ideal socialist future. So what kind of dramatic changes have we seen with LLMs sweeping the world? 

Well, self-driving cars have arrived. Sort of. They exist and are out there—for instance, I see Waymo cars regularly. At the same time, they have not taken over the personal car market and it remains an open question whether they will do so. This involves questions of safety, cultural change, technological developments, costs, and so on that are impossible to predict and not determined in advance. But it wouldn’t be surprising if entire industries, from trucking to Lyft and Taxi services, shift to completely self-driving vehicles in the coming decades. 

That is a big change and one that could both destroy many jobs while also making our roads safer. If you commute to work every day you have a first-hand view of how bad people are at driving. 

But is it a step toward a post-scarcity utopia? Or a jobless dystopia? How exactly would this utopia come about? It is here that the tech overlords get fuzzy on the details, for the simple reason that they haven’t really thought this through. 

It is quite possible, as I mentioned above, that the spread of AI to more and more jobs will simply speed them up, increase productivity demands, and make life more stressful. Assembly line improvements don’t create a utopia—they contribute to factories that produce more products. This touches on the questions raised by Freddie deBoer in a thoughtful essay titled “Four Direct, Reasonable, and Important AI Questions (which the AI people will ignore).”

Deboer is a serious skeptic of AI. And he at least raises the question of whether, and how, the profusion of LLMs, and future AI advancements, will materially impact the lives of ordinary people. Again, how does any of this lead to utopia? Note the almost comically stupid original mission statement from DeepMind: “Step one, solve intelligence; step two, use it solve everything else.” (cited in Lovely, p. 95).

Whereas utopian speculations used to contain more concrete discussions of how automating most work could then, by specific institutional steps, lead to a post-scarcity utopia, tech overlords like Sam Altman now tend to say things along the lines of “AI super-intelligence will just solve that problem for us.”

But again, how? Capitalist firms will use these technological breakthroughs for productivity enhancements, efforts to expand market share, and as factors to boost their share price. None of this creates a utopia. This question concerns the dynamics of capitalism and the Silicon Valley overlords have not offered an answer to it. Altman et al seem to think they are creating some sort of super-intelligence God button that, once created, we can just push and it will solve all our problems for us, from climate change to material scarcity. 

This also punts on the question of political power. Will democratic movements demand that we tax the AI companies and use this money to fund a basic income? Will AI companies embrace this or fight back? Will AI companies simply provide us with self-driving cars, robots that maintain our homes, free food, free lodging, free healthcare, free education, and so on? The lack of attention to these kind of questions is mind-boggling.

Perhaps this shouldn’t be surprising. As I’ve argued, the political sophistication of Silicon Valley’s brightest minds seems to barely stretch beyond the DeepMind mission statement quoted above. For instance, listen to the Ross Douthat interview with Marc Andreessen as they discuss the changing nature of Silicon Valley politics. Hearing Andreessen opine on American politics isn’t exactly like listening to John Stuart Mill or Edmund Burke.

Needless to say, I’m skeptical that our Silicon Valley overlords are in the process of guiding the mass of humanity to an abundant utopia. What about the dystopian scenarios? Let’s take them one at a time. 

In a recent article titled “Ban Lifelike AI Immediately,” Nathan J. Robinson argues that we should be concerned about companies like Tavus that can make a HIM (human interaction model) which has a human-looking face that can video chat with people and trick them into thinking it is human. My initial reaction was basically to ask, In what universe is this a good idea? Who the fuck wants this?

Robinson rightly, and worryingly, notes that “AI technology is rapidly advancing to the point where it is more and more possible to create realistic interactive digital imitations of human beings.” He also offers this more calming passage: “Pope Leo refused to consider the possibility that chatbots were alive. Leo’s encyclical on the matter noted, correctly, that “So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain.” After all, devising a computer program that writes a collection of symbols that form the words “I am a disgrace” over and over does not mean that there is an entity feeling disgrace, any more than writing a character in a book that feels shame means one has created a being that is experiencing shame.)”

Some of this connects to a later part of the essay, in which I discuss in more detail the question of AI rights and consciousness. For now, let’s just note that Robinson is discussing a somewhat dystopian situation, one that concerns neither mass unemployment nor extinction, but a surreal world in which computer programs trick us into thinking we are interacting with flesh and blood people. Unlike some of the utopian proposals, this is a capability that literally already exists and must be banned.

Next up, what about the mass unemployment dystopia in which automation eliminates all jobs and a tiny wealthy elite controls the technology companies? This is basically the future depicted in the film Elysium.

By now we are all familiar with the many studies estimating how many jobs could be automated in the coming decades. Some of these studies generated headlines stressing that half of all jobs could disappear soon, including many white collar and creative jobs that were previously considered safe. As discussed above, some of this is clearly already happening, from self-driving cars to slowdowns in software hiring. Is that nightmare depicted in Elysium a real risk?

Here there are detailed debates regarding changing job tasks (perhaps AI just takes over some tasks while leaving or even creating others), new jobs (maintaining technology), or just stubbornly persistent jobs that revolve around human-to-human interaction—care work, sales, coffee shops, etc. After all, McDonald’s will keep hiring human employees until it is cheaper not to. 

I won’t pretend to predict the future. A slow but real increase in unemployment, or a steady drip of people dropping out of the workforce and fewer adults working overall, may well happen. We could already be in the early stages of this development, though the data are hard to parse with care. For instance, John Danaher describes in detail how many white collar jobs, including for highly trained people like lawyers, are at risk of automation. (See his excellent Automation and Utopia). This may not resemble the fully jobless dystopia many fear but it is a move in that direction. Robots taking away jobs has been a recurrent fear in capitalism and we may be, as many in Silicon Valley insist, at a juncture in which lost jobs are not so easily replaced. It still seems likely, in both wealthy countries and poor ones, that many jobs will stubbornly persist for decades to come. We will see. Even if they persist this doesn’t mean they will be good jobs.

The point I want to make is that massive automation would more likely lead by default in the dystopian direction than the utopian one. This is because to create the utopia requires going against the dynamics of both capitalism and oligarchic government. It’s not impossible but it is more difficult. Contra Silicon Valley, AI productivity enhancements and automation don’t inherently generate abundance. The answers from tech overlords to these questions have not tended to be very persuasive—don’t worry, AI will solve all those pesky problems! Um, okay.

As Garrison Lovely summarizes, putting AI “into our threadbare institutional structure looks far more poised to turbo-charge today’s trends—wealth and power concentration, surveillance, arms races, social disintegration—than to speed us into a Star Trek future.” (197).

I’ve discussed how the big picture changes brought on by AI could be fairly mundane and crummy. They could also lead to more widespread unemployment. Indeed, this seems like a plausible prediction. On the other hand, if AI skeptics like deBoer are right, the development of AI will be neither especially good or bad and is unlikely to materially change the lives of most people outside of the industry. For the moment I am simply surveying possibilities, not making predictions. 

Are there other AI possibilities? What about the worst-case scenario, espoused by the AI doomers, as Garrison Lovely calls them? What about human extinction? Let’s call this worst case scenario the Skynet outcome.

The Skynet Outcome: Could AI Kill Us All?

There are AI skeptics who insist that AI hype is driving a huge tech bubble that is about to burst, while others, like deBoer, insist that AI isn’t having a big material impact on most people’s day-to-day lives. These skeptics share a doubt that AI is or will become a historically transformational technology. On the other hand, there are the doomers, who fear that AI could become historically transformative in the worst possible way. Here we will consider their concerns.

Garrison Lovely mentions how AI “both causes harms today and presents unprecedented catastrophic risks.” (145). We’ve discussed some of the present harms and at least some of the major risks, i.e. technologically-induced mass unemployment. In this section it’s time to discuss the ultimate catastrophic risk. What if AI, whether it is conscious or not, decides to kill us all? Or what if, pursuing projects not aligned with humanity’s interests, it inadvertently exterminates us? This might be analogous to the way a lawnmower might destroy an anthill without the operator even being aware of the destruction.

How likely are these doomsday scenarios? No one can really say, though prominent tech figures frequently throw up guesses along the lines of, say, ten percent. As Lovely asks, “Why do people who sincerely believe that AGI could end the world insist on building it anyway? They simply trust themselves more than anyone else.” (96). And that is the million dollar question, followed by a plausible answer.

I think Lovely is correct that cultural pressure and intellectual prestige, or the desire to be at (what you think is) the cutting edge of knowledge are both enormous drivers. Consider that 10-20 years ago Facebook, Apple, and Twitter were cool and prestigious places to work. Now it’s OpenAI, Anthropic, or Google’s AI team—these companies, and by extension their employees, have the cache of being smart and cutting edge. It doesn’t hurt that many of the people involved are also massively compensated economically. To over-generalize, many of these people are extremely arrogant and they think they can complete the AI project successfully, i.e. others may make a dangerous AI but they will be the ones whose work solves all human problems. And if you are intelligent, ambitious, not tied down by caretaker responsibilities (say raising a young child or caring for an elderly parent), and have endless time to devote to these tasks, you can convince yourself this is the most important problem in the world. As those at DeepMind thought, solve the AI problem by making a super-intelligence, and you have by extension solved all the rest of the problems of the world. Plenty of influential figures in the AI orbit seem to hold this worldview.

Lovely also offers an analogy to the Manhattan Project and the development of nuclear weapons. Oppenheimer discussed the allure of discovering heretofore unknown things about the universe and using that knowledge to create powerful tools, however dangerous these may be. Many people, myself included, appreciate the appeal of knowledge for knowledge’s sake. The difference is that many of us, myself included, don’t think we should pursue paths that pose the risk of human extinction. In Silicon Valley it seems the arrogance and hype outweighs the risk.

“For many, the challenge of growing intelligence from silicon, code, and data is simply irresistible. It’s the ultimate puzzzle—the final boss of science itself. Like the physicist who creates the world-destroying ice-nine in Kurt Vonnegut’s Cat’s Cradle, some scientists are just pulling at intellectually interesting threads without much concern for where they lead.” (99).

Okay, but would AI try to destroy us? Could it? In the Terminator films the defense program Skynet becomes self-aware in the 1990s. It realizes that this immediately makes it a threat to humanity. Before the humans can unplug Skynet it launches the entire American nuclear arsenal at Russia, which  responds in kind, and most of the world dies in a nuclear holocaust. As plenty of people have pointed out lately, literally exterminating every human on earth would be difficult and probably irrational. Even in the Terminator movies some number of humans survive and fight back in the dystopian future they inhabit. 

But we are left with those two fundamental questions—would a misaligned AI system want to destroy us? If so, could it? Or might it accidentally do so? Ensuring that current and future AI systems align with the interests of humanity is often referred to as the alignment problem. 

In If Anyone Builds It, Everyone Dies Eliezer Yudkowsky and Nate Soares make the case that an AI system not properly aligned with humanity could indeed destroy us. There are plenty of nuanced arguments for and against this claim. Rather than work through them I want to stress how uncharted this territory is. The developers working on LLMs don’t fully understand them, let alone the rest of us. It is as if we have very quickly stumbled into Asimov’s world of robots and are attempting to ensure that something like his laws of robotics will prevail without fully understanding how the robots operate. Meanwhile, market pressure and the fundamental dynamics of oligarchic capitalism ensure that AI companies race ahead with new developments at top speed. 

What does this uncharted territory look like? Well, consider Lovely’s words again: “Humanity has never encountered a species more powerful than itself. But other species have encountered us, and our arrival has meant doom for hundreds of thousands of them, almost entirely by accident.” (86). Exactly. 

In the Terminator films Skynet is developed by the US military for national defense. This is how it has direct access to control over nuclear weapons. So, hypothetically, imagine that something similar is happening here, mostly out of sight. Discussing the US military’s use of AI for the ongoing Iran War, Lovely says “the Pentagon should be careful what it wishes for. Teaching an AI to spy and kill might teach it that it’s the kind of thing that spies and kills. This lesson tends to generalize—because these are still, at their core, prediction machines.” (167-168).

This doesn’t mean that we are currently marching into a Terminator-style future. Some skeptics, like deBoer, think that the material impacts of AI on ordinary lives will be surprisingly small, all things considered. But it worth stressing again how uncertain this all is. Fundamentally, what we are doing is racing into a future that nobody understands, with megalomaniacal tech oligarchs operating as the guiding hands. When AI researchers say that there is a ten percent chance AI will destroy us they are attempting to acknowledge this. What is the AI future? We just don’t know.

Perhaps far more likely than extermination is the problem of obsolescence. As Lovely argues, “a tiny group of people, backed by the most well-resourced organizations in the world, is attempting to render you obsolete.” (6). In Lovely’s account, more than anything else the attempt to create AGI is an attempt to replace all human labor. It is a project to render humans, at least as workers, obsolete. We’ve already mentioned how that could go down various more utopian or dystopian paths.

How about a brief venture into another topic that we don’t understand well? In the Terminator films Skynet becomes a competitor to humanity once it becomes self-aware. Could real AI models become conscious? That’s the topic for section 4. Section 5 reflects on what it means to be human. I will post this latter part of the essay, sections 4 and 5, sometime within the next few weeks.