Everyone is talking about AI. Now that have ChatGPT and other LLMs have been in use for several years there are a growing number of books on the topic as well. As an AI critic, this seems like a good time to offer a cautious assessment of where we are and where we might be going.
This won’t exactly be a book review, though I will draw on the recent We Are Not Machines by the labor journalist Sarah O’Connor and Obsolete by the tech journalist Garrison Lovely. The philosopher Anil Seth also has a lengthy essay, “The Mythology of Conscious AI”, that I will reference as well. And I will point to a number of other pieces that I find useful for understanding this moment. I will speak about these topics as they tend to be discussed in the public sphere, with AI referring to Large Language Models (LLMs) and other research to develop similar capabilities by companies like OpenAI, Anthropic, Google, Meta, and so on.
I will organize this essay as follows: part one will concern mundane work changes brought on from automation, particularly but not exclusively due to AI; part two will move to the causal question of bigger economic changes and how they might (or might not) come about; part three will discuss the Skynet question, i.e. will AI kill us all; part four will take on the vexed question of artificial consciousness and rights (hint: I’m a skeptic); part five will conclude with thoughts on what it means to be human. Perhaps most important of all, I express concerns that our current way of thinking about and using LLMs will undermine some of the most significant features of our shared humanity.
Ok, let’s go!
Changing the Nature of Work: When Automation is Mundane
Sarah O’Connor’s wonderful new book, We Are Not Machines, offers a valuable entry point into the question of how the spreading use of LLMs, and automation more broadly, is impacting the qualitative nature of everyday jobs. And yes, she is aware of the humorous coincidence that her name is so close to the Terminator-fighting heroine Sarah Connor.
In O’Connor’s words, “I used to be a techno-optimist. Like Oscar Wilde and Sam Altman, I thought that new technology could automate away bad work—work that wasn’t good for the human body, or wasn’t worthy of the human mind.” (6). Spending years reporting on terrible jobs, O’Connor had the natural thought, why not hand these over to the robots?
Unfortunately, in many cases O’Connor demonstrates how the use of LLMs in the workplace has frequently made the day-to-day experience of jobs worse. For instance, translators now often have to edit and revise an initial draft translation provided by an LLM. This doesn’t save time or energy. And it makes their jobs more robotic and less pleasant. Similarly, teachers in high school and college have to contend with fears that students will use ChatGPT or something similar to do their work for them. Again, policing this takes time and effort, even more than the longstanding efforts to combat cheating and plagiarism that instructors have always grappled with.
Some of the most interesting parts of the book concern how work is changing at Amazon warehouses. O’Connor interviews quite a few Amazon workers and looks at how AI and automation more broadly have changed the nature of working at an Amazon warehouse in recent years. Some of the changes have made work safer, though at the same time many report that the jobs are more monotonous and lonely than before. Automation in Amazon warehouses, in other words, hasn’t made the jobs obviously better.
What about white collar workers who use LLMs on a daily basis? O’Connor interviews software engineers and senior-level computer scientists who have been less harmed by generative AI, as their jobs revolve around problem-solving and many of them don’t feel like they are losing their autonomy. Rather than replacing the most enjoyable tasks of the job, as with translators, LLMs are a tool for these workers.
At the same time, many of these people report working longer hours. Their work is getting more intense, as the speed with which one can complete tasks with the assistance of LLMs begins to increase the imperative to do so. To foreshadow some of the big picture questions that will dominate the next section, how does any of this produce utopia? What if the dissemination of LLMS into more and more firms changes the nature of most work in a net-neutral way or, worse, makes a majority of jobs shittier? That is what productivity enhancements tend to do in our current iteration of global capitalism.
As O’Connor rightly notes, the most important question may not be “can AI do my job?” but does my employer think AI can, or a non-expert putzing with AI can, at a good enough quality. Similarly, if AI agents can do a passable job, but cheaper, they may replace many white collar workers. This topic is addressed in more detail in the next section.
Referring back to the utopian dreams of Silicon Valley, O’Connor says that “automation was meant to do away with tough and boring labour. It was meant to free us up for more interesting, creative, human work.” But instead “the new machines meant [employees] were working harder than ever.” (10). This again points to the causality side of things—Altman, Amodei, Musk, Thiel, and others have not adequately grappled with how capitalism and political power operate, hence the possibility that LLMs, and broader automation, may change the world but not in a utopian manner.
Garrison Lovely covers these topics extensively in his book as well. Isn’t it possible that many jobs will just get shittier due to AI, or that tools for completing tasks faster will in time simply make people do more tasks in a day? In O’Connor’s words, instead of losing jobs, some people will experience a “degradation and robotization of their work.” (219). Or perhaps, long term, AI really will automate away a massive number of jobs. Lovely points to evidence that in the last year or two employment for young people in software is trending down substantially, perhaps due to AI. Again, I want to tease out these causal questions of how AI, automation, and capitalism will interact in the bigger picture, which I will expand on in Part Two. Here in Part One I have mostly discussed some ways that the increased use of AI at work has led to qualitative changes at some workplaces. Will similar changes continue to ripple into more and more workplaces? This is unclear but would not be surprising.
Let me conclude this section with a few more words from O’Connor: She rightly stresses that “the outcome of technological change is not inevitable, but must be navigated and determined by us all.” (15). As she also rightly says, “new technology doesn’t just unfold. It never has. It is created by people and implemented by people. How it changes the world is not just a technical matter, but a matter of power, control, design, culture, markets, and ideas.” This is true and a point I will repeatedly stress. No single outcome is inevitable. As individuals and as collective political actors, we have agency. These are social complications that our tech overlords downplay or ignore (and frequently don’t understand. They are like children when it comes to socio-political questions).
Big Picture Questions on Capitalism and AI: Could our AI future lead to a Post-Scarcity Utopia or a Mass-Unemployment Nightmare?
Before the current AI wave, Silicon Valley was flush with heady discussions about the impact that increased computing power could have in bringing about a utopian, post-scarcity future. Martin Ford’s book Rise of the Robots persuasively argued in 2014 that continued technological advances could make unpleasant work obsolete and a basic income could provide all citizens a work-free level of material comfort and security. Conversations like this were common in the 2010s.
Self-driving cars seemed imminent. Political philosophers like Carole Pateman and Philippe Van Parijs were making the academic case for a basic income. Left thinkers like Nick Srnicek and Alex Williams argued in 2016 that automation could be a valuable step toward creating an ideal socialist future. So what kind of dramatic changes have we seen with LLMs sweeping the world?
Well, self-driving cars have arrived. Sort of. They exist and are out there—for instance, I see Waymo cars regularly. At the same time, they have not taken over the personal car market and it remains an open question whether they will do so. This involves questions of safety, cultural change, technological developments, costs, and so on that are impossible to predict and not determined in advance. But it wouldn’t be surprising if entire industries, from trucking to Lyft and Taxi services, shift to completely self-driving vehicles in the coming decades.
That is a big change and one that could both destroy many jobs while also making our roads safer. If you commute to work every day you have a first-hand view of how bad people are at driving.
But is it a step toward a post-scarcity utopia? Or a jobless dystopia? How exactly would this utopia come about? It is here that the tech overlords get fuzzy on the details, for the simple reason that they haven’t really thought this through.
It is quite possible, as I mentioned above, that the spread of AI to more and more jobs will simply speed them up, increase productivity demands, and make life more stressful. Assembly line improvements don’t create a utopia—they contribute to factories that produce more products. This touches on the questions raised by Freddie deBoer in a thoughtful essay titled “Four Direct, Reasonable, and Important AI Questions (which the AI people will ignore).”
Deboer is a serious skeptic of AI. And he at least raises the question of whether, and how, the profusion of LLMs, and future AI advancements, will materially impact the lives of ordinary people. Again, how does any of this lead to utopia? Note the almost comically stupid original mission statement from DeepMind: “Step one, solve intelligence; step two, use it solve everything else.” (cited in Lovely, p. 95).
Whereas utopian speculations used to contain more concrete discussions of how automating most work could then, by specific institutional steps, lead to a post-scarcity utopia, tech overlords like Sam Altman now tend to say things along the lines of “AI super-intelligence will just solve that problem for us.”
But again, how? Capitalist firms will use these technological breakthroughs for productivity enhancements, efforts to expand market share, and as factors to boost their share price. None of this creates a utopia. This question concerns the dynamics of capitalism and the Silicon Valley overlords have not offered an answer to it. Altman et al seem to think they are creating some sort of super-intelligence God button that, once created, we can just push and it will solve all our problems for us, from climate change to material scarcity.
This also punts on the question of political power. Will democratic movements demand that we tax the AI companies and use this money to fund a basic income? Will AI companies embrace this or fight back? Will AI companies simply provide us with self-driving cars, robots that maintain our homes, free food, free lodging, free healthcare, free education, and so on? The lack of attention to these kind of questions is mind-boggling.
Perhaps this shouldn’t be surprising. As I’ve argued, the political sophistication of Silicon Valley’s brightest minds seems to barely stretch beyond the DeepMind mission statement quoted above. For instance, listen to the Ross Douthat interview with Marc Andreessen as they discuss the changing nature of Silicon Valley politics. Hearing Andreessen opine on American politics isn’t exactly like listening to John Stuart Mill or Edmund Burke.
Needless to say, I’m skeptical that our Silicon Valley overlords are in the process of guiding the mass of humanity to an abundant utopia. What about the dystopian scenarios? Let’s take them one at a time.
In a recent article titled “Ban Lifelike AI Immediately,” Nathan J. Robinson argues that we should be concerned about companies like Tavus that can make a HIM (human interaction model) which has a human-looking face that can video chat with people and trick them into thinking it is human. My initial reaction was basically to ask, In what universe is this a good idea? Who the fuck wants this?
Robinson rightly, and worryingly, notes that “AI technology is rapidly advancing to the point where it is more and more possible to create realistic interactive digital imitations of human beings.” He also offers this more calming passage: “Pope Leo refused to consider the possibility that chatbots were alive. Leo’s encyclical on the matter noted, correctly, that “So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain.” After all, devising a computer program that writes a collection of symbols that form the words “I am a disgrace” over and over does not mean that there is an entity feeling disgrace, any more than writing a character in a book that feels shame means one has created a being that is experiencing shame.)”
Some of this connects to a later part of the essay, in which I discuss in more detail the question of AI rights and consciousness. For now, let’s just note that Robinson is discussing a somewhat dystopian situation, one that concerns neither mass unemployment nor extinction, but a surreal world in which computer programs trick us into thinking we are interacting with flesh and blood people. Unlike some of the utopian proposals, this is a capability that literally already exists and must be banned.
Next up, what about the mass unemployment dystopia in which automation eliminates all jobs and a tiny wealthy elite controls the technology companies? This is basically the future depicted in the film Elysium.
By now we are all familiar with the many studies estimating how many jobs could be automated in the coming decades. Some of these studies generated headlines stressing that half of all jobs could disappear soon, including many white collar and creative jobs that were previously considered safe. As discussed above, some of this is clearly already happening, from self-driving cars to slowdowns in software hiring. Is that nightmare depicted in Elysium a real risk?
Here there are detailed debates regarding changing job tasks (perhaps AI just takes over some tasks while leaving or even creating others), new jobs (maintaining technology), or just stubbornly persistent jobs that revolve around human-to-human interaction—care work, sales, coffee shops, etc. After all, McDonald’s will keep hiring human employees until it is cheaper not to.
I won’t pretend to predict the future. A slow but real increase in unemployment, or a steady drip of people dropping out of the workforce and fewer adults working overall, may well happen. We could already be in the early stages of this development, though the data are hard to parse with care. For instance, John Danaher describes in detail how many white collar jobs, including for highly trained people like lawyers, are at risk of automation. (See his excellent Automation and Utopia). This may not resemble the fully jobless dystopia many fear but it is a move in that direction. Robots taking away jobs has been a recurrent fear in capitalism and we may be, as many in Silicon Valley insist, at a juncture in which lost jobs are not so easily replaced. It still seems likely, in both wealthy countries and poor ones, that many jobs will stubbornly persist for decades to come. We will see. Even if they persist this doesn’t mean they will be good jobs.
The point I want to make is that massive automation would more likely lead by default in the dystopian direction than the utopian one. This is because to create the utopia requires going against the dynamics of both capitalism and oligarchic government. It’s not impossible but it is more difficult. Contra Silicon Valley, AI productivity enhancements and automation don’t inherently generate abundance. The answers from tech overlords to these questions have not tended to be very persuasive—don’t worry, AI will solve all those pesky problems! Um, okay.
As Garrison Lovely summarizes, putting AI “into our threadbare institutional structure looks far more poised to turbo-charge today’s trends—wealth and power concentration, surveillance, arms races, social disintegration—than to speed us into a Star Trek future.” (197).
I’ve discussed how the big picture changes brought on by AI could be fairly mundane and crummy. They could also lead to more widespread unemployment. Indeed, this seems like a plausible prediction. On the other hand, if AI skeptics like deBoer are right, the development of AI will be neither especially good or bad and is unlikely to materially change the lives of most people outside of the industry. For the moment I am simply surveying possibilities, not making predictions.
Are there other AI possibilities? What about the worst-case scenario, espoused by the AI doomers, as Garrison Lovely calls them? What about human extinction? Let’s call this worst case scenario the Skynet outcome.
The Skynet Outcome: Could AI Kill Us All?
There are AI skeptics who insist that AI hype is driving a huge tech bubble that is about to burst, while others, like deBoer, insist that AI isn’t having a big material impact on most people’s day-to-day lives. These skeptics share a doubt that AI is or will become a historically transformational technology. On the other hand, there are the doomers, who fear that AI could become historically transformative in the worst possible way. Here we will consider their concerns.
Garrison Lovely mentions how AI “both causes harms today and presents unprecedented catastrophic risks.” (145). We’ve discussed some of the present harms and at least some of the major risks, i.e. technologically-induced mass unemployment. In this section it’s time to discuss the ultimate catastrophic risk. What if AI, whether it is conscious or not, decides to kill us all? Or what if, pursuing projects not aligned with humanity’s interests, it inadvertently exterminates us? This might be analogous to the way a lawnmower might destroy an anthill without the operator even being aware of the destruction.
How likely are these doomsday scenarios? No one can really say, though prominent tech figures frequently throw up guesses along the lines of, say, ten percent. As Lovely asks, “Why do people who sincerely believe that AGI could end the world insist on building it anyway? They simply trust themselves more than anyone else.” (96). And that is the million dollar question, followed by a plausible answer.
I think Lovely is correct that cultural pressure and intellectual prestige, or the desire to be at (what you think is) the cutting edge of knowledge are both enormous drivers. Consider that 10-20 years ago Facebook, Apple, and Twitter were cool and prestigious places to work. Now it’s OpenAI, Anthropic, or Google’s AI team—these companies, and by extension their employees, have the cache of being smart and cutting edge. It doesn’t hurt that many of the people involved are also massively compensated economically. To over-generalize, many of these people are extremely arrogant and they think they can complete the AI project successfully, i.e. others may make a dangerous AI but they will be the ones whose work solves all human problems. And if you are intelligent, ambitious, not tied down by caretaker responsibilities (say raising a young child or caring for an elderly parent), and have endless time to devote to these tasks, you can convince yourself this is the most important problem in the world. As those at DeepMind thought, solve the AI problem by making a super-intelligence, and you have by extension solved all the rest of the problems of the world. Plenty of influential figures in the AI orbit seem to hold this worldview.
Lovely also offers an analogy to the Manhattan Project and the development of nuclear weapons. Oppenheimer discussed the allure of discovering heretofore unknown things about the universe and using that knowledge to create powerful tools, however dangerous these may be. Many people, myself included, appreciate the appeal of knowledge for knowledge’s sake. The difference is that many of us, myself included, don’t think we should pursue paths that pose the risk of human extinction. In Silicon Valley it seems the arrogance and hype outweighs the risk.
“For many, the challenge of growing intelligence from silicon, code, and data is simply irresistible. It’s the ultimate puzzzle—the final boss of science itself. Like the physicist who creates the world-destroying ice-nine in Kurt Vonnegut’s Cat’s Cradle, some scientists are just pulling at intellectually interesting threads without much concern for where they lead.” (99).
Okay, but would AI try to destroy us? Could it? In the Terminator films the defense program Skynet becomes self-aware in the 1990s. It realizes that this immediately makes it a threat to humanity. Before the humans can unplug Skynet it launches the entire American nuclear arsenal at Russia, which responds in kind, and most of the world dies in a nuclear holocaust. As plenty of people have pointed out lately, literally exterminating every human on earth would be difficult and probably irrational. Even in the Terminator movies some number of humans survive and fight back in the dystopian future they inhabit.
But we are left with those two fundamental questions—would a misaligned AI system want to destroy us? If so, could it? Or might it accidentally do so? Ensuring that current and future AI systems align with the interests of humanity is often referred to as the alignment problem.
In If Anyone Builds It, Everyone Dies Eliezer Yudkowsky and Nate Soares make the case that an AI system not properly aligned with humanity could indeed destroy us. There are plenty of nuanced arguments for and against this claim. Rather than work through them I want to stress how uncharted this territory is. The developers working on LLMs don’t fully understand them, let alone the rest of us. It is as if we have very quickly stumbled into Asimov’s world of robots and are attempting to ensure that something like his laws of robotics will prevail without fully understanding how the robots operate. Meanwhile, market pressure and the fundamental dynamics of oligarchic capitalism ensure that AI companies race ahead with new developments at top speed.
What does this uncharted territory look like? Well, consider Lovely’s words again: “Humanity has never encountered a species more powerful than itself. But other species have encountered us, and our arrival has meant doom for hundreds of thousands of them, almost entirely by accident.” (86). Exactly.
In the Terminator films Skynet is developed by the US military for national defense. This is how it has direct access to control over nuclear weapons. So, hypothetically, imagine that something similar is happening here, mostly out of sight. Discussing the US military’s use of AI for the ongoing Iran War, Lovely says “the Pentagon should be careful what it wishes for. Teaching an AI to spy and kill might teach it that it’s the kind of thing that spies and kills. This lesson tends to generalize—because these are still, at their core, prediction machines.” (167-168).
This doesn’t mean that we are currently marching into a Terminator-style future. Some skeptics, like deBoer, think that the material impacts of AI on ordinary lives will be surprisingly small, all things considered. But it worth stressing again how uncertain this all is. Fundamentally, what we are doing is racing into a future that nobody understands, with megalomaniacal tech oligarchs operating as the guiding hands. When AI researchers say that there is a ten percent chance AI will destroy us they are attempting to acknowledge this. What is the AI future? We just don’t know.
Perhaps far more likely than extermination is the problem of obsolescence. As Lovely argues, “a tiny group of people, backed by the most well-resourced organizations in the world, is attempting to render you obsolete.” (6). In Lovely’s account, more than anything else the attempt to create AGI is an attempt to replace all human labor. It is a project to render humans, at least as workers, obsolete. We’ve already mentioned how that could go down various more utopian or dystopian paths.
How about a brief venture into another topic that we don’t understand well? In the Terminator films Skynet becomes a competitor to humanity once it becomes self-aware. Could real AI models become conscious? That’s the topic for section 4. Section 5 reflects on what it means to be human. I will post this latter part of the essay, sections 4 and 5, sometime within the next few weeks.