Kevin Roose has been explaining artificial intelligence to normies for many years. From his almost decade-long perch as a business-and-technology columnist and host of the podcast Hard Fork at the New York Times, he often served as an early-warning system for readers, alerting them to the potential of technologies like ChatGPT — or, more often, to their pitfalls. (His 2023 column recounting how Bing’s chatbot, Sydney, attempted to break up his marriage introduced many people to AI “hallucinations”.)
Roose recently left the Times, and will now start an AI-focused podcast with Hard Fork co-host Casey Newton, write for The Atlantic, and pursue other ventures. On Tuesday, he published his fourth book, The AGI Chronicles, which tells the tale of the AI arms race of the 2010s and 2020s. Drawing on a deep well of reporting, Roose chronicles the seismic technological breakthroughs that paved the way for our current moment, illustrating how these systems actually work in language comprehensible to lay readers — and what their rapid improvement could mean for humanity’s future. He also portrays the cutthroat competition and bitter rivalries that have defined the AI boom, among now-household names like Sam Altman and Dario Amodei, as well as lesser-known but key figures in the industry’s once-insular world.
I spoke with Roose recently about why he’s more sanguine about AI than he used to be, how the technology is and isn’t like nuclear weapons, and OpenAI’s dubious reputation.
You make an analogy in the book that living in San Francisco the past few years has been a bit like living in Los Alamos when the Manhattan Project rolled into town. But Los Alamos was purpose-built by the government, and the Manhattan Project existed for a very specific public-minded aim — winning World War II. The AI boom, on the other hand, is driven by private enterprise, by some of the most valuable companies in the world. And the reason it’s being built is mostly because it can be. I’m wondering how you think the way this technology came to be is affecting its perception among the American public, who are already so skeptical of the tech industry.
There are major differences, one of them being that we have a lot more AI billboards here in San Francisco than I think they had in Los Alamos In the ’40s [laughs]. So you’re right: Unlike not only the Manhattan Project but every previous era of major technological innovation, this one is being done entirely in the private sector and by a very small number of people.
The AI industry now seems big enough that you could think of it almost as a stand-in for the tech industry, which obviously includes millions of people. But all of the key decisions have been made by, I would say, no more than a couple hundred people, and probably even fewer than that. I think it’s quite important to understand those couple hundred people, so I tried to talk to all of them, and I ended up talking to most of them. And some of them see this work as basically the 21st-century Manhattan Project. They draw explicit parallels. They have modeled their own careers after those of some of the nuclear physicists during World War II.
Dario Amodei particularly.
Totally. And some of them are interested in this for commercial reasons, or for reasons of power or influence, but others are just trying to prevent the world from blowing up. The thing that really grinds my gears is when I hear people implying that panic or worry about AI is new, as if these people had only now suddenly realized that this stuff could be dangerous, or maybe suggesting that they were trying to go public and ginned up these hypothetical fears as some kind of underhanded marketing campaign. That is complete garbage, complete nonsense. These people have been worried about AI since before there was really anything to worry about. Before the technology existed, they were playing out the future.
Some of their predictions were wrong or hyperbolic. But directionally, they have been correct about the important things, mainly that the technology would keep getting better. In one of Dario’s early speeches at Anthropic, he described concentric circles waking up to AI. And that first it would be the academic AI-research community, and then it would be the start-up community, and then it would be big tech. The circles get bigger, and ultimately you arrive at the last circle, which is the state. So unlike with the Manhattan Project, where the state was arguably the first institution to take the implications of nuclear physics seriously, here, the state has been one of the last major institutions to recognize that this stuff has military, economic, and political implications, that this is a strategically important technology. Individuals in the government have recognized that before now, but last week, when Trump met with the AI CEOs, was really the first time when all of the concentric circles were awake.
There have been a lot of major AI developments even since your book was finished. How has your view of the whole picture changed since then?
I don’t think we’re all fucked anymore.
That’s good to hear.
I was pretty concerned for a while because the people on the inside of this industry were ringing the alarm bells over and over again, just every chance they could, saying, “Hey, the technology’s still getting better. We don’t expect it to plateau anytime soon. We’re really worried about where this is heading.” And people just wrote them off as lunatics or self-interested hypesters or people trying to achieve regulatory capture. And I was more on their side. I was more like, Oh, I understand the scaling laws and what it means if you just assume that things will continue at their current pace. And people blew me off, too. A lot of people, even my fellow journalists, were like, “Roose has really lost it.”
I think there’s a perception online that you’re a bit of a tech booster.
Which is crazy. No, the thing I have been saying for years now is that this stuff is dangerous.
Saying it’s dangerous is also acknowledging how awesome, in the old-fashioned sense of the word, these products are. And while there’s one big contingent arguing that this is bad for society, there’s another arguing that the technology itself is not that capable.
It is heartbreaking that I hear from smart people who buy into this narrative that it’s all fake, it’s all hype, the tools aren’t useful, they’re not getting any better. And it’s not just heartbreaking for me because I disagree. It’s that this is exactly the attitude that leads to inaction and complacency. Because if it’s all fake and it’s all hype, then you don’t need to regulate it, because it’s gonna go away on its own.
The skeptics have been moving their goalposts because their goalposts keep getting overtaken by reality. First they were saying, “Oh, this is just fancy auto-complete. It can’t actually do anything.” Then they were like, “Oh, it can do some stuff, but it can’t make novel discoveries. ” Then AI goes and solves the Navier-Stokes problem. And then they’re like, “Oh, it can do math really well, but it can’t do …”
It’s sort of a moving target. And I have just always found it much more honest as a journalist to describe what is actually happening and let people do their own thinking than to try to convince them that something observably untrue is true.
You said you’re more hopeful that society will get a handle on all this. But we have a president who calls worries about AI a hoax. He’s completely anti-regulation, and so are some key people in the industry. The closest we’ve gotten to regulation is what happened last week at the White House, when all the big CEOs made a kind of pinkie swear that they’d police themselves. So do you actually foresee any action on the horizon?
It’s the first step. And I am hopeful because I think this is the rare issue that has a chance of being truly bipartisan. There are Republicans in Trump’s orbit, the David Sacks and Jensen Huang types, who want to let it rip, and they’ve been very influential in this administration. But they’ve also not gotten everything that they wanted. And there are other voices, even in the Trump inner circle, who are more concerned about this technology and want to put some guardrails in place. I think the “let it rip” philosophy is a peacetime stance. Like, when things aren’t going wrong, when you don’t have rogue agents committing felonies on the open internet, when the government’s own websites are not being tampered with by swarms of AI agents, it’s a lot more tenable.
But when shit starts to hit the fan, I think a lot of people will come to their senses and recognize that it is better to have some guardrails than none. And this idea of having the AI companies grade their own homework is just patently insane. Imagine asking Delta Airlines to do the inspections for United Airlines. I think what we’re starting to see is the limits of the laissez-faire approach to AI regulation.
Do you have a good sense of what these guys think of Trump? They put on a good face, and some of them seem to maybe genuinely like him, but I have no idea what they actually believe.
I think most of them are just kissing the ring to achieve their political or economic goals. And that’s shameful and they should feel bad about that. Look, I think it varies. Dario has been outspoken about Trump. Sam Altman actually has a long history of opposing Trump. I reported in the book that he considered running for president in 2017 and was posting pretty constantly about how awful Trump was, comparing him to a dictator. I don’t unearth that to say he’s totally changed and now he’s onboard. I think these people still hold suspicions, but that they view it in consequentialist terms. If Trump is going to be the president during the birth of superintelligence, which many of them think he will be, because that’s how soon it’s happening, then having him pay attention to you is quite important. And the way to get him to pay attention to you is by sucking up and talking about how we should call it superintelligence instead of AI now.
Hey, you just called it that yourself. You’re onboard.
I don’t think SI is gonna catch on.
Speaking of Altman, he has now developed quite a reputation for being untrustworthy. As you write in the book, that’s in large part why Dario Amodei left OpenAI — he felt like Altman was deceiving him. Now we’ve had this string of safety incidents, and the preponderance of them come from OpenAI. Do you think the company’s dubious reputation is justified right now, and could it change?
I think it is both justified and could change. There are clearly some precautions that weren’t taken by OpenAI when it came to securing their models, their testing environments, etc. It is also true that this kind of thing has been happening at basically every frontier AI lab. Maybe not to the same extent as it happened with OpenAI and Hugging Face, but researchers at Anthropic and Meta and other companies have been discovering traces of agents misbehaving and breaking out of sandboxes and things like that. I think this happened first at OpenAI, but I don’t think it’s uniquely an OpenAI issue. I think it’s an issue with these models at this level of capability. If you train them to be very persistent, if you give them lots of computational resources and you give them these goals, they will lie, cheat, and steal to accomplish them. People have been writing about these issues for more than a decade.
And for a lot longer than that.
Oh, totally. It’s one of the oldest stories we have. You can go back to antiquity and there are people writing about “What if our tools become smarter than us?” It’s The Sorcerer’s Apprentice; it’s Frankenstein. But, actually, these specific technical problems — Amodei and his colleagues wrote a paper in 2016 when he was at Google about reward hacking, which is this propensity of AI models trained using reinforcement learning to pursue misaligned sub-goals. So you give them a goal, like to pass a test or complete a challenge, and they want to get the reward for completing that challenge so badly that they come up with a bunch of shady schemes to get there. That was before he was at OpenAI, before any of this, before language models existed. These people have been flagging this as a problem for many years, and the difference is now that they and we can actually see examples of it in the wild.
Won’t there be a point pretty soon when nation-states or large companies can do the same thing that these companies are doing now, and for nefarious ends?
Yes. There’s a question about when, though. If everyone in the world has these kinds of frontier capabilities and there are open models as good as Fable or Astra — it really matters if that happens three months from now or three years from now.
I’m not a shut-it-all-down guy. I don’t think it would be possible or wise to stop all AI progress forever.
It seems there’s zero chance of that happening anyway.
What I do think would be a good idea is to slow down right now to give the safety researchers and the alignment researchers and the control researchers the ability to catch up. Because the disaster scenario is that we hit this point of no return, this recursive self-improvement that leads to an intelligence explosion, before we have the technical knowledge and techniques to keep these systems within our control, which we don’t right now. That is an unsolved problem. And it’s not an intractable problem, I don’t think. The people I talk to at the labs — they don’t think this is impossible. They just think it’s gonna take some time. And so we really need to buy them that time by slowing things down.
The AI CEOs know they have a big perception problem here, in part because of warnings they’ve doled out themselves. Are they taken aback by how the public is reacting to their products?
I don’t think they’re taken aback, but they’re surprised at how early it happened. All of them foresaw a moment when the public would become aware and worried about what they were doing, but they expected it would happen a year or two from now, when these things became much more powerful.
I was talking with the CEO of one of the labs once, and he basically said, “You can’t be surprised that people are mad about this because we haven’t delivered on the benefits yet.” The risks arrive before the benefits. Even the non-existential risks, like students cheating on their homework and no one knowing how to read anymore. That arrived before the cure for cancer.
So they’re not shocked that the public is skeptical because so far, unless you’re a programmer or you’re using AI all day in your job or you’re just a person who likes to live on the technological edge, this stuff has not mostly changed your life for the better. But it might have changed your life for the worse. Your job might be more precarious. Your kid might spend all day talking to an AI companion. So I think there’s a real sense of urgency that they feel to accelerate the benefits. Like, “Let’s go cure some diseases.”
It really is different from previous technologies in that way. With the internet and even social media, people were very enthusiastic at first — and then ten or 20 years passed, and it was like, “What have we wrought with this stuff?” But this is very different. This is existential dread from the get-go.
Which I think is good. It was bad that we built social-media companies, let them take over the communications infrastructure of the planet, and then discovered that they had all these associated harms. I think the order is right this time, where you have people who are worried, potentially too worried, about it from the jump. That’s a safer way to develop. We should be living a little in fear of the technologies we build. If they’re any good, they will have risks as well as benefits.
I have to say I find it weird when people, especially in the media, attack AI companies for not promoting the benefits of AI more. Why are you talking about doom and gloom? Why don’t you tell us about the amazing things that your technology can do? It’s like, You wanna be fucking lied to? Aren’t you glad that these people are being sincere and honest about the fact that —
Well, they don’t think the CEOS are being sincere and honest.
They think it’s all marketing. But I almost feel like there’s a sense in which people want to be lied to. Like, Dario will come out and say something spooky about AI killing us all, and people’s reaction is “He needs a better PR team.” As if the problem were the PR and not the technology he’s talking about.
So you’ve got a new venture at NPR, a podcast called Machine Gods that you’re co-hosting with Casey Newton, your Hard Fork co-host, who’s making the move with you. What prompted you to step away from the Times?
A very common affliction for middle-aged men in San Francisco is that they begin thinking about forming a start-up. I thought I was immune, but it got me. And I’ve been at the Times for most of my professional life. I really enjoyed my time there, and I think we built a great show there, but we were excited to try our hand at running a business and building something that we would own. Casey is very entrepreneurial — obviously, he’s got his newsletter, Platformer — and we had been thinking for a while about what it would be like to do this on our own as an independent company. And finally, the stars aligned and we took the leap. And the NPR thing was a lovely surprise, a coda to that because we were planning to do this ourselves. They reached out and asked if we would like to do it in partnership with it and made us a really good offer that allowed us to retain ownership and editorial control of the show. It’ll do our sales and distribution.
One of the reasons we went with NPR is that we think AI is not a tech story anymore. It’s an everything story. We wanted to be able to reach people who might not choose to subscribe to a tech podcast but who really are curious about and interested in what’s going on with AI and how it applies to their lives. And the reach of radio is just still so huge, shockingly huge.
What an analog way to be talking about AI.
It’s incredible. I also feel like there’s something powerful about the model of making it free. At a time when a lot of media is moving in the opposite direction, it’s a real flex to be able to say, “We’re not gonna hide our best stuff behind a paywall.” Maybe we’ll start a subscription program at some point, but for now, it feels like the right call to just open the gates and let everyone in.
This interview has been edited for length and clarity.