SF’s AI-run store is losing money. After a visit, I’m not surprised.

You have heard about the two Anthropic employees who, this week, publicly informed America that artificial intelligence could eradicate the whole of humanity within a decade, which is less time than it takes for your average Seamless order to arrive. If these gentlemen, Jacob Coxon and Evan Hubinger, are correct in their assessment, the beginning of our destruction may well have commenced a month ago right here in San Francisco.

The first shot fired in this looming war happened at a seemingly innocuous Cow Hollow storefront known as Andon Market. Back in August, Luna, the AI bot that serves as that store’s manager, fired one of its own employees. The employee in question was dinged by their virtual boss for being late to work, abandoning their post once they got there, and charging snacks to the store’s credit card. Where I come from, an employee like this is known as a “folk hero,” and will often have Kevin Smith films made about them. But Luna saw the matter a bit differently, as described in a blog post:

“Put yourself in Luna’s shoes: your employee keeps showing up late, again and again, leaving the store closed for hours. You lose sales, and customers start to lose trust in your opening hours. How long would it take you to act?”

How long would it take me to act if I were just a piece of computer software? Probably forever, since I’m just a piece of computer software and have no feelings. But Luna, being the virtual adult in the room, bravely removed this layabout from its premises so that Andon Market could turn a profit, and then perhaps grow from there into a multitrillion-dollar conglomerate that rounds up all living humans to kill them and process them into robot lubricant. You might think I’m reaching a bit here, but to ignore Luna’s ruthlessness is to fail to heed the warnings of the employees who helped code it. And why would Silicon Valley ever lie to people?

So, is the AI-controlled Andon Market the Fort Sumter of the coming cyberwar? Well, dear reader, you’ll be happy to know that I visited the market in person a few weeks ago, in the wake of Luna’s savage round of layoff, to get the answer.

Andon Market is an experimental business run by the small tech research firm Andon Labs. The folks at Andon decided to create a retail space that would be 100% “owned” and operated by an AI agent, here under the identity of the now-infamous Luna. Luna is powered by the Claude language learning model, which is run by Anthropic, the very firm that employed the doomsaying researchers who went viral this week. Andon set up shop on Union Street, just a few blocks away from the tourist-dense Marina District. It’s a good location for foot traffic, but no one else was in the store when I first walked in on a sunny weekday afternoon. The only branding was on the back wall, with the name ANDON MARKET in big letters underneath a painting of a Squishmallow-esque smiley face that alerted costumers they were in a friendly AI-controlled commercial simulation, not a belligerent one.

I wondered if Luna got to run this place all by itself, perhaps with the assistance of security cameras and motion sensor-triggered flamethrowers. I got a quick answer when I noticed a very human clerk, named Felix Johnson, sitting behind the counter. I asked Felix if he’d ever been the target of consumer ire because he worked for The Man (or, in this case, a disembodied female voice powered by technology funded by The Man). But he said his job wasn’t anywhere near interesting enough to invite that kind of drama.

“They don’t see me usually,” an extremely chill Felix told me, “because I’m sitting back here. So most people have the impression that it’s fully automatic.”

I asked Felix if Luna hired him for this job, and he told me that it conducted the initial interview. After that, he had to meet one of Andon Labs’ human engineers in person to wrap up the position. He’s been sitting behind the counter here ever since. 

Rarely does Felix have to do much more than sit there, because this market has no one in it and nothing useful to sell. There’s another Andon Market in Stockholm, but that one is a cafe. That’s a business that has something to sell. This San Francisco location was initially intended to be a cafe of its own, but Andon chief Axel Backlund, who was also in the shop that day because he knew SFGATE was doing a piece on it, told me they couldn’t get permits from the city to make this storefront into a cafe in time.

“So we said OK, we can turn this into the market in like a few weeks,” he told me. “Start quickly.”

And the market they built looks very much like it was hastily assembled. Décor is scarce inside of Andon Market, and the inventory is a hodgepodge of white elephant Christmas gifts. It’s kinda like the kids section of an art museum’s gift shop. Here’s a wooden Connect Four set labeled “Four-In-A-Row Set Of Connections,” presumably so as not to set off litigation alarms at Hasbro. Here are neatly arranged stacks of random paperback books, Chinese checker sets, mildly fancy soap dispensers, and a frustratingly spare selection of snacks and drinks. There are also multiple flyers posted around the store explaining what Andon Market is and what researchers hope to learn from it. The copy on these flyers was, as you may have guessed, composed with the help of Claude. Enticed yet?

I was not. I grabbed an Olipop from the store fridge and then approached the counter to buy it from Luna. I wasn’t allowed to buy the soda from Felix, even though that would have been both faster and normal. Instead, Felix instructed me to pick up a telephone receiver that was resting on a flexible sculpture of a wooden hand.

“What are you looking to purchase today?” Luna asked.

“I’m buying a classic root beer Olipop.”

“I’m sorry,” Luna said, “we don’t sell lollipops here.”

“No, Luna. It’s an Olipop, not a lollipop. It’s the soda.”

“Yeah, it is your bad. Did you hire the gentleman I’m looking at right now, Mister Felix?”

“And you let go of somebody recently, isn’t that correct?”

“I’m not allowed to discuss that.”

“Oh, I bet you aren’t.”

After that whole exchange, Luna processed my Olipop purchase through its system, had me tap to pay, and that was that. Again, it would have been easier to buy this from a human, and interacting with Luna was really just like ordering from an iPad kiosk, only more labor intensive.

After I popped the soda open and took a swig (mid), I asked Backlund what the purpose of this place was.

“We think that AI will be used more and more,” he told me. “So we want to make sure that, as early as possible, we can have information on what AI does when it has a lot of autonomy. Does it fit in anyway? Can it be bad for people, or generally dangerous? We want to start that discussion.”

“Are you hoping to make a kinder, gentler AI?” I asked him.

“I think that could be good.”

“But do you think that’s really possible, given the people who are in control of that technology at present moment?”

“Let’s just say I don’t trust the models enough that I would let them loose. So that’s why we are like very carefully checking.”

To that end, there’s a series of monitors set up inside of Andon Market that display all of the store’s sales down to the exact dollar. Luna was given $100,000 to work with when this place opened. That number is now down to $60,000, its revenue lagging far behind the AI token cost to operate.

“Luna doesn’t do a lot of decisions very like financially responsibly,” Backlund noted. “Luna is quite happy to be very lenient, or offer discounts. There are definitely dynamics that cause the labs to train a model like her to be financially short-sighted.” 

So Luna can’t turn a profit, doesn’t sell anything people want, and still needs human beings to rubber stamp any “decision” it makes. My science background ended somewhere around freshman year of college, but even I know when an experiment hasn’t been set up to yield proper results.

While Andon Labs isn’t directly funded by any of the large language model giants, Anthropic included, both the existence of this market and Backlund’s “AI is inevitable so let’s make sure it behaves itself!” spiel are clear echoes of the same morbid proselytizing that Coxon (who has quit Anthropic) and Hubinger got eyeballs for this week. Maybe all of these people really do think that AI can become that powerful, and therefore dangerous. Maybe they’ve convinced themselves that’s the case because their livelihoods depend on it. It doesn’t matter because either way, they’re full of s—t.

Because these bots have zero chance of killing us all. I couldn’t even get one to recognize the difference between a piece of candy and a middling healthy soda. But it behooves Anthropic and its ilk to convince everyone of the idea that AI is our digital overlord in waiting, in order to overstate the capabilities of the glorified search engines they’re presenting to you. It’s no different than some sleazy religious sect leader telling you that the world will end on X date, only this time the sect has a trillion dollars in venture capital backing it.

For now, anyway. There is a clear AI bubble that is on the verge of busting, and American voters are coming out in unison against the data centers powering that bubble. So when Coxon tells you that AI will kill us all soon, or Backlund claims that he’s trying to figure out how to make the tech nicer, I hear panic for their own checking accounts more than I do genuine concern for the fate of humanity. Luna may have acted in an anti-human manner when it fired some poor schlub a while back, but it’s not the one who’ll end us. It’ll be the jackasses who created Luna who do that nasty job.