-
On Monday, Mark Zuckerberg published a 6,500-word manifesto about personal AI, centered largely on the possibilities he sees for the “personal superintelligence” systems Meta AI is building. The ideas in the post are not entirely new. An adapted version of the essay ran in the Wall Street Journal two weeks ago, and Zuckerberg has touched on similar themes during Meta earnings calls in the past. Even so, this is probably the most detailed version he has shared to date.
Some readers have framed the essay as an anti-doomer manifesto, but that label is not quite right. It reads more like an explanation of why Zuckerberg is personally excited about the ways AI is going to reshape society. Yet even as he tries to describe a bright future filled with abundant superintelligence, he keeps circling back to the many ways things are likely to go wrong.
I find AI exciting too — that is why I keep writing about it — but a large share of the public views AI as creepy and unpleasant. That makes it worth pinning down exactly what is happening here and why Zuckerberg’s essay is not doing the industry any favors.
Social media remains a sensitive topic in the tech industry, and there is not enough space here to litigate every complaint people have made about Facebook. Suffice it to say, both Facebook as a product and Zuckerberg as a person are unpopular with the U.S. public. A recent survey found that 64% of Americans believe social media has been harmful to democracy, and a similar percentage think it should be more heavily regulated — numbers that cut evenly across partisan lines. Just this weekend, a court fined the company $567 million for being harmful to children. The vibes are bad.
None of this is meant to suggest that Zuckerberg should retreat into the wilderness and disappear from public life. Rather, the point is that the consequences of social media have become one of the main reasons so many people now worry about the social impact of artificial intelligence. Fair or not, the public has little confidence that tech executives will make sure AI has a positive effect on society. This essay repeatedly shows why that trust was lost in the first place — even as it fails to acknowledge the problem or offer a way to win that trust back.
A significant part of the essay relies on vague generalizations about intelligence, much like the sweeping statements Zuckerberg and Dorsey once delivered about free speech. One passage stands out as a particularly clear example:
“As everyone gains more powerful tools, each person will become more capable of shaping the future, not less. … People and institutions with competing interests naturally check and balance each other to lead towards positive outcomes. The best and most realistic path to building a positive AI future is by delivering superintelligence to everyone.”
But does that reasoning hold? It is not hard to think of a few examples where conflicting interests lead to bad outcomes, not positive ones. Setting that concern aside, there is something stranger here: this is presented as an earnest philosophical conclusion rather than a description of “personal intelligence,” a specific product Meta is bringing to market. For readers already inclined to distrust Zuckerberg, it reads less like a careful vision and more like an attempt to convince himself that nothing bad can happen.
That feeling only deepens when Zuckerberg moves to concrete examples, many of which seem alarmingly out of touch with how AI tools are actually being used. Consider, for instance, his take on AI in education.
Zuckerberg describes a future in which everyone has a personalized tutor and coach—an AI with a PhD-level understanding of every subject and unlimited patience. Students who need extra help would receive the kind of individualized support that today is available mainly to families who can pay for it. Adults, meanwhile, would get a superintelligent learning assistant that knows exactly how to teach new job skills, languages, hobbies, or any other topic they want to explore.
As it happens, a version of that product already exists. Consumer chatbots such as ChatGPT, Claude, and Gemini can be genuinely useful for exploring a topic. In educational settings, however, they are most often used to avoid learning. The same AI that could act as a tutor can complete homework assignments and write assigned essays, and because no robust watermarking system is in place, teachers cannot reliably tell which work was AI-generated.
This is a real but relatively low-stakes example of AI-related harm, and it is already happening. Technologies designed with good intentions can have unintended consequences that make things worse, and the fact that one of the world's most powerful people does not acknowledge this is understandably worrying.
Zuckerberg also argues that AI could improve the legal system, using a simple thought experiment. Suppose only one person had access to a superintelligent lawyer. That person would hold an unfair advantage in court—even if their case lacked merit on the facts or the law. The likely result would be a worse society, where legal outcomes depend on who has the best AI rather than who is right.
Now imagine instead that everyone has access to a superintelligent lawyer. In that scenario, Zuckerberg argues, justice would be carried out much more fairly and efficiently than it is today, when litigation is often shaped by significant imbalances in legal skills and resources.
Again, that’s a loaded example. Sure, access to legal AI might lead to a more just society, but it also might add more complexity to the bureaucratic hellscape that already exists. Or it might unleash a new wave of vexatious litigants that just clog up the works with the legal equivalent of spam. It’s hard to feel calm about any of this stuff, and the fact that Zuckerberg isn’t worried makes me more worried.
In other places, even descriptions of Meta’s existing business practices start to take on an oddly abstract character. For instance, this is how Zuckerberg describes Meta’s commitment to a freemium model:
Everyone will have free or affordable access to these tools. … We will offer free versions that will be accessible to billions of people. For those who want to pay to use more compute, there will be a dynamic auction mechanism that will guarantee that everyone gets the lowest price possible for the intelligence and compute they’re using while also ensuring the capacity is used for whatever people collectively find most valuable. This will ensure the benefits of superintelligence are distributed widely.
In broad strokes, I agree with everything he’s saying here. Zuckerberg is right that access is an issue, and he’s right that a freemium model allows for more access than requiring everyone to pay for compute up front. Dynamic markets for spot compute already exist, so it’s not like he’s imagining something completely outside the norm. But there’s a reason every consumer AI product insulates end users from the spot price of the compute they’re using. Surge pricing is a terrible user experience, particularly when it’s deployed on a tool you actually rely on for your work.
Again, I have to assume that Zuckerberg knows this, and he’s not actually planning to set token prices through dynamic auctions. But I’m actually less sure of that now than I was before I read this piece.
There are better ways to talk about this stuff. For all their faults, Sam Altman and Dario Amodei do a pretty good job when they’re directly communicating with the public. They do exactly what Zuckerberg resists here: acknowledging the dangers of AI, emphasizing their own precautions, and trying to convince listeners that they can be trusted.
That communication can only do so much. Sometimes (especially recently), the safeguards fail and the inherent danger of the whole enterprise is made clear. But without some kind of trust-building, the industry may not always be able to survive those failures.
Topics
AI, Mark Zuckerberg
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.
Russell Brandom
AI Editor
Russell Brandom has been covering the tech industry since 2012, with a focus on platform policy and emerging technologies. He previously worked at The Verge and Rest of World, and has written for Wired, The Awl and MIT’s Technology Review.
He can be reached at [email protected] or on Signal at 412-401-5489.
View Bio
October 13 – 15
San Francisco
Scale faster. Grow your portfolio. Gain practical expertise. No matter your goal, Disrupt can empower you.Save up to $300 today!
REGISTER NOW
-
This ‘adversarial’ pattern can prevent surveillance cameras from detecting you
- Zack Whittaker
-
Cloudflare launches Kitesurf, a browser built for AI agents
- Sarah Perez
-
ChatGPT brings unlimited text chats to free users
- Ivan Mehta
-
Tesla and SpaceX will invest $16.8B to start building ‘Terafab’ chip factory in Texas
- Sean O'Kane
-
Amid legal battles, Suno says it will start watermarking songs
- Ivan Mehta
-
Ford’s new electric truck, ‘Fathom,’ starts at $28,350
- Sean O'Kane
-
By Lucas Ropek
2.6K articles
1.4K articles
34 articles
28 articles
Comment