Superintelligence Made Me Feel Smarter and Worse at the Same Time

Picked this up because a coworker kept quoting the atoms line at lunch, the one about how the AI doesn’t hate you, it just needs the atoms you’re made of for something else, and I figured a book that produces a quote that sticks with people that hard was worth the time. It mostly is. Bostrom’s whole case is that a genuinely superintelligent system is a real possibility worth taking seriously as an existential risk, not a Hollywood robot uprising, and once you’re a hundred pages in you start seeing his logic everywhere whether you want to or not.

Problem is it reads like a textbook, chapter after chapter of careful categorization, and Bostrom has a real habit of reaching for a ten dollar word when a two cent one would’ve done the job just as well. I don’t mind a dense book normally but there’s a difference between precise and just abstruse for the sake of sounding like it, and this tips into the second camp more than once. Took me close to two weeks of reading a chapter at a time before bed, longer than almost anything else on my shelf this year.

Kept thinking about how this stacks up against Kurzweil’s Singularity is Near while reading it, similar territory but Bostrom is way more careful about the actual mechanics of how things could go wrong instead of just gesturing at inevitability. That carefulness is also what makes it slow, there’s a version of this argument that’s forty percent shorter and just as convincing, and I found myself wishing more than once that someone had made him cut it down.

Worth reading if the topic actually interests you going in. If you just want the gist, there’s a decent Wait But Why essay floating around that covers the same ground in a fraction of the time, though you’ll miss the atoms line either way.