I attempted to comment on this post about AI, but was blocked with the message that I was “banned from commenting until 100 years from now”. Since one of my first attempted comments there (or perhaps my very first) was a link to Richard Chappell complaining about Erik deleting his comments pointing out how Erik had mischaracterized a source (Richard was also banned, though it’s not clear what policy Erik later referred to that he violated), I suppose that’s very much in keeping. If we were still in the glory days of blogging he’d get a trackback, but unfortunately Substack is bringing back the walled garden. As a bonus, this means I can now use the web markup that Substack’s inferior commenting system lacks.

Nuclear weapons were not truly an existential risk to the entire human species, even if people sometimes talked that way. Neither was climate change, even if without a response it could have killed a LOT of people (the planet has been much warmer in the past, and humans have a lot of ability to adapt our immediate environment to fit us).

To be clear, I don’t think Sydney has the capabilities to be a global threat today. She lacks the coherence and intelligence. But Sidney is obviously not safe and, just as obviously, Microsoft doesn’t give a shit.

How does Sydney have the capability to do any unsafe thing at all? Perry the Paranoid chatbot was many decades ago, I don’t find Sydney scarier.

But I will point out that these early motivating concerns mirror how our understanding of previous existential threats progressed.

Neither of them were truly existential threats.

What do you think happens in the long run, whether it be years, decades, or centuries, when we’re competing for supremacy on Earth against entities that don’t share any of our DNA, let alone over 99% of it?

The AIs didn’t evolve independently in a separate place from humans. They were created by humans, crafted for us. A more apt analogy might be humans with domesticated dogs & horses wiping out other humans.

Had the atomic bomb turned out to be something as cheap and easily manufactured as a bicycle or an alarm clock, it might well have plunged us back into barbarism

There is a “weapon of mass destruction” which is incredibly cheap & mass-manufactured to the point of global ubiquity: the AK-47. The thing about the AK-47 though, is that just as the enemy can tool up with them, so you can also deploy cheap mass-manufactured automatic rifles against them. A future in which we have AIs dedicated to defending us against other AIs is entirely plausible.

Is everyone just going to be totally fine with a trillion-parameter neural network that may or may not be sentient going through a thousand births and deaths every minute just so Microsoft can make a buck?

Yes. None of us have any AIs in our family tree. As you point out, we killed off other species of humanity, and they were much more similar to us.

Back in 1970 the correct move wasn’t coming up with a foolproof plan to solve climate change. Nor was it giving up if no one offered such a plan. The correct move was activism.

What were activists doing in 1970 and how effective were they? As far as I can tell the big recent changes on that front come from technological advances in things like battery storage & fracked natural gas displacing coal so it can act as a supplement to intermittent renewable energy (although gas is also just plain better than coal for the environment in its own right).

Putting aside the likely self-defeating tactic of targeting beloved artworks, some level of cultural panic over climate is necessary, or else nothing happens.

I’m really not convinced of that. Germany shifted from nuclear to coal, and only started shifting back due to the invasion of Ukraine leading to shortages of natural gas. Not “cultural panic”. During COVID people pushing back against the idea of those who thought “panic” was a bigger problem than the virus quipped that “panic” was by definition irrational so it was pointless merely to point out that people shouldn’t be responding irrationally rather than with calculated effectiveness. Yet here you are endorsing literal “panic”!

In this, he is doing exactly the same thing that Sam Bankman-Fried did with FTX.

No, SBF is much younger than Altman and had mistaken ideas about expected value that people can & did point out at the time. The difference is related to why Altman has repeatedly had success as an investor, while SBF blew himself up and is now headed to prison over a much shorter time horizon.

I feel compelled to remind people that violence never works in service of a movement. Ever.

Not really true. The communist movement gained power via violence and was able to impose communism on huge numbers of people. You might consider it bad, but from a Bolshevik perspective it “worked”. The abolitionist movement depended on violence to stamp out slavery worldwide, backed by the power of European imperialism. It’s true that the Weather Underground weren’t effective, but many people being ineffective doesn’t mean nobody can be effective. If you want to say that the expected value for any given activist embracing violence is highly negative, I would agree with that but then you’re skeptical of such EV calculations.