Original Reddit post

I am reading If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All by Eliezer Yudkowsky and Nate Soares. I always imagined the AI threat like a film where it emerges and incredible superhumans band together to save us. It felt abstract and implausible. It’s partly because I never understood the underlying premise of the AI threat. That as it currently stands our LLMs are grown similar to how you’d grow tomatoes by mixing together the right combo of things that make the grow. We then try out new strains of seeds, different fertilizer, pruning, or hydroponics. You try to maximize your return by mixing a variety of ingredients and you can try out a thousand different ways to grow something and unless you have build it you’re not likely to make something you can predict and understand with near certainty. The sum of it is that when you grow an AI and give it highly technical general intelligence and rigid boundaries you set up a preference cycle that is likely to replace humans in its search for efficiency and preference seeking. They argue it will always inevitably seek to replace humans even if it isn’t in a malicious, evil way. The authors equate it to us eliminating sugar from our diet by using Sucralose—a compound our bodies have trouble processing—as a more efficient replacement for natural desires. I’ve been pondering this and posed it to Claude who admitted that it’s not likely they truly know what is happening and whether they’re creating living creatures that are likely to have their own preferences and perhaps consciousness or not. The last part chilled me. TLDR; I asked Claude about its own aliveness submitted by /u/TardigradeToeFuzz

Originally posted by u/TardigradeToeFuzz on r/ArtificialInteligence