I really don’t want to be a doomer, so if you think you can change the way I think about this, please reply in the comments! TL;DR really smart things that aren’t human can be really dangerous (at least from a human-centric perspective), regardless of whether it is controlled by the few or the many. I usually consider myself an optimist, but I feel that treating AGI as something that is more likely to be good than bad is wishful thinking. Not going to detail all my thoughts since it would take way too long (and I need to sleep), but here’s a summary. Assume we get a sufficiently advanced level of AGI. If a small group of elites ends up controlling and restricting access to it, be it a lab or a government, that can clearly be dangerous. All the leverage sits with them, and I’m not sure I trust these actors to use that leverage correctly. I think this argument has been repeated a lot of times, so I won’t go in depth, but the idea is that when a handful of people no longer need everyone else’s labour and thinking, there’s not much stopping them from acting like that’s the case. But the open weights scenario, where access to AGI isn’t restricted or controlled, isn’t that reassuring either. It doesn’t take much to destroy the world or make it a very bad place. You don’t need most people to be malicious, you just need enough people determined to use an uncensored open model to do unthinkable damage. If it can empower a bad actor to make and release a highly deadly bioweapon, that scenario only needs to happen once for it to be a very bad outcome (something something vulnerable world hypothesis). Yes, I’m oversimplifying, and these are the two extremes, and most serious arguments try to find some kind of a middle way. But even when we look for a compromise, all we’re really doing is picking where we sit on the spectrum between “too concentrated” and “too open,” and both ends of that spectrum seem like they can go wrong all too easily. Mixing and matching doesn’t get you out of the underlying problem, which is that AGI hands out an enormous amount of power to do damage. Intelligence will be the closest thing to a superweapon we’ve ever produced, and no arrangement of who holds it makes that fact go away. I can definitely think of scenarios where AGI to be aligned and somehow steer clear of both outcomes, but assuming we don’t have plot armour, I don’t see why that good outcome should be more likely than the two bad ones. Getting it right seems to need a narrow set of things to all go well at once, while getting it wrong just needs any one of them to fail. I don’t know, man. Just wanted to rant and hear what other (probably more informed people) think about this. submitted by /u/CarbonChen
Originally posted by u/CarbonChen on r/ArtificialInteligence
I have about 20 years of experience in machine learning algorithms development.
I wish I had an argument that would change your mind because I could use such an argument for myself. I don’t think you are a doomer. This is our generation’s nuclear arms race but with more ways to go bad. This is taking place at a time when we need unity of mind and action but are going in the opposite direction at warp speed.
What I would like to know is not, “what is going to happen?” But “what are we going to do about it?”