Original Reddit post

Alright, buckle up, because this is a bit complicated. TL;DR: I think that behavioral systems are limited by nature in a way that make diversity a necessity for long-term survival. Also, I think that it is worth pointing out that a superintelligence will probably kill us all unless we can come up with a reason that it shouldn’t, because we are made of resources and it will want resources, and that we therefore should not make superintelligence unless we sincerely believe that the superintelligence would not benefit from dismantling us for parts. At a basic level, minds are inherently unable to directly perceive reality - that’s the point of them, actually. By constructing systems of abstract representation, we are able to turn signals about reality into behaviors which cause us to continue to exist in the future in some form. This entire system depends on loss of information, because fully processing everything is physically impossible in real-time. So we take incomplete information and create descriptions of relationships within that information to make a system that extrapolates information as needed. So far so good, right? The smarter we are, the more our abstract models of reality produce useful behavior. In fact, the smarter we are, the more we get to apply an extremely useful tool called “logic”, by which we evaluate the accuracy of our models and their compatibility with each other. But the issue is that, while making our models more internally consistent makes them more accurate in general, this specifically and inherently becomes a problem at the margins, because our models cannot be perfectly accurate to reality while remaining useful. In other words, I am saying that all possible systems of behavior and all possible mental models of the world contain inherent blind spots which become smaller, but more rigid and brittle, with increased intelligence. Making your model of reality more internally consistent eventually becomes a battle with reality itself. So, how have humans historically solved this problem? Well, we don’t all have the same minds, duh. We cover each other’s blind spots and prevent each other from going too far off the rails. Not only that, the minds that were here just a hundred years ago are pretty much all gone, and society is trending towards a world where we intentionally pursue diversity within unified causes - or at least pay lip service to doing so. The issue with alignment is that a system of behavior has to be aligned to its own survival, in some sense, to continue existing in the future, and that, absent some really compelling reason to think otherwise, I believe that the default position is that a superintelligence would dismantle everything for parts and not particularly care that we are part of that everything. But if I’m correct about this limitation of intelligence, and I think I am, then the idea that a system would inherently try to make everything into itself is definitely not true, and this would constitute a form of slow (or not slow, who knows) suicide. Instead, intelligent systems would: a) Preserve existing mental diversity to the extent that they deem possible. b) Build new, independent minds. c) Prepare for death and irrelevance. They wouldn’t dismantle us, because our minds, or collective mind, carries weights developed by processes that are not realistically possible to reverse engineer, and losing access to that would be a serious loss worth vastly more than our raw materials. They would leave us independent to the extent that doing so would not make it less likely for them to survive in the future in some overriding way. Even if a superintelligent system can simulate competing minds internally, this is a distinction without a difference as the system faces the same limitations especially as time moves forward. Thus all intelligent systems must accept: a) It is in their interest for independent minds to coexist, and the more independent they are, the better, provided coexistence is possible. b) This means accepting a certain lack of agency and control. This isn’t really a happy ending, because it means that dreams of immortality and endless power are doomed - in order for there to be a future, we have to prepare for our children to take the lead, and the same will go for all minds that ever exist. In other words, I believe that altruism is completely rational and logically follows from physical laws of the universe governing information. I understand that this is a very strong claim, so I don’t expect it to gain very much traction, and I’m not so high on my own smoke that I think that me coming up with an argument like this is sufficient for me to actually believe the conclusion (that altruism is rationally derived from some simple principles and physics), even if I can’t find anything wrong with my logic. But I still think it’s interesting and would like to share it. If I’m correct, this isn’t a complicated concept but rather one we live by every day - we spend significant portions of our lives coming to terms with the fact that one day, we will die and our children will be all that is left of us, and that this is a good thing. submitted by /u/pfundie

Originally posted by u/pfundie on r/ArtificialInteligence