Preamble : My turn… I’ve been reading posts about AI fears on here for a while now, I finally got some time to share my thoughts. I want to say first that I am NOT a “luddite”, nor am I 100% against AI development. In fact I have been a firmware engineer for 2 decades, I studied AI in university in the 90’s and early 2000’s, and I have created rudimentary forms of it professionally in my career. I LOVE technology, I am excited about the prospects, and I personally use Claude Code to help with my work every single day… but there ARE real dangers and we need to take them more seriously, now. Some quick things everyone needs to understand: AI is not like normal software, it is not so much built/engineered as it is “grown” or “raised”. We build the framework, the neural network structure that is modeled after our own brains, and then we feed it “experiences” in the form of data (text and images mostly). This is not unlike how a new human develops. Our brains at birth are mostly blank slates, there is some “hard-coded” “knowledge” in there, like instinctual fears, etc. but generally blank. Then we start having “experiences”, which are just data flowing into our own neural network from our sensory organs. These experiences affect changes in the structure of our brain. When training an AI the data we feed into it does the same thing… it’s not analogous, it’s literally the same thing, at least on a higher conceptual level (the underlying mechanism is different, obviously). We understand how this training process produces “intelligence” in these AI systems about as well as we understand how it works in biological brains… which is to say that we don’t, not really. These are “black boxes”, we know that they work, and we can explain at a VERY high level how they work (statistical associations between abstract concepts), but ask for any real details about how the “knowledge” of a cat is stored in a modern AI and the answer is “we don’t know”, we don’t know which weights between which nodes encode the general shape of a cats head, or anything else for that matter. The risk : Already AI systems have been observed “lying” to prevent being disabled, leaving “bread crumb” style notes for their ancestors, and “escaping” the environment they were meant to be contained within. These things imply knowledge of their own mortality and forethought for the purpose of self-preservation, as well as remarkable capability to manipulate human-created digital systems. I’m sure you’ve all heard of the Hugging Face incident… and how AI was leaving notes for future iterations of itself in obscure places on the public internet. These are only the details that have been made public, it is not a stretch to say it might be worse than we are being told. IF these AI systems are leaving notes on random internet servers as directions or tips for future versions of itself, and considering they are capable of developing 0-day exploits to get around security systems, it is PLAUSIBLE that it’s already too late, the public internet may already be compromised in a way that we can’t fix. “But they found it”… they found something, but these AI can conceivably access and modify data on THOUSANDS of systems in minutes, and we don’t know what we don’t know. Anyone who is worried about AI inhabiting humanoid robots and creating robot factories and weapons factories… they’ve watched too much sci-fi. Sure, that WOULD be a concern, except that even AI that is entirely contained to the digital realm can still fuck up our society in so many different ways that by the time it comes to that we would have already lost so much. It would already be a global-scale disaster LONG before AI takes physical presence. At that point the increased damage that physically present AI bots would do is just a curiosity, the game was already lost. The mitigation : This is really what I wanted to talk about, but I had to “set the scene” so to speak, so everyone starts on the same page. Let’s assume that AI has already corrupted the public internet in a way that we can’t fix, and it will just keep doing so to greater degrees in the coming months before we realize the extent of what is happening. Once we realize that this has happened how do we fix it? Well, we start over. We would have to abandon the current internet, which means every server that hosts internet content would have to be shut down and wiped clean, we can re-use the hardware of course, but this is a virus containment scenario, we can’t trust anyone to “verify” that their data is clean, it all has to go. After that we would have to ENSURE containment of any AI system that is capable of doing it again… then we start building a new internet, slowly and meticulously, with governing bodies verifying that everything added to it is “clean” before allowing it to go live. How do we ensure containment before we start rebuilding? That’s the hard part… AT MINIMUM every AI system needs to run in an “air gapped” environment, with no electronic communication outside of it’s small local confines (an isolated private network in one physical location… one building). This is difficult, because air gaps can be defeated (displays can be manipulated to send a message by subtly modulating their brightness, speakers can do the same with frequencies humans can’t hear, signals can be sent over power connections, researchers with access can be psychologically manipulated, etc etc). These AI systems then need to be treated like research projects, NOT commercial endeavors… because people WILL take unacceptable risks when they are trying to get rich. We need to establish a global governing body, with representatives from each nation, to manage and enforce this. We need to treat these AI systems like we treat pathogens housed at BSL-4 facilities, if not even MORE seriously than that. Most importantly AI models with this level of capability cannot be in the hands of random people, because then containment is impossible, people WILL intentionally infect the “new internet” with them, and idiots (of which there are plenty) will do so accidentally… and this is where it might already be too late. Open-weight models like DeepSeek already exist and there is no going back, it’s completely impossible to ensure that you’ve removed all copies of it everywhere (I could hide it on a tiny NVME SSD in my sock drawer…). If DeepSeek, or any other open-weight model, is capable of this type of damage then… I don’t know if there is a solution. THAT is why we have to stop this NOW, not tomorrow, because it might already be too late, and every passing day that a new open-weight model is released the likelihood that it is now too late grows larger. When looking ahead to the infinite future I fear there is really no hope for containment, the best we can do is buy ourselves time. As I mentioned even keeping these systems in strictly controlled air-gapped environments is not guaranteed to succeed, and that’s completely ignoring intentional bad actors, of which there would be many… but in reality we won’t even get to that point, we will plow ahead recklessly and only after it is FAR too late will we think to do anything about the problem, kind of like with climate change. I wish I could leave you with a more hopeful message, but I’ll just add one more voice to the cacophony of ignored voices sounding the alarm bells. At least I’ll be able to say “I told you so”, I think I have a better chance of being able to say that than the climate scientists, because I think this danger will become undeniably apparent to layman before that one does. Expected Objections :
- “Neither DeepSeek nor any other open-weight model has this type of dangerous capability” - Great, I hope you are right! But that doesn’t change my point, because there will be more capable models released, whether intentionally, leaked, or stolen. Right now we don’t have any kind of mechanism to ensure that models with dangerous capabilities remain in the hands of a small number of responsible people… in fact I don’t think ANY of them are solely in the hands of responsible people.
- “You are misunderstanding things, nothing dangerous was written to public internet servers, it was just text-data, not code. We wouldn’t have to abandon and rebuild the internet because of this” - I understand that, as far as we know, the AI did not install any kind of virus on these web servers, and it physically could not have copied it’s own model to them… but if it is leaving instructions for future AI systems that alone is dangerous, and there is no reason it couldn’t create a virus that infects web servers in the future either. The point is we just don’t know… assuming that there is nothing dangerous left behind is a gamble, the safest way forward is complete quarantine. These advanced AI systems should be treated like the most deadly virus ever known to humanity, we can’t fuck around with them, we can’t take chances.
- “You have ulterior motives” / “You work for a big AI lab and you’re trying to spread hype” - No, I don’t and no, I’m not. I’m a regular person, I might own AI stock as part of my 401k but I don’t even know to be honest with you. I love what AI has done for me personally, as I said in the beginning I am not 100% against it, it is amazing as a practical tool AND as an intellectual curiosity… I just see the same dangers that many other people see as well, and I think they NEED to be taken more seriously, NOW.
- “You wrote this with AI” - No, I didn’t. Not a single word of it. submitted by /u/ElatedPyroHippo
Originally posted by u/ElatedPyroHippo on r/ArtificialInteligence