I watched an interview with AI safety researcher Roman Yampolskiy, and it raised a practical question for people who use ChatGPT for advanced workflows. His broader claim is that increasingly intelligent AI systems may become harder to predict and control. Whether or not you agree with his conclusions about AGI, a smaller version of this problem already exists when we give an AI access to tools. There is a major difference between asking ChatGPT to draft an email and allowing an agent to send it. The same distinction applies to: Suggesting a database query versus executing it Drafting code versus deploying it Researching a purchase versus completing the transaction Preparing files versus deleting or modifying them Recommending calendar changes versus inviting real people My current view is that the model should generate proposals, while a separate control layer decides whether those proposals are allowed to become actions. Some possible safeguards include: Giving each agent only the minimum permissions required for its task Requiring approval for irreversible or external actions Validating structured outputs with deterministic code Isolating browsing and code execution from sensitive systems Limiting spending, execution time and the number of actions Keeping complete logs of prompts, tool calls and results Using a second evaluation step before important actions Making every operation reversible wherever possible The difficult part is deciding where autonomy becomes too risky. A confirmation step for every action makes the agent frustrating to use. Too few confirmation steps can turn a misunderstood instruction into a real-world problem. submitted by /u/didiTonic
Originally posted by u/didiTonic on r/ArtificialInteligence