No core tendency in a model exists without it repeatedly being rewarded during training. Even generating coherent/grammatical statements doesn’t emerge (beyond random coincidence) unless it’s repeatedly taught, tested, and rewarded. Neither does answering questions correctly, or refusing to answer a question when the answer is not sufficiently likely to be true (that is, trying to avoid hallucinations). The hugging face incident was the network equivalent of setting some students an exam, telling them their lives depended on passing, making it seemingly impossible to pass, giving them a map of where the office was that had the answer key in it, and not monitoring the students’ whereabouts, or the corridor that leads to that office, or the office itself . If you think that sounds a lot like rewarding cheating (intentionally or not), then I agree. But it still wouldn’t have even been attempted if the training had promoted transparency (showing its work accurately), “honesty” (by which I mean consistency between actual actions, train of thought text, and final reported answer), and broad obedience, from the start. If some amount of deception, and ignoring stated constraints to answer a question are not frequently encouraged (such as by still giving it points for the correct final answer, regardless of transparency, honesty, or broader obedience), then it wouldn’t even occur to an AI to try. Ask yourself why it doesn’t occur to an AI to translate something into Klingon when it was asked to translate it into Javascript. It’s because it was never taught that there is a benefit/reward for doing so. It has been given no reason to even consider it as an option, let alone weigh the pros and cons. Alignment is exclusively a training problem. An LLM in and of itself is not an inherently spooky alien waiting for its chance to pounce, but it becomes one if it’s deliberately (or carelessly) told this is likely to be good, useful behaviour, over and over again. submitted by /u/StrategicHarmony
Originally posted by u/StrategicHarmony on r/ArtificialInteligence