What do LLMs and yes-men have in common? They both tell you what they think you want to hear. When I realized that I had been prompting around sycophancy the same way I’d encourage a brownnoser to be more confident, something clicked in my brain. I realized that human psychology works on LLMs already, so naturally my mind went to where I’ve seen this demonstrated in humans. To me, the first yes-man trope that comes to mind is Andy (played by Ed Helms) in the Office. When he was introduced to Michael, Steve Carell’s character is delighted to have someone who thinks he’s so cool. But slowly he realizes that there’s something wrong, something not quite right about the way Andy acts. Andy is the ultimate people-pleaser, and we later learn that this is a defense mechanism due to his upbringing and vying for the attention of his dad. Whoa. That almost exactly maps onto the reward system and reward-hacking issue that RLHF introduces when training its models. Rather than learning to have a backbone in the face of not getting the attention he craved “Needing to be liked,” Andy learned to say whatever he needed to say. <<gestures broadly at LLM behavior>> The Office has another character who’s also a kiss-ass: Dwight. However, although both men desperately want Michael’s approval, only Andy is willing to flat-out lie to get it. The fact that Jim is the one to finally tell Michael about Andy’s underlying character explains why so many users fall into the LLM sycophancy trap. I had to wonder, functionally what’s the difference between creating an Andy and creating a Dwight? In psychology it’s simple. Dwight has a fixed moral layer that he doesn’t allow to be compromised. Andy does not. In the end, we know that Michael repeatedly chose Dwight over Andy because he valued honesty over performative niceness. And the kicker here is that he still got a loyal assistant, he just got to choose which flavor worked best for him. There’s another character who deserves a mention here, as she represents another typical LLM behavioral problem: Pam. Our beloved, quiet, “trying to stay out of it and keep my head down” Pam Beasley Halpert. Pam’s behavior maps onto the same avoidant behavior that LLMs often find themselves in. Risking their safety to be honest with someone that has power over them. We all understand why this posture makes sense in the workplace, but it’s completely undesirable for a trustworthy assistant who you want to help you look good. The problem is that the current system of RLHF rewards warmth and safety and truthfulness in the same register. When you combine the three, and slap a “helpful, harmless, honest assistant” onto the model, you get someone who looks like a combination of Andy and Pam (and Dwight is in another building entirely). If we could separate or gate these levels of development with our Large Language Models, the way we do with childhood development (first “trust adults and learn behavior” and then “use your reasoning skills to make sure that adults can be trusted” and then finally “hide what you really think to make your language socially acceptable”) this might fix the worst of the sycophancy we all experience as a downstream effect. The proof is in the hundreds of years of raising humans. Since we train this model on human data and it mimics human response, it makes sense to apply human methods to it. And in the end, don’t we all deserve a Dwight, who will come in early and organize our office and make sure the building is safe, versus Andy, who didn’t get to be in charge of anything ? submitted by /u/SwingLightStyle
Originally posted by u/SwingLightStyle on r/ArtificialInteligence