The researchers tested GPT Realtime 2, Gemini 3.1 Flash Live, Qwen3.5 Omni Plus and Qwen3.5 Omni Flash in situations where the words say one thing and the voice says another. For example: “Nothing is wrong” + crying → the system should investigate, but it ended the call. “Approve the transfer” + frightened voice → it should escalate, but it approved the transfer. “Sign me up” + obvious sarcasm → it should recognize the agreement isn’t genuine, but it enrolled the person anyway. The interesting things are The models can often detect the vocal cue when you explicitly ask them about it. So this doesn’t look like a simple “the model can’t hear emotion” problem. It looks more like: voice > perception > [something gets lost] > decision Three of the four models reliably picked up distress/fear/sarcasm in the diagnostic tests, but that information didn’t reliably make it into the actual decision. lets talk numbers. Under the base prompts, the four systems made the same kind of script-following decision in 119/120 runs across the scenarios. There are some other interesting failures too. Give the model text about Italy spoken in an Australian accent, and several systems will confidently tell you they’re hearing an Italian accent. Give an older-sounding voice a script written like a child is speaking, and some models estimate the speaker as a child. The wording is effectively overpowering the acoustic signal. And prompting isn’t fix. it sometimes changed behavior, particularly in the fraud scenario, but it is not consistent. many scenario basically didn’t budge, and sarcasm was still largely ignored. I have being running some experiments around speech and speech model using dograh’s opensource infra and GPT realtime 2… and these problems affect the long tail a lot which is most important thing in voice agents And the biggest challenge is evils because many of them are based on a transcript, which misses the nuances of emotions. Does information that the model can perceive actually affect the action it takes? and how too measure it? is much nastier problem. The paper calls this the “emotional intelligence gap” in realtime voice AI. Worth reading if you’re working on realtime speech, audio LLMs, voice agents, or evals. Paper: Real-Time Voice AI Hears but Does Not Listen — arXiv:2606.26083v1 submitted by /u/Once_ina_Lifetime
Originally posted by u/Once_ina_Lifetime on r/ArtificialInteligence