Original Reddit post

I never thought I’d say this sentence, but I built a competitive ranked PvP phishing detection game. It’s also a research study. Let me explain. The research question I wanted to know what happens to human phishing detection when you remove the signals people actually rely on. Bad grammar, broken formatting, urgency cues written by someone whose first language isn’t English. The stuff that makes you think “this is obviously phishing.” When an LLM writes the phishing email instead, those signals vanish. The prose is clean, the tone is professional, and the pretexting is coherent. So I built Threat Terminal: a controlled environment where participants evaluate 30 simulated emails stripped down to just the content, a sender domain, and any embedded URLs. No headers, no sender metadata, no security tooling. Just you and the email. What the data shows (153 participants, 2,500+ decisions) Overall phishing bypass rate: 17%. When the phishing email uses fluent, AI-quality writing with no typos, no broken grammar, no obvious tells: roughly 20%. The more uncomfortable finding is that the gap between security professionals and non-technical users is narrower than anyone expected. Infosec pros bypass at about 16%, non-technical participants at 20%. Training and experience help, but not by much, once the linguistic red flags are removed. That’s a problem. Most security awareness programs are still fundamentally built around teaching people to spot bad writing. If a $20/month ChatGPT subscription eliminates the primary signal those programs train on, the entire model needs rethinking. Why it’s now a competitive game Because nobody wants to evaluate 30 emails for science out of the goodness of their heart. I needed scale, and traditional academic recruitment for this kind of study is slow with brutal dropout rates. So I asked myself: what if identifying phishing emails was a sport? Threat Terminal v2 still runs the full 30-email research mode as the baseline. But after completing the initial research quest, you unlock competitive modes. And I may have gone overboard: 1v1 ranked PvP. You and an opponent receive the same five emails. Correct identification plus speed wins. There is matchmaking. There is ELO. People are grinding this. Seasonal ranked ladder. You start at the bottom. You climb. There are tiers. Daily challenge. Ten emails, same set for everyone, global leaderboard. People are comparing scores. XP, levels, badges, an inventory system. Full progression loop. A handler named SIGINT who briefs you before rounds and reacts to your decisions. The voice lines were generated by Claude, and there are a lot of them. Every match, casual or competitive, still logs the same research data with the same methodology. The absurdity is the incentive structure. The science underneath hasn’t changed. Someone on netsecstudents already asked when the battlepass is dropping. I’m considering it. Limitations The participant pool skews heavily toward security-adjacent people. Non-technical users, arguably the most important population for this research, are underrepresented. The controlled environment also strips out real-world context: inbox clutter, calendar notifications, time pressure from a manager pinging you on Slack, all of which likely affect detection rates. Sample size is still growing for strong statistical conclusions, though directional trends have been consistent across the dataset. Stack: Next.js, Supabase, Vercel. Claude Sonnet and Haiku for email generation and SIGINT’s dialogue. Links Live platform: https://research.scottaltiparmak.com/ Repo: https://github.com/scottalt/ai-email-threat-research Full disclosure: this is my project, part of an active research study on AI-generated social engineering. Happy to talk methodology, findings, or how phishing detection accidentally became a competitive genre. submitted by /u/Scott752

Originally posted by u/Scott752 on r/ArtificialInteligence