Original Reddit post

When designing a survey, you should always remember the 3 Rs or Reliability: A survey works best when it is Random, Representative and Rare The quest surveys fail on all 3 points. The surveys are technically uniform chance across demographics, but the uniform chance is 100%. In practice, it’s a lot more effective to randomly present the survey to a fixed percent of each demographic, and scale the responses based on each demographics. This works because you can measure the demographics and responses independently, and not be skewed if one demographic is more likely to respond. They are not representative because they are shown more often for early game quests than late game quests. There are far more completions of Sheep Herder than WGS. On top of this, someone’s first enounter with the survey is far more likely to be in an older quest, and someone’s 100th encounter is far more likely to be in a newer quest. As such, the baseline is skewed towards long-standing early game quests. They are also obviously not rare. Psychologically, people are far more likely to give honest feedback if they feel their voice will not otherwise be heard. Practically, a responder will quickly get tired of answering the same question, and likely start answering out of habit or spite after a few iterations. Making surveys rare introduces a sense of urgency and prevents anticipated response. The fix is extremely simple: Add a small, fixed chance the survey appears on each completion. If a user only encounters 2 or 3 random surveys across their play-through, they’re far more likely to answer honestly. Across thousands of completions, you’ll now have accurate data you can compare against the measured demographics of the player base. You’ll have fewer total completions, but these completions will represent the genuine feelings of the community, instead of the frustrations of seeing the 150th dialogue box. submitted by /u/CallIllustrious2538

Originally posted by u/CallIllustrious2538 on r/2007scape