Today I play media theorist and examine how survey questionnaires reinforce survey designers’ biases:
The knowledge that biases emit from survey questionnaires is nothing new. The extreme case, “push-polling”, intentionally guides the questionnaire reader toward a viewpoint, without real interest in their prior opinion. Any survey writer willing to push-poll already understands my concerns about bias (because they are propagandists).
It is the unintended or “honest” biases that concern me here.
Consider for example the common belief that individuals can be categorized as a member of one out of four or five distinct racial groups, a belief reflected in many survey questionnaires that ask respondents to indicate which race they belong to. This is an example of what I call an “honestly” projected bias; the survey writer likely has limited awareness that there is even a problem, and does not expect their respondents to question the belief. In these cases, the bias enters the survey questionnaire through the questionnaire writers’ phrasing and provided options, and is confirmed when each respondent chooses one of the options.
Stepping back, we observe “bias in, bias out” where the belief itself gains strength across the survey process. It strengthens among the respondents as they accept the belief when answering the questions, and strengthens in the mind of the survey creator when they see tacit acceptance of the bias in the responses. At each step, neural pathways supporting the belief become stronger due to exercise.
I’ve mapped this process below, illustrating the cumulative bias amplification by degree of red in the arrows’ color:
While we cannot completely escape projecting our biases through our measurement instruments, I call on questionnaire writers to step back and consider what we might be propagating. We may have to become more creative to limit the damage. (For one example of a creative approach, see here for an idea on how to avoid propagating the binary sex/gender bias through survey questionnaires).