9 Tips to Help Survey Respondents Understand What You’re Asking (and Feel Understood by You)

Cropped aerial view of a person wearing pink pyjama bottoms and typing at a laptop while sitting in bed. One leg is bent sharply at the knee as though it is tucked under their body, which is not visible. The other is fully extended and lines up with the laptop. Photo credit: Sincerely Media, Unsplash.

This is part two of a series on cognitively accessible survey design.

The first part of this series discussed the process of optimizing and considered how understanding and designing for survey respondents’ motivations can increase your chances of receiving optimized responses.

This post will look at some common cognitive access barriers that respondents may encounter when optimizing, and suggest some ways to mitigate them.

To recap, optimizing refers to the ideal cognitive process through which a person responds to an item on a questionnaire. It is generally accepted as a series of four steps:

• Interpreting a question and deciding what information it is asking for
• Remember and reflect on relevant opinions and/or experiences
• Synthesize those thoughts into a single conclusion
• Determine which of the available responses most closely matches that conclusion

Cognitive Barriers to Optimizing

The first step of optimizing constitutes a significant potential access barrier for neurodivergent survey participants. Many neurodivergent people find it especially challenging to attribute meaning to language used by other people. This can be due to a number of factors, such as use of metaphorical or imprecise language, complicated sentence structure, or uncommon vocabulary.

Some neurodivergent people frequently identify multiple possible meanings when reading a statement or question, and struggle with determining which is the intended meaning. This can substantially increase the cognitive burden and energetic cost of responding to a questionnaire, and some respondents may find that they are completely unable to respond to some questions. It can also create a rupture in trust with marginalized respondents who are accustomed to being misunderstood and misinterpreted.

Closed-ended questions, which require respondents to choose from a limited set of predefined responses, can also be an access barrier. Not only do they require another round of interpretation, but the available response options may fail to account for the distinct ways that neurodivergent people may experience and interpret certain things. The latter may also be a problem for other people whose experiences and perspectives are routinely marginalized, and it creates a risk of rupture with participants who are motivated by a desire for self-expression, recognition and understanding.

Techniques to Support Interpretation and Understanding

The following are approaches that can help facilitate interpretation and understanding:

• Use common vocabulary and simple sentence structure whenever possible
• Avoid use of metaphorical or idiomatic language
• Where uncommon or idiomatic vocabulary is needed for a particular reason, an explanation of its meaning should be provided
• Be precise with your language: when the wording of a question or a response option has multiple possible interpretations, alternative wording should be chosen or an explanation should be provided
• Where arbitrary frameworks are used to characterize subjective judgments, the parameters of the framework should be explained (for example, when asking respondents to “rate” an experience, each possible rating should be defined and the way that respondents are expected to assess their experience should be explained)
• Write-in, “don’t know,” and/or “uncertain” options should be provided for all closed-ended questions unless there is a specific reason they need to be excluded
• Open-ended questions should be provided for each question block, to allow respondents to add relevant context and/or clarify their interpretations of questions
• Include a back button so that respondents can revisit and amend previous answers if their understanding changes in responses to subsequent questions

Necessary Unnecessary Questions

Conventional wisdom says that every question in a questionnaire must serve the purpose of providing data that researchers can use to answer their research questions. [Vannette 2014] This is predicated on an assumption that the sole purpose of the questionnaire is to gather usable data (where the characteristics of ‘usability’ have already been decided).

However, if it is expected that a substantial number of respondents will be motivated by a desire for self-expression, the questionnaire must fulfill a second purpose: providing opportunities for that self-expression. Moreover, if respondents are being asked to reflect on experiences of marginalization, including questions that can make them feel recognized and understood may be useful both in enhancing their motivation and, even more importantly, in making the survey a positive experience for them.

With this in mind, question blocks dealing with these topics should be designed with the goal of facilitating an opportunity for respondents to provide a full and nuanced account of their experiences, rather than deciding in advance what questions will provide you with the most direct and concise answers to your research questions. This can be supported by one of the tactics discussed in the first part of this series: structuring question blocks to create a narrative flow that begins with concrete experiences, progresses to participants’ impressions of those experiences, asks for their interpretation, and offers an opportunity for contextual clarification.