5 Ways to Ensure Optimized Survey Responses
This is part one of a series on cognitively accessible survey design.
A Survey Is a User Experience
A questionnaire is a tool for researchers to gather data in order to answer research questions, but it is also both an interface and an experience for respondents. Best practices for questionnaire design resemble best practices for user experience design, in that they are oriented toward considering how a user is likely to perceive and interpret aspects of an interface, and how their interactions with each part of an interface will impact their overall journey — what and how they do and experience things within it.</p>
It is still somewhat uncommon for user experience guidance to consider how disabled people may experience the world in non-normative ways. It is even less common to encounter such consideration within the literature on survey design.
In this series of posts, I will be discussing some established principles and best practices of both UX and survey design while rethinking others through a lens of disability ethnography. This first post will look at some approaches for understanding marginalized participants’ motivations and supporting them to optimize their responses.
Optimizing and Motivation
Optimizing refers to the ideal cognitive process through which a person responds to an item on a questionnaire. It is generally accepted as a series of four steps:
- Interpreting a question and deciding what information it is asking for
- Remember and reflect on relevant opinions and/or experiences
- Synthesize those thoughts into a single conclusion
- Determine which of the available responses most closely matches that conclusion
This is a cognitively, and potentially emotionally, demanding process. Therefore, a questionnaire must be designed to make the cost of optimizing worthwhile for its respondents. Doing this requires an understanding of respondents’ motivations for participating.
When a survey population is predefined, if possible, it is good practice to conduct small-scale, preliminary qualitative research to identify respondents potential motivations for participating. However, when this is not suitable or possible, it is reasonable to assume that marginalized participants may be motivated by any or all of these three common factors:
- A desire to express oneself, to be understood and recognized
- A desire to effect change in policies and practices
- Monetary compensation
In my practice, I adopt the following five tactics to ensure that my surveys are aligned with these motivations.
1. Structure questionnaires to create a narrative flow, beginning with concrete experiences, progressing to participants’ impressions of those experiences, asking for interpretation, and offering an opportunity for contextual clarification
When the questionnaire deals with a procedure or other sequential experience, it is useful to identify the possible user journey(s), and structure the questionnaire according to these, offering branching logic to accommodate each possible journey and following the experiences, impressions, interpretations and clarification approach for each step of the journey
2. Provide opportunities for respondents to explain their experiences and ideas in their own words, with optional open-ended questions for each section
This unquestionably adds time and complexity to the data analysis stage; however, if feasible within project parameters, it is a worthwhile tradeoff to establish trust with marginalized respondents who may be accustomed to being misunderstood and misinterpreted
3. Take care to explicitly acknowledge and validate the possibility of respondents’ pain or distress
This can be accomplished through content notes and informed consent language that makes clear what respondents can expect from the survey and that they are not obligated to complete the survey if it causes them distress, as well by including additional content notes at the top of question blocks that may be uncomfortable or distressing, and by making potentially distressing questions optional to answer
4. Respect the expertise and compensate the cognitive and emotional effort expended by respondents
Like all workers, survey respondents will put in the most effort when they are compensated fairly for it. As an aside, it’s worth keeping in mind that as researchers, we are learning from our respondents and asking them to share their expertise with us. Thank them for that rather than “their time.” And consider how you can go beyond lip service and demonstrate that you truly recognize and respect their expertise.
5. Cultivate a pool of respondents through long-term community engagement
One way to show respect for the lived and embodied expertise of research informants is by making your research an ongoing dialogue with them. Knowledge translation and research dissemination is an important recruitment tool that not only increases awareness of projects while recruitment is happening, but increases the likelihood that prospective respondents will be interested in the research and motivated to provide optimized responses.
