Questionnaire Design | Methods, Question Types & Examples
Questionnaire Design | Methods, Question Types & Examples
A questionnaire is a list of questions or items used to gather data from respondents about their attitudes, experiences, or opinions. Questionnaires can be used to collect quantitative and/or qualitative information.
Questionnaires are commonly used in market research as well as in the social and health sciences. For example, a company may ask for feedback about a recent customer service experience, or psychology researchers may investigate health risk perceptions using questionnaires.
Questionnaires vs. surveys
A survey is a research method where you collect and analyze data from a group of people. A questionnaire is a specific tool or instrument for collecting the data.
Designing a questionnaire means creating valid and reliable questions that address your research objectives, placing them in a useful order, and selecting an appropriate method for administration.
But designing a questionnaire is only one component of survey research. Survey research also involves defining the population you’re interested in, choosing an appropriate sampling method, administering questionnaires, data cleansing and analysis, and interpretation.
Sampling is important in survey research because you’ll often aim to generalize your results to the population. Gather data from a sample that represents the range of views in the population for externally valid results. There will always be some differences between the population and the sample, but minimizing these will help you avoid several types of research bias, including sampling bias, ascertainment bias, and undercoverage bias.
Questionnaire methods
Questionnaires can be self-administered or researcher-administered. Self-administered questionnaires are more common because they are easy to implement and inexpensive, but researcher-administered questionnaires allow deeper insights.
Self-administered questionnaires
Self-administered questionnaires can be delivered online or in paper-and-pen formats, in person or through mail. All questions are standardized so that all respondents receive the same questions with identical wording.
Self-administered questionnaires can be:
- cost-effective
- easy to administer for small and large groups
- anonymous and suitable for sensitive topics
- self-paced
But they may also be:
- unsuitable for people with limited literacy or verbal skills
- susceptible to a nonresponse bias (most people invited may not complete the questionnaire)
- biased towards people who volunteerbecause impersonal survey requests often go ignored.
Researcher-administered questionnaires
Researcher-administered questionnaires are interviews that take place by phone, in-person, or online between researchers and respondents.
Researcher-administered questionnaires can:
- help you ensure the respondents are representative of your target audience
- allow clarifications of ambiguous or unclear questions and answers
- have high response rates because it’s harder to refuse an interview when personal attention is given to respondents
But researcher-administered questionnaires can be limiting in terms of resources. They are:
- costly and time-consuming to perform
- more difficult to analyze if you have qualitative responses
- likely to contain experimenter bias or demand characteristics
- likely to encourage social desirability biasin responses because of a lack of anonymity
Open-ended vs. closed-ended questions
Your questionnaire can include open-ended or closed-ended questions or a combination of both.
Using closed-ended questions limits your responses, while open-ended questions enable a broad range of answers. You’ll need to balance these considerations with your available time and resources.
Closed-ended questions
Closed-ended, or restricted-choice, questions offer respondents a fixed set of choices to select from. Closed-ended questions are best for collecting data on categorical or quantitative variables.
Categorical variables can be nominal or ordinal. Quantitative variables can be interval or ratio. Understanding the type of variable and level of measurement means you can perform appropriate statistical analyses for generalizable results.
Examples of closed-ended questions for different variables
Nominal variables include categories that can’t be ranked, such as race or ethnicity. This includes binary or dichotomous categories.
It’s best to include categories that cover all possible answers and are mutually exclusive. There should be no overlap between response items.
In binary or dichotomous questions, you’ll give respondents only two options to choose from.
White
Black or African American
American Indian or Alaska Native
Asian
Native Hawaiian or Other Pacific Islander
Are you satisfied with the current work-from-home policies?
Yes
No
Ordinal variables include categories that can be ranked. Consider how wide or narrow a range you’ll include in your response items, and their relevance to your respondents.
15 or younger
16–35
36–60
61–75
76 or older
Likert scale questions collect ordinal data using rating scales with 5 or 7 points.
Very dissatisfied
Somewhat dissatisfied
Neither satisfied nor dissatisfied
Somewhat satisfied
Very satisfied
When you have four or more Likert-type questions, you can treat the composite data as quantitative data on an interval scale. Intelligence tests, psychological scales, and personality inventories use multiple Likert-type questions to collect interval data.
With interval or ratio scales, you can apply strong statistical hypothesis tests to address your research aims.
Pros and cons of closed-ended questions
Well-designed closed-ended questions are easy to understand and can be answered quickly. However, you might still miss important answers that are relevant to respondents. An incomplete set of response items may force some respondents to pick the closest alternative to their true answer. These types of questions may also miss out on valuable detail.
To solve these problems, you can make questions partially closed-ended, and include an open-ended option where respondents can fill in their own answer.
Open-ended questions
Open-ended, or long-form, questions allow respondents to give answers in their own words. Because there are no restrictions on their choices, respondents can answer in ways that researchers may not have otherwise considered. For example, respondents may want to answer “multiracial” for the question on race rather than selecting from a restricted list.
- How do you feel about open science?
- How would you describe your personality?
- In your opinion, what is the biggest obstacle for productivity in remote work?
Open-ended questions have a few downsides.
They require more time and effort from respondents, which may deter them from completing the questionnaire.
For researchers, understanding and summarizing responses to these questions can take a lot of time and resources. You’ll need to develop a systematic coding scheme to categorize answers, and you may also need to involve other researchers in data analysis for high reliability.
Question wording
Question wording can influence your respondents’ answers, especially if the language is unclear, ambiguous, or biased. Good questions need to be understood by all respondents in the same way (reliable) and measure exactly what you’re interested in (valid).
Use clear language
You should design questions with your target audience in mind. Consider their familiarity with your questionnaire topics and language and tailor your questions to them.
For readability and clarity, avoid jargon or overly complex language. Don’t use double negatives because they can be harder to understand.
Use balanced framing
Respondents often answer in different ways depending on the question framing. Positive frames are interpreted as more neutral than negative frames and may encourage more socially desirable answers.
| Should protests of pandemic-related restrictions be allowed? | Should protests of pandemic-related restrictions be forbidden? |
Use a mix of both positive and negative frames to avoid research bias, and ensure that your question wording is balanced wherever possible.
Unbalanced questions focus on only one side of an argument. Respondents may be less likely to oppose the question if it is framed in a particular direction. It’s best practice to provide a counter argument within the question as well.
| Do you favor…? | Do you favor or oppose…? |
| Do you agree that…? | Do you agree or disagree that…? |
Avoid leading questions
Leading questions guide respondents towards answering in specific ways, even if that’s not how they truly feel, by explicitly or implicitly providing them with extra information.
It’s best to keep your questions short and specific to your topic of interest.
- The average daily work commute in the US takes 54.2 minutes and costs $29 per day. Since 2020, working from home has saved many employees time and money. Do you favor flexible work-from-home policies even after it’s safe to return to offices?
- Experts agree that a well-balanced diet provides sufficient vitamins and minerals, and multivitamins and supplements are not necessary or effective. Do you agree or disagree that multivitamins are helpful for balanced nutrition?
Keep your questions focused
Ask about only one idea at a time and avoid double-barreled questions. Double-barreled questions ask about more than one item at a time, which can confuse respondents.
Strongly Agree
Agree
Undecided
Disagree
Strongly Disagree
This question could be difficult to answer for respondents who feel strongly about the right to clean drinking water but not high-speed internet. They might only answer about the topic they feel passionate about or provide a neutral answer instead – but neither of these options capture their true answers.
Instead, you should ask two separate questions to gauge respondents’ opinions.
Strongly Agree
Agree
Undecided
Disagree
Strongly Disagree
Do you agree or disagree that the government should be responsible for providing high-speed internet to everyone?
Strongly Agree
Agree
Undecided
Disagree
Strongly Disagree
Question order
You can organize the questions logically, with a clear progression from simple to complex. Alternatively, you can randomize the question order between respondents.
Logical flow
Using a logical flow to your question order means starting with simple questions, such as behavioral or opinion questions, and ending with more complex, sensitive, or controversial questions.
The question order that you use can significantly affect the responses by priming them in specific directions. Question order effects, or context effects, occur when earlier questions influence the responses to later questions, reducing the validity of your questionnaire.
While demographic questions are usually unaffected by order effects, questions about opinions and attitudes are more susceptible to them.
- How knowledgeable are you about Joe Biden’s executive orders in his first 100 days?
- Are you satisfied or dissatisfied with the way Joe Biden is managing the economy?
- Do you approve or disapprove of the way Joe Biden is handling his job as president?
For this reason, presidential approval ratings questions are asked at the start of surveys to measure opinions more accurately.
It’s important to minimize order effects because they can be a source of systematic error or bias in your study.
Randomization
Randomization involves presenting individual respondents with the same questionnaire but with different question orders.
When you use randomization, order effects will be minimized in your dataset. But a randomized order may also make it harder for respondents to process your questionnaire. Some questions may need more cognitive effort, while others are easier to answer, so a random order could require more time or mental capacity for respondents to switch between questions.
Step-by-step guide to design
Step 1: Define your goals and objectives
The first step of designing a questionnaire is determining your aims.
- What topics or experiences are you studying?
- What specifically do you want to find out?
- Is a self-report questionnaire an appropriate tool for investigating this topic?
Once you’ve specified your research aims, you can operationalize your variables of interest into questionnaire items. Operationalizing concepts means turning them from abstract ideas into concrete measurements. Every question needs to address a defined need and have a clear purpose.
Step 2: Use questions that are suitable for your sample
Create appropriate questions by taking the perspective of your respondents. Consider their language proficiency and available time and energy when designing your questionnaire.
- Are the respondents familiar with the language and terms used in your questions?
- Would any of the questions insult, confuse, or embarrass them?
- Do the response items for any closed-ended questions capture all possible answers?
- Are the response items mutually exclusive?
- Do the respondents have time to respond to open-ended questions?
Consider all possible options for responses to closed-ended questions. From a respondent’s perspective, a lack of response options reflecting their point of view or true answer may make them feel alienated or excluded. In turn, they’ll become disengaged or inattentive to the rest of the questionnaire.
Step 3: Decide on your questionnaire length and question order
Once you have your questions, make sure that the length and order of your questions are appropriate for your sample.
If respondents are not being incentivized or compensated, keep your questionnaire short and easy to answer. Otherwise, your sample may be biased with only highly motivated respondents completing the questionnaire.
Decide on your question order based on your aims and resources. Use a logical flow if your respondents have limited time or if you cannot randomize questions. Randomizing questions helps you avoid bias, but it can take more complex statistical analysis to interpret your data.
Step 4: Pretest your questionnaire
When you have a complete list of questions, you’ll need to pretest it to make sure what you’re asking is always clear and unambiguous. Pretesting helps you catch any errors or points of confusion before performing your study.
Ask friends, classmates, or members of your target audience to complete your questionnaire using the same method you’ll use for your research. Find out if any questions were particularly difficult to answer or if the directions were unclear or inconsistent, and make changes as necessary.
If you have the resources, running a pilot study will help you test the validity and reliability of your questionnaire. A pilot study is a practice run of the full study, and it includes sampling, data collection, and analysis. You can find out whether your procedures are unfeasible or susceptible to bias and make changes in time, but you can’t test a hypothesis with this type of study because it’s usually statistically underpowered.
टिप्पणियाँ
एक टिप्पणी भेजें