QUESTIONNAIRES

Written or electronic survey instruments comprised of a series of questions, designed to measure a specific item or set of items.
Questionnaires are an inexpensive way to gather data from a potentially large number of respondents. Often they are the only feasible way to reach a number of reviewers large enough to allow statistically analysis of the results. A well-designed questionnaire that is used effectively can gather information on both the overall performance of the test system as well as information on specific components of the system. If the questionnaire includes demographic questions on the participants, they can be used to correlate performance and satisfaction with the test system among different groups of users.
It is important to remember that a questionnaire should be viewed as a multi-stage process beginning with definition of the aspects to be examined and ending with interpretation of the results. Every step needs to be designed carefully because the final results are only as good as the weakest link in the questionnaire process. Although questionnaires may be cheap to administer compared to other data collection methods, they are every bit as expensive in terms of design time and interpretation.
The steps required to design and administer a questionnaire include:
Defining the Objectives of the survey
Determining the Sampling Group
Writing the Questionnaire
Administering the Questionnaire
Interpretation of the Results
This document will concentrate on how to formulate objectives and write the questionnaire. Before these steps are examined in detail, it is good to consider what questionnaires are good at measuring and when it is appropriate to use questionnaires.
What can questionnaires measure?
Questionnaires are quite flexible in what they can measure, however they are not equally suited to measuring all types of data. We can classify data in two ways, Subjective vs. Objective and Quantitative vs. Qualitative.
When a questionnaire is administered, the researchers control over the environment will be somewhat limited. This is why questionnaires are inexpensive to administer. This loss of control means the validity of the results are more reliant on the honesty of the respondent. Consequently, it is more difficult to claim complete objectivity with questionnaire data then with results of a tightly controlled lab test. For example, if a group of participants are asked on a questionnaire how long it took them to learn a particular function on a piece of software, it is likely that they will be biased towards themselves and answer, on average, with a lower than actual time. A more objective usability test of the same function with a similar group of participants may return a significantly higher learning time. More elaborate questionnaire design or administration may provide slightly better objective data, but the cost of such a questionnaire can be much higher and offset their economic advantage. In general, questionnaires are better suited to gathering reliable subjective measures, such as user satisfaction, of the system or interface in question.
Questions may be designed to gather either qualitative or quantitative data. By their very nature, quantitative questions are more exact then qualitative. For example, the word "easy" and "difficult" can mean radically different things to different people. Any question must be carefully crafted, but in particular questions that assess a qualitative measure must be phrased to avoid ambiguity. Qualitative questions may also require more thought on the part of the participant and may cause them to become bored with the questionnaire sooner. In general, we can say that questionnaires can measure both qualitative and quantitative data well, but that qualitative questions require more care in design, administration, and interpretation.
When to use a questionnaire?
There is no all encompassing rule for when to use a questionnaire. The choice will be made based on a variety of factors including the type of information to be gathered and the available resources for the experiment. A questionnaire should be considered in the following circumstances.
When resources and money are limited. A Questionnaire can be quite inexpensive to administer. Although preparation may be costly, any data collection scheme will have similar preparation expenses. The administration cost per person of a questionnaire can be as low as postage and a few photocopies. Time is also an important resource that questionnaires can maximize. If a questionnaire is self-administering, such as a e-mail questionnaire, potentially several thousand people could respond in a few days. It would be impossible to get a similar number of usability tests completed in the same short time.
When it is necessary to protect the privacy of the participants. Questionnaires are easy to administer confidentially. Often confidentiality is the necessary to ensure participants will respond honestly if at all. Examples of such cases would include studies that need to ask embarrassing questions about private or personal behavior.
When corroborating other findings. In studies that have resources to pursue other data collection strategies, questionnaires can be a useful confirmation tools. More costly schemes may turn up interesting trends, but occasionally there will not be resources to run these other tests on large enough participant groups to make the results statistically significant. A follow-up large scale questionnaire may be necessary to corroborate these earlier results.

I. Defining the Objectives of the Survey

The importance of well-defined objectives can not be over emphasized. A questionnaire that is written without a clear goal and purpose is inevitably going to overlook important issues and waste participants' time by asking useless questions. The questionnaire may lack a logical flow and thereby cause the participant to lose interest. Consequential, what useful data you may have collected could be further compromised. The problems of a poorly defined questionnaire do not end here, but continue on to the analysis stage. It is difficult to imagine identifying a problem and its cause, let alone its solution, from responses to broad and generalizing questions. In other words, how would it be possible to reach insightful conclusions if one didn't actually know what they had been looking for or planning to observe.
A objective such as "to identify points of user dissatisfaction with the interface and how these negatively affect the software's performance" may sound clear and to the point, but it is not. The questionnaire designer must clarify what is meant by user dissatisfaction. Is this dissatisfaction with the learning of the software, the power of the software, of the ease of learning the software? Is it important for the users to learn the software quickly if they learn it well? What is meant by the software's performance? How accurate must the measurements be? All of these issues must be narrowed and focused before a single question is formulated. A good rule of thumb is that if you are finding it difficult to write the questions, then you haven't spent enough time defining the objectives of the questionnaire. Go back and do this step again. The questions should follow quite naturally from the objectives.

II. Writing the Questionnaire

At this point, we assume that we have already decided what kind of data we are to measure, formulated the objectives of the investigation, and decided on a participant group. Now we must compose our questions.
If the preceding steps have been faithfully executed, most of the questions will be on obvious topics. Most questionnaires, however, also gather demographic data on the participants. This is used to correlate response sets between different groups of people. It is important to see whether responses are consistent across groups. For example, if one group of participants is noticeably less satisfied with the test interface, it is likely that the interface was designed without fair consideration of this group's specific needs. This may signify the need for fundamental redesign of the interface. In addition, certain questions simply may only be applicable to certain kinds of users. For example, if one is asking the participants whether they find the new tutorial helpful, we do not want to include in our final tally the responses of experienced users who learned the system with an older tutorial. There is no accurate way to filter out these responses without simply asking the users when they learned the interface.
Typically, demographic data is collected at the beginning of the questionnaire, but such questions could be located anywhere or even scattered throughout the questionnaire. One obvious argument in favor of the beginning of the questionnaire is that normally background questions are easier to answer and can ease the respondent into the questionnaire. One does not want to put off the participant by jumping in to the most difficult questions. We are all familiar with such kinds of questions.
It is important to ask only those background questions that are necessary. Do not ask income of the respondent unless there is at least some rational for suspecting a variance across income levels. There is often only a fine line between background and personal information. You do not want to cross over in to the personal realm unless absolutely necessary. If you need to solicit personal information, phrase your questions as unobtrusively as possible to avoid ruffling your participants and causing them to answer less than truthfully.
What kind of questions do we ask?
In general, there are two types of questions one will ask, open format or closed format.
Open format questions are those that ask for unprompted opinions. In other words, there are no predetermined set of responses, and the participant is free to answer however he chooses. Open format questions are good for soliciting subjective data or when the range of responses is not tightly defined. An obvious advantage is that the variety of responses should be wider and more truly reflect the opinions of the respondents. This increases the likelihood of you receiving unexpected and insightful suggestions, for it is impossible to predict the full range of opinion. It is common for a questionnaire to end with and open format question asking the respondent for her unabashed ideas for changes or improvements.
Open format questions have several disadvantages. First, their very nature requires them to be read individually. There is no way to automatically tabulate or perform statistical analysis on them. This is obviously more costly in both time and money, and may not be practical for lower budget or time sensitive evaluations. They are also open to the influence of the reader, for no two people will interpret an answer in precisely the same way. This conflict can be eliminated by using a single reader, but a large number of responses can make this impossible. Finally, open format questions require more thought and time on the part of the respondent. Whenever more is asked of the respondent, the chance of tiring or boring the respondent increases.
Closed format questions usually take the form of a multiple-choice question. They are easy for the respondent, give
There is no clear consensus on the number of options that should be given in an closed format question. Obviously, there needs to be sufficient choices to fully cover the range of answers but not so many that the distinction between them becomes blurred. Usually this translates into five to ten possible answers per questions. For questions that measure a single variable or opinion, such as ease of use or liability, over a complete range (easy to difficult, like to dislike), conventional wisdom says that there should be an odd number of alternatives. This allows a neutral or no opinion response. Other schools of thought contend that an even number of choices is best because it forces the respondent to get off the fence. This may induce the some inaccuracies for often the respondent may actually have no opinion. However, it is equally arguable that the neutral answer is over utilized, especially by bored questionnaire takers. For larger questionnaires that test opinions on a very large number of items, such as a music test, it may be best to use an even number of choices to prevent large numbers of no-thought neutral answers.
Closed format questions offer many advantages in time and money. By restricting the answer set, it is easy to calculate percentages and other hard statistical data over the whole group or over any subgroup of participants. Modern scanners and computers make it possible to administer, tabulate, and perform preliminary analysis in a matter of days. Closed format questions also make it easier to track opinion over time by administering the same questionnaire to different but similar participant groups at regular intervals. Finally closed format questions allow the researcher to filter out useless or extreme answers that might occur in an open format question.
Whether your questions are open or closed format, there are several points that must by considered when writing and interpreting questionnaires:
Clarity: This is probably the area that causes the greatest source of mistakes in questionnaires. Questions must be clear, succinct, and unambiguous. The goal is to eliminate the chance that the question will mean different things to different people. If the designers fails to do this, then essentially participants will be answering different questions.
To this end, it is best to phrase your questions empirically if possible and to avoid the use of necessary adjectives. For example, it asking a question about frequency, rather than supplying choices that are open to interpretation such as:
Very Often
Often
Sometimes
Rarely
Never

It is better to quantify the choices, such as:
Every Day or More
2-6 Times a Week
About Once a Week
About Once a Month
Never

There are other more subtle aspects to consider such as language and culture. Avoid the use of colloquial or ethnic expressions that might not be equally used by all participants. Technical terms that assume a certain background should also be avoided.

Leading Questions: A leading question is one that forces or implies a certain type of answer. It is easy to make this mistake not in the question, but in the choice of answers. A closed format question must supply answers that not only cover the whole range of responses, but that are also equally distributed throughout the range. All answers should be equally likely. An obvious, nearly comical, example would be a question that supplied these answer choices:
Superb
Excellent
Great
Good
Fair
Not so Great

A less blatant example would be a Yes/No question that asked:
Is this the best CAD interface you have every used?

In this case, even if the participant loved the interface, but had an favorite that was preferred, she would be forced to answer No. Clearly, the negative response covers too wide a range of opinions. A better way would be to ask the same question but supply the following choices:
Totally Agree
Partially Agree
Neither Agree or Disagree
Partially Disagree
Totally Agree

This example is also poor in the way it asks the question. It's choice of words makes it a leading question and a good example for the next section on phrasing.

Phrasing: Most adjectives, verbs, and nouns in English have either a positive or negative connotation. Two words may have equivalent meaning, yet one may be a compliment and the other an insult. Consider the two words "child-like" and "childish", which have virtually identical meaning. Child-like is an affectionate term that can be applied to both men and women, and young and old, yet no one wishes to be thought of as childish.
In the above example of "Is this the best CAD interface you have every used?" clearly "best" has strong overtones that deny the participant an objective environment to consider the interface. The signal sent the reader is that the designers surely think it is the best interface, and so should everyone else. Though this may seem like an extreme example, this kind of superlative question is common practice.
A more subtle, but no less troublesome, example can be made with verbs that have neither strong negative or positive overtones. Consider the following two questions:
Do you agree with the Governor's plan to oppose increased development of wetlands?
Do you agree with the Governor's plan to support curtailed development of wetlands?
They both ask the same thing, but will likely produce different data. One asks in a positive way, and the other in a negative. It is impossible to predict how the outcomes will vary, so one method to counter this is to be aware of different ways to word questions and provide a mix in your questionnaire. If the participant pool is very large, several versions may be prepared and distributed to cancel out these effects.
Embarrassing Questions: Embarrassing questions dealing with personal or private matters should be avoided. Your data is only as good as the trust and care that your respondents give you. If you make them feel uncomfortable, you will lose their trust. Do not ask embarrassing questions.

Hypothetical Questions Hypothetical are based, at best, on conjecture and, at worst, on fantasy. I simple question such as:
If you were governor, what would you do to stop crime?
This forces the respondent to give thought to something he may have never considered. This does not produce clear and consistent data representing real opinion. Do not ask hypothetical questions.

Prestige Bias: Prestige bias is the tendency for respondents to answer in a way that make them feel better. People may not lie directly, but may try to put a better light on themselves. For example, it is not uncommon for people to respond to a political opinion poll by saying they support Samaritan social programs, such as food stamps, but then go on to vote for candidates who oppose those very programs. Data from other questions, such as those that ask how long it takes to learn an interface, must be viewed with a little skepticism. People tend to say they are faster learners than they are.
There is little that can be done to prevent prestige bias. Sometimes there just is no way to phrase a question so that all the answers are noble. The best means to deal with prestige bias is to make the questionnaire as private as possible. Telephone interviews are better than person-to-person interviews, and written questionnaires mailed to participants are even better still. The farther away the critical eye of the researcher is, the more honest the answers.

Now What?
Now that you've completed you questionnaire, you are still not ready to send it out. Just like any manufactured product, your questionnaire needs to go through quality testing. The major hurdle in questionnaire design is making it clear and understandable to all. Though you have taken great care to be clear and concise, it is still unreasonable to think that any one person can anticipate all the potential problems. Just as a usability test observes a test user with the actual interface, you must observe a few test questionnaire takers. You will then review the questionnaire with the test takers and discuss all points that were in any way confusing and work together to solve the problems. You will then produce a new questionnaire. It is possible that this step may need to be repeated more than once depending on resources and the need for accuracy.
Conclusions
Questionnaire design is a long process that demands careful attention. A questionnaire is a powerful evaluation tool and should not be taken lightly. Design begins with an understanding of the capabilities of a questionnaire and how they can help your research. If it is determined that a questionnaire is to be used, the greatest care goes into the planning of the objectives. Questionnaires are like any scientific experiment. One does not collect data and then see if they found something interesting. One forms a hypothesis and an experiment that will help prove or disprove the hypothesis.
Questionnaires are versatile, allowing the collection of both subjective and objective data through the use of open or closed format questions. Modern computers have only made the task of collecting and extracting valuable material more efficient. However, a questionnaire is only as good as the questions it contains. There are many guidelines that must be met before you questionnaire can be considered a sound research tool. The majority deal with making the questionnaire understandable and free of bias. Mindful review and testing is necessary to weed out minor mistakes that can cause great changes in meaning and interpretation. When these guidelines are followed, the questionnaire becomes a powerful and economic evaluation tool.

The key to designing effective interviews: structure
Although many different approaches can be incorporated into the interview, effective interviews have three characteristics. When these characteristics are present, the interview can be referred to as a "structured" interview.
"A structured interview may be defined as a series of job-related questions with predetermined answers that are consistently applied across all interviews for a particular job." Pursell, Campion, & Gaylord, Personnel Journal, 1980.
A structured selection interview involves more than just asking the same questions of all candidates. In order to be called "structured", an interview must have three characteristics:
Questions are rooted in the duties and responsibilities of the position.
Questions are developed systematically to tap specific qualifications.
Answers are evaluated against established criteria.
Interviews developed to include the above characteristics are more effective in identifying competent candidates.
Designing a structured interview
Some interviewers ask questions that are designed to get the candidates to "reveal themselves". These questions are usually formulated without any particular qualification in mind. Candidates' answers to such questions are difficult to interpret, and the information elicited is rarely relevant to the qualifications being assessed.
Avoid fishing expeditions
For example, unless related to the qualifications being assessed, the following questions would not be appropriate:
"What kind of people annoy you the most?"
"Describe the best person who ever worked for or with you."
"What did you like best about your last position?"
A better alternative is to develop questions that will provide information relevant to the qualification(s) being assessed. The three steps described below outline how you can develop these questions.
· Step #1: Consider the kind of actions, responses and behaviours that are relevant to the qualification(s) being assessed.
· Step #2: Develop questions that will elicit relevant information.
· Step #3: Evaluate candidates' answers.
These three steps apply to the development of interview questions for the assessment of various qualifications.
Step #1: Consider the kind of actions, responses, and behaviours that are relevant to the qualification(s) being assessed.
Some groundwork needs to be done before the interview questions can be written. The information elicited at this step will be used to develop the questions and the rating schemes for the candidates' answers.
Ask yourself the following questions about the qualification(s) being assessed:
In what way is the possession of the qualification demonstrated? What is it that employees who are strong on the qualification do that employees who are weak on that qualification don't or can't do?
In what situations is the qualification demonstrated? What aspects of these situations are relevant to the qualification?
What is the impact of different courses of action? What makes a particular action or response effective? What makes it ineffective?
An example
Let's consider the "ability to plan". What does it mean to plan in the context of the position being staffed? You would want to know what, specifically, good planners would do and what bad planners would do. Then, you would want to know about the kinds of activities which are planned. You would want to know about their complexity, their time constraints and other obstacles and challenges that are usually encountered in planning. Finally, you want to know about the impact of different planning actions, or behaviours. Are different approaches to planning equally effective?
Step #2: Develop questions that will elicit relevant information
Develop questions specifically for each qualification being assessed. In this way, you'll ensure that relevant information is elicited for all qualifications to be assessed. It's true that some questions may provide information about more than one qualification. Nonetheless, it's a good idea to develop questions with a particular qualification in mind.
Three kinds of interview questions have evolved:
the job knowledge question,
the situational question and
the behavioural question.
Let's look at each in turn.
A. The job knowledge question
These questions most often deal with technical or basic knowledge required to perform the duties and responsibilities of the position; they can also tap more complex aspects of knowledge. This kind of question could just as well be presented in written form, say, on a paper-and-pencil instrument. The interview context, however, provides the advantage of being able to ask follow-up questions and probe more deeply on the basis of the answers given.
Here are some examples of knowledge questions:
"What are the steps involved in the investigation of this kind of complaint?"
" What can cause this kind of malfunction in this machine? "
In developing job knowledge questions, consider the following:
To determine the content to be covered, break down the knowledge area into sub-areas. Identify the content areas, or the specific facts, concepts, techniques, or regulations that are crucial. Take into account the relative importance of each of these knowledge areas in deciding how many questions you will require.
Consider the desired level of competence in deciding how difficult to make the questions. Questions that are too difficult or too easy are not really informative.
Make sure that your questions are clear and unambiguous. In fact, it's a good idea to have a colleague or incumbent review the questions.
Avoid "double barreled" questions. Don't ask for more than one kind of information in the same question as this will only confuse candidates.
Ask follow-up questions, but only to clarify ambiguous answers, not to help the candidate figure out the answer.
B. The situational question
These questions describe a hypothetical job-related situation that focusses on a relevant qualification. These questions require the candidates to reply with what they would do in a given situation.
The situations described in these questions are often quite similar to the scenarios used in work samples and simulations. The difference is mainly that, in the interview, the candidates will be asked to describe what they would do; whereas, in the work sample or simulation, candidates would be asked to actually perform the task. The situational question is based on the notion that what people say they would do is related to what, in fact, they actually will do in the situation.
Situational questions are developed from the kind of incidents that were identified in step #1. Consider incidents in which there have been, or would be, clear differences between the actions of good and bad performers. Then, turn these incidents into questions by creating descriptions of situations which require some immediate action. Add realistic detail to the situations. Create as many questions as will be needed to provide sufficient information to assess the qualification.
Here is an example (Latham, Saari, Pursell and Campion, Journal of Applied Psychology, 1980) of a situational question which was developed to elicit information about "judgement": "You have received a call from an angry client. He has been trying to reach your supervisor to obtain a report recently prepared by your division, however, his calls have not been returned. He asks you to send him a copy of the report. What would you do?"
The answers to situational questions are evaluated against established criteria. The development of these criteria will be discussed at step #3.
In developing situational questions, consider the following points:
Your questions should not be too obvious or transparent. To have a valuable tool, candidates should not be able to "see through the questions".
Try out situational questions on colleagues, or job incumbents, to see if they are clearly understandable and whether they elicit the types of answers expected.
During the interview, read the questions to each candidate so that no details are forgotten. Stick to the questions as written.
C. The behavioural question
The behavioural question is based on the observation that the best predictor of future performance is past performance. For example, a competent technician in the past will probably continue to be a competent technician in the future.
Behavioural questions are aimed at obtaining information from a candidate's past that's relevant to a given qualification or qualifications. In many ways, the objective of behavioural questions is similar to that of the reference check. The difference is that it's the candidates themselves who are the source of information.
Behavioural questions are designed to gather information relevant to the qualification(s) being assessed by having candidates identify what they actually did in similar situations, tasks, or contexts in the past. When you ask behavioural questions, you want to get at:
the particulars of the situation, task, problems, or context;
the actions that the candidate took, or failed to take, in response to the above; and
the impact of the actions taken, or not taken.
Behavioural questions consist of a question that describes a scenario and follow-up questions. The first question asks the candidate to describe a particular situation that's relevant to the qualification being assessed.
For example, if you need to obtain information about stress tolerance, you could ask:
"Can you tell me about some recent situations where you have had to deal with more than the usual amount of stress?"
On the basis of the information you receive, follow-up questions are then asked to flesh out the particular situation, clarify the candidate's actions and determine the impact.
As follow-up questions, you could ask:
"How did you cope with this situation?"
"And how did your reaction affect others?"
In developing behavioural interview questions, the following pointers are useful:
Balance the interview towards the positive. Give candidates the opportunity to talk about successes; they may then be more forthcoming about their failures.
To get at behavioural incidents where a candidate may not have done so well, begin with a rationale for the question. For example, using stress tolerance again, you could ask: "we've all had situations where stress has gotten to us; can you describe one such incident that happened to you?"
Make sure that you get the candidates to identify actions rather than feelings or opinions.
Collect at least a few incidents for each qualification assessed. Continue collecting information until you've exhausted the topic or until you're satisfied that you have enough information to evaluate the candidate on the relevant qualification.
Combine the different types of interview questions
Although each of the above types of interview questions can be effective in gathering relevant information, it's sometimes even more effective to combine these different types in the assessment of a given qualification. For instance, both situational and behavioural questions can be used to elicit information relevant to the same qualification.
During the interview
While conducting the interview, consider the following:
Begin by establishing rapport. Just because an interview is structured and the questions have been developed beforehand does not mean that the interview cannot be conducted in a natural manner.
Follow your plan. Whatever approach you use, write out your questions and use this plan to keep on track.
Explain the format of the interview to candidate.