In Singapore, we have been discussing experimenter bias that is also known as researcher bias and it is a process where the researcher performing the research has his influence on the results and this occurs mostly when the researcher wants to manipulate a certain outcome. It is a psychological predisposition causing people to draw erroneous conclusions. These preconceived notions are mostly based on rules of thumb. For example, a researcher selects the subjects who are more likely to produce a desired outcome. Though, not all biases in a research are a conscious attempt on part of the experimenter. Sometimes the researcher fails to take into consideration all of the possible variables. Let’s discuss the possibility of researcher bias or response in the various research methods like survey research, focus groups, contextual enquiries, and retrospective think aloud methods.
The most significant bias that occurs in a survey research is that it’s not measurable. The researcher’s own perspective affects the design and analysis of the survey questions. Another kind of bias is the response bias, which is the inability to get a random sample of population. This type of survey mostly assumes that all those people who didn’t respond would give the same response as those who did. Market research normally ignores response bias because correcting such errors requires a lot of time and money.
Responses resulting from Focus Groups are subjective in nature and a researcher can yield to experimenter bias. He may unconsciously try to seek desired responses. Then there is another bias known as group thinking where participants tend to give consistent opinions in order to maintain group harmony. They may also infer a frame of reference from a previous answer or activity. Sometimes the participants may want to please the experimenter and tell him what he wanted to hear.
Ethnographic or Contextual Inquiry
Contextual inquiry focuses on tasks and processes that people are involved in. It helps discover what they do, why they do certain things, and how they do things. Experimenter demand effects can considerably affect the findings of an ethnographic study. A UX researcher must set his ethnographic inquiry objectives in the manner that could meet the constraints they are subject to. For instance, the research participants may perceive the presence of the UX researcher as an authority figure and demonstrate optimal behavior in spite of their actual behavior. The researcher may also demonstrate some social cues to his subjects that could further lead participants to display the researcher’s desired behavior.
Retrospective Think Aloud
The think aloud interview is usually conducted in two different ways: Concurrent Think Aloud (CTA) and Retrospective Think Aloud (RTA). CTA involves the subjects speaking at the time of performing a task whereas the RTA requires the subjects to think aloud after the completion of a task. Both methods have their own limitations and benefits however RTA in combination with eye tracking data is highly recommended in usability testing.
Retrospective Think Aloud is a usability research method where a user’s performance is verbalized when it is over. There is not much work done to validate the findings of this method of research but the protocol of RTA is often stimulated with the use of a visual reminder i.e. replay of already recorded video of the task and screen shots. This provides a more reliable account of what people concentrated on while completing tasks. This method also lessens the risk of fabrications. The complexity of the task doesn’t affect the validity of stimulated RTA findings. RTA and eye tracking methods when combined, yields extremely influential data in a unique way.
The eye tracking videos are retrospectively played back to the subjects. This data gives them a really powerful cue known as ‘cued retrospective recall’. Subjects are thus able to see their own eye movements and the decisions they made. It acts as a reminder and allows the subjects to recall their unconscious behavior and what motivated them to do what they did. So the experimenter bias is itself removed from this context since subjects first see what they did and then they judge from what they have seen and the entire process doesn’t require any communication between the subjects and the experimenter.
When research participants verbalize their thoughts retrospectively, they are possibly focusing on their memory alone. Moreover the visual stimulus works as a means to retrieve a more detailed answer from the participants and they are less likely to give a fabricated response since they themselves can see the evidence. Consequently the effect of response bias is reduced with the help of this method. In order to reduce social desirability effects, the participants are gently asked explanatory questions to de-stress the experience and subjects are given the confidence that the aim is to focus on the stimuli.
Research has shown that eye tracking combined with retrospective think aloud method actually delivers certain benefits especially when employed in usability testing. The findings also demonstrate that by using this method, subjects take less time to complete their tasks and produce a higher rate of the words and comments as compared with other methods.