Why negatively worded items

However, some recent research [pdf] Jim Lewis and I conducted found little evidence for these biases. We found response bias effects are at best small and outweighed by the real effects of miscoding and misinterpreting by users. In fact, of the most frequently used questionnaires to measure attitudes about usability all but one use a mix of positive and negative items.

By including a mix of both positive and negative items, respondents are forced to consider the question and hopefully provide a more meaningful response which should reduce these biases.

To our knowledge there is no research documenting the magnitude of acquiescence bias in general, or whether it specifically affects the measurement of attitudes toward usability. Does alternating item wording outweigh the real negatives of misinterpreting, mistaking and miscoding? To find out, we created an all positively worded version of the SUS and tested it against the original alternating SUS in a series of remote unmoderated usability studies.

We had users in the US attempt two representative tasks on one of seven websites third party automotive or primary financial services websites: Cars. The tasks included finding the best price for a new car, estimating the trade-in value of a used-car and finding information about mutual funds and minimum required investments.

At the end of the study users randomly completed either the standard or the positively worded SUS. There were between 15 and 17 users for each website and questionnaire type.

The mix of gender, age and education levels were not statistically different between groups. We found little evidence that the purported advantages of the alternating items outweighed the disadvantages.

Negatives Outweigh the Positives : There is little evidence that the purported advantages of including negative and positive items in usability questionnaires outweigh the disadvantages.

This finding certainly applies to the SUS when evaluating websites using remote unmoderated tests. It also likely applies to usability questionnaires with similar designs in unmoderated testing of any application. Future research with a similar experimental setup should be conducted using a moderated setting to confirm whether these findings also apply to tests when users are more closely monitored.

No Reason to stop using the original SUS just watch your coding! Researchers who use the standard SUS have no need to change to the all positive version provided that they verify the proper coding of scores for example by using the error-checking spreadsheet included in the SUSPackage.

All Positive SUS generates similar results: Researchers who do not have a current investment in the standard SUS can use the all positive version with confidence because respondents are less likely to make mistakes when responding, researchers are less likely to make errors in coding, and the scores will be similar to the standard SUS.

We only examined questionnaires that measure usability or system satisfaction. It could be that in other areas of behavioral research where the emphasis is on the individual e.

For more detail on the experiments and related research into this topic see the full paper [pdf] to be presented at CHI in May Skip to content. Are both positive and negative items necessary in questionnaires?

Jeff Sauro, PhD. April 26, This website was easy to use. It was difficult to find what I needed on this website. This is somewhat related to the acquiescent bias except respondents basically pick the most extreme rating and provide it to many or all items. Disadvantages to Alternating There is a dark side to alternating items. We are aware of at least three. There is evidence that this lowers the internal reliability, distorts the factor-structure and is more problematic in cross-cultural settings.

Methods: This was a cross-sectional study including 90 second-year nursing students. Seven pairs of positively and negatively worded items were evaluated for differences in response as well as agreement between the items. Results: Two pairs of items showed higher mean values if the item was negatively worded. Between-item agreement for positively worded item scores and their corresponding reverse-coded negatively worded item scores was poor-to-moderate intraclass correlation coefficient: 0.

A moderate agreement was found when testing all positively worded items against the recoded negatively worded items intraclass correlation coefficient: 0. Internal consistency was 0.