The Standardized User Experience Percentile Rank Questionnaire
The SUPR-Q is a Rating Scale to Measure perceptions of Usability, Trust, Credibility, Appearance and Loyalty for WebsitesHow usable is your website? Will your visitors think the website is credible and trustworthy? While it is important to continuously make improvements in your user experience, it’s good to know how well your website stacks up against other websites on the important aspects of usability, trust, credibility and loyalty. Are users able to find what they’re looking for? Do they trust your website? Do they feel comfortable providing personal information? Will they return and become loyal customers?
How usable is my website, what should I fix?It's important to have a mix of both qualitative comments from your users and quantitative benchmarks to know how much effort you should invest in improving your website. Interpreting open ended feedback is an important step in identifying what to improve on your website. One problem with open-ended comments is that they can be difficult to put in context.
A SUPR-Q score will tell you how usable and easy to use visitors think your website is. If your SUPR-Q score is low relative to your peers, then start fixing the things identified in your comments and consider commissioning a larger usability study to generate more substantial design improvements. If your SUPR-Q score is high relative to your peers then at least you know you’re doing something right! You should constantly be updating your website based on feedback, but high SUPR-Q scores suggest major usability improvements would be a lower priority.
What exactly is a SUPR-Q score?The SUPR-Q score is made up of a global score which is a percentile ranking based on four factors:
- Trust & Credibility
The SUPR-Q was developed through a process of psychometric validation. Over 75 candidate questions were tested on thousands of users across hundreds of websites to arrive at the 8 which provide the most reliable, sensitive and valid picture of the attitudes which shape the success of a website.
The SUPR-Q is an 8 item standardized questionnaire which generates a normalized score. The scoring system is based on the relative ranking from a database of over 150 websites and thousands of users.
What are the SUPR-Q Items?
- This website is easy to use.
- It is easy to navigate within the website.
- The information on the website is credible.
- The information on the website is trustworthy.
- I feel comfortable purchasing from this website. (Alternate for eCommerce websites)
- I feel confident conducting business with this website. (Alternate eCommerce websites)
- How likely are you to recommend this website to a friend or colleague?
- I will likely visit this website in the future.
- I found the website to be attractive.
- The website has a clean and simple presentation.
Credibility (Trust, Value & Comfort)
What are the response options?There are five response options for 7 of the questions, and can be displayed in a grid like the example below.
You can randomize the order of the responses as long as you keep track of the item number when scoring the responses. Amazon is used as an example website; substitute the name of your website.
The final question “How likely are you to recommend this website to a friend or colleague?” has 11 response options and can be displayed like the example below. It needs to start with a 0 and end with a 10 and have the end-points labeled and the middle option labeled. It also provides the commonly used "Net Promoter Score".
How do you score the responses?Scoring is simple. Average the responses for the first 7 questions and add this to 1/2 of the score for the Likelihood to Recommend question. This is your SUPR-Q score. This can then be compared to the industry benchmarks.
How is the SUPR-Q score generated?The SUPR-Q score is expressed as a percentile (hence percentile rank), so a SUPR-Q score of 75% would mean the global score is higher than 75% of all websites tested in the database. The same percentile ranking procedure is then done for all of the four factors and by each of the 8 questions. This can also be done within a specific industry. It is common to have a range of rankings for each of the four subscales. Some websites might score high in usability but low with respect to loyalty.
How is SUPR-Q like SUS?
The System Usability Scale (SUS) is a popular 10 item questionnaire which measures the perceptions of usability. The SUS was developed over 20 years ago and provides a reliable estimate of usability. Its questions are generic enough to allow it to be used on a wide variety of software and hardware applications. However, as a result of this generic quality, some questions don’t fit well with websites (e.g. needing the support of a technical person). The result is that while SUS still provides a reliable picture of users’ attitudes toward usability, it is limited to only perceptions of usability and excludes other important factors such as trust, credibility and loyalty.
The SUPR-Q overall score has a strong correlation with a SUS score, r = .87, p < .001, meaning just four questions account for 87 percent of the variation in SUS (.87 squared). These questions are a good substitute for SUS on websites.
How it Works
The SUPR-Q can be used as part of a usability test or placed in a conspicuous location on your website as an ongoing assessment tool. For a usability test, have your users answer the 8 questions after they’ve completed realistic task scenarios. Your scored results can then be compared to hundreds of other websites.
Who Uses the SUPR-Q?
Current customers include:
Is the SUPR-Q Reliable?
The SUPR-Q has been used on over 200 websites and by over 4500 users from 13 countries (90% US & Canada). As part of the validation of the instrument, the internal reliability was measured using Cronbach Alpha. The global score and each factor exhibits strong internal reliability as can be seen in Figure 1 below.
|Trust, Value & Comfort||0.85||0.73|
Is the SUPR-Q Valid?
To ensure the SUPR-Q provides a valid measure of usability, credibility, trust and appearance it was compared to other known instruments which measure similar constructs. The SUPR-Q global score and usability score have a very strong correlation with the System Usability Scale (SUS) r > .88 p <.01. Additionally, 108 users across 8 different websites completed both the SUPR-Q and the WAMMI questionnaire. The usability factor and global SUPR-Q score also have strong correlations (r > .88, p <.01) with the average responses from the 20 WAMMI questions. A combination of just two questions from the SUPR-Q account for 88% of the variability in the 20 WAMMI questions. The SUPR-Q was able to discriminate well against websites which have been known to have poor usability, trust, loyalty and appearance against others which score highly on these attributes.
How is the SUPR-Q different than other standardized questionnaires?
A good standardized questionnaire provides both a valid and reliable instrument as well as a relative ranking based on a dataset. The SUS is a good instrument but a raw SUS score is difficult to interpret without knowing how other websites score (e.g. is a 70 good?). Questionnaires like the WAMMI and SUPR-Q both provide relative rankings expressed as percentages. Therefore a percentile score of 50 is an average score, or roughly half the websites have received better scores and half received worse scores. One problem with the WAMMI is that you don’t know what websites make up the scoring system. In any large dataset there will be some websites that are good comparisons and others which are irrelevant. What makes SUPR-Q different is that you know what websites are in the dataset. What’s more, you can see how well your SUPR-Q scores compares to a subset of up to 10 other websites or compared to the industry.
What websites make up the SUPR-Q database?Data from over 200 websites make up the SUPR-Q database. Most websites are US based and were tested on users from North America. Roughly 10% of the users come from countries outside of the US & Canada including: UK, Australia, Romania, India, Japan, Mexico and Hong Kong. A partial list of the websites in the database is shown in the table below. They include large and small ecommerce websites, mobile phone carriers, travel websites and governments.
3rd Party Travel
3rd Party Automotive
News & Information