How to minimize human bias in design testing (using benchmarks to track UI progress)
UI Design happens at the intersection of business needs and user needs. It’s easy to see if business needs have been met -- a simple checklist will work.
But user needs are different, and require reliable tools to measure. For decades, design professionals have used the System Usability Scale (SUS) to collect a single measurement that expresses the usability and the learnability of a given interface. This technology-independent measurement has become an industry standard, referenced in over 600 publications.
Most people respond positively when asked “How are you?” in casual conversation. Humans generally strive for social acceptability, and this can bias the results of usability tests and user satisfaction studies.
SUS is designed to control this bias through using a rating system rather than direct questions. It’s theoretically easier for a user to provide a response that could be construed as being “negative” if the user is responding to a statement rather than a question.
Study participants are asked to respond to ten statements using a five-point scale. For example, the first statement is “I think that I would like to use this system frequently” and the possible responses are:
1. Strongly Disagree
5. Strongly Agree
The second statement is a “negative” statement, “I found the system unnecessarily complex”, with the same set of possible responses. The positive/negative pattern continues throughout.
The numerical responses are then normalized to produce a range of zero to four, with four being the most positive. Since there are ten statements, the highest possible total is 40. The total is multiplied by 2.5 to bring the possible total to 100. This final number is used as a benchmark.
What is a “good” SUS score?
Well, anything higher than your benchmark is good, but the average SUS score derived from over 500 studies is 68, so a score higher than 68 would mean that your interface has a better than average SUS score. A score of 80 means that users are likely to recommend the product being studied to a friend or colleague.
For a simpler measure of user satisfaction, UXD at ACI uses a Net Promoter Score. This is similar to SUS, but it uses only a single question: “How likely are you to recommend this product to a friend or colleague?” with a response scale ranging from 0, “Not likely at all” to 10, “Highly Likely”.
The scoring is also simple: Any score from 0 – 6 is labeled a “detractor” while a 9 or a 10 is a “Promoter”. Detractors are subtracted from Promoters to produce the Net Promoter Score. Since this method is widely used in many industries, benchmarks are available, making it possible to compare your scores with other companies in your industry.
Both SUS and NPS are useful throughout the life of a project. They are used to justify design work and to support design decisions through scientifically gathered data directly from users.
Related Blog Posts
Why Banks Must Democratize Machine Learning for Fraud Prevention and Payments Intelligence
Banks are already actively on the path to digital transformation, considering new technologies, new customer experiences and new business models. A critical piece of this digital transformation centers on better understanding the wealth of data within the banks’ systems and mining it for improved customer insight. In the New Payments Ecosystem, data is as valuable to the bank and its customers as the deposits held in their accounts, and it should be protected, and leveraged for the benefit of the customer.
Payments and Fraud: The Paradox Twins
Digital commerce through web and mobile is where merchants predominantly experience shopper growth today. This has become a hugely important domain for their focus. It offers a means for international growth, new market penetration and a way to engage with shopper-hungry Millennials in their culture. Merchants frequently adopt a Digital-First, eCommerce-First or Mobile-First strategy to ensure full corporate buy-in to this strategy.
Building Trust in Open Banking with Behavioral Biometrics and Machine Learning
Strategies for fraud prevention in payments are having to evolve quickly, as new technologies emerge and digitalization of the banking ecosystem continues at pace. I spoke with Giselle Lindley, Principal Financial Crime Consultant at ACI Worldwide and Tim Dalgleish, Head of Threat Analytics, Asia Pacific at BioCatch to understand how financial institutions can use payments intelligence to build trust in this challenging environment.
Knowing New Customers – And How Shared Data Helps in Fighting Fraud
As the eCommerce industry continues its rapid growth, the lines between physical and digital shopping are becoming increasingly blurred. These changes are creating a number of challenges for merchants, not least around customer visibility and fraud prevention.
Reducing Fraud and Improving Customer Experience with Machine Learning
Julie Conroy is research director for Aite Group’s Retail Banking practice and covers fraud, data security, anti-money laundering, and compliance issues. Recently, Julie teamed up with ACI’s Marc Trepanier for a webinar, Key Trends in Payments Intelligence – Machine Learning for Fraud Prevention. I sat down with Julie to get her take on the topic.
Account Takeover: The Burgeoning Fraud Profession
Data security features frequently in the news headlines these days, in some form or other. Today’s consumers are increasingly protective of their personal data and concerned about its use. Yet an increasing number of apps and websites allow new customers to use external logins (from social media accounts for instance) to set up new accounts.
Hitting Back Against Telco Fraud with Positive Profiling
Fraudulent activity costs telecommunications companies billions of dollars every year – and it hits these telcos in many different forms. According to reports from CFCA (Communications Fraud Control Association), over USD $12 billion is lost annually purely to consumer-based fraud attacks such as subscription fraud and account takeover. These are fast becoming the biggest growth areas for fraudsters in the telco industry.
How Fresh, First-Hand Research Leads to Interfaces That Make Users Feel Understood
In 1890, my great-great-grandfather fled Czarist Ukraine for a new home in London. Amongst the few possessions that he brought with him was a book that had been given to him in 1860 by his grandfather.
Cybersecurity: Risks, Controls and What to Expect in 2019
The world of Cybersecurity has brought about several subtle changes in 2018. For example, malware and targeted 'Spear Phishing' were on the rise, while the focus on protecting the perimeter has begun to take a back seat to hardening internal controls. As we enter 2019, the changing threat landscape is certain to result in a barrage of additional considerations in how we protect data and systems.
‘Soup To Nuts’ – A Multi-Layered Fraud Menu for the Holiday Season
The holiday shopping season is well underway, with Black Friday now behind us and many retailers around the world braced for higher levels of eCommerce fraud, from Cyber Monday all the way though until Christmas.