UX Metrics: Evaluating Usability with the SUM Metric

Written by Anderson Batista & Elisa Zanoni

Usability stands as a pivotal aspect in UX design and digital system development, directly influencing user experience and product effectiveness. In this context, the Single Usability Metric (SUM) emerges as a valuable tool to assess and measure the ease of use of digital systems, interfaces, and products.

In this article, we’ll delve into how Sensorama Design leverages SUM to provide metric-based insights into usability testing for digital product design.

UX Metric for Usability: Exploring SUM Metric for Digital Product Evaluation

A Bit of Context

Developed by Jeff Sauro in 2005, the SUM metric aims to deliver a single numerical measure that synthesizes various aspects of user experience, such as efficiency, effectiveness, satisfaction, and ease of learning. Through this methodology, we streamline the usability evaluation process, enabling a quick and straightforward understanding of the system’s performance.

Application of SUM Metric in Product Design

This methodology relies on collecting data about user interaction with the system through usability tests, questionnaires, or other research techniques. These data are then processed and analyzed to calculate the SUM score, representing the overall usability quality of the system on a specific scale, thereby assessing how the system performs.

It’s important to note that SUM usage should be directed towards measuring a limited number of specific tasks within the system, not for measuring the entire platform. Focusing on a few tasks allows for precise and targeted usability analysis, ensuring relevant insights for specific improvements.

Combined with detailed qualitative analysis, the evaluation of digital product UX through a SUM test can be further enriched. Here at Sensorama Design, we often connect SUM’s quantitative data with qualitative insights obtained through user observations, interviews, or feedback analysis, enabling a deeper and contextualized understanding of user experience.

Qualitative analysis goes beyond SUM numbers, revealing:

• Subtle aspects of user behavior not captured by numbers.
• Standardized behaviors indicating how users interact with the system.
• Individual desires and expectations varying among users.

Practical examples of the qualitative analysis impact:

• Users are facing frustrations due to the complexity of screen X. Qualitative analysis identified the cause: a confusing interface and complex navigation.
• Low task completion rate: We found that users don’t understand the instructions.
• Areas of the system with high usage: Qualitative analysis confirms the relevance of these areas and directs development focus.

How We Utilize the SUM in Usability Testing

To exemplify how Sensorama utilizes the SUM metric to evaluate user experience, let’s discuss a project where we were tasked with comparing usability between the client’s product and its competitors.

Our first challenge was mapping the flows of competing platforms. To achieve this, we conducted guided navigation with users, allowing us to collect the necessary screens to build the prototype for testing.

The 3 stages of usability testing to capture the SUM metric:

1. Navigation and task execution in the prototype: We limited the number to three tasks to explore in our script.
2. Completion of ASQ and CCS questionnaires on a Likert scale: Used to understand the user’s perception of ease of use, satisfaction, and time spent on each task.
3. Post-test micro-interview: A moment where we ask in-depth questions to gain understanding beyond the tasks.

Calculating the SUM Score

After completing usability tests, we proceed with the Keystroke Level Model (KLM) method analysis to deepen the evaluation of user experience. This method allows us to predict what the ideal execution time of a task would be performed without errors or deviations. It is through KLM that we obtain the essential parameters to calculate the SUM score. Basically speaking, KLM would be the ideal time, while SUM is the actual time executed versus the ideal time.

KLM should be fed with the following data:

Subsequently, we extract all numerical deviation data, task execution time, completion rate, and satisfaction scores to feed our SUM calculator, cross-referencing this data with KLM spreadsheet data.

After obtaining the SUM score, we develop the report with the consolidation of all quantitative data and merge it with qualitative analysis we conducted through interviews and observations during testing. By combining this data, we gain a holistic view of user experience, allowing us to identify:

• Recurring user behaviors.
• System points that needed improvement.
• Insights into user needs and expectations.
• Opportunities for new features that could add value to the business.

As a conclusion, the SUM test is a valuable tool for:

• Gaining insights into user experience.
• Identifying usability issues.
• Evaluating product performance.
• Prioritizing system improvements.
• Making data-driven decisions.

At Sensorama Design, we apply the SUM metric to bring depth to digital product UX evaluations. As a result, our recommendations are based on specific data and metrics of digital product design. Consequently, we ensure that the solutions we design yield results from investments in UX design and innovation projects.