TL;DR: The UMUX is a short, 4 items questionnaire that correlates to the System Usability Scale. It is a valid and reliable way of measuring perceived usability. Its short form makes it a good asset in a researcher’s toolbox of UX Metrics that can be administered in-the-wild or on user testings.
There are a lot of definitions of usability; the ISO 9241-11 defines usability as a combination of effectiveness, efficiency, and satisfaction. There are also a lot of ways to measure usability, in regards to quantitative usability measurement questionnaires are frontrunners, with the System Usability Scale being the most commonly used and researched. However, the SUS consists of 10 questions and can be difficult to administer in large, in-the-wild, studies. Kraig Finstad understood that and in 2010 introduced the Usability Metric for User Experience (UMUX) with 4 questions.
Setting up the UMUX
The questions of the UMUX
The UMUX consists of 4 questions, these questions measure the perceived usability as described in the relevant ISO. As in the SUS the tone of the questions changes between positive and negative.
- [This system’s] capabilities meet my requirements.
- Using [this system] is a frustrating experience.
- [This system] is easy to use
- I have to spend too much time correcting things with [this system].
Example of questions with changed system’s wording
The design of the questionnaire allows for changing the content of the brackets ([This system’s]) to the actual name of the system, or a more suitable word for the application.
- This application’s capabilities meet my requirements.
- Using Google Forms is a frustrating experience.
etc.
The responses of the UMUX
The UMUX can be administered either verbally or through some written format (including digital questionnaires). Each question is scored on a Likert scale, from 1 (strongly disagree) to 7 (strongly agree).
Calculating the UMUX score
The rules:
- Positively worded questions are 1 and 3
- Negatively worded questions are 2 and 4
Scoring each question:
- For positively worded questions, take the number of the reply and subtract 1 (e.g. if the answer to the question 1 is Slight Agreement—5, then the question is scored: 5 – 1 = 4)
- For negatively worded questions, take the number of the reply and subtract it from 7 (e.g. if the answer to the question 2 is Slight Agreement—5, then the question is scored: 7 – 5 = 2)
Find the participant score:
We have the score of each question, to calculate the UMUX score for each participant:
- Add all the questions’ score
- Multiply by 4.1667 (or 100 / 24)
- You will get a score between 0 and 100, this is the UMUX score
To sum up, the formula for calculating the participant’s UMUX score is:
UMUX = ((Item_1 − 1) + (Item_3 − 1) + (7 − Item_2) + (7 − Item_4)) * (100/24)
The overall score of the UMUX (the system’s score)
Often one would use the UMUX to “put a number” to the usability of the whole system / application in question. In that case, the System’s score (or overall UMUX score) is the average of all the participant’s scores.
Presenting the UMUX results
Except for presenting the mean UMUX score of the system, usually the Standard Deviation and participants volume (sample size) are presented as well.
How many participants should someone include to get significant results?
To the point of the writing (will update if new research is published), unlike in the SUS, there are no general guidelines of the minimum number of participants required to achieve accurate results. Finstad created the questionnaire in a study with 42 participants, while Berkman and Karahoca validated it with 3 studies of 363, 42, and 151 participants — to give some general numbers. When working with relatively low numbers of participants, Laplace’s rule of succession is your friend.
Is the UMUX a good metric for measuring User Experience?
UMUX is a reliable metric. It is a great metric of usability, which is a part of User Experience. It has been validated numerous times. Berkman and Karahoca investigated deeply the structure, validity, and reliability of the UMUX, they also found that the UMUX scores correlate with the scores of the System Usability Scale and the Computer System
Usability Questionnaire. If one wants to evaluate a system they can use UMUX and obtain similar results to other validated questionnaires. The age and genre of the participants shouldn’t affect the UMUX scores.
Is the UMUX affected by participant’s characteristics?
As with other usability questionnaires, the previous experience of the participant with the system affects the resulting score. Participants that have used the system in the past would tend to rate the system higher, than participants who haven’t. Berkman and Karahoca also reported that non-native English speakers gave slightly different UMUX scores than native English speakers.
Translations of the UMUX
UMUX has been translated and validated in a couple of languages, here is a list of all translations as of 2023:
The Arabic UMUX
The adaptation can be found in the original paper by Khalid S. Al-Tahat.
The Chinese UMUX
- 该系统的功能能够满足我的要求.
- 使用该系统是令人沮丧的经历.
- 该系统易于使用.
- 我必须花大量时间去适应该系统.
Adapted by Yuhui Wang, Jing Chai & Tian Lei.
The Slovene UMUX
- Zmogljivosti [] izpolnjujejo moje zahteve.
- Uporaba [] me frustrira.
- [] je enostaven za uporabo.
- Preveč časa porabim za popravljanje napak ob uporabi [].
Adapted by Urška Laha, James R. Lewis, and Boštjan Šumak.
The Turkish UMUX
The adaptation can be found in the original paper by Mİ Berkman, Ş Şahin.