Someone suggests you "do a UX benchmark of the competition" before your redesign. It is a good idea. The risk is that three weeks later you receive a folder of forty screenshots, marked "good" and "not good", that nobody opens again once the mock-ups start. The screenshots are not the problem. What was missing was one sentence, written at the start, saying which decisions the benchmark was meant to support.
It is an early-stage exercise, and I run a very stripped-down version of it every day in my content machine. Here is what I take from it: which exercise to ask for, how many competitors to look at, how to build the grid, and how to stop the benchmark turning into a copy of everyone else.
01The phrase UX benchmark covers three different exercises
The word is used for things that have neither the same cost nor the same value. Nielsen Norman Group describes two ways to compare your product with competing designs: the review, where a practitioner examines the sites against criteria, and comparative testing, where real users carry out the same tasks on several sites.
There is a third, stricter meaning. The same organisation defines UX benchmarking as evaluating an experience with metrics, against a meaningful standard: an earlier version, a competitor or a goal. It is often a programme repeated with each release, not a one-off study.
Before a redesign, most SMEs need the first, sometimes the second on one specific journey, and rarely the third. I say so early, because the three are not priced the same way and the word on its own does not tell you what you are buying.
02Pick two to four competitors, not twenty
Nielsen Norman Group recommends sticking to two to four competitor sites: beyond that, the study becomes too expensive and too heavy to analyse. If you have more, a quick first pass lets you sort them.
I keep, first, the ones your customers really compare you with, not the ones the board watches. Then one or two sites that do well what you want to do, even outside your sector. The same article suggests it: a neighbouring business, solving the same problem with different habits, often teaches more than a direct competitor.
In the video Competitive Reviews vs. Competitive Research, Nielsen Norman Group also points out that there is always competition, even when nobody does exactly what you do. For business software it is often the spreadsheet and email. That competitor almost never appears in the benchmarks I have seen, yet it is often the one you need to beat.
03Build the grid: tasks before criteria
The grid is the heart of a UX benchmark. I build it in two columns: what the user is trying to do, then how to judge each site on that point.
Start from the tasks
I list your users' main tasks, in their words. For a B2B site that might be understanding the offer in a minute, finding a price or a price range, booking a demo. Nielsen Norman Group's benchmarking guide suggests keeping five to ten of the most important ones. For a competitive review I often go down to four or five. If you already have personas that help you decide, they give you these tasks almost directly.
Add stable criteria
To judge each task I use a known basis rather than my taste of the day: Jakob Nielsen's ten usability heuristics. Does the user know where they are? Is the vocabulary theirs? Can they go back? Are errors prevented before they happen? These are broad rules of thumb, not a checklist, but they force you to justify every remark.
I then add two or three criteria specific to your project: how clear the pricing is, the proof offered (case studies, figures, reviews), the time before the first useful screen. Each line of the grid gets a simple score and one sentence of explanation. The sentence matters more than the score.
04Running the benchmark step by step
This is the order I follow for a competitive review before a redesign.
- Write the three to five design questions the benchmark has to answer: should prices be shown, how many steps to sign up, where to put the proof.
- Choose the sites, using the two-to-four rule, and note why each one is in.
- Run each task on each site, in a private window, on a computer and then on a phone. I assess as a visitor would, not as someone who already knows the brand.
- Fill in the grid as you go, with one screenshot per notable point, filed under the task it belongs to.
- Reread the grid line by line and answer the questions from step 1, in one or two sentences each.
- Present those answers first, then the grid, and the screenshots only in an appendix.
Step 3 comes from a lesson learnt on an assignment. A site can look fine to the logged-in team and be broken for visitors, because of caching. Since then I assess everything as a stranger would, competitors included, and I do the same in a technical audit before a redesign.
If a question is still open after the review, that is the moment for a comparative test on that one journey. Nielsen Norman Group then advises having each participant use no more than two or three sites, and alternating the order so that learning effects do not skew the results.
05What I do with it in my own work
In my content machine, every article starts with a brief, and that brief has an "angle" field: what the article offers that other pages do not. As long as the field is empty or too short, the tool rejects the brief and writing does not start. To fill it in, you have to have read what already exists. In effect it is a competitive review reduced to a single question, and the tool does not let me skip it.
On Moonworkers, my payroll software for small British businesses, the most useful benchmark was not against a competitor. The free calculators on the site were checked one by one against the official tools. Comparing with a reliable reference tells you whether the product is right, which no design comparison will.
One last thing I learnt on assignments: showing several mock-up directions at the same level of finish unblocks a decision. Nielsen Norman Group describes the same idea, a comparative evaluation of several variations of your own design rather than of competitors. After a benchmark, that is often the most useful next step: two or three directions, tested on a clickable prototype.
06The limits of a UX benchmark
A practitioner's review is quick, because it needs neither a research plan nor participants. The same Nielsen Norman Group video names the downside: without user data, it can produce highly biased results and keep unfounded assumptions alive. So I present my reviews for what they are, reasoned hypotheses, and I say which ones deserve a test.
Second limit: a competitor is not proof. What they do is not necessarily what works for them, let alone for you. They may have tested that choice, or not. Copying their pricing page means copying a decision whose reasons you do not know.
Third limit, more down to earth: measured benchmarking is expensive. The benchmarking guide warns that bad numbers are worse than no numbers, and that you should not set up a measurement you cannot repeat. For an SME I would rather have an honest review and a targeted test than a scoreboard nobody knows how to redo.
Finally, the aim is not to name a winner. As Nielsen Norman Group puts it, you want to beat the competition, not copy it. A good benchmark often ends with an idea that nobody on the list had had.
07Where to start before your redesign
If a UX benchmark is already planned, ask first for the list of questions it has to answer. If nobody can write it in five lines, that is where to begin, ideally during the project scoping week.
If you are starting from scratch, keep it simple: three competitors, four tasks, half a day in a private window, and one page of conclusions. You will have an imperfect but useful base, which the mock-ups and tests will then correct.
This is the kind of work I do during scoping or design, and you will find the details in my services. If your redesign is waiting on a benchmark, we can talk about it.