Looking for Signals: When to Do Evaluative Research

What is Evaluative Research vs Generative Research?

Evaluative research is what most people think of right away when they think about UX Research. Usability sessions are a method for carrying out evaluative research, which happens at the end of a design or development cycle. The goal is to evaluate whether assumptions that have been made are true, and that the output is not only usable, but grounded in real wants and needs from real people.

Conversely, Generative Research happens at the beginning of a design cycle during the discovery phase. It involves talking to people to find hidden opportunities to innovate or problems to solve with design and technology. ‍

Research requires a genuine curiosity and open mind; but more importantly the vulnerability to leave your ego and expertise and at the door, two things that stand in the way between your user and their needs.

In this article, I’m sharing when to do evaluative research, who should be involved, and how to prepare for the session. 

Looking for Early Signals to Kick Off Evaluative Research

Teams often struggle with how to schedule regular evaluative research alongside product development, as if it were another ceremonial activity inside a sprint. While you should regularly talk to users as generative research, I prefer to look for early signals that the team needs clarity and confidence around a solution rather than sticking to a rigid cadence for evaluative research.

In general, you will always learn something by getting your solution into the hands of real people, as soon as possible. The sooner you do evaluative research, the less married to the solution the team will be, and the less risk there is to the business.

Some signals it might be time to do some research:

• Have we started to make compounding decisions based on assumptions about our users?
• Is the team bringing up an increasing number of unknowns about our users, creating confusion or misalignment?
• Are we doing something new & novel in a workflow, but deviating from standard or expected interaction patterns?
• Is customer service receiving consistent feedback from users around a specific point in a workflow?
• Did we complete an end-to-end workflow that is core to our product experience?
  

Any one of these signals would be enough to kick off a usability session. The more you work with your team the earlier you can get ahead of these signals. 

Reducing Distance Between Research & The Team

At a minimum, a moderator and note-taker should be involved in the usability session. I’ve seen this team as a mix of researchers, product managers, designers, and developers, but usually moderated by the designer or user researcher. 

In general it’s best to keep these sessions intimate, like a conversation rather than a panel interview. The more comfortable the participant feels, the better and more authentic the results will be. What this *doesn’t* mean it to keep the unsynthesized research to yourself. 

The more you can reduce the distance between the team who uses the research with the actual research itself, the better. The more approachable research is to the team, the more involvement there will be. You’ll start to see greater advocacy for research when they experience the research firsthand. The team will start to bring up a specific person and their needs from a research session, rather than referring to everyone point blank as “users.”

Some ideas to involve the team: 

• Cycle them through the notetaking role
• Record and share the videos
• Call in from one account and stream the session to the wider team
• Invite them to the debrief after each session
  

As a rule of thumb, it’s best not to have “silent” attendees just observing. If they’re in, they should be actively involved in some way. Otherwise, share the video with them later.

Figuring Out What Your Team Wants to Learn

Once you’ve identified an early signal that it’s time to do some testing, you’ll need to extract from the team what it is you’re setting out to learn. 

I suggest always having a running list of questions to answer that pop up organically through doing the day-to-day work. At Notably, we use Notion to track potential research programs as a simple “questions to be answered”, tagged by feature, and link back to any conversations, tickets, or emails that surfaced this question. 

This habit builds a culture of humility, admitting that we don’t know everything about our users and are constantly making assumptions and taking bets. Research helps you make better assumptions and take smarter bets.

It also jumpstarts this next activity for planning your usability session.

Team Planning Activity

Map out the workflow you plan to test, complete with screenshots of each step in the flow. This can be done on a physical whiteboard or digital whiteboard tool. Create a row for each team member or role (depending on the size of your org) that spans across the workflow. Facilitate a working session to pull questions about this workflow from the team:

• What’s the peak experience?
• Where might user’s get stuck or hung up?
• What are our known unknowns?
• What are we assuming?
• What are our fears and concerns?
• Where are we doing something new/novel?
  

This becomes the framework to inform what will become your research plan: what you’re setting out to learn. This is a good team building exercise that gives everyone a chance to be heard and contribute to the research plan, or as one of our clients referred to as “product therapy.” 

‍