As a tech consultant, I help companies optimize how they build software. I talk to a lot of people about implementing new practices and processes that will help them put the user value at the center of the development practice. But with many of the companies I work with full assimilation of these concepts is a distant goal. Until yesterday, I’d never seen an organization learn about a process and start implementing it full-force the next day. That, my friends, is catching sight of a unicorn in the wild.
This unicorn is called OpenSignal and they came to visit me in a Pivotal Labs Office Hours session to chat about beta testing their new app. Ellie, Teresa, and Jaleh, told me about WifiMapper, a brilliant (IMO) native app tool that would help users locate public and private sources of wifi and provide some info about the location.
They had done a lot of work towards their research plan. They had recruited 60 people for a group of super-user participants. They had two focus groups scheduled in their office. They were preparing to conduct some pretty huge global surveys. They wanted the feedback that they got to drive their next stage of development and they came to me to figure out some ways to help them do that. How could they best decide which usability issues to work on and where they should focus enhancements?
I took a deep breath and broke the news that if I would recommend anything, it would be to cancel the focus groups and redo their whole research strategy.
Instead of getting mad, OpenSignal got curious. Here are some of the problems we talked about.
Problem 1: Focus Groups are Dumb
Focus groups are highly risky way of gathering information. If people are shy they won’t speak up. If they feel like others know more, they won’t say that they don’t understand. One aggressive person can take over the session and run it off the rails.
Essentially, people pollute people.
Additionally, a focus group format would not allow OpenSignal to watch their users play with the app, only hear about how people felt about it after the fact. Getting late feedback means that they were going to miss out on 99% of the valuable information that they would have caught if they had seen the app used for the first time in front of their eyeballs.
Solution: 1-on-1’s
Scrap the focus group and reschedule each participant as an individual user interview. This will allow OpenSignal to go in-depth with each person, digging into their personal experiences, challenges and hearing how they are reacting to the tool as they using it.
The transcripts of these interviews will contain nuggets of pure gold and uncut diamonds. The added bonus is that user interviews have a relatively small overhead. Generally, five 1-hour interviews are enough to help surface patterns in user behavior and usability issues strongly enough to help set priorities. This leads us to the second major problem that OpenSignal had…
Problem 2: No synthesis means you’re losing stuff
No plan for how to use the feedback and incorporate it into their development pipeline. Currently, OpenSignal fed bugs directly into their development team, but no one was really prioritizing them. Suggestions, usability complaints, and opportunities were tackled on an ad hoc, scattershot basis, and the work wasn’t really well connected to overall feature goals or metrics.
When you don’t take the time to synthesis and plan, all of the value of user research is lost.
Solution: Download and find the themes
OpenSignal needed a few baseline synthesis activities that would help them shake out the nuggets of wisdom from their interviews, as well as their general feedback pipeline. A really simple one is the whole team going through the raw feedback and write one take-away per sticky, talking them through and then bucketing them into similarities. Then you pull out themes, challenges and opportunities out of the buckets. These can then be mapped to specific business goals.
The main point of having the synthesis session is to clearly identify the patterns that emerged from your conversation and map them to your overall business goals. This helps the team understand where user value maps to goals and also sets priorities. BUT in order to achieve this you need your whole team to participate, which leads us to their final problem.
Problem 3: Not Enough Org Participation
The three OpenSignal ladies who joined me for Product Office Hours were collaborating on the user research stuff, but were finding it difficult to get other parts of the organization consistently involved. They hadn’t found a way to involve the developers, and so in turn devs didn’t see user feedback as part of their own workflow. This has made it doubly difficult for them to incorporate the feedback organically into the development backlog.
It can be difficult for small start-ups when there is so much going on, but I tend to say:
Not making the time to participate in user research is like ignoring a chainsaw as you try to cut down a tree with a butter-knife.
Solution: Make it really available
I encouraged these ladies to start having conversations with OpenSignal leadership about user interviews and ask them to try to listen to at least 1–2 of the user interview sessions. With OpenSignal, an invite is probably all that is needed, but if there is resistance, sometimes a knuckle cracking is in order to get higher-ups to tune in. At a small organization like OpenSignal, making sure that leadership is as close to direct user feedback as possible is critical to keeping priorities straight.
With developers, I take a really soft approach. I book space for them to listen to the interviews. I invite them all to both the interviews and the synthesis, and then announce to the whole team whenever the interviews are happening, day before, morning of and five minutes before the interview starts. This helps it get it in front of their faces. The same goes for synthesis. Leave the door wide open, and bang a drum that it’s going on. More often than not, curiosity will get the better of them.
The Full Monty
The next afternoon I went down to the OpenSignal office and was floored by how quickly they had implemented some of my suggestions.
- They had cancelled the focus groups and scheduled several user interviews.
- They had taken all of the existing customer feedback that was in their ZenDesk and put it through a synthesis process on a wall in the middle of the office.
- They spoke about the new user research process with their CEO and invited him and other leadership folks to participate in the upcoming user interviews.
- Finally, they sat me down and conducted their first user interview with me as their guinea pig, with an amazing script that was copied from a script template. They nailed the exercise and delivered it like true pros. It was this process that offered the ah-ha moment for them as to why this stuff is the absolute Poo.
“So much of what you said is what I’ve been saying, but can’t get anyone to take action on.”
“We had so many conversations around whether or not people knew what to do there, but couldn’t decide how to proceed.”
This is what this type of direct user research helps organizations do, understand what’s important vs. what can be put off.
There is very little ambiguity when you’re sitting next to a user and they do not notice, ever, that a button is clickable, or that there is another page of information that they can access. A few days later I got an update from Ellie saying that they had several interviews, videoed them, and shared out the live-feed with the whole organization and a ton of people listened. They also did a synthesis exercise and presented the synthesis to the whole organization. In the words of Ellie herself: “It was fascinating seeing the users go through our app, and at points frustrating because we were thinking ‘why can’t you see the button? it’s right there!” I have never been so impressed by a company’s commitment to putting their customer at the center of their process. OpenSignal, kudos for just doing it. You guys rock.
About the Author
