“So...what is this supposed to do?” I hear.

Then, “This could get nerve-wrecking. I would probably just give up.” 

A few minutes later, “Ah, I get it! This was what I was looking for earlier.” 

This is a typical User Testing Day with my teammates at Appcues. We’re sitting in the Davenport building in Cambridge, MA, where HubSpot’s headquarters are located.

We’re here to gather feedback on our onboarding experience and our improved WYSIWYG editor. Both are crucial to our customer experience and tie back to larger business goals. In order to get both right, we know that we need user feedback, and lots of it.

That’s where User Testing Day comes in. 

On the last Thursday of every month, the entire product team at Appcues spends the whole day gathering user feedback. A regularly scheduled User Testing Day has helped us create meaningful deadlines and a cadence around shipping. It’s also sharpened our skills as product owners and creators.

User Testing Day has become one of the most high-energy and productive days at Appcues. We get 10-20 tests done in a typical day, and the feedback has helped us improve our own onboarding, scope new features, and launch Appcues 2.0. 

We’ve gathered our best tips for everything from recruiting participants to running interviews to put together this end-to-end guide. We hope that it will be useful for anyone who wants to implement a User Testing Day or just learn a few more strategies for conducting more meaningful user tests.

1) Overcome user testing anxiety

User testing can be time-consuming and scary. If you don’t plan ahead with intention and dedicate time to it, there’s a good chance that it’d come too late or happen haphazardly. 

While it may feel uncomfortable to recruit people and ask for their time and thoughts, remember that you’re ultimately turning that feedback into better customer experiences. That’s well worth the initial dose of jitters. 

Tristan, our Head of Product Design, recently told the folks at User Interviews that the biggest lesson he has learned about user research is “just to talk to as many users as you can.”

“Nothing takes the place of experience and practice. Drop all your fears and schedule 5 calls this week and just do your best, get feedback, mess up a lot, and try it all again next week.”

It’s easier said than done, but just like with personal habit formation, building a routine around customer feedback can ease anxieties and prioritize it within your own workload and across teams.

2) Establish a cadence around user testing

The most productive user tests help progress critical features and garner feedback that can translate into gains for the business.

While goals may come from business or product leaders, we give engineers and designers autonomy over what to build and test every month. This has led to sharper hypotheses and more constructive user testing tasks.

To keep everyone aligned amidst autonomy, we follow a regular schedule of meetings that build towards User Testing Day: 

• Kick-off Meeting: Begin each month with a product planning meeting to determine what will be tested. 
• Weekly Product Planning Meetings: Collaborate and cover any adjustments.
• Daily stand-ups: Stay on track towards weekly goals.
• Pre-User Testing Day Prep: Create scripts and plan out the specific tasks a few days before User Testing Day. 
• User Testing Day: Run through a full day of tests.
• Post-User Testing Day: Analyze feedback, make quick improvements, and deploy if applicable. 

This schedule won’t work for all teams and can take some getting used to. Regardless of what routine you follow, sticking to one increases the likelihood of getting more tests done. 

3) Identify your target users and recruit according to use case

Recruitment methods depend on whether you’re testing something a new user would see (e.g. landing pages, sign-up process, new user onboarding experience) or something an existing customer would see (e.g. lifecycle nudge, a feature for power users). 

Here are some of the tactics we’ve tried for both groups: 

Hallway testing for new users

Hallway testing is a tried-and-true method for recruiting new users to provide their first impressions of your product. This is a great approach for getting unbiased thoughts on steps that occur early in your customer lifecycle.

While there’s some unavoidable loss of control when testing with complete strangers, you can try to optimize for your user base by targeting certain locations.

We share our office building with over a dozen of other startups, and it’s a prime location for testing a tech product. In the morning, we put up posters around the office to recruit participants. 

The team also gets out of the building and finds participants in nearby coffee shops and other areas with tech-savvy (or just tech-friendly) users nearby, like the startup-filled Kendall Square in Cambridge and WeWork offices around Boston.

Hallway testing is not for the faint of heart. It can take many tries before you feel comfortable approaching a stranger for help. For us, positioning ourselves in locations where we think people would be friendly and willing to connect has helped. We typically try to approach people who appear to be by themselves and just ask for 20 minutes of their time to help user test new software. Straight-forward without any frills. 

In-app messages for customers

Testing new features requires customers who are familiar with the product already. Our most effective channel for recruiting customers is in-app messages (built with Appcues). 

We use this simple modal window to target paying customers who are not brand new.

The modal shows up on the dashboard page and links directly to Calendly for easy scheduling. 

Customer slack channel

The in-app announcement is often sufficient enough to fill the schedule. If we need more participants or if there are any last-minute cancellations, we will recruit more through email and our customer Slack channel. 

Each test is scheduled for 45 minutes, on the hour, with 15 minutes of buffer. This usually gives us enough time to dig deep and take much-needed breaks to process the feedback and prepare for the next test.

4) Do as much as possible to fill the schedule

The worst thing you could do is under-invest in a User Testing Day. Without testers you'll be sitting alone twiddling your thumbs. 

We do everything we can to ensure a full schedule. While we try to convey excitement and gratitude in our communications, we also gently stay on top of people who sign up.

A week before User Testing Day, we confirm everyone’s slot by email.

If we don’t receive a confirmation, we auto-cancel the test and try to fill up the slot. Rewarding users provides an extra incentive too. $25 for 45 minutes has worked for us.

5) Welcome users and build a human connection from the start

Researchers who are solely focused on getting feedback may be tempted to jump straight to the tasks; this is a huge missed opportunity. 

Try to form a genuine human connection and build empathy with users from the start. Not only does this leave a better impression of your company, but building a rapport can multiply the gains of the interview that follows. Tristan explains how: 

“My goal is to make an honest human connection with the person on the other end, and set it up so it’s all but guaranteed that you get truth and honesty just oozing out of the conversation.”  

Asking basic questions and going one level deeper can go a long way. For example, a good follow-up question to “What is your role?” could be “How did you get to that?” or, “How do you like your role?”, or “What’s your management style?” 

A few tries of this could surface a shared passion or similarity in career paths. Taking the time to build this connection can make the rest of the interview more easy-going and productive for both parties. 

Here’s our pre-test interview checklist: 

Before diving into the tasks, a short pre-test problem interview can also help separate the problem from the solution you’ve built. Even if people have no usability issues, it doesn’t mean that your product has solved the problem you were hoping to solve. Ask what the user is currently doing to address the problem, how they would expect to solve it, and what solution would be most ideal.

6) Continue guiding users through tasks with empathy and excitement

Maintaining that strong sense of empathy and excitement once the testing part begins is paramount too. It’d be hard to muster up excitement in participants if it’s not obvious that you’re proud of what you’ve built and that you’re grateful for their time. 

Some participants may be apologetic and feel like they’re doing something “wrong” if they don’t immediately complete the tasks. Ensure them that there’s no such thing as wrong in a user test, and convey empathy when guiding them through the actions. 

Once users start going through the tasks, they are encouraged to say what they are thinking and doing out loud. Helen, one of our engineers, says that this is helpful in providing her an outsider point of view: 

“Because I’ve been working on it for so long, it’s hard not to have biases and assumptions. User testing helps me get out of the bias and think about how to use the product from another person’s shoes.”

When people express initial confusion or frustration, the most natural response might be to give them the answer or help them through the task. Resist this urge, and instead voice empathy and try to understand their expected outcomes. 

Try something like, “I understand that can be really confusing. What would you have expected it to do instead?” or “We’ve heard that from a few people today. If you were designer the feature, where would you have put it?” 

While difficult, teasing out users’ frustrations can help you identify product gaps and fine-tune solutions with a customer-first mindset.  

7) Don’t forget lunch

User Testing Day can get pretty hectic once it starts, so don’t rely too much on time between tests to get work done or loop back on important questions. 

Ordering a few pizzas can be the difference between a productive product team and a very grumpy one.  

Making customer empathy a priority

User Testing Day is ultimately just another venue for getting user feedback. It’s not the only one out there (and it’s not the only one we use—we also host Drunk User Testing and conduct user research as needed), but we’ve found it to be so effective that we ingrained it into our product development process. 

Even if the features that are tested don’t pan out, product owners will hear why first-hand and have a better sense of what to build next. “Anyone who is involved in the process of building products benefits from hearing what people think in real time and interpreting that information,” says Jonathan. “We’re getting good at teasing out what really has more value to the customer.” 

User Testing Day creates a buzz in the office that spans beyond designers and engineers. It makes the entire team excited about product and keeps customers at the forefront of what we do.