Introduction
Hey there, curious minds and daring innovators! Welcome to the ultimate guide to in-app experimentation, where we'll show you how to shake things up, push boundaries, and unlock the full potential of your in-product experiences. If you're ready to embrace the thrill of the unknown and discover what makes your users tick, then grab a cup of coffee (or your preferred beverage), and let's dive in!
Why experiment?
Let's start with the burning question: Why should you bother experimenting with your product and in-product experiences? Well, the answer is simple—experimentation is the secret sauce that can take your product from good to spectacular! By experimenting with something new, you get validation of your idea, and when you go a step further by testing and iterating, you get even more valuable insights into what resonates with your users and what falls flat. It's like being a mad scientist, but without the lab coat and crazy hairdo (unless that's your thing).
How to experiment
Now that you're convinced of the power of experimentation, let's talk about how to get started. Remember, experimentation is a mindset, an approach to continuously improving and delighting your users. Here's a step-by-step guide to fuel your experimentation endeavors:
1. Experimentation framework
Before you unleash your creative genius, it's essential to establish a solid experimentation framework. This framework will help you organize your experiments, set goals, form hypotheses, and measure results effectively. Think of it as the blueprint for your experimental journey.
Here’s one example of what that looks like for us at Appcues:
2. Goals
Every experiment needs a goal—a North Star to guide your efforts. Whether it's increasing onboarding success, boosting adoption rates, or enhancing retention, clearly define your objectives. Without goals, you'll be wandering in the dark like a lost unicorn (spoiler alert: this is why Appcues control experiments require a goal to get started).
3. Hypothesis
Now it's time to put on your detective hat and formulate a hypothesis. These are educated guesses about what changes or improvements might lead to the desired outcome. Remember, a hypothesis is like a spark that ignites the fire of experimentation.
4. Measuring results
Once your experiments are in motion, you need a way to measure the impact. Collect data, analyze it like a boss, and determine if your changes brought about the desired effect. After all, you're not experimenting just for the fun of it—you want results!
Why experiment with your product?
You might be wondering, "Okay, experimenting, in general, sounds great, but why should I do it specifically within my product?"
Data shows that in-product changes can have a profound impact on onboarding, adoption, retention, and overall user satisfaction. By running experiments within your product, you have the power to create a build-measure-learn feedback loop for your organization.
Experimentation overview
Before we delve into specific experiments you can run, let's quickly cover some key terms and concepts you'll encounter on your experimentation journey.
At Appcues, the way we look at experimentation is through two lenses:
Control/Holdback Testing: This is when you run an experiment, showing content to one group and showing no content to the other (the control group). Then you compare the difference in conversion between the two groups. Typically, you’re assuming the content will product at least some positive conversion lift (although in some cases you may find out it produced negative results!).
A/B Testing: This is when you run an experiment between multiple groups who are presented with slightly different versions of content. At Appcues, you can technically have as many groups as you like, although a typical 50/50 split is most common for this experiment type.
Lastly, you’ll also see this term used around:
Statistical significance: Ah, statistics—the friend who tells you if your experiment results are legit or just a fluke. Statistical significance helps you determine if the changes you made had a genuine impact or were merely a roll of the dice.
Top in-product experiments to run
Now it's time to roll up your sleeves. These are some of the top places where experiments can be useful to leverage:
Test something new: When you're introducing something new for the first time, like a Flow designed around encouraging users to book time with support, it's a great practice to start with a control group experiment. This allows you to measure the impact of the change and ensure it's a step in the right direction. It also gives you great piece of mind because you can test with a subset of users before rolling it out to your entire user base. Remember, even Batman tested his gadgets before fighting crime!
Onboarding: First impressions matter. Before you roll out that new user onboarding experience you’ve been working hard on, test it with a controlled group of users. This will add a valuable level of insight into its performance, so when your team wants to know how it’s performing, you’ll have an arsenal of receipts.
New feature announcements: How you communicate new features can make all the difference. Experiment with different approaches, from imagery vs. text only, different button colors, or variations on the same copy. A/B testing will help you learn what drives your users to engage so you can map out best practices for your team moving forward.
In-your-face vs. On-demand: Should you present information directly in users' faces or let them discover it on demand? Experiment with different placements and timing to see what grabs attention without causing annoyance. It's like finding the perfect balance between a surprise party and giving someone space to breathe.
Empty states: When users encounter an empty state (like a blank canvas), it can be overwhelming. Test providing them with a starting point—a gentle nudge in the right direction—to make them feel less lost. It's like a lifeguard throwing a buoy to a swimmer who's uncertain about which way to paddle.
Related experiments to run
Experimentation shouldn't be confined to just your product. Consider running experiments in other areas to uncover valuable insights:
Pricing: Test different pricing models, discounts, or subscription options to find the sweet spot that maximizes conversions and revenue. It's like finding the perfect balance between making a profit and making your users smile.
Marketing Campaigns: Experiment with different messaging, imagery, or channels to attract and engage your target audience. It's like playing Cupid, finding the right arrow to shoot straight into your users' hearts.
How to run a control experiment in Appcues
Get your lab coat on, because it's time to dive into how you actually run a control experiment directly in Appcues.
- As mentioned, start with a hypothesis on what you’d like to test (i.e. new onboarding flow, feature announcement, etc.) and how the results will influence what you are building
- Start a new experiment
- Then, you’ll select the flow you are experimenting with and provide some details to understand the sample size and duration the experiment needs to run
- That’s it!
Once you've set everything up, it's time to publish your experiment and let the magic happen. Sit back, relax, and watch the results roll in. 🎉 (Take that advice to heart, as stopping an experiment early throws off the results, so make sure you let it run its entire course).
Next steps
Last but not least, make sure you leverage the insights gained from your experiments within your product. Use the data to refine and optimize your user experience, constantly striving for greatness. After all, experimentation isn't a one-time affair—it's an ongoing journey of continuous improvement.
So, my fellow explorers of the unknown, it's time to unleash the power of experimentation. Embrace the excitement, the challenges, and the triumphs that come with pushing the boundaries of your product. Now, go forth and experiment like there's no tomorrow! If you’re ready to get started, chat with our team of product experts.
Happy experimenting!
