[Editor's Note: Since the publication of this article, HubSpot's Sidekick product has been absorbed into their sales software platform HubSpot Sales.]

Retention is a tricky beast. It can sneak up on you and sink your entire endeavor. Getting your new user retention experience right often takes iteration and patience, but it’s always, always, always worth the effort.

We’ve been working to improve retention at Sidekick (now known as HubSpot Sales)—for the better part of a year now. And with the help of our excellent team, we’ve found some success after running numerous growth experiments to solve our retention challenge.

Note: the data in these—and the following—charts are all fake; but the lessons are 100% real.

The Importance of Retention

“If you have poor retention, nothing else matters.” This was said by my colleague, renowned growth marketer, Brian Balfour, and it’s absolutely true.

With bad retention, monthly cohorts slowly slink into sadness. It makes all the effort you put into acquisition seem useless.

But good retention keeps a company energized. It can give you more at bats to get your marketing or your product right, and it brings in more revenue.

Do you know what’s even better than good retention? Negative churn. Negative churn is that holy-smokes-awesome phenomenon that reflects users that sign up for your product, and spend more money over time; offsetting revenue lost with churn with upgrade revenue.

This could be from upgrading to a paid or higher paying plan, or from adding team members in what we call a B2C2B strategy.

The benefits of better retention are vast:

• A higher LTV of your customers, which can be reinvested into a greater customer acquisition budget.
• An increased virality as there’s more happy users to spread the word, which lowers your CAC significantly.
• Shorter payback periods, especially with a free-to-paid business model like that of Sidekick.

Sidekick’s Retention Challenge

A user’s journey can be broken up into three stages of retention, each with its own challenges:

1. Week 1 - can you get your users to use your product more than once?
2. Mid-term (Weeks 2-4) - can you establish any pattern of usage?
3. Long-term (Weeks 5 and on) - how does your product become an indispensable tool?

At Sidekick, we had 2 main problems we wanted to fix. The first was that cohorts were getting worse month over month.

We tied this problem back to week 1 retention issues. We seemed to have a high volume of “drive by” users that only took one action and then bounced.

And the second problem was that retention never plateaued. New users were churning at a consistent clip and never found their way into routine usage.

How We Improved Week 1 Retention at Sidekick

We sent out a survey

Sometimes the easiest way to get the insight you are looking for is to go straight to the source. In this case, we sent out a survey asking churned users why they stopped using Sidekick. We then isolated the feedback from users that churned within week 1, and broke it down by reasons like this:

This data gave us enough confidence to start experimenting. Each insight was—with the exception of competition—things we believed we could fix with the right user onboarding experience.

Because so many of these users never returned to app even once, we focused on the welcome screen to engage users when they first login.

Experiment 1: Add Explanation to Empty State

We stripped out a ton of CTAs that we thought were confusing. There are still too many CTAs here, but, at the time, we thought it was an improvement. It turned out not to make a difference. Users didn’t know how to interpret what we implemented.

Experiment 2: Adding Sample Data

Adding sample data confused users. We got a lot of people thinking we were sending emails on their behalf already. This may be a common danger when using sample data.

Experiment 3: Inserting an Explainer Video

Even though it looked like people watched a decent amount of the video, we didn’t know what their reaction was. They may not have listened to the sound, they may have thought “I still don’t get how I would use it”—the examples may not have been pertinent—, or they didn’t think it would be valuable for them.

7 Failed Experiments Later…

After 7 failed experiments, we went back to the drawing board and we asked ourselves, “is the empty activity stream providing any value to new users?” We decided it didn’t.

So we figured it was worth trying to get them to send more emails instead, and this modal text was the simplest way to test that. Ultimately, this message pointed users to a better place to start engaging with HubSpot Sales, and lead to a significant increase in user retention.

How We Improved Long-Term Retention

Long-Term Survey Results

The survey feedback from users that churned after week 5 was also easy to sort by reason.

We focused on the actionable reasons and ignored the others, and ran experiments toward actually solving churn.

Experiment 1: Cut Down Noise

By consolidating Sidekick notifications, we reduced the cognitive load that we were impressing on our users. This fix helps avoid annoying them to the point where they turn off Sidekick entirely.

Experiment 2: Resurrecting Users

The checkbox shown above allows users to toggle the product on / off. Once it’s off, a user has to remember to toggle it back on.

It turns out a lot of people had forgotten they had turned their checkboxes off, and they didn’t remember to turn it back on. We found this through the qualitative feedback, many users said “Oh yeah! I was wondering what happened to Sidekick.”

Turning it back on was something that most users appreciated and was a genuine way to bring active users back instead of tricking them into turning it back on.

Experiment 3: Investing in Outlook

It’s hard to point to one bug fix, or one issue that we resolved for Outlook users that fixed our retention issues, but we were pleasantly surprised to see that after focusing on stability improvements our retention started to improve.

Retention Takeaways

Retention can be a tricky beast. But I hope you know how much there is to gain by improving your user retention if even by a few percent. As you implement your own retention experiments, here are some tips to keep in mind:

• Retention problems sneak up on you: your top level metrics may be going up, but if your users are not sticking around, you’re in trouble. Don’t be blind to this.
• Ownership: it really helps to have a directly responsible individual (DRI) on new user onboarding. We set up a team that was specifically focused on week 1 retention.
• Alignment of goals: make sure all teams are excited about optimizing for growth and are willing to make sacrifices and invest time in helping improving retention.
• Soak time: weekly/monthly cohorts take a while to mature. Make sure to give them enough runway to understand their engagement habits—or lack thereof.
• Cohort sizes: you really need a large cohort size to run and effectively measure many experiments.