As humans, we evaluate software in much the same way as we evaluate relationships with other humans.

Consider a hypothetical relationship between two highly compatible people as a series of questions:

Can I buy you a drink?
Can I have your number?
Can we go out to dinner on Saturday?
Can I come up?
Will you meet my friends?
Can you come to dinner with my parents?
Will you move in with me?
Will you marry me?

What would happen if the sequence were changed just slightly:

Can I buy you a drink?
Will you move in with me?
Can I have your number?
Can we go out to dinner on Saturday?
Can I come up?
Will you meet my friends?
Can you come to dinner with my parents?
Will you move in with me?
Will you marry me?

This subtle change can mean the difference between falling madly in love or being forever forgotten as “that creepy guy from the other night.”

This happens all the time in software. Products overwhelm users and ask for too much too soon. If you know the 10 actions users must take to become successful, it doesn’t mean you should force them to take all 10 in their first sitting. Similarly, just because you want to improve your conversion rate doesn’t doesn’t mean you should ask users to pay right away.

Rather, it’s important to exercise restraint and build a relationship with your user that flourishes in the way the best human-to-human relationships do. You want to start with small commitments and build consistency. Your user should gracefully grow from a newbie to a power user in a way that is natural to them.

If you do this, more users will inevitably upgrade their account because it’s most beneficial to them.

Crunch the numbers

When people think about analyzing their product’s user onboarding experience, they often think of charts like this:

This type of funnel is hugely important to track your conversion rates at each step. But here’s the problem: it tells us nothing about why. For that, we have to look much deeper at your usage data.

An example: mai.ly

Let’s examine a fictitious email automation tool, mai.ly. mai.ly has a 30-day free trial before it limits the number of emails users can send per month. The trial currently converts at 3%, and our goal is to help more customers become successful within 30 days.

In order to better understand the behavior that leads to success, we want to work backwards. Since our trial is 30 days, we’ll use that as our observation period, and look at usage behavior for one month’s worth of signups (selecting everyone who signed up between 60 and 30 days ago).

Step 1: Segment users based on adoption

First, separate these users into 2-4 mutually exclusive segments based on their adoption. For mai.ly, we’ll choose 3 groups:

• Not engaged (has not sent or created an email in last 7 days)
• Engaged, not paid (has sent or created an email in last 7 days)
• Paid (engaged, converted)

Assume that after 30 days, 60% of mai.ly users are not engaged, 37% are engaged, and 3% are paid.

PROTIP: Remove anomalies from this analysis. If someone paid mai.ly yet has neither created nor sent an email in the last 7 days, they shouldn’t be included. That customer is likely a churn risk and didn’t follow the success path you want to build for other users.

Step 2: Identify user activities that lead to customer success

Comb through usage data and identify the core activities that lead to customer success. Some of these can be volume milestones, Facebook has found that successful users add at least 7 friends in their first 7 days after sign up.

Complex products might have a lot of user activities to analyze, but for simplicity’s sake, let’s say there are 5 core user activities for mai.ly:

• Installing your javascript snippet
• Creating a custom email template
• Sending your first email
• Setting up 5+ triggered emails
• Launching an A/B test

Step 3: Compare usage patterns among your segments

Review the user data in each segment and calculate what percentage of the segment completed each activity by the end of the observation period. This is when you’ll start to see real patterns. Some of them will be quite obvious (like sending your first mai.ly email), while others may be surprising (A/B testing).

The data above tells us the percentage of users in each segment who performed each action before the time interval (in this case, 30 days). So you can see that after 30 days, 42% of users who eventually paid launched an A/B test, while only 19% of engaged users have.

Bonus: Save hours with this Google doc to help you organize data & get insights faster.

Step 4: Repeat this analysis at different time intervals

What do users who are successful after 30 days look like after 1 day? After 5 days? Are there early behavioral traits that lead to user success? Yes.

Do the same analysis for at least 3 different time intervals. This can be a cumbersome process, but it’s how you’ll discover clear insights about the success path that is natural to your user.

Note: Even as you change time intervals, your segmentation should remain based on data from the end of your observation period.

Get insights from the data

This analysis not only tells us what needs to happen in order for customers to become successful, but also what needs to happen when.

In mai.ly’s case, for instance, we find 27% of users who convert install mai.ly’s javascript snippet on the first day (and 64% within the first 5 days). In fact, there’s a startling difference in the free-to-paid conversion between users who install the code within 5 days and those who don’t:

We also notice there are other activities that highly correlate with success that typically don’t happen until later in the observation period (A/B testing, setting up 5+ triggered emails).

Support your quantitative hypothesis with qualitative findings

When seeing data like this, it can be tempting to jump to conclusions based on the numbers alone. If conversion is that much higher, shouldn’t I just lock users out of the webapp until they install the code snippet?

Maybe.

Installing the code snippet isn’t the only activity that is highly correlated with success. 42% of users who convert customize an email template in their first five days too.

Are users sufficiently motivated to install the code snippet without taking other actions? This is where it’s important to gather qualitative findings to support your quantitative hypothesis. Customer development, jobs-to-be-done interviews, and usability testing are all good ways of better understanding you user’s motivations and abilities.

Your success path can also be quite different based on your user persona. If it’s a marketer signing up for mai.ly, it may be best for them to get started by drafting emails and creating templates. If it’s an engineer, perhaps they’re most comfortable going straight to the code install.

The more you can learn about your user during the signup process, the better.

Use automation to craft the relationship with your user

Marketing automation is how you make the conversation about the unique needs and behaviors of your user.

mai.ly would want to focus first on early stage success activities like customizing email templates, installing the code snippet, and sending the user’s first email campaign. But as users complete these tasks, they can gradually shift the messaging to focus on activities that deepen engagement like A/B testing and adding additional triggered emails.

You can use automation inside your product, where your user is best prepared to act on the message. For instance, mai.ly could use Appcues to get users who have sent at least 3 emails but have yet to A/B test to launch their first:

Lifecycle emails are the most common form of automation, and they’re highly effective. At Appcues, we send this one to users who have yet to install our code on day 10 of our 14 day trial. It has a 52% open rate and 4.7% of users who receive it install.

Subject: Help with your Appcues Embed Code

Hi there,

I’m Spencer, one of the engineers at Appcues. I was just taking a look through your account and noticed you still haven’t installed your embed script.

I can help with that! Here’s a link to my calendar where you can choose a time for me to help you or one of your engineers set up Appcues. It shouldn’t take more than 6 minutes, and once it is configured properly you’ll be able to launch personalized in-product experiences without changing any code.

If you just forgot and are ready to do this by yourself I’ve pasted your personalized installation instructions below my signature.

Let me know if you have any questions,

Spencer Davis

The bottom line

As humans, we evaluate software in much the same way as we evaluate relationships with other humans. So it’s important to carefully craft your success path to escalate over time in the same way real relationships do. There are three important ways to do this:

• Dig deep into your data to understand when successful users are taking important actions
• Contextualize your quantitative findings with qualitative observations and user personas
• Use marketing automation (email + in-product) to get users to take actions that are meaningful to them when they are most relevant

One final note: mai.ly is a fictitious company with fictitious data. As Appcues grows, we’re learning and iterating from this process ourselves. Once we have enough insights for it to be helpful for others, we’ll publish our real data with actual findings.