Lesson 6

6 mins

Measuring What Matters

Cut through the noise of metrics to focus on what really indicates success. Learn which metrics matter for each channel while keeping focus on business outcomes.

Measuring What Matters

Time to tackle everyone's favorite topic: metrics! 📊

Measuring engagement effectively means focusing on metrics that actually drive decisions, not just collecting data for its own sake. That’s especially true when scaling your strategy across thousands of users.

Beyond the Vanity Metrics ✨

We've all been there: staring at dashboards full of numbers, wondering what they actually mean. Let's focus on metrics that tell a real story about your user engagement.

Instead of:

Tracking opens, clicks, and views

Try:

Measuring completion rates, retention impact, and behavior changes

Instead of:

Looking at isolated channel metrics

Try:

Tracking journey completion across channels

The Metrics That Matter

Activation Metrics

These tell you if users are getting started right:

Time to first value: Days between signup and first meaningful action
Key action completion: Percentage of users who complete critical setup steps
Initial engagement rate: How many new users take a second action after their first

How to measure without a data team:

Use your engagement platform's built-in analytics for basic completion rates. For time-based metrics, create a simple spreadsheet tracking 10-20 new users' journey days to identify patterns. Most email tools provide basic engagement data you can export regularly.

Adoption Metrics

These show if users are building habits:

Feature adoption rate: Percentage of users engaging with specific features
Engagement frequency: How often users return to your product
Usage patterns: Which features are used together or in sequence

How to measure without a data team:

Set up a weekly manual check of 5-10 random active users to observe their feature usage. Many products have usage logs you can export to CSV and analyze in spreadsheets. For engagement frequency, track weekly/monthly active users divided by total users.

Impact Metrics

The numbers that actually move the needle:

Conversion rate: % of users who take desired business actions
Retention improvements: Reduced churn compared to baseline
Revenue impact: Direct correlation between engagement and revenue

How to measure without a data team:

Start with simple before/after comparisons when you make changes. Track conversion rates manually for small cohorts (50-100 users) by checking how many complete a key revenue action. For retention, create monthly cohorts in a spreadsheet and compare their 30/60/90 day retention rates.

The Testing Framework

We've all been there: staring at dashboards full of numbers, wondering what they actually mean. Let's focus on metrics that tell a real story about your user engagement.

Pick one metric to improve

Choose a clear focus: Select one specific metric (e.g., "onboarding completion rate" rather than "engagement")
Ensure it's measurable: Verify you can track this metric before and after your test
Connect to business goals: Link your chosen metric to a key business outcome
Set a specific target: Define what improvement would be meaningful (e.g., "increase from 25% to 40%")

Change one element at a time

Isolate variables: Modify just one component (message copy, timing, channel, etc.)
Document your hypothesis: Write down why you think this change will improve your metric
Create a clear control: Maintain an unchanged version for comparison
Consider A/B testing: Split your audience if you have enough users (at least 300-500 per variant)

Give it enough time to show results

Define test duration: Set a specific timeframe based on your typical user behavior cycles
Account for multiple use cases: Allow enough time for different user types to encounter your change
Resist premature conclusions: Avoid judging results until you have statistical significance
Capture external factors: Note any unusual events during your test period (e.g., holidays, outages)

Document what you learn

Record all details: Save the what, when, and how of your test
Analyze both expected and unexpected outcomes: Look beyond your primary metric
Create a testing knowledge base: Build a simple repository of all tests and results
Share insights across teams: Circulate a brief summary of what worked and what didn't

Real-world success: Hotjar’s testing win

When Hotjar noticed a drop-off between signup and installation, they didn’t guess—they tested. By asking “Have you used Hotjar before?”, they showed users one of three tailored onboarding checklists. Each started with the same first step—install Hotjar—then adapted based on experience level. Read the full story →

The result:

26% increase in installations

Your Measurement Toolkit

Channel Metrics vs. Business Outcomes

Channel-specific metrics are useful indicators of message performance, but they're not the same as business success. High open rates or engagement percentages don't necessarily translate to meaningful outcomes.

Instead of:

Optimizing solely for channel metrics (like open rates or click-throughs)

Try:

Connecting channel metrics to actual business goals (like feature adoption or conversion)

For example, a feature announcement email with a 60% open rate might seem successful, but if only 5% of openers actually adopt the feature, the campaign isn't achieving its true purpose. Whenever possible, measure the end result: what percentage of users who engaged with your message ultimately took the desired business action?

That being said, each channel has its own unique set of key performance metrics that can be quite useful when optimizing specific messages. Let’s review:

In-App Messages

Interaction rate:

% of users who click on your message after viewing it

Completion rate:

% of users who finish a multi-step flow without abandoning

Goal attainment rate:

% of messages that achieve their intended goal

Open rate:

% of recipients who open your email

Click-through rate:

% of opened emails that receive clicks

Delivery timing:

When emails receive the highest open/click rates

Push Notifications

Open rate:

% of recipients who open your notification

Opt-out rate:

% of users who disable notifications after receiving one

Delivery success:

% of notifications successfully delivered to devices

QUICK EXERCISE

Adding Measurement Plan to Your Journey Map

Now let's complete your Journey Map with a robust measurement strategy:

For each key touchpoint and flow on your map:

Define 1-2 specific success metrics (beyond basic opens/clicks)

Set benchmark goals for these metrics

Note which metrics indicate progression to the next stage

Identify what would trigger intervention or flow adjustment

For the overall journey:

Determine the primary success metric for the entire journey

Identify intermediate metrics that predict overall success

Note which data sources you'll need to collect these measurements

Plan A/B testing opportunities at critical decision points

This measurement layer ensures your engagement strategy can be evaluated and improved over time. In our final lesson, we'll prioritize which parts of your journey map to implement first.

Remember ✨

Good measurement isn't about tracking everything—it's about tracking the right things. Start small, test thoughtfully, and let the data guide your next move.

Measurement Planning Template

Next up: We'll turn all this knowledge into action with a practical implementation plan. See you in Lesson 7! 🎯