Playlists and Pirates

Let’s say you’re a data scientist on the playlists pod at Spotify. It’s your responsibility to measure the success of playlists and report back to your team.

Are people seeing playlists? Clicking them? Should playlists exist, or are they a distraction from consuming content? Should Spotify invest more or less into supporting and developing playlist functionality?

So, you work with your product manager to workshop tangible product success metrics: The primary metric might be daily minutes spent listening to playlists per user. The secondary metrics might include the share of playlist listening versus total listening, click-through rate of playlists on the homepage, new playlist listens per week, or playlist retention.

After aligning on the definition of success, you work with your data engineering team to source the metrics and models, build a few charts in your BI tool of choice (or ask your AI bot to do it for you), present the dashboard to your PM, set alerts for daily changes of +/-1%, and ship it to the rest of the team.¹

Great! A job well done.

Take the scenario further: In Spotify’s weekly leadership meeting, Daniel Ek asks his Chief Product Officer: “Hey, how are things going?”

Playlists are only a single facet of media consumption within Spotify. There’s a search feature, a library, genres, a home page, and AI-powered DJs, all of which are different entry points to consumption. Furthermore, consumption occurs on various platforms: phones, laptops, CarPlay, and smart TVs.

Is his CPO going to respond with a ten-minute monologue that describes each possible breakout?

Perhaps, but probably not. When Daniel EK asks, “How are things going?” He’s really asking, “How is our product progressing towards our company’s goals of increasing the number of people who have heard of Spotify, signed up, consumed content, subscribed, and told their friends to subscribe?”

Yes, this is a rudimentary example - Spotify has the necessary processes in place so Daniel Ek does not need to ask his CPO “how things are going.” However, it drives home the following point: software is complex, and individual feature success metrics have little inherent value unless they bubble up to a broader product-level goal.

The Pirate Metrics: AAARRR

AAARRR stands for Awareness, Acquisition, Activation, Retention, Revenue, and Referral.  These are achievable steps that lead to positive user and revenue growth for your product. It’s a framework developed by Silicon Valley alums during in the early aughts and, while has taken different forms over the years (most recently, Product Led Growth or PLG), the central concept is the same: measuring how people transition from not knowing about your product to yelling about it from the rooftops for all to hear, creating the flywheel effect to grow your business organically.

The AAARRR Funnel is the north star for any product team, and the way for these teams to measure holistic success as they iterate through the Product Development Lifecycle.

How does data play into this? In a perfect world, your PM and engineering pod can instrument all events and self-serve a funnel in a product analytics tool. However, that’s often a far-fetched goal. It typically requires disparate sources of data, such as backend databases, marketing partners, e-commerce fulfillment, payment processors, and complex time-based logic that tools like Amplitude or Mixpanel have trouble handling. More often than not, the model ends up in your warehouse, owned by your data team.

In comes your product-focused data scientists to the rescue!

The 0-to-1 Guide

Map User Journeys

If there’s one hard-and-fast rule to live by in a data role, it’s that before writing a single line of SQL, gather as much context as possible.² Gain insight into user feedback by speaking with your product and customer success counterparts and reviewing feedback directly from your users. You should look to answer the following questions:

1. How do users typically hear about your product?

   Example: Ads, word-of-mouth.

2. How can users sign up, and what drives them to do so?

   Example: Most new users sign up after receiving a free trial, but many are referred by existing users. We also drive many sign-ups through organic content sources like newsletters.

3. When do users realize the value of the product? When do they reach the “aha!” moment of delight?

   Example: Discovering playlists curated to the user’s listening tastes.

4. What keeps users coming back, paying for a subscription, and telling their friends? What features engage them most? What are their favorite paid features? How do they share content with their friends?

   Example: Listening to their favorite podcast for three weeks in a row. After reaching usage limits, they upgrade to unlimited and eventually share the podcast with friends and family.

Understanding the answers to these questions will inform future steps in the process, such as defining high value actions and exploratory data analysis.

Define High Value Actions (HVAs)

HVAs are the measurable actions that imply value received by the user. In the Spotify case, these may be:

1. Playing content

2. Engaging with content (i.e., comments, likes)

3. Engaging with tours (i.e., buying concert tickets)

4. Building playlists

5. … and plenty more

As users complete more HVAs, both in quantity and frequency, they are more likely to move down the funnel. In other words, they are more likely to return to the product, pay for a subscription, and share with peers.

Given that your product team is building the product with specific use cases in mind, they will have strong opinions on an initial set of HVAs. Heed their advice!

Exploratory Data Analysis (EDA)

Using the user journey and HVAs defined in the previous steps, visualize everything. The goal is purely exploratory; you should work to validate earlier claims (including ones from direct user feedback) to fill in the gaps of the user journey.

One effective tactic is to set a few target attributes from downstream funnel activity, such as power users with the top 10% engagement in specific features (retention), payers (revenue), and referrers (referral). Think of these attributes as the goals you want users to reach, then visualize how they reach these goals by segmenting up-funnel engagement by downstream activity.

Example: Paying subscribers consist of a 15% share of total users, but a 45% share of all “share -> copy link to Song” activity. Directionally, share link actions are strong candidates for HVAs. 

This will also act as an audit exercise. You’ll often find that something as simple as counting the number of users has more than one source of truth. Getting your hands dirty with preliminary measurement will help unveil untracked and mistracked user actions.

(Basic) Statistical Modeling

At this point in the process, you should have a detailed understanding of the following:

1. The most common and desired user journeys

2. Usage distribution across multiple HVAs

3. How HVA usage translates to downstream funnel activity

It’s now time to use this context to build and test a few hypotheses.

Don’t be afraid! We’re not suggesting causal inference or reinforcement modeling (not yet, at least). Simply understanding the basics of logistic regression and random forest models (Data Science for Business is a classic) and running a few tests with scikit-learn will suffice for now. 

The output of these modeling techniques will be statistically driven insights into the drivers behind activation, revenue, retention, and referrals, essentially taking the previous EDA one step further. 

For example, if a user performs more than 2 HVAs in the past 30 days, they are three times more likely to become paid users. That’s your retention stage logic.

In fact, depending on your organization’s stack, data size, and propensity to incorporate AI-powered workflows, you could let an LLM do this for you. (Note: proceed cautiously, follow organizational guidelines related to sharing PII externally, etc.).

Define the funnel in centralized models

You did the challenging work; now it’s time to regroup with the organization, present your findings, and align on a funnel that the entire organization can rally around.

Regarding funnel buildout, both the top (awareness, acquisition) and bottom (revenue, referral) are typically cut and dry. 

In other words, it’s easy (well, maybe not easy, but easier) to measure how many people have signed up to your product and how many paying users you have. However, determining the activation and retention logic will depend on your previous findings: both peer-driven context and data-informed insights. This is typically the time to move logic into your transformation tool and visualize the AAARRR funnel in your BI tool.

Lastly, don’t forget to bridge the last-mile analytics gap. Like any analytics work, the funnel is fruitless if teams don’t adhere to it as the centralized way to measure product success.

Recalibrate over time

Your product will inevitably change over time. Further, you’ll have more outcomes to measure against. Stay close to those changes and recalibrate the funnel as needed.

One effective SOP is to build an intake form where specific stakeholders around the company (typically leadership) submit recommendations for new HVAs and funnel logic. The data and product team can review these on a rolling basis and, when accepted, build them into the data team’s roadmap.

Closing thoughts

The beauty of building a product activation funnel is that you don’t need a fully stacked data science team to perform this workflow. All you need are strong peer partnerships, a data warehouse, a basic BI tool, and some foundational data modeling to get started.

And remember, it’s less about perfecting a single framework and more about creating a shared language across your company. It’s in aligning product, engineering, and business stakeholders around what success actually means for your users.

1  This nets out to roughly ~7M people.

2  The under appreciation of context building is forever unmatched. Could this be a future post? 👀