M
Malek Damiri
Guest
Most SaaS companies have a customer health score. Green, yellow, red. A number from 0 to 100, built out of logins, seats, NPS, support tickets, and weights someone picked in a workshop.
Very few of those scores have ever been tested. Not "does it look reasonable." Tested: when the score said an account was red, did that account actually churn more often than the green ones? By how much?
If a health score were a drug, it would need a trial before anyone prescribed it. Here's the trial. It takes about thirty minutes and three queries.
Don't test today's score against past churn.
An account that churned in July has a terrible score today. Of course it does: its usage went to zero when it left. Test current scores against historical churn and the score will look brilliant at predicting churn that already happened.
You need the score as it stood before the outcome. That means point-in-time snapshots: one row per account per week or month, with the score as it was that day. If you don't keep them, start today. It's one table, and in a quarter you'll have something to test.
For each snapshot, ask one question: did this account churn in the following 90 days?
Three filters do the work. One snapshot a month keeps any single account from dominating. Snapshots from the last 90 days are dropped, because their outcome isn't known yet. And snapshots taken after an account already left are dropped, because a score on a departed customer tells you nothing.
The data in this post is simulated: 1,500 accounts, 18 months of weekly snapshots, and a score built the usual way, from logins, NPS, and a penalty for support tickets. In the simulation, churn is driven by falling usage. That's an assumption of the demo, not a fact about your customers. The queries are what carry over.
Red churns more than green. That's the good news, and it's where most score reviews stop.
Look at the gap, though. Red churns at 1.5 times the rate of green. A CSM working the red list is working accounts that are only somewhat riskier than the ones they're ignoring. And red is 7% of snapshots. If red were rare and deadly, it would be useful. Rare and mildly worse isn't much of a list.
Bands depend on where you draw the lines. AUC doesn't. It answers one question: pick a random account that churned and a random account that stayed. How often did the one that churned have the worse score?
0.5 is a coin flip. 1.0 is perfect. As a rough guide for churn, 0.7 and up is genuinely useful, and around 0.6 is a hint.
It's a plain SQL query, using the rank-sum form of the Mann-Whitney U statistic:
Barely better than a coin. Now, change the
A one-line rule, "usage fell 20% or more versus three months ago," beats the whole score. And support tickets come in below 0.5: in this data, accounts that file tickets churn less, because filing a ticket means someone is using the product. The score subtracts points for tickets, so one of its own inputs points the wrong way.
Again, that's the simulation, and your tickets might mean something else. The point is you can find out with one query per input before anyone builds a QBR around the score.
Test each input on its own. Anything near 0.5 is noise. Anything below 0.5 points the wrong way. Drop it or flip it.
Beat a dumb baseline. If "usage dropped" outperforms your score, use "usage dropped" until the score earns its complexity.
Calibrate the bands to churn rates. Red should mean something you can say out loud, like "about one in four red accounts churns within 90 days." If you can't say that sentence, the colors are decoration.
Publish the backtest next to the score. One small table, churn rate by band plus the AUC, refreshed every quarter. People trust a score more when they can see how often it has been right.
Too few churn events. With thirty churns, AUC bounces around a lot. Pool more history or a longer window before you draw conclusions.
Contract timing. Annual contracts churn at renewal. A snapshot nine months before renewal and one a month before aren't comparable. Add days-to-renewal to the analysis, or test within renewal windows.
Your CSMs doing their job. If the team saves red accounts, red accounts churn less than they otherwise would, and the score looks worse than it is. That's a good problem to have. Log the interventions, so you can tell "the score was wrong" apart from "the score worked, and someone acted on it."
None of this needs a data science team. It needs score history and an afternoon. Plenty of health scores have survived for years because nobody ran these three queries. Run them before renewal season, then either trust the score for real or replace it with something simpler that works.
I'm Malek Damiri, head of revenue operations in Atlanta. I write about revenue metrics and how to check whether they mean anything at malekdamiri.com.
Very few of those scores have ever been tested. Not "does it look reasonable." Tested: when the score said an account was red, did that account actually churn more often than the green ones? By how much?
If a health score were a drug, it would need a trial before anyone prescribed it. Here's the trial. It takes about thirty minutes and three queries.
The mistake that ruins the test.
Don't test today's score against past churn.
An account that churned in July has a terrible score today. Of course it does: its usage went to zero when it left. Test current scores against historical churn and the score will look brilliant at predicting churn that already happened.
You need the score as it stood before the outcome. That means point-in-time snapshots: one row per account per week or month, with the score as it was that day. If you don't keep them, start today. It's one table, and in a quarter you'll have something to test.
Code:
create table health_snapshots (
account_id int,
snapshot_date date,
score int
-- plus each input, so you can test them one at a time
);Step 1: Label every snapshot with what happened next.
For each snapshot, ask one question: did this account churn in the following 90 days?
Code:
create or replace table labeled as
select s.account_id,
s.snapshot_date,
s.score,
s.usage_change,
s.tickets,
(c.churned_on is not null
and c.churned_on > s.snapshot_date
and c.churned_on <= s.snapshot_date + interval 90 day) as churned_90d
from health_snapshots s
left join churn_events c using (account_id)
where day(s.snapshot_date) <= 7 -- first snapshot of each month
and s.snapshot_date <= date '2026-06-29' - interval 90 day -- outcome is known
and (c.churned_on is null or c.churned_on > s.snapshot_date); -- still a customerThree filters do the work. One snapshot a month keeps any single account from dominating. Snapshots from the last 90 days are dropped, because their outcome isn't known yet. And snapshots taken after an account already left are dropped, because a score on a departed customer tells you nothing.
The data in this post is simulated: 1,500 accounts, 18 months of weekly snapshots, and a score built the usual way, from logins, NPS, and a penalty for support tickets. In the simulation, churn is driven by falling usage. That's an assumption of the demo, not a fact about your customers. The queries are what carry over.
Step 2: Does red churn more than green?
Code:
select case when score < 60 then '1 red (<60)'
when score < 75 then '2 yellow (60-74)'
else '3 green (75+)' end as band,
count(*) as snapshots,
round(100.0 * avg(churned_90d::int), 1) as churn_pct_90d
from labeled
group by 1
order by 1;
Code:
band snapshots churn_pct_90d
1 red (<60) 1433 4.3
2 yellow (60-74) 11098 3.6
3 green (75+) 8205 2.8Red churns more than green. That's the good news, and it's where most score reviews stop.
Look at the gap, though. Red churns at 1.5 times the rate of green. A CSM working the red list is working accounts that are only somewhat riskier than the ones they're ignoring. And red is 7% of snapshots. If red were rare and deadly, it would be useful. Rare and mildly worse isn't much of a list.
Step 3: One number for "does it rank risk?"
Bands depend on where you draw the lines. AUC doesn't. It answers one question: pick a random account that churned and a random account that stayed. How often did the one that churned have the worse score?
0.5 is a coin flip. 1.0 is perfect. As a rough guide for churn, 0.7 and up is genuinely useful, and around 0.6 is a hint.
It's a plain SQL query, using the rank-sum form of the Mann-Whitney U statistic:
Code:
-- AUC: the chance that a random account that churned looked riskier than a random
-- account that stayed. 0.5 is a coin flip, 1.0 is perfect. Ties get the average rank.
with scored as (
select churned_90d, -score as risk -- lower score = riskier
from labeled
),
ranked as (
select churned_90d,
rank() over (order by risk)
+ (count(*) over (partition by risk) - 1) / 2.0 as rnk
from scored
)
select round(
(sum(rnk) filter (where churned_90d)
- count(*) filter (where churned_90d)
* (count(*) filter (where churned_90d) + 1) / 2.0)
/ (count(*) filter (where churned_90d)
* count(*) filter (where not churned_90d)), 3) as auc
from ranked;
Code:
auc
0.556Barely better than a coin. Now, change the
risk expression to test anything else, which is where it gets uncomfortable:
Code:
signal risk expression auc
usage trend -usage_change 0.643
usage fell 20%+ (usage_change <= -0.2)::int 0.589
health score -score 0.556
support tickets tickets 0.434A one-line rule, "usage fell 20% or more versus three months ago," beats the whole score. And support tickets come in below 0.5: in this data, accounts that file tickets churn less, because filing a ticket means someone is using the product. The score subtracts points for tickets, so one of its own inputs points the wrong way.
Again, that's the simulation, and your tickets might mean something else. The point is you can find out with one query per input before anyone builds a QBR around the score.
Step 4: Act on it.
Test each input on its own. Anything near 0.5 is noise. Anything below 0.5 points the wrong way. Drop it or flip it.
Beat a dumb baseline. If "usage dropped" outperforms your score, use "usage dropped" until the score earns its complexity.
Calibrate the bands to churn rates. Red should mean something you can say out loud, like "about one in four red accounts churns within 90 days." If you can't say that sentence, the colors are decoration.
Publish the backtest next to the score. One small table, churn rate by band plus the AUC, refreshed every quarter. People trust a score more when they can see how often it has been right.
Three things that will fool you.
Too few churn events. With thirty churns, AUC bounces around a lot. Pool more history or a longer window before you draw conclusions.
Contract timing. Annual contracts churn at renewal. A snapshot nine months before renewal and one a month before aren't comparable. Add days-to-renewal to the analysis, or test within renewal windows.
Your CSMs doing their job. If the team saves red accounts, red accounts churn less than they otherwise would, and the score looks worse than it is. That's a good problem to have. Log the interventions, so you can tell "the score was wrong" apart from "the score worked, and someone acted on it."
None of this needs a data science team. It needs score history and an afternoon. Plenty of health scores have survived for years because nobody ran these three queries. Run them before renewal season, then either trust the score for real or replace it with something simpler that works.
I'm Malek Damiri, head of revenue operations in Atlanta. I write about revenue metrics and how to check whether they mean anything at malekdamiri.com.