The Case for Context-Aware Model Evaluation

[David Taiwo Balogun]

Every model review I have sat in eventually arrives at the same slide: one big number in a large font. Accuracy 92%. F1 0.88. AUC 0.91. Heads nod. Someone asks about latency—the model ships.



The problem is not that the number is wrong. The problem is that it answers a question nobody actually asked. "How often is the model right, averaged over everyone in the test set?" is not the same as "Is this model safe to make decisions about the people it will be used on?"

Those two questions come apart whenever the population is not uniform, which is to say, always. Your users are not one distribution. They are several distributions stacked on top of each other, and the biggest one dominates the average.

Accuracy tells you how good the model is for the average user but nobody is the average user.

A concrete example: the thin-file problem​

Let us make this tangible with a setting I care about: credit for small borrowers. Imagine a lender scoring applicants for small loans. Most applicants are salaried. Their repayment behaviour is well captured by credit-bureau data and income stability, because those are exactly the things bureaus record.



But a meaningful minority, say 15%, are informal traders: market sellers, artisans, small-business owners. Many have perfectly healthy businesses, but their financial life runs through cash and mobile money, not bank payslips and credit cards. Their bureau files are thin: few reported accounts, noisy scores, systematically lower numbers at the same true risk. The signal that actually predicts whether they repay is cash flow.



This is not a hypothetical edge case. In many markets, including much of West Africa, the “minority segment” here is a large share of the real economy. I have simulated it so every number below is reproducible, but the structure is the one you meet in the wild.

def make_data(n):
    segment = np.where(rng.random(n) < 0.15, "informal", "formal")
    is_inf = segment == "informal"
 
    bureau_score     = rng.normal(0, 1, n)
    income_stability = rng.normal(0, 1, n)
    cash_flow        = rng.normal(0, 1, n)   # mobile-money signal
 
    # Formal: repayment driven by bureau + stability.
    # Informal: repayment driven mostly by cash flow.
    logit = np.where(
        is_inf,
        0.3*bureau_score + 0.2*income_stability + 1.8*cash_flow + 0.6,
        1.6*bureau_score + 1.2*income_stability + 0.2*cash_flow + 1.4,
    )
    repaid = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
 
    # Thin files: informal bureau scores are noisier and shifted down.
    bureau_obs = np.where(is_inf,
                          0.5*bureau_score - 0.8 + rng.normal(0, 1, n),
                          bureau_score)
 
    return pd.DataFrame({"segment": segment, "bureau_score": bureau_obs,
                         "income_stability": income_stability,
                         "cash_flow": cash_flow, "repaid": repaid})

Listing 1: Synthetic data with two segments whose repayment depends on different signals. Full script at the end of the article.



Now we do what a typical team would do: train on the standard bureau features, hold out 30%, and evaluate.

BASE_FEATS = ["bureau_score", "income_stability"]
 
model = LogisticRegression().fit(train[BASE_FEATS], train["repaid"])
p = model.predict_proba(test[BASE_FEATS])[:, 1]
 
print(f"accuracy: {accuracy_score(test.repaid, p >= 0.5):.3f}")
print(f"AUC:      {roc_auc_score(test.repaid, p):.3f}")
accuracy: 0.765
AUC:      0.814

Output: the numbers that end up on the review slide.



An AUC of 0.81 on a credit problem is perfectly respectable. The model is calibrated on average too: it predicts a mean repayment rate of 68.6% against an actual 69.0%. If you stopped here, you would ship it. Many teams do.

Step one: slice before you celebrate​

Context-aware evaluation starts with a simple act: compute every metric per segment that matters in the deployment context, not just over the whole test set. Here is a small, reusable function that does that, plus the metric that matters most in lending: of the people who would have repaid, how many did we approve?

def slice_report(frame, prob_col, threshold=0.5):
    rows = []
    for name, g in [("ALL", frame)] + list(frame.groupby("segment")):
        y, p = g["repaid"].values, g[prob_col].values
        yhat = (p >= threshold).astype(int)
        good = y == 1
        rows.append({
            "slice": name, "n": len(g),
            "accuracy": accuracy_score(y, yhat),
            "auc": roc_auc_score(y, p),
            "approval_rate": yhat.mean(),
            # Equal opportunity: share of genuine good payers we approve
            "tpr_good_payers": yhat[good].mean(),
            "wrongly_rejected": int((~yhat.astype(bool) & good).sum()),
            "mean_pred": p.mean(),
            "actual_repay": y.mean(),
        })
    return pd.DataFrame(rows).set_index("slice")

Listing 2: A sliced evaluation report. Swap 'segment' for whatever groups matter in your context.

Slice	n	Accuracy	AUC	Approval rate	Good payers approved	Good payers rejected
Overall	12,000	76.5%	0.814	77.4%	89.0%	907
Formal income	10,206	80.4%	0.867	80.6%	93.3%	478
Informal income	1,794	54.3%	0.527	58.9%	60.8%	429

Table 1: Baseline model, sliced by income segment.

Read that table slowly, because each column is telling you something different:

• AUC of 0.53 for informal applicants means the model's ranking of them is barely better than random. It is not a weaker model for this group; it is effectively no model.

• Only 60.8% of informal applicants who would have repaid get approved, versus 93.3% of formal ones. Same lender, same creditworthiness, very different odds.

• Informal applicants are 15% of the population but 429 of the 907 creditworthy people wrongly turned away (47%). The harm is concentrated, not spread.

• The overall numbers stayed respectable because the formal segment is large enough to carry the average.

Step two: check calibration per slice​

Discrimination (can the model rank people?) is half the picture. The other half is calibration: when the model says 30%, does 30% actually happen? Downstream systems such as pricing, limits and human review queues usually consume the probability, not the yes/no decision, so miscalibration quietly propagates.

The orange line is the most damning chart in this article. Informal applicants the model scores at 10% repay at roughly the same rate as those it scores at 90%. Any interest rate, credit limit or “auto-decline below 0.3” rule built on top of that score is arbitrary for this group, and the arbitrariness always lands on the same people.

Why aggregate metrics miss this​

None of this is exotic. It falls out of three ordinary facts:

• Averages are weighted by headcount. A segment that is 15% of your test set can be 100% broken and move the overall accuracy by only a few points. On a small enough slice, a total failure is statistically invisible in the aggregate.
• Features encode who was measured well. Bureau data is a record of participation in formal finance. It is informative for people who participate and noise for those who do not. The model is not prejudiced; the measurement is uneven.
• The loss function is indifferent to who pays for errors. Log-loss treats a wrongly rejected trader and a wrongly rejected salaried employee identically, and there are simply more of the latter to optimise for.

This is why “the model is accurate” and “the model is unfair” are not contradictory statements. Accuracy is a property of a model on a dataset. Fairness is a property of a model in a context: who it is used on, what decision it drives, and who bears the cost of its mistakes.

Step three: fix the cause, not the symptom​

The tempting fix is a group-specific threshold: lower the bar for informal applicants until their approval rate looks right. Sometimes that is a legitimate policy decision, but here it would be treating a symptom. If the score has an AUC of 0.53 for a group, moving the threshold just changes how many people you approve at random; it does not make the decisions any better.



The real problem is that the model cannot see the signal that matters for this group. So we give it that signal (cash flow) and let it learn that the signals carry different weight in each segment:

FULL_FEATS = ["bureau_score", "income_stability", "cash_flow"]
 
def add_interactions(df):
    df = df.assign(inf=(df.segment == "informal").astype(int))
    for f in FULL_FEATS:
        df[f + "_x_inf"] = df[f] * df["inf"]
    return df
 
train_x, test_x = add_interactions(train), add_interactions(test)
FIX_FEATS = FULL_FEATS + ["inf"] + [f + "_x_inf" for f in FULL_FEATS]
 
fixed = LogisticRegression(max_iter=1000).fit(train_x[FIX_FEATS], train_x["repaid"])
test["p_fixed"] = fixed.predict_proba(test_x[FIX_FEATS])[:, 1]
print(slice_report(test, "p_fixed"))

Listing 3: Adding the missing signal and letting its weight differ by segment.

Slice	Metric	Baseline	Context-aware	Change
Informal	AUC	0.527	0.837	+0.310
Informal	Accuracy	54.3%	76.2%	+22.0 pts
Informal	Good payers approved	60.8%	83.5%	+22.8 pts
Formal	AUC	0.867	0.870	+0.003
Formal	Accuracy	80.4%	81.0%	+0.6 pts
Formal	Good payers approved	93.3%	90.2%	-3.2 pts
Overall	AUC	0.814	0.866	+0.052
Overall	Accuracy	76.5%	80.3%	+3.8 pts
Overall	Good payers approved	89.0%	89.3%	+0.3 pts

Table 2: Before vs after, per slice.

Informal applicants go from an AUC of 0.53 to 0.84, and the number of creditworthy traders wrongly rejected falls from 429 to 180. The overall model also improves (AUC 0.814 to 0.866), which is the usual result when you fix a blind spot rather than paper over it.



Be honest about the trade-off. Notice that the share of formal good payers approved dips from 93.3% to 90.2%. That is not the formal group being penalised for fairness; it is the new model being better calibrated and slightly more conservative at the same 0.5 threshold. Whether that is acceptable is a business and policy call, and it should be made explicitly, by people who can see the per-slice numbers, rather than implicitly by an average.

A context-aware evaluation checklist​

Slicing by one segment is the start, not the finish. Here is the checklist I now use before any model touches real decisions. None of it requires a fairness library; it requires asking the right questions before writing the eval script.

Question	What to actually do
Who will this model be used on?	Compare the deployment population to the test set. If production skews towards a segment the test set barely contains, your headline metric is measuring someone else.
Which slices matter here?	Protected attributes where legal, plus context-specific ones: income type, region, language, device, data completeness (thin vs thick files). Ask domain people, not just the data.
What does an error cost, and for whom?	In lending, a false reject denies capital; a false approve costs the lender. Pick metrics (TPR, FPR, precision) that match the harm, per slice.
Is the score calibrated per slice?	Plot reliability curves by segment. Downstream rules consume probabilities.
Is the slice large enough to trust?	Report n and confidence intervals. A 10-point gap on 40 people may be noise; on 4,000 it is not.
Is the gap a data problem or a policy problem?	Low AUC in a slice means missing signal: fix features or data. Similar AUC but different approval rates may be a threshold or policy choice: decide it explicitly.
Will it stay fair?	Monitor sliced metrics in production, not just the global one. Drift often starts in a minority slice.

Table 3: Questions to answer before signing off a model.

Make it a gate, not a report.​

The most effective change is mechanical: make sliced metrics a blocking check in your evaluation pipeline, the same way a failing unit test blocks a merge.

GATES = {"auc": 0.70, "tpr_good_payers": 0.75}   # minimum per slice
MAX_TPR_GAP = 0.10                                 # equal-opportunity tolerance
MIN_SLICE_N = 200
 
def evaluation_gate(report):
    failures = []
    slices = report.drop(index="ALL")
    for name, row in slices.iterrows():
        if row["n"] < MIN_SLICE_N:
            failures.append(f"{name}: only {row['n']} samples, collect more data")
            continue
        for metric, floor in GATES.items():
            if row[metric] < floor:
                failures.append(f"{name}: {metric}={row[metric]:.2f} < {floor}")
    gap = slices["tpr_good_payers"].max() - slices["tpr_good_payers"].min()
    if gap > MAX_TPR_GAP:
        failures.append(f"TPR gap across slices = {gap:.2f} > {MAX_TPR_GAP}")
    return failures
 
assert not evaluation_gate(slice_report(test, "p_fixed")), "Model blocked"

Listing 4: A sliced evaluation gate for CI. The baseline model fails it; the context-aware model passes.



Run against the baseline, this gate fails on three counts (informal AUC 0.53, informal TPR 0.61, and a TPR gap of 0.33). The context-aware model passes. That is exactly the conversation you want to have before deployment, not after a complaint.

What this is not​

• It is not a single fairness metric. Equal opportunity, demographic parity and calibration across groups generally cannot all hold at once when base rates differ. Context tells you which one matters for the decision at hand.
• It is not only about protected attributes. The segment here is income type. In practice, unfairness often hides behind proxies such as region, language or data completeness.
• It is not a substitute for domain knowledge. The fix worked because someone knew that mobile-money cash flow is the meaningful signal for traders. No metric would have told us that.
• It is a simulation. The data is synthetic so the article is reproducible. Real datasets are messier, and the gaps are usually harder to close, which is all the more reason to measure them.

Takeaway​

A model's accuracy is a fact about a dataset. Whether that model is fair is a fact about a context: the people it serves, the decision it drives, and who absorbs its mistakes. If your evaluation never looks at the context, it cannot tell you whether the model is fit for it.

So before the next big number goes on the slide, ask one more question: accurate for whom? Then slice, plot, gate, and only then ship.

Reproduce it​

Everything above comes from a single Python script (numpy, pandas, scikit-learn, matplotlib), seeded for reproducibility: experiment.py. Run python experiment.py to regenerate every table and chart.