How to Test Product Hypotheses When Your User Base is Tiny

[Daria Kasatkina]

Chances are you, as a product manager or a startup founder, took some time to learn about designing A/B tests to make informed decisions about which features land, and which need to be thrown in the bin.

But there’s one thing you may have noticed doing a Coursera course or a Python exercise on DataCamp: oftentimes, A/B test designs rely on having a lot of data from a large sample of users.

While these exercises are very useful, and big data is a reality for many tech products, there are also many examples where you may end up working with smaller samples and way less data:

• An early-stage startup: you just don’t have that many users yet;
• An internal tool: your entire user population may be under a couple thousand users: in-house developers and analysts, customer support specialists, recruiters;
• A product or a service with a lower event volume or a lower usage frequency: a number of B2B tools, property tech, etc.

While it’s common knowledge that if your user population is small the best thing you can do is talk directly to the customers, I believe using data is still important for a well-rounded understanding of where the product metrics are going. There’s a famous book that’s frequently recommended to Product Managers named “The Mom Test”, and its whole premise is basically that: people can lie or sugar-coat information to be polite and spare your feelings, just like your mom (hopefully) would. The solution Rob Fitzpatrick proposes is, instead of asking if an idea is good, you must act like a detective investigating the problems customers face. And, while asking better questions in customer interviews and gathering qualitative data is a must, it doesn’t remove the need for real, actionable quantitative data to supplement it.

Today, I want to share what I’ve learned working within these constraints as a Product Manager of internal products in a number of big tech and fintech companies. My goal is to show that making data-driven decisions in products with small populations is possible, and can give you a lot of insights that simply “talking to customers” cannot.

A Customer Support Specialist’s Workday​

I will be using customer support as an example, because it’s what I know best, and also because it ticks a lot of the boxes I described above: oftentimes, a customer support specialist is an internal user, you may have a smaller population of them (~100-2000 in-house employees).

Metrics volatility​

If you’re a Product Manager or an Analyst working with customer support products, you will have to test hypotheses and design A/B tests around two main constraints:

• Your population is small;
• The productivity metrics (such as average handling time (AHT), occupancy, or after-call work time) are very volatile.

If you look at one customer support specialist’s productivity during one shift, it may look like this:

It makes total sense: some cases are easy and require a short, templated answer, while others require a full investigation to get to the bottom of the problem.

So, we can see that our key metric is very volatile, and, since the population is small, we can also assume that each individual customer support specialist’s actions would heavily influence the averages in an A/B test.

Outside influences​

One thing we always try to rule out in test and control groups in an A/B test is something other than the change we’re testing influencing the result. In internal tools testing, such influences are usually known before the test even starts:

• Internal tool users are employees, not customers, and in a small population they are most certainly communicating about what’s going on with the tools they’re using (a great channel for feedback, by the way);
• The human factor can be a heavier influence here: annual leave, team switches, training, etc.;
• Their actions are more often scripted or bound by internal targets and regulations.

So, when you’re testing a change in a tool the internal users use daily, you would also have to factor in that you’re not testing in a vacuum: your users may be talking to each other, influencing the decisions and actions. This may cause a spillover effect that will cause test interference, or a SUTVA violation.

How to design a valid test for a small population​

Even within the constraints of having a small population, it’s still possible to design a test that will yield valid results and also interpret these results correctly.

Stratification​

In most cases of testing on internal users, stratification is key to good test design. Stratification is segmentation's more rigorous cousin. It balances the groups on the variables that drive most of the variance, so the comparison isn't contaminated by group composition.

In customer support tests, we would stratify the agents by their tenure, support channels they’re working in (e.g. chat or phone), grade, and other parameters, depending on the test.

Avoiding long test runs when possible​

We usually try to design tests on small samples to run quickly, (<30 days), to reduce the influence of outside factors and get clearer data. That is not always the case: in one anecdote an analyst shared with me, they ran a test for 6 months on a small sample of stores (about 50), and it would be impossible to gather enough data faster. But as a rule of thumb, you have to aim for shorter time frames.

It doesn’t always have to be an A/B test​

Sometimes the classic A/B split genuinely isn’t an option. In that case, there are some alternatives that can still deliver useful data.

• Limited release: if a randomized sample is something you can’t reliably achieve, you can choose who will be the test group, in our customer support example these can be teams or business units, in a startup or a B2B service you can choose a region to test it in or a certain segment of a population, like a timed cohort;
• The test group a week ago is the control group: while this is one of the weakest ways to get valid data, in some cases it can still be useful, and I’ve used it myself to get some good data.

Use variance-robust estimators​

• Analysis of covariance or ANCOVA: the default when your test group is randomized. Regress the outcome on the test group flag plus each unit's pre-period baseline. In our case, it runs on the assumption that an agent's handle time last month likely predicts their handle time this month. Conditioning on it strips out the between-agent variance that has nothing to do with the feature you’re testing. This is the single cheapest power gain available in a small sample, and it's what stratification is doing at design time. ANCOVA does it again at analysis time, with the added benefit of catching any in-group imbalance that stratification missed;
• Difference-in-differences: the fallback when the groups aren't comparable in levels, only in trend. You're not randomizing (a limited rollout, e.g. a per-team or per-business unit split), so groups start at different baselines; DiD compares the change in each group instead of the level, which cancels any fixed difference between them. However, you must be sure the groups have been moving together before the intervention (parallel trends assumption), and you should show that with pre-test data beforehand rather than assert it;
• Bootstrap: the choice when the metric's shape breaks the standard error, not when the sample is small. For example, ticket handle time is heavy-tailed and truncated at zero, and metrics like occupancy are ratios rather than means, so the usual t-based interval is quoting a precision it doesn't have. Resampling builds the interval from the data's own distribution instead. Resample at the level you randomized — whole agents or agent-days.

One caveat is that none of these statistical methods manufacture power. What they do is remove specific nuisance variance and give you truthful intervals, sometimes wider ones.

You still need to talk to the users​

While having a small population may be a clear constraint for testing and a headache for an analyst, for a Product Manager it can be an advantage that will be a great addition to the testing data. I’ve already mentioned “The Mom Test” and how customer interviews can provide you with great insights if you know what and how to ask. That’s product management 101. However, there’s also another advantage you get with direct access to your users, and that’s showing you where you may have messed up the test design.

One time, feedback from our customer support helped us understand that the feature’s poor test metrics and low adoption were not due to a design flaw or a bad feature, but due to the fact that no one knew about its release, and the on-screen prompts to use it were too easy to overlook.

Another time we were testing a new queue distribution mechanism, and only with the help of our users were we able to understand that a new, faster queue was prioritizing older, more complicated tickets, and that appeared as a dramatic AHT increase in the test group in our dashboards. What made it a test design failure is that both groups drew from the same backlog — so the test group's speed changed what was left over for control, and the two groups were no longer independent. And whenever your groups share a finite pool (a queue, inventory, a lead list), randomizing within it may not yield the results you expect.

Conclusion​

Making data-driven decisions in a small sample is possible, if you follow these simple rules:

• Smaller sample usually means more sensitive metrics, so the test design and data interpretation carry the weight that sample size normally would;
• If you can't run a clean randomized A/B test, run a designed alternative, and gather as much prior data as possible, so that you have a defensible dataset, even if you’re doing a before-after comparison;
• You still need to talk to your users. In small-sample environments qualitative work isn't a softer supplement. You need both the numbers and the real feedback to get the full picture of what’s going on with your product.

Did you have experience testing your new features on a smaller sample? Share your thoughts!