D
David Taiwo Balogun
Guest
Dropping every feature whose correlation with the target is below 0.1 feels like hygiene. It is often the opposite: you are deleting signal your model could have used, and you will never see it in your metrics because the comparison you would need is the one you never ran.
Most of us learned feature selection from a heatmap. Compute Pearson's r between each column and the target, sort, keep the top k, move on. It is fast, explainable to a stakeholder in one slide, and wrong often enough to matter.
Pearson's r answers a narrow question: if I draw the best straight line through this feature against the target, how tightly do the points hug it?
r_{XY} = \frac{\operatorname{cov}(X, Y)}{\sigma_X \, \sigma_Y}
That definition has four blind spots, and each one maps to a way you lose information:
1. It is linear. A feature with a perfect U-shaped relationship to the target can score r ≈ 0.
2. It is marginal. It looks at one feature at a time, so it cannot see signal that only appears when two features are combined.
3. It is fragile. A handful of outliers can inflate or flatten r, and it says nothing useful about categorical or heavy-tailed features without extra work.
4. It confuses relevance with uniqueness. Two near-identical features both score high, so a top-k filter happily keeps both copies while discarding a weaker but genuinely new signal.
Spearman's rank correlation fixes some of the linearity problem (it captures any monotonic trend), but it is still marginal and still blind to non-monotonic shapes. Swapping one coefficient for another does not solve the structural issue.
Suppose y = x² + noise, with x symmetric around zero. Every positive x has a mirror negative x that produces the same y, so the linear trend cancels out and r lands near 0. A tree model or even a squared term would predict y almost perfectly from x. Real examples: customer age vs churn (young and old churn more), temperature vs energy demand (heating and cooling), transaction hour vs fraud.
The classic case is XOR: y = 1 when exactly one of a or b is positive. Individually, each feature has zero correlation with y. Together, they determine it completely. A marginal filter drops both, and the model never gets a chance to learn the rule.
This is not a toy problem. In credit and fraud work, "high amount" means little on its own but a lot when combined with "new device" or "first transaction with this merchant".
A suppressor is a feature with little or no correlation with the target that makes other features more predictive by removing noise from them. Example: a test score s measures ability plus test-taking speed, and you also have a pure speed measure v. v alone barely predicts job performance, but including it lets the model subtract the speed component from s, sharpening the signal. Filter out v and s gets noisier.
If income_monthly and income_annual both have r = 0.6 with the target, a top-k filter keeps both and spends two slots on one piece of information. Meanwhile, months_at_address with r = 0.15 adds something neither income column knows, and it gets cut.
No single method is "the right one". Each answers a different question, so the useful move is knowing which question you are asking.
A few notes on using them well:
• Mutual information is the natural upgrade from correlation as a cheap filter. It is zero only when the feature and target are truly independent, so the U-shape case scores high. It is still marginal, so XOR features still look useless.
• Permutation importance should be computed on held-out data, not training data, or it will reward features the model overfit to. With correlated features, importance gets split between them, so a feature can look weak because its twin is carrying the load.
• SHAP is the best tool for understanding why a feature matters, but it explains a model, not the world. A biased or leaky model produces confident, misleading SHAP values.
• Boruta and stability selection are what you reach for when the selection itself needs to be defensible: they ask whether a feature survives noise and resampling, not just whether it ranked well once.
On a synthetic dataset with one linear signal, one U-shaped signal, and one XOR pair, a correlation filter (|r| > 0.1) keeps only the linear feature and its duplicate. A random forest trained on those scores R² = 0.006. The same model on all features scores R² = 0.846.
The target is built from three known effects plus noise, so we know the ground truth:
Output (scikit-learn 1.9, sorted by permutation importance):
Read the columns left to right, and you can watch each method's blind spots:
• Correlation ranks the two least useful signals first and calls everything else noise, including the feature that explains most of the variance.
• Mutual information rescues the U-shape, but still scores the XOR pair as worthless, because each half is independent of the target on its own.
• Permutation importance on a fitted model finds all three real effects and correctly puts noise at zero. Notice it also splits credit between linear and linear_copy, which is the redundancy caveat in action.
Use cheap filters to remove what is clearly useless or duplicated, then let a model decide what is relevant, and check that its choice is stable.
1. Split first. Hold out a test set before you compute any importance score. Selecting features on the full dataset and then cross-validating leaks the target into the selection step and inflates your scores.
2. Remove the obviously dead. Drop constant and near-constant columns, IDs, and anything that will not exist at prediction time.
3. Use correlation for redundancy, not relevance. Compute correlation between features. For clusters above roughly 0.9, keep one representative (or combine them). This is where correlation genuinely earns its place.
4. Screen with mutual information, generously. Rank by MI and cut only the bottom tail. Remember it misses interactions, so do not set an aggressive threshold.
5. Train a flexible baseline on what remains. Gradient boosting or a random forest is usually enough.
6. Measure model-based importance on held-out data. Permutation importance or SHAP, computed on the validation fold, not the training data.
7. Test stability. Repeat steps 4–6 across CV folds or bootstrap samples. Keep features that are selected consistently, and treat ones that flicker in and out as suspect.
8. Confirm with an ablation. Compare the cross-validated score of the reduced set against the full set. If the reduced set is meaningfully worse, you cut signal. If it is the same, you have earned the simpler model.
Three traps to watch for:
Feature selection is supposed to remove noise, not information. A correlation threshold removes whatever does not look like a straight line, and in real data a lot of the signal does not.
Keep correlation in your toolbox for what it is good at: spotting duplicated features and giving a quick linear sanity check. For deciding what your model actually needs, ask the model, on held-out data, more than once. The extra compute is small. The accuracy you stop throwing away usually is not.
If you have a war story about a feature you almost dropped, or one that turned out to be a leak, I would like to hear it in the comments.
Most of us learned feature selection from a heatmap. Compute Pearson's r between each column and the target, sort, keep the top k, move on. It is fast, explainable to a stakeholder in one slide, and wrong often enough to matter.
What correlation actually measures.
Pearson's r answers a narrow question: if I draw the best straight line through this feature against the target, how tightly do the points hug it?
r_{XY} = \frac{\operatorname{cov}(X, Y)}{\sigma_X \, \sigma_Y}
That definition has four blind spots, and each one maps to a way you lose information:
1. It is linear. A feature with a perfect U-shaped relationship to the target can score r ≈ 0.
2. It is marginal. It looks at one feature at a time, so it cannot see signal that only appears when two features are combined.
3. It is fragile. A handful of outliers can inflate or flatten r, and it says nothing useful about categorical or heavy-tailed features without extra work.
4. It confuses relevance with uniqueness. Two near-identical features both score high, so a top-k filter happily keeps both copies while discarding a weaker but genuinely new signal.
Spearman's rank correlation fixes some of the linearity problem (it captures any monotonic trend), but it is still marginal and still blind to non-monotonic shapes. Swapping one coefficient for another does not solve the structural issue.
Four ways a correlation filter throws away signal.
1. Nonlinear relationships
Suppose y = x² + noise, with x symmetric around zero. Every positive x has a mirror negative x that produces the same y, so the linear trend cancels out and r lands near 0. A tree model or even a squared term would predict y almost perfectly from x. Real examples: customer age vs churn (young and old churn more), temperature vs energy demand (heating and cooling), transaction hour vs fraud.
2. Interactions
The classic case is XOR: y = 1 when exactly one of a or b is positive. Individually, each feature has zero correlation with y. Together, they determine it completely. A marginal filter drops both, and the model never gets a chance to learn the rule.
This is not a toy problem. In credit and fraud work, "high amount" means little on its own but a lot when combined with "new device" or "first transaction with this merchant".
3. Suppressor variables
A suppressor is a feature with little or no correlation with the target that makes other features more predictive by removing noise from them. Example: a test score s measures ability plus test-taking speed, and you also have a pure speed measure v. v alone barely predicts job performance, but including it lets the model subtract the speed component from s, sharpening the signal. Filter out v and s gets noisier.
4. Redundancy masquerading as relevance
If income_monthly and income_annual both have r = 0.6 with the target, a top-k filter keeps both and spends two slots on one piece of information. Meanwhile, months_at_address with r = 0.15 adds something neither income column knows, and it gets cut.
Failure | What correlation sees | What the model could use |
Nonlinear (U-shape) | r ≈ 0 | Strong single-feature signal |
Interaction (XOR) | r ≈ 0 for each feature | Perfect signal from the pair |
Suppressor | r ≈ 0 | Improves other features' accuracy |
Redundant pair | High r, twice | One signal, counted twice |
A better toolkit.
No single method is "the right one". Each answers a different question, so the useful move is knowing which question you are asking.
Method | Question it answers | Catches nonlinear? | Catches interactions? | Cost |
Pearson / Spearman | Does this feature move linearly (or monotonically) with the target? | No / monotonic only | No | Very low |
Mutual information | Does knowing this feature reduce uncertainty about the target at all? | Yes | No (marginal) | Low |
mRMR | Which features are relevant and not redundant with ones already picked? | Yes (MI-based) | Partly | Low–medium |
L1 / Lasso | Which features does a sparse linear model need? | No (unless you engineer terms) | Only engineered ones | Low |
Permutation importance | How much does the trained model's score drop if this feature is scrambled? | Yes | Yes | Medium |
SHAP | How much does each feature contribute to individual predictions? | Yes | Yes (interaction values) | Medium–high |
Boruta | Is this feature more useful than a randomly shuffled copy of itself? | Yes | Yes | High |
Stability selection | Is this feature picked consistently across resampled data? | Depends on base selector | Depends | Medium–high |
A few notes on using them well:
• Mutual information is the natural upgrade from correlation as a cheap filter. It is zero only when the feature and target are truly independent, so the U-shape case scores high. It is still marginal, so XOR features still look useless.
• Permutation importance should be computed on held-out data, not training data, or it will reward features the model overfit to. With correlated features, importance gets split between them, so a feature can look weak because its twin is carrying the load.
• SHAP is the best tool for understanding why a feature matters, but it explains a model, not the world. A biased or leaky model produces confident, misleading SHAP values.
• Boruta and stability selection are what you reach for when the selection itself needs to be defensible: they ask whether a feature survives noise and resampling, not just whether it ranked well once.
A 40-line Experiment: Correlation keeps the wrong features.
On a synthetic dataset with one linear signal, one U-shaped signal, and one XOR pair, a correlation filter (|r| > 0.1) keeps only the linear feature and its duplicate. A random forest trained on those scores R² = 0.006. The same model on all features scores R² = 0.846.
The target is built from three known effects plus noise, so we know the ground truth:
Code:
import numpy as np, pandas as pd
from sklearn.feature_selection import mutual_info_regression
from sklearn.ensemble import RandomForestRegressor
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split, cross_val_score
rng = np.random.default_rng(42)
n = 5000
X = pd.DataFrame({
"linear": rng.normal(size=n),
"u_shape": rng.normal(size=n),
"xor_a": rng.normal(size=n),
"xor_b": rng.normal(size=n),
"noise": rng.normal(size=n),
})
X["linear_copy"] = X["linear"] + rng.normal(scale=0.05, size=n) # redundant twin
y = (
1.0 * X["linear"]
+ 1.5 * X["u_shape"] ** 2 # nonlinear
+ 2.0 * np.sign(X["xor_a"] * X["xor_b"]) # interaction
+ rng.normal(scale=0.5, size=n)
)
# Three views of "importance"
corr = X.corrwith(y).abs()
mi = pd.Series(mutual_info_regression(X, y, random_state=0), index=X.columns)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
model = RandomForestRegressor(n_estimators=300, min_samples_leaf=5, random_state=0, n_jobs=-1)
model.fit(X_tr, y_tr)
perm = permutation_importance(model, X_te, y_te, n_repeats=10, random_state=0, n_jobs=-1)
pi = pd.Series(perm.importances_mean, index=X.columns)
print(pd.DataFrame({"abs_corr": corr, "mutual_info": mi, "perm_importance": pi}).round(3))
def cv_r2(cols):
m = RandomForestRegressor(n_estimators=200, min_samples_leaf=5, random_state=0, n_jobs=-1)
return cross_val_score(m, X[cols], y, cv=5, scoring="r2").mean()
keep = corr[corr > 0.1].index.tolist()
print("Correlation filter keeps:", keep, "R2 =", round(cv_r2(keep), 3))
print("All features: R2 =", round(cv_r2(list(X.columns)), 3))Output (scikit-learn 1.9, sorted by permutation importance):
Feature | True role | abs(r) | Mutual information | Permutation importance |
u_shape | Nonlinear signal | 0.022 | 0.333 | 0.912 |
xor_b | Interaction | 0.011 | 0.000 | 0.411 |
xor_a | Interaction | 0.024 | 0.008 | 0.408 |
linear | Linear signal | 0.319 | 0.201 | 0.063 |
linear_copy | Redundant twin | 0.317 | 0.195 | 0.034 |
noise | Nothing | 0.015 | 0.010 | -0.001 |
Read the columns left to right, and you can watch each method's blind spots:
• Correlation ranks the two least useful signals first and calls everything else noise, including the feature that explains most of the variance.
• Mutual information rescues the U-shape, but still scores the XOR pair as worthless, because each half is independent of the target on its own.
• Permutation importance on a fitted model finds all three real effects and correctly puts noise at zero. Notice it also splits credit between linear and linear_copy, which is the redundancy caveat in action.
A workflow that keeps the signal.
Use cheap filters to remove what is clearly useless or duplicated, then let a model decide what is relevant, and check that its choice is stable.
1. Split first. Hold out a test set before you compute any importance score. Selecting features on the full dataset and then cross-validating leaks the target into the selection step and inflates your scores.
2. Remove the obviously dead. Drop constant and near-constant columns, IDs, and anything that will not exist at prediction time.
3. Use correlation for redundancy, not relevance. Compute correlation between features. For clusters above roughly 0.9, keep one representative (or combine them). This is where correlation genuinely earns its place.
4. Screen with mutual information, generously. Rank by MI and cut only the bottom tail. Remember it misses interactions, so do not set an aggressive threshold.
5. Train a flexible baseline on what remains. Gradient boosting or a random forest is usually enough.
6. Measure model-based importance on held-out data. Permutation importance or SHAP, computed on the validation fold, not the training data.
7. Test stability. Repeat steps 4–6 across CV folds or bootstrap samples. Keep features that are selected consistently, and treat ones that flicker in and out as suspect.
8. Confirm with an ablation. Compare the cross-validated score of the reduced set against the full set. If the reduced set is meaningfully worse, you cut signal. If it is the same, you have earned the simpler model.
Three traps to watch for:
- Leakage looks like importance. A feature that is suspiciously dominant (a timestamp, a post-outcome status field) is often a leak, not a discovery.
- Correlated features share credit. Permutation importance and SHAP can both make each twin look weak. Group them, or permute them together, before concluding either is unimportant.
- Importance describes your model, not causation. A feature can be highly predictive and still useless as a lever for intervention.
The Point
Feature selection is supposed to remove noise, not information. A correlation threshold removes whatever does not look like a straight line, and in real data a lot of the signal does not.
Keep correlation in your toolbox for what it is good at: spotting duplicated features and giving a quick linear sanity check. For deciding what your model actually needs, ask the model, on held-out data, more than once. The extra compute is small. The accuracy you stop throwing away usually is not.
If you have a war story about a feature you almost dropped, or one that turned out to be a leak, I would like to hear it in the comments.