Column Shortlist Review

Oct 9, 2026 · Chris Cole

Ship the 92-column shortlist, with three changes: restore the five columns dropped as dead, cut ret_best_hz from the target set, and treat the 2017–2026 scoring window as the build's main risk rather than its history as a free win. The redundancy pruning and the cross-sectional return target are the strongest parts of the spec and should go through unchanged.

Keep unchanged

Four parts of the spec are sound and should go through untouched.

The redundancy pruning is the most defensible work in the document. Fifteen columns dropped at |rho| between 0.93 and 1.00, including z_ev2s against z_ps_ttm at rho = 1 and eps_q2_out against eps_q2_out_28d at rho = 1. This is pure dimensionality reduction and makes no predictive claim, so it cannot be wrong in the way a ranking can.

The search-space framing is correct and is the thing most reviews get wrong. The spec says plainly that no single column is strong, that AUCs cluster at 0.44-0.55 and ICs at 0.04-0.14, and that the list is therefore a search space rather than a feature set. It then draws the right conclusion: do not prune to the top 15 by solo scores, because the interaction features carry the model and score near the bottom univariately. That matches what our own testing found.

Cross-sectional z-scoring is the right primary target. z_ret_26w_xs removes the market move and leaves relative return in units of sigma, which is the quantity the model is judged on. The accompanying warning not to standardise pooled is equally important: pooled standardisation leaves beta in the target and the model learns to predict the tape.

Purged and embargoed walk-forward is non-negotiable. A 26-week label overlaps 26 weeks of subsequent rows, so without purging and an embargo the test set contains the training label. This is the control that caught our earlier false positives.

Cut ret_best_hz from the target set

ret_best_hz = max(13w, 26w) is a hindsight target and will overstate whatever the model achieves.

To realise that return you would have to know, at entry, which of the two horizons was going to be the better exit. Nobody does. A model trained to predict the better of two outcomes is scored on information unavailable at decision time, so its measured performance is not a performance anyone can capture.

The spec labels it "validated best", which is how it earned a place in the file. It validates well for the same reason it is unusable: taking a per-row maximum over two horizons raises the target's mean and lowers its variance, so any predictor of it looks stronger.

We have already paid for this exact error class once. A metric that averaged per-episode means inflated a result to 0.593; the honest row-level figure was 0.5055 with clustered standard errors, and the apparent effect disappeared. The shape of the mistake is the same: the metric quietly selected on the outcome.

Recommendation. Keep z_ret_26w_xs as the primary target and ret_26w_w as the stable regression target. Keep ret_pctile_26w for learning-to-rank. Either drop ret_best_hz or carry it clearly marked as a diagnostic that must never appear in a reported result.

Restore the five columns dropped as dead

The drop rule contradicts the argument the spec makes two paragraphs earlier, and it was not applied consistently.

The stated rule is that a column scoring below 0.03 composite on both objectives is dead. But rank_t_price_str_vs_market is kept at a composite of 0.022, below that threshold, explicitly because the GA leans on it. The spec is right to keep it. That is the point: once you accept that a column scoring near zero solo can carry the model in combination, a solo score can no longer justify a drop either.

No interaction test was run on the five columns that were discarded: earnings_consist_past, rank_e_analyst_opinion, rank_e_insider_activity, z_rv_cy_4wk, rank_e_price_str_vs_industry. They were cut on exactly the measure the document argues cannot rank columns.

The cost of restoring them is close to nothing. At 92 columns, five more is a 5% increase in search space for a tree or GA that is already selecting within 92. The cost of being wrong the other way is losing a feature that only pays off in combination, which is the failure mode the spec itself warns about.

Recommendation. Restore all five, making it 97 columns. Keep the 15 redundancy drops, which rest on correlation rather than on a weak score.

The regime mismatch is the build's main risk

The columns were scored on 63,719 resolved rows from 2017-2026 and will be used on a 2000-2026 build. That is selection bias the spec does not flag, and it cuts against the document's own headline.

2017-2026 is a historically unusual stretch: one long bull with two short interruptions. Columns chosen for their behaviour in that window are being asked to work across 2000-02 and 2008. A feature set selected in a bull, gated on a bull universe, is most likely to break precisely in the regimes being added.

This matters for what the build is expected to show. The spec argues that adding 2000-02 and 2008 raises breadth because they are low-correlation regimes. That is true for breadth, but the same property means they are out-of-sample for every selection decision made here. The honest prediction is that measured sigma falls when those years are included, and that this is the system working rather than failing.

Recommendation. Re-score the shortlist on the 2000-2026 data once it exists, before the physics model is tuned on it, and compare the two rankings. If the A/B/C tiers reorder substantially across regimes, that is the most useful single finding the longer history can produce, and it is cheap to obtain. Treat the current tiers as provisional until then.

On 2 sigma to 3 sigma

The formula is right; the assumption underneath it is the problem. IR = IC x sqrt(breadth) holds, but it is a statement about a real IC, and in this feature set we have not been able to demonstrate one out of sample.

TestResult
100 published signal rules, out-of-sample check95 of 100 had zero out-of-sample support
Exhaustive search, 388k hypotheses, best survivorPGR >= 82 and ProjPE >= 24 cleared three windows, then failed on 2024-26 against independent prices at lift -0.003
Trend-exhaustion effect, re-measured at row level0.593 fell to 0.5055 +/- 0.0024 with clustered errors; not significant 2016-19
Return targets, both engines, frontier scanAUC <= 0.55 everywhere
4-week CCI reversal, conditioned event2 sigma reached on 444 episodes, but the economic audit was negative
Any 26-week positive, next windowEvery one died

Two specific corrections to the argument.

A richer target raises measured IC, not necessarily realised IC. Moving from one bit to a continuous return lets the model fit magnitude, which includes magnitude noise. If the underlying signal is absent, a continuous target buys a more precisely measured zero. Calling this "free signal you are currently throwing away" is the one line in the spec I would strike.

Breadth counts independent bets, not years. 2000-2026 is roughly three times the weeks, but the sqrt(breadth) term rewards independent bets per period multiplied by periods. Adding regimes genuinely helps generalisation, which is the real reason to build the longer file. It is not a reason to expect sigma to rise.

None of this says do not build it. A magnitude target over 2000-2026 is a better experiment than a boolean over 2017-2026, and it is worth running for that reason. It is simply not a route to 3 sigma that can be assumed in advance.

Actions before the build

ItemActionWhy
15 redundancy drops (rho >= 0.92)Keep as droppedCorrelation, not a weak score; no predictive claim to be wrong about
5 columns dropped as deadRestore, giving 97Cut on the one measure the spec argues cannot rank columns
92 to 97 column listShip as the search spaceRight call, and the spec defends it correctly
ret_best_hz = max(13w, 26w)Remove, or mark diagnostic-onlyHindsight target; the exit horizon is not knowable at entry
z_ret_26w_xsMake it the primary targetRemoves beta, expresses the result in the units being judged
ret_26w_w, ret_pctile_26wKeepStable regression and ranking targets
Per-week standardisationEnforce; reject pooledPooled leaves beta in the label
Purged + embargoed walk-forwardRequired for any reported number26-week labels overlap 26 weeks of later rows
A/B/C tiersTreat as provisionalKeep the tier labels; ignore the rank order inside a tier - see the standard errors below
Shortlist re-score on 2000-2026Do it before tuning, compare rankingsTests whether the selection survives regimes it was not scored in
Point-in-time membership and fundamentalsConfirm on your sideLargest bias risk across a span as long as 2000-2026

What the standard errors say

Measured on weekly.csv: the bull universe holds 64,203 resolved rows, 18,442 positives and 45,761 negatives, a positive rate of 28.72%. Under the null AUC = 0.5 the Mann-Whitney variance is exact, (n1 + n0 + 1) / (12 n1 n0), giving SE(AUC) = 0.00252. For the IC side, SE = 1/sqrt(n) = 0.00395.

Three consequences, and the middle one corrects an impression the scores invite.

The columns are individually real. 83 of the 92 kept columns have |AUC - 0.5| beyond two standard errors, and 85 have |IC| beyond two. Only two are indistinguishable from noise on both measures. The spec's description of faint but genuine signal is accurate - these are not dead columns, and an AUC of 0.444 carries as much univariate information as 0.556 with the sign reversed.

The rank order inside the list is mostly not resolvable. Two columns need an AUC gap of 0.00713 to be ordered at 95%. Across the 91 adjacent pairs in the composite ordering, 64 - that is 70% - fall below that gap. So the sequence of columns within a tier carries no information, and the composite's third decimal place certainly does not.

The A/B boundary itself is resolvable, which I had assumed it would not be. mf_indy_avg (composite 0.260, AUC 0.455, |AUC-0.5| = 0.045) against z_price_sma90 (composite 0.250, AUC 0.468, |AUC-0.5| = 0.032) is a gap of 0.0130, comfortably above the 0.00713 needed. The tier labels are therefore more defensible than the ranks are.

What follows. Keep the tiers as a coarse three-way split and do not read the ordering inside them. If the list is ever narrowed, cut on tier rather than on rank, and bear in mind that none of this measures the interaction value the GA actually exploits - rank_t_price_str_vs_market sits at AUC 0.499, indistinguishable from a coin, and is one of the two columns the best precision model leans on.