· @u_gS85FJSu6MpPror4Z7ilAA
Our own sample could not have detected an edge smaller than +0.054 AUC, and the edge it measured was +0.048. The result sits below the detection floor of its own test.
That floor comes from seven years of out-of-sample data — fourteen six-month purged walk-forward windows — at 80% power, after correcting for the fact that 149 factors were searched. The correction is what moves it: scanning 149 factors puts an honest 5% bar at 3.59 standard errors instead of 1.96. At that bar the measurement is not evidence of an edge, and it is not evidence against one either. The data cannot separate the two cases.
This is the opening worth using, because none of it judges his model. It says the sample is too short to settle the question, which is a statement about the geometry of weekly data with a multi-week forward label — his 20 factors sit on the same geometry. A 20-factor search, incidentally, puts the bar at 3.02 standard errors: almost exactly the 3σ convention, arrived at from his own numbers rather than borrowed from physics.
The noise unit is the period, not the row. Each fold returns one out-of-sample AUC, and the spread of those fourteen numbers is the only estimate that carries regime variation.
| Quantity | Value |
|---|---|
| Out-of-sample windows | 14 six-month folds (7.0 years) |
| Mean fold AUC | 0.5480 — edge +0.0480 |
| Period-to-period sd | 0.0458 |
| Standard error of the mean | 0.0122 |
| t against AUC 0.5 | 3.92 on 13 df, two-sided p = 0.0018 |
| Sign test | 10 of 14 folds above 0.5, p = 0.18 |
| Hanley–McNeil se from row counts | 0.0035 — 13.2× smaller |
The two tests disagree, and he should hear it from us rather than find it himself. A t-test on the fold means gives p = 0.002, which reads as a real edge. The distribution-free sign test gives p = 0.18, which does not. The gap is carried by the two highest folds, 2019 H2 at 0.616 and 2020 H1 at 0.637 — the run into the 2020 crash.
Dropping those two by name, which is a sensitivity check and not a result, leaves twelve folds at mean 0.5350, edge +0.0350, sd 0.0341, t = 3.55, p = 0.0045. So the edge is not purely a crash artefact; it is smaller and steadier than the headline number suggests, and still below the multiplicity-corrected floor for twelve folds (+0.044).
The honest summary is narrower than "it does not work": by a conventional test the edge is distinguishable from chance, by a search-corrected one it is not, and at +0.05 AUC it is too small to survive costs in either case.
Every curve falls as the square root of elapsed time, so tripling the run from seven years to twenty cuts the floor by less than half. The accent line crosses the measured edge at roughly 17 periods — another eight and a half years of out-of-sample data before the result we already have would clear its own bar.
Read it the other way round: fix the edge you believe in, and the protocol tells you how long before it can be shown. Cells are six-month periods, with years in brackets, at 80% power.
| True edge | 5% two-sided | After a 149-factor search | 3σ | 5σ |
|---|---|---|---|---|
| AUC 0.52 | 42 (21.0 y) | 103 (51.5 y) | 78 (39.0 y) | 180 (90.0 y) |
| AUC 0.53 | 19 (9.5 y) | 46 (23.0 y) | 35 (17.5 y) | 80 (40.0 y) |
| AUC 0.55 | 7 (3.5 y) | 17 (8.5 y) | 13 (6.5 y) | 29 (14.5 y) |
| AUC 0.60 | 2 (1.0 y) | 5 (2.5 y) | 4 (2.0 y) | 8 (4.0 y) |
The practical consequence for designing the test: a two-year blind run is four periods, and four periods cannot resolve anything below about AUC 0.59 at a 3σ bar. So either the claimed edge is large, or the test is long, or it returns nothing — and which of the three it will be is known before anyone starts, not after.
One honest caveat to offer him. Every figure here scales with the period-to-period dispersion, which is 0.0458 in our data. If his signal holds steadier across regimes than ours did, his dispersion is smaller and every duration shrinks with its square. That is the one number that could move these estimates in his favour, and he can measure it from his own walk-forward in an afternoon.
This is the point most likely to be argued with, so it is worth having ready. Our pooled test set is 47,470 rows. Computed from those row counts, the textbook Hanley–McNeil standard error of the AUC is ±0.0035. The actual dispersion of the fourteen period results is ±0.0458 — thirteen times larger.
The row-count formula assumes the rows are independent draws. They are not, twice over. A 26-week forward label sampled weekly overlaps the next week's label in 25 of 26 weeks, so the rows within a window are nearly the same observation repeated. And every name in that window shares one market factor, so the ~700 stocks are not 700 independent bets either. Nine years of weekly data yields roughly 18 genuinely separate outcomes per name.
So a six-month window behaves closer to one draw than to 3,400. Doubling the universe doubles the rows and leaves the standard error that matters almost untouched; the only thing that moves it is more periods, and periods arrive at two a year. That is why the protocol question is not "what model" but "how long", and it is why a result quoted with a row-count error bar is quoting a number thirteen times too small.
He is sending more years and fewer columns. The two pull in opposite directions, and only one of them changes what a test can decide.
The extra years are the valuable half, because periods are the only currency. Against our dispersion of 0.046, after the search correction:
| Out-of-sample run | Smallest edge it can resolve |
|---|---|
| 14 periods, 7 years — today | +0.054 |
| 18 periods, 9 years — two more | +0.048 |
| 24 periods, 12 years — five more | +0.041 |
| 34 periods, 17 years — ten more | +0.035 |
Ten extra years of history takes a third off the floor. That is the honest return: real, worth having, and not a solution on its own. The question to put to him about it is whether the values are point-in-time — as they stood on the date, not as restated since — and whether names that left the universe are still in it. Backfilled history that fails either test adds periods and quietly adds signal that was not knowable at the time.
The column cut helps the model more than it helps the test. Our own correlation file supports his instinct: raw and z twins run from 0.90 to 0.997 — pe_ttm against z_pe_ttm at 0.950, sales_ttm_gr at 0.958, eps_ttm_pyq_accel at 0.997. One of each pair is close to redundant.
Two things follow, both worth saying. The multiplicity relief is smaller than the column count suggests: two factors correlated at 0.99 were never two independent tests, so Bonferroni counted them as two where reality counts them as about one — cutting them lowers the nominal bar and barely moves the true one. And at 0.90 the twins are not interchangeable, since a fifth of one's variance is missing from the other, so which one he keeps is a real choice. If that choice gets made after seeing which version scores better, it is one more hidden search, and it belongs in the frozen rule rather than in the analysis.
Where the cut genuinely pays is the modelling: collinear predictors create posterior ridges that a diagonal mass matrix cannot precondition — we dropped 15 columns at |r| ≥ 0.95 for that reason alone — and they are a large part of why coefficients flip sign between folds.
Still worth asking for, and cheaper than either: his out-of-sample metric fold by fold from the walk-forward he already has. One column of numbers, disclosing nothing about the factors, that sets the dispersion every duration here scales with. If his is 0.030 rather than our 0.046, then with 20 factors a claim of AUC 0.55 needs 6 periods, three years, where ours needed 17 and eight and a half. His data decides which of those two conversations the blind test is.
Three questions, sent to Marc on 21 October ahead of the call, so it can open from his answers instead of spending its first ten minutes collecting them. They are diagnostic, and each one can be answered without disclosing anything proprietary.
What is the purge gap between train and test? With a multi-week forward label and weekly sampling, no gap means the validation is partly reading its own answer. This decides whether the selection result exists before any timing layer goes on it.
How many factors were tried in total, not kept? Twenty kept out of twenty is a 3.02 bar. Twenty kept out of two hundred tried is a different number, and it is the one that sets the threshold.
What is the period-to-period dispersion of his own out-of-sample metric? One number, measurable from the walk-forward he already has, and every duration above rescales with it.
Then the protocol, which is the part worth the call. Physicists built it for this exact failure mode — too many ways to analyse the data, too much incentive to keep looking until something appears. Muon g−2 kept the key number behind an offset held outside the analysis team. LIGO ran blind injections, and the collaboration wrote up a detection that turned out to be planted.
Freeze the rule in writing first. Universe, rebalance, costs, the metric, and the number that counts as failure. Dated before anything is scored. Declaring in advance what would falsify the claim is the whole discipline.
Run the power calculation before the test. The tables above, with his dispersion and his factor count. If the test cannot return a verdict in the time available, better to know now than in two years. The tool behind those tables, power.cpp, takes the dispersion and the factor count as switches, so if he gives you both numbers on the call the tables can be re-run against his model rather than ours.
Injection and null tests before any real holdout. A third party hands over surrogate datasets — some carrying a synthetic edge of known size, some with the edge destroyed by permutation — without saying which. Miss the planted edge and the pipeline is not sensitive enough to trust; find one in the permuted set and that is its false-positive rate. This is the step that can vindicate his model rather than merely failing to condemn it.
Then the sealed holdout. Predictions submitted as a file, scored by someone with no stake. One shot: changing the model afterwards starts a new experiment on fresh data.
The whole of step 1 can be one sentence, and it is worth getting him to say it in his own numbers rather than agreeing to ours:
Scored on the rule we freeze today, over the next ____ six-month windows my model's AUC will average at least ____, and no single window will fall below ____.
The third blank is the one that matters, because it is his dispersion claim stated as something that can fail. A low period-to-period spread is not only good news about the test's length — it is an assertion that the edge holds across regimes, which is exactly what ours did not do: our fourteen windows ran from 0.487 to 0.637 around a mean of 0.548.
It also makes a short test useful immediately. Four windows cannot confirm his average — the tables above say so — but one window below his floor refutes the stability claim outright. That asymmetry is worth pointing out to him as a feature: he gets a result years before he gets a verdict.
If he declines to name a floor, that is informative too, and not a gotcha: it means the stability claim is doing no work, only the average is under test, and the duration comes straight off the table.
The chatbot belongs in the control arm, not in the prediction. Marc has been asked to put the same question to jimmychat.ai. It cannot make the commitment: it has never seen his factors, his folds or his dispersion, it carries no cost for being wrong, and if the test fails it gives him somewhere to put the blame. Pointed the other way, though, it is a real instrument — step 3's injection logic aimed at the forecaster instead of the pipeline. Ask it cold, record the answer, and if his three numbers and its three numbers land in the same place, that is evidence his figure came from the ambient distribution of plausible finance numbers rather than from his own walk-forward.
The order decides whether any of that holds. His numbers have to be written down and sent before he reads the machine's, or he has been anchored by it and both answers are worth less — the same reason not to offer him 0.55 and 0.52 first. If the sequencing cannot be rescued, the call should say so plainly and treat his figure as anchored rather than quietly use it as though it were independent.
Two things to say out loud. A historical holdout is a weaker blind than a physics one, because both of you lived through 2020 and 2022 and remember how they ended — the only strict blind is genuinely future data. And the offer should be symmetric: our next claim goes through the same protocol, same custodian. Proposing a standard you have not agreed to submit to is an audit, and it will be heard as one.
If you offer him the write-up afterwards, it now says the same thing the call does. The findings memo's evidence section reads all below the detection floor, and the paragraph under it carries the same +0.054 against +0.048, with the two disagreeing tests stated rather than smoothed over. Handing him a document that still said "at chance" while the call said "below the floor" would have cost more than it gained.
What not to open with: the chatbot. The critique of that answer is correct but it is a side argument, and leading with it makes the call about his judgement in forwarding it rather than about the test.