We tested three things: whether the word-frequency database predicts Eric's calibration scores, whether it predicts the scores in the consolidated crossword list, and whether that crossword list predicts Eric's scores.
Short answer: no. Frequency data ranks Eric's list only weakly (correlation about 0.4) and does no better than "accept everything" at a 50 cut. On the crossword list it runs backwards: the more common a string, the lower its score. The crossword list's score says nothing about Eric's (correlation 0.02); only being on the list at all helps, by about 8 points.
Each scored item was looked up in word_frequency.db (21 million entries). Its row gives 21 features: counts in Google Web, Google Books (2009, 2012, 2020), Usenet and OpenSubtitles; membership of the canonical list, Wiktionary, BNC, Broda, XWord Info and Gooch's BIGDICT; Gooch's dictionary codes (OED, Chambers, Web2, OSPD, place, phrase); and the database's English probability. A ridge regression maps these to the 0-100 score. Every prediction is made by a model that never saw that item (leave-one-out for Eric's list, 10-fold for the crossword list). The baseline predicts the average score.
167 items; 131 have a database row. The other 36, mostly spaced phrases such as bit the bullet and Nobel Peace Prize, cannot be scored at all.
| Measure | Model | Baseline |
|---|---|---|
| Rank correlation with Eric | 0.43 | — |
| Variance explained (R²) | 0.20 | 0 |
| Mean absolute error | 6.5 | 7.4 |
| Right at the 50 cut | 64% | 66% (accept all) |
| Right at the 60 cut | 79% | 76% (reject all) |
The best single signals: membership of the canonical list (+0.38), Usenet count (+0.37), number of sources (+0.36). The full model barely improves on any one of them. Its predictions are compressed: about 52 for items Eric scored 40-49 and about 57 for 70-79.
| Largest misses | Eric | Model |
|---|---|---|
| bigamy | 90 | 61 |
| werewolf | 86 | 63 |
| offboarding | 70 | 48 |
| liveblog | 17 | 56 |
| rajbansi | 26 | 53 |
Weak ranking, no use as an accept/reject gate.
The consolidated crossword list (Broda + XWord Info, 556,686 entries); 530,230 have a database row. Broda and XWord Info membership were left out, since they are the source of the score.
| Measure | All entries | Without the default 50s |
|---|---|---|
| Entries | 530,230 | 216,658 |
| Rank correlation | 0.31 | 0.38 |
| Variance explained (R²) | 0.09 | 0.20 |
| Mean absolute error (baseline) | 7.6 (7.9) | 11.6 (12.5) |
| Separating 50+ from below (AUC) | 0.75 | 0.83 |
56% of the list sits at exactly 50, which looks like a default, so the second column leaves those out. Every frequency feature correlates negatively with the score (total count −0.20, Google Books 2020 −0.21): the lowest-scored entries are common words editors reject (raped, bugger) and the highest are lively phrases the corpora barely contain. Predictions stay between 61 and 64 for every true band from 50 to 100.
Frequency can weakly flag bad fill. It cannot rank good fill.
67 of the 167 items are on the crossword list.
| Items | Eric's mean | |
|---|---|---|
| On the crossword list | 67 | 56.5 |
| Not on it | 100 | 48.7 |
Among the 67, the correlation between the list's score and Eric's is +0.02: none. His mean stays between 51 and 60 in every band of list score.
| Word | Eric | Crossword list |
|---|---|---|
| bigamy | 90 | 50 |
| werewolf | 86 | 50 |
| fatberg | 45 | 75 |
| snowclone | 40 | 70 |
| liveblog | 17 | 64 |
Adding the crossword list to the frequency model changes nothing (rank correlation 0.35 vs 0.36, R² 0.16 vs 0.15, on all 167 items).
Membership of the crossword list is worth about 8 points. Its score is noise against Eric's.
131 and 167 items make the results on Eric's list noisy. The regularisation strength was chosen on the same held-out error that is reported, which flatters the models slightly. The crossword list's scores rate crossword fill, which is a different target from Eric's wordplay grades. Tools: freqmodel.cpp and xwjoin.cpp in the wordlist project.