For what it's worth, I took the SEDE query posted earlier in the comments and tweaked it to generate some statistics on the historical effectiveness of the proposed autofilter.
Specifically, my query returns a monthly count of:
- the total number of questions selected for HNQ in that month,
- the number of those questions that were either closed or manually removed from HNQ by a moderator, and
- both of the above, but limited only to questions whose title (at the time they were picked for HNQ) would pass the proposed autofilter.
Anyway, looking at the results, I'm not too hopeful about the effectiveness of this proposed filter in keeping bad questions out of HNQ:
As you can see from the graph, the autofilter only matches about 10% of all HNQs on math.SE. Furthermore, it appears to be matching roughly the same fraction of both good and bad questions (or, at least, of questions that were and weren't closed or manually removed from HNQ in the past).
In other words, at least based on this historical data, the proposed filter — regardless of its other merits, or lack thereof — would appear to have no significant effect on the fraction of HNQs that are closed or manually removed.
(Of course, the query can't tell us what other questions might have been selected to replace the 10% that would've been filtered out, had the autofilter been in place when they were hot. However, it seems reasonable to assume that those questions would not have been statistically too dissimilar from the ones that were actually selected.)
Ps. If you'd like to examine the underlying data in more detail, here's a version of the query that returns the raw list of past HNQs and whether or not they match the filter.
Pps. Here's a version of the query the returns percentage statistics on the fraction of questions closed or removed from HNQ among those that either pass or match the proposed autofilter:
Of course, there's no actual new data here — you could easily generate these percentages just by dropping the raw question counts from the first query above into a spreadsheet — but this might provide a useful alternative visualization.
As this graph shows, the questions that match the proposed filter aren't historically any more likely to get closed or removed from HNQ than those that don't, at least not until the last couple of months. Since February 2020, questions matching the filter do seem to get removed from HNQ a bit more often than those that don't, but this could be just a statistical coincidence. (It also seems to correlate with an uptick in the number of questions removed from HNQ in general.)
Even within the last few months, however, automatically applying the autofilter would not have made much difference to the overall removal rate: the yellow and blue lines track each other very closely over the whole period for which data is available. At best, in March 2020, having the autofilter in place would've reduced the manual HNQ removal rate from 87% to 85%.
(Again, this is assuming that the questions that pass the filter and did get selected as HNQs are also statistically representative of the questions that would've been picked to replace the ones filtered out, which cannot be proven but seems like a reasonable assumption.)
Ppps. Just for the sake of curiosity, prompted by Paul Plummer's comment below, I did a little bit of exploratory data analysis using the data from the detailed query above. In particular, I tried looking for words* that have occurred only in the titles of removed (or closed) HNQs, and which have occurred in more than two such titles. Here they are, grouped by the number of such titles they occur in:
- 7: delta
- 5: h, l, students, let
- 4: rolls, whole, distributing, separable, extended, trace, start, reciprocal, letter, hold
- 3: finish, proposition, require, i'm, seeking, negation, morphism, add, puzzle, cut, entries, direction, associative, roll, symmetry, eight, variation, symbol, moves, average, english, squeeze, clear, cancellation, uncountably, hypothesis, physics, towards
So, what does that list tell us? I'd say not much, except that it's kind of hard to reliably detect bad HNQs using word filters.
Would we want to add an autofilter for "delta"? Maybe. Doing so would've saved us from a whole seven (presumably) low-quality HNQs over the past year or so. Adding in "let" and "students" would increase that to 17, assuming that none of those three words have occurred in the same title (which I haven't checked).
On the other hand, I'd assume we probably don't want to autofilter "separable", "extended" or "trace" (or the letters "h" and "l"), even though statistically those all seem to be better indicators for bad HNQs than any of the proposed filter keywords above.
*) For the purposes of this exploratory study, I'm defining a word as a sequence of the letters A–Z, digits 0–9 and apostrophes. All titles were converted to lower case before analysis.
FWIW, I also looked at which words would appear to be the best overall predictors of a HNQ getting removed or closed, using add-one additive smoothing to discount infrequently occurring words. (In other words, I ranked the words based on the score $(1 + n_b) \mathbin/ (2 + n_b + n_g)$, where $n_b$ and $n_g$ are the number of "bad" and "good" titles the word occurs in.) Here are all the words with score > 0.75:
- delta: 0.89 (7 bad, 0 good)
- h, l, students, let: 0.86 (5 bad, 0 good)
- rolls, whole, distributing, separable, extended, trace, start, reciprocal, letter, hold: 0.83 (4 bad, 0 good)
- expectation, properties, 20: 0.82 (8 bad, 1 good)
- finish, proposition, require, i'm, seeking, negation, morphism, add, puzzle, cut, entries, direction, associative, roll, symmetry, eight, variation, symbol, moves, average, english, squeeze, clear, cancellation, uncountably, hypothesis, physics, towards: 0.8 (3 bad, 0 good)
- verify, column, school: 0.78 (6 bad, 1 good)
(The reason for setting the cutoff at > 0.75 is that after that comes a long list of words that each appear in exactly two "bad" HNQ titles and no "good" titles. Most of those look even more like just random coincidences than the words already listed above.)
Not too surprisingly, this looks very similar to the previous list. The only difference is the addition of the words "expectation", "properties", "verify", "column", "school" and the number "20", all of which appear in the titles of sufficiently many HNQs that were removed or closed to counterbalance their appearance in the title of one HNQ that wasn't as well.
FWIW, trying add-two additive smoothing instead (to favor common words over rare ones even more) still gives the same old list of usual suspects as before, but also brings up a few more words that occur notably more often in the titles of HNQs that get removed or closed than of those that don't, such as "true" (17 bad, 5 good), "countable" (14 bad, 4 good), "contains" (8 bad, 2 good), "three" (22 bad, 8 good), "up" (10 bad, 3 good) and "determinant" (19 bad, 7 good), as well as "maximal", "rotation", "gives", "epsilon", "black" and "sine" (5 bad, 1 good each).
I'll leave it up to the reader to decide whether any of those words would actually make good autofilter entries.