Autofilters for Hot Network Questions

Question

The SE software allows us to request certain regular expressions be automatically excluded from the Hot Network Questions.

I was asked by the CMs to make this post on the meta, and include the list. This way the community can express agreement or disagreement, as well as suggest improvements.

Let me start by stating that the goal is to filter out bad titles. If it happens that some good questions get filtered out, it is unfortunate, but the default state of a question is off the HNQ, and the current consensus of the moderators is that it is more important to keep questions which we do not want to represent our site off this list.

With that in mind, note that sometimes good questions have bad titles. I still hope that people will push to edit and improve titles of questions just as well as their bodies.

One final clarification. There will be a zero tolerance policy towards people who make titles worse just to get a question on or off the HNQ. People who abuse this list will be punished.

Status of this thread: Discussion.

This thread is aimed to have a discussion phase (until April 30) $\to$ escalation phase $\to$ implementation stage.

---

• this
• problem
• question
• puzzle
• contest
• advice
• help
• please
• the following
• exercise
• easy
• opinion
• Solv[e|ing]
• difficult
• urgent
• tricky
• simplify

Answer 1

This request is, specifically, declined but we have added a subset of words to the HNQ filter just now.

Using the responses to the question asked by quid: Which words (if any) in the title of a question should prevent inclusion in the HNQ list? I've requested the top upvoted answers there to be added to the HNQ block and I've marked those words status-completed.

The filter regex looks like this:

(\W|^)(?i)(urgent|ASAP|please|pls|help)(\W|$)

Some things to note

• Unlike a moderator removal, editing the title will make a question eligible for the HNQ list the next time it compiles.
• Based on the excellent analysis by Ilmari Karonen, I'm not sure how impactful the filters we did add will be. There don't seem to be any super common words in titles that indicate problems.

Neither of these things are particularly surprising. The first is by design and the second... is designed less to prevent low-quality content getting in - which is what y'all are trying to do - and more to prevent abusive or offensive content from getting in.

If y'all have thoughts on better ways we could help address this issue, please let me know.

Answer 2

For what it's worth, I took the SEDE query posted earlier in the comments and tweaked it to generate some statistics on the historical effectiveness of the proposed autofilter.

Specifically, my query returns a monthly count of:

• the total number of questions selected for HNQ in that month,
• the number of those questions that were either closed or manually removed from HNQ by a moderator, and
• both of the above, but limited only to questions whose title (at the time they were picked for HNQ) would pass the proposed autofilter.

Anyway, looking at the results, I'm not too hopeful about the effectiveness of this proposed filter in keeping bad questions out of HNQ:

As you can see from the graph, the autofilter only matches about 10% of all HNQs on math.SE. Furthermore, it appears to be matching roughly the same fraction of both good and bad questions (or, at least, of questions that were and weren't closed or manually removed from HNQ in the past).

In other words, at least based on this historical data, the proposed filter — regardless of its other merits, or lack thereof — would appear to have no significant effect on the fraction of HNQs that are closed or manually removed.

(Of course, the query can't tell us what other questions might have been selected to replace the 10% that would've been filtered out, had the autofilter been in place when they were hot. However, it seems reasonable to assume that those questions would not have been statistically too dissimilar from the ones that were actually selected.)

---

Ps. If you'd like to examine the underlying data in more detail, here's a version of the query that returns the raw list of past HNQs and whether or not they match the filter.

---

Pps. Here's a version of the query the returns percentage statistics on the fraction of questions closed or removed from HNQ among those that either pass or match the proposed autofilter:

Of course, there's no actual new data here — you could easily generate these percentages just by dropping the raw question counts from the first query above into a spreadsheet — but this might provide a useful alternative visualization.

As this graph shows, the questions that match the proposed filter aren't historically any more likely to get closed or removed from HNQ than those that don't, at least not until the last couple of months. Since February 2020, questions matching the filter do seem to get removed from HNQ a bit more often than those that don't, but this could be just a statistical coincidence. (It also seems to correlate with an uptick in the number of questions removed from HNQ in general.)

Even within the last few months, however, automatically applying the autofilter would not have made much difference to the overall removal rate: the yellow and blue lines track each other very closely over the whole period for which data is available. At best, in March 2020, having the autofilter in place would've reduced the manual HNQ removal rate from 87% to 85%.

(Again, this is assuming that the questions that pass the filter and did get selected as HNQs are also statistically representative of the questions that would've been picked to replace the ones filtered out, which cannot be proven but seems like a reasonable assumption.)

---

Ppps. Just for the sake of curiosity, prompted by Paul Plummer's comment below, I did a little bit of exploratory data analysis using the data from the detailed query above. In particular, I tried looking for words* that have occurred only in the titles of removed (or closed) HNQs, and which have occurred in more than two such titles. Here they are, grouped by the number of such titles they occur in:

• 7: delta
• 5: h, l, students, let
• 4: rolls, whole, distributing, separable, extended, trace, start, reciprocal, letter, hold
• 3: finish, proposition, require, i'm, seeking, negation, morphism, add, puzzle, cut, entries, direction, associative, roll, symmetry, eight, variation, symbol, moves, average, english, squeeze, clear, cancellation, uncountably, hypothesis, physics, towards

So, what does that list tell us? I'd say not much, except that it's kind of hard to reliably detect bad HNQs using word filters.

Would we want to add an autofilter for "delta"? Maybe. Doing so would've saved us from a whole seven (presumably) low-quality HNQs over the past year or so. Adding in "let" and "students" would increase that to 17, assuming that none of those three words have occurred in the same title (which I haven't checked).

On the other hand, I'd assume we probably don't want to autofilter "separable", "extended" or "trace" (or the letters "h" and "l"), even though statistically those all seem to be better indicators for bad HNQs than any of the proposed filter keywords above.

^{*) For the purposes of this exploratory study, I'm defining a word as a sequence of the letters A–Z, digits 0–9 and apostrophes. All titles were converted to lower case before analysis.}

---

FWIW, I also looked at which words would appear to be the best overall predictors of a HNQ getting removed or closed, using add-one additive smoothing to discount infrequently occurring words. (In other words, I ranked the words based on the score $(1 + n_b) \mathbin/ (2 + n_b + n_g)$, where $n_b$ and $n_g$ are the number of "bad" and "good" titles the word occurs in.) Here are all the words with score > 0.75:

• delta: 0.89 (7 bad, 0 good)
• h, l, students, let: 0.86 (5 bad, 0 good)
• rolls, whole, distributing, separable, extended, trace, start, reciprocal, letter, hold: 0.83 (4 bad, 0 good)
• expectation, properties, 20: 0.82 (8 bad, 1 good)
• finish, proposition, require, i'm, seeking, negation, morphism, add, puzzle, cut, entries, direction, associative, roll, symmetry, eight, variation, symbol, moves, average, english, squeeze, clear, cancellation, uncountably, hypothesis, physics, towards: 0.8 (3 bad, 0 good)
• verify, column, school: 0.78 (6 bad, 1 good)

^{(The reason for setting the cutoff at > 0.75 is that after that comes a long list of words that each appear in exactly two "bad" HNQ titles and no "good" titles. Most of those look even more like just random coincidences than the words already listed above.)}

Not too surprisingly, this looks very similar to the previous list. The only difference is the addition of the words "expectation", "properties", "verify", "column", "school" and the number "20", all of which appear in the titles of sufficiently many HNQs that were removed or closed to counterbalance their appearance in the title of one HNQ that wasn't as well.

FWIW, trying add-two additive smoothing instead (to favor common words over rare ones even more) still gives the same old list of usual suspects as before, but also brings up a few more words that occur notably more often in the titles of HNQs that get removed or closed than of those that don't, such as "true" (17 bad, 5 good), "countable" (14 bad, 4 good), "contains" (8 bad, 2 good), "three" (22 bad, 8 good), "up" (10 bad, 3 good) and "determinant" (19 bad, 7 good), as well as "maximal", "rotation", "gives", "epsilon", "black" and "sine" (5 bad, 1 good each).

I'll leave it up to the reader to decide whether any of those words would actually make good autofilter entries.

Answer 3

I haven't logged into this site for some time, but I felt I needed to to register my displeasure if our community indeed marches further down this route. This might not come as a surprise, given my last meta question: Under what criteria should we remove questions from the HNQ list?

Let me start by stating that the goal is to filter out bad titles. If it happens that some good questions get filtered out, it is unfortunate, but the default state of a question is off the HNQ, and the current consensus of the moderators is that it is more important to keep questions which we do not want to represent our site off this list.

I wholeheartedly disagree with the conclusion here, as well as the reasoning leading to it. The default state of a question is not necessarily on or off the HNQ, and liable to be placed on the HNQ list should whatever algorithms in place decide that it should be there. The moderators (typically Asaf, but others as well) have taken a heavily interventionalist approach to removing questions from the HNQ list, which strikes me as the opposite of the "natural state".

In the actual natural state, we see the questions that our readers wish to see are promoted visibly, bringing people to our site, and promoting mathematics in the community. It gives people in the network of sites, including people who don't frequent MSE, an opportunity to see digestible nuggets of mathematics that's of interest to them.

Clearly, this will bias towards the elementary questions. I understand that we might see fewer, original, deep questions with meaty, informative answers in the HNQ if we have a flood of elementary questions in the HNQ. But, and let me highlight this, having questions on the HNQ is not to benefit our regular contributors.

We have bounties for this; if you think a question is particularly interesting, or has a particularly nice answer that you wish others to see, you can leave a bounty on the question and/or answer. This will bump the question to the list of bountied questions for enthusiasts to peruse.

I still think we should leave the HNQ alone, in a vast majority of situations. None of us have to actually like the questions on there, since we are not the target audience. It feels like our moderation team is attempting to reduce our site's relevance to the community.

---

So, as you'd imagine, I'm fairly against this proposal of blacklisting certain words, and some of these words more than others. Some words, like "solve" and (to a lesser extent) "simplify", strike me as extremely prejudicial. There are plenty of extremely interesting equations and algebraic expressions that can make for interesting questions.

It also heavily biases against elementary questions, which frequently involve "solving" and "simplifying". This smacks of that icky feeling that we are profiling people, not questions. It feels like we are implicitly taking the position that certain levels of maths are too basic to be sufficiently interesting (a certain comment on my previous question springs to mind).

I understand that we don't wish to publicise the questions we don't want on MSE, but I just don't think that the HNQ questions are any significant causal agent here. Just look at the number of new contributors, asking low quality questions, with less than 15 reputation. I don't think that any of them read the guidelines, thought that they shouldn't post their question as it was, then saw a low quality HNQ and decided that it was actually fine. I just don't think that happens. I'm just not sure what we're getting in exchange for all the babies we're prepared to flush with the bath water.

If we do implement such a blacklist, I think we should restrict ourselves to things like "plz", "urgent", "help", and other things that shouldn't be in the title of any question, and can be edited out/replaced without damaging the question. This way, it doesn't feel like we are censoring content and dictating what people should be interested in, merely helping ensure that whatever is published on the HNQ is well-presented.

I know that this opinion is somewhat unpopular in a certain sector of MSE, and unfortunately, this sector of MSE encompasses all the moderators, but I needed to say my piece.

Answer 4

My intention is that an upvote of this answer indicates agreement with the following statement:

I am in favor of creating a blacklist of terms which will prevent a post from appearing on the HNQ. The precise composition of that list be determined once we have agreed that such a list should exist in the first place.

---

Pontification

What follows are an expression of my opinion, intended to justify my support for the existence of a blacklist, and to elicit comments and engender discussion. Please not that my intention is that an upvote to this answer should indicate agreement with the statement above. One needn't agree with the following in order to agree with the above statement—comments would be an appropriate place (I think) to continue the discussion and voice disagreement with my reasoning. I considered placing the following in the comments directly after this answer, but, as the writer of the answer, I get the privilege of using my answer to kick of discussion in manner that is less ephemeral than comments.

In general, I am in favor of any proposal which cuts down on the amount of low-quality posts which end up on the HNQ. My understanding is as follows.

1. Each sub-site is allotted only a finite number of slots on the HNQ (five, with the possibility of reducing that number if the community agrees).

2. We have no lack of active questions which can be promoted to the HNQ. Hence removing a question from the HNQ does not greatly alter the amount of MSE content that is on the HNQ; it only change what content is on the HNQ.

3. The HNQ should be a showcase for the best that MSE has to offer. However, for various technical and cultural reasons, posts which are not a good reflection of MSE often end up on the HNQ. For example, no post with MathJax in the title can appear on the HNQ; homework dumps often become quite active because they are easy to understand and easy to answer, even if such questions do not put forward the very best of MSE.

Assuming that the above is true (and that there is general consensus about point 3—perhaps some people have no problem with MSE being (or being portrayed to the wider community as) a site for dumping homework problems), then it makes sense to filter and prune the HNQ somewhat actively. A list of verboten terms is a step in that direction, and is, perhaps, more transparent than the current status quo. However, it seems that we don't yet have consensus even on the idea that there should or could be a blacklist of terms which should be barred from the HNQ (note: such a list only keeps bad titles off the HNQ; a good question with a bad title can be fixed).

Answer 5

I'll try to address some issues I see here.

First, there is an underlying tone of this being an interventionist approach and that opposition is out of principle against interventionism.

That is fair, but if you are against interventionism out of principle and you think that such a list is an intervention, then you should be against the HNQ too. (A disclaimer: I am not against the HNQ.)

Why? The HNQ involves an algorithm with some parameters that displays questions which satisfy those parameters with more prominence.

This list involves an algorithm with some parameters that avoids displaying some questions which satisfy those parameters with more prominence.

The actions are the same in nature, the intention is different, but interventionism is defined by the actions (or perhaps absence thereof or not) and objects, not the intentions, regardless if they are understood as positive or negative. A political analogy is that if someone is against interventionism out of principle, then they ought to be against both government helping companies which are well-established and/or companies which are in bankruptcy. They should not help big businesses, nor small ones.

One can say that they are against manual intervention, which is what Asaf is/has been doing, and that is fine, but:

1. This is not what this topic is about.
2. If anything, this topic is precisely a way to avoid or lessen the manual intervention which you oppose.

It is somewhat frustrating that people are conflating Asaf's actions with the maintenance of such a list and lumping all kinds of things together. It looks like a knee-jerk reaction and it is adding a lot of noise to the discussion.

There are now two possible branches of reply which I can see:

1. "Well, you see, the HNQ has always been here, so it is natural to me. Such a list is new, so it is an intervention."
2. "OK, but that is a bad intervention, whereas the HNQ is a good one because it helps displaying what our community thinks is best."

Branch 1 is just silly. To elaborate more: if you follow branch 1, you are not against interventionism, you are against changing the status quo, which is entirely different and not that much helpful when we are discussing... changes. Regarding 2, do you honestly think that adding things such as "urgent" or "ASAP" is bad intervention? I mean, honestly? If yes, then fine... but please understand and acknowledge that this is not out of principle in the sense mentioned previously. It is for some reason that I don't understand, and I am still waiting for a sensible explanation.

Answer 6

Proposal: MSE should have an auto-filter. That filter should consist of the string *.

(Look! I've answered the question. This answer is therefore technically on-topic. And as we all know, that is the best kind of on-topic. Having technically answered the question, I'm going to go off the rails a little.)

In other words, remove Mathematics StackExchange from the HNQ.

The majority of the best questions on MSE are already ineligible for the HNQ because they contain MathJax in the title. The StackExchange network is unwilling to implement any kind of technical fix for this (and there is a quite reasonable argument that technical fix is more complicated than it is worth). As such, the HNQ already fails to be a showcase for the best of what MSE has to offer.

Indeed, my rough impression is that the HNQ is primarily driven by routine homework exercises. A common path to the HNQ is that a mathematically immature or unsophisticated^[1] user will post a verbatim question from some homework assignment. Several FGITW users will very quickly provide answers to the question, while a few other users use the comments to elicit improvement from the original asker. The answers, being correct solutions to the stated problem, are upvoted, and the question appears on the HNQ a few hours later.

I do not believe that MSE should be (or be known as) a homework dump. As such, homework problems rarely belong on the HNQ^[2]. Unfortunately, the HNQ broadcasts to the rest of SE community that MSE is exactly that: a dumping ground for homework problems.

Moreover, I don't really see an upside to participating in the HNQ. We don't need the traffic (MSE is already one of the most active SEs, and students taking classes is a huge driver of traffic). What benefit does MSE gain from participation?

Perhaps it is time to abandon the HNQ altogether?

---

[1] Note that "mathematically immature and unsophisticated" is not a critique of the level of the question being asked. I have had precalculus students who show tremendous maturity in their write-ups of homework problems, and have read draft Ph.D. theses which look like they were written by a third-grader. Rather, it is a critique of the quality of the presentation.

[2] It should be noted that Physics SE has an explicit "No Homework" policy. I have often wondered by MSE does not have a similar policy, but it seems like the ship sailed on that one long before I got an account.

Answer 7

Although this would require more work to implement, consider that instead of posts being excluded based on the autofilters just not appearing on the HNQ list, they are also placed in a separate list. Then somebody (at least the moderators, but perhaps it can also be accessed by certain other members, e.g., ones with particularly high reputations) can occasionally, if and when they wish to, check these items to see if any should actually be considered for the HNQ. If so, then perhaps the easiest way to have them be reconsidered is just for the person to edit the title in such a way that it's no longer excluded from the HNQ.

To avoid having too many items in this list, especially ones which no longer would reasonably be considered for the HNQ anyway, perhaps have them age off after some period of time, e.g., a few hours or perhaps a day or so.

Note if nobody ever looks at this list, then with all else being equal (e.g., the list of terms for the autofilter is not different based on this feature being available) the behavior would be effectively the same as without this feature, so there's not much downside in that respect. However, having this extra list does help to address certain concerns, such as mentioned in the question comments, about otherwise good questions not being included just due to their titles containing one or more words that are being filtered out.

Note I don't have, or claim to, enough knowledge and experience with the questions involved which should, or should not, reasonably be on the HNQ to state whether or not this additional option would be worth the implementation effort and considering how often it's expected anybody will check this list. I'm just proposing it here so other members, especially ones who are more knowledgeable about this than me, can give their feedback and opinions.

Update: I believe a fairly reasonable option to consider for anybody other than moderators to see the list is for other members to see only the questions with tags where they have a gold tag badge (i.e., as described in Badges, "You must have a total score of $1000$ in at least $200$ non-community wiki answers to achieve this badge."). As they already are believed to be quite expert in this area, so they can for example single-handedly mark these questions as duplicates and reopen as needed, they should also be able to tell, better than most other members (including moderators), which questions are quite good & how to best possibly change the titles so the questions may appear on the HNQ.

You may also consider adding other restrictions, e.g., requiring a reputation of at least $20000$ so they're considered a "trusted user". However, I suspect the large majority of gold tag badge holders would already meet any requirement like that.

Answer 8

I support the proposal.

• Please consider upvoting this post if you agree with the proposal of the question & the current proposed list. Else downvote.

Answer 9

I don't know how many people share my view, but I think if it is a minority it is certainly not a small one based on the vote counts of the answers to this question at this time.

I don't think there should be any autofilters to remove hot network questions, and I also don't think any moderators should be tampering with the hot network questions manually. In practice, what ends up remaining are what the moderators think are good questions, and this is not what moderators are for. Moderators are not supposed to impose their views, they are there to handle exceptional problems that regular users cannot deal with. I don't count certain types of questions being on the list of hot network questions as being exceptional problems.

The terms to filter out that have been proposed are, in my opinion, ludicrously general. I'd be happy to review examples provided by the moderators that they think warranted removal, but the probability that I'll disagree is very close to $1$, especially considering this list.

On prompting by a moderator I will note that I disagree with having a list at all since I don't think hot network questions should be filtered out, period. I think the algorithm should be allowed to select the questions.

Answer 10

Please consider upvoting this post if you agree with having the filter for the HNQ & by removing only the following words in list. Else downvote.

• question

• the following

• urgent

• opinion

• please

• help

(Feel free to suggest more words to be removed from the HNQ, they might be taking into consideration for the creation of a new list (new poll post)).