# Hot Network Questions. How are they selected?

Now that we get to see the SE-network wide list of "hot" questions, I am just shaking my head in disbelief. At the time I am writing this, the two hot questions from Math.SE are titled (get a barf-bag, quick)

Who gets to select these questions? How? Irrespective of how this is done, this is ridiculous, as neither question has any even remotely serious content (the latter one is more or less a common fake-proof).

My proposal:

The representatives of Math.SE on this list should be based only on the votes of people who are active on Math.SE. Not just all voting members (suspecting/pointing finger at SOers, who get the right to vote from association bonus alone).

The exact rep limit (if any) is open to debate, may be 1000? Probably shouldn't put the bar too high, for that would introduce different kind of problems. But something that ensures a valued history of contributions on this site - not elsewhere on the SE network.

• I think there should be no community wiki questions shown. I would also ban certain tags such as "homework" from being shown there. – Michael Greinecker Dec 10 '13 at 12:14
• I'm afraid that's not a technical problem. The real problem is that question without «any even remotely serious content» quickly get highly upvoted (and high-rep users oppose closing obviously offtopic questions like poems question). – Grigory M Dec 10 '13 at 12:17
• As someone mentioned, these questions are accessible to a wide audience and are therefor upvoted, like other online communities (which will remain nameless). As an aside, math is not the only SE that suffers from this. A hot Physics SE question for a while was one about why a helium balloon in a car went right when you turned right. Nothing to do with physics, really, but the question was so simple it appealed to a wide audience. – JFA Dec 16 '13 at 2:12

I've criticized the hot questions algorithm quite heavily on MSO in the past, it does favor certain types of questions without a good reason, and it has a strong self-reinforcing effect where questions that manage to get into the hot questions list get disproportionally more votes, which keeps them there longer.

But the algorithm is not the only thing responsible for the result, it has to work with the information it can get about the questions. It can't actually judge the quality of questions, the only measure it has available is the score of the question and its answers. Voting generally favors questions that are accessible to a wide audience, very specialized question tend to get rather low scores. The algorithm will prefer questions that appeal to a wider audience due to that, but that isn't necessarily a bad thing for the hot questions list. The votes from users that arrive at the question from the hot questions list will exaggerate this effect, but they are not the primary cause of it.

One aspect particular to MSE is that this site is one of the very few sites that still allows certain kinds of questions (the type often tagged as or ) as community wiki question. If this kind of question is so ridiculous or embarrassing, why are they still allowed? If ridiculous questions are not getting closed, but instead upvoted, the problem is not the selection algorithm for the hot questions list, but that those questions are upvoted and left open.

I've proposed a similar change and a few others in the following feature request on MSO: Better criteria for the hot questions list

There are several aspect that could be improved in my opinion:

Questions can stay in the list for too long, making the list a bit too static for regular SE users. I've previously posted a feature request about this, and I still think that no question should be able to stay in the hot questions list for a whole week.

The current criteria are prone to select problematic questions. They value a high number of answers, which you find for example in list questions. So questions like this example from Math.SE have it easy to get into the list, while this type of question is regarded as problematic by most of the SE network and explicitly disallowed on many sites.

Being in the list of hot questions is self-perpetuating. Questions on the list get significantly more exposure, which means more votes, which then improves the position in the list of hot questions. The votes from outside users are also problematic as they represent mostly the popular appeal of the posts, not necessarily a judgement of the quality by an expert.

I've some ideas on what could be changed:

Exclude community wiki questions from the list. Those are usually big list-style questions that aren't a good example for high-quality content across the network.

Put less value on a high number of answers. The current method values (as far as I understand it) a high number of answers and a high total score of all answers. This preferentially puts more subjective or list-type questions to the top. A good question that got one high-quality answer that is highly upvoted shouldn't be at a disadvantage compared to a popular question that gets lots of opinions. Maybe counting only the first two or three answers and their combined score would be enough.

Weight external votes differently from internal votes. External votes from users that discover the question via the hot questions list shouldn't be counted in full, they mostly represent the additional exposure and the mass-appeal of the question, not necessarily the quality. But they also shouldn't be disregarded completely, a question that doesn't get any votes from the external users is probably too specialized to be of interest to a wide audience.

Normalize votes for each site. The voting behaviour varies a lot between different sites. Currently sites that have an above average number of votes are favoured compared to other sites. Normalizing the votes on a per-site base would put a stronger emphasis on the number of votes a specific questions gets compared to other questions on the same sites. It would prevent certain sites from being overrepresented due to their general voting behaviour.

• I fully agree with the proposal, but not your lament on the existence of soft-questions and big lists. You might think it is tolerable that soldiers play drinking games, but may want to display something else at a military parade. – Michael Greinecker Dec 10 '13 at 12:52
• Thank you very much for sharing your thoughts! – Jyrki Lahtonen Dec 17 '13 at 22:08

There is at least something that can be done to reduce the degree of embarrassment that "hot" questions bring to Math.SE: improve their titles. The titles are what ~3 million daily visitors to SE actually see; relatively few will click through to the question (although the absolute number of visits may still be high). Misspelled words, poor grammar, multiple question signs, lack of actual information in the title... these things tell the rest of network and its visitors that Math.SE does not care about the quality of its content. Current example:

How to show some function is constant ??

Yes, removing one ? and the space between text and ? looks like a small edit; but when the text gets shown to millions of people, maybe the fix isn't so minor after all. No other SE site contributes similarly embarrassing titles to Hot questions. (Arqade is often on the quirky side, but this is different.)

Proposal: whenever you see a less-than-stellar title in the sidebar (and have 2000 points in the bank), just go ahead and edit it. If applicable, make it more specific: this will reduce the number of passers-by who click the title just to find out there's nothing for them there. On the scale of Math.SE traffic, one edit bump is nothing; especially since hot questions get bumped by answers anyway.

• That extra space before the question mark is normal in some languages (e.g., French), and Math.SE has a very high proportion of non-native writers of English. This fact also accounts for a great deal of the odd grammar and spellings. I’m not suggesting that they shouldn’t be fixed, but I don’t consider them an embarrassment. – Brian M. Scott Dec 15 '13 at 23:07
• A good point! Thanks for the time you spent on this. I will pick the most upvoted answer to allow this question to drift to the background. – Jyrki Lahtonen Dec 17 '13 at 22:07
• @BrianM.Scott: I wouldn’t consider such errors as reflecting badly on their authors, for the reason you give. However, I think they to end up as a minor embarrassment to the site nonetheless: we may understand their cause, but many visitors won’t. – Peter LeFanu Lumsdaine Dec 19 '13 at 2:48

These lists are automatically generated by a formula as described here.

According to this accepted MSO answer, in the past there was no intent to change the formula to account for the reputation of voters.1 And who's to say that "active" math.SE users are not upvoting our poetry question? Five answers were given by $\geq 1000$ reputation users. Note, also, that the current formula does account for the number of answers given, and many "serious" mathematical questions get only very few answers.

• Yeah. May be a better idea would be to have a separate voting for questions that we feel should be in the showcase? – Jyrki Lahtonen Dec 10 '13 at 17:09
• @Jyrki: I'm not sure that decoupling "hot" voting from question voting is the right route. I do think that the "hotness" of a question should be measured by how the community naturally interacts with it: votes, answers, &c. Adding a "hot or not" button would only be an ad hoc way to avoid the embarrassing situation of being represented by substandard questions. (This is not uncommon for us.) But aren't we partially to blame for this by allowing a lot of soft questions with wide appeal while neglecting to upvote many of our more serious Qs and As? – user642796 Dec 10 '13 at 18:15
• Quite. Thanks for explaining these points, Arthur. – Jyrki Lahtonen Dec 17 '13 at 22:09

I could say a lot about this, but I'll do my best to be as brief as possible. The hot questions list is something that I've been collecting data on for a few months now. Barring a few strange cases, I have a formula which seems to be approximately correct for most questions. Note that the formula is different from the one Arthur Fischer's answer references; that formula is known to be out of date. However, I think the most important thing is not that, but the qualitative description in MadScientist's answer.

I'm significantly more radical when it comes to this than the OP's proposal. I personally think that the hot questions list should be removed from the network entirely (though I'm not formally proposing this here). IMO it does very little good for the purpose of answering questions. Though it does sometimes communicate interesting questions, I think that's more the job for other sites e.g. social networking like Reddit. What it does is bring in a lot of non-experts to vote on questions and answers. The sorts of things that make it on the list are typically the most basic questions allowed by the site. It also creates a positive feedback loop which disproportionately favors the interests of SO and essentially lets SO take over smaller communities on individual questions. This really distorts the reputation awarded from these questions as well. I've played around with answering some hot questions both on this account and anonymously, and the results were quite startling (see e.g. this answer of mine which is my top answer in the network despite being a joke). I can not think of any case in my memory where a hot question got a much better answer than the existing ones because it was hot. I can however find numerous cases of mediocre answers getting a lot of upvotes solely because the question was hot.

However, it seems that the MSO crowd for some reason really likes this list. I can't see why, since to me it's basically like a social networking site like reddit and has almost nothing to do with generating good questions and/or answers. But I'm aware that's not a fight I'm going to win. I think a case can be made that this is not doing us much good as a community, and in many cases it's actively harming this site by distorting reputation and emphasizing mediocre/basic content. This to me seems like a problem. While it seems extreme, I think we should at least consider whether we want MSE questions to remain on the hot questions list at all. I'm not formally making this proposal yet though.

You might argue that this is really a non-issue. Sure, some questions get hot, and the voting on those can somewhat disrupt the reputation, but it's a pretty small effect, and the rep cap is there for the worst cases. I've actually personally found it to be rather exploitable, answering only hot questions on anonymous accounts quickly got me much more rep than I could by answering the questions I'd normally see. I do still sympathize with this viewpoint though. In this case, the main thing that we should do as a community is make sure that those questions that do get hot are at least edited to be well-written and good content. Any particularly bad questions that happen to be on the list should be edited if they are improvable, or downvoted and/or closed if not.

I will note that advanced questions are quite rare on the hot questions list. I've only seen a few questions which made it anywhere on the list that were fairly advanced. For example, How did we know to invent homological algebra? and categorical interpretation of quantification reached around #25 on the list for a few hours before falling back off. These just don't do very well in the long run, because the SO crowd doesn't appreciate them as much and so the positive feedback effect is much smaller. That's the highest I've seen any questions reach which was at a higher level than basic calculus/linear algebra/number theory.

The suggestion of the OP to ignore votes cast by users with very low participation here might fix some of the problem of being dominated by external voters, but it's likely programatically unfeasible. The hot questions formula has to be calculated many times per question, which rules out anything but the simplest possible formulas. Querying exactly which users has upvoted the question and checking each of their user profiles would be a much more computationally expensive operation than just looking at the vote total. I've seen much less expensive suggestions be shut down as not feasible on MSO, so I'm almost 100% certain this is out of the question. One might be able to come up with subtle ways to try to do something similar without requiring so many expensive database queries, but at the end of the day I just don't see a lot of marginal benefit of doing this just so that we can have MSE appear on the hot questions list but only a little bit of the time.

I am sure some reading this will think "why not just do the simplest thing and penalize Math questions some?". IMO This will just cause more problems along the lines of those mentioned in the OP. The problem of math questions on the hot questions list is (at least) 2-fold. Questions from Math make it onto the hot questions list too frequently, to the point that even bad ones can make it on if they quickly get answered and voted on. That could be fixed by penalizing math questions so that only the really good ones make it. However, the second problem is that the questions which do make it to the top and stay there are more controlled by the masses and less by the experts here at Math SE. Applying a penalty would just make that worse, since the few questions that would make it to the top of the list would be the ones that can bring in lots of votes from outside Math SE's regular voter base.

For reference, in case anyone wants some statistics, I've compiled a list of all the questions which I know of from the past few months which topped the hot questions list. There's no data on how long they topped it, and I did not put ones which made the list but not the top of the list. The full hot questions list has 100 questions at any given time, and most of the bottom ones are only hot for a few minutes, so that data would not be very useful and would not be systematic. By contrast, only looking at the top questions, I can be fairly sure that I've not missed very many important ones over the span of this time period since most of them remain hot for quite a while and I've collected data on this roughly every few hours.

I don't intend to keep updating this list, so keep in mind that it only goes from the dates July 28-December 21, 2013. I'm primarily including this list so that anyone who wants to can look at the statistical distribution of these "worst offender" questions. Also, apologies that this list is so long, but I could not find a better method for listing it.

• +1 for the data, but I disagree with your proposal. Re: "nominally supposed to be expert-level" -- MSE is explicitly for all levels of mathematics. In any subject, the distribution of people by level of mastery forms a pyramid, with majority at the most basic levels. Voting, naturally, reflects this. As long as the Earth orbits the Sun, expert-level q/a will not be appreciated more than the basic ones. The hot network list brings something mathy to the masses outside of MSE; most of it is - inevitably - quite shallow, but some of the questions on your list are not bad at all. – Post No Bulls Dec 22 '13 at 5:57
• @PostNoBills That's fair criticism, and I realize that I'm in the minority on this. My proposal is rather extreme, and I'm fairly certain it won't happen. I'm considering removing it for now from this answer. – Logan M Dec 22 '13 at 6:00
• @PostNoBills I've edited my post to be more conservative for now, and I'm no longer formally proposing anything. If the community eventually comes to agree that MSE shouldn't be on the list, then we can make that decision later. I was probably jumping the gun here since I've been looking at data on this for months, while the list has only been highly visible to this community for a few weeks. – Logan M Dec 22 '13 at 6:25
• A ${\Huge +1}$ for all the work you have put into this. I agree with the conclusions, too. Many people cite that "MSE is for all levels of math". I rather feel that people take that too literally. I haven't checked, but wouldn't SO similarly for all programming questions. Yet they are allowed to form some community standards of what's appropriate and what's not. I think that we can do the same. And I also think that unless we do, the site will devolve into a HW help site for freshmen. – Jyrki Lahtonen Dec 22 '13 at 7:21
• @JyrkiLahtonen I shouldn't take too much credit for the data. While I did compile it, I do low level data mining as a leisure activity, and SE provides a lot of fairly noisy data so it's a good source. That's really the only reason I know or care about this. I expressed some strong opinions about the list and I agree with them, but in reality I don't really care that much about this site and I rarely post here nowadays. In any event, you may find this answer in a similar vein if you want to know more about what sorts of things the list favors. – Logan M Dec 22 '13 at 7:54