# Examples of poor review audits

Since threads about review audits seem to be turning up fairly frequently now, it seemed like a good idea to open a thread to collect examples of those audits where people failed but feel like they should not have.

This serves both the purpose of getting at least a small idea of the scale of this (ie, how often does it happen), and also means that we will potentially get information enough to suggest concrete changes to the algorithms used for picking the audit cases (if this seems necessary).

• board of shame is a better name for this question – Norbert Sep 11 '13 at 14:52
• And for venting! It's frustrating to spend time thinking about a post, only to be told that I've chosen the wrong answer. – Douglas S. Stones Sep 11 '13 at 16:50
• @DouglasS.Stones that too, without each person needing to vent having to open a new thread for it (assuming they actually see this one). – Tobias Kildetoft Sep 11 '13 at 17:52
• I seem to recall that it is possible to remove audits from the audit system -- this post could be useful for that. But I can't find the source. Is there anyone who can verify this? – Lord_Farin Sep 12 '13 at 14:05
• I get quite a few, but they are glaringly obvious and quick to dismiss. – copper.hat Oct 27 '13 at 20:38
• math.stackexchange.com/questions/535020/… is too broad for sure and to some extent lacks context. – dfeuer Nov 3 '13 at 2:53
• It appears that the audits have now been redesigned - clicking close on an audit wasn't an automatic pass for me just now, but rather gave me a chance to click an option first. – user61527 May 22 '14 at 5:46
• While the thread is obviously useful as a way to vent frustration, seeing it bumped repeatedly gets old. I think it would be better implemented as a chat room, similar to Whining Room on Meta. Or just add the (whining) tag. – user147263 Jul 30 '14 at 17:46
• @900sit-upsaday I agree. It was originally supposed to provide examples that could show how it might be improved, but it seems clear that this will not happen. And if you think it is annoying to see it bumped, consider getting a notification each time making you think "ahh, someone is speaking to me" only to see that it is just another answer to this question. – Tobias Kildetoft Jul 30 '14 at 17:58
• @TobiasKildetoft You can ask SE to be disassociated from this thing; I'm sure that seeing the situation they'll honor the request at once. – user147263 Jul 30 '14 at 18:00
• Should we add the tag (big-list) to this question ? – GNUSupporter 8964民主女神 地下教會 Dec 12 '17 at 17:17

Side complaint: I seem to be getting an awful lot of review audits, maybe one per five legitimate review requests.

If you are asking me to spend my time performing community service, I would prefer you not waste my time with pointless "fake" tasks.

• That would definitely be excessive. – Graphth Sep 17 '13 at 0:42
• I too am getting a lot of them. – dfeuer Sep 17 '13 at 3:34
• Recently the fake reviews have died down. Maybe I made it through some kind of "probationary period" with a ton of audits. – user7530 Oct 2 '13 at 0:06

The newest and one of the most ridiculous examples to date:

https://math.stackexchange.com/review/first-posts/109542

I failed this by clicking "add comment". That's right. You're not allowed to comment when you're being audited. I'm filing a bug report.

• I just had a similar experience trying to leave a Comment noting Related Questions. The audit complains: "Your review was inappropriate. This was a high quality post and you should have considered leaving it as-is or even upvoting." – hardmath May 20 '15 at 23:30

I have failed a few. One interesting thing is that, for "close" audits at least, the system lies about the vote count, showing 0 instead of the actual vote count. Which seems to be counterproductive: it appears they change the vote count to encourage reviewers to evaluate the question independently, but then they criticize the reviewer being too independent and not realizing that a question with a high vote count should not be closed!

Now that I realize this, I don't fail them anymore, because I check the vote count of the real question first. One examples I have been presented this way was A "non-trivial" example of a Cauchy sequence that does not converge?

• Good idea. I took your suggestion to heart and found that a question I was supposed to review had been deleted, so selecting "close" earned me kudos for making the right choice. While I understand the reason behind audits, I think that on balance (at least for me) the chilling effect outweighs the didactic benefits. – Rick Decker Sep 13 '13 at 1:04
• Just for curiosity, what the system actually says if one fails the audit? EDIT: ok, I think that's the text shown at meta.math.stackexchange.com/questions/10836/… – mau Sep 13 '13 at 13:30
• @mau: meta.math.stackexchange.com/questions/10836/… is one of them, but the message is slightly different depending on which type of review you were performning – Carl Mummert Sep 13 '13 at 13:33

https://math.stackexchange.com/a/508096/17596 showed up as a "first post" review. I found the animation far more distracting than helpful, so I downvoted (disagreeing, apparently, with fifteen folks who think it's the cat's meow). Review audit failed.

• First post audits are the most ridiculous of the bunch! I've failed audits for leaving alone posts that, while not of the highest quality, had no actionable problems. – user7530 Oct 27 '13 at 20:04
• I just had to review the first post math.stackexchange.com/questions/578662/… (with fake 0 score and fake user reputation). I added a comment because the OP made a significant change to the question and the thing was in front of me, but apparently adding a comment to a post that was better before an edit means I'm a bad auditor. – Mark S. Nov 24 '13 at 16:44

I failed a review audit for this question because I went to close it as duplicate as I recognised the title and content of the post so thought it was an exact duplicate of itself (the review had artificially made it appear that the question was written 2 hours ago). This highlights a flaw in the system in my opinion.

The first time I got an audit, I thought it was a good idea. Well, not anymore.

I've failed some when we changed the policy, initiated by - I believe - this topic. Since the new "close" reason was included, I go for it almost on every question that was nominated for closing and lacks any context or effort (i.e., copy/paste questions). I myself nominate such questions for closing if there was a request for edit in the comments and the OP didn't respond in a few hours.

But, there are many questions, especially those that were asked before we took this more serious stand on copy/paste questions, that were upvoted quite nicely, so I've expected to fail each such audit.

Because of that, like Carl, I now open almost every question in new tab, and check that it's not a trap. It's a pointless waste of time, but my alternative is to get "caught" simply for not agreeing with other voters of that time, possibly before the new "close" reason was added (when I myself voted against closing such questions, due to the lack of a proper "close" reason).

The alleged purpose of the audits is to force reviewers -- actually voters, which is important! -- to pay attention. But that's the problem! As the site's help says (my emphasis):

Our sites are all intended to be a sort of representative democracy. Moderator elections are an important part of that plan, but voting on questions and answers is the primary mechanism through which the community governs the site on a day to day basis. Every user with sufficient reputation can exercise their right to vote, every day that they visit the site.

In a representative democracy, people vote according to their personal views. Be it for the views different than those of a majority, or simply for being lazy, a voter in a representative democracy is allowed to vote differently than the majority. That is the core of a democratic system, and audits measure how close a person's vote is to that of a majority, punishing those who are not in accordance. Completely antidemocratic, and so anti-self-description of the site. I don't mind strict rules, but then don't make audits based on votes which can be -- and often are -- not in accordance with those.

If lazy voters are a problem, then all the incentive should be removed from such voting, i.e., badges. Who wants to help the community, doesn't need badges; those who vote for those little coloured circles won't do it anymore if those "medals" are gone.

But the system, as it is, is rotten. IMO, of course.

Edit: I forgot... I sometimes click "close" not to actually vote for it, but to see what did others select as the reason. Just "off-topic" from the description is not as specific as seeing which of these, which helps me focus on that particular reason and see if I consider it valid. An audit would "catch" me there as well.

• +1 There is something distinctly pointy-haired about the whole audit system – user7530 Oct 28 '13 at 18:46
• @user7530 A perfect description! :-) – Vedran Šego Oct 28 '13 at 18:55

I failed this close-vote audit. I wanted to vote to close it as a duplicate of this question, particularly since there was a comment on the original question suggesting it to be a duplicate. The suggested close reason was not as duplicate (I believe it may have been "too broad," but I don't recall exactly), but I do not agree that a close vote was inappropriate here.

I just "failed" a reopen review audit. My philosophy is that the principle of res judicata applies to all reopen requests: that is, I will automatically reject any request to reopen a question that has not been edited since it was closed, unless strong reasons to reopen are given in the comments. The audit I failed showed no edits since closure and no comments at all. Although the question was allegedly closed as "unclear", I believed it to be actually closure-worthy as "off-topic: homework without a meaningful attempt" (although it turns out to have been answered by the asker some time later, so there's no actual need to close it). I just don't think that "no major problems" is a legitimate reason to reopen—if enough users voted to close, why should I substitute my judgement for theirs?

• I disagree with your last line, in that sometimes closing is not clear cut. Perhaps starting a meta thread is appropriate when you come across debateable questions, but simply going "the first five people to the post held this opinion and who am I to question them?" isn't helpful. Although I do agree with the sentiment :-) – user1729 Sep 18 '13 at 15:57
• @user1729: Certainly closing is not always clear cut. But the notion that the second set of five people to consider it are inherently more likely than the first five to come up with the "correct" response is absurd. I haven't been around long enough to know whether comments on the question or a thread on meta are the better approach, but it certainly seems to me that something either in or surrounding the question should change before it is reconsidered. "Give me another chance at the lotto please" isn't a reason. – dfeuer Sep 18 '13 at 16:34
• Yes notion that the second set of five people to consider it are inherently more likely than the first five to come up with the "correct" response is absurd, but so is the converse notion, that the first lot of people are inherently more likely than the first five to come up with the "correct" response. Also, when a question is in the reopen queue it is often because someone has voted for it to be reopened. So someone disagrees with the five. I believe their opinion should be considered. – user1729 Sep 19 '13 at 9:11
• @user1729: I think it's reasonable to require that they explain that opinion in order for in to be considered. – dfeuer Sep 19 '13 at 15:32
• I would agree that an explanation would help, but I still think it is better to consider it than just to ignore it an pretend it hasn't happened. – user1729 Sep 19 '13 at 18:39
• I feel the "res judicata" reasoning is wrong, because one cannot vote to (re-)open before a question is closed. So it is not the first five people that voted to close, but just some set of five people, among an unknown number of people who might have had other ideas. Also it is impossible to undo a close vote, I think even if the question is edited (before it is closed). – Marc van Leeuwen Sep 21 '13 at 5:21
• Your res judicata doctrine is an abnegation of responsibility, pure and simple. And since you do assume the responsibility of adding your voice to close votes, you’re demonstrably biassed in favor of closing questions. – Brian M. Scott Dec 1 '13 at 4:43
• @BrianM.Scott, I doubt you can see my close or reopen votes, but I take them all seriously. I feel great when I cast a reopen vote on a question I previously voted to close because the OP went back and addressed the problems that led to closure. When someone initiates a vote to reopen when nothing has changed, I think it's on them to explain why. – dfeuer Dec 1 '13 at 5:13
• @dfeuer: I can see the ones that you make through the review queues, and I can see the close votes that I’ve been recording as I come across them, which in your case are the main source since I started paying close attention. Your last sentence is just more evidence of your bias in the matter, since the reason is obvious: that person did not think that the question should have been closed in the first place. – Brian M. Scott Dec 1 '13 at 5:21

This question post is a horrible "exemplar" of a post that should be reopened, according to a "reopen review queue audit."

In fact, the question should be put on hold, so the asker can improve it, and perhaps, and only then, reopened.

Including it as an exemplar of a post that users should reopen teaches users to reopen very poor posts, which is antithetical to the purpose of the reopen review queue.

The set of questions with a vote count of 5 or more, with no previous close votes, is not identical to the set of questions that should be reopened. The algorithm reflects the mistaken algorithms used in the review queue audits: high vote count greater than or equal to five net votes is not a reliable indicator of a good post. (This is evident in many cases on point point: there exist a sizable percentage of former Hot Network Questions that received "popular" votes, not because the question is one that should be kept.)

I failed a First Post with this question for one of two reasons. I noticed two things at first glance:

1. There was a mix of unTeXified and TeXified variables.
2. The post mentioned that the numbers could be "very large," but didn't specify how large they could be, nor give much other context aside from "it's for a computer program."

What I intended to do (in some order) was to fix the mixed variables (which I have since done) and leave a comment to the effect of: "Out of curiosity, how large can $x$ and $y$ be? The best solution for you may depend on that, and on other context (like what sort of program this is for), so the more you can tell us, the better." I am not sure at this point whether I pressed 'edit' or 'add comment' first--I was too surprised to discover that I had failed. Once I failed and actually went to the question, of course, I discovered that I was not the only person who wondered how large the numbers $x,y$ could be--the $4$th comment down (at present) asks that very question, and there are other comments asking for more context.

I rarely review, but I thought I'd give it a try, and almost immediately failed an audit. I was presented with a very poor answer in the “late answers” queue. I opened the thread to which it was attached. I saw that the answer I was reviewing had been deleted several weeks ago, so I clicked the “no action necessary” button, and got this obnoxious response:

So that was a poke in the eye from the review system, and I will immediately return to my former practice of ignoring the review queue.

Probability and expectancy problem

This is supposed to be a "reopen queue audit question" and to "pass" the audit, one is supposed to vote to reopen.

This is a problem statement question and deserves to be downvoted and closed, and not reopened!

I failed this one. After the fact, I voted to close it and it has since been closed.

https://math.stackexchange.com/review/close/246038

It has no context or details. I hope the fact that it is now closed removes it from the review audit.

• Now it will be moved from "known-good" to "known-bad" category, and used to trip up reviewers in the opposite direction. :) – user147263 Jul 30 '14 at 20:17

https://math.stackexchange.com/review/close/1056774

Are there any necessary or sufficient condition for when in a general convex hexagon, the lines formed by joining the midpoints of opposite sides are concurrent (intersect in a common point)?

This question lacks context. I can see that it might be interesting, but it appears entirely unmotivated to me. Based on the single not-deleted answer, it appears that this might have originally been a contest problem (and an original source would be valuable context for future users who might be reading off of the same list of contest problems). Frankly, not only do I think that this is a poor audit review question, I honestly think that it should be closed until the original asker provides some context.

Yesterday I failed this Review audit, and when I posted an answer in the Reopen/Undelete thread, another user reported failing two similar audits, and someone else later failed another one. That same person failed another one of the same type less than 6 hours later.

The number of such posts by the same user (who also had a question deleted on MO, though I can't see it), their frequency (all posted within a few days) and the fact that they all link to the same website is suspect, but taken in isolation they don't look so bad. So these make poor examples for review audits.

• The weirdest thing is why the deletion happened 7 months later. – YuiTo Cheng Apr 26 at 6:13

I failed this audit because I wanted it add a comment. Not that I added a bad comment. For all any computer might know I wanted to say this problem is the best research direction for mathematics since the Langland's program. Que sera sera.

Anyway I was already thinking the review audit sucked and was wasting my time-- can't it at least adjust so that those who pass more audits get fewer in future? But when you fail them it's a disincentive to continue.

https://math.stackexchange.com/review/close/1015074

This question is a fairly straight-forward "problem statement question" that I voted to close as "Lacking Context." I think that the level of the material (advanced undergraduate or introductory graduate level complex analysis) coupled with the (non-obvious) difficulty of the problem earned it several upvotes, and I am willing to concede that the dan_fulea's answer may be one of the few instances of a great answer justifying the preservation of a question-of-debatable-merit (note that answers are invisible in the "Close Votes" review queue), but I don't think that this is a good audit question.

https://math.stackexchange.com/review/close/1076507

I have been stuck on this problem for hours and have absolutely no clue how to go about it. Any help would be amazing.

The question itself is in the title, and is a pretty standard combinatorics question. I honestly don't understand how the question garnered five upvotes in the first place...

Black holes and the Schwartzschild solution

This Question turned up in my Close Vote review (as an audit). Because the Question garnered five upvotes, the system considers it a model of something that ought not be closed. But despite the OP's declaration, "This is probably more a mathematical problem as far as I can see...", it was cross-posted to Physics.SE and consists of four sub-queries that, judging by the OP's comments under the Question, have a strong nexus with the physics of gravitational fields.

The OP accepted the lengthy Answer (with references) that was given at Physics.SE and did not accept either of the Answers given to the Math.SE version. Therefore I feel this really should be closed as off-topic here. Those interested in the subject matter will still be able to follow links to the Physics.SE post (no migration is required).

I got this one:

https://math.stackexchange.com/review/close/1194064

I passed it by leaving it open, but it seems not nearly clear-cut enough to qualify as a good review audit. The asker has not provided much in the way of evidence of effort.

I also floated this in a discrete Meta question, before posting here.

https://math.stackexchange.com/review/first-posts/174827

I failed this audit, because I tried to mark it as a duplicate. The audit said it has been asked (I think) 1 hour ago and I was absolutely sure, that I have read exactly the same question a few days ago. However, I had read exactly this question before, because that first post was a fake of the orginal good question...

• Indeed, it seems that a minimal change that ought to be made to this system is that clicking the "close" button is not an automatic failure, when selecting an inappropriate reason would be the actual failure (after all, you have not actually chosen an action by clicking the "close" button). – Tobias Kildetoft Mar 5 '14 at 13:05

I passed this first post audit by selecting no action needed (which I think is the appropriate result). The question has $4$ close votes currently, but is sitting at $+17$; considering that at least $4$ relatively high-reputation users think that the question is unclear to the point it should be closed (while many other users disagree), it seems inappropriate to use this question to test the reviewer.

I failed this one: https://math.stackexchange.com/review/low-quality-posts/512046

I understand that it is an interesting question that does provide a bit of background with a link to the problem. But there is no attempt at solving the problem, so I voted to close. Some will disagree on this, but the question doesn't (IMO) belong in the list of audit questions.

https://math.stackexchange.com/review/close/781215

I think this is a poor review audit. I voted to close as "too broad" because well...it's too broad! The first comment even says it's a "very broad question".

I voted to close as too broad outside of review queue. So hopefully it won't show up again.

Failed this audit: https://math.stackexchange.com/review/first-posts/141607

I call BS; this is an adequate (if short) answer and I would in fact upvote it in preference to the accepted answer, if it hadn't been deleted.

Today I was asked to review an answer this question. The answer I was to review has been deleted. The deleted answer stated that the real number group may be equipped with the operation "$+$". I was warned that this was a bad answer. I am supposed to click the "I understand button." This I will refuse to do. It seems to me that is is an excellent hint for the problem.

• I don't agree that this is a good answer (although I completely understand not clicking the "I understand button." I generally refuse to): I read it as a new question separate from the post, not a hint towards the problem. – user61527 Mar 14 '14 at 19:34

I failed this reopen audit. The question was closed as "unclear," although I think it's perfectly clear what's being asked (and contains some initial thoughts from the OP about the issue).

Although this thread seems fairly quiet now, I have to report this one:

Although the Question has problems, these are not currently of the unclear what you are asking variety. No doubt the Question was properly closed and downvoted on that account, but subsequently it was edited in a way that makes the problem posed clear and precise.

I bring it up, not to argue for reopening, but to point out that the criteria for Review Audits do not seem to take into account post closure edits, other than to presumably require that the item remains closed currently.

• Almost the same happened to me. The question had problems, probably leading to a close, but was market as "unclear what you are asking". Nevertheless, the question was so, so clear that somebody commented the answer. I suppose I had to vote for the formatting issues without considering anything else... – Rafa Budría Jun 3 '17 at 12:43
• @RafaBudría: If it's simply formatting, I often try to fix them and reopen, though in some cases the edit preview flummoxes me and I fail. More dubious are cases that might be closed for lack of context. Sometimes, if I think the Question is of genuine interest, I vote to reopen despite that lack. – hardmath Jun 3 '17 at 14:23

Just got this one:

https://math.stackexchange.com/review/close/532401

The question shows no effort, starts out with "community, I need to solve the following problem" and then ends with (bold face) "I try several ways, and the hint given below but I have no luck to overcome this problem, I hope someone knows how to do it. It will be really appreciated if someone knows the actual proof, I'd like to learn how did they arrived that such a series converges. Thanks, and I will give some of my reputation for the actual proof. Thanks to all the community" Give me a break.

• Did you happen to notice who last edited it? – Daniel Fischer Nov 27 '15 at 19:21
• Meh. The annoying bold stuff could be edited out. And the question does have some redeeming qualities to offset the perceived lack of effort. – user147263 Nov 27 '15 at 20:46