63
$\begingroup$

Since threads about review audits seem to be turning up fairly frequently now, it seemed like a good idea to open a thread to collect examples of those audits where people failed but feel like they should not have.

This serves both the purpose of getting at least a small idea of the scale of this (ie, how often does it happen), and also means that we will potentially get information enough to suggest concrete changes to the algorithms used for picking the audit cases (if this seems necessary).

$\endgroup$
11
  • 16
    $\begingroup$ board of shame is a better name for this question $\endgroup$
    – Norbert
    Commented Sep 11, 2013 at 14:52
  • 9
    $\begingroup$ And for venting! It's frustrating to spend time thinking about a post, only to be told that I've chosen the wrong answer. $\endgroup$
    – Douglas S. Stones
    Commented Sep 11, 2013 at 16:50
  • $\begingroup$ @DouglasS.Stones that too, without each person needing to vent having to open a new thread for it (assuming they actually see this one). $\endgroup$
    – Tobias Kildetoft
    Commented Sep 11, 2013 at 17:52
  • 1
    $\begingroup$ I seem to recall that it is possible to remove audits from the audit system -- this post could be useful for that. But I can't find the source. Is there anyone who can verify this? $\endgroup$
    – Lord_Farin
    Commented Sep 12, 2013 at 14:05
  • 1
    $\begingroup$ I get quite a few, but they are glaringly obvious and quick to dismiss. $\endgroup$
    – copper.hat
    Commented Oct 27, 2013 at 20:38
  • $\begingroup$ math.stackexchange.com/questions/535020/… is too broad for sure and to some extent lacks context. $\endgroup$
    – dfeuer
    Commented Nov 3, 2013 at 2:53
  • $\begingroup$ It appears that the audits have now been redesigned - clicking close on an audit wasn't an automatic pass for me just now, but rather gave me a chance to click an option first. $\endgroup$
    – user61527
    Commented May 22, 2014 at 5:46
  • $\begingroup$ While the thread is obviously useful as a way to vent frustration, seeing it bumped repeatedly gets old. I think it would be better implemented as a chat room, similar to Whining Room on Meta. Or just add the (whining) tag. $\endgroup$
    – user147263
    Commented Jul 30, 2014 at 17:46
  • $\begingroup$ @900sit-upsaday I agree. It was originally supposed to provide examples that could show how it might be improved, but it seems clear that this will not happen. And if you think it is annoying to see it bumped, consider getting a notification each time making you think "ahh, someone is speaking to me" only to see that it is just another answer to this question. $\endgroup$
    – Tobias Kildetoft
    Commented Jul 30, 2014 at 17:58
  • 2
    $\begingroup$ @TobiasKildetoft You can ask SE to be disassociated from this thing; I'm sure that seeing the situation they'll honor the request at once. $\endgroup$
    – user147263
    Commented Jul 30, 2014 at 18:00
  • $\begingroup$ Should we add the tag (big-list) to this question ? $\endgroup$
    – GNUSupporter 8964民主女神 地下教會
    Commented Dec 12, 2017 at 17:17

62 Answers 62

Reset to default
1
2 3
32
$\begingroup$

Side complaint: I seem to be getting an awful lot of review audits, maybe one per five legitimate review requests.

If you are asking me to spend my time performing community service, I would prefer you not waste my time with pointless "fake" tasks.

$\endgroup$
3
  • $\begingroup$ That would definitely be excessive. $\endgroup$
    – GeoffDS
    Commented Sep 17, 2013 at 0:42
  • $\begingroup$ I too am getting a lot of them. $\endgroup$
    – dfeuer
    Commented Sep 17, 2013 at 3:34
  • 1
    $\begingroup$ Recently the fake reviews have died down. Maybe I made it through some kind of "probationary period" with a ton of audits. $\endgroup$
    – user7530
    Commented Oct 2, 2013 at 0:06
31
$\begingroup$

The newest and one of the most ridiculous examples to date:

https://math.stackexchange.com/review/first-posts/109542

I failed this by clicking "add comment". That's right. You're not allowed to comment when you're being audited. I'm filing a bug report.

$\endgroup$
1
  • $\begingroup$ I just had a similar experience trying to leave a Comment noting Related Questions. The audit complains: "Your review was inappropriate. This was a high quality post and you should have considered leaving it as-is or even upvoting." $\endgroup$
    – hardmath
    Commented May 20, 2015 at 23:30
26
$\begingroup$

I have failed a few. One interesting thing is that, for "close" audits at least, the system lies about the vote count, showing 0 instead of the actual vote count. Which seems to be counterproductive: it appears they change the vote count to encourage reviewers to evaluate the question independently, but then they criticize the reviewer being too independent and not realizing that a question with a high vote count should not be closed!

Now that I realize this, I don't fail them anymore, because I check the vote count of the real question first. One examples I have been presented this way was A "non-trivial" example of a Cauchy sequence that does not converge?

$\endgroup$
3
  • 5
    $\begingroup$ Good idea. I took your suggestion to heart and found that a question I was supposed to review had been deleted, so selecting "close" earned me kudos for making the right choice. While I understand the reason behind audits, I think that on balance (at least for me) the chilling effect outweighs the didactic benefits. $\endgroup$
    – Rick Decker
    Commented Sep 13, 2013 at 1:04
  • $\begingroup$ Just for curiosity, what the system actually says if one fails the audit? EDIT: ok, I think that's the text shown at meta.math.stackexchange.com/questions/10836/… $\endgroup$
    – mau
    Commented Sep 13, 2013 at 13:30
  • $\begingroup$ @mau: meta.math.stackexchange.com/questions/10836/… is one of them, but the message is slightly different depending on which type of review you were performning $\endgroup$
    – Carl Mummert
    Commented Sep 13, 2013 at 13:33
19
$\begingroup$

https://math.stackexchange.com/a/508096/17596 showed up as a "first post" review. I found the animation far more distracting than helpful, so I downvoted (disagreeing, apparently, with fifteen folks who think it's the cat's meow). Review audit failed.

$\endgroup$
2
  • 3
    $\begingroup$ First post audits are the most ridiculous of the bunch! I've failed audits for leaving alone posts that, while not of the highest quality, had no actionable problems. $\endgroup$
    – user7530
    Commented Oct 27, 2013 at 20:04
  • 3
    $\begingroup$ I just had to review the first post math.stackexchange.com/questions/578662/… (with fake 0 score and fake user reputation). I added a comment because the OP made a significant change to the question and the thing was in front of me, but apparently adding a comment to a post that was better before an edit means I'm a bad auditor. $\endgroup$
    – Mark S.
    Commented Nov 24, 2013 at 16:44
19
$\begingroup$

I failed a review audit for this question because I went to close it as duplicate as I recognised the title and content of the post so thought it was an exact duplicate of itself (the review had artificially made it appear that the question was written 2 hours ago). This highlights a flaw in the system in my opinion.

$\endgroup$
13
$\begingroup$

The first time I got an audit, I thought it was a good idea. Well, not anymore.

I've failed some when we changed the policy, initiated by - I believe - this topic. Since the new "close" reason was included, I go for it almost on every question that was nominated for closing and lacks any context or effort (i.e., copy/paste questions). I myself nominate such questions for closing if there was a request for edit in the comments and the OP didn't respond in a few hours.

But, there are many questions, especially those that were asked before we took this more serious stand on copy/paste questions, that were upvoted quite nicely, so I've expected to fail each such audit.

Because of that, like Carl, I now open almost every question in new tab, and check that it's not a trap. It's a pointless waste of time, but my alternative is to get "caught" simply for not agreeing with other voters of that time, possibly before the new "close" reason was added (when I myself voted against closing such questions, due to the lack of a proper "close" reason).

The alleged purpose of the audits is to force reviewers -- actually voters, which is important! -- to pay attention. But that's the problem! As the site's help says (my emphasis):

Our sites are all intended to be a sort of representative democracy. Moderator elections are an important part of that plan, but voting on questions and answers is the primary mechanism through which the community governs the site on a day to day basis. Every user with sufficient reputation can exercise their right to vote, every day that they visit the site.

In a representative democracy, people vote according to their personal views. Be it for the views different than those of a majority, or simply for being lazy, a voter in a representative democracy is allowed to vote differently than the majority. That is the core of a democratic system, and audits measure how close a person's vote is to that of a majority, punishing those who are not in accordance. Completely antidemocratic, and so anti-self-description of the site. I don't mind strict rules, but then don't make audits based on votes which can be -- and often are -- not in accordance with those.

If lazy voters are a problem, then all the incentive should be removed from such voting, i.e., badges. Who wants to help the community, doesn't need badges; those who vote for those little coloured circles won't do it anymore if those "medals" are gone.

But the system, as it is, is rotten. IMO, of course.

Edit: I forgot... I sometimes click "close" not to actually vote for it, but to see what did others select as the reason. Just "off-topic" from the description is not as specific as seeing which of these, which helps me focus on that particular reason and see if I consider it valid. An audit would "catch" me there as well.

$\endgroup$
2
  • 4
    $\begingroup$ +1 There is something distinctly pointy-haired about the whole audit system $\endgroup$
    – user7530
    Commented Oct 28, 2013 at 18:46
  • $\begingroup$ @user7530 A perfect description! :-) $\endgroup$
    – Vedran Šego
    Commented Oct 28, 2013 at 18:55
11
$\begingroup$

I failed this close-vote audit. I wanted to vote to close it as a duplicate of this question, particularly since there was a comment on the original question suggesting it to be a duplicate. The suggested close reason was not as duplicate (I believe it may have been "too broad," but I don't recall exactly), but I do not agree that a close vote was inappropriate here.

$\endgroup$
0
9
$\begingroup$

This question post is a horrible "exemplar" of a post that should be reopened, according to a "reopen review queue audit."

In fact, the question should be put on hold, so the asker can improve it, and perhaps, and only then, reopened.

Including it as an exemplar of a post that users should reopen teaches users to reopen very poor posts, which is antithetical to the purpose of the reopen review queue.

The set of questions with a vote count of 5 or more, with no previous close votes, is not identical to the set of questions that should be reopened. The algorithm reflects the mistaken algorithms used in the review queue audits: high vote count greater than or equal to five net votes is not a reliable indicator of a good post. (This is evident in many cases on point point: there exist a sizable percentage of former Hot Network Questions that received "popular" votes, not because the question is one that should be kept.)

$\endgroup$
8
$\begingroup$

I just "failed" a reopen review audit. My philosophy is that the principle of res judicata applies to all reopen requests: that is, I will automatically reject any request to reopen a question that has not been edited since it was closed, unless strong reasons to reopen are given in the comments. The audit I failed showed no edits since closure and no comments at all. Although the question was allegedly closed as "unclear", I believed it to be actually closure-worthy as "off-topic: homework without a meaningful attempt" (although it turns out to have been answered by the asker some time later, so there's no actual need to close it). I just don't think that "no major problems" is a legitimate reason to reopen—if enough users voted to close, why should I substitute my judgement for theirs?

$\endgroup$
9
  • 2
    $\begingroup$ I disagree with your last line, in that sometimes closing is not clear cut. Perhaps starting a meta thread is appropriate when you come across debateable questions, but simply going "the first five people to the post held this opinion and who am I to question them?" isn't helpful. Although I do agree with the sentiment :-) $\endgroup$
    – user1729
    Commented Sep 18, 2013 at 15:57
  • $\begingroup$ @user1729: Certainly closing is not always clear cut. But the notion that the second set of five people to consider it are inherently more likely than the first five to come up with the "correct" response is absurd. I haven't been around long enough to know whether comments on the question or a thread on meta are the better approach, but it certainly seems to me that something either in or surrounding the question should change before it is reconsidered. "Give me another chance at the lotto please" isn't a reason. $\endgroup$
    – dfeuer
    Commented Sep 18, 2013 at 16:34
  • 5
    $\begingroup$ Yes notion that the second set of five people to consider it are inherently more likely than the first five to come up with the "correct" response is absurd, but so is the converse notion, that the first lot of people are inherently more likely than the first five to come up with the "correct" response. Also, when a question is in the reopen queue it is often because someone has voted for it to be reopened. So someone disagrees with the five. I believe their opinion should be considered. $\endgroup$
    – user1729
    Commented Sep 19, 2013 at 9:11
  • $\begingroup$ @user1729: I think it's reasonable to require that they explain that opinion in order for in to be considered. $\endgroup$
    – dfeuer
    Commented Sep 19, 2013 at 15:32
  • 1
    $\begingroup$ I would agree that an explanation would help, but I still think it is better to consider it than just to ignore it an pretend it hasn't happened. $\endgroup$
    – user1729
    Commented Sep 19, 2013 at 18:39
  • 5
    $\begingroup$ I feel the "res judicata" reasoning is wrong, because one cannot vote to (re-)open before a question is closed. So it is not the first five people that voted to close, but just some set of five people, among an unknown number of people who might have had other ideas. Also it is impossible to undo a close vote, I think even if the question is edited (before it is closed). $\endgroup$
    – Marc van Leeuwen
    Commented Sep 21, 2013 at 5:21
  • 3
    $\begingroup$ Your res judicata doctrine is an abnegation of responsibility, pure and simple. And since you do assume the responsibility of adding your voice to close votes, you’re demonstrably biassed in favor of closing questions. $\endgroup$
    – Brian M. Scott
    Commented Dec 1, 2013 at 4:43
  • $\begingroup$ @BrianM.Scott, I doubt you can see my close or reopen votes, but I take them all seriously. I feel great when I cast a reopen vote on a question I previously voted to close because the OP went back and addressed the problems that led to closure. When someone initiates a vote to reopen when nothing has changed, I think it's on them to explain why. $\endgroup$
    – dfeuer
    Commented Dec 1, 2013 at 5:13
  • 3
    $\begingroup$ @dfeuer: I can see the ones that you make through the review queues, and I can see the close votes that I’ve been recording as I come across them, which in your case are the main source since I started paying close attention. Your last sentence is just more evidence of your bias in the matter, since the reason is obvious: that person did not think that the question should have been closed in the first place. $\endgroup$
    – Brian M. Scott
    Commented Dec 1, 2013 at 5:21
8
$\begingroup$

I failed a First Post with this question for one of two reasons. I noticed two things at first glance:

  1. There was a mix of unTeXified and TeXified variables.
  2. The post mentioned that the numbers could be "very large," but didn't specify how large they could be, nor give much other context aside from "it's for a computer program."

What I intended to do (in some order) was to fix the mixed variables (which I have since done) and leave a comment to the effect of: "Out of curiosity, how large can $x$ and $y$ be? The best solution for you may depend on that, and on other context (like what sort of program this is for), so the more you can tell us, the better." I am not sure at this point whether I pressed 'edit' or 'add comment' first--I was too surprised to discover that I had failed. Once I failed and actually went to the question, of course, I discovered that I was not the only person who wondered how large the numbers $x,y$ could be--the $4$th comment down (at present) asks that very question, and there are other comments asking for more context.

$\endgroup$
8
$\begingroup$

I rarely review, but I thought I'd give it a try, and almost immediately failed an audit. I was presented with a very poor answer in the “late answers” queue. I opened the thread to which it was attached. I saw that the answer I was reviewing had been deleted several weeks ago, so I clicked the “no action necessary” button, and got this obnoxious response:

enter image description here

So that was a poke in the eye from the review system, and I will immediately return to my former practice of ignoring the review queue.

$\endgroup$
7
$\begingroup$

Probability and expectancy problem

This is supposed to be a "reopen queue audit question" and to "pass" the audit, one is supposed to vote to reopen.

This is a problem statement question and deserves to be downvoted and closed, and not reopened!

$\endgroup$
6
$\begingroup$

I failed this one. After the fact, I voted to close it and it has since been closed.

https://math.stackexchange.com/review/close/246038

It has no context or details. I hope the fact that it is now closed removes it from the review audit.

$\endgroup$
1
  • 9
    $\begingroup$ Now it will be moved from "known-good" to "known-bad" category, and used to trip up reviewers in the opposite direction. :) $\endgroup$
    – user147263
    Commented Jul 30, 2014 at 20:17
6
$\begingroup$

https://math.stackexchange.com/review/close/1056774

The entire question reads:

Are there any necessary or sufficient condition for when in a general convex hexagon, the lines formed by joining the midpoints of opposite sides are concurrent (intersect in a common point)?

This question lacks context. I can see that it might be interesting, but it appears entirely unmotivated to me. Based on the single not-deleted answer, it appears that this might have originally been a contest problem (and an original source would be valuable context for future users who might be reading off of the same list of contest problems). Frankly, not only do I think that this is a poor audit review question, I honestly think that it should be closed until the original asker provides some context.

$\endgroup$
5
$\begingroup$

I failed this audit because I wanted it add a comment. Not that I added a bad comment. For all any computer might know I wanted to say this problem is the best research direction for mathematics since the Langland's program. Que sera sera.

$\endgroup$
5
$\begingroup$

I failed this one, ironically because I put too much effort into it. It was presented as a first post with 0 votes, on a question that had been answered 10 days ago. I navigated to the question and saw it had good answers v similar to the one I was reviewing so tried to add a comment to say it had already been answered => FAIL. (What I should have spotted was that I couldn't see the username of the answer I was reviewing on the page. And one of the answers was suspiciously similar identical to the one I was reviewing...:)

Anyway I was already thinking the review audit sucked and was wasting my time-- can't it at least adjust so that those who pass more audits get fewer in future? But when you fail them it's a disincentive to continue.

$\endgroup$
1
5
$\begingroup$

https://math.stackexchange.com/review/close/1015074

This question is a fairly straight-forward "problem statement question" that I voted to close as "Lacking Context." I think that the level of the material (advanced undergraduate or introductory graduate level complex analysis) coupled with the (non-obvious) difficulty of the problem earned it several upvotes, and I am willing to concede that the dan_fulea's answer may be one of the few instances of a great answer justifying the preservation of a question-of-debatable-merit (note that answers are invisible in the "Close Votes" review queue), but I don't think that this is a good audit question.

$\endgroup$
5
$\begingroup$

https://math.stackexchange.com/review/close/1076507

The entire question reads:

I have been stuck on this problem for hours and have absolutely no clue how to go about it. Any help would be amazing.

The question itself is in the title, and is a pretty standard combinatorics question. I honestly don't understand how the question garnered five upvotes in the first place...

$\endgroup$
5
$\begingroup$

Yesterday I failed this Review audit, and when I posted an answer in the Reopen/Undelete thread, another user reported failing two similar audits, and someone else later failed another one. That same person failed another one of the same type less than 6 hours later.

The number of such posts by the same user (who also had a question deleted on MO, though I can't see it), their frequency (all posted within a few days) and the fact that they all link to the same website is suspect, but taken in isolation they don't look so bad. So these make poor examples for review audits.

$\endgroup$
1
  • 1
    $\begingroup$ The weirdest thing is why the deletion happened 7 months later. $\endgroup$
    – YuiTo Cheng
    Commented Apr 26, 2019 at 6:13
5
$\begingroup$

https://math.stackexchange.com/review/low-quality-posts/1480735

As far as I can see, the question is not visible in the audit link. The question is

Prove this sequence converges to $1/2$

There is a comment under the question providing a link to a duplicate. So, I voted to close as a duplicate and failed the review audit.

$\endgroup$
4
$\begingroup$

https://math.stackexchange.com/review/first-posts/174827

I failed this audit, because I tried to mark it as a duplicate. The audit said it has been asked (I think) 1 hour ago and I was absolutely sure, that I have read exactly the same question a few days ago. However, I had read exactly this question before, because that first post was a fake of the orginal good question...

$\endgroup$
1
  • 2
    $\begingroup$ Indeed, it seems that a minimal change that ought to be made to this system is that clicking the "close" button is not an automatic failure, when selecting an inappropriate reason would be the actual failure (after all, you have not actually chosen an action by clicking the "close" button). $\endgroup$
    – Tobias Kildetoft
    Commented Mar 5, 2014 at 13:05
4
$\begingroup$

I passed this first post audit by selecting no action needed (which I think is the appropriate result). The question has $4$ close votes currently, but is sitting at $+17$; considering that at least $4$ relatively high-reputation users think that the question is unclear to the point it should be closed (while many other users disagree), it seems inappropriate to use this question to test the reviewer.

$\endgroup$
4
$\begingroup$

I failed this one: https://math.stackexchange.com/review/low-quality-posts/512046

I understand that it is an interesting question that does provide a bit of background with a link to the problem. But there is no attempt at solving the problem, so I voted to close. Some will disagree on this, but the question doesn't (IMO) belong in the list of audit questions.

$\endgroup$
4
$\begingroup$

https://math.stackexchange.com/review/close/781215

I think this is a poor review audit. I voted to close as "too broad" because well...it's too broad! The first comment even says it's a "very broad question".

I voted to close as too broad outside of review queue. So hopefully it won't show up again.

$\endgroup$
4
$\begingroup$

Black holes and the Schwartzschild solution

This Question turned up in my Close Vote review (as an audit). Because the Question garnered five upvotes, the system considers it a model of something that ought not be closed. But despite the OP's declaration, "This is probably more a mathematical problem as far as I can see...", it was cross-posted to Physics.SE and consists of four sub-queries that, judging by the OP's comments under the Question, have a strong nexus with the physics of gravitational fields.

The OP accepted the lengthy Answer (with references) that was given at Physics.SE and did not accept either of the Answers given to the Math.SE version. Therefore I feel this really should be closed as off-topic here. Those interested in the subject matter will still be able to follow links to the Physics.SE post (no migration is required).

$\endgroup$
4
$\begingroup$

I got this one:

https://math.stackexchange.com/review/close/1194064

I passed it by leaving it open, but it seems not nearly clear-cut enough to qualify as a good review audit. The asker has not provided much in the way of evidence of effort.

I also floated this in a discrete Meta question, before posting here.

$\endgroup$
4
$\begingroup$

Are there any properties of sup?

The question title is not informative, and suggests a very broad question (rather than the fairly focused question which is actually presented). The question itself reads:

Can I say: $$\sup |f(x)-h(x)+g(x)-g(x)| = \sup|f(x)-g(x)|+ \sup|g(x)-h(x)|$$

I can't seem to find any properties of $\sup$ , so i am wondering if there is any properties that we can generalize to sup of some functions restricted on domain $x \in [0,1]$ for example

This question is (to me) unclear. I don't know if the asker is trying to prove this identity (which isn'g going to be possible), or if they are looking for conditions under which the identity will hold.

It may be worth noting that I am not the only one to have been stung by this audit: https://math.stackexchange.com/review/reopen/1395894 .

$\endgroup$
1
  • $\begingroup$ Yep, not a sound exemplar for a review audit. $\endgroup$
    – hardmath
    Commented Jun 5, 2020 at 1:55
4
$\begingroup$

https://math.stackexchange.com/review/low-quality-posts/1407444

Even if we accept that questions may be appropriately answered with hints (and the consensus seems to be that hint answers are okay in certain circumstances), it is not at all clear to me that this is, in fact, a good hint. I can see definite differences of opinion about the quality of this answer or its appropriateness for Math SE. As such, it seems like a poor choice for audits.

$\endgroup$
1
  • 2
    $\begingroup$ I agree with you. This would have been a good Comment, and if intended to provoke a Socratic dialog, perhaps Robert could be cajoled to add something more to the post. $\endgroup$
    – hardmath
    Commented Jun 20, 2020 at 2:42
4
$\begingroup$

https://math.stackexchange.com/review/close/1408914

The question is long but, at the end of the day, it is nothing more than a statement of a result which the asker wants proved. There is no additional context. This is even pointed out in the comments:

It would be helpful if you could share some of your thoughts on the problem: Where did you encounter the problem; do we have a reason to suspect that this problem has a nice solution? What have you tried? How far did you get before you got stuck? Are there any results/techniques that you suspect would be applicable here? — Omnomnomnom

$\endgroup$
4
$\begingroup$

enter image description here

Though not a major one. I tried to edit the ${\boxed {tea+wine=s}}$ into ${\boxed {\text{tea+wine=s}}}$ and the one following -> (s-wine)...And I failed.

$\endgroup$
2
1
2 3

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .