Some questions in the review queue are there only to test the reviewer. Who chooses those?

  • $\begingroup$ I think a side-effect (even if occasionally contentious answers/questions occur) is "to keep reviewers on their toes" (not an auto pilot). I've been very occasionally off put by what I think to be a poorly chosen model for such reviews, but I think it does more good than harm. $\endgroup$ – amWhy Aug 8 '17 at 23:30
  • 1
    $\begingroup$ All of the proposed edits that were tests that I've ever seen were edits that were absurd. I've seen several tests in the review queue where I voted to close a question and I was told that it was a test and I passed. I've seen only one where I voted to leave a question open that turned out to be a test and it said I failed, and not only that I failed but that the question was seriously flawed. I read the question patiently and carefully with that in mind and continued to think it should be reopened. Conjoin that with the overwhelming fact that decisions to close questions$\,\ldots\qquad$ $\endgroup$ – Michael Hardy Aug 9 '17 at 0:33
  • $\begingroup$ $\ldots\,$on mse are generally done ineptly except in the most obvious cases, and it's clear that the "test" is garbage. $\endgroup$ – Michael Hardy Aug 9 '17 at 0:33

From the capital-M Meta FAQ on review audits,

Review audits are chosen automatically. The system isn't perfect, meaning that every so often a post slips through normal community detection, causing the system to expect the wrong type of action against a post.

The answer goes into a little bit more detail about the criteria, for some of these. For example, questions that are "highly-voted with no close votes" can be chosen as tests where the reviewer is expected to keep vote to leave open.

  • $\begingroup$ Why are votes to close as a duplicate never tested? Why aren't suggested edits tested with more realistic edits (any one who can't spot a dismal edit (which in testing, should all be rejected) should be banned from the suggested edit review, given the yawn-triggering edits proposed currently)? $\endgroup$ – amWhy Aug 8 '17 at 23:28
  • $\begingroup$ With respect to the comment above, I have never seen, as a test suggested edit post, a post which should be approved, and not rejected. Currently, the rule of thumb is that when confronting preposterous (suggested) edits, always reject it. $\endgroup$ – amWhy Aug 8 '17 at 23:33
  • 1
    $\begingroup$ Don't get me wrong; but currently, I think the current "suggested edit" tests are so absurd that only a zombie can miss it's absurdity. Are there any ways that the pool of potential test questions in the "suggested edit" queue" which are actually significantly improved by an editor, so that those rejecting it (in a test) would see $$\bf{\Large\text{STOP!}}$$ $\endgroup$ – amWhy Aug 9 '17 at 17:21
  • $\begingroup$ I agree that the edit testing could be improved a lot, and should involve some sort of real-world examples instead of the random nonsense it currently is. But one issue, I suppose, is that edits are pretty much never reviewed by more than 3 people. On the other hand, close/leave-open audits are based not only on the 5+ review votes, but also the overall score on the post. There's a lot more data there. $\endgroup$ – user296602 Aug 9 '17 at 17:26
  • $\begingroup$ user296602 I do acknowledge those points you make. So it would require other sorts of data, which isn't so readily counted. When is SE getting into the artificial intelligence business? ;) $\endgroup$ – amWhy Aug 9 '17 at 17:44
  • $\begingroup$ @amWhy Sounds like a good question for AI SE :) $\endgroup$ – user296602 Aug 9 '17 at 19:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .