1. Take only real examples from real review queues. Don't tell me there aren't enough real rejected edit suggestions and the system has to make up absurdly unbelievable ones.

  2. Present the audit exactly as it appeared when it first entered the review queue, with the same text, comments, votes, closure reasons, etc.

  3. Test the audit samples before use by presenting them to several more reviewers. Only samples that indicate a strong consensus should be used for audits.

  4. Measure audit sample performance in use: if more than, say, 10% of reviewers get it "wrong", scrap it.

  • 3
    $\begingroup$ The "suggested edit" review audits are particularly bad. You can spot them at first sight, without reading a single letter. $\endgroup$
    – azimut
    Nov 3 '13 at 21:11
  • 2
    $\begingroup$ Re: 1, approving or rejecting a suggested edit involves judgment calls. When is an edit too minor? Is it acceptable to remove "Thanks you so much in advance!" from the problem statement? Reasonable people can disagree about such things. Disagreeing with an earlier decision on a review item is not a sign that the reviewer isn't paying attention. $\endgroup$
    – user103254
    Nov 3 '13 at 21:52
  • 2
    $\begingroup$ To add to the reasons that edit review audits are particularly bad - the generating algorithm seems to be especially slow and can take up to 10 seconds to load a review. Based on this alone it's almost trivial to know when an audit is coming because it takes so long to be generated. $\endgroup$
    – Dan Rust
    Nov 3 '13 at 23:56
  • $\begingroup$ @user103254, that's why the samples should be tested to make sure they're not controversial. $\endgroup$
    – dfeuer
    Nov 4 '13 at 0:50
  • 6
    $\begingroup$ @dfeuer: the point is that if the system makes up absurdly unbelievable ones, then those people who do fail the review audits have absolutely zero cause for complaint. Your version would be testing a user to see if he or she agrees with community norms; the current system version tests to see if the user is paying a nontrivial amount of attention (instead of just clicking through every review item in order to earn a badge or two). $\endgroup$ Nov 4 '13 at 8:26
  • $\begingroup$ I agree with @WillieWong, here. I actually find the "suggested edit" audits entertaining! (Some have actually made me laugh out loud at the absurdity.) Absurd as some might be, they do keep reviewers "awake" and reveal reviewers who are operating with "automatic pilot" approvals. $\endgroup$
    – amWhy
    Nov 4 '13 at 14:22
  • $\begingroup$ @WillieWong Question not worthy of a meta post: Do reviewers who (correctly) respond to review audits at least given credit for the proper review? (i.e., a +1 in their review tallies?) $\endgroup$
    – amWhy
    Nov 4 '13 at 14:26
  • $\begingroup$ @WillieWong, this may be reasonable, but in that case, all the other kinds of audits fail to live up to that standard! $\endgroup$
    – dfeuer
    Nov 4 '13 at 15:07
  • $\begingroup$ @amWhy: I don't think so (disclosure: I have not fully checked MSO). I think that if you receive certain number of strikes within a moving window, you will temporarily have your reviewing privileges revoked, and that's it. So I suppose doing it "right" will help prevent that from happening. $\endgroup$ Nov 6 '13 at 8:53
  • $\begingroup$ @amWhy I haven't paid attention (oops), but I have the firm impression that my review tallies are incremented after passing a review audit. $\endgroup$ Nov 6 '13 at 14:18

You must log in to answer this question.

Browse other questions tagged .