26
$\begingroup$

Last August, so-called "review audits" were introduced on MSE.

These are fake review samples added to the review queues, designed to catch those users who (in a dash to get a badge) practice mindless button mashing instead of careful consideration while reviewing.

Prior to the introduction, people were enthusiastic -- me included.

However, in three months, we have generated:

in which threads, more and more people start stating their resentment of the whole auditing thing, with wordings like Vedran Šego's:

The first time I got an audit, I thought it was a good idea. Well, not anymore.

It is time to evaluate. Some valid concerns have been raised, but not addressed by the SE developers. A tour around Meta.StackOverflow reveals a similar picture. The developers are unwilling to invest the time to do something about at least the most-heard complaints about the audit system.

However, the current implementation has a far too high "false negative" statistic: Too many conscientious reviewers are being shouted at by the audit system for trying to be helpful to their best judgement.

Therefore, pending improvements to the audit system to reduce the "false negatives", I want the audits to be disabled on Math.StackExchange. Do you agree?

Please find two polling answers below -- you can use an upvote to indicate your preference.

Seeing as (to the best of my knowledge) the audit system is standardised SE-network-wide, the relevant feature requests for improving the audit system should be filed at Meta.StackOverflow.

The vote tallies have more or less crystallised. Disregarding downvotes, and taking into account my own, the final result is:

  • Disable audits: 24 votes
  • Keep audits: 15 votes

This falls short (and has consistently fallen short throughout the poll) of the "qualified majority" (two thirds) I had in mind for requesting the termination of the audits. That means the audits remain, if I am to decide.

$\endgroup$
9
  • 1
    $\begingroup$ Where can I see the false negative statistic? And aside from bruised egos, what suffering has the system inflicted on any particular users? Did we have a bunch of people get review ability disabled or something? $\endgroup$
    – rschwieb
    Nov 15, 2013 at 19:51
  • 1
    $\begingroup$ @rschwieb It's a mood thing -- I took it that people won't be posting every incident on meta. There's generally enough thick skin around MSE. I guess the occasional failure due to mere unclarity is bearable to most. But there have been pointed out quite a number of design flaws in the audit system. E.g., the prospect of getting yelled at for trying to comment is just infuriating. It is these problems that instigated my request. $\endgroup$
    – Lord_Farin
    Nov 15, 2013 at 20:06
  • 2
    $\begingroup$ We're agreed it's got lots of bugs, and it is a bit disappointing that it might not get serviced soon. I'm very interested in seeing hard data on how "real" all of the anecdotal complaints have been. My own experience is that (even with its current flaws) the system is completely tolerable. While people might have thick skins in their interactions with other users, I don't believe that necessarily applies to this situation, where a newfangled machine has come along to taunt us. I think people are far more likely to complain loudly about that. $\endgroup$
    – rschwieb
    Nov 15, 2013 at 20:13
  • $\begingroup$ Anyway, is audit performance data accessible from the data here? $\endgroup$
    – rschwieb
    Nov 15, 2013 at 20:15
  • $\begingroup$ @rschwieb Not that I know of. $\endgroup$
    – Lord_Farin
    Nov 15, 2013 at 22:24
  • $\begingroup$ s/false negative/false positive/ $\endgroup$
    – dfeuer
    Nov 18, 2013 at 11:05
  • $\begingroup$ Hi @dfeuer : I don't know much about accessing data here. Does that comment mean you've found a way? If you did, I'd be grateful for a link to the query. (Or if it's a joke that I don't understand, apologies :) ) $\endgroup$
    – rschwieb
    Nov 20, 2013 at 13:59
  • $\begingroup$ @rschwieb, the term "false negative" in this post should probably be replaced by the term "false positive" in order to get the right connotation. A "false negative" in medical/psychological testing is a test that fails to detect a disease that is actually present. Lord_Farin should have used the term "false positive", meaning that a disease was detected when none exists. $\endgroup$
    – dfeuer
    Nov 21, 2013 at 1:53
  • $\begingroup$ @dfeuer No. If we take the perspective that review audits check for "proper reviews" (which they do, IMO), a "fail" amounts to a "false negative". I was a bit ambivalent on which I should choose, and therefore I explained verbosely what I meant with it. "Negative" corresponds better to what is being conveyed by it. $\endgroup$
    – Lord_Farin
    Nov 21, 2013 at 8:23

2 Answers 2

Reset to default
31
$\begingroup$

Yes.

Please disable the review audits.

$\endgroup$
1
  • 1
    $\begingroup$ The vote count for this option should be increased by 1 to account for my own vote. $\endgroup$
    – Lord_Farin
    Nov 14, 2013 at 12:12
9
$\begingroup$

No.

Please keep the review audits.

$\endgroup$
15
  • 8
    $\begingroup$ I've voted for this option, not because I think the audit system is perfect as it stands, but because I think it serves a useful purpose (keeping reviewers on their toes and discouraging lazy reviews). I do agree that there need to be improvements, but the problems with the current system are not, in my opinion, drastic enough to warrant disabling them altogether. $\endgroup$
    – Dan Rust
    Nov 15, 2013 at 0:09
  • 2
    $\begingroup$ Most of the comments I've seen sure like people are just annoyed an automatic bell is bzzting them. I'm not sure what my raw number of reviews was since I started, but I know for sure I only ever failed an audit once, compared to lots of audits. Even if you get zinged once every ten audits... who cares? Nobody is expecting anyone to have a perfect audit record. How many people have failed more than 10% of their audits? $\endgroup$
    – rschwieb
    Nov 15, 2013 at 19:45
  • 2
    $\begingroup$ By the way, does anyone know if questions that cause an inordinate amount of false positives are filtered out of the audit system? It seems like if a large percentage of reviewers failed a particular audit, it should just get yanked and whatever demerits it gave out be reversed. $\endgroup$
    – rschwieb
    Nov 15, 2013 at 19:47
  • $\begingroup$ or false negatives I guess is what I meant... $\endgroup$
    – rschwieb
    Nov 15, 2013 at 19:53
  • 11
    $\begingroup$ Keep in mind that this option has received downvotes for some reason (which means some people effectively 'voted twice'). Counting just the upvotes, the gap isn't quite as big as it looks. $\endgroup$
    – Dan Rust
    Nov 16, 2013 at 12:27
  • 5
    $\begingroup$ @Daniel Rust: I think that the main effect of the review system, for me, is to discourage careful reviews rather than discouraging lazy reviews. To pass the review audits, I have to ask "what do they think a generic person would say" instead of considering the circumstances and giving my opinion. $\endgroup$
    – Carl Mummert
    Nov 18, 2013 at 13:26
  • $\begingroup$ Keeping reviewers on their toes is important. So if reviews are deactivited temporarily, please do not make it known. $\endgroup$
    – Nameless
    Nov 18, 2013 at 14:49
  • 2
    $\begingroup$ @rschwieb: Actually, I would expect a significant number of users to have perfect audit records, if the audits were actually correct. $\endgroup$
    – Brian M. Scott
    Nov 20, 2013 at 2:49
  • 1
    $\begingroup$ @BrianM.Scott And surely there are: there are probably a lot of people who only had one or two audits and aced them. "Significant" is too vague of a word to base anything on, but if you're expecting a majority of users with a relatively high level of review usage to have perfect records, that seems excessively unrealistic. Expecting a program to execute a dynamic subjective test of human users such that the aggregate result is perfect most of the time is just a pipe-dream. Expecting a high percentage of users to have a high percentage of passed audits seems right, though. $\endgroup$
    – rschwieb
    Nov 20, 2013 at 13:53
  • $\begingroup$ @rschwieb: Oh, I don’t have any such expectations. My comment was mostly a condemnation of the idea of automated audits. $\endgroup$
    – Brian M. Scott
    Nov 20, 2013 at 19:04
  • 2
    $\begingroup$ @ whoever is reading: Bad editing practices damage site content. Audits retard bad editing practices. The main side effect is a little wasted time on the part of reviewers but not really that much. As long as the system isn't causing major problems, it seems better than nothing. Getting caught by a dumb audit every once in a while shouldn't be upsetting or even surprising. It's just a "dumb machine," after all. $\endgroup$
    – rschwieb
    Nov 20, 2013 at 19:29
  • 2
    $\begingroup$ @rschwieb: It’s not clear to me that MSE suffers much from the kind of bad editing practices that the audit system was designed to combat. Moreover, most of the bad edits that I’ve noticed were apparently spontaneous edits of new questions, not from the review queue. Thus, it’s not at all clear to me that the benefit of the audits to MSE outweighs the drawbacks. $\endgroup$
    – Brian M. Scott
    Nov 24, 2013 at 19:30
  • $\begingroup$ @BrianM.Scott that's certainly true, and it's unclear how one would determine anything from data. Unfortunate... $\endgroup$
    – rschwieb
    Nov 24, 2013 at 23:55
  • $\begingroup$ I think the audits are a good idea, they are pretty obvious and take little time to click through. If only mistakes in my answers were so obvious... $\endgroup$
    – copper.hat
    Nov 25, 2013 at 1:01
  • 3
    $\begingroup$ I have been reviewing a bit more lately to get a feel. I have gotten to see a dozen or so of these inserted extras, and they were all fairly obvious (to close or not to close). While going through the review queue I do "right click->open the question in a new tab" often, so may be I "cheated". I do recommend that every body would do that. Reviewing should not be done on autopilot. $\endgroup$
    – Jyrki Lahtonen
    Nov 26, 2013 at 17:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .