6
$\begingroup$

I got this message today.

Who is assessing the quality of my reviews, and how?

What am I supposed to be "more careful and critical" about? I haven't failed any audit recently, at least not one that the system informed me about.

Edit: to answer questions in the comments, if I click on the review icon, this is what I see

And by searching on this issue I found the "review" url, where I see this

$\endgroup$
  • 1
    $\begingroup$ So, I can only speculate about "issues" with reviews that might arise outside of Review Audits. One is tangential, submission of flags that are "declined" in substantial numbers. Another possibility is failure of consensus with other reviewers, something that would only emerge in larger statistical samplings for a given Review Queue. $\endgroup$ – hardmath Jan 25 '19 at 2:14
  • $\begingroup$ The consensus thing is what worries me. I know I don't share the view of several of the people who are most active reviewing. I find it worrisome that the system penalizes me for that. $\endgroup$ – Martin Argerami Jan 25 '19 at 3:47
  • 3
    $\begingroup$ Perhaps screenshot would increase the chance that somebody is able to help. Does the message contain a link? Did it appear in the inbox? Did it appear when reviewing? Isn't this a moderator message? $\endgroup$ – Martin Sleziak Jan 25 '19 at 6:02
  • $\begingroup$ This sounds like a canned message the system autogenerated. Are you actually getting "penalized"? For example, are you currently banned from reviewing? I'm not 100% but I think only diamond mods can issue review bans. Those are very rare. I only ever issued a handful of those during my term. All for blatant violations (roboreviewing at a rate of 6 seconds per post, and/or letting spam posts through). If your close/do not close stats are highly unbalanced that may count against you, but I would be surprised if that were the case. $\endgroup$ – Jyrki Lahtonen Jan 25 '19 at 7:09
  • $\begingroup$ @JyrkiLahtonen I think that temporary review bans are automated, not issued by mods. (I mean the one accompanied by the message: "You've failed too many recent review audits - looks like you might need a break. Come back in in 7 days and try again." Meta: Why am I getting the message “You've failed too many recent review audits”?) However, since the OP says he did not fail any review audits this is not the case here - I am just reacting to "I think only diamond mods can issue review bans" from your comment. $\endgroup$ – Martin Sleziak Jan 25 '19 at 7:30
  • $\begingroup$ Ok, @MartinSleziak. I forgot about those. Thanks for the clarification. I was similarly reacting to "penalized". $\endgroup$ – Jyrki Lahtonen Jan 25 '19 at 7:41
  • $\begingroup$ The posts discussing the "Come back in 2 days to continue reviewing." message that I found seem to be about review audits (and the first part of the message is different). $\endgroup$ – Martin Sleziak Jan 25 '19 at 11:08
  • 4
    $\begingroup$ Perhaps there is some merit to this message, @MartinArgerami $\endgroup$ – amWhy Jan 25 '19 at 14:09
6
$\begingroup$

The mod team periodically evaluates the review stats of the community to see if there are outliers with respect to some statistics, and we act upon those if we perceive problems.

We understand that there is a wide spectrum of different opinions regarding what should be closed/reopened etc. However, when doing review, people are contributing to the site, so its standards (as a community) should also contribute. One should attempt to consider both their personal perspective and the community perspective to some extent. (This is intended for all people in review, not one side or another or yet another.)

This last paragraph may be a good rule of thumb to strive for, and is essentially what you should be "more careful and critical". But, more practically speaking, it suffices to have a balanced review process.

$\endgroup$
  • 5
    $\begingroup$ Thanks for the reply, Aloizio. But I still don't understand what's expected. The message you are giving me seems to be clear, that I should vote to close questions or not, even if I don't think they should be, to be "balanced". Or, if I'm doing something explicitly wrong, please let me know which of my votes is "wrong". $\endgroup$ – Martin Argerami Jan 25 '19 at 10:42
  • 2
    $\begingroup$ @Aloizio Thanks for the explanation and alert to review abuse. Stats reveal a lot: Some measures need to be taken when, unless after one checks and realizes it's an audit question, one, on principle votes to keep open each and every non-audit post nominated for closure, and/or to reopen each and every non-audit candidate for reopening. $\endgroup$ – amWhy Jan 25 '19 at 14:06
  • 4
    $\begingroup$ @MartinArgerami Yes, sorry if the answer is not completely explicit, but there is an intention to not divulge your specific statistics and also to avoid Goodhart's law. Ideally (in order to avoid said law), you should invert the order: you should try to be more balanced, and then vote to close questions or not, even if a priori you don't think they should be. $\endgroup$ – Aloizio Macedo♦ Jan 25 '19 at 18:15
  • 3
    $\begingroup$ @Aloizio: I would still like an answer on the meaning of "balanced". Now that I have access again to the review stats, I see lots of users who vote to close 20/20, or 19/20. Are those people being penalized for being "unbalanced"? If they aren't, I'm really not understanding where you are going with your answer and your comment. $\endgroup$ – Martin Argerami Jan 28 '19 at 1:45
  • $\begingroup$ @MartinArgerami The total amount of reviews being low is a relevant variable to consider. But yes, for the question in general, people who vote to close inadvertently are also penalized. If you feel that someone isn't, please flag it so that we can take a look. $\endgroup$ – Aloizio Macedo♦ Jan 29 '19 at 15:28
  • $\begingroup$ Fair enough, but I don't think I have a way to see that someone is penalized? $\endgroup$ – Martin Argerami Jan 29 '19 at 15:32
  • $\begingroup$ @MartinArgerami I don't think that is an issue. Flag it if you perceive it as a recurrent problem. If they have not been penalized, then it may be helpful to bring it to our attention. If they have been, then it is just no harm and would be marked a helpful flag nonetheless. $\endgroup$ – Aloizio Macedo♦ Jan 29 '19 at 15:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .