Can hints and PSQ be removed from the test of the review queue?

The reason I ask is because some PSQ can be extremely interesting and don't get closed or not interesting and get closed. The PSQ that are considered high quality are easy to fail because there is no work shown so recommending close or deletion is reasonable. Now, I could just right click and open the post in another window to see if that post was removed or not but then there is no point of testing me if I am just going to do that. Hints have similar fates. Some stay and others go.

Wouldn't it be better to not have test question be ambiguous in choice of action or on topics that the community is divided on?

| |
  • 14
    $\begingroup$ How exactly is the audit system supposed to figure out which questions are PSQ and which are not? $\endgroup$ – user9733 Feb 13 '15 at 6:54
  • $\begingroup$ @MadScientist by human intervention. $\endgroup$ – dustin Feb 13 '15 at 13:30
  • $\begingroup$ probabaly another SE site there can be created for interesting PSQs. $\endgroup$ – user795571 Feb 14 '15 at 2:19
  • $\begingroup$ @user795571 I am only speaking of implementing this here. Every site has their own standards so I by no means say enforce this on them. $\endgroup$ – dustin Feb 14 '15 at 2:21
  • $\begingroup$ That was just a suggestion so that the interesting PSQ can survive and the problem of PSQs can be solved. $\endgroup$ – user795571 Feb 14 '15 at 2:29
  • $\begingroup$ @user795571 I am not saying remove the PSQs from the site just the audits. The interesting PSQs will still be on the actually site. $\endgroup$ – dustin Feb 14 '15 at 2:30

The questions used for review audits are selected automatically, based on some algorithm that among other things incorporates vote tally.

Since the response to PSQ-like questions on this site can be quite erratic, to say the least, and varies wildly from person to person, they cannot really make for good review audits.

On the other hand, there are ample review audits that do the job of sifting out robo-reviewers properly. You're not among those, so you will pass a good fraction of your review audits.

Thus, the review audit system is not ideal, but there's not too much too be done about it; I would say that as long as you're not failing a lot of audits, you shouldn't care too much. The system was designed while keeping in mind the existence of questions with a mixed response, so the odds of getting review-banned while doing bona fide reviews are very small.

| |
  • 2
    $\begingroup$ Or, you know, check the question outside the review queue (i.e. use the "link" option first) and see if the votes, user reputation, and whatnot matches the information on the review. If not, then it's an audit. $\endgroup$ – Asaf Karagila Feb 13 '15 at 11:21
  • $\begingroup$ With automated systems, most companies check on the system and make adjustments and instead of just letting it do its thing indefinitely. All we would need is some human intervention by say the mods to look at a few review questions in the test queue to deem if they should stay or go. That could be done by a simple process of every 6 months the mods scan the questions and mark the ones that shouldn't be included. This process would slowly refine the test queue. $\endgroup$ – dustin Feb 13 '15 at 21:41
  • $\begingroup$ @dustin It looks like your "simple process" involves machine learning. (It also looks like it lets a very small subset of the community (moderators) more or less define the parameters for everyone else) $\endgroup$ – Milo Brandt Feb 14 '15 at 0:44
  • $\begingroup$ @Meelo then wouldn't be defining parameters just removing post that the community is undecided on like PSQs and hints. If a mod deems a post not a good fit, the algorithm would then know that post is off limits. Eventually, over time, all the PSQs and hints would be taken out of the test queue. $\endgroup$ – dustin Feb 14 '15 at 0:53

Review audits work under the assumption that the community is reasonably consistent when closing questions. If that assumption is violated, audits will be far more unpredictable than they should be.

There is almost nothing the review system can do about this, the audits are selected automatically. This selection relies on signals from the community, if the community is not consistent, audits can't be consistent as well.

But breaking audits to some extent is the least of the problems caused by inconsistent moderation. It can also lead to a very confusing experience for new users, close/reopen wars and the associated drama.

The way to solve this is to get the community to agree on a useful and consistent position on this kind of question. This is of course far easier said than done, but it is the only way to resolve the issue.

| |

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .