The site asked me today to review 10 questions for a "site self-evaluation."
What is this? How will the data be used?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
7
The evaluations were meant for beta sites, but SE is also trying them out on graduated sites. This kind of evaluations used to be performed by SE employees themselves, but that didn't scale with the ever increasing number of sites. Additionally, SE employees often don't have the necessary domain knowledge to accurately judge the usefulness of the content.
The existence of this evaluation doesn not mean that SE is considering to shut down any graduated site. No graduated site was ever shut down, and I don't expect something like that to happen outside of exceptional circumstances.
The community evaluation is a tool, it is far more useful on beta sites than on graduated sites. If you don't see the value in the self-evalution, just don't participate. Participation is completely voluntary, you're free to ignore it entirely. Just marking every question as excellent without an honest evaluation is just wasting your time and producing noise in the result which you would have to filter out, if you want to try to get anything useful out of the data.
-
4$\begingroup$ But how were these ten posts chosen? Are they arbitrary? Isn't that like testing one cup of water from the ocean and deducing that there are no fish and hardly any pollution in the entire ocean; or that the entire ocean is full of amoebas and pollution? (Depending where you got that cup of water from...) $\endgroup$– Asaf Karagila ModCommented Feb 3, 2013 at 21:25
-
$\begingroup$ It's a random sample of questions from the past 90 days, @Asaf. As it was intended for smaller sites (those still in beta), the sample size may easily be too small to be useful on graduated sites. $\endgroup$– Shog9Commented Feb 3, 2013 at 21:32
-
2$\begingroup$ @Shog: I see. This was my point about sample size, it's too small to possibly conclude anything meaningful. Perhaps set it to a percentage of the questions asked within the last 90 days (which should, by my estimates, be around 6-10k posts), maybe only questions which have votes or answers, which may cut down the amount by a third or so, then take like 1% which would be 40-60 questions instead. But what do I know about these things! :-) $\endgroup$– Asaf Karagila ModCommented Feb 3, 2013 at 21:35
-
$\begingroup$ @Asaf: well, I suspect before picking a sample size it would be necessary to have a specific goal in mind for the results. 10 was chosen for beta sites since it's small enough that you can reasonably expect folks to actually review all of them in the set, and large enough to give a reasonable overview of the (usually fairly small) volume of traffic on such sites (in some particularly quiet betas there are actually less than 10 questions eligible!) But since obviously we're not checking Mathematics for graduation eligibility, we've no specific interest in the results of these reviews. $\endgroup$– Shog9Commented Feb 3, 2013 at 21:38
-
1$\begingroup$ @Shog: That is also very true. But you can always allow each user to review at most 10-15 questions, even if your sample size is roughly 60-70 questions. You simply define that after $n$ reviews a question is taken off the queue of everyone. $\endgroup$– Asaf Karagila ModCommented Feb 3, 2013 at 21:41
-
6$\begingroup$ I think it is worth noting that, at least on mathematics, few people are competent to really judge the quality of all test questions. Specialization matters a lot here. $\endgroup$ Commented Feb 4, 2013 at 6:51
-
1$\begingroup$ On the beta sites, the ten questions were actually discussed which makes a lot more sense if people articulate why they do not deem a question excellent. $\endgroup$– PhiraCommented Feb 5, 2013 at 10:55
$\begingroup$
$\endgroup$
2
As Mad notes, this was designed for beta sites - we do periodic evaluations internally to decide whether or not a site is ready to graduate, and exposing this part publicly provides a way for a growing site to engage in a bit of self-reflection.
While this was never intended for sites that had already graduated, after its introduction we received several requests for it to be made available on graduated sites, so as of Friday it's available everywhere.
That said, if it's not useful here we can disable it. We're not using the data ourselves, so if no one here wants it there's no point in keeping it.
-
1$\begingroup$ How does one even access this information in a useful manner? $\endgroup$– Asaf Karagila ModCommented Feb 3, 2013 at 21:51
-
$\begingroup$ There's a summary page; only available to mods right now, so they'd have to post it somewhere after the eval ended. $\endgroup$– Shog9Commented Feb 3, 2013 at 22:44