42
$\begingroup$

Since October 2015 a "How to ask?" page is shown before a user asks their first question on this site, and they need to confirm that they read it; see here: Show “how to ask” advice before a new user asks a question. (The post announcing that this was implemented is from October 15.)

Do we have some statistics which can help to evaluate whether this change has made some impact? (This question is somewhat similar to: Registration to ask a question – has it made a difference?)

Basically, I am asking two questions in one:

  • Which statistics should we look at?
  • Where can we get those statistics?

Which statistics should we look at? Which numbers would describe changes in positive direction? Should we look at percentage of questions that were deleted? Percentage of question (or first questions) that were closed?

Where can we get those statistics? There is a page with site analytics, however, I am not sure whether this is of much help here. (I do not see anything about closures and deletion there. It is possible that moderators have access to more detailed stats in the site analytics.) We could also use some data from Data Explorer (SEDE). (Some data will be necessarily skewed, since SEDE contain only limited information about deleted posts in the PostsWithDeleted table. However, I have been told by more experienced users that CMs have the option to run SEDE queries also on the data-set, including the deleted posts - so if we are able to come up with something useful, we might ask them for help.)

$\endgroup$
6
  • 12
    $\begingroup$ I should mention that my motivation to ask this is partly because of discussion on Meta MathOverflow: Should users be shown some basic information before posting the first question? Knowing to which extent this is actually efficient might be helpful when discussing whether such change should be done on other SE sites. $\endgroup$
    – Martin Sleziak
    Commented Jan 14, 2019 at 5:57
  • 9
    $\begingroup$ I suppose the proper way to do it would be to randomly show the page to some new posters but not others, then compare suitable statistics for the two groups—assuming (i) that was practical and (ii) people would tolerate getting lots of questions from people who hadn't been shown the page. That way you'd avoid inadvertently measuring things like changed toleration levels towards poor quality posts. $\endgroup$
    – timtfj
    Commented Jan 14, 2019 at 15:34
  • 4
    $\begingroup$ You'd also be measuring what happens now (with more people posting from smartphones, for example) rather than what happened a few years ago under possibly-different conditions. $\endgroup$
    – timtfj
    Commented Jan 14, 2019 at 15:43
  • 5
    $\begingroup$ You could also look at votes and edits on first posts before Oct 2015 and after Oct 2015. $\endgroup$
    – rene
    Commented Jan 14, 2019 at 20:33
  • 2
    $\begingroup$ I don't know the answer to this question, but I should mention we've been thinking about ammending that page lately. Suggestions about what to change or add would be helpful. Here's a chat so the comments don't get cluttered up. $\endgroup$
    – Alexander Gruber Mod
    Commented Jan 15, 2019 at 8:20
  • $\begingroup$ I will just add a link to a post which discusses statistics related to the same change on ServerFault: Should folks have to click through an interstitial page to ask questions on Server Fault? $\endgroup$
    – Martin Sleziak
    Commented Jun 5, 2019 at 5:51

2 Answers 2

Reset to default
6
$\begingroup$

Here is a CW answer where we could collect various SEDE queries - they could potentially serve as a source of data to get some insights into behavior of the first-time askers before and after introducing this feature. (We need to get some data first, before we can analyse them.)

Some queries related to first questions:

  • Percentage of first questions which get closed and for comparison the same stats for all questions.1,2,3
  • I think that also percentage of first questions which get deleted might give some information about quality of the questions. However, I can't use similar query as above. (The table Posts does not contain deleted posts. The table PostsWithDeleted contains info about deleted posts, but information about the owner is removed.) But such data could possibly be obtained from similar query as above - with ClosedDate replaced by DeletionDate - if it is run on the internal SEDE instance which has data about deleted posts.2
  • Average score of a first question and the same stats for all questions.1,2,3

1The data here is grouped per day - of course, it is possible to use other periods instead (for example by month).
2The query uses Posts table, which does not contain data about the deleted posts. Using the same query on the PostsWithDeleted table would not give stats for all posts, since the information about owner is not shown there. So if we want the data to include the stats for deleted questions, we have to ask a CM who has access to an internal SEDE instance with more data. (Probably this should be done after we gather all queries we are interested in.)
3 A bit of discussion related to these queries can be seen in the Data Explorer chatroom, see this conversation: Statistics about the first-time askers. (I will be grateful for any suggestions how to improve the queries both here in the comments or in the linked chatroom.)

$\endgroup$
4
$\begingroup$

To check this using the premises/query already mentioned in the previous answer, I first checked for the percentage of questions that were closed vs the total of undeleted questions (Intentionally excluding deleted ones to remove the deleted & closed noise). This is what we get when analyzing the first question of each user:

Closed vs Total

The median for the year before the change (Oct '14 - Oct '15) is 9.25%, with an average of 7.63%. The month with the highest percentage is 11.15% in Sep '15 and the lowest is 4.57% in Oct '14.

A year after the change (Nov 15' - Nov '16) we actually have a higher percentage of closed questions. A median of 11.41%, an average of 11.15% (Same as the highest in the year before!). The highest month is Aug '16 with 14.59% and the minimum is 6.57% in Dec '15.

So we see an overall increase in closed questions - a higher minimum and maximum and an increase in average and median.

Here's a zoom in the percentages: Percentage of closed questions

There is a quite different picture for deleted ones, though. This is what we get when we look at deleted questions for the first question of each user:

Deleted vs total

The median for the year before the change (Oct '14 - Oct '15) is 36.2%, with an average of 36.2% too. The month with the highest percentage is Sep '15 with 39.26% and the lowest is Jan '15 with 33.17%.

A year after the change (Nov 15' - Nov '16) there's a clear diminish in the percentage of deleted questions. The median is 32.72%, with an average of 33.43%. Nov '16 has the highest with 37.40% and Jun '16 has the lowest with 29.7%.

Here's a zoom in the percentages: Percentage of deleted

Another thing worth considering is if the tool served as a deterrent for people asking questions, here's the percentage of new users that asked a question on the month they created the account: Percentage overtime

There's a noticeable drop from a median of 60.92% and an average of 61.25% to a median of 55.31% and an average of 55.39% after the tool was implemented.

However, from the line in the graph, there seems to be a general decline over time and no significant spike when the tool was implemented - so that's a dead end.

Conclusions

I do not think the date is enough to say with clear certainty that the tool worked or didn't - there are contradictory indications here. One possibility is that it did help, and thus we saw a decline in the number of deleted questions. Though it didn't help that much, because a lot of the questions that would be deleted are still not good enough and thus get closed. It is hard to say that though without looking at the specific close reasons for each question.

One more point of noise is that it seems that October is particularly a busy month on Math, notice the spike in 2014 (not as significant), 2015, 2016 and 2017. And then a rapid fall-off in the following months. October has spikes

My next suggestion would be looking at Flag Types and see if there's a noticeable change if you folks really want to dig deep into it.

It seems there isn't strong causation of the tool being implemented and an improvement in the quality of questions. There's definitely a correlation that there are less deleted ones after the tool, but causation cannot be proven.

$\endgroup$
1
  • $\begingroup$ I will mention that there is a chatroom related to this question - in case somebody wants to comment a bit on the data, their interpretation, further datapoints to look at, etc. $\endgroup$
    – Martin Sleziak
    Commented Jun 18, 2019 at 7:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .