41
$\begingroup$

Since October 2015 a "How to ask?" page is shown before a user asks their first question on this site, and they need to confirm that they read it; see here: Show “how to ask” advice before a new user asks a question. (The post announcing that this was implemented is from October 15.)

Do we have some statistics which can help to evaluate whether this change has made some impact? (This question is somewhat similar to: Registration to ask a question – has it made a difference?)

Basically, I am asking two questions in one:

  • Which statistics should we look at?
  • Where can we get those statistics?

Which statistics should we look at? Which numbers would describe changes in positive direction? Should we look at percentage of questions that were deleted? Percentage of question (or first questions) that were closed?

Where can we get those statistics? There is a page with site analytics, however, I am not sure whether this is of much help here. (I do not see anything about closures and deletion there. It is possible that moderators have access to more detailed stats in the site analytics.) We could also use some data from Data Explorer (SEDE). (Some data will be necessarily skewed, since SEDE contain only limited information about deleted posts in the PostsWithDeleted table. However, I have been told by more experienced users that CMs have the option to run SEDE queries also on the data-set, including the deleted posts - so if we are able to come up with something useful, we might ask them for help.)

$\endgroup$
  • 12
    $\begingroup$ I should mention that my motivation to ask this is partly because of discussion on Meta MathOverflow: Should users be shown some basic information before posting the first question? Knowing to which extent this is actually efficient might be helpful when discussing whether such change should be done on other SE sites. $\endgroup$ – Martin Sleziak Jan 14 at 5:57
  • 9
    $\begingroup$ I suppose the proper way to do it would be to randomly show the page to some new posters but not others, then compare suitable statistics for the two groups—assuming (i) that was practical and (ii) people would tolerate getting lots of questions from people who hadn't been shown the page. That way you'd avoid inadvertently measuring things like changed toleration levels towards poor quality posts. $\endgroup$ – timtfj Jan 14 at 15:34
  • 4
    $\begingroup$ You'd also be measuring what happens now (with more people posting from smartphones, for example) rather than what happened a few years ago under possibly-different conditions. $\endgroup$ – timtfj Jan 14 at 15:43
  • 5
    $\begingroup$ You could also look at votes and edits on first posts before Oct 2015 and after Oct 2015. $\endgroup$ – rene Jan 14 at 20:33
  • 2
    $\begingroup$ I don't know the answer to this question, but I should mention we've been thinking about ammending that page lately. Suggestions about what to change or add would be helpful. Here's a chat so the comments don't get cluttered up. $\endgroup$ – Alexander Gruber Jan 15 at 8:20
  • 1
    $\begingroup$ Just going to point out that from a certain standpoint one could argue that the requirement to scroll through some page isn't going to make much difference in the quality of what someone asks if they never cared about quality in the first place. But that's a cynical view. I am intrigued on what the results will be from the data. $\endgroup$ – The Great Duck Jan 19 at 20:12
6
$\begingroup$

Here is a CW answer where we could collect various SEDE queries - they could potentially serve as a source of data to get some insights into behavior of the first-time askers before and after introducing this feature. (We need to get some data first, before we can analyse them.)

Some queries related to first questions:

  • Percentage of first questions which get closed and for comparison the same stats for all questions.1,2,3
  • I think that also percentage of first questions which get deleted might give some information about quality of the questions. However, I can't use similar query as above. (The table Posts does not contain deleted posts. The table PostsWithDeleted contains info about deleted posts, but information about the owner is removed.) But such data could possibly be obtained from similar query as above - with ClosedDate replaced by DeletionDate - if it is run on the internal SEDE instance which has data about deleted posts.2
  • Average score of a first question and the same stats for all questions.1,2,3

1The data here is grouped per day - of course, it is possible to use other periods instead (for example by month).
2The query uses Posts table, which does not contain data about the deleted posts. Using the same query on the PostsWithDeleted table would not give stats for all posts, since the information about owner is not shown there. So if we want the data to include the stats for deleted questions, we have to ask a CM who has access to an internal SEDE instance with more data. (Probably this should be done after we gather all queries we are interested in.)
3 A bit of discussion related to these queries can be seen in the Data Explorer chatroom, see this conversation: Statistics about the first-time askers. (I will be grateful for any suggestions how to improve the queries both here in the comments or in the linked chatroom.)

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .