27
$\begingroup$

Since the new site theme has been introduced, I find that I hardly read math.SE anymore. In essence, I find it illegible ... including that the topics that interest me hardly 'show up' anymore. The site used to be a paragon of design excellence ... now I avoid it, but I still enjoy the statistics (stats.SE) and Mathematica.SE sites whose user interface has not changed.

It seems, based on links such as:

Mathematics new site theme is live

... that others share the same view.

My question is:

  • Are there are any statistics on how site usage has changed before and after the change? e.g. number of visitors per day, number of questions posted per day, number of answers posted per day ... or similar?
$\endgroup$
7
  • $\begingroup$ It's now going to be difficult to study now, as the site's theme changed, and the introduction of the banner "New contributor" have taken place. So since yesterday, any stats might be confounded. But it would be interesting to know, until the day before yesterday, how usership had changed. $\endgroup$
    – amWhy
    Commented Aug 22, 2018 at 16:05
  • 1
    $\begingroup$ I will try to pull some stats on questions asked and page views, which are probably two of the most interesting and critical top level stats in regards to site health. $\endgroup$
    – Joe Friend
    Commented Aug 23, 2018 at 21:18
  • 4
    $\begingroup$ Oh, and the short answer is that there doesn't seem to be any significant impact. Or, it is small enough that it is hidden by the normal annual summer slump. $\endgroup$
    – Joe Friend
    Commented Aug 23, 2018 at 21:47
  • 1
    $\begingroup$ It's all dropped. The site is dying now like never before. (this is the answers most people want to hear anyway)...Anyhow, act like that is the case and start answering questions like fire from hell, ok. $\endgroup$
    – Nick
    Commented Aug 24, 2018 at 21:26
  • $\begingroup$ I don't think the 'demand for questions' is the appropriate test ... if people need help, they need help. The statistic I would look at is the number of page reads of the site, and the number of answers per day (before and now). $\endgroup$
    – wolfies
    Commented Aug 28, 2018 at 17:21
  • 2
    $\begingroup$ It's worth noting that there are probably a large bulk of users who are not regular users, and may have zero investment in the old theme. I'd be more interested in seeing how members with, say, over 1000 reputation and 1 year of experience on the site are affected. $\endgroup$
    – Theo Bendit
    Commented Aug 30, 2018 at 8:22
  • 2
    $\begingroup$ Is there any good reason why we can't get the old site theme back? $\endgroup$
    – SAINT
    Commented Aug 30, 2018 at 15:59

1 Answer 1

Reset to default
9
$\begingroup$

This data has been a bit longer time coming than first expected... but the wait is over and I have some interesting results for you... but not a lot of answers.

Based on the question and comments here I wanted to be careful about what information I looked at for a few reasons...

  1. The summer already has lower visits and there's an increase in September, so looking at this year alone might make it look like there was more activity after the introduction of the theme due to school starting in late August/September.
  2. The number of questions/day may not necessarily be a good indicator.

To address the first point, we'll compare the most recent year to the prior one for the same period and the second by looking at additional stats, like the number of visits from users >1k rep.

Through querying the internal SEDE, we see the following info:

For the period from 2017-08-10 to 2018-04-06

  • 4,502 questions / week
  • 5,336 answers / week
  • 4,281 visits from users with at least 1,000 rep (as of 2018-04-06)

After the change we saw

For the period from 2018-08-10 to 2019-04-06

  • 4,142 questions / week
  • 4,607 answers / week
  • 4,670 visits from users with at least 1,000 rep (as of 2019-04-06)

So, there is a decrease in the number of questions and answers dropped in a proportional number - the answers to question ratio before was 1.19 and after the change it was 1.11 but the raw numbers there are a loss of 360 questions and 729 answers... which is about double (more on this after the break).

Despite the slight slump in answers - which could be due to other factors like if more questions are being closed/deleted before they're answered - the number of visits from higher rep users increased, which is great.

And, if you like something a bit more visual, here's a chart of the posts created for the weeks before and after the change. You can see that it's on the expected start-of-school year incline but there's no obvious point where the post volume changed dramatically.

Posts per day by type between August 1 and October 1 2018

So, all-in-all, my first instinct is that the change of layout has had little to no impact on your site health.

I wanted to go looking for an explanation for the difference in the questions vs the answers so I decided to increase the spread of my search a bit... and I started digging and saw that there is actually a big drop in questions asked on a monthly basis and it looks like it starts in August.

Graph of posts, questions, answers per month from Feb 2016 - March 2019 with an arrow indicating the time since August 2018

Those numbers in the last hump look remarkably lower than the prior one and that startled me. I wanted to get a comparison, so I pulled the numbers into a spreadsheet and saw that there were between 1800 and 5800 fewer posts over the course of the months. This looks horrible!

I went searching for an explanation there were a lot of little things that changed on the network around the same time - the ability of low-rep users to flag comments on their posts (August 10th), the new contributor icon (August 17th), Google changed their algorithm a bit but when I looked around the network at our other big sites... there wasn't any similar drop off in participation and yet, all of those sites had also had the same changes.

So, I went back to the numbers... and I realized that I'd missed the difference between July 2018 and 2017 - it's pretty big at around 3400 posts short... and that extends all the way back to about January 2018, long before any of the changes we made. Here's a year of numbers for your reference:

          2017   2018    Diff      %
Apr      46255  44226   -2029    -4.39%
May      45244  42959   -2285    -5.05%
Jun      37462  33393   -4069   -10.86%
Jul      33824  30348   -3476   -10.28%
Aug      34413  32583   -1830    -5.32%
Sep      39430  36112   -3318    -8.41%
Oct      46771  42422   -4349    -9.30%
Nov      46767  41013   -5754   -12.30%
Dec      41827  36284   -5543   -13.25%
Jan      43364  37499   -5865   -13.53%
Feb      40867  35162   -5705   -13.96%
Mar      44432  39164   -5268   -11.86%

The numbers are certainly more consistently down than they were before August but the June and July numbers indicate that there was already something causing a downward trend in posts here on Math that doesn't seem to be reflected to this degree on the other sites I looked at (Super User, Ask Ubuntu, Server Fault, Unix & Linux, Cross Validated, and Physics).

Wrap up / TL;DR:

So, I think it's safe to say that the design didn't cause the change but there's definitely fewer posts being created - fewer people both asking questions and answering them over the last year but the returning users over 1000 rep are more likely to be around.

$\endgroup$
9
  • 2
    $\begingroup$ Thanks for compiling this. Easy, off-the-cuff explanations might be that we are, indeed, now better ogranized to catch duplicates and poorly researched questions. Whether we currently are too good or not yet good enough at that is subject to a heavy debate (somebody will use your data as a hammer to press home their related agenda point here in ...10...9...). The number of returning 1k+ users is, of course, great. I don't think it needs any explanation other than that the total number of such users is steadily increasing. $\endgroup$
    – Jyrki Lahtonen
    Commented Apr 9, 2019 at 7:46
  • $\begingroup$ Thanks for the interesting reply, though it seems you are bending backwards to avoid the obvious: that site usage is down since the changes, and that the same has not happened on your other sites. The obvious counter-experiment is to undo the changes, put it back the way it was, and see if it slows down or indeed reverses the loss in numbers. $\endgroup$
    – wolfies
    Commented Apr 9, 2019 at 13:31
  • $\begingroup$ @wolfies I'm not sure how that makes sense as the conclusion to be drawn here. The site usage has been down for a while. Every site on the network had the same exact changes made - some more drastic than Math - and they're not experiencing this. Also, with the higher-rep users having more visits, that means it's the askers being impacted, not the answerers... and that's not really likely to be impacted by the site design... and, finally, going back is literally impossible because the code for the old design has been stripped out entirely. $\endgroup$
    – Catija
    Commented Apr 9, 2019 at 13:48
  • $\begingroup$ Hi again. I think you must be talking about different user-interface changes than I am, since I have only noticed the unhelpful user-interface/design changes on the math.se site, and not on the other se sites that I use (including stats.se and mathematica.se). $\endgroup$
    – wolfies
    Commented Apr 9, 2019 at 15:03
  • $\begingroup$ And if your code changes are causing your user base to fall, I think the correct solution is not to bury one's head in the sand, but to re-examine the decisions that have been made in the past, and place user experience and growth as the optimising variable ... with design subservient to that goal (as Google does) and reign in mistakes of the past before the negative growth problems becomes irreversible. $\endgroup$
    – wolfies
    Commented Apr 9, 2019 at 15:03
  • $\begingroup$ @wolfies I'm really not sure then what you're talking about. The changes in that post you linked are everywhere on the network. Could you be more specific? $\endgroup$
    – Catija
    Commented Apr 9, 2019 at 16:16
  • $\begingroup$ See: math.meta.stackexchange.com/questions/28842/… where some 136 people have downvoted the changes. The issue are change of font, layout, lack of contrast, inability to see faved tags, etc: these problems of contrast and legibility did not occur on other SE sites. $\endgroup$
    – wolfies
    Commented Apr 9, 2019 at 16:41
  • $\begingroup$ The contrast issues are all marked as being fixed, @wolfies ... and looking around, they seem to be to me. If there are still issues with contrast and legibility, please, open a new question about it with the design tag and I will see it and get the design team to look at it. There were some initial issues with link colors being too similar to text colors but that seems to have changed... and I'm not sure what legibility issues you mean. The word doesn't appear on that page at all. We can absolutely fix these things but we are not going to roll the entire layout back to what it was. $\endgroup$
    – Catija
    Commented Apr 9, 2019 at 16:45
  • $\begingroup$ @wolfies : "The obvious counter-experiment is to undo the changes, put it back the way it was, and see if it slows down or indeed reverses the loss in numbers" - for me this would not happen this way. If they would even go and play around more with that interface my interest in the site would decrease even more - assuming even less sincerity of the company in the providing of a consistent GUI and in any non-destruction of user-automatisms, which have been learnt in long (and hard) self-training. $\endgroup$
    – Gottfried Helms
    Commented Apr 11, 2019 at 14:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .