19
$\begingroup$

When accessing the main site, I see a banner with the text "This site is currently in read-only mode. We’ll return with full functionality soon. Follow @StackStatus, visit our status blog or search our recent meta posts on the topic for more info." But the Twitter account says the maintenance was done over 8 hours ago, and the other SE sites seem to be up and running, including MSE meta! So why is MSE still down --- or is this somehow an issue just for me?

$\endgroup$
6

1 Answer 1

Reset to default
20
$\begingroup$

What happened?

During our planned maintenance window on March 17th/18th, a sequence of events in our Colorado data center left Mathematics Stack Exchange in read-only mode for several hours after the maintenance was finished.

TL;DR

The site is back up. The maintenance window was officially closed on March 18th at 01:28 UTC, at which point all other sites were fully operational, but Mathematics remained in read-only mode until it was manually corrected at roughly 11:46 UTC.

What caused the issue?

A number of factors contributed to the issue itself and our failure to recognize it. The long version:

  • As part of our manual failover process, we move all sites to read-only mode due to a performance issue we’ve been experiencing for months but have yet to solve.
  • This involves shifting site traffic to our web servers in Colorado, where they can read data from the secondaries there without any impact from operations happening in New York.
  • Fundamentally, the application should not try to write to a read-only secondary; if successive attempts fail, the source site is automatically switched to read-only mode for safety, because it’s a clear sign that something is awry.
  • In this case, the Mathematics site did try to write to a read-only secondary following a very specific (non-nefarious) sequence of events from an end user.
  • Because this happened while all of the other sites were in read-only mode, any notifications about this site specifically were silenced for maintenance, as we were intentionally setting the entire network read-only and also doing maintenance that would cause significant alert noise anyway.

What contributed to the duration?

At the end of our maintenance window, we spot check a handful of things, including a few specific web sites (but certainly not all of them). We don’t manually open every single web site in a browser to ensure the banner is gone, or test that we can successfully post a comment or vote on a post.

In this case, all of the places we checked at 01:30 UTC looked good, including our open-source monitoring tool, OpServer - which provides a fantastic pulse on the state of our databases. If a specific site’s primary database couldn’t be written to, we certainly would have seen it there. But the site itself does not surface its status there - that is shown elsewhere, on an internal dashboard. So we went to bed.

How can we improve?

We can improve internal notifications and time to resolution for this scenario:

  • We can manually detect when a site is automatically shifted to read-only in response to a scenario like the above {done}
  • We can automate and alert on that monitor
  • We can further improve the way the application attempts to write to the database during maintenance
  • We can add to our maintenance window runbooks:
    • Checking the internal dashboard in addition to OpServer and other checks {done}
    • Increasing the number of sites we spot check
    • Waiting an hour and checking meta (though there are a lot of individual meta sites to check, too)
$\endgroup$
7
  • 2
    $\begingroup$ Wouldn't this be very amenable to automation to check that all sites and their metas are out of read only? $\endgroup$
    – Asaf Karagila Mod
    Mar 19, 2022 at 9:32
  • 5
    $\begingroup$ @Asaf I was being a little facetious about manually opening browser windows for every site. We open a couple as a sanity check but we don’t need automation here - we already have an internal dashboard that shows the read only status of all sites and pushes those in read only to the top. We just need to remember to check it, which we didn’t do the other night simply because there are two different settings that control read only - globally and per site. The miss was that we didn’t predict an individual site would trip its setting while the global one was in effect. $\endgroup$
    – Aaron Bertrand Staff
    Mar 19, 2022 at 11:07
  • $\begingroup$ I see. Okay, that makes more sense. Still, there should be a big red flashing thing if it detects any one site is in read-only mode. $\endgroup$
    – Asaf Karagila Mod
    Mar 19, 2022 at 11:15
  • 2
    $\begingroup$ @Asaf It already is a red flashing light but we have to be looking in its direction. We’re human, and we didn’t. But we sure will next time. $\endgroup$
    – Aaron Bertrand Staff
    Mar 19, 2022 at 11:21
  • 1
    $\begingroup$ Okay, it just seemed like there's a lack of automation in the process that relied a lot more on human interaction. I was a bit surprised, but I am glad to hear that this is just a lack of explanation in the answer. :-) $\endgroup$
    – Asaf Karagila Mod
    Mar 19, 2022 at 11:25
  • 1
    $\begingroup$ @Asaf And here I thought we were being pretty transparent about what happened and how we’ve learned from it. $\endgroup$
    – Aaron Bertrand Staff
    Mar 19, 2022 at 11:37
  • 1
    $\begingroup$ Well, that you were. :-) $\endgroup$
    – Asaf Karagila Mod
    Mar 19, 2022 at 11:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .