34
$\begingroup$

I was exploring this question. The answer in particular links to a CS Theory question (that I assume was similar or the same) that had been deleted/moved.

I imagine that there A LOT of questions throughout the stackexchange network that reference OTHER network posts.

Would the community be interested in a bot that can crawl user posts and look to detect dead/moved links, and if such are found, notify the poster that their question/answer contains a dead Stack Exchange link.

$\endgroup$
  • 14
    $\begingroup$ I would personally appreciate something like this. $\endgroup$ – Theoretical Economist Oct 1 at 6:34
  • $\begingroup$ about the only hard part about that in google is checking the body of the linked page. $\endgroup$ – Roddy MacPhee Oct 1 at 11:53
  • 5
    $\begingroup$ It might be worth a trial run to see how many owners of the posts with dead links are still members of Mathematics.SE. Otherwise it would be effort for little gain. Or do you plan to implement across the StackExchange empire? $\endgroup$ – Peter Phipps Oct 1 at 15:20
  • $\begingroup$ some are comments... $\endgroup$ – Roddy MacPhee Oct 1 at 15:44
  • 1
    $\begingroup$ This is something the StackExchange software should do. But given that it doesn't, I do think the bot would be useful. $\endgroup$ – 6005 Oct 3 at 2:41
  • $\begingroup$ ill probably start with an exploratory analysis, and if i see a big number then writing the bot can commence $\endgroup$ – frogeyedpeas Oct 3 at 2:59
  • $\begingroup$ Your choice of example is not great. Note that the Math SE question and the CS Theory SE question have similar URLs. I assume the question was posted on both SE sites, someone commented on the CS Theory SE question, then that question was deleted due to cross-posting. The commenter in question then posted on the Math SE question. $\endgroup$ – Rodrigo de Azevedo Oct 7 at 10:59
  • $\begingroup$ @RoddyMacPhee, AIUI Google has semi-retired link:, and it can't be relied upon. $\endgroup$ – Peter Taylor Oct 8 at 8:03
  • $\begingroup$ okay so use related: $\endgroup$ – Roddy MacPhee Oct 9 at 12:13
  • $\begingroup$ FWIW, I wrote an SEDE query to find links to posts on other SE sites. It's long, ugly and slow, but it seems to work. It should be possible to extend it to only return links to deleted posts on other sites, except the cross-site SEDE queries are currently broken. :( $\endgroup$ – Ilmari Karonen Oct 11 at 0:41
  • 1
    $\begingroup$ ...and here's the finished query, now that cross-site querying works again. Looks like we currently have 343 broken cross-site links on math.SE (although I already fixed one of them that was an obvious typo). $\endgroup$ – Ilmari Karonen Oct 11 at 19:11
  • $\begingroup$ ah beat me to it! I was going to start this weekend :) $\endgroup$ – frogeyedpeas Oct 11 at 23:51
6
$\begingroup$

I don't know if a bot is needed, but I did write a query on the SE Data Explorer that finds links to deleted posts on other SE sites.

Screenshot

As the screenshot shows, there were 343 such broken links on math.SE as of last weekend's SEDE update. I've fixed about a dozen of them since then, but they'll still show up in the results until the next weekly update.

The query itself is a huge mess of half-baked T-SQL, and takes several minutes to run on the math.SE database after an update. Fortunately the results are cached, so after the first slow run it becomes really quick.

FWIW, the query works by first compiling a list of all links to posts on other SE sites and then querying each site to filter out any links that point to an actual existing question or answer. The fundamental problem here is that there's no actual table containing a list of such links in the SE database (or, if there is, it's not available via SEDE) so the query has to find them the hard way, by searching through the HTML text of every single post.

This requires a whole bunch of non-trivial text processing and pattern matching, which T-SQL (at least as far as I know it) isn't really particularly good at, while being almost completely unable to make use of the its strengths as a relational database query language. The query thus spends almost all of its time laboriously searching through the full math.SE database for posts that look like they might contain links to another SE site; once it's reduced that list down to about 10,000 or so plausible candidate posts, actually extracting the links from them and checking their validity is pretty quick in comparison.

You can also use the "site switcher" on SEDE (just below the code and above the results) run the same query for other SE sites besides math.SE. For example, here's a link to the same query for MO, which currently returns 261 results. It times out if you try to run it on the Stack Overflow database, though.

$\endgroup$
  • $\begingroup$ I suppose running something like this regularly and identifying dead/broken links manually would be of benefit to the SE community, I guess once I get lazy and if SE ever approves a bot could be made to auto-comment when finding dead links $\endgroup$ – frogeyedpeas Oct 11 at 23:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .