34
$\begingroup$

I was exploring this question. The answer in particular links to a CS Theory question (that I assume was similar or the same) that had been deleted/moved.

I imagine that there A LOT of questions throughout the stackexchange network that reference OTHER network posts.

Would the community be interested in a bot that can crawl user posts and look to detect dead/moved links, and if such are found, notify the poster that their question/answer contains a dead Stack Exchange link.

$\endgroup$
13
  • 14
    $\begingroup$ I would personally appreciate something like this. $\endgroup$
    – Theoretical Economist
    Oct 1, 2019 at 6:34
  • $\begingroup$ about the only hard part about that in google is checking the body of the linked page. $\endgroup$
    – user645636
    Oct 1, 2019 at 11:53
  • 5
    $\begingroup$ It might be worth a trial run to see how many owners of the posts with dead links are still members of Mathematics.SE. Otherwise it would be effort for little gain. Or do you plan to implement across the StackExchange empire? $\endgroup$
    – Peter Phipps
    Oct 1, 2019 at 15:20
  • $\begingroup$ some are comments... $\endgroup$
    – user645636
    Oct 1, 2019 at 15:44
  • 1
    $\begingroup$ This is something the StackExchange software should do. But given that it doesn't, I do think the bot would be useful. $\endgroup$
    – Caleb Stanford
    Oct 3, 2019 at 2:41
  • $\begingroup$ ill probably start with an exploratory analysis, and if i see a big number then writing the bot can commence $\endgroup$
    – Sidharth Ghoshal
    Oct 3, 2019 at 2:59
  • $\begingroup$ google.com/search?q=inurl%3Astackexchange+link%3Astackexchange $\endgroup$
    – user645636
    Oct 3, 2019 at 11:52
  • $\begingroup$ Your choice of example is not great. Note that the Math SE question and the CS Theory SE question have similar URLs. I assume the question was posted on both SE sites, someone commented on the CS Theory SE question, then that question was deleted due to cross-posting. The commenter in question then posted on the Math SE question. $\endgroup$
    – Rodrigo de Azevedo
    Oct 7, 2019 at 10:59
  • $\begingroup$ @RoddyMacPhee, AIUI Google has semi-retired link:, and it can't be relied upon. $\endgroup$
    – Peter Taylor
    Oct 8, 2019 at 8:03
  • $\begingroup$ okay so use related: $\endgroup$
    – user645636
    Oct 9, 2019 at 12:13
  • $\begingroup$ FWIW, I wrote an SEDE query to find links to posts on other SE sites. It's long, ugly and slow, but it seems to work. It should be possible to extend it to only return links to deleted posts on other sites, except the cross-site SEDE queries are currently broken. :( $\endgroup$
    – Ilmari Karonen
    Oct 11, 2019 at 0:41
  • 1
    $\begingroup$ ...and here's the finished query, now that cross-site querying works again. Looks like we currently have 343 broken cross-site links on math.SE (although I already fixed one of them that was an obvious typo). $\endgroup$
    – Ilmari Karonen
    Oct 11, 2019 at 19:11
  • $\begingroup$ ah beat me to it! I was going to start this weekend :) $\endgroup$
    – Sidharth Ghoshal
    Oct 11, 2019 at 23:51

1 Answer 1

Reset to default
7
$\begingroup$

I don't know if a bot is needed, but I did write a query on the SE Data Explorer that finds links to deleted posts on other SE sites.

Screenshot

As the screenshot shows, there were 343 such broken links on math.SE as of last weekend's SEDE update. I've fixed about a dozen of them since then, but they'll still show up in the results until the next weekly update.

The query itself is a huge mess of half-baked T-SQL, and takes several minutes to run on the math.SE database after an update. Fortunately the results are cached, so after the first slow run it becomes really quick.

FWIW, the query works by first compiling a list of all links to posts on other SE sites and then querying each site to filter out any links that point to an actual existing question or answer. The fundamental problem here is that there's no actual table containing a list of such links in the SE database (or, if there is, it's not available via SEDE) so the query has to find them the hard way, by searching through the HTML text of every single post.

This requires a whole bunch of non-trivial text processing and pattern matching, which T-SQL (at least as far as I know it) isn't really particularly good at, while being almost completely unable to make use of the its strengths as a relational database query language. The query thus spends almost all of its time laboriously searching through the full math.SE database for posts that look like they might contain links to another SE site; once it's reduced that list down to about 10,000 or so plausible candidate posts, actually extracting the links from them and checking their validity is pretty quick in comparison.

You can also use the "site switcher" on SEDE (just below the code and above the results) run the same query for other SE sites besides math.SE. For example, here's a link to the same query for MO, which currently returns 261 results. It times out if you try to run it on the Stack Overflow database, though.

$\endgroup$
1
  • $\begingroup$ I suppose running something like this regularly and identifying dead/broken links manually would be of benefit to the SE community, I guess once I get lazy and if SE ever approves a bot could be made to auto-comment when finding dead links $\endgroup$
    – Sidharth Ghoshal
    Oct 11, 2019 at 23:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .