I recently found out about the stackoverflow network data dump (now hosted by the internet archive). Incidentally, I have also been looking for a playground to test some knowledge representation ideas.
So far, I have explored the Posts and Tags datasets for mathoverflow and math.stackexchange, for which I set out to find the hidden hierarchical structure in the tags. The idea is really simple: to say whether a certain tag $X$ subsumes a tag $Y$, a count the co-occurrences of $X$ and $Y$ and the occurrences of $Y$ without $X$. If the second number is 'much' smaller than the first then I say that $Y \subset X$.
Just using this one piece of evidence in a single iteration, I think the accuracy of the result is already good. I am sure there are other pieces of evidence one could use.
I have created a directed graph based on this relation (nodes are tags and edges are inclusions). Below are some connected parts of the graphs (which is bigger - there are about 1500 tags currently).
Now, time to explain why this post is here:
- Stackexchange seems a community that would appreciate such use of their data but I can't find a suitable place to start the discussion
- Giving a graph structure to tags can improve the tag engine. If we 'train' the system to know that finite-groups is in fact a part of group theory, then asking users don't need to use both tags (and can instead focus on "horizontal" tagging) or we can have the system remove the including (highest) tag. Similarly, responding users do not need to set both tags as favourites and can just use the lowest tag, or interpret the presence of both in novel ways.
If this is not the appropriate place for this discussion, could you point me at a better site?
Anyone has any other ideas on what kind of information could be dug up from the system (particularly with knowledge representation in mind)? I am sure StackOverflow must be working on this, perhaps it's already implemented at some level.