On my setup it appears as though the limited preview rectangle, which shows as a pop-up when the mouse hovers above a question and contains a text excerpt of the question's main text, doesn't support TeX. Is it supposed to work like that or is this a bug?
Yes, it's supposed to work like that. The hover effect is achieved by giving the
<a> (link) tag a
title property containing the excerpt, and then the tooltip is produced by the browser. There's no way (on SE's end anyway, the browser could theoretically be changed to do it) for this tooltip to support MathJax: the browser just reads the
title attribute as plain text.