Swag is pretty nice, I think that useful swag is better.
For the mathematician case, I can suggest the coffee mug (as Erdos' said "a mathematician is a machine turning coffee into theorems") and even better a writing pad... I could sure use a new one, and as Isaac suggests in the comments pens and such. Preferably black, but blue works out just as well.
As for a shirt, I am not sure if I would wear one.
Edit: Since Willie is surprised I suggested some sort of a tool for drugs (well, caffeine... but still a psychoactive drug!), I will suggest things that will be useful to me in the same spirit: low-ball glasses and coasters.
On the same spirit, but not as useful to me (as I am not smoking), ashtrays and cigarette boxes, as well lighters and matches.
(All the addition is meant to be tongue in cheek, but I will gladly sit and drink scotch from a math.SE low ball, on a math.SE coaster!)
Edit II: Reviewing the current swag, I think that it is more than just what sort of swag, but also what should be printed on it.
I would enjoy a coffee mug with the quote by Erdos, or some theorem about the consistency of happiness and the contents of the mug (e.g. if it has hard liquor)
For a lowball and other tumblers, this should be more sophisticated.
As per shirts, writing pads and cardboard blocks to write on, there we can put proofs, cheatsheets for math and so on.