This is just my personal opinion on the three points in particular. (Yeah, I know the other answer is already good and already accepted.)
- Reuse a formula from a math book? (probably not, but why?)
I would say this never constitutes a breach of copyright. But always, always cite where you found the formula.
- I give a proof from a math book, in my own words, but clearly heavily inspired?
Again, I would say this never, ever consitutes a breach of copyright, but always quote the math book or any other source of heavy inspiration for a proof. In a paper this means discussions with colleagues (who may be anonymous), personal communication, other papers, books, conference proceedings, and so on.
- I quote from a math book?
As in a large block of text taken directly from the book? Opposite to the last two points I would say never do this. I can not think of any time it would be a good idea. Also, apart from this, it may also constitute a breach of copyright. I seem to recall photocopying large portions of any book is a breach of copyright (university library mass e-mails, who reads them?) and can not really see that posting said slabs to the internet is any less 'damaging' than photocopying.
Hope that helps.