For instance, recently I have earned two gold badges: commutative-algebra and abstract-algebra tags badges. I wonder if such a badge (or any other) can be withdrawn if, for instance, people downvote many of my answers.
Tag badges are withdrawn when one no longer meets the criteria (general badges are basically never withdrawn, that is they are certainly not automatically taken on failing the criteria later, yet only possibly in cases of cheating and related).
See "What can cause a badge to be lost/revoked/taken away after it is awarded?" in the relevant FAQ on the general meta: How do "badges" work?