My understanding is that you lose $2$ reputation when you do so. I would like my reputation to be as high as possible, so I don't want to voluntarily give up points. I do my best to assist in low-effort posts by inlining images, asking the OP what their thoughts are on the problem, and voting to close for lack of context when it is clear that it is a PSQ. However, am I expected to downvote a question or answer that is "bad," for lack of a better word? Or is this up to my discretion?
It's your time that you're putting into keeping MSE in line, to say nothing of your reputation. Spend those resources as you wish. Everything I've read from SE has lead me to believe that they appreciate our moderation efforts and our reputation is the basis for the trust they have that our judgment represents the will of the community.
Speaking for myself, I have cast 1357 votes, and only six were downvotes. I think they may have all been in my first month. (It's hard to tell, since your profile page only shows you the undeleted posts you have downvoted.) If I were to cast a downvote because a question or answer were bad, I would feel obligated to go back and undownvote it if the question had been improved. So the only time I seem to do it is when a question is bad and the poster has doubled-down on the badness in comments.
If someone posts an answer that is wrong, I'll give them a comment (which sends a more precise message to the questioner and the community). If it's trivial or repeating a previous answer, I just move on with my day. And if it's not mathematical, I flag it. Even if my rep wasn't penalized for downvoting an answer, I wouldn't feel a need to have that arrow in my personal quiver. But we all have our own strategies, so blessings to the people who do use it!