I have seen that when I have down voted some answer that is just posted and has no votes. My reputation gets decreased by 1. Also it does not happen every time it, it only happens when the person who has answered has a high reputation ? Why is it so or is there some other rule I don't know of ?
Downvoting answers - except community wiki answers - costs one reputation point, downvoting questions is free. It is independent from the reputation of the poster (or yours). See the help centre.