I'm sure there's a lot of truth in the Answers posted above, but the elephant in the room is: trolls.
I had a recent encounter where a user became combative when I pointed out errors in his Answer, and within the hour I had several down-votes on Q's and A's I had posted months before.
I also once got several down-votes on an Answer where I made a distinction between "divergent" and "non-convergent" that some people apparently hadn't been aware of. Rather than lose the Rep points, I deleted a useful Answer.
Revenge down-votes would be a fairly easy pattern for the site to detect, warn abusers about, and flag to Moderators. This feature should be implemented, IMHO.