Yes, if one suspects that a particular user has downvoted one's answer, then tracking the reputation of the suspect can confirm the suspicion. This has been described on meta already:
Open the page of the user X in question and record his reputation. Delete your downvoted answer. Wait a few minutes so that the system implements the reputation changes (in particular, your own reputation should decrease by appropriate amount) and then refresh the page of X. If he was at the origin of the downvote, his reputation will be increased by 1. Then undelete your answer, wait again and refresh the page of X. His reputation will decrease back by 1.
The system already delays the reputation recalc somewhat. Setting the deduction to $-2$ or another number would not change much: users can still see the expected changes of reputation upon deleting/undeleting the answer (and they can do several rounds of delete/undelete, choosing time when the site is not very active and the suspected user is not online).
So: yes, downvotes on answers are not $100\%$ anonymous. This is widely known: see Anonymity of downvotes has a loophole where a Stack Exchange employee commented:
Yes, it is a loophole of sorts [...] this does bother me a bit. A dedicated user with a vendetta could confirm his target much more readily now than in the past. We'll want to keep an eye on it.
That said, there is an easy solution to the problem of revenge downvotes, serial downvotes, etc. It's something I call "not giving a damn".