I understand that the consensus of the community about downvoting is that each one has its own personal reasons to downvote, and should not have to justify that.
Yes, but there is one major exception, which is that one must not vote based on the identity of the post-owner.
Shortly after I explained the downvote, my answer was downvoted with no reasoning. The question had approximately $ 14$ views when that happened, and it was really in a short time span after my explanation. I can't help but feel that this was retaliation.
This seems plausible, but it is not certain.
Is this under "fair use" of downvoting?
If the vote was actually cast in reaction to your vote, it is not admissible. This use of votes can be, and has been, penalized in the past.
Do I have to deal with this, and move on?
In case it stays an isolated instance, there is nothing much to be done locally.
If not, what can I do? Is flagging appropriate?
You can still flag and explain the problem. It might be that for you locally it is isolated but the user in question exhibited a pattern of behavior of this form already. Yet be prepared that we cannot do anything. (It also makes sense to phrase your flag in this spirit, as information, as opposed to a call for action.)
My first thought-reaction was to flag, since I believe moderators can see downvoters and would be able to solve this without exposing anyone etc. Is my reasoning innapropriate?
As said, feel free to flag to let us now. But actually we elected moderators can neither see the origins of individual votes nor can be undo them. Only if there is an accumulation of votes, then we can do something.
Technically, SE staff could do that or give us the means. But it is not really feasible for them to track individual votes. Moreover, there is also the issue that even if it should be the case that the origin of the vote is what you suspect it is a bit tricky to ascertain the motivation based on a single vote. Surely, the voter could come up with some excuse like: "Of course it was not a retaliation, I would never do that. It is true my attention was drawn back to that thread because of the comment, thus the temporal coincidence. But my vote is perfectly justified, because {some argument why they downvote}."
What do you do then? Say the reasoning is not credible enough? But maybe it is not completely implausible in a given case, rare is the answer that is so perfect that one could not cook up a reason to downvote. Anyway, it would just lead to all kinds of discussion and make the problem grow and grow.
Thus, to recap, if it stays at one downvote, just move on. It's not nice, but it is not that big a deal either. But if it becomes a persistent or bigger problem, do inform us, and we have means to deal with it.
Generally, isolated and somewhat minor conflicts are often best handled by just disengaging and letting the thing slide away.
