No. Not unfair.
If you put an ad in the paper, looking for a carpenter, say, to do some work. But then you decide to buckle down and do the thing yourself. Do you ask for a refund on your ad from the paper?
No. You paid for the exposure. What you did or didn't do afterwards is irrelevant.
Bounties are payments for higher foot traffic through a question. Because in the general case, more visibility means more people might see it means more people who could answer might see it and it is an incentive for these people to post an answer that is as good as they can make it. But you paid for the exposure. Not the answer.
If nothing else, it means that your answer is now receiving higher exposure and may garner even more upvotes than it would have otherwise. This is why when you bounty a question that you already answered, the minimum amount is higher than if you just bounty a random question. You're paying, in part, to get your answer exposed, and so you have something to gain.
Not to mention that these are, at the end of the day, made up internet points. But I guess it's easy for me to say that, having so many of them, but then again, that's my point... it's just internet points.