My reputation was 1172 this morning. I added my Yahoo account open ID to my profile, and now my reputation is only 1132, without receiving any downvotes or something which could affect negatively my reputation. What is the problem?
As others have pointed out, users can undo accepted answers at any time, or upvotes within a 5 minute vote grace period. Also, if the post has been edited since the last time they voted, they can also undo the vote at will.
Check the reputation tab on your user page, and https://math.stackexchange.com/reputation for details, but remember that reverted votes will not be shown anywhere.