This question already has an answer here:
So this just came into my rep notifications. What is it about?
I am guessing that "User was removed" means that someone had once upvoted a question (because of the 5) of mine and now that $+5$ rep is being undone because his account is no more (for whatever reason).
If that is the case, then I guess it makes sense and it is completely fine, but I wanted to make sure just in case it had to do with some (wrong) action I took. I am guessing the older users here know what this is, so is it my initial guess? Or is it something else?