This is answered in the Help Center article titled Who are the site moderators, and what is their role here?.
At Stack Exchange, we believe moderation starts with the community itself, so in addition to all users gaining privileges through reputation earned, each site has moderators elected through popular vote. We hold regular elections to determine who these community moderators will be …. Moderators are elected for life, though they may resign (or, in very rare cases, be removed).
Details about the kinds of scenarios under which a moderator's diamond can be removed can be found in the FAQ on Meta Stack Exchange titled Who are the diamond moderators, and what is their role?. From the section "How long does a moderator's term last?":
[T]here are a few cases where a user's moderator diamond may be removed:
First, moderators may resign from their post at any time.
Second, inactive moderators who haven't performed a single moderator action for the past six months will be contacted and asked if they wish to continue being a moderator. If there is no response or if they respond "no", their privileges will be removed.
Third, there are procedures for removing moderators in case the moderator's actions are causing issues with the community or among the moderator team, if the moderator violates the moderator agreement, or if rights need to be immediately revoked in an emergency.
You can also find some discussion about why the moderation system is set up like this in the post Should Community Moderators be "elected for life", or have terms? over on Meta Stack Exchange, as well as its linked questions.