Let me add a second point which Mariano does not address in his question, that there are "not an answer"s and there are "not an answer"s, and please use the not an answer flag only to indicate that the post should in fact be a comment.
- Some answers are not, by virtue of them being actually comments, but posted by users without the requisite 50 rep who thus abuse the system slightly. Those the moderators can easily convert to a comment. (In the case that the comment really belongs to one particular answer out of many [as opposed to it being a comment on the question, or it being a comment on the only non-comment-in-disguise answer to the post], it would help enormously if the flagger gives an indication where the comment should be transferred to.)
- Some answers are not, by virtue of them being SPAM. (SPAM bots now sometimes post quasi-comprehensible posts, and rely on a link to the link-farm or SPAM advertisement in their "user" profiles. If you see a very strange post that doesn't really make sense from a 1 rep user with nondescript name, try checking the user profile.) Those posts should be flagged as SPAM, and not as "not an answer".
- Some answer are not, by virtue of them, well, not being an answer to the question, either by answering the wrong question or answering the question wrong. Please don't flag those. You should instead down-vote that answer and/or post a comment indicating the problem.
In general I am in favour of efforts to help clean-up the community. In cases where it may not be clear what you, as a user, is expecting the moderator to do (for example, a long, slightly off-topic answer that is not offensive nor spam, but just off-the-mark; we can't convert that to a comment [it is too long, and will still miss the point of the question], nor should we bin it), it will help a lot if you use the free-form "needs moderator attention" option and describe what you see wrong and what you want us to do, instead of one of the pre-defined flags which usually carry more specific meanings than the dictionary reading.