As quid already said in their answer, it is perfectly fine to discuss in comments things which are not related to the question's content, but more to the issues of formatting, editing, tagging, closing, reopening (of the particular question), etc.
I will also add that the other user wanted to get your attention to some issue. They did not have many other options to do that rather than make a comment on some post where you would notice it (ideally accompanied by a ping). And if the issue was related to that particular question, it was the most natural place to leave the comment.
Personally I think that it is not always needed to delete all comments related to such issues, since they might be useful for other users, too. (For example, let us say there is a comment explaining that some tag was used incorrectly. This comment is useful for other users, which can learn from the comment about correct usage of the particular tag.) But in your question you are asking probably about longer exchanges which will look as noise and do not add to much information for other users.
If the comment exchange grows long, you will be offered by the software the possibility to move comments to chat. See here for some more details: What happens when the "move comments discussion to chat" link is used?
Or you don't have to wait, you can create a chatroom yourself. If it is useful (for archiving the comments or for giving some context to the conversation in chat), you can copy the comments from the site one by one.
If you continue in the discussion with the other users in chat, you can than agree there whether the comments on the main site are no longer needed and you can each delete your comments. (You can find here and here example of me doing something like that.)
The advantage of moving to chat is also that the comments are archived at least in chat after they are deleted on main site. (So it is useful especially in case where the comments contain some information which might be useful for you, some useful links, etc.)
Other option is removing your comments and flagging the other user's comments as obsolete. (Moderator will then handle the flag and delete the comments, if they agree that it is a correct thing to do.) Some useful advice about this can be found here: Preferred course of action for flagging of multiple obsolete comments.