Occasionally, I get involved in exchanges of comments on MSE and the other party eventually says "let's discuss this in chat". Am I being antisocial in never taking up this offer? I feel that my interaction with MSE is via the question/answer/comment interface, where everything is shared and public. "Let's discuss this in chat" feels like "let's discuss this outside" which in the UK is an invitation to sort something out with a fight in the pub carpark.
The "Let's discuss this in chat..." invitation is a "canned" line that the Stack Exchange software produces automatically when the exchange of comments between two users passes a certain predefined threshold, i.e. the software is programmed to step into the conversation after a number of comments has been made and try to stifle it.
This means that this canned line should be understood less as an honest invitation to chat, and more like a signal that the conversation in the comments section has extended for too much and threatens to clutter the page, and that it would be a good thing for the two users engaged in it to stop it, or at least move it elsewhere.
Since this canned line is not produced by the user that you are discussing with, but by an anonymous piece of software, ignoring it does not automatically make you antisocial (since you are not ignoring a real human being).
It's not my place to judge what is or isn't antisocial, but that certainly wouldn't be my reaction. I don't view it as antisocial. You are always free to decline to contribute to chat rooms, and to step away from any questions where commenting gets moved to chat. That's entirely within your rights. No judgement whatsoever, and no obligation. This is a site of volunteers, and you can always choose not to volunteer. I hope you won't feel any shame or guilt about it.
I would suggest a slightly different way of thinking about this. Stack Exchange's mission (in my view) is to build up an archive of knowledge, in the form of high-quality questions and answers. Comments are a means to an end, but are not the primary goal. As a first approximation, comments exist primarily to help people improve questions and answers (not, e.g., to support interesting discussions).
If you find yourself in a long comment thread, with many comments, or with back-and-forth exchanges with other users, then that is often a "bad smell" that indicates that something has gone awry. Perhaps the question wasn't a good fit for this site, or perhaps comments are not being used for the intended purpose that the site was designed for. So in that situation, it might be a good time to take stock of the ultimate goal (the site mission) and how best to advance that goal.
In that situation, if you can answer the question yourself, you might consider whether you can write your own answer. If there are problems with the question, you might consider whether you can edit the question to improve it, or vote to put it on hold until it is improved. (In theory, if there are problems with an answer that can be fixed, you might consider whether you can edit it to make those improvements -- though on this site, such edits are often viewed unfavorably, so in those situations you might have more luck with walking away or writing your own answer.)
It's definitely not an invitation to "step outside", i.e., to sort it out with a fight or a debate. Indeed, we probably don't want either fights or debates here on this site, as a general rule of thumb. (But perhaps it would not be completely wrong to view it as a suggestion that "you don't have to take this outside, but please don't continue it in here".)
You can think of the "move to chat" as a nudge. It is a suggestion that, hey, something seems like it might be going astray here. There's a good chance that comments are being used in a way that they weren't intended for. And it is offering one possible alternative. You don't have to take that alternative. You have agency to consider how best to respond, based on your best judgement. Regardless, I do think it's worth paying attention to the "something might be going astray here" signal, and using it to consider what is the best way to make a positive contribution on the site, in a way that is consistent with the Stack Exchange format.
I find it hard to say more without looking at concrete examples.