I hope it isn't too late for this answer. Much of it is general, but relevant I feel.
The central point of MSE is, as someone pointed out, providing answers to appropriate questions. There appears to be a slight debate over what form should those answers take.
To be strict about it, just about everyone has the necessary background knowledge to solve any solveable mathematics problem - it's just that the number of intermediate steps from where a person feels comfortable beginning his or her analysis and the solution may differ. This is partly because some may have to start nearer to the bottom of the knowledge pyramid, and partly because some are more comfortable skipping the details between steps.
I believe what most people need to overcome a mental obstacle is
- a way of looking at things
- freedom from ego/fear of seeming stupid
I would guess point 2. rarely features on the internet where anonymity is possible.
In theory, one could answer every question in a form similar to "this follows from the axioms" and be done with it. Strictly speaking, if the goal is merely to provide answers that should suffice.
Of course it does not, and the only purpose of that contrived example is to point out that the directive of MSE is perhaps slightly more subtle than described in my first paragraph. Truly, I feel the goal is to provide explanations/solutions that others can understand, with a strong focus on helping the OP to understand. Now I'm not trying to dictate to the administrators of the site what its purpose should be, but rather observing what I perceive the purpose really is. I may be wrong.
If I am right, however, then part of the "job description" of the community is to lead others to answers when the path is blocked for them. It is not any one person's job however, so I think it is perfectly reasonable to quit helping someone if you feel it is futile and leave it to whomever may wish to try their hand at it. This is part of the code of the community, but was this user truly a member of the community:
I am very curious about the reputation and sign up date for the poster whose question you were patiently addressing. If he/she had signed up that day and had a score of $1$, I must say this would color my view of the situation. But I think the most telling part of the dialogue was the moment when he/she asked which assignments to make in the integration by parts. This is something one can simply try by making a choice and then trying the other possibility if the choice fails (I know that strictly speaking there are more than $2$ choices, but I think you see my point).
It is possible that this person was so flustered from banging his or her head against a problem for a long time that he or she had trouble solving problems within his or her wheelhouse. I've been there. In such (real life) cases I try to help people relax and know that no matter what the problem will get solved, in an attempt to sap the feeling of urgency and panic. If it is part of the path to the solution I think this a valid approach here, though perhaps most appropriate in chat.
It is also possible that this person isn't really a member of the community, but rather a vacationer or worse someone here to plunder our resources. I do not believe the code I alluded to above should apply to such people.
Finally, I do not think you were out of line. I believe you were admirably patient. However, what I might have done is ask a leading question or two to try to determine whether this person is a true member of the community or just using a visitor's visa. If my determination were that the really are a member of the community, then I may have taken the discussion to chat where some less conventional approaches may be more appropriate.