I am pretty new in this community and honestly do not know if this question is too out of place here, please let me know and I will delete it.
It's fine. Indeed, if anything it came up too frequently already... :-)
When I try to give an answer to something that looks like part of a homework, I avoid going into too many details, just giving enough information for the OP to figure the rest out. But then I get bombarded with comments like 'what do I do next?', 'should I then do ...?' 'what happens if ...?' Should I reply to these comments?
In my opinion, you should reply something. However, you can very well reply what you said here: "I tr[ied] to give an answer to something that looks like part of a homework, I avoid[ed] going into too many details, just giving enough information for [you] to figure the rest out."
My impression so far is that, more often than not, there's another user in the community that is willing to post a full reply, with a step-by-step solution to the original post. So even if I don't answer these comments, somebody else will, which kind of defeats the purpose of being subtle to begin with.
There are competing views on this subject. I for one am rather of the opinion that one should either be willing to give a complete answer (which still might be terse) or not answer at all. The principal idea of the site is to provide answers that are "ready to use." That is, if you feel not comfortable answering a question completely, maybe just do not answer the question and possibly vote to close it for "lack of context," if applicable.
I do realize that sometimes this may be the result of a race for votes,[...]
Well, incomplete answers or "hints" are often quicker to write down and they can be very popular, so one could just as well argue that what you do may be the result of a race for votes. I take it, it is not, just to point out that there is whole different way to look at it, too.
[...] I do not know if I should just post a possible reply and forget about it, or should I keep in the discussion?
As said above, I think it is generally a good idea to be explicit about ones intentions. No one here is a mind-reader (or who knows...but then in any case I'd think at least hardly anyone :-)). Somebody else could just as well reply to the questions directed at you or complete your answer, because they think you might be busy with something else now.
downvote answers that are too explicit for homework?
It is a somewhat common practice. It is also controversial. Personally, I would not answer a question incompletely and downvote complete competing answers. I sometimes (but rarely) downvote answers to questions that I believe are clearly off-topic; yet, if I do so at all than this is rather directed at poor answers.