I can share my own heuristics, first with respect to Questions in the First Posts Review Queue, then with respect to Answers that show up either there or in the Late Answers Review Queue.
For Question Reviews
When a Question makes no sense, I use the "very low quality" option. I find a range of such posts, from "keyboard banging" to word-salad to rants against The Mathematical Powers that Be. When it is very difficult to imagine edits that would fix such shortcomings, it is best to use "very low quality" rather than "needs improvement".
When a post attempts to ask about something in the on-topic realm for Math.SE, but not clearly enough that Readers would be able to give a cogent answer without more information, then "needs improvement" is the better option. One is offering the OP a chance to realize that what they had in mind has not been sufficiently articulated.
There remains a further range of Questions that, while clear enough for some Readers to jump at answering, fail to meet Quality Standards, typically by lacking context for the problem posed. In these cases my practice is to leave a Comment, alerting the OP to the oversight in not providing context or other issues like stating the problem only in the title of their Question. [Should I encounter these kinds of posts in the Close Review, I would use the "community-specific reason" version of "missing context or other details".]
Leaving a Comment permits me to check back later (from my Comment history under All actions tab) to see if the OP has made useful edits, and thus I can defer downvoting a First Post when no improvement is made.
For Answer Reviews
The same distinction applies broadly to reviewing Answers, noting that the Late Answers queue should be populated solely with answer posts. If you cannot see a prospect of an edit that would bring a post up to guidelines, use Very Low Quality. If you can envision an edit turning a poorly articulated Answer into good content, mark it as Needs Improvement.
Here is what the FAQ tells us about the two Review Queues to which one is given early access:
First posts - 500 reputation. The First Posts queue contains the first posts made by users. These posts may need special attention because the users are not familiar with site quality rules.
Late answers - 500 reputation. This queue contains answers that were posted much later than the question. Reviewing these posts makes sure that great new answers get seen and problematic answers get removed.
I've had occasion to chime in previously on how to review a first post, so let me just add a few thoughts about the Late Answers queue, which is regarded as the one with lowest activity.
One sometimes sees late "answers" that are effectively "I'm having this same problem too" or even posting an unrelated problem. Since these are not going to become good Answers short of a miracle, I'd leave a Comment about this not being the right way to ask a new Question, etc. and mark it as Very Low Quality.
Another easily identified case is the "link only" Answer, which might bear a little scrutiny before choosing between Very Low Quality and Needs Improvement. There's a stock response that applies to link-only answers, but not all links really fit the text that proposes a solution could be found by quoting or summarizing material referenced by the link. It is possible the link represents spam or other abuse of posting privileges, and sadly one sees new accounts being created for this end.
Finally, thanks for being diligent in exercising your new responsibilities for Community Moderation.