Which flag should I use when somebody did not try anything and wants help?
I tried the off-topic, not about what is in the help-center, but the flag was declined.
Technically, you used basically the correct flag.
"Should be closed, Off-topic, Lack of context" is the standard sequence for this case. (It is not clear from your description if you picked the last but that's a detail.)
Opinions on the closure of such questions just are mixed; some are of the opinion nothing should be done in such a case. Thus, a declined flag can happen and is not a big deal.
If a considerable proportion of your flags is declined you might reconsider your standards though.
Anyway, the flag was the correct one. It may be you should not have flagged at all, but there is no other flag you should have used.