What you describe seems like the normal behavior.
When you flag as off-topic, you are first presented with several options, a few giving reasons why the question is not suitable for this site, and one of them being "belongs to another site."
When you chose that last option, which you should only do when you are sure the question is off-topic here yet a good on-topic question on another site, then you are indeed presented only with a few of all the SE sites as potential target.
The ones that are presented are fixed by SE staff globally for the site, based on various consideration. The total number for every site is kept very small; there may even be none except the site's meta. For this site it are those that you saw.
What you could do when you are sure a question should be moved elsewhere is:
flag as 'off-topic' with the generic off-topic reason "not about mathematics within the scope of the help center".
write a comment pointing to the other site.
optionally flag for moderator attention "other" an recommend the question should be moved. (Moderators can move questions to any site in the SE network.)
Yet reserve this to cases where you are sure the question is a good fit on the other site. As a rule of thumb, only recommend migrations to sites where you are a user.