I could imagine a system of categorizing the questions that would work alongside the current tagging system. If you select the "homework" tag (or some special tag or option), it would give you the option to specify which textbook problem your question pertains to in terms of title/chapter/section/problem number. Maybe the site could present a list of textbooks that have already had one solution entered and the user could navigate through a hierarchical tree of problems.
Now the users would be able to search or browse by textbook title, go to the chapter/section/problem and see if there is a solution present from someone else who has asked a question about it before.
This feature would make the site a lot more organized and eliminate redundant questions, or at least make it possible to find all the questions related to a specific problem in a textbook by providing explicit links. Math Stackexchange could eventually turn into the authoritative source of solutions for textbook problems. I think it would make the site a little less intimidating, too, if it was easier to use without having to ask questions all the time. Searching for mathematical symbols is pretty hard to do.