The first thing is that this site is open to everyone, I strongly doubt if there is an age limit (correct me if I am wrong). From grade school math to high school math to college math to post graduate studies and research works. This difference in level/difficulty can upset some people who are looking to involve only in a certain difficulty. If we take some statistics and find out that the community has a higher percentage of high school students for example, these students will (most probably) only be able to answer and post questions at high school level thus leading to a vast amount of questions in high school level which may render the community low in quality to some people, most probably of higher level.
Looking at the period of the year where the site receives higher traffic, I couldn't get any stats here from http://data.stackexchange.com/ but using a simple logic; if this site has a higher percentage of college students, one will observe that the traffic of their activities will depend on the situation of schools around the world. When school resumes, there will be low activity. Close to mid terms, traffic reduces and close to finals, the traffic explodes.
I am not saying that the only thing people come to MSE to do is school work but if this site has 60% high school + college students, then pick an average student from high school or college and ask him why he would be interested in a site like MSE. The first thing his mind would go to is probably homework or classwork or study for exam. in this situation, we all know how wild the media is these days, if one student finds out a trick on how to get his homework done here lazily, he will most definitely share this trink with his other friends and then the trend will go on and the percentage increases and quality reduces.
Moving up to a higher level with post graduate, research works and professors. in this situation, I would compare MSE with SO and SuperUser SE. These two sites involve a large percentage of members that have almost constant interaction with the subject of the community everyday of their lives. Take for example in the summer one would find less of high school and college students here on MSE because well, it's summer and everyone should be on break. But a user in SO or SuperUser didn't stop using his gadget and probably didn't stop coding and then wouldn't need an homework to encounter a problem or situation that will bring him to the club. The same goes to post graduates and researchers and professors here on MSE, Maths is no longer a school subject to them, it's now their life so their traffic on the site won't vary much from time to time. To some people Math is work, to some a burden, to some fun, to some it is their life and each and everyone of us must go through Maths in life to have a formal education. In this situation the question is; What percentage of the community do they hold?
If the percentage is high like on MO then it is good for everyone and most people will agree on the quality of the community. But if the percentage is low in comparison with school and college students then there may be a problem. there may be a problem because members in this higher level are the voices and leaders of the community and the community is seen through their eyes so if the community is filled with high school and college students who ask questions that is way lower than their level, they may tend to believe that the community has a low quality and this belief can spread.
In summary, it is difficult to find a solution that will make this community interesting for anyone as the number of users explodes without chasing away some users. One solution may be to split the site into various branches of Mathematics but then Mathematics wouldn't have a meaning. Another solution may be to enforce a tag that denotes the level/difficulty of the question as presumed by the asker. Of course these level must follow a well defined norm and moderators and editors will be able to correct a wrongly placed question. Just because the difficulty level of a question is easy shouldn't necessarily mean the answerer can't upgrade the level of his answer but must do so after having successfully answered the asker's question to his level of understanding. In this way, users who feel some questions are low in quality because of their difficulty can easily filter them out.