The problem with proof verification

What I am going to bring up is not a new problem. It was hashed again and again and again. It was brought up as far as 2013, and repeatedly almost every year. See 1, 2, 3, 4, 5, 6, 7, and 8.

Some of these questions discuss whether or not it makes sense, or what to do when the supposed proof tries to prove a famous open problem, or how to even answer such question, or how to make them more attractive.

These are all valid concerns. But most of these were asked quite some time ago. And the growth of the site makes it harder to fix any future damages, so I think we need to talk about this again. In this question, I am not offering any solutions. I just want to highlight the problems and bring this back to the table, and engage with a discussion on what are possible solutions in the answers below.

I had recently asked a user who had been posting a lot of fairly elaborated proofs (sometimes in response to suggestions on previous questions). With a little discussion in the comments, this is what I managed to extract as far as pros to proof verification questions:

1. Not everyone in this site is a student, not everyone has a teacher, assistant, or a grader to check their work and make remarks. Many people here are hobbyists or self-educating themselves, and they need help from others to make sure that they are not making mistakes in their work.

2. Supposedly, there is a contribution of an elaborate proof to the corpus of proofs on the site. This helps future visitors, since they can find this proof and learn from it, and learn from its mistakes if those are pointed out in the answers.

3. In one instance, the proof seemed fine, and even one person said it's fine, but then someone else pointed out a subtle mistake. Which requires closer attention to details. This is something you might not be able to do yourself if you're asking these sort of verification related questions, and this is how we truly learn mathematics.

On the contra side:

1. Future visitors will also want their proofs to be verified. They will simply post their proof, and this will not be a duplicate, because the question is not about the problem, but rather about the solution, which is likely to be somewhat different than the previous instances. This leads to an inflation of questions which are asking essentially the same, but are not "formally duplicates".

2. It might be that someone looks at the proof says "Yeah, that looks good", but they glanced over the details. In that case, especially if this user is more established, any mistake might be overlooked by future readers. (The issue here is that finding a mistake is a $$\Sigma_1$$-type problem: until you found it, you don't know if it's there.)

3. If you are looking to contribute an elaborate proof, then you're looking to contribute an answer to this website. Not a question. It is far more reasonable to post the suggested proof as an answer to a question about the statement you're proving (or post a question and an answer if there is no such question). Of course the problem is that not everyone bothers to read the answers, you cannot award people a bounty for proof reading your answers, and so there is no "points incentive" for people to engage with your answer. This is why sometimes a user would post an answer to their question after receiving a partial or broad-strokes answer, and a few days later move this to a proof verification question, where they can incentivize more people to look at it.

What do you think we should be doing with proof verification?

The issues here are that the site is growing fast in users, and this means that it grows even faster with more questions. If we decide that we enable a proof verification culture, this might cause an even more serious problem in the future in the form of repeated questions with the same proofs.

For example "Posts containing 'algebraic numbers is:q'" show $$9$$ questions, out of the first $$10$$, asking to verify the proof that the algebraic numbers are countable.

This will continue to blow up, rapidly. Whereas the problem with "homework mill questions" (for lack of a better term) can be solved by dedicated people who point out duplicates and close those questions where the OP is truly disinterested in providing any contribution, proof verification are hardly like that. The OP usually shows their work (often to a degree one might call excessive), and seem to be willing to contribute to the site their version of the solution.

• Proving things in mathematics is both a skill and a discipline. People need to try these things for themselves, and importantly, receive feedback from their efforts in order to gain (or lose) confidence. I'd prefer to accept them, as they are a really great way of helping people. Commented Sep 20, 2018 at 10:33
• Yes. But if you don't put a tourniquet on it, the in flux of proof verification will not decline. And since none of them are formally duplicates, whereas any other question would be eventually a duplicate, this causes a problem.
– Asaf Karagila Mod
Commented Sep 20, 2018 at 10:38
• This touches on the reason why I think the dual purposes of MSE of both helping people with mathematics and creating a repository of mathematical knowledge are not always compatible. Again, I would tend to err towards the former rather than the latter. Is it feasible to delete these questions after they've been answered? Commented Sep 20, 2018 at 10:42
• Yes, in principle, if there is at most one answer and that one possible answer has score 0, it is possible for the OP to delete that. But why would they? They have no incentive to do so. As "outsiders", yes, every question can be closed and then deleted with enough people willing to vote on these things.
– Asaf Karagila Mod
Commented Sep 20, 2018 at 10:45
• @Theo: That there are a lot of proof verification questions that essentially ask to verify possibly different proofs of the same statement?
– Asaf Karagila Mod
Commented Sep 20, 2018 at 12:13
• How about creating a canonical proof verification question, where each answer links to a set of all proof verification questions asked so far for a particular problem? Then, posters any new proof verification question for these problems could first be pointed to these links, to check if they already have an answer. This allows linking questions to "duplicates" without closing them explicitly as duplicates. Commented Sep 20, 2018 at 17:50
• @GoodDeeds: About half of the proof verification requests essentially repeat proofs that were given before in "proper questions". But it's not enough, because it requires the OP to say "Okay, this is how my proof goes, and therefore it's correct/incorrect", which if they can do, they can also do on their own without posting it to the site.
– Asaf Karagila Mod
Commented Sep 20, 2018 at 18:49
• I think there is a point at which proof-verification questions, say, a proof that $\sqrt 2$ is irrational, have been asked so so so often that closing as a dupe of any one of the many questions, or merges of questions, which show likely every way to proceed. For an asker to insist: "But I want my proof evaluated" comes pretty close to the close reason "seeking personal advice". I don't think MSE was designed to address each user, personally. I think the use of chatrooms like "Constructive Feedback" or any number of chatrooms are a better place for users to ask for proof verifications. Commented Sep 20, 2018 at 20:19
• @TheoBendit I don't believe anyone wants math.se to be "a repository of mathematical knowledge" but I do believe they want it to be "a repository of mathematical questions and answers." I am curious to see what you think the incompatibilities are. It seems like maintaining a good repository of questions and answers "helps everyone learn." I think it's possible to practice both without incompatibility. I do not see what one is forced to trade off. Commented Sep 21, 2018 at 17:30
• I am curious what everyone’s idea of a good question is. What is the perfect balance between “homework mill” and “proof with excessive work shown”. Is there some perfect middle ground that new users should be striving for? Commented Sep 22, 2018 at 6:57
• Also, completely serious, why is it such a treacherous problem for the same proof to be verified more than once? Are we running out of space in the cloud or whatever Commented Sep 22, 2018 at 7:11
• @Prince: If the search engine was perfect, and everyone would have written easily findable questions, and everyone would have known how to use the search engine, then it's the amount of noise to signal is irrelevant. But that's almost the opposite of reality. If you are searching for a proof of, say, "the algebraic numbers are countable", and you come across 100 "is this proof okay", that seemed to have no answer but a few comments (since how would you answer that?), you might not be sure about the correctness of these proofs (as evident by many posters who find that insufficient too). [...]
– Asaf Karagila Mod
Commented Sep 22, 2018 at 7:19
• [...] But if you found only a small handful of questions, mostly about specific gaps in proofs, and one or two with people actually outlining the proofs, or verification seekers writing their solution as actual answers, then it's less confusing. I'm not talking about the quality of the proof, sure, one proof is enough. Just about how confusing it is to treat 100 almost-similar-and-usually-ignored questions "is this correct?", this is really tiring. I know from experience.
– Asaf Karagila Mod
Commented Sep 22, 2018 at 7:21
• @AsafKaragila (to answer to "how would you answer that") In questions which I tag with 'proof-verification', I usually hope to receive more than a mere "yes, that's ok" comment: as someone who's trying to learn mathematics by itself, I find very important to receive suggestions and corrections on how I could have done my work better, even if it is correct. Watching the problem from the other side, I find instructive to read someone else proof of something related to what at the moment I'm studying, and then read some advice on how this can be done better, right below the post. [...]
– user457568
Commented Sep 24, 2018 at 16:23
• @amWhy offering personal advice means giving someone advice on something outside of a mathematical problem. Someone can ask for help on how to solve a problem that has never been presented on the site and we can say that helping is "giving personal advice", but that's by taking the meaning a bit too literal. I don't think a user should be brushed off because they want to know what they did wrong rather than just being told "this is how you actually prove it" and being given a proof that looks nothing like their attack vector. There can be many proofs of a statement from different directions... Commented Sep 30, 2018 at 3:58

2 Answers

I do not at all agree with rschwieb's statement that "the way we get feedback for solutions is to post them as a solution, and then people have at them". People who have a very weak foundation in mathematics should not be dumping their so-called 'proof attempt' of any mathematical problem onto an existing thread asking about that problem!

Proof-verification questions exist to serve the needs of students who need guidance in writing proofs, perhaps because they were not taught basic first-order logic, and thus cannot quite put together a logical argument from their ideas.

I think that proof-verification questions can be closed as duplicate if roughly the same approach had been asked about before. In my opinion, it is the responsibility of the asker to do sufficient research to determine whether his/her approach has been tried before, and only post a question if he/she cannot find it (at least on Math SE). As amWhy commented, proof-verification questions of very common problems have been asked so often that it is almost surely a duplicate of a prior question that shows likely every way to proceed.

I also think that a good proof-verification question will avoid the problem of insisting "But I want my proof evaluated..." as follows. After a bit of research one should be able to (or learn to be able to) identify in what key ways one's proof attempt is different compared to existing proofs, and thus ask for verification about those specific new parts. If one cannot identify any part of the proof that is different, then there is no new question to be answered.

This criterion for proof-verification questions would largely solve the key issues in Asaf's post:

• A proof-verification question will not just be a dump of a proof and a request to check it. Rather, it would specifically identify something that distinguishes it from other known proofs (unless it is a new problem, in which case the issue of duplication does not arise). That makes it not only a contribution to the site but also a self-classifying one; future readers can quickly classify the myriad of proof-verification questions about the same theorem by their distinguishing features.

• Mistakes are also more easily identified; if the asker takes effort to highlight how his/her proof differs from the known ones, experts can much more quickly tell whether the difference is inessential or not.

• Future visitors will also have an easier time searching for their proof attempt, if past proof-verification questions each highlight the part the asker is most concerned about. After all, it is likely that short wrong proofs by sincere students are wrong in similar ways, and most proofs on Math SE are short.

• Inflation of questions that are essentially the same will also be curbed, not only because highlighting specific points of uncertainty in a proof makes it more likely to be a duplicate, but also that it is easier for experts to find duplicates.

• My reactions to the four points, in that order are: 1) I think a proof with a specific issue that distinguishes it falls into the category of "i have a specific question about my steps" which is one of the exceptions I enumerated. 2) Why are mistakes more easily identified if they are in a Post with PostId=1 rather than 2? 3) I don't buy that this setup is more searchable. Given the choice between a few posts with many answers of varying quality, and many posts with single answers of varying quality, I think the former is searchable. 4) This seems optimistic, although I wish it were true. Commented Oct 16, 2018 at 13:23
• Nevertheless, I agree that if the advice in this post were followed, the quality of verification questions would be greatly increased! So perhaps it points to another valid method we could be using, which just concerns a set of higher standards for what constitutes an appropriate use. I would advocate that if I thought it would be easier to enforce (which I don't think it is.) Commented Oct 16, 2018 at 13:26
• @rschwieb: I feel rather strongly about not encouraging people to post answers that they are not qualified to post. Ultimately, this is not a forum, but a Q&A site, so the notion of "answer" should be reserved for those who are confident of their mathematical contribution to the question at hand. I've seen many many Math SE answers posted by users who think they have a proof but actually do not, and many of them lie around unchallenged. To respond to your responses: (1) I think we should gently enforce that all proof-verification questions follow my criterion. (2) I don't get your query. Commented Oct 16, 2018 at 15:40
• (3,4) When we require proof attempts to come with a highlighted portion of interest, askers will naturally focus more of the post on that portion. That makes it easier for experts to find duplicates, because they can search for that same focus. Commented Oct 16, 2018 at 15:45
• @rschwieb: I think I get your query about (2), but you misread it. I stated that "if the asker takes effort to highlight how his/her proof differs from the known ones, experts can much more quickly tell whether the difference is inessential or not." I don't see how it can be disputed. Often, such posts save every reader a lot of time in zooming in on the key issue, as opposed to having a long post ending with "Please help me check my proof. Thank you!". Commented Oct 16, 2018 at 15:48
• You undoubtedly saw the recent deluge of proof verification questions about set theoretic questions. How do you feel that your approach handles that sort of situation?
– Asaf Karagila Mod
Commented Oct 16, 2018 at 16:53
• @AsafKaragila: Most proof-verification questions I've seen do not compare it to an existing proof, and that is part of my criterion. One should compare one's attempt with a known proof and identify what are the key differences, as part of the 'research effort'. So common elementary set theory problems that are given every semester shouldn't cause as big a deluge as it might now. Commented Oct 16, 2018 at 17:26
• I am talking about someone who is self-studying, and has repeatedly replied to criticism with "I have only my book, and nobody else" sort of approach to justify their behavior. And they're right, it's harder without friends to talk to or teachers to teach you, or graders to grade your proofs. So how does your approach handles that sort of situation?
– Asaf Karagila Mod
Commented Oct 16, 2018 at 17:27
• @AsafKaragila: I think you'd have to provide an example. I am saying that it falls into two cases. If the problem has not been looked at before on Math SE, then there is no issue. If it has, then the asker should take some effort to compare his/her proof with an existing one. If that effort is not shown, I think it is perfectly fine to close as duplicate. Commented Oct 17, 2018 at 3:03
• I rather not link to any specific user. Suffice to say that perusing the combination of the elementary set theory tag and the proof verification tag should single them right out at this time of writing.
– Asaf Karagila Mod
Commented Oct 17, 2018 at 9:38
• @AsafKaragila: Almost all questions under both those two tags fail to satisfy my criterion. So empirically, my claim that my criterion would solve the proof-verification duplication issue seems sound. =) Commented Oct 17, 2018 at 12:31

I thought my take on the situation existed somewhere, but I think it must only be in comments, or possibly in deleted comments. But thanks for the chance to get it down in a solution!

Answers should almost always be posted as answers.

If the question exists elsewhere on the site, and you have an answer for it, it should go on that question. if the question doesn't exist, you should write it up and self-answer it.

This is the best fit for our format.

The primary goal is to get feedback about your solution. And the way we get feedback for solutions is to post them as a solution, and then people have at them.

I imagine there are a couple reasons some users might shy away from this suggestion.

1. "I didn't know you could self-answer/that it was allowed." Maybe they worry that it'll look like they don't actually have a question, or just plain don't know that there's no block against it.

2. "I don't know if it's right, so I can't post it as a solution." (aka "afraid of getting it wrong") . Maybe putting the answer with the question affords some comfort for this. Of course, we shouldn't post things we know are wrong, but there's nothing wrong with putting forth a solution we think is best. That is, essentially, what all solutions are, with varying levels of success. If this is not already the default, readers should already know that every answer is potentially wrong, and that it requires evaluation on the viewer's part. There is no guarantee a solution is correct on this site.

3. "My question won't get any attention if I put the answer as a solution." This ought not to be true, and I sure hope it isn't. I'm pretty sure that new answers to old questions get attention bumps. (And we might consider having a mild attention bump for new self-answered questions. Mild enough that it hopefully can't be abused. But let's not discuss that particular idea in this thread.)

Does that mean all types of "check" questions should be like that?

No, not necessarily. There are still species of "check" type questions that are actually questions. For example

1. "I came up with this proof <...> but I also know that <...> and that's a contradiction. Where am I going wrong? . ("I have reasonable cause to believe that there's a problem and I need help finding it.") (Example where that's well done)

2. "in this proof of mine <...> I think I should be able to say that , but I'm not sure if I'm justified in doing that step. Am I? ("I have trouble validating this step.")

A common thread for these exceptions is that there is a concrete direction for the question, rather than an open-ended "Are all the steps right?"

Asaf's "pro points"

To address the points about the contras:

I think 1+2 are both handled equally well by solutions-as-solutions. I don't see why solutions-in-questions handles the situation any better. If people are worried their proposed solution will go unnoticed, it should be a fine thing to point out in the question that a proposed solution follows.

3 seems like a clear case of one of the exceptions I listed above: i.e., there is actually a question ('It doesn't work for this reason... how can I fix this gap?')

I hope I'm making myself clear that I am not taking a very hard line. I am not trying to obliterate anyone's options. I'm pretty sure that given any such question I can reason why it should be in one of the buckets above, without alienating anyone.

And by going this extra mile and organizing things so, I think we get a leaner body of questions with a fuller body of answers which is easier to learn from. Moreover, the behavior I'm suggesting has a few healthy aspects in it that are good to encourage, namely:

1. Thinking harder about how to formulate your question (rather than just "here's some stuff I wrote, amirite?")
2. Not being afraid to put your work out there as a solution
3. Being aware of already existing questions and answers on the site

I also feel like I am forgetting a couple of other points I had previously made too, but hopefully they come back to me in ensuing discussion.

Update:

Until now, the existence of a user who asks too many proof verification questions was hypothetical, but recently I stumbled across a real user in the wild. I won't name the user directly, but I'd like to share some data from SEDE queries that may (unavoidably) be enough to identify them.

The user has 700+ questions. I didn't wade through them all, but I looked back until 2017 since the posts seemed to have a regular repetitive structure ("Here is Prob. blah of book blah is this all correct?") There are ~275 since 2017, and at least 170 begin that way. Before then there appears to have been a 2-3 year inactivity period: I didn't do any analysis on those. Some of the earliest posts are what I'd call good quality, so they're not all bad.

This is admittedly an outlier, and I suppose a couple hundred posts is a drop in the bucket, but knowing it is actually happening is a little harrowing.

Another user, distinct from the one above, has come to my attention. There are 194 questions asked between May 2021 and October 2022, seemingly (as discerned from a sample) someone posting their solutions to standard books by Munkres, by Rudin, and Hoffman. As of the writing of this post, 90 were unanswered. Probably some of the single-answer ones are just "yeah that's right" too.

• I like this stance, especially the focus on people asking questions of their own proof to create a better post. Oftentimes we tend to have some sense of where our proof holds are what statements may be dubious if we aren’t confident in a solution. It would be nice to have a how-to on proof verification that reviewers could point such questions to if they fall under the “amirite?” format, as we do for the “how to ask a question” page. Commented Sep 22, 2018 at 3:24
• Re "I have reasonable cause to believe that there's a problem and I need help finding it." - What if "reasonable cause" is "I just 'proved' the Riemann Hypothesis in five lines of logic, but it can't possibly be that easy, so obviously something has gone terribly wrong." Are those good questions or not? Commented Sep 23, 2018 at 17:01
• @Kevin I don't see why that would be a bad question at all. It will probably be a fairly low level answer (i.e. "you flipped your quantifiers in this step" + example why this is invalid), but I don't think that as described it would be a bad question. If it is 14 pages of indecipherable mess and "I proved RH but I doubt I am correct" then it needs more work to be a good Q. Commented Sep 29, 2018 at 11:10
• @Mario: Thank you for taking my question seriously. Commented Sep 29, 2018 at 12:25
• @Kevin If you would have responded to my last comment, I would take it seriously too. It appeared you were complaining that the "Where am I going wrong?" exception is too permissive and would qualify a "crank" post as "good." My response to that would be that nothing about my suggestion requires us to abandon common sense and use of other quality filters. If you didn't intend to talk about crank posts, then why isn't that type of question clearly covered by the "Where am I going wrong?" point above? Commented Sep 29, 2018 at 13:17
• @rschwieb I don't associate Kevin's example with a crank post. One of the characteristic behaviors of cranks is overconfidence in the correctness of the proof. I'm not sure I've ever seen a crank approach their proof of RH from the angle "I'm sure this is wrong, but not why". Here again I see the value in your recommendation for questioners to analyze their proofs critically rather than just "amirite". Commented Sep 30, 2018 at 2:20
• This is wrong though. It invites people to write answers to the problem rather than for the proof. Plus if someone writes an incorrect answer on this site it usually gets downvoted and deleted as being incorrect. The question is "what is wrong with this proof and why does it fail" and it is not "how do I prove this statement". Changing the question to have that meaning will actually prevent the user from getting what they desire. An incorrect proof being asked about in a question gets a fix. A deliberately incorrect answer gets a delete vote and a ton of down votes. Don't break the site. Commented Sep 30, 2018 at 4:01
• Also the instant an answer gets deleted from the canonical question to use for proof verification (due to being incorrect and considered spam for some reason) the answer will be protected and require a certain amount of reputation to post in. This effectively roadblocks users from using as (if as you claim) attempts should be posted as answers looking for help and all other attempts be closed and deleted as off topic. We're just denying users from asking their question. Commented Sep 30, 2018 at 4:05
• @MarioCarneiro Yes, verifying whether it is a cranky post or not is a grey area which needs to be handled on a case-by-case basis. But using the subject matter of unsolved problems was highly suggestive. But then as I said, if that topic of the question is not at issue but rather the idea of someone asking "where does this argument break down," I thought I completely covered that in the caveat section above. That is, when a user knows there is a problem and is seeking help locating it, that is acceptable. Commented Oct 1, 2018 at 15:19
• @TheGreatDuck When you write The question is "what is wrong with this proof and why does it fail" and it is not "how do I prove this statement". it indicates you have not read what I wrote completely. I said that when the user knows there is a mistake but is seeking help finding it, that is not necessarily eliminated. Quite frequently, the user is not contending that something in their proof is wrong. They are not asking "where does it go wrong' but rather is this all correct? The latter type of question is what I think we need to minimize. Commented Oct 1, 2018 at 15:23
• @TheGreatDuck Also the instant an answer gets deleted from the canonical question to use for proof verification (due to being incorrect and considered spam for some reason) the answer will be protected and require a certain amount of reputation to post in I think what you are describing is unrealistic. It isn't our policy to delete answers for those reasons. I'm also not aware that people are rep-limited from answering their own questions, and at any rate the users who would be restricted that way do not represent a significantly large portion of posers facing this problem. Commented Oct 1, 2018 at 15:26
• If anyone can confirm whether or not answer-rep-limiting applies when you are answering your own question, I would be interested in knowing for sure. Thanks. Commented Oct 1, 2018 at 15:28
• @rschwieb If you are answering your own question to verify your proof without causing a problem then it isn’t producing a duplicate. If it is producing a duplicate then that does not solve the problem outlined above of these question possibly needing a delete and close vote for reasons of duplicate. My assumption was that things that get a large volume of failed proof attempts need to have their own canon question that everyone uses for that one statement. Hence why I said that it would get closed from new users looking to get help with such proofs. Commented Oct 1, 2018 at 23:15