31
$\begingroup$

The author of this answer (now deleted) claims that it was generated by AI (artificial intelligence). This claim could certainly be false (for example, if the user wants plausible deniability for an answer they're not confident about), but assuming it's true it raises a general question: what's our policy on AI-generated content like this? Specifically:

  • Do we / should we allow AI-generated user content (questions, answers, comments) on the site at all, assuming it otherwise meets quality standards?
  • Does the AI-generated content still need to be posted by a human user?
  • Does there need to be a disclosure that the content is AI-generated?
  • Should a human user be able to receive reputation (positive or negative) for their AI-generated content? Should the content be Community Wiki?
  • What actually constitutes "AI-generated" content?

Note here I'm not asking about content generated by something like the Community Bot.

I didn't find anything directly addressing these policy questions in the help documentation or here on meta, or at a quick glance in Stack Exchange's own TOS or AUP. Given the rise of AI-generated content in the world, I feel like we should probably have some guidelines.

Other related questions:

Note: some folks have provided helpful links to related posts about banning of ChatGPT on other Stack Exchange network sites and/or the network as a whole. I feel my question here is still relevant because (i) it is not specific to ChatGPT and (ii) it is specific to the Math Stack Exchange site, for which there may be some special considerations different from other network sites.

Update: there is now policy documentation posted here.

$\endgroup$
10
  • 7
    $\begingroup$ See related (MSO): Temporary policy: ChatGPT is banned $\endgroup$
    – Paramanand Singh Mod
    Commented Dec 8, 2022 at 16:08
  • 8
    $\begingroup$ Related on Meta SE: Ban ChatGPT network-wide $\endgroup$
    – The Amplitwist
    Commented Dec 8, 2022 at 16:29
  • 6
    $\begingroup$ See the highly upvoted: Temporary policy: ChatGPT is banned. Although this applies to StackOverflow, the reasoning will IMHO apply to Math.SE. $\endgroup$
    – hardmath
    Commented Dec 8, 2022 at 18:17
  • 4
    $\begingroup$ There is now an "official" help page on SO about this: stackoverflow.com/help/gpt-policy . Also, the moderators have had some internal discussion about this. We have no official policy yet. I am hoping to post an answer here soonish. $\endgroup$
    – Xander Henderson Mod
    Commented Dec 8, 2022 at 18:57
  • 3
    $\begingroup$ A certain user has taken to posting wall-of-text questions and answers around here lately. I wonder whether AI-gone-rogue is at work. $\endgroup$
    – Gerry Myerson
    Commented Dec 8, 2022 at 21:17
  • 3
    $\begingroup$ I wonder, could AI produce good math questions? Or good math answers? $\endgroup$
    – Dan
    Commented Dec 9, 2022 at 4:49
  • 4
    $\begingroup$ Now being discussed at MathOverflow Meta: ChatGPT strikes MathOverflow $\endgroup$
    – Gerry Myerson
    Commented Dec 9, 2022 at 5:51
  • 1
    $\begingroup$ An example of a post I just came across - Flaw in AI generated Riemann hypothesis proof $\endgroup$
    – Calvin Khor
    Commented Dec 9, 2022 at 6:21
  • 6
    $\begingroup$ @Dan The problem is that the AI answers which were posted/linked above are absolutely flushbunking. They're easier to spot than Shaquille O'Neal on Lilliput island, and are about as mathematically correct as the Earth is flat. I think there will be a day when we have to be more concerned about them being more accurate than us, but today, nah. So far the contribution of AI here is more like AS, artificial stupidity, and the quicker we get rid of it the better. $\endgroup$
    – Sarvesh Ravichandran Iyer
    Commented Dec 10, 2022 at 4:48
  • $\begingroup$ (*Stack Exchange (the last section), not StackExchange.) $\endgroup$
    – Peter Mortensen
    Commented Apr 26, 2023 at 11:29

5 Answers 5

Reset to default
28
$\begingroup$

Current Policy

The current policy on generative large language models ("AI") on Math StackExchange is that it is impermissible to use such tools to draft questions or answers. For details, please see the Help Center. If you believe that a question or answer was written using tools like ChatGPT, Gemini, or other similar large language models, please flag the suspicious post for moderator attention.

The following, which predates the implementation of the above policy still remains largely relevant—I am leaving the answer intact, with strikethroughs, to mitigate against the need to click through to read the edit history:

Math SE Policy

Currently, there is no policy.

In general, we believe that existing policies are sufficient to handle AI generated content on this site. Specifically, answers here should be correct, and should not be plagiarized. The output of ChatGPT is typically incorrect, and posting the output of a ChatGPT session without disclosing that the text was generated by ChatGPT is a violation of OpenAI's terms of service (hence plagiarism).

So, again, existing policies on content quality seem to be sufficient for the time being. We'll revisit this if things change in the future.

Larger Context

To provide a bit of inside baseball, the moderators at SO made the decision to ban ChatGPT generated posts explicitly because they were being flooded with this kind of content, and the human moderators could not handle the the influx of these posts under existing protocols. An explicit, temporary ban on ChatGPT-generated content at SO was needed, as the usual procedures could not keep up (the same quality and anti-plagiarism policies apply, but the procedures for handling low-quality and plagiarized content could not keep up).

Math SE does not have nearly the volume of content generation as SO, and there is not yet clear evidence that more than a handful of users are trying to post ChatGPT answers here. We have not yet felt any need to enact an official policy.

$\endgroup$
14
  • 10
    $\begingroup$ I find myself unpersuaded. The current policy is that mods generally do not judge the correctness of answers. Therefore, if someone flags an answer as incorrect, mods are rightly going to decline the flag on the basis that they don't judge correctness. So the policy of "correctness" doesn't seem like it addresses the problem. It does set an expectation for users, which is good, but mods will not act on violations of it. $\endgroup$
    – D.W.
    Commented Dec 17, 2022 at 23:33
  • 2
    $\begingroup$ Regarding plagiarism, if there is a declaration that including ChatGPT output without attribution is considered a violation of the rules regarding attribution, then I think that should be very clearly stated. I am concerned that someone is going to flag a ChatGPT answer, and mods are going to respond by saying "no evidence of plagiarism, where's the link showing where it was copied from?". Finally, regarding OpenAI's terms of service, violation of another web site's terms of service is not considered grounds for deletion/action here, so that doesn't seem to address the issue. $\endgroup$
    – D.W.
    Commented Dec 17, 2022 at 23:34
  • $\begingroup$ That policy was established almost a week ago. Using machine-generated text without attribution to the source is plagiarism on Stack Exchange, and it is not acceptable. $\endgroup$
    – Nij
    Commented Dec 18, 2022 at 5:49
  • 4
    $\begingroup$ @D.W. We actually do have ways of verifying that a post has been plagiarized from ChatGPT. Don't hesitate to flag. $\endgroup$
    – Alexander Gruber Mod
    Commented Dec 18, 2022 at 6:36
  • 4
    $\begingroup$ I agree that we should have a linkable meta post explaining how the plagiarism rules specifically apply to ChatGPT and other AI generated text. $\endgroup$
    – Alexander Gruber Mod
    Commented Dec 18, 2022 at 6:42
  • $\begingroup$ @AlexanderGruber, thank you for the reply. You've addressed my concern. That all makes sense. Thanks again. $\endgroup$
    – D.W.
    Commented Dec 18, 2022 at 8:47
  • 2
    $\begingroup$ @D.W. My a priori assumption is that a ChatGPT generated answer is incorrect. Hence I have no problem deleting such answers. On the other hand, while moderators are not responsible for judging correctness, the community can (and does). I would recommend (1) voting to delete ChatGPT generated content as "not an answer", (2) leaving a comment suggesting that it is incorrect (and why) so that others can vote, and (3) raising a mod flag. It shouldn't require moderator attention to delete wrong answers, but I, at least, am happy to delete AI-generated answers on sight. $\endgroup$
    – Xander Henderson Mod
    Commented Dec 18, 2022 at 17:25
  • $\begingroup$ @AlexanderGruber: how can you verify that a post has been plagiarized from ChatGPT? I am curious about it $\endgroup$
    – Taladris
    Commented Dec 19, 2022 at 1:42
  • 2
    $\begingroup$ @Taladris That's being kept confidential, at least for now. $\endgroup$
    – Alexander Gruber Mod
    Commented Dec 19, 2022 at 18:18
  • 2
    $\begingroup$ @AlexanderGruber Re: "we do have ways of verifying ... ChatGPT (generated answers)". If you are refer to tools like ZeroGPT then beware they may be highly inaccurate - yielding many false positives, e.g. ZeroGPT claims many of Arturo's recents posts are AI/GPT generated. $\endgroup$
    – Bill Dubuque
    Commented Apr 16, 2023 at 21:49
  • $\begingroup$ @BillDubuque That's not our method. I wish I could elaborate further, but we've been asked to stay confidential, as having any details out there can aid people in trying to subvert it. $\endgroup$
    – Alexander Gruber Mod
    Commented Apr 17, 2023 at 1:28
  • 1
    $\begingroup$ @AlexanderGruber Do you think the method you have is much better? I highly doubt any method currently available be good enough for such purposes. $\endgroup$
    – Bill Dubuque
    Commented Apr 17, 2023 at 2:53
  • 1
    $\begingroup$ @BillDubuque I can't really say with any more precision than it seems to work pretty well to me. I didn't develop this method and am not in charge of its QA. It's better than the GPT detectors out there, but again, I can't really go into why. $\endgroup$
    – Alexander Gruber Mod
    Commented Apr 17, 2023 at 5:00
  • $\begingroup$ Updating @AlexanderGruber's comment: a group of per-site moderators, Community Moderators, and interested users with appropriate background in LLMs has been working over the last year to craft guidelines for detecting and handling LLM-generated content. When we delete this kind of content, we (diamond moderators) are expected to make reference to the tools and heuristics developed (and being developed) by this team. I am not at liberty to get into the details of when we classify content as "almost certainly LLM-generated", but I think that the risk of false positives is low. $\endgroup$
    – Xander Henderson Mod
    Commented Nov 28 at 1:04
43
$\begingroup$

AI-generated content should be banned from MSE (except potentially from the profile page). I have had direct experience with evaluating answers produced via an AI tool (some chucklehead posted some AI-generated answers in a tag I watch) and the community experience with them was poor: they were nonsense, but required extra effort to deal with compared to garden-variety nonsense. Add to that the fact that these AI-generated answers are easy to produce at scale and there is a big potential problem there.

$\endgroup$
6
  • 6
    $\begingroup$ "Chucklehead" , "garden-variety nonsense", brilliant, +1 then and there! I think the scale argument is the most important : it is so, so possible to churn out balderdash on an industrial scale using an AI, and make it believable. I never worked with an AI, but I did forward some "random poems" to people which were generated by Markov chains on English words/phrases. It is quite easy to generate AI answers on a scale, for example a modular arithmetic question like (simplify $a^x \pmod{c}$) could be automated, so it's a threat we ought to take seriously. $\endgroup$
    – Sarvesh Ravichandran Iyer
    Commented Dec 9, 2022 at 4:38
  • $\begingroup$ Note that we aren't... or rather we're assuming that AI-generated content can't be quality post/comment/content. This possibility is definitely there. Would there be a problem still if there is a flood of quality AI-generated answers? Though I don't think it matters to me in any way but my only concern is that AI can be very useful. $\endgroup$
    – InanimateBeing
    Commented Dec 10, 2022 at 15:52
  • 12
    $\begingroup$ The rate at which AI tools currently produce correct answers is so low that that should be banned to prevent us from wasting our time. If that changes in the future, you're welcome to start the discussion again, but that's not where we are. $\endgroup$
    – KReiser
    Commented Dec 10, 2022 at 16:11
  • $\begingroup$ @InanimateBeing Yes, I think it's safe to assume that. Having said that, you're right in that if that assumption is broken, then we need a deeper conversation. $\endgroup$
    – Sarvesh Ravichandran Iyer
    Commented Dec 11, 2022 at 3:31
  • 1
    $\begingroup$ @SarveshRavichandranIyer sure, thanks. KReiser, though I'm not fully aware of the scale of the problem, but if it is really that big (and hardly useful) then it makes sense to do away with it. Perhaps when someone can actually really put it to good use here (in a big way) then we can again have a meta discussion then. Thanks for clearing my concern. $\endgroup$
    – InanimateBeing
    Commented Dec 11, 2022 at 5:33
  • $\begingroup$ @InanimateBeing Perhaps this discussion should be titled "what's our current policy on ...", given that any change in the way AI answers are viewed could affect this. At this point, let me just say that it will be very difficult for me to accept AI-generated answers on this site, even if they do become a majority at some time. $\endgroup$
    – Sarvesh Ravichandran Iyer
    Commented Dec 13, 2022 at 6:26
15
$\begingroup$

I agree with the position stated by @KReiser that such content should be banned from posting Answers. (I'm also inclined to ban it from Questions.)

Let me point out some characteristics of the example Answer linked in the above meta Question. The post lacks MarkDown and MathJax formatting, even though it responds to a Question that makes substantive use of $\LaTeX$.

Beyond that it lacks suitable whitespace formatting, presenting a "wall of text".

In this case the post (from a really new account) makes a point of saying the material is generated by an AI, and there are those confirming marks in the post to make that claim plausible:

I am not sure this is a great answer, but I ran this through an AI program and id be incredibly interested if what came out is at all helpful. If it is even coherent a little, that would be incredible.

A cynical interpretation is that the poster is using Math.SE to develop a training set for a neural net. Better received posts will reinforce tuning of an algorithm.

For this reason I support not only deleting such posts but removing the user accounts on evidence of repeated behavior. Indeed based on moderator experience across StackExchange communities, a rapid posting of Answers on a wide variety of topics is a strong diagnostic signal.

SEDE queries have been developed to automate this kind of detection on StackOverflow. If there is interest I'll adapt them to Math.SE for comparison. Of course the SEDE data will necessarily be something of a lagging indicator.

$\endgroup$
6
$\begingroup$

You probably want to ban/downvote the life out of AI generated answers UNTIL the AI generated answers are actually good enough to be useful. I suspect that MIGHT even be possible for very elementary math but I highly DOUBT at the moment that is possible for even undergraduate level topics.

Maybe an AI + Proof Assistant needs to be created to generate actually useful content. (Even then the use will be questionable, it will merely be correct but not necessarily useful without significant improvements)

$\endgroup$
2
  • 8
    $\begingroup$ I think the notion of "actually good enough to be useful" bears a critical analysis. Do we require that the AI generated answers are useful to all of the people some of the time? Some of the people all of the time? But I think this misses the heart of the matter, which is that copying of an AI output without understanding what it means will be unsatisfactory Math.SE content even if someone finds it "useful". The point is someone will need to understand it in order to curate it, and if the person posting cannot do that function, they just create a burden for others. $\endgroup$
    – hardmath
    Commented Dec 9, 2022 at 6:31
  • 11
    $\begingroup$ On homework I'll often tell students that they can draw from many sources, but ultimately they are personally responsible for everything they put down on the paper they turn in. It feels like a similar standard should be adapted here. In particular, anyone posting stuff and then responding "I don't know what it means; it's what the AI generated." should be sharply reprimanded, if not penalized. $\endgroup$
    – JonathanZ
    Commented Dec 10, 2022 at 11:01
-6
$\begingroup$

I think it would make sense to ban AI-generated answers until their probability of being correct and useful matches the corresponding probability for human-generated answers. Can we estimate this probability for human answers from the up/down votes or from comments and subsequent corrections? Similar reasoning applies to questions.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .