It’s hard out there for a bot. Just when you thought we’d all be writing everything with AI from now until eternity, engineers have started to develop new ways to detect whether text has been written by ChatGPT or another AI text generator.
Here are three detection tools you (or your college professor, or your boss, or your editor) can use right now (or very soon). We asked ChatGPT to “write an article about tools to detect if text is written by ChatGPT” and fed the results to the tools below. All of them determined—or suspected—that the text had probably been written by an AI. Then we fed the text of this article to these same tools—and, thankfully, they confirmed our humanity.
GPT-2 output detector
OpenAI has been wowing the internet with its efforts to replicate human intelligence and artistic ability since way back in 2015. But this past November, the company finally went mega-viral with the release of the AI text generator ChatGPT. Users of the beta tool posted examples of AI-generated text responses to prompts that looked so legit, it struck fear in the hearts of teachers and even made Google worry that the tool would kill its search business.
If OpenAI engineers are capable of creating a bot that can write as well as, or better than, the average human, then it stands to reason that they can also create a bot that’s better than the average human at detecting whether text has been generated by AI.
The online demo of the GPT-2 output detector model lets you paste text into a box and immediately see the likelihood that the text was written by AI. According to research from OpenAI, the tool has a relatively high detection rate, but “needs to be paired with metadata-based approaches, human judgment, and public education to be more effective.”
GLTR (Giant Language model Test Room)
When OpenAI released GPT-2 in 2019, the folks from the MIT-IBM Watson AI Lab and the Harvard Natural Language Processing Group joined forces to create an algorithm that attempts to detect if text was written by bot.
Computer-generated text might look like it was written by a human, but a human writer is more likely to select unpredictable words. Using the “it takes one to know one” method, if the GLTR algorithm can predict the next word in a sentence, then it will assume that sentence has been written by a bot.
GPTZero
While you were having cocktails and watching the ball drop on New Year’s Eve, Edward Tian was busy creating GPTZero, an app that can help determine if text has been written by a human or a bot. As a 22-year-old senior at Princeton, Tian understands how college professors might have a vested interest in detecting “AIgiarism” (plagiarism, but with the help of AI.)
Tian says his tool measures randomness in sentences (“perplexity”) plus overall randomness (“burstiness”) to calculate the probability that the text was written by ChatGPT. Since tweeting about GPTZero on January 2, Tian says he’s already been approached by VCs wanting to invest and will be developing updated versions soon.
Watermarking (coming soon)
In addition to approaching the problem like plagiarism detectors have in the past, OpenAI is trying to address the problem by watermarking all ChatGPT text. According to a recent AI safety talk by OpenAI guest researcher Scott Aaronson, engineers have already built a working prototype that adds a watermark to any text created by OpenAI.
“Basically, whenever GPT generates some long text, we want there to be an otherwise unnoticeable secret signal in its choices of words, which you can use to prove later that, yes, this came from GPT,” says Aaronson. The tool would use what he calls a “cryptographic pseudorandom function.” And only OpenAI would have access to that key.
Recognize your brand’s excellence by applying to this year’s Brands That Matter Awards before the early-rate deadline, May 3.
