Trying to tell the difference between AI-generated and human-generated content will be one of the more difficult challenges of the coming years. We’ve already seen deepfake videos that are nearly indistinguishable from reality. Now people are using OpenAI’s ChatGPT AI to write conversational text that could be mistaken for human writing. Thankfully, a recent study suggests that there is at least one way to tell the difference between ChatGPT and human writing.
Last week, researchers from the University of Applied Sciences and Arts of Southern Switzerland (SUPSI) published a study about a machine learning model that could detect ChatGPT-generated text (via ZDNet). The researchers tasked ChatGPT with generating restaurant reviews on its own and rephrasing human-written restaurant reviews.
Based on the test, the researchers found that the key difference between a ChatGPT review and a human’s review was whether or not feelings were expressed. ChatGPT frequently describes the experiences but fails to share how it reacted to those experiences. Meanwhile, as you might know from reading Yelp reviews, humans are more than happy to let others know exactly how a dining experience made them feel, often in great detail.
Other key signs that text was generated by AI included a disinclination to use personal pronouns, the appearance of unusual words, and a lack of any aggressive or rude language. Once again, if it’s not rude, is it really a user-generated restaurant review?
“Looking into SHAP (SHapley Additive exPlanations) explanations of the predictions gives some insights about ChatGPT writing style,” the researchers conclude. “It is extremely polite, aiming to please different types of requests from various domains fairly well mimicking humans, but that still does not have the profoundness of human language (e.g. irony, metaphors,…).”
While the machine learning model was successful at distinguishing between AI and human text, it’s worth noting that OpenAI’s language models are only going to get smarter. ChatGPT currently runs on GPT-3.5, but GPT-4 is already in the works. As these language models grow and evolve, their ability to mimic human writing will undoubtedly improve.
