Spotting fake text: AI knows machine-written text when it sees one

• Author:
• Author name

  Quantumrun Foresight
• August 2, 2022

Insight summary

Artificial intelligence (AI) is both creating and fighting the spread of fake news, impacting everything from elections to public health information. While AI can now detect and delete fake news, its creators are constantly adapting, making fake content more believable and harder to spot. This evolving battle between AI-generated disinformation and detection tools has significant implications for society, including the potential for biases and the need for human oversight in identifying false content.

Spotting fake text context

There is a growing danger of propagandists (across the political spectrum) using AI systems to auto-generate convincing text-based social media posts and long-form articles designed to sway public opinion toward extreme positions. Fortunately, new AI systems are employing advanced algorithms to spot fake digital media before they spread their disinformation. 

Competing global actors are exploiting AI software to either write fake news or to detect it. The detection process involves analyzing the text of a message to determine how likely it is that a particular message is a real communication from an actual person or a mass-distributed piece of propaganda. A machine learning (ML) algorithm can also evaluate how well a post’s text, or a headline, compares with the content of an article someone shares online. Another method could examine related articles to see whether other news media have contradicting facts. 

Similar technologies can distinguish fake news sources and identify specific accounts that produce them. However, those techniques assume that people who spread false information do not alter their behavior. In reality, they frequently modify their methods to disguise the content of phony posts to make them appear more genuine. 

Fortunately, the main breakthrough powering modern fake-spotting systems is the Giant Language Model Test Room (GLTR) tool. The GLTR was developed by researchers from Harvard University and the MIT-IBM Watson AI Lab and can identify statistical patterns in text that are not typically found in human-written content. As misinformation becomes increasingly automated, tools utilizing GLTR may become valuable weapons in the fight against fake news, deepfakes, and Twitter and Instagram bots. 

Disruptive impact

Companies that employ AI exclusively to spot fakes (particularly social media companies) may reveal (and worsen) societal prejudices by how they evaluate data, especially in the context of said data. The meaning of words can vary over time. And the same word might have different meanings on left-wing and right-wing websites. For example, while a post with certain terms on a more liberal website may be regarded as newsworthy, those same terms may instead refer to a specific set of conspiracy theories on a conservative site. The outcomes of these contextual misunderstandings may lead to increased prejudices based on gender, race, or geography. And depending on the political party in power, these misunderstandings may lead to increased restrictions on the free expression of specific viewpoints.

Likewise, as synthetic text becomes easier to produce, it presents new challenges for detecting fraudulent content. Artificial text can create convincing disinformation that is easy to tweak and spread. This trait could significantly impact the information ecosystem as machine-generated content becomes increasingly common. Thus, human intelligence and intervention are still vital to help AI systems combat how they are being used to generate fake content.

Long term, GLTR initiatives aim to create human and AI collaboration systems. In other words, the tool is designed to help humans work together with AI to identify fake text. To test this collaboration, the researchers asked Harvard students to identify AI-generated text—both with and without the help of GLTR. The results showed that the students could identify 72 percent of all fakes when using the tool, as opposed to only 50 percent when they were working independently.

Implications of spotting fake AI text

Wider implications of spotting fake AI-generated text may include: 

• AI systems increasingly flagging and deleting fake news during critical events like elections and global health crises, enhancing public information reliability and reducing misinformation spread.


• Machine learning identifying various fake news creation methods, leading to search engines and social media platforms auto-suppressing such content, thus improving online information quality.


• Governments investing in fraudulent content detection technology to protect national security, simultaneously enabling rogue nations to use similar tech to create discord in adversarial states.


• Financial institutions employing GLTR-powered AI to intercept phishing in email and SMS, significantly reducing financial fraud and enhancing consumer protection.


• AI systems learning from low-quality fake news to produce more convincing content, requiring stricter regulations to combat an intensifying misinformation crisis.


• Education systems incorporating fake text detection tools, equipping students with critical thinking skills essential for discerning accurate information.


• Enhanced fake text detection contributing to a decline in the effectiveness of online propaganda, promoting more authentic and transparent public discourse.

Questions to consider

• How might your job or company benefit from having tools that can detect fake news or misinformation?


• How else do you think AI can both improve and worsen the misinformation crisis?