In the wake of Black Lives Matter protests spreading across the nation, Facebook content moderators are revolting over the way the company and its leadership are handling political speech that incites violence, specifically posts from U.S. President Donald Trump. Yet this struggle represents only the latest episode in the long-running saga moderators have faced as they have tried to combat the storm of online toxicity that has plagued the internet.
Following years of allegations that Facebook had failed to protect its moderators from the traumatic effects of viewing violent and harmful content, the social media giant finally agreed in mid-May to a $52 million settlement to compensate for content moderators’ mental health struggles. But this has done little to help moderators still on the job.
It’s clear that content moderation as we’ve long known it is not working as it should. The vast majority of the internet may be “clean”—yet moderators have struggled to keep up with the proliferation of toxic content, such as racist hate speech and images of sexual violence, in recent years. Not only that, the leaders of popular online platforms have too often failed to adequately support moderators facing such emotionally overwhelming content.
The disadvantages of the status quo run deeper. Human moderators are different from one another—two different moderators might make two different decisions on the same piece of content, complicating standardized procedures. It is naive to pretend that constant exposure to harmful content on the job doesn’t further impact moderators’ judgment and their ability to discern the wheat from the chaff. Over time, moderators become desensitized and exhausted, making them more prone to errors that allow toxic content to spread like wildfire. A moderator may approach the same image with a different mindset after two hours of browsing hateful materials as opposed to at the start of a shift.
Given these pitfalls, platforms like Facebook have increasingly turned to AI-based tools to filter out damaging content. One prominent method is identifying potentially toxic content based on keyword detection in texts. But this is far from a foolproof approach: False positives are common since the algorithm usually focuses on the context of a sole sentence. That means it misses a great deal of truly harmful content, so the heavy lifting is still left to human professionals. Simply put, traditional AI hasn’t solved the problems of scale or traumatic exposure it was supposed to address.
If moderators so frequently end up overburdened and standard AI has its own limitations, how can online platforms maximize the efficiency of their process to filter out harmful content while still pinpointing where human moderators can best intervene? How do online platforms lessen moderators’ fatigue and minimize their exposure to explicit, often traumatizing, material? The answer lies in contextual AI. Whereas traditional AI flags words, the contextual approach can take the entire history of an interaction into account.
Looking at online conversations in context
While traditional moderation tools often erroneously flag innocuous content as harmful while allowing toxic content to pass through the filter, contextual AI relies on multidisciplinary methods such as psychology and behavioral science to predict toxic content and filter it out before it causes significant harm. Rather than inserting a range of keywords to flag for detection, contextual AI solutions are trained to distinguish the harmless from the harmful by distilling the nuances of online speech.
Contextual AI’s algorithms mimic human behavior. Instead of being trained only on text, like more traditional solutions, contextual AI gets smart with the help of images, photos, videos, and more, making it possible to predict with greater accuracy whether a conversation is likely to lead to a toxic outcome.
Imagine two friends going back and forth about the COVID-19 pandemic. Asked how he’s dealing with the crisis, one friend replies that he’s “angry,” blaming “the Chinese” for engineering the novel coronavirus to tank the economy. He then approvingly shares an image from an anti-lockdown protest featuring neo-Nazi iconography. Contextual analysis is key here as it can cover online interactions lasting weeks or even months—which is essential because toxic relationships usually develop over the long term. Given the negative emotional sentiment expressed, the singling out of a racial group, and the peddling of hateful rhetoric, a contextual AI system would be able to discern that this was almost certainly a case of racial hate speech—even before the Nazi imagery emerged.
Or consider a chat room conversation in which a pedophile is “grooming” a young girl. A contextual AI solution would go beyond scanning for lewd and predatory words and phrases. In most cases of grooming, stand-alone messages wouldn’t necessarily appear problematic. That’s why contextual AI is necessary to analyze whether the two have expressed an intent to meet in real life, look at context clues to confirm that one of the parties is likely underage, and gauge the likelihood of predatory intent.
Making moderators’ jobs easier
Sophisticated AI can dramatically reduce strained moderators’ workloads by intelligently filtering out open-and-shut cases of harmful content—mitigating moderators’ exposure to traumatizing words and images.
But policing toxic content is not possible with AI alone. When a photo, post, or social media comment may cross a legal line, for instance, a human professional will ultimately need to make the call as to whether the content is problematic and should be taken down. Content moderators are also needed to determine what criteria the AI should look for based on observed patterns in online speech.
So far, tech giants have been reluctant to invest in and deploy the most advanced methods—exposing millions of ordinary users and moderators alike to harmful content. Detecting and eradicating toxicity simply isn’t the field in which these online platforms expend most of their time, technology, or money. The ongoing debate about Section 230, the statute that protects platform owners from liability for what their users post in the United States, and the introduction of more comprehensive anti-hate speech legislation in Europe will hopefully spur leading platforms into taking action and seeking more contextual-based solutions to proactively tackle issues of toxicity.
Content moderators have an essential role to play in making the internet safer for everyone—but that shouldn’t come at the expense of their own safety. Online platforms must rise to the occasion to keep their workers protected.
Zohar Levkovitz is the cofounder and CEO of L1ght.