AI attacks destroy content filters. That increases misinformation and terrorist recruitment.

Content filters are society’s digital immune systems. By removing foreign assets that are dangerous, illegal, or against the terms-of-service of a particular application, they keep platforms healthy and root out infections. Content filters are also uniquely qualified to police content at the scale the Internet requires. The content uploaded to the Internet each minute is a staggering amount growing at a staggering rate. Over three billion images are shared every day on the Internet.28 AI-based content filters have emerged as the primary, if not only, tool able to operate at this scale, and have been widely adopted by industry. For example, Facebook removed 21 million pieces of lewd content in the first quarter of 2018 alone, 96% of which was flagged by these algorithms.29 Content filtering has taken on increased urgency in the past years. ISIS successfully used social media as one of its main recruitment avenues.30 Nationalists in Myanmar used Facebook as its mouthpiece to incite a campaign of Rohingyan genocide.31 Misinformation campaigns deploying fake content on social networks have been used to influence democratic elections in the U.S. and Europe.32 As this content successfully weaponizes US-based platforms, the efficacy of AI-based content filters has broad-ranging implications, including the defense of both national security and oppressed populations. Beyond these newest matters, content filters must continue to be effective with important tasks already within their purview, such as the detection of child pornography. Perhaps the only concept that could make strange bedfellows of the Americans, Russians, Chinese, and Iranians, child pornography was universally accepted as a target of censorship even from the early days of the Internet. AI-based content filters allow website and platform operators to efficiently and effectively scan the millions of images uploaded each minute for illicit content, and immediately destroy offending images. In addition to custom tools companies built for their own use, this detection software was eventually provided via the Software as a Service (SaaS) distribution model, where one large company like Microsoft offered an API-based content filter that website owners could use instead of building their own.33 Even in more banal uses, content filters are tied to many business models. As advertisers begin to be held responsible in the court of public opinion for the content appearing next to their advertisements, there is a growing need to detect an increasing number of objectionable content types. This extends to detection of nudity, violence, hate crimes, weapons, adult pornography, profanity, and inappropriate comments. YouTube faced the boycott of advertisers including AT&T, Disney, Hasbro, and Nestle for failing to effectively filter sexual comments left by viewers on videos in which children appeared.34 As content filters are drafted into these battles, there will be strong incentives both to attack them and to generate tools making these attacks easier to execute. Adversaries have already seen the power of using digital platforms in pursuit of their mission. ISIS organically grew an international following and successfully executed a large-scale recruitment program using social media. These are successes that, morals aside, may have evoked jealousy from the marketing departments of Fortune 500 companies. Future organizations of malice are likely to follow the same playbook. If confronted with better content filters, they are likely to be the first adopters of AI attacks against these filters. In an environment with AI attacks, content filters cannot be trusted to perform their job. Because content filters are now being used as the first and, in many respects, only line of defense against terrorism, extremism, and political attack on the Internet, important parts of society would be left defenseless in the face of successful AI attacks. These attacks give adversaries free reign to employ these platforms with abandon, and leave these societal platforms unprotected when protection is needed more than ever. Further, it will be difficult to stop or even detect these attacks on content filters because they will likely go wholly unnoticed. Because content filtering is applied to digital assets, it is particularly well suited to the “imperceivable” input attacks. Further, unlike many other cyberattacks in which a large-scale theft of information or system shutdown makes detection evident, attacks on content filters will not set off any alarms. The content will simply fall through the filter unnoticed. In this respect, entities such as social networks may not even know they are under attack until it is too late, a situation echoing the 2016 U.S. presidential election misinformation campaigns. As a result, as is discussed in the policy response section, content-centric site operators must take proactive steps to protect against, audit for, and respond to these attacks.