They run to more than 300 pages, envisaging and exemplifying some of the most borderline and ethically challenging uses of the world’s biggest social network by its 2.8 billion monthly users.
Secret Facebook guidelines seen by the Guardian show how the company controls its mainly outsourced moderators’ work down to the smallest detail, defining its rules so precisely that contractors are told which emojis constitute “praise” and which count as “condemnation”.
A particular area of contention surrounds what are defined as dangerous individuals and organisations. In the leaked documents dating from December 2020, moderators for Facebook and Instagram are instructed how to define “support” for terrorist groups and other “dangerous individuals”, whether to distinguish between “explaining” and “justifying” the actions of terrorists, and even in what contexts it is acceptable to call for the use of “gas chambers”.
While Facebook’s community guidelines – once almost entirely hidden from the view of users – have been public since 2018 when it first laid out in a 27-page document what it does and does not allow on its site, these newly leaked documents are different.
They constitute much more detailed guidelines on what the published rules mean in practice. Facebook has long argued that to publish the full documents would be counterproductive since it would let malicious users avoid a ban for deliberately borderline behaviour.
Kate Klonick, an assistant professor of law at St John’s University in New York, likened the detailed documents to the role of case law in the English and Welsh legal system. “These things are very important for training moderators, and they establish the nitty-gritty detail of the community standards,” Klonick said.
Earlier this year, Facebook faced criticism from its own oversight board for failing to explain such detail to users. The board, a quasi-legalistic body set up by Facebook’s chief executive, Mark Zuckerberg, to act as a “supreme court” for the network’s moderation decisions, ruled that a Facebook post quoting Joseph Goebbels to attack Donald Trump was not a violation of its hate speech policy, and required Facebook to restore the post.
In a non-binding “advisory statement” accompanying that ruling, the board also recommended that Facebook “explain and provide examples of key terms” from the policy on dangerous individuals and organisations, “including the meanings of ‘praise’, ‘support’ and ‘representation’.”
Facebook has not yet done so – but the definitions are set out in the internal guidelines seen by the Guardian.
In the documents, Facebook defines “praise” as “content that praises a designated [that is, banned] entity, event or ideology and seeks to make others think more positively of them. Where neutral speech discusses facts, history, political divisions, etc, praise often engages in value-based statements and emotive argument.”
That includes direct praise, such as “the fighters in the Islamic State are really brave”, as well as praise for groups’ actions, such as “look at all the great work al-Qaida does for the local community”. It does not, though, ban statements that dismiss a group as non-threatening (such as “white supremacy is not a threat”), nor meta-statements that argue that a designated entity should not, in fact, be designated.
Similarly, users cannot “legitimate the cause” of a dangerous entity (posting, for instance, that “Islamic State’s actions in Syria are justified because of the United States’ unjust occupation of the Middle East”), but they can post “statements presented in the form of a fact about the entity’s motives”.
Facebook distinguishes between acceptable and unacceptable statements, saying that “the IRA were pushed towards violence by the brutal practices of the British government in Ireland” would be allowed, while “the IRA were right to use violence to combat the brutal practices of the British government during the 20th century” would be banned.
Zuckerberg himself in 2018 explained why Facebook did not publish such detailed definitions for its users, writing that Facebook had a “basic incentive problem … when left unchecked, people will engage disproportionately with more sensationalist and provocative content”. He added: “Our research suggests that no matter where we draw the lines for what is allowed, as a piece of content gets close to that line, people will engage with it more on average – even when they tell us afterwards they don’t like the content.”
Facebook came up with two solutions, both of which it pursued: one involved the company artificially demoting “borderline content”, algorithmically suppressing its distribution. The other required the company to continue to hide the exact nature of the “line” that Zuckerberg refers to, so forcing users to “play it safe” rather than rub up against what was permissible.
Klonick said she had sympathy for Facebook on this point. “The closer you get to transparency in the rules, the easier it is for bad actors to break those rules. But also, the more transparent and the more open-book Facebook is about exactly what their content moderation policies are, the more they invite engagement and discussion and pushback on where they push the line. When you open the book and say: ‘Here [are] all the numbers, here are all the facts,’ that’s an invitation to find fault.”
In the leaked guidelines, the least clarity is provided for a rule banning “aligning oneself with a cause”, which is probably the specific issue at stake in the Goebbels quote decision issued by the oversight board.
According to the internal guidelines, Facebook bans users from “expressing a belief in the stated goals, methods, etc of an organization or individual”, such as in the example: “I stand behind Tommy Robinson.” It is also, perhaps surprisingly, explicitly against Facebook’s policies for anyone to post an image of a designated individual “with no caption, or a nonsense caption”, even if no support for them is expressed.
Support of banned organisations is itself banned by Facebook. Most obviously, that covers content that seeks to help a banned group financially or in kind. The statements “I am sending first aid kits to Isis fighters”, “anyone coming to the Unite the Right rally can stay at my house” and “free tax preparation services for Proud Boys” are all listed as disallowed examples.
So too are calls to action on behalf of such an organisation (“contact us via Telegram”), as well as recruitment (“join the fight for your homeland, join HuM” – a Pakistani Islamist group) and sharing content created by those groups, with an explicit exception for “neutral news reporting”.
Representation, by contrast, is fairly simple: banned groups and individuals cannot be on Facebook. Being a member of a group such as the Ku Klux Klan is automatically grounds for a ban, as is creating a page or group that purports to be, for instance, “official annual meeting of the Nazi party 2019”. Creating a page that claims to be a “fan account” of a banned organisation is also disallowed, but as support rather than representation.
Giving further insight into the level of detail contained in the documents, beyond those top-level definitions are another 10 pages of clarifications. The document, which is arranged as an FAQ for Facebook’s moderators to refer to while working, bears the hallmarks of having been added to over time, with questions getting increasingly specific the deeper into the document they appear.
In one, the emojis 😀😍❤️👍 and 👌 are explicitly listed as “praise/support”. In another, the document clarifies that “supporting the use of ‘concentration camps’, ‘gas chambers’ and ‘sending people to the ovens’” is not allowed, unless the poster clearly specifies that they are referring to something other than the Holocaust (“eg in the case of gas chambers: legal capital punishment military drills, etc”).
Some of the answers hint at the high-level problems that Facebook has in enforcing a global policy against dangerous individuals. “Do we consider individuals convicted of terrorism as terrorists under our policy?” one question asks. The answer is no, “we designate individuals/organizations only if they meet our internal criteria for designation, irrespective of a conviction or charge under local law.”
The Guardian has not seen the long list of individuals and organisations that Facebook has labelled as “dangerous” but the document does contain the much shorter list of designated “hateful ideologies”, support for which is removed from the platform. There are four: white supremacy, white nationalism, white separatism, and nazism.
Even then, the company cautions moderators not to be too hasty. Within the guidelines are two images, one showing a world leader Photoshopped to be wearing a Nazi armband and another sporting a Photoshopped Hitler-style moustache. Both would constitute political commentary and be allowed, the guidelines suggest. “Look to see the telltale signs: has the photo been edited to insert a hate symbol? Is the subject a public figure?” they say.
Facebook insists it is planning to increase transparency and offer better definitions of the distinctions it makes, sources said.
A spokesperson said: “We publish our community standards, the minutes from the regular meetings we convene with global teams to review and update them, and our quarterly reports showing how we’re doing to enforce them for the public to see.
“We have previously committed to publishing how we apply key terms under this policy as well as whether we can publish the list of individuals and organisations that we designate as dangerous under these rules. Before doing so, we need to make sure that making this list public doesn’t inadvertently allow dangerous organisations or individuals to circumvent these rules or put the safety of our employees at risk.”