Como Is Infosec
Content moderation is a security problem.
--
in·fo·sec (/ˈinfōˌsek/): information security
co·mo (/koh-moh/): content moderation
Content moderation is really, really hard.
Content moderators:
- seek to set conversational norms, steering transgressors toward resources that help them better understand the local conversational rules;
- respond to complaints from users about uncivil, illegal, deceptive or threatening posts;
- flag or delete material that crosses some boundary (for example, deleting posts that dox other users, or flagging posts with content warning for adult material or other topics);
- elevate or feature material that is exemplary of community values;
- adjudicate disputes about impersonation, harassment and other misconduct.
This is by no means a complete list!
There are many things that make content moderation hard. For starters, there are a lot of instances in which there is no right answer. A user may sincerely not intend to harass another user, who might, in turn, sincerely feel themself to be harassed. Deciding which of these is the case is an art, not a science.
But there is an another side to como, one that’s far more adversarial: detecting, interdicting and undoing deliberate attempts to circumvent the limitations imposed by content moderation policies. Some examples:
- Using alternative spellings (e.g. “phuck” or “5hit”) to bypass filters intended to block profanity, racial slurs, or other language prohibited within a community or discussion;
- Using typographical conventions to circumvent prohibitions on slurs (for example, using (((multiple parentheses))) to express antisemitism);
- Using euphemisms to express racist views;
- Degrading or altering video or audio files to bypass content filters;
- Using automated or semi-automated sock-puppets to disguise coordination and make organized harassment campaigns look like spontaneous community disapprobation;
Some of these tactics are social, others are technical, and many straddle the boundary…
