Frances Zlotnick

GitHub

Using Anomaly Detection on User Demographic Distributions to Identify Fake Account Bursts (pdf, video)

Mass generation of fake accounts for malicious purposes is a problem that faces many online platforms. Identifying and removing such accounts is an increasingly high priority for security and integrity teams in commercial, governmental, and other contexts, as prevalent misrepresentation on a platform degrades user trust, injects uncertainty into performance and business metrics, and presents opportunities for serious security incidents. Malicious users generating such accounts often go to great lengths to make such accounts appear legitimate, by adding plausible names, photos scraped from other websites, and other details to fake account profiles.

This habit presents an opportunity for automated detection. Names—to a greater or lesser extent depending on cultural context and language—encode demographic attributes such as gender, the distribution of which can be monitored among legitimate users. Bad actors rarely have sufficient knowledge of a platform's user base to accurately mimic these expected distributions. Sharp departures from known distributions can be used to identify bursts of fake account generation for closer inspection. We present empirical examples using data from our work detecting malicious users.

While potentially useful, use of such methodology sits within a minefield of technical and, most importantly, ethical challenges. We discuss a number of these, including the challenges of detecting gender across cultural contexts, and the inherent dangers of using gender-related features to identify potential bad actors. Particularly in contexts where women are already severely underrepresented, false-positives among this cohort might have the effect of further discouraging participation, running counter to goals of increasing diversity, inclusion, and belonging.