Mechanical Turk's Unsavory Side Effect: Massive Spam Generation [Updated, Amazon Comments]

Mechanical Turk

Amazon's Mechanical Turk human computing system is a handy way to get repetitive tasks done cheaply, quickly, and in ways that code can't compete with. Essentially it's a system where users get paid small sums for digital piece-work, doing repetitive tasks that it would be impossible or too expensive or time-consuming to run through code. But is the Turk being used to perpetrate massive amounts of spam? New research suggests yes.

Panos Ipeirotis, an associate professor at the Stern School of Business of New York University, has been working with colleagues to investigate a problem he noticed cropping up over and over again in the infrastructure of Amazon's Mechanical Turk system: How much of Turk activity is being used to generate spam? By polling trusted Turk workers on an array of new Human Intelligence Tasks (HITs—the Turk activities that users work on, for cash) from new requesters who joined between September and October 2010, Ipeirotis could get a feel for how many HITs on the system were being generated as part of spamming activity.

The results are surprising: Nearly 41% of HITs from these new requesters were regarded as being related to spam. Ipeirotis was concerned the data was being skewed, so he tested to see if spam was being generated more by some folks than others, and discovered that nearly 32% of new requesters were using Mechanical Turk to only generate spam, and nothing else. Even more interestingly, spam requesters on MT seem to offer more pay-per-activity than non-spammers—although Ipeirotis was denied access to deeper data on the matter to determine if spammers ended up paying out or not.

What kinds of spam activity did this research uncover? It's everything from giving "likes" on Facebook, to giving fake votes, Diggs, and reviews to products and services online. Then there're HITs for generating fake Twitter and Facebook accounts to send on spam, and HITs for clicking on ads on specific websites, to drive up web traffic to the advertisers account. Of these, the Facebook and Twitter activities are the most easily identifiable as actually being "spam"—we've all received one or two weird @ comments on Twitter, for example, directing us to a particular shortened web link with a teaser text of some kind. But the voting and "liking" is equally spammy behavior if it biases the review value of a product or service that's available online away from its true crowd-sourced figure. In essence, spammers are using Mechanical Turk's crowd sourcing powers to develop spam and distort other, genuine, crowd sourced actions online.

Quora, a new question/answer online service, found itself in the spotlight recently for using MT in a way that seemed very spammy: Quora was offering HITs for $0.15 per task for mass-generating Twitter accounts, a process that involves MT users solving CAPTCHA security codes which Twitter's API imposes to prevent mass spam accounts being set up. Quora defended its actions by noting it's using Twitter as a promotional/news dissemination tool instead of traditional RSS feeds, but it does seem a gentle abuse, or at least circumvention, of Twitter's system.

Other MT spammers are, obviously, far less scrupulous, and are essentially gaming MT's legion of dedicated digital piece-workers to generate lucrative spam. And that begs one big question: Why doesn't Amazon stamp on this sort of behavior? You would think it was obliged to, since in many places spamming is an illegal activity. Is it because Amazon's actually profiting from the spammers? Amazon isn't saying, and it seems like it doesn't want to help Ipeirotis to find out. We've quizzed Amazon on the matter, but haven't received a reply by the time of publication.

There is one argument that this form of spamming may not exactly be a bad thing, either—some social commenters have proposed that by building a paid-for framework for spam, the very worst sort of spam may become de-incentivised. But whichever way you look at it, if you combine this research with news that spam traffic surged to 77.4% of all email traffic in October, one conclusion is inescapable, and maybe it's one of those ephemeral Internet Laws: "Spam expands to maximize the potential of every digital system available."

Update: Amazon got back to us on this matter. A spokesman noted that "we monitor our marketplace closely and our data indicates that the majority of the work [on Mechanical Turk] is valid," before noting that "the Amazon Mechanical Turk PA outlines our terms of service for Requesters including the type of work considered invalid," meaning they have clear guidelines. "When we find invalid work in the system, we work quickly to remove those HITs," and the offending HITs may include:

  • Require disclosure of the Worker's identity or e-mail address, either directly or indirectly 
  • Require registration at another website or group  
  • Directly or indirectly promoting a site, service, or opinion
  • Violate the terms and conditions of an activity or website (for instance asking Workers to vote for something
  • Have explicit or offensive content, such as nudity, without the Adult Content Qualification
  • Ask workers to solicit third parties § Generate "referred" site visits or click through traffic 
  • Ask workers to take action to manipulate a search engine's relationship data
  • Violate copyrights

Amazon's contact also noted that there's a reliable mechanism for reporting invalid HITs and the company's staff "monitor and analyze the work in Mechanical Turk at a significant level of granularity." Their data on abusive HITs indicate that "This [reported] analysis is not only inaccurate but also off by at least an order of magnitude." Which would indicate that around 4% of HITs are invalid, rather than 40%—is that still enough to generate significant amounts of spam, though?

To read more news on this, and similar stuff, keep up with my updates by following me, Kit Eaton, on Twitter.

Add New Comment

4 Comments

  • Stephanie Costello

    I work full time (and by full time I mean as much as 12 hours per day) on MT and I can tell you that although there is some spam, it isn't even close to 40%. Workers also have the option to report the offending HITS and I do so on a daily basis because I want a productive, safe work environment. I also regularly accept those HITS and then just let them "time out" so that neither do they get done, nor does the person who posted it get the information they hoped to. It doesn't affect my rating and it frustrates those who put that crap on the site. 

  • Todd

    Almost nothing. it;s chump change and a min wage job would be 10x more money than this. I would know

  • Kyle Hawke

    This is an excellent article. Well researched and data-driven.

    Does this represent the down-fall of the super-low-end, task-based crowdsourcing market?

    Don't worry... this comment was not generated by a Turk worker. :)