Hot Off The Twitter Bot: How To Train Your Newspaper To Survive The Digital Age

By embracing a “digital first” approach, The Guardian has seen readership on its website shoot up by over 40% year over the past two years. Its latest feature, the Twitter-scaling search bot @GuardianTagBot, should only help. “It has fun and charm, but it’s also fantastically useful and structurally sound,” Janine Gibson, who leads the papers’ digital operations in the U.S., tells us.

The guardian and the observer


The New York Times recently reported a quarterly profit of $15.7 million driven, in part, by its success at drawing paying subscribers to its website. The news put some much-needed wind in the sails of an industry navigating the choppiest of waters. But to see a newspaper that may be innovating even faster and more creatively than the Times, and which may point the way toward a brighter future for old-school media outlets, one must gaze across the Atlantic at the London-based Guardian.

Ironically, the Guardian is taking the opposite approach: It’s making the open, pay-wall free approach work by being leaps ahead of everyone else in the digital space. And it’s paying off: Readership on is up by over 40% year over the past two years.

The newspaper’s latest quirky digital experiment is a Twitter-scaling search bot called @GuardianTagBot, launched in beta last week, which is already getting a warm reception. When someone tweets a few search terms (though just one at a time may work better) at the bot, it tweets back with a link to a list of the latest news stories on the Guardian‘s site, making use of the tagging system on the website, to quickly deliver relevant search results on command. “It has fun and charm, but its also fantastically useful and structurally sound,” Janine Gibson, who leads Guardian News & Media’s digital operations in the U.S., tells Fast Company. “It’s all the best things that our tech team does.”

The new service is a work in progress. The bot has a limit of 3,000 searches per day, and it can’t process full sentences. But it’s plum evidence of the Guardian‘s recent creative efforts to use the web to their best editorial and commercial advantage. “@GuardianTagBot is currently both an editorial, tech, and commercial experiment,” Nina Lovelace, Content Development Manager at the Guardian wrote in an email to Fast Company. “We built it to see if it would be popular, and provide a new service for Twitter-using Guardian fans.” 

To create TagBot, the Guardian‘s editorial and digital team worked with developers at Smesh, a social media monitoring company, to build “a kind of ‘Twitter bridge’ between Twitter and the Guardian‘s content API,” Lovelace explains. When queries come in via the Twitter stream, the Smesh software runs a semantic analysis, matching the terms up to tags on their content database.

At the bottom of the link list page that the tagbot sends back, users are asked to rate if it’s been a “goodtagbot” or “badtagbot.” Ratings from here are part of what Lovelace and the rest of the team will use to decide if the experiment should continue, and if it does, what direction it should take. “It’s already showing us how our tagging system could be improved to better suit users–but we’ll be collecting usage data over the next month and analyzing it to decide next steps,” Lovelace writes. 


Guardian News & Media officially announced it was going “digital first” in June this year, but that’s a way of thinking that’s been taking over for some years. Since its launch, the digital face of the Guardian has actively worked to intermesh itself with the rest of the organization–so much so that now “it is no longer the ‘other’ to our newspaper,” Gibson says. “We’ve always been at the forefront in terms of media orgs on the web… We were integrating and putting our website together before others, and this is just the next phase of that.”

Reaching out to readers across the Atlantic, the Guardian recently opened offices in New York, and launched a U.S. website: The New York offices will also lab-test their concept of a digital newsroom, in which animators, developers, and other digital wizards will rub elbows with the journalists on staff, together creating stories which fly on digital fairy dust. By April, Gibson says, 20% of the New York newsroom will be developers–one techie for every four members of the editorial staff.

All this is an extension of the Guardian’s existing comfort with creating stories that make use of the web. In fact, some of their stories couldn’t have existed without their alternative, interactive approach to reporting. When the WikiLeaks cables were released, the Guardian kept those that contained potentially sensitive information private. “We wanted to redact and interpret and publish responsibly as a news org, Gibson explains, “But we realized that people would have lots of questions that we wouldn’t think of or wouldn’t think were important.” So they had a facility by which people could write and comment and ask questions, which the journalists on the project would follow up on, and either write a story about it, or simply answer within the comments. “We got so many questions from academics emailing [us],” Gibson said. “It was the perfect thing to do.”

A more recent experiment is the Open News List. Here, Guardian editors share carefully selected sections of the upcoming news schedule on an Excel spreadsheet posted on their website. Readers can stop by, and tweet suggestions to the news team via #opennews, toppling the closed, secretive working model in traditional newsrooms.

“It feels like a state of mind,” Gibson says. “You let the spirit of it infuse the readers so that you develop a readership that’s active and engaged and wants to participate.”

The organization is also good about cultivating tech talent, and keeping developers involved and interested. Hack days are held a few times a year, when the Guardian offices are thrown open to developers, where they work and compete side by side with in-house developers to come up with creative tech projects that might work well on the Guardian website. Eventually, some of those creations thought up in these jam sessions do end up on the website.


On its Open Platform website, the Guardian declares its digital mission: “Our vision is to weave the Guardian into the fabric of the Internet, to become ‘of’ the Web rather than ‘on’ the Web.” In the 12 months that the Platform’s been up in beta, the Guardian says it has partnered with 2,000 developers to create applications that mine years of the Guardian‘s content in a friendly and usable way. 

With these changes in place, there’s plenty more we can expect from the Guardian in terms of how to make the best use of the web as a news space. “Why wait 5 years? Why not figure it out now?” Gibson says. “I’m baffled that no one else is doing it.” 

[Image: Flickr user Michael Bruntonspall]

Nidhi Subbaraman writes about technology and science. Follow on Twitter, Google+.