The EMBERS Project Can Predict the Future With Twitter


Since its inception in April 2012, an average of 80 to 90 percent of the forecasts it generates have turned out to be accurate—and they arrive an average of seven days in advance of the predicted event. EMBERS (short for Early Model Based Event Recognition using Surrogates) derives its intelligence from what data geeks call “open-source indicators”—social media, satellite imagery and more than 200,000 blogs that are publicly available. It mines up to 2,000 messages a second and purchases open-source data such as Twitter’s “firehose,” which streams hundreds of millions of real-time tweets a day.

While much has been made of the government’s secret surveillance operations—particularly those that spy on Americans—the EMBERS project is focused on tracking human behavior overseas and publishing its findings, even if negative. “We are not looking at anything classified and we aren’t forecasting terrorism, because we don’t have access to those kinds of back channels,” Ramakrishnan says. “We are looking at data anyone can get.”...

EMBERS was the product of a 2012 contest organized by Jason Matheny, an associate director of the government’s Office for Anticipating Surprise (yes, that’s the name of a real office) and a program manager at the Office of the Director of National Intelligence’s [Intelligence] Advanced Projects Research Activity program. Three teams—from Virginia Tech, quantum computing firm Raytheon BBN Technologies in Cambridge, Massachusetts, and HRL in Malibu, California, formerly Hughes Research Laboratories—were asked to build the best possible forecasting model based on open-source indicators. The most successful of these was EMBERS, which ended up integrating several members of the other teams into its own, including Raytheon BBN, which now builds some of EMBERS's social media models, like the ones trying to forecast civil unrest from reading Twitter feeds. Some of the guiding principles of the research, says Scott Miller, senior technical director of Raytheon BBN’s speech and language group, are astoundingly simple.