Decision makers rely on the Intelligence Community to provide accurate and relevant forecasts, and IARPA is working to identify methods to maximize the quality of these forecasts. IARPA’s Geopolitical Forecasting (GF) Challenge invites individuals or teams to develop innovative solutions and methods for integrating crowdsourced forecasts and other data into accurate, timely forecasts of geopolitical events.
Solvers will be competing against each other and in parallel with a similarly structured research program funded by IARPA. Overall, IARPA is seeking novel approaches that can help further current methodologies and improve the accuracy and timeliness of geopolitical forecasts. Over the course of seven months, the GF Challenge solvers will be asked to produce forecasts to a series of questions like:
- Who will win the upcoming presidential election in Egypt?
- What will the spot price of Brent Crude oil be on [date]?
In order to answer these questions, solvers will be given access to a continuously updated stream of forecast judgments produced by a crowd of human forecasters and will be allowed to use other data sources to produce their solutions. The challenge presents an opportunity for individuals and teams to earn prizes by creating methods that successfully forecast a wide variety of geopolitical events, such as political elections, international conflict, disease outbreaks, and macro-economic indicators.
Solvers will be invited to the GF Challenge platform to compete. Each solver or team will be assigned a unique application programming interface (API) key and login account for the platform. A steady stream of questions (roughly 25 per month) will be released for solvers to produce probabilistic forecasts against (specific requirements will be described in the rules and eligibility document). Questions may be binary (yes/no), multiple choice, or ordered outcome (binned quantity/date). In addition to the questions, solvers will gain access to a continuously updated stream of forecast judgments produced by a crowd of human forecasters. Solvers are encouraged to create solutions that use this stream of human judgments alongside other publicly available data streams and information to create their forecasting solutions.
- Questions will be released on the platform and can be pulled through the API
Each forecast question will include a set of exhaustive, and mutually exclusive response options, and will have a final resolution date that may range from a day to many months in the future. Challenge participants will be asked to produce forecasts until the question resolves.
Examples of potential questions*:
- Will the WHO confirm more than 10 cases of Marburg in 2018? (Yes/No)
Before March 2018 will South Korea file a World Trade Organization dispute against the United States related to solar panels? (Yes/No)
*Additional sample questions and criteria for forecast submissions will be available in the challenge rules.
- All forecasts will need to be submitted through the API
- Forecasts should be updated on a daily basis for each question that is open
Scores will be calculated using a metric which incorporates forecast accuracy and confidence. This metric is based on the Brier score, is a quadratic measure of the forecaster’s closeness to ground truth for the outcome that occurs. For example, an accurate forecast with 90% confidence for an event that occurs will score four times better than a forecast with 80% confidence.
As questions will range in duration, milestone check-ins for questions resolved during a certain period will be used to award progress-based incentives.
This means solvers who start after the challenge has launched are still able to compete for some of the prizes. Finalists who wish to compete for monetary prizes will need to provide the solution package for review by the GF Challenge Team at the end of the challenge, or as appropriate, if requested. Solutions will need to be documented and all data streams used in prize-winning solutions will need to be outlined and verified by the GF Challenge Team. A complete list of rules and eligibility requirements will be available soon.