The Problem

Decision makers rely on the Intelligence Community to provide accurate and relevant forecasts, and IARPA is working to identify methods to maximize the quality of these forecasts. The Geopolitical Forecasting (GF) Challenge invites Solvers from around the world to develop innovative solutions and methods for integrating crowdsourced forecasts and other data into accurate, timely forecasts on worldwide issues.

GF Challenge Solvers will develop solutions that produce probabilistic forecasts in response to numerous closed-ended forecasting questions that concern specific, objectively verifiable geopolitical events containing timeframes with deadlines and locations. The effort will run in parallel to IARPA’s geopolitical forecasting research program - Hybrid Forecasting Competition (HFC). Challenge Solvers will be competing on a largely overlapping set of Individual Forecasting Problems (IFPs) as HFC research teams and given access to the same human forecaster data stream. In addition to the provided data stream, Solvers may use other data streams and their own data and models for the challenge.

Over the course of seven months, the GF Challenge solvers will be asked to develop solutions capable of producing forecasts to a series of questions like:

  •  
  • Who will win the upcoming presidential election in Mexico?
  • What will the spot price of Brent Crude oil be on [date]?

The challenge presents an opportunity for individuals and teams to earn prizes by creating methods that can successfully forecast a wide variety of geopolitical events, such as political elections, disease outbreaks, and macro-economic indicators.

Challenge Details

Overview

Participating in the challenge is a two-step process. Solvers will register on the HeroX GF Challenge page and acknowledge their understanding and intent to follow the rules of the challenge. Following registration, instructions will be sent to access the application programming interface (API) within the Cultivate Labs platform. Solvers will be granted an API token and documentation for interacting with the challenge platform. There is only one API account allowed per Solver or Team (a single representative from each Team will need to setup the Team’s API account, see the Eligibility section within the Challenge Rules Document for details.)

Forecasting questions, referred to as IFPs, will be released periodically and will be open for a specified period of time. Solvers should check daily for new IFPs. While an IFP is open, Solvers may update their forecasts daily until the IFP closes. Each Solver will be permitted to submit forecasts using up to 25 different methodological “slots.” Once the IFP closes, the forecasts will be locked and scores for the IFP for each method will be calculated. Scores will be tracked on the leaderboard within the Cultivate Labs platform. For additional details on scoring and registration, please see the Challenge Rules Document.

A steady stream of IFPs (roughly 25 per month) will be released for Solvers to produce probabilistic forecasts against. IFPs may be binary (yes/no), multiple choice, or ordered outcome (binned quantity/date). In addition to the IFPs, Solvers will gain access to a continuously updated human forecaster data stream produced by the HFC research team.

  • IFPs will be released via the API
  • Each IFP will include a set of exhaustive, and mutually exclusive response options, and will have a final resolution date that may range from a few days to a few months in the future. Solvers will be asked to produce forecasts until the IFP resolves
  • Examples of potential IFPs*:
    1. Will the WHO confirm more than 10 cases of Marburg in 2018? (Yes/No)
    2. Before March 2018 will South Korea file a World Trade Organization dispute against the United States related to solar panels? (Yes/No)
  • Solver’s forecasts need to be submitted through the API
  • Solvers should update their forecasts on a daily basis

Scoring

Scores will be calculated using a metric which incorporates forecast accuracy and confidence. This metric is based on the Brier score, is a quadratic measure of the forecaster’s closeness to ground truth for the outcome that occurs. For example, an accurate forecast with 90% confidence for an event that occurs will score four times better than a forecast with 80% confidence.

As IFPs will range in duration, milestone check-ins for IFPs resolved during a certain period will be used to award progress-based incentives.

Registrations will be accepted on a rolling basis, so Solvers may register at any point. However, Solvers will be rewarded by accuracy scores that incentivize participation on their ability to forecast on more IFPs. Eligibility for prizes have minimum participation requirements for the resolved IFPs included for each prize. Solutions will need to be documented and all data streams used in prize-winning solutions will need to be outlined and verified by the GF Challenge Team. Read the Judging and Scoring Criteria section in the Challenge Rules Document for more detailed information.

Congratulations to the challenge winners.

Overall Winners

  • First place — DigitalDelphi (Johns Hopkins University Applied Physics Lab)
    • Anna L. Buczak (Lead)
    • Daniel Berman
    • Jared Zook
    • Kayla Scharfstein (Summer intern from Brown University)

  • Second place — Steven Erly and Brian Erly

  • Third place — Lars Ericson, Catskills Research Company

  • Fourth place — truthLover

  • Fifth place — SISLers
    • Jonathan Cox, NIO
    • Maxim Egorov, A^3 by Airbus
    • Yegor Tkachenko, Columbia Business School

  • Star Forecaster — Steven Erly and Brian Erly

Additional Prizes

  • Election Forecaster — DigitalDelphi (Johns Hopkins University Applied Physics Lab)
    • Anna L. Buczak (Lead)
    • Daniel Berman
    • Jared Zook
    • Kayla Scharfstein (Summer intern from Brown University)

    Domain Region

  • Politics/International Relations & Middle East — truthLover

  • Geopolitical Grab Bag & Asia — Lars Ericson, Catskills Research Company

  • Macroeconomics/Finance & Europe — LIA (EPFL)
    • Boi Faltings (Lead)
    • Naman Goel
    • Diego Antognini
    • Jean-Thomas Furrer
    • Jiaxi Gu

  • Politics/International Relations & Europe — Steven Erly and Brian Erly

  • Politics/International Relations & Asia — truthLover

Rules & Eligibility

You may participate in this competition if you are 18 years of age and over and register via the HeroX GF Challenge page if you are an:
  • Individual or Team (all team members must be 18 years of age and over)
  • Organization or company*
  • International participant or a foreign national*
Some individuals or organizations may not be eligible for monetary prizes. However, these Solvers may, upon IARPA approval, participate in the challenge and be eligible for ranking on the leaderboard. Approval will need to be obtained in advance of participation by emailing: gfchallenge@iarpa.gov. HFC Performer team members and HFC Good Judgment Forecasters are not able to participate in the challenge.

*Additional exclusions may apply, for a full list of eligible participants see the Eligibility section within the Challenge Rules Document.

Resources

The following resources have been compiled to help familiarize solvers with geopolitical forecasting, scoring, and the platform for the challenge. These resources are intended to serve as examples of relevant background materials, and identification here does not constitute a recommendation from IARPA to purchase any for-pay services related to these resources.

GF Challenge Resources

CONNECT WITH US