Decision makers rely on the Intelligence Community to provide accurate and relevant forecasts, and IARPA is working to identify methods to maximize the quality of these forecasts. The Geopolitical Forecasting (GF) Challenge invites Solvers from around the world to develop innovative solutions and methods for integrating crowdsourced forecasts and other data into accurate, timely forecasts on worldwide issues.
GF Challenge Solvers will develop solutions that produce probabilistic forecasts in response to numerous closed-ended forecasting questions that concern specific, objectively verifiable geopolitical events containing timeframes with deadlines and locations. The effort will run in parallel to IARPA’s geopolitical forecasting research program - Hybrid Forecasting Competition (HFC). Challenge Solvers will be competing on a largely overlapping set of Individual Forecasting Problems (IFPs) as HFC research teams and given access to the same human forecaster data stream. In addition to the provided data stream, Solvers may use other data streams and their own data and models for the challenge.
Over the course of seven months, the GF Challenge solvers will be asked to develop solutions capable of producing forecasts to a series of questions like:
- Who will win the upcoming presidential election in Mexico?
- What will the spot price of Brent Crude oil be on [date]?
The challenge presents an opportunity for individuals and teams to earn prizes by creating methods that can successfully forecast a wide variety of geopolitical events, such as political elections, disease outbreaks, and macro-economic indicators.
Participating in the challenge is a two-step process. Solvers will register on the HeroX GF Challenge page and acknowledge their understanding and intent to follow the rules of the challenge. Following registration, instructions will be sent to access the application programming interface (API) within the Cultivate Labs platform. Solvers will be granted an API token and documentation for interacting with the challenge platform. There is only one API account allowed per Solver or Team (a single representative from each Team will need to setup the Team’s API account, see the Eligibility section within the Challenge Rules Document for details.)
Forecasting questions, referred to as IFPs, will be released periodically and will be open for a specified period of time. Solvers should check daily for new IFPs. While an IFP is open, Solvers may update their forecasts daily until the IFP closes. Each Solver will be permitted to submit forecasts using up to 25 different methodological “slots.” Once the IFP closes, the forecasts will be locked and scores for the IFP for each method will be calculated. Scores will be tracked on the leaderboard within the Cultivate Labs platform. For additional details on scoring and registration, please see the Challenge Rules Document.
A steady stream of IFPs (roughly 25 per month) will be released for Solvers to produce probabilistic forecasts against. IFPs may be binary (yes/no), multiple choice, or ordered outcome (binned quantity/date). In addition to the IFPs, Solvers will gain access to a continuously updated human forecaster data stream produced by the HFC research team.
- IFPs will be released via the API
- Each IFP will include a set of exhaustive, and mutually exclusive response options, and will have a final resolution date that may range from a few days to a few months in the future. Solvers will be asked to produce forecasts until the IFP resolves
- Examples of potential IFPs*:
- Will the WHO confirm more than 10 cases of Marburg in 2018? (Yes/No)
- Before March 2018 will South Korea file a World Trade Organization dispute against the United States related to solar panels? (Yes/No)
- Solver’s forecasts need to be submitted through the API
- Solvers should update their forecasts on a daily basis
Scores will be calculated using a metric which incorporates forecast accuracy and confidence. This metric is based on the Brier score, is a quadratic measure of the forecaster’s closeness to ground truth for the outcome that occurs. For example, an accurate forecast with 90% confidence for an event that occurs will score four times better than a forecast with 80% confidence.
As IFPs will range in duration, milestone check-ins for IFPs resolved during a certain period will be used to award progress-based incentives.
Registrations will be accepted on a rolling basis, so Solvers may register at any point. However, Solvers will be rewarded by accuracy scores that incentivize participation on their ability to forecast on more IFPs. Eligibility for prizes have minimum participation requirements for the resolved IFPs included for each prize. Solutions will need to be documented and all data streams used in prize-winning solutions will need to be outlined and verified by the GF Challenge Team. Read the Judging and Scoring Criteria section in the Challenge Rules Document for more detailed information.
Overall Performance Prizes
BONUS*: Star Forecasters Second through Fifth place may be eligible for a share of an additional $30,000.
*Bonuses have additional eligibility requirements. Please see the challenge rules for details.
Rules & Eligibility
- Individual or Team (all team members must be 18 years of age and over)
- Organization or company*
- International participant or a foreign national*
*Additional exclusions may apply, for a full list of eligible participants see the Eligibility section within the Challenge Rules Document.