Forecasting Rare Events
The Intelligence Advanced Research Projects Activity (IARPA) is seeking information on methods to model and forecast rare events. This request for information (RFI) is issued solely for information gathering and planning purposes; this RFI does not constitute a formal solicitation for proposals. The following sections of this announcement contain details of the scope of technical efforts of interest, along with instructions for the submission of responses.
Background & Scope
The purpose of this RFI is to identify existing and emerging methods to model and forecast rare events, and approaches for assessing the performance of these methods. Of particular interest are methods that can support fundamental advances in rare event forecasting that may be generalized or adapted for a range of domains. For present purposes, a “forecast” is defined as a probability assigned to an event or class of events, while a “rare event” is defined as an event observed with very low temporal or spatial frequency relative to the parent data population or reference class, such as less than one instance in one thousand observations. The usefulness of methods depends heavily on their performance (e.g., accuracy, lead time, false positive and negative rates). The low frequency of rare events makes model and forecast evaluation difficult, as the number of observations may be too small to make conventional statistical inferences about performance. This RFI also seeks to identify alternative approaches for assessing the performance of methods for forecasting rare events.
RFI respondents are encouraged to address methodological issues and describe the state-of-art in rare event modeling and forecasting. Domains of interest include, but are not limited to: International Relations (e.g., interstate wars), Political Science (e.g., coups), Epidemiology (e.g., pandemics), Economics (e.g., market crashes), Technological Hazards (e.g., nuclear incidents), Natural Hazards (e.g., earthquakes, extreme space weather, etc.) and changes in Scientific and Technological knowledge (e.g., breakthrough discoveries).
Responses to this RFI should answer any or all of the following questions:
- Forecasting Methods
- What are existing methods for forecasting rare events? What are specific examples that implement these methods?
- What are the statistical assumptions of the underlying models? For example, is the model based on a specific heavy-tailed distribution relating probability to frequency and magnitude? Is the modeled distribution empirical or theoretical? Is stationarity of the underlying process assumed?
- What are the limits of these models (e.g., time horizon of forecasts, data requirements, availability of outcome/evaluation data, resource requirements, or other limiting factors)?
- Test and Evaluation of Methods
- How accurate are the methods listed in your answer to Question 1? What approaches have been used to validate or assess the accuracy and/or usefulness of these forecasting methods? What are their strengths and limitations?
- More generally, how can truly prospective forecasts and/or forecasting models for infrequent events be evaluated, given practical limits to study duration and cost?
- To what extent are the methods listed in your answer to Question 1 applicable and adaptable to other domains?
- Data Sets and/or Data Collection Tools and Platforms
- What structured data sets exist for the analysis of rare events, to include data that would be used in formulation of model-based forecast estimates and/or evaluation of model performance against “ground truth”? In particular, what data sets exist that are not already listed on http://en.wikipedia.org/wiki/Rare_events?
- How are those data sets gathered, compiled, maintained, updated, and documented, and at what cost?
- Emerging and/or Cutting Edge Methods
- What novel methods could be developed, expanded, or adapted to improve or replace existing approaches to rare event forecasting or rare event forecast evaluation?
- What recent or underappreciated publications are of critical relevance to the development, improvement, or evaluation of rare event forecasting?
- What are the key, cross-domain challenges or limiting factors in the modeling and forecasting of rare events?
- What are the key, cross-domain challenges or limiting factors in evaluating the accuracy or performance of rare event models and forecasts?
- What are the most promising solutions to the challenges in your answers to Questions 6 and 7?
The responses to this RFI may be used to help in planning a one-day workshop on rare events. An expected result of such a workshop is the identification of promising areas for research investment.
Preparation Instructions to RespondantsIARPA requests that submittals briefly and clearly describe approach(es) and concept(s), outline critical technical obstacles, describe how critical obstacles may be addressed, and comment on the expected performance and robustness of the proposed approach(es). If appropriate, respondents may also choose to provide a non-proprietary rough order of magnitude (ROM) estimate of the cost of developing relevant approaches, in terms of funding and other resources for one or more years. This announcement contains all of the information required to submit a response. No additional forms, kits, or other materials are needed. IARPA appreciates responses from all capable and qualified sources from within and outside of the US. Because IARPA is interested in an integrated approach, responses from teams with complementary areas of expertise are encouraged. Responses have the following formatting requirements:
- A one-page cover sheet that identifies the response title, organization(s), respondent's technical and administrative points of contact - including names, addresses, phone and fax numbers, and email addresses of all co-authors, and clearly indicating its association with RFI-15-03;
- A substantive, focused, one-half page executive summary;
- A description (limited to 5 pages in minimum 12 point Times New Roman font, appropriate for single-sided, single-spaced 8.5 by 11 inch paper, with 1-inch margins) of the technical challenges and suggested approach(es);
- A list of citations (any significant claims or reports of success must be accompanied by citations);
- Optionally, a single overview briefing chart graphically depicting the key ideas.
Submission Instructions to RespondantsResponses to this RFI are due no later than 4:00pm, Local Time, College Park, MD on September 18, 2015. All submissions must be electronically submitted to firstname.lastname@example.org as a PDF document. Inquiries to this RFI must be submitted to email@example.com . Do not send questions with proprietary content. No telephone inquiries will be accepted.
Disclaimers and Important NotesThis is an RFI issued solely for information and planning purposes and does not constitute a solicitation. Respondents are advised that IARPA is under no obligation to acknowledge receipt of the information received, or provide feedback to respondents with respect to any information submitted under this RFI. Responses to this notice are not offers and cannot be accepted by the Government to form a binding contract. Respondents are solely responsible for all expenses associated with responding to this RFI. IARPA will not provide reimbursement for costs incurred in responding to this RFI. It is the respondent's responsibility to ensure that the submitted material has been approved for public release by the information owner. The Government does not intend to award a contract on the basis of this RFI or to otherwise pay for the information solicited, nor is the Government obligated to issue a solicitation based on responses received. Neither proprietary nor classified concepts or information should be included in the submittal. Input on technical aspects of the responses may be solicited by IARPA from non-Government consultants/experts who are bound by appropriate non-disclosure requirements.