Model Drift

The Intelligence Advanced Research Projects Activity (IARPA) is seeking information on automated methods to detect, quantify, and correct model drift. This request for information (RFI) is issued solely for information gathering and planning purposes; this RFI does not constitute a formal solicitation for proposals. The following sections of this announcement contain details of the scope of technical efforts of interest, along with instructions for the submission of responses.

Background & Scope

IARPA develops technologies to forecast a broad set of events relevant to national security. Once these technologies are deployed in an operational environment, the Intelligence Community needs automated methods to detect model drift, including:

  • When algorithms require re-training (e.g. to include more historical data or significant recent events), or
  • When the assumptions (overt or implied) underlying a model do not apply (e.g. mobility assumptions underlying an epidemiological model), or
  • When the input data are significantly different from the data on which the model was trained (e.g. applying a model trained on one part of the world to forecast events in another part of the world), or
  • When the context in which the events are being forecasted has changed significantly (e.g. forecasting societal events before and after a coup), or, more generally,
  • When confidence in the output of a model has been diminished enough to require model retraining, updating, forking for specific conditions, or even withholding.

When model drift is detected, it is important to determine whether correction is required. Correction can be resource-intensive, so must be done only when needed. For example, a small degradation of performance may be acceptable to some users and applications. This is a risk-management problem that requires quantifying the drift. If model drift causes decrements in performance, a model may need to be corrected, through retraining, updating, or complete rebuilding. The purpose of this RFI is to identify existing automated methods to detect, quantify, and correct model drift. These methods will likely include an efficient instrumentation of models, and computational techniques to process the data generated by that instrumentation. This RFI also seeks to identify innovative approaches to long-term model maintenance that would advance the state-of-the-art and improve greatly the long-term viability of technologies transitioned by IARPA.

Responses to this RFI should answer any or all of the following questions:

  1. What are existing automated methods for detecting model drift? Please provide instantiations of such methods for specific models or domains.
  2. What are existing automated methods for quantifying model drift? Please provide instantiations of such methods for specific models or domains.
  3. What are existing automated methods for correcting model drift? Please provide instantiations of such methods for specific models or domains.
  4. Can these existing automated methods be applied to systems by measuring the input/output characteristics, or do systems that employ these methods require instruments built into the system itself?
  5. What are metrics and protocols to assess the performance of such methods?
  6. What novel approaches to long-term model maintenance do you propose that would advance the state-of-the-art as described in your answers to questions 1-3?
  7. What are the key, cross-domain challenges or limiting factors in the development of methods to detect, quantify, and correct model drift?

Preparation Instructions to Respondents

IARPA requests that respondents submit ideas related to this topic for use by the Government in formulating a potential program. IARPA requests that submittals briefly and clearly describe the potential approach or concept, outline critical technical issues/obstacles, describe how the approach may address those issues/obstacles and comment on the expected performance and robustness of the proposed approach. If appropriate, respondents may also choose to provide a non-proprietary rough order of magnitude (ROM) regarding what such approaches might require in terms of funding and other resources for one or more years. This announcement contains all of the information required to submit a response. No additional forms, kits, or other materials are needed.

IARPA appreciates responses from all capable and qualified sources from within and outside of the US. Because IARPA is interested in an integrated approach, responses from teams with complementary areas of expertise are encouraged.

Responses have the following formatting requirements:

  1. A one page cover sheet that identifies the title, organization(s), respondent's technical and administrative points of contact - including names, addresses, phone and fax numbers, and email addresses of all co-authors, and clearly indicating its association with RFI-14-02;
  2. A substantive, focused, one-half page executive summary;
  3. A description (limited to 5 pages in minimum 12 point Times New Roman font, appropriate for single-sided, single-spaced 8.5 by 11 inch paper, with 1-inch margins) of the technical challenges and suggested approach(es);
  4. A list of citations (any significant claims or reports of success must be accompanied by citations, and reference material MUST be attached);
  5. Optionally, a single overview briefing chart graphically depicting the key ideas.
Submission Instructions to Respondents Responses to this RFI are due no later than 4:00pm, Local Time, College Park, MD on March 14, 2014. All submissions must be electronically submitted to dni-iarpa-rfi-14-02@iarpa.gov as a PDF document. Inquiries to this RFI must be submitted to dni-iarpa-rfi-14-02@iarpa.gov. Do not send questions with proprietary content. No telephone inquiries will be accepted.

New Approach and New Technologies Needed

This RFI seeks inputs on the feasibility of making significant progress on the challenge of tracking event emergence as reported in unstructured text. First, it is crucially important that event be clearly defined and that the concept of emergence be specified. Secondly, tracking emerging events will require methods for identifying the events and their participants beginning at a relatively granular level and moving through stages of event emergence. Techniques must move beyond verb-based event definition and extraction to other, richer characterizations of events and to approaches that find and link events as higher-level events emerge. For example, Communication Events may lead to Planning Events. Planning Events will specify topics or issues to be addressed. A Planning Event may result in a Purchase Event. At any stage along this emergence, it may not be known what the ultimate outcome will be; but reasoning about topics, activities, related or past events and the participating entities will be crucial to understanding emergence.

Automating the process of identifying and tracking emerging events and their participants must address a wide range of challenges, including:

  • Defining the concept of event, various precursor events and event emergence
  • Addressing event co-reference within and across documents as well as entity co-reference (e.g., person, place, equipment)
  • Representing and reasoning over time
  • Reasoning with uncertainty

This RFI invites discussion of substantive advancements in the development of technologies and architectures that support identifying emerging events, their components and their participants and that can reason about the events over time.

The RFI is not interested in building reference lists (e.g., nicknames, place names); in improved processing of phone numbers, dates of birth, etc.; nor in resolving database semantics across existing databases. Also not of interest are "strategic" or "society-level" events or trends that involve populations, e.g., predicting civil unrest, national rebellions, etc.

The responses to this RFI will be used to help in the planning of a 1-day workshop on new approaches to understanding, characterizing and capturing emerging events. An expected result of such a workshop is the identification of promising areas for investment through vehicles such as seedlings, grand challenges, and programs. It is anticipated that this workshop will be held in the Washington, D.C., Metro Area in June, 2014. A separate workshop announcement will be posted with further details.

Preparation Instructions to Respondents

IARPA is seeking respondents to submit ideas related to this topic for use by the Government in formulating a potential program. IARPA requests that submittals briefly and clearly describe the potential approach or concept, outline critical technical issues, and comment on the expected performance, robustness, and estimated cost of the proposed approach. This announcement contains all of the information required to submit a response. No additional forms, kits, or other materials are needed.

IARPA appreciates responses from all capable and qualified sources from within and outside of the U.S.

Responses have the following formatting requirements:

A one page cover sheet that identifies the title, organization(s), respondent's technical and administrative points of contact - including names, addresses, phone and fax numbers, and email addresses of all co-authors, and clearly indicating its association with IARPA-RFI-14-02: 

  1. A one page cover sheet that identifies the title, organization(s), respondent's technical and administrative points of contact - including names, addresses, phone and fax numbers, and email addresses of all co-authors, and clearly indicating the association with RFI-14-02;
  2. A substantive, focused one-half page executive summary;
  3. A description (limited to 5 pages in minimum 12 point Times New Roman font, appropriate for single-sided, single-spaced 8.5 by 11 inch paper, with 1-inch margins) of the technical challenges and suggested approach(es);
  4. A list of citations (any significant claims or reports of success must be accompanied by citations, and reference material MUST be attached);
  5. Optionally, a single overview briefing chart graphically depicting the key ideas.

For information contact:

dni-iarpa-info@iarpa.gov

IARPA-RFI-14-02   CLOSED

Posted Date: January 28, 2014
Responses Due: March 14, 2014