Seeking Data Sources for Technology Performance Prediction

The Intelligence Advanced Research Projects Activity (IARPA) invests in high-risk, high pay off research programs to tackle some of the most difficult challenges of the agencies and disciplines in the Intelligence Community.

Background & Scope

IARPA is exploring methods to predict the future performance of critical technologies. Making and evaluating such predictions will require historical data on technologies’ performance over time, ideally over the span of decades (e.g., the TOP500 dataset illustrating computation performance improvements). Examples of prior work with such data include:

Koh, Heebyung, and Christopher L. Magee. “A Functional Approach for Studying Technological Progress: Application to Information Technology.” Technol. Forecast. Soc. Change 73, no. 9 (November 2006): 1061–83. https://doi.org/10.1016/j.techfore.2006.06.001

Koh, Heebyung, and Christopher L. Magee. “A Functional Approach for Studying Technological Progress: Extension to Energy Technology.” Technological Forecasting and Social Change 75, no. 6 (July 2008): 735–58. https://doi.org/10.1016/j.techfore.2007.05.007

Martínez-Plumed, Fernando, Shahar Avin, Miles Brundage, Allan Dafoe, Sean Ó hÉigeartaigh, and José Hernández-Orallo. “Accounting for the Neglected Dimensions of of AI Progress.” ArXiv:1806.00610 [Cs], June 2, 2018. http://arxiv.org/abs/1806.00610

IARPA requests information on: 

Datasets that document the historical performance of technologies. Datasets must have historical coverage of at least a decade and, most importantly, have accurate information on when the technology was first developed. For data about products, first development of the technology means “first offered for sale”. For data about pre-commercial or non-commercial technologies, date of journal publication or other public disclosure will be defined as first development. A temporal resolution of at least 1 year is needed for any data occurring after 1900; earlier data may have coarser temporal resolution. - Performance metrics for technologies. These must be objective, quantitative, and recoverable from the historical record. Many performance metrics will likely be multi-dimensional and represent tradeoffs in some aspect of the technology (e.g., FLOPs per Watt). IARPA is interested in all areas of technology, but particularly the areas of:

  • Artificial Intelligence
  • Collection & Sensing
  • Communication
  • Computing
  • Cybersecurity
  • Energy & Power
  • Human Performance Modification
  • Materials
  • Quantum Sciences
  • Space Sciences

These are broad areas of technology that are composed of constituent technologies, which are themselves composed of systems and components. It is the performance of these systems and components that can potentially be tracked and predicted in an objective and quantitative manner. For example, within power and energy, the ability to deliver high pulsed power can be accomplished with capacitor technology. This capacitor technology is effectively represented by the performance metric of power density (kilowatts per kilogram). As capacitor technology improves, increasing power density is a good indicator of performance improvement, and ultimately historical improvements in power density may be used to predict future improvements in power density.

Respondents may suggest data sources for one or several technology areas within their submission. For any technology area respondents choose to respond to, one or more of the following questions should be explicitly answered in their submission:

  1. What is the constituent technology, system or component whose performance serves as a suitable measure of the broad technology area specified?
  2. What are the appropriate metrics (including units) that best track the performance of the items listed in (1)? For these metrics, are there any tradeoff metrics that should also be considered?
  3. What are good data sources from which historical performance data could be obtained for (2)? “Good” data sources should accurately list the metrics in (2) and additionally have an accurate date when the technology became available and should continuously span at least 10 years. The data source must be orderly in the presentation and interpretation of the individual data points. It is desirable, but not necessary, that the data be non-proprietary, machine-readable, unclassified, and either documented by a credible or authoritative source and/or corroborated by an independent 3rd party. These data sources can be public or private, readily available or otherwise. Detailed references are encouraged where possible.

This RFI is issued solely for information gathering and planning purposes and does not constitute a formal solicitation for proposals.

Preparation Instructions to Respondents

IARPA requests that respondents submit suggestions for potential data sources related to this topic for use by the government in formulating a potential program. If appropriate, repsondents may also chose to provide a non-proprietary rough order of magnitude estimate regarding what such data sources may cost, or what the cost would be to gather from these data sources. This announcement contains all of the information required to submit a response. No additional forms, kits, or other materials are needed.

IARPA appreciates responses from all capable and qualified sources from within and outside of the U.S. Classified reponses will not be accepted.

Responses include the following requirements:

  1. A one-page cover sheet that indentifies the title, organization(s), respondent's technical and administrative points of contact, including names, addresses, phone and fax numbers and email addresses of all co-authors, and clearly indicating its association with IARPA-RFI-19-07
  2. Answers to one or more of the questions listed above, as they relate to each area of technology selected by the respondent from the bulleted list

Submission Instructions to Respondents

Responses to this RFI are due no later than 08/06/2019, 4pm Eastern Daylight Time. All submissions or inquiries must be electronically submitted to DNI-IARPA-RFI-19-07@iarpa.gov as a PDF or Word document. Do not send questions with proprietary content. No telephone inquiries will be accepted.

Disclaimers and Important Notes

This is an RFI issued solely for information and planning purposes and does not constitute a solicitation. Respondents are advised that IARPA is under no obligation to acknowledge receipt of the information received, or provide feedback to respondents with respect to any information submitted under this RFI.

Responses to this notice are not offers and cannot be accepted by the government to form a binding contract. Respondents are solely responsible for all expenses associated with responding to this RFI. IARPA will not provide reimbursement for costs incurred in responding to this RFI. It is the respondent's responsibility to ensure that the submitted material has been approved for public release by the information owner.

The government does not intend to award a contract on the basis of this RFI or to otherwise pay for the information solicited, nor is the government obligated to issue a solicitation based on responses received. Neither proprietary nor classified concepts nor information should be included in the submittal. Input on technical aspects of the responses may be solicited by IARPA from non-government consultants/experts who are bound by appropriate non-disclosure requirements.

Contracting Office Address:

Office of the Director of National Intelligence
Intelligence Advanced Research Projects Activity
Washington, District of Columbia 20511
United States

Primary Point of Contact:

Jeff Alstott
Intelligence Advanced Research Projects Activity
dni-iarpa-rfi-19-07@iarpa.gov

IARPA-RFI-19-07 CLOSED

Posted Date: June 12, 2019
Responses Due: August 08, 2019