Using Neural Tools to Augment Prediction of Performance, Expertise, and Domain-knowledge (UNTAPPED)
The Intelligence Advanced Research Projects Activity is seeking information on novel approaches and tools for predicting an individual’s future cognitive performance in complex environments using structural and functional measures of the brain. This request for information is issued solely for information gathering and planning purposes; this RFI does not constitute a formal solicitation for proposals. The following sections of this announcement contain details of the scope of technical efforts of interest, along with instructions for the submission of responses.
Background & Scope
Many organizations – from universities and companies to competitive sports teams and the military – are interested in accurately predicting an individual’s future cognitive performance and potential for different domain-knowledge and expertise.1 Such a predictive capability would allow organizations to determine in advance who is most likely to be able to learn and master complex skills and accomplish tasks in real-world environments that are important for the organization’s mission and success, thereby increasing return on investment for training activities and optimizing matching of personnel to tasks/environments. Such a capability would be particularly valuable in professions where job demands and required skills change rapidly due to new environments, new competitors, and/or advances in tools and methods. Indeed, given the accelerating rate of technical and social change impacting many organizations, as well as increasing costs for training and sustaining human resources, the ability to improve accuracy in predicting future cognitive performance for even a small percentage of current or future personnel could be significant.
Although conventional tools and measures such as academic achievement, pencil-and-paper exams, and previous experience can be informative for predicting future performance, some recent research2 suggests that it may be possible to supplement (or supplant) traditional evaluation tools with direct measurements of the brain to confer additional predictive power. However, the extent to which neural tools can improve prediction of performance and expertise over and above more conventional tests remains in question, and the extent to which neural tools and measures have been tested using meaningful real-world outcome measures is also unclear.
Therefore, IARPA is soliciting responses to this RFI to better characterize the state of the art in brain-based predictors of future cognitive performance. In particular, IARPA is interested in non-invasive assays of brain structure and/or function that can be used to predict who will best learn complex skills and accomplish tasks within real-world environments, and with outcome measures, that are relevant to national security. Beyond characterizing the state of the art, IARPA also seeks insight into credible next-generation (0 – 5 years out) tools, methods, and/or analyses that may overcome current technical and/or practical hurdles in predicting future cognitive performance, or that have been developed for other applications (e.g. predicting or assessing mental illness or psychological disorders) but have not yet been tested for this application.
Responses to this RFI should include information about all of the following topics, to the extent that information is available:
- What characteristics of brain structure and/or function are (or could be) used to predict future cognitive performance or potential expertise in different domains?
- Which technologies are used to collect this information?
- What are the experimental protocol(s) employed?
- Have the measures been used in conjunction with a specific intervention (e.g. collected prior to or in parallel with task-related training to predict training outcome)?
- What are the key technical or logistical challenges in collecting these data (e.g. multiple samples are needed per person and/or overall; extensive time is required to collect the data; imaging requires dye injection; analysis requires supercomputer; etc.)?
- Which of these challenges (if any) are likely to be resolved with advances in primary or enabling technologies?
- What types of cognitive performance have been (or could be) predicted with these measures? If this performance isn’t demonstrated in terms of real-world outcomes, is there evidence in the literature that these predictions could affect real-world outcomes?
- For which types of cognitive tasks are neural measures likely to offer the most predictive power relative to more easily assessable behavioral predictors (e.g., pen and paper tests)?
- How far in advance have these measures been shown to be predictive (i.e. do they predict performance one hour/day/week/year from when they are collected)?
- What data support the finding that the measures can be used to predict performance in the future vice simply correlating with previous or current performance?
- In what kind and size of populations have these results been demonstrated? How much variance is exhibited in the accuracy of prediction? How much additional variance could be expected by increasing the diversity of the subject population in terms of sex, age, IQ, occupation, nationality, culture, etc.?
- What support, if any, have these measures received in peer-reviewed literature? Have there been any reports (anecdotal or published) of negative findings using the same types of measures?
- Have these brain-based predictors been compared to or combined with conventional (non-neural) measures? If so, what are the relative costs/benefits? If not, against which conventional measures should the proposed approach be tested?
- Given that performance on a complex cognitive task or expertise in a specific domain may be mediated by a host of intermediate factors, what are likely to be fundamental limitations (theoretical and/or practical) to the development, testing, or use of neural tools and measure(s)?
Preparation Instructions to Respondents
IARPA requests that respondents submit ideas related to this topic for use by the Government in formulating a potential program. IARPA requests that submittals briefly and clearly describe the potential approach or concept, outline critical technical issues/obstacles, describe how the approach may address those issues/obstacles and comment on the expected performance and robustness of the proposed approach. If appropriate, respondents may also choose to provide a non-proprietary rough order of magnitude (ROM) regarding what such approaches might require in terms of funding and other resources for one or more years. This announcement contains all of the information required to submit a response. No additional forms, kits, or other materials are needed.
IARPA appreciates responses from all capable and qualified sources from within and outside of the US. Because IARPA is interested in an integrated approach, responses from teams with complementary areas of expertise are encouraged.
Responses have the following formatting requirements:
- A one page cover sheet that identifies the title, organization(s), respondent's technical and administrative points of contact - including names, addresses, phone and fax numbers, and email addresses of all co-authors, and clearly indicating its association with RFI-14-08;
- A substantive, focused, one page executive summary;
- A description of the technical challenges and suggested approach(es), limited to ten (10) pages in minimum 12-point Times New Roman font, appropriate for single-sided, single-spaced 8.5 by 11 inch paper, with 1-inch margins;
- A list of citations with all reference material attached or linked to accessible repositories;
- A single overview briefing chart graphically depicting the key ideas.
Submission Instructions to Respondents
Responses to this RFI are due no later than 5:00 PM Eastern Standard Time on September 30, 2014. All submissions must be electronically submitted to firstname.lastname@example.org as a PDF document. Inquiries to this RFI must be submitted to email@example.com. Do not send questions with proprietary content. No telephone inquiries will be accepted.
Disclaimers and Important Notes
This is an RFI issued solely for information and planning purposes and does not constitute a solicitation. Respondents are advised that IARPA is under no obligation to acknowledge receipt of the information received, or provide feedback to respondents with respect to any information submitted under this RFI.
Responses to this notice are not offers and cannot be accepted by the Government to form a binding contract. Respondents are solely responsible for all expenses associated with responding to this RFI. IARPA will not provide reimbursement for costs incurred in responding to this RFI. It is the respondent's responsibility to ensure that the submitted material has been approved for public release by the information owner.
The Government does not intend to award a contract on the basis of this RFI or to otherwise pay for the information solicited, nor is the Government obligated to issue a solicitation based on responses received. Neither proprietary, nor classified concepts nor information, should be included in the submittal. Input on technical aspects of the responses may be solicited by IARPA from non-Government consultants/experts who are bound by appropriate non-disclosure requirements.
2. See for example: Acuna, D.E., Allesina, S., Kording, K.P. (2012) Predicting scientific success. Nature, 489, 201- 202. Berkman, E.T., & Falk, E.B. (2013). Beyond brain mapping: Using the brain to predict real-world outcomes. Current Directions in Psychological Science. 22(1), 45-50. Erickson, K. I., Boot, W. R., Basak, C., Neider, M. B., Prakash, R. S., Voss, M. W., Graybiel, A. M., Simons, D. J., Fabiani, M., Gratton, G., & Kramer, A. F. (2010). Striatal Volume Predicts Level of Video Game Skill Acquisition. Cerebral Cortex, 20(11), 2522-2530. Ferguson, R.B. (2013, April). Predicting the Performance of Analytics Talent. MIT Sloan Management Review. Retrieved from sloanreview.mit.edu. Mathewson, K.E., Basak, C., Maclin, E.L., Low, K.A., Boot, W.R., Kramer, A.F., Fabiani, M., Gratton, G. (2012) Different slopes for different folks: Alpha and delta EEG power predict subsequent video game learning rate and improvements in cognitive control tasks. Psychophysiology, 49, 1558-70. Morgan, C.A. 3rd, Rasmusson, A., Pietrzak, R.H., Coric, V., Southwick, S.M. (2009) Relationships among plasma dehydroepiandrosterone and dehydroepiandrosterone sulfate, cortisol, symptoms of dissociation, and objective performance in humans exposed to underwater navigation stress. Biological Psychiatry, 66(4), 334-40. Supekar, K., Swigart, A.G., Tenison, C., Jolles, D.D., Rosenberg-Lee, M., Fuchs, L., Menon, V. (2013) Neural Predictors of Individual Differences in Response to Math Tutoring in Primary-Grade School Children. PNAS, 110(20), 8230-35. Tan, L.H., Chen, L., Yip, V., Chan, A.H.D., Yang, J., Gao, J-H., Siok, W.T. (2011) Activity levels in the left hemisphere caudate–fusiform circuit predict how well a second language will be learned. PNAS, 108(6), 2540-44. Ullman, H., Almeida, R., Klingberg, T. (2014) Structural Maturation and Brain Activity Predict Future Working Memory Capacity during Childhood Development. Journal of Neuroscience, 34(5),1592-8. Ventura-Campos, N., Sanjuan, A., Gonzalez, J., Palomar-Garcia, M. A., Rodriguez-Pujadas, A., Sebastian-Galles, N., Deco, G., & Avila, C. (2013). Spontaneous brain activity predicts learning ability of foreign sounds. Journal of Neuroscience, 33(22), 9295-9305. Vo, L., Walther, D., Kramer, A., Erickson, K., Boot, W., Voss, M., Prakash, R., Lee, H., Fabiani, M., Gratton, G., Simons, D., Sutton, B., & Wang, M. (2011). Predicting individuals' learning success from patterns of pre-learning MRI activity. PLoS One, 6(1), e16093. Wong, P. C., Morgan-Short, K., Ettlinger, M., & Zheng, J. (2012). Linking neurogenetics and individual differences in language learning: the dopamine hypothesis. Cortex, 48(9), 1091-1102. Wong, P. C. M., & Ettlinger, M. (2011). Predictors of spoken language learning. Journal of Communication Disorders, 44(5), 564-567. Yadav, S.K. and Pal, S. (2012). Data Mining: A Prediction for Performance Improvement of Engineering Students using Classification. World of Computer Science and Information Technology Journal, 2(2), 51-56. arXiv:1203.3832 [cs.LG]