Credibility Assessment Datasets
The Intelligence Advanced Research Projects Activity is seeking information on data sources for evaluation of credibility assessment techniques and technologies. This Request For Information is issued solely for information gathering and planning purposes; this RFI does not constitute a formal solicitation for proposals. The following sections of this announcement contain details of the scope of technical efforts of interest, along with instructions for the submission of responses.
Background & Scope
The assessment of a source’s credibility, or their information, is a core challenge for a broad range of intelligence, defense, homeland security, and law enforcement applications. There are a variety of techniques and technologies that are purported to be useful for establishing the credibility of a source, such as an individual, and/or their information, but it is difficult to evaluate and compare the accuracy, sensitivity, specificity, and practical utility of such solutions. Effective evaluations and comparisons of different credibility assessment methods are hampered by a diversity of testing and demonstration environments, the use of experimental methods that lack construct validity, and other testing conditions that impact the relevance to national security applications, such as poor ecological validity, low stakes or low consequence conditions, and unrepresentative participant cohorts. These concerns are further exacerbated by the potential financial gain for developers of such approaches, which can introduce biased methods and potentially limited reporting about the performance of such systems. Together, these limitations highlight the need for an independent body to motivate and evaluate different credibility assessment techniques and technologies.
To enable such an open and fair evaluation, IARPA seeks data sources that could be compiled and used to independently evaluate and verify the state of the art in credibility assessment techniques and technologies. For this RFI, credibility refers to the veracity of information and/or the person or source providing that information. Assessments of credibility are often complex and may involve an evaluation of many factors of a source and/or the information, to include, but not limited to, veracity, trustworthiness, motivation, considerations about what may be withheld or concealed, and if credibility of information and/or a source has evolved across environments or over time, such as through transmission (e.g. second or third hand information). This RFI seeks data sources that include an evaluation of one, some, or all of these factors, as well as others that are not listed here, as long as they can be theoretically or empirically linked to an evaluation of source and/or information credibility.
Of critical interest are datasets that can be made publically accessible, or accessible via a data use agreement, and which have objective ground truth about the credibility of the source and/or the information. Data sources of interest include, but are not limited to:
- Experimental data from real or mock/simulated scenarios;
- Data from real world events;
- Information that is exchanged in physical or virtual spaces, either face to face or via other channels (e.g. exclusively text based);
- Information that is delivered directly by the individual whose credibility is being evaluated, or through a proxy or intermediate (e.g. messenger, translator, avatar);
- The credibility of the information and/or source may evolve over time, through transmission, and/or across environments/contexts;
- Video, audio, behavioral, text, physiological, or other data types.
Responses to this RFI should address the following points:
1. Please describe the dataset.
- When and how were the data collected?
- What were the steps in the protocol? How were data collected from a real environment or event?
- What/who is the source that is being assessed (to include characteristics, such as demographics, as relevant)? And/or what information is being assessed?
- How is credible vs. not credible defined in the dataset? How is this determined for credibility that may evolve over time, context, or as the information and/or source may evolve?
- How is credibility demonstrated, evaluated, and verified?
- What is the motivation or incentive to be/not be credible, or have information be/not be credible?
2. Were the data collected with an approved protocol from an Institutional Review Board? Was it determined to be exempt by an IRB? Were there other policies, legal requirements, privacy standards that were adhered to or implemented in the collection or storage of the data?
3. What is the size of the dataset? What is the sample size (e.g. number of participants, unique events, pieces of information)? What is the proportion of credible versus not credible?
4. What specific data types are available? What is the format of the data?
5. Are there restrictions on the use or accessibility of the data? Is there any cost associated with accessing or using the data? What permissions or use agreements are necessary? How are the data accessible (e.g. cloud-based or local repository, local hard drive)? Who has permissions to hold, host, and/or work with the data? What other challenges exist with sharing such data, how might these challenges be overcome?
6. Have the data been analyzed, reported in any publications (e.g. conference papers, informal presentations, manuscripts, official reports), or otherwise been cited in any peer-reviewed, open source, or media reports?
7. Other details that would be useful to understand the data, methods, or usability.
Preparation Instructions to Respondents
IARPA requests that respondents submit ideas related to this topic for use by the government in formulating a potential program. IARPA requests that submittals briefly and clearly describe the potential dataset, address questions related to accessibility and usability, outline critical technical issues/obstacles, and describe how those issues/obstacles may be addressed. This announcement contains all of the information required to submit a response. No additional forms, kits, or other materials are needed.
IARPA appreciates responses from all capable and qualified sources from within and outside of the US. Because IARPA is interested in an integrated approach, responses from teams with complementary areas of expertise are encouraged. Responses have the following formatting requirements:
1. A one-page cover sheet that identifies the title, organization(s), respondent's technical and administrative points of contact - including names, addresses, phone and fax numbers, and email addresses of all co-authors, and clearly indicating its association with RFI-18-02;
2. A substantive, focused, one-half page executive summary;
3. A description (limited to 8 pages in minimum 12 point Times New Roman font, appropriate for single-sided, single-spaced 8.5 by 11-inch paper, with 1-inch margins) of the technical challenges and suggested approach(es);
4. A list of citations (any significant claims or reports of success must be accompanied by citations);
5. Optional, a single overview briefing chart graphically depicting the key ideas.
Submission Instructions to Respondents
Responses to this RFI are due no later than 5 p.m., Eastern Daylight Time, on May 31, 2018. The government anticipates that responses submitted to this RFI may be unclassified or classified.
All unclassified submissions must be electronically submitted to email@example.com as a PDF document. Inquiries to this RFI must be submitted to firstname.lastname@example.org. Do not send responses with proprietary content. No telephone inquiries will be accepted.
All classified submissions must first receive permission from IARPA Security prior to responding to this RFI.
For classified submissions, contact the IARPA Security Office at 301-851-7580 for further guidance and instructions prior to writing or transmitting classified information to IARPA. IARPA will not accept classified responses without prior validation of accredited spaces and systems on which to write the response. If the response is classified at the SCI level, a Co-Use/Joint-Use Agreement must be coordinated by IARPA with the cognizant security organization prior to writing a response to this RFI. Respondents choosing to submit a classified response are reminded that the proposal deadline remains the same regardless of whether the responses, in whole or in part, is classified. Additional processing time may be required if all or part of a response is classified.
IARPA will not sponsor facility accreditations or personnel security clearances. IARPA will not accept classified deliveries without prior coordination with IARPA Security staff.
Disclaimers and Important Notes
This is an RFI issued solely for information and planning purposes and does not constitute a solicitation. Respondents are advised that IARPA is under no obligation to acknowledge receipt of the information received, or provide feedback to respondents with respect to any information submitted under this RFI.
Responses to this notice are not offers and cannot be accepted by the government to form a binding contract. Respondents are solely responsible for all expenses associated with responding to this RFI. IARPA will not provide reimbursement for costs incurred in responding to this RFI. It is the respondent's responsibility to ensure that the submitted material has been approved for public release by the information owner.
The government does not intend to award a contract on the basis of this RFI or to otherwise pay for the information solicited, nor is the government obligated to issue a solicitation based on responses received. Proprietary information should not be included in the submittal. Input on technical aspects of the responses may be solicited by IARPA from non-government consultants/experts who are bound by appropriate non-disclosure requirements.
Contracting Office Address:
Office of the Director of National Intelligence
Intelligence Advanced Research Projects Activity
Washington, District of Columbia 20511
Primary Point of Contact:
IARPA-RFI-18-02 CLOSEDPosted Date: February 27, 2018
Responses Due: May 31, 2018