Knowing when someone is telling the truth plays a critical role in law enforcement and national security events, to include criminal investigations, screening new employees before hiring, and interviewing potential sources and witnesses. The polygraph is one tool that members of the Intelligence Community (IC) and law enforcement look to for help, but there is a long-standing debate among researchers and polygraph practitioners about the accuracy and reliability of this tool. How can we evaluate how good the polygraph is, and how much better new tools may be? The Intelligence Advanced Research Projects Activity (IARPA), within the Office of the Director of National Intelligence (ODNI), intends to launch the Credibility Assessment Standardized Evaluation (CASE) Challenge to address this critical question.
The goal of the CASE challenge is to develop novel procedures to evaluate the accuracy, reliability, and utility of current and future credibility assessment techniques and technologies, such as the polygraph. Credibility refers to the truthfulness of information and/or to the person providing that information. Assessments of credibility are often complex and may involve an evaluation of many factors of a person and/or their information, to include, but not limited to, veracity, trustworthiness, motivation, and considerations about what may be withheld or concealed. To evaluate the credibility of an individual and/or their information the IC and law enforcement often use human judgment and complement this with additional techniques and technologies, such as specific interviewing techniques or devices, like the polygraph, to record behavioral or physiological responses when someone responds to a question.
The polygraph test remains the standard for using technology-assisted assessments of credibility, despite having undergone only modest improvements over roughly the last fifty years. Other methods and systems have been proposed, but for varying reasons have not achieved widespread use for national security or law enforcement applications. A major limitation is that current procedures for evaluating these technologies frequently have little applicability to real-world credibility assessment situations, resulting in levels of performance in everyday use that sometimes differ dramatically from those reported by laboratory researchers. Real applications of credibility assessment tools and techniques might include the following:
- Issue-Specific: An investigator must evaluate the credibility of the information a person may provide on a specific event or topic, for example after a crime has been committed.
- Screening: A manager must determine the credibility of a person and/or their information, such as a potential new employee. In such situations the manager may ask more general and prospective questions, as well as retrospective ones (e.g. Do you intend to keep the company’s trade secrets? Have you ever used illegal substances?). The truth may be difficult to determine and it may not be completely objective.
The CASE Challenge is the first concerted effort to invite interested individuals to develop credibility assessment evaluation methods that can be used to objectively evaluate both existing and novel credibility assessment techniques/technologies. In doing this the CASE Challenge strives to incentivize a broad range of new ideas, while still ensuring their utility to real-world applications. To meet this goal, a panel of experts will evaluate each solution based on: its novelty and innovation; the potential utility of the protocol in practice; its scientific credibility and how likely it will be to generate valid and repeatable outcomes and objective ground truth about whether someone is truly credible or not; its applicability to real-world use cases for the IC and law enforcement; and its responsible care and protection of people who would participate in the protocol.
In addition to the financial incentives for prize winners, participants will be invited to a workshop where winners will present their solutions and people who use credibility assessment techniques and technologies will mix with researchers and other challenge participants. It is anticipated that there will be several prizes with a total prize purse of $100,000. The prizes will focus on ideas that are novel, innovative, safe, have the potential to be useful, and are scientifically credible.
A Participants’ Day WebEx was held on Tuesday, July 31, 2018 in anticipation of the challenge launch. The slides and audio from the WebEx is archived here. This event served as a pre-prize challenge notice and was issued solely for information and planning purposes. The official details of the prize challenge will be released at a later date. Participation in this WebEx was voluntary and not required for challenge participation.
Other Government Agencies, Federally Funded Research and Development Centers (FFRDCs), University Affiliated Research Centers (UARCs), or any other similar organizations that have a special relationship with the Government, that gives them access to privileged or proprietary information, or access to Government equipment or real property, will not be eligible to participate in the prize challenge.
Questions concerning the challenge, Participants’ Day WebEx, or registration can be sent to CASEChallenge@iarpa.gov.