Crowdsourcing Evidence, Argumentation, Thinking and Evaluation

Intelligence Value

The CREATE program developed tools and methods designed to improve analytic reasoning through the use of crowdsourcing and structured analytic techniques. These new resources empower multi-disciplinary collaboration among analysts to provide the Intelligence Community with accurate, timely, and evidence-based analyses.


Structured analytic techniques hold promise for increasing the rigor and transparency of analyses. These techniques can help reveal underlying logic and identify unstated assumptions. To make these techniques straightforward to implement, the CREATE program developed software platforms that allow geographically dispersed teams to collaborate seamlessly to assess evidence, identify key information gaps, challenge assumptions, and benefit from the collective wisdom of individual team members. The collaboration platforms can be used not only for intelligence analysis, but also for domains, such as science, law, or policy in which people must reason their way through complex questions.

As an example of the value CREATE has delivered, one of the research teams from the University of Melbourne used their platform, called SWARM -- the Smartly-assembled Wiki-style Argument Marshalling -- as the means for distributed collaboration among roughly 100 researchers from Australia’s eight leading universities. SWARM was used to analyze possible outcomes of policy decisions to relax COVID-19 social distancing restrictions. The result of the three-week, quick turnaround collaboration was a report, titled “Roadmap to Recovery,” which was delivered to the Australian minister for health and the chief medical officer at the end of April 2020. The “Roadmap to Recovery” report was a comprehensive analysis of the outcomes of alternative policy decisions under evaluation for Australia’s recovery. The CREATE program ended in February 2020.

Related Publications

To access CREATE program-related publications, please visit Google Scholar.


Contact Information

Program Manager

Main Office

Related Program(s)

Broad Agency Announcement (BAA)

Link(s) to BAA


Solicitation Status


Proposers' Day Date

June 30, 2015

BAA Release Date

February 16, 2016

BAA Question Period

February 16, 2016 — March 15, 2016

Proposal Due Date

May 9, 2016

Program Summary

Testing and Evaluation Partners

  • Johns Hopkins University Applied Physics Laboratory

Prime Performers

  • Melbourne University
  • Monash University
  • George Mason University
  • Syracuse University 

Additional Information

Frequently Asked Questions (FAQs)

Join the CREATE study

We are looking for highly motivated, inquisitive, open-minded individuals to participate in CREATE. Please visit our experiment website to join in and help us improve analytic reasoning!