Aggregative Contingent Estimation (ACE) BAA Questions

#QuestionAnswerDate Posted
001 Who are the "representative users" mentioned on p.10 of the BAA? Are they the same people who provide forecasts, or a different population? They are a different population. The "representative users" will be recruited and managed by the Government test and evaluation team. 07/16/10
002 How many forecasts will be aggregated in the unweighted linear opinion pool (ULinOP) mentioned on p.10 of the BAA? The Government test and evaluation team will recruit hundreds of study participants, who will self-select into providing forecasts for particular problems. The number of forecasts aggregated for each problem will vary, but is likely to be in the tens. 7/16/10
003 Does the tool need to be capable of performing the segmentation analysis and cluster analysis mentioned on p.11 of the BAA? Yes. 7/16/10
004 Should proposals be written for 1 year or 4 years? 4 years. Per Section 4.A.1 of the BAA, "Offerors should submit proposals for a Base Period of 12 months plus three possible 12-month Option Periods." 7/16/10
005 Is there a maximum budget for proposals? Per Section 2 of the BAA, "The amount of resources made available under this BAA will depend on the quality of the proposals received and the availability of funds." Cost realism is a criterion for evaluation of proposals, per Section 5.A.5 of the BAA. 7/16/10
006 Can cost proposals from universities include university overhead? Yes. Offerors may apply Government-approved indirect rates as applicable. 7/16/10
007 P.24 of the BAA mentions a "cost or pricing data" requirement if offerors seek a procurement contract award of $650,000 or greater. Is that $650,000 for the base year, or for the four-year total? The four-year total. 08/04/10
008 Are innovative, unconventional, non-monetary prediction markets within the scope of this program? Yes, but they must offer a substantial improvement over the state of the art. Per Section 1.A of the BAA, "The ACE Program seeks aggregation methods that . . . [g]enerate forecasts that are substantially more accurate than both the unweighted linear opinion pool and conventional prediction market forecasts." 08/04/10
009 Will forecasts of conditional events with unrealized conditions be evaluated? For instance, suppose the forecast is: there is probability P of event X given event Y. If Y does not occur, will the estimate P be evaluated? No, only forecasts on realized conditions will be evaluated. 08/04/10
010 Can proposals be single-spaced? Yes. 08/04/10
011 Are appendices to Volume 1 allowed that do not count towards the page limit? Per Section 4.B.1 of the BAA, "Volume 1, Technical and Management Proposal, may include an attached bibliography of relevant technical papers or research notes (published and unpublished) which document the technical ideas and approach on which the proposal is based. Copies of not more than three relevant papers can be included with the submission. The submission of other supporting materials along with the proposal is strongly discouraged and will not be considered for review. Except for the cover sheet, transmittal letter, signed Academic Institution Acknowledgement Letter(s) if required, OCI waiver/certification, table of contents, bibliography, and relevant papers, Volume 1 shall not exceed 30 pages." In addition, in accordance with Section 4.B.1 Section 3, the Human Use documentation and the written verification are not included in the total page count. 08/04/10
012 Do we have to deliver source code if we develop software on top of an existing commercial product? Per Section 4.B.1 of the BAA, "For all software deliverables, the offeror shall include all as delivered version source code produced in the course of software development. These deliverables must include source code and the appropriate scripting, subordinate libraries, release notes, and other necessary components, data, and documentation. These and all other deliverables developed as part of the IARPA ACE Program shall be delivered prior to the end of the contract Period of Performance. The Government desires Government Purpose Rights for all deliverables, anything less will be considered a significant weakness in the proposal." See also the requirements in Section 6.B.3 (Intellectual Property). 08/04/10
013 The announcement has an August 18th deadline for the initial round of selections, but also says that proposals may be considered for up to a year from the date of the announcement. Do you anticipate making the majority of selections during the initial round? Per Section 4.C.2 of the BAA, "Proposals must be received by the time and date specified in section 4.C.1 in order to be considered during the initial round of selections. IARPA may evaluate proposals received after this date for a period of up to one year from the date of initial posting on FedBizOpps. Selection remains contingent on availability of funds." 08/04/10
014 Is it required to submit an IRB draft protocol to IARPA as part of the proposal? No. Per Section 6.B.5 of the BAA, "For all proposed research that will involve human subjects, the institution must provide evidence of or a plan for review by an Institutional Review Board (IRB) on proposal submission to IARPA. The IRB conducting the review must be the IRB identified on the institution’s Assurance." 08/04/10
015 Are there any Communication deliverables due in year 1? In Year 1, performers will provide waypoint deliverables for Communication methods. Per Section 1.B of the BAA, "Offerors must also identify waypoints that will provide the Government with insight into the development of the key aspects of the research beyond the annual Program evaluations. These waypoints are to consist of self-evaluations of key research and development activities, and progress in the empirical work. Waypoints are to be reviewed during site visits (Table 4). Proposals must include rationale, definition, metrics, performance goals and an evaluation plan for each waypoint. Waypoints must provide a clear measure of progress toward meeting the Program milestones." Per Section 4.B.1 of the BAA, "Other deliverables are to include problem set nominations, research status reports including waypoint results, and significant completed tool prototypes, publications, and data." 08/09/10
016 Is it up to us to determine how much and what kind of information is shared among forecasters while forecasts are being updated? Performers will determine how much and what kind of information is shared among forecasters. Per Section 1.A of the BAA, "Each performer will choose how judgments are elicited, how judgments are aggregated, which scoring rules are applied to individual judgments, the amount and type of feedback given to forecasters, and the non-monetary incentives that are used. In its proposal, the offeror should justify initial approaches, alternatives to be explored, and measures to prevent or mitigate intentional and unintentional misreporting of judgments." 08/09/10
017 Are we precluded from generating training data to be used for the aggregation algorithm -- for instance, by having study participants answer training questions generated by our team? No, you are not precluded from using training data during research. However, in forecasting problems assigned by IARPA, per Section 1.A of the BAA, "elicitation should require no more than a few minutes per elicitation per forecaster," and data collected prior to elicitation "should take no more than 30 minutes of their time and should be self-administered online." 08/09/10
018 For a given event, is a performer's evaluation based only on the accuracy of its last aggregated forecast? No. Per Section 1.A of the BAA, "Since the performers' methods must allow forecasters to continuously update their forecasts, the mean forecast accuracy will be calculated per event, from the time the test problem is delivered to performers, until the time the forecast expires." 08/09/10
019 Do waypoints for option years have to be specified? Yes. Per Section 2 of the BAA, "Offerors must also identify waypoints that will provide the Government with insight into the development of the key aspects of the research beyond the annual Program evaluations. These waypoints are to consist of self-evaluations of key research and development activities, and progress in the empirical work. Waypoints are to be reviewed during site visits (Table 4). Proposals must include rationale, definition, metrics, performance goals and an evaluation plan for each waypoint. Waypoints must provide a clear measure of progress toward meeting the Program milestones." 08/09/10
020 The BAA on p. 19 calls for costs by task and then by deliverable. Costs by deliverable are also called for on p. 21. However, for the cost proposal, detail at the task level is called for (p.23 of BAA). Are we required to show costs for deliverables? Yes, per Section 4.B.1 of the BAA, "Summarize, in table form, the cost, schedule and milestones for the proposed research, including estimates of cost for each deliverable, total cost and company cost share, if applicable." 08/09/10
021 Can deliverable costs be bundled across several closely-related but distinct deliverables? No. 08/09/10
022 P.23 requests that major cost times be broken down by major task and phase. Please clarify the definition of "phase." "Phase" refers here to one of the four periods. 08/09/10
023 Can research subjects be paid? Per Section 1.A of the BAA, "Study participants may be reimbursed for their time, in accordance with the performer's Institutional Review Board (IRB) guidance. Such reimbursement should be minimal, uniform across participants, and consistent with compensation provided in other research on human judgment. (See also Section 6.B.5.) In its proposal, the offeror should also describe non-monetary incentives of negligible face value that will be used to motivate participants' level of participation, accuracy, and reasoning." 08/09/10
024 Must all research subjects be U.S. citizens? No. Research subjects do not have to be U.S. citizens. However, offerors who propose a significant number of non-U.S. citizens as research subjects may be assessed a technical weakness, as these individuals may not be representative of Intelligence Community analysts. Per Section 1.A of the BAA, "In its proposal, the offeror must describe the selection criteria for its study population, how the population will be recruited, and how it can be considered generally representative of IC analysts. The IC employs many thousands of analysts across 16 agencies. Common attributes of analysts are U.S. citizenship, which is required to hold a security clearance, and interest in national security. Otherwise, analysts are highly diverse. They vary widely in age, education, and training. Analysts range in experience from less than a year to more than 30 years on the job, and their educational levels range from bachelor's degrees to PhDs, with degrees in a wide range of disciplines including history, linguistics, law, international policy, science, mathematics, and engineering. Performers must therefore be able to recruit a study population from communities with similar diversity." 08/09/10
025 What is the minimum time horizon for a forecast? For example, could a forecasting problem be tied to an event that is as little as one month away? Per Section 1.A of the BAA, "Forecasting problems should include a variety of global political, military, economic, science and technology, and social events, on one-to-12+ month time horizons." 08/09/10
026 Can a forecaster be given more than one forecasting problem? Yes. 08/09/10
027 WiWill IARPA consider a three-year (base plus two option years) period of performance, instead of four? See response #004 above. 08/09/10
028 Should a performer disagree with verification references tied to a forecasting problem, can the performer nominate alternative verification criteria? Per Section 1.A of the BAA, "At months 1, 9, 21, and 33, each performer will nominate 50 real-world, unclassified forecasting problems and the references for verification, to be used in IARPA's assessment of all performers." Performers will nominate verification references but IARPA will choose which references are used. 08/09/10
029 What does it mean that "elicitation methods should allow . . . continuous updating of forecaster judgments and rationales"? The methods should allow forecasters to provide a new judgment and/or rationale at any time. 08/09/10
030 Is an organization precluded from serving as a subcontractor to multiple (meaning more than one) prime(s) bidding on this BAA? No. Per Section 6.B.8 of the BAA, "IARPA neither promotes, nor discourages the establishment of exclusive teaming agreements within offeror teams. Individuals or organizations associated with multiple teams must take care not to over-commit those resources being applied." 08/09/10
031 Section 4.B.2 indicates that IARPA is requesting a CPFF type contract or other type of procurement (specify). This seems to indicate that the Offeror may request a different contract type for their proposal. Please verify this is accurate. Per page 3 of the BAA, procurement contracts are the only types of instruments that may be awarded. Per Section 4.B.2 of the BAA, offerors should specify the type of procurement contract requested: "(9) Award instrument requested: cost-plus-fixed-fee (CPFF), cost-contract—no fee, cost sharing contract – no fee, or other type of procurement contract (specify)." Per Section 2 of the BAA, "the Government reserves the right to negotiate the type of award instrument it determines appropriate under the circumstances." 08/09/10