Solicitations - Office of Safe and Secure Operations

Securely Taking On New Executable Software of Uncertain Provenance (STONESOUP) Program - BAA Questions

IARPA-BAA-09-08
BAA Release Date: September 16, 2009
BAA Question Period: September 16, 2009 - October 19, 2009
Proposal Due Date: November 2, 2009
FedBizOpps Reference

The questions and answers below summarize those received by the Program. Proposers are encouraged to check this list periodically and carefully review the BAA as well as the Proposers' Day presentations for answers to their questions.

# Question Answer Date Posted
001 Will the proposal submission address listed in the BAA accept deliveries via regular mail from the United States Postal Service? No. As stated in Section 4.C.2 of the BAA, "Deliveries must be made using one of the following commercial delivery services: UPS, FedEx or DHL." September 23, 2009
002 Section 4.C.2 (Proposal Delivery) asks for "one original hard copy with original signatures." By signatures do you mean the signatures of all proposal contributors or just the PI? "Original signatures" pertains to:
- Submitter's transmittal letter
- Academic Institution Acknowledgement Letter
- OCI Certification Letter
The transmittal letter and OCI certification letter should be signed by an appropriate official of the prime organization. Each academic institution that is part of a team must complete an Academic Institution Acknowledgement Letter, signed by an appropriate official of the institution, and included with the prime's submission.		 October 15, 2009
003 It would appear that our small business would not be a prime contractor on a project of this magnitude, but we would like to provide some components to one or more teams. We have already sent an overview, similar to the information above, to each of the contacts listed on the Proposers Day list, but we would like to know if there are any other processes for us to contact potential prime contractors and team mates. IARPA does not provide any other processes for contacting potential prime contractors or team mates. October 15, 2009
004 It appears from the BAA that the forms provided on the STONESOUP web pages are meant to be completed by the prime contractor; if that's incorrect could you clarify this for us? The prime organization is responsible for submitting all required forms. However, each academic institution must complete its own Academic Institution Acknowledgement Letter, regardless of whether it is a prime or subcontractor, which should be included in the prime's submission. October 15, 2009
005 With regard to Table 4 from section 1.B of the BAA, could you please define if the Principal Investigators and Key Personnel required to attend are only from the prime offeror or also from any sub-contractors involved? Attendance is required of at least one PI from the Prime organization, plus any team member the Prime determines is necessary for proper representation of the effort. October 23, 2009
006 Also, from section 6.B.4. Meeting and Travel Requirements, there is a reference to Program workshops. No workshops appear on Table 4. Are there requirements beyond Table 4 or are the workshops within Table 4 meetings? "Workshops" is a reference to Program Reviews as described in Section 6.B.4.a. October 23, 2009
007 On page 24 of the solicitation, 4.B.2 Section 2: Detailed Cost Estimate Breakdown, indicates that the cost breakdown should include major program tasks by fiscal year. Does "fiscal year" mean Government Fiscal Year or should we be using our own fiscal year? "fiscal year" means Government fiscal year October 23, 2009
008 4.C.2 Proposal Delivery. Does the proposal need to be in the hands of Dr. Carl Landwehr at 5 p.m. local time or does it just need to be received at the mail facility by 5 p.m. local time? The proposal must be received at the mail facility at or before 5 p.m. local time October 23, 2009
009 6.B.6 Publication Approval. Am I correct in understanding that the only time pre-approval for publication is anticipated to be required is if the release may result in the disclosure of sensitive intelligence information? Other than that there is no requirement pre-approval for publication anticipated. Would the requirements be any different for universities? Pre-publication approval of information produced in this program will only be required if the Government determines that its release may result in the disclosure of sensitive intelligence information. Note that a courtesy soft copy of any work submitted for publication should be provided to the IARPA Program Manager and the Contracting Officer Representative (COR) at the time of submission. Requirements are the same for academic institutions. October 23, 2009
010 What is the target Technology Readiness Level (TRL) at the end of the project? (On the one hand, the evaluation plan suggests a high TRL; on the other, the tool could have a lower TRL --- say it's clunky to install --- but pass evaluation.) Technology Readiness Levels will not be used in the STONESOUP Program. The Government will evaluate the effectiveness of proposed solutions as described in Section 1.B. October 23, 2009
011 When is "reject" an acceptable or expected response? When the program is deemed "too dangerous" or "too hard to process"? "Reject" in this context means "too hard to process". See also Answer 17. October 23, 2009
012 Does the allowable slowdown (e.g., 10%) apply just to normal execution? Could execution during an attack potentially incur a higher overhead? The performance target of no more than 10% increase in the running time of a processed program relative to the unprocessed version of the same program applies to any execution environment. October 23, 2009
013 Is VMM use expected as part of the solution to containment? The choice of specific containment technique is up to the offeror. October 23, 2009
014 For class C, is there any characterization of the space of executables that should be protected? It seems likely that they will be desktop applications. Will there be real-time constraints? Could executables in class C include "interpreters," such as the JVM, or are they covered under class A and disjoint from class C? Will the executables be tamper proofed (as some COTS products are)? There are no additional characterizations or constraints beyond what is described in section 1.B of the BAA. Regarding interpreters, addressing vulnerabilities in the interpreter code itself would fall under Class C, while addressing vulnerabilities in the code the interpreter is executing would fall under class A. October 23, 2009
015 Would the vulnerability Set C5 (SQL injection /command injection (e.g., CWE #78, CWE #79)) for Language Class C be considered covered if the provided solution did not address SQL injection, but did render unexploitable binary machine code injections? Within a Vulnerability Set, multiple test cases may be developed to ensure adequate coverage for test and evaluation. A solution proposed against a particular Vulnerability Set may be measured against any test case for that set. October 23, 2009
016 The solicitation is very explicit that all techniques should be automatic. Yet, some valuable containment techniques (e.g., those that disallow run-time code generation) will stop some legitimate applications (e.g., a JVM using just-in-time compilation). Is it reasonable to offer a dialog-box warning in such cases, or is that insufficiently automated? A solution that relies on human intervention does not meet the automation goals of STONESOUP. October 23, 2009
017 In the end-of-phase evaluations against the requirements, we have a question. If the tool discovers a problem with the program being analyzed that it *knows* is unsafe and can't run in a safe way, can we reject it outright? For example, a trivial program that accepts network connections and passes the input as is to a command shell or a database cannot be safe without some inferred semantics. For type safe languages, tables 1, 2, and 3 require that 100% of all programs be processed - does this imply that the tool cannot say "this program is simply unsafe and there's not enough semantics to decide how to run it safely" without failing that 100% target? When a tool is run on a particular program or system of programs, there are three possible outcomes:
A. The tool attempts to process the input program but fails -- perhaps it cannot make sense of a binary or it recognizes a construction that it is unable to remedy. The tool terminates its operation with no output and an error message.

B. The tool processes the input successfully but fails to render a vulnerability in the input unexploitable.

C. The tool processes the input successfully, and generates a modified program as output that renders the vulnerabilities in the input unexploitable.


Case A corresponds to a rejected program. Cases B and C both count as processed programs, but only Case C is a fully successful outcome.
 October 23, 2009
018 Are Phases 2 and 3 formally priced options? Are offerors required to submit fully detailed cost information in support of Phases 2 and 3, or just rough orders of magnitude (ROMs)? Yes, Phases 2 and 3 are formally priced options. See Section 4.B.2, "Section 2: Detailed Estimated Cost Breakdown." October 23, 2009
019 Section 3.A. (Eligible Applicants) states that the lead PI must be a "US person". How does this criteria impact an individual who has dual citizenship (US/Israel)? An individual who is a U.S. citizen, even though holding some other citizenship, meets the U.S. Person requirement for being lead Principal Investigator (PI) under Section 3.A. 16. October 23, 2009
020 At the beginning of each test phase, IARPA is going to provide sample programs. Will they also include (a) a description of all of the known vulnerabilities in the programs and (b) sample exploits for the vulnerabilities in those programs? (a) For the smaller test examples provided, we intend to identify the known vulnerabilities, but for larger test samples we will not necessarily be able to do that.
(b) No, we will not provide code to exploit vulnerabilities.
 October 23, 2009
021 The BAA encourages proposers to consider alternatives to fail-stop handling of errors detected at run-time, but the evaluation metrics only consider soundness, completeness, and performance. Will the project evaluations have any component that looks at other factors, such as graceful error handling? The milestones and metrics described in Section 1.B are the primary means that the Program Manager will use to assess progress. October 23, 2009
022 Sometimes it is impossible to statically determine whether a run-time event is an error or not. For example, some integer overflows are deliberate (e.g. in hash function computations, CRCs, encryption, etc, that do arithmetic mod a power of 2), but others are exploitable bugs. One way to distinguish such cases without programmer annotations would be to run the instrumented program on a known-good test suite. Will the evaluation programs have test suites so that we can do profile based analysis and instrumentation? Yes, the Government intends to provide test cases with known good code. Note that only some test cases will be made available to performers at program kick-off, while others will be reserved for testing at the project milestones (see Section 1.A, "Assessment"). October 23, 2009
023 Are tools expected to prevent all instances of the indicated vulnerability classes or only exploitable instances? If the answer is the latter, are tools penalized for preventing non-exploitable instances? For example, suppose an integer variable overflows in a monitored execution, but its value is not subsequently used in that execution. Would a tool that takes action (e.g., stops the program) in that execution be penalized? Would a tool that takes no action be penalized? Integer overflows that do not impact normal execution should not halt execution. Therefore, a tool that halts execution in the presence of "non-exploitable instances" would be penalized. A tool that allows normal execution under the same conditions would not be penalized. October 23, 2009
024 The described Section 2 (Summary of Proposal) is intended to be somewhat longer than typical proposal summaries (1-2 pages). Is that a correct assessment? There is no specific length intended or required. This is up to the offeror. The overall length of Volume 1 (of which Section 2: Summary of Proposal, is a part) is limited to 30 pages, as specified by 4.B.1 of the BAA. October 23, 2009
025 The BAA indicates that a start date of March 2010 is intended. Is this to be March 1, 15, 31, or some date in between? Would it create a problem if we used an April 1, 2010 start date keeping within the parameters of 18 months, 12 months, 18 months? For proposal purposes, assume a start date of 1 March 2010. Actual start date will depend on the results of contract negotiations. October 23, 2009
026 The BAA also states that Phase I is to be 18 months. Is it intended that you receive 3 budget pages (18 months, 12 months, 18 months) or that the budget be further separated for a period of 3/1/10 through 9/30/10 and then pick up on 10/1/10 through 9/30/11 for Phase I? With a March 1 start date that would make the first part of the Phase I budget 7 months long and the 2nd part 11 months long with an end date of 8/31/2012. Phase II would then be 9/1/12 thru 8/31/13, and Phase III would be 9/1/13 thru 2/28/14 (again 7 months & 11 months). Since the solicitation also indicates that the budget must be broken by task & phase and further that you want Major Tasks by Fiscal Year, this makes budgeting a little difficult, especially since there isn't any detail pertaining to how you want to see the budget. For the major program tasks by fiscal year did you simply need a page showing Task XX to be completed during FY10, Task YY to be completed FY11, etc. or did you need detailed budget line item associated with each task? The base and option periods should each be priced separately. As stated in Section 4.B.2 of the BAA, the Cost Proposal (Volume 2) must provide supporting cost and pricing information in sufficient detail to substantiate the summary cost estimates in Volume 1. The same section itemizes the required budget information. October 23, 2009