Reynard Program - BAA Questions

#QuestionAnswerDate Posted
001 What type of contract is expected? See BAA Part One (Overview Information) and BAA Part Two, Section IV.A.2.b (Format of Volume II, Cost Proposal) for a list of possible contract types. The type of contract for each successful proposal will be the most appropriate for that proposal. 05-01-09
002 What is the rough order of magnitude of expected contract awards? See BAA Section V.A.5 (Cost Realism). There is no predetermined award size for Reynard proposals. Proposers should submit proposals that will result in successful research efforts within the BAA criteria and timeline. Proposals should advance compelling research ideas based on sound science that respond to the stated challenges and identify how much they will cost. 05-01-09
003 Are Reynard proposals restricted to the broad definition of culture given in the BAA (e.g. region of the world), or would you consider finer gradations of culture especially with respect to group dynamics? See BAA Section I.A (Program Overview). Researchers may empirically discover that some RW characteristics, such as culture, are able to be deduced at greater (or lesser) levels of discrimination than described in the example. 05-01-09
004 Can you tell us the composition of reviewers on the proposal evaluation team? Reviewers will be experts in disciplines relevant to this BAA. 05-01-09
005 What if a proposed project attempts to discover real world (RW) properties of groups or communities (vs individuals) from VW behaviors and indicators? See BAA Section I.A (Program Overview). Behavioral Indicators (BIs) of VW groups and communities that relate to RW characteristics of individuals, groups and cultures would be within scope of Reynard research. 05-01-09
006 You say that you are not specifically interested in how RW behavior is influenced by VW experience, but if we focus on the correlation rather than the mechanism of the adaptive behavior-Is that in scope? See BAA Section I.A (Program Overview). Reynard research goals are to determine behavioral indicators in the realm of Massive Multiplayer Online Games (MMOGs) and VWs that are predictive of RW characteristics of the users. 05-01-09
007 Can you address the proposal format, forms and length of proposal? See BAA Section IV.A.2 (Proposal Format). 05-01-09
008 We have a University Affiliated Research Centers (UARC) on our campus. Can other campus organizations, not affiliated with the UARC participate? Yes, other campus organizations not affiliated with the UARC may propose to the BAA. Only members of the specific UARC sponsored organization are prohibited from proposing. Individuals who hold joint appointments at both the UARC and the university are prohibited from proposing. See BAA Section II.A.1 (Procurement Integrity, Standards of Conduct, Ethical Considerations, and Organizational Conflicts of Interest (OCI)). 05-01-09
009 Can you further clarify "special relationship with the government," and the prohibition against companies that have access to Government property, e.g. Government Furnished Equipment (GFE)? See BAA Section III.A (Eligible Applicants). Organizations such as Federally Funded Research and Development Center (FFRDCs) and University Affiliated Research Centers (UARCs) have access, beyond that which is common in a normal contractual relationship. This special relationship frequently entails access to supplier and/or proprietary data, Government employees, installations, equipment, and property. Proposers who are not categorized as either an OGA, FFDRC, or UARC may submit a proposal even if they have been issued Government Furnished Equipment (GFE) on other Government programs. Per BAA Section III.A.1, proposals should always be mindful of Ethical and Organization Conflicts of Interest (OCI) issues associated with their activities. 05-01-09
010 Do you intend to down select after Phase I? See BAA Section II (Award Information). Subject to the availability of funds, participants in the Option Period (Phase II) will be those teams that have made significant progress in the Base Period (Phase I) and have correctly understood and contributed to the overarching goals of the Program. Teams that fail to make sufficient progress in achieving Phase 1 metrics will not be invited to continue with the Program. 05-01-09
011 If someone has a general capability such as natural language processing, could they participate on multiple teams? See BAA Section II (Award Information). Yes. There is no limit to the number of sound proposals that can be submitted by a team. Furthermore, IARPA neither promotes, nor discourages the establishment of exclusive teaming agreements within proposer teams. Individuals or organizations associated with multiple teams must take care not to over-commit those resources being applied. 05-01-09
012 What is the relationship between Reynard and the recent SCIL BAA? Are you interested in leveraging potential dual-purpose analytical techniques, or are you interested in differentiation between the programs? Reynard and the Socio-cultural Content in Language (SCIL) Program are not directly related. However, both programs may share insights and results as appropriate. 05-01-09
013 Why are you not interested in web usage patterns and query styles from an intelligence perspective? See BAA Section I.A (Program Overview). The program scope is limited to the study of VWs. 05-01-09
014 Has IARPA discussed the Reynard BAA with any specific VW or MMOG vendors? Are they open to the research being conducted for the effort? Reynard researchers are responsible for gaining vendor cooperation and access to VWs as required for their proposed studies. 05-01-09
015 Will there be a platform for sharing resources and findings between teams? Yes. 05-01-09
016 Will IARPA directly issue contracts or will you be using an external agent for contract administration. IARPA anticipates using a DoD organization as the contracting agent for Reynard. 05-01-09
017 If research findings lead to an effort possibly being classified as it proceeds further, will it be disqualified for future funding? See BAA Section VI.B.1 (Security). All Reynard research is intended to be unclassified and results will be publicly releasable. No classified information will be accepted in response to this BAA. 05-01-09
018 Will the BAA spell-out requirements for supporting validation of results? BAA Section I.A (Program Overview) provides guidance for estimating resources to support NIST technical assessments. 05-01-09
019 Will this effort use 6.1-Basic Research), 6.2-Applied Research, or 6.3-Developmental Research funding? The Reynard Program will sponsor fundamental applied research. 05-01-09
020 What is the due date for proposals? See BAA Section IV.B.1 (Proposal Due Date). Proposals are due June 16, 2009 at 5:00pm ET. 05-01-09
021 What could be the motivation for grown-ups with real responsibilities to participate in virtual worlds? If more, younger individuals participate in general, will not the experiments be very statistically biased? See BAA Section I.A (Program Overview). Proposers are highly encouraged to consider study of behavioral indicators across a demographically heterogeneous set of VWs as opposed to a single demographically homogeneous VW. 05-01-09
022 Do you expect algorithm development in this effort? See BAA Section I.A (Program Overview). Algorithm development is not a deliverable for this effort. Reynard researchers may collaterally develop algorithms to support their data collection and analysis as part of their research methodology. 05-01-09
023 What is an extant context? For example, is observation of students engaged in coursework activities relevant to your interests? See BAA Section I.A (Program Overview). Observation of student activities in VWs or MMOGs is relevant to the extent that activities and subject demographics meet Reynard research criteria and goals as stated in the BAA. Proposers should also consider study of VWs that have a considerable concentration of non-US players. Proposals that examine VWs whose player populations are primarily US-based will receive lower evaluation ratings. 05-01-09
024 Do proposals need IRB approval before submission? See BAA Section VI.B.5 (Human Use). IRB approval is not required prior to submission. However, for all proposed research that will involve human subjects in the first year of the program, the institution must provide evidence of or a plan for review by an Institutional Review Board (IRB) upon final proposal submission to IARPA. The IRB conducting the review must be the IRB identified on the institution's Assurance of Compliance. No IARPA funding can be used towards human subject research until ALL approvals are granted. 05-01-09
025 In many MMOs, the RW gender ratio is about 80:20. Thus, any universal VW behavior (such as having a name with at least one letter) would predict whether a player is male at an accuracy of 80%. One solution might be to increase the target metric for these cases. On the other hand, due to the high base rate of males, it might also be difficult to provide a prediction model that outputs a probability value higher than the base rate. Or for example, consider a skew in another RW variable where there is a 95:5 base rate. It would be hard to outdo 95%. Can IARPA provide guidelines for how to deal with such cases?

The Reynard Program Manager recognizes that some RW characteristics will be easier to identify, such as gender or approximate age. For that reason, the Program has different target metric values for different types of RW characteristics; see BAA Section I.B (Program Milestones and Metrics). It is anticipated that gender and age RW characteristics will be easier to empirically determine, and so have been set with higher target metric values. Proposers should describe, to the extent that the information is known, the base RW demographics of the VW populations being researched, so that the utility of the quantitative Behavioral Indicator can be determined. See BAA Section V.A.1 (Overall Scientific and Technical Merit).

Proposers should also propose additional metrics for assessing progress of their research. See BAA Section I.B (Program Milestones and Metrics).

05-01-09
026 Due to differences in base rates and other parameters, it is possible to arrive at a scenario where we could predict whether a player is male at 75% likelihood, but whether a player is female at only 25% likelihood. When the BAA states the target metric for gender, does "gender" mean both male AND female (i.e., the average probability of the two), or is it sufficient to predict either male OR female at 75% accuracy? And does that answer also extend to the other characteristic categories (such as age-is it ok if we can only predict minors well or is it necessary to predict all age groups)?

It is highly desirable to reliably infer both genders, and all age groups. It may not be possible for one BI to permit positive inference about both genders (or all age groupings), in which case a different BI may be required for each.

See BAA Section I.B (Program Milestones and Metrics). Proposers should also propose additional metrics for assessing progress of their research.

See also BAA Section V.A.1 (Overall Scientific and Technical Merit).

05-01-09
027 The BAA gives the example of Russian flags in profiles indicating Russian nationality (pg. 12), but consider the following scenario: In Eve Online, there are 1000 Russian players. 80 of them display a national Russian flag in their profile. 20 other players (of other nationalities) also display a Russian flag in their profile. Thus, this scenario supports the claim that "in Eve Online, 80% of players who display a Russian flag in their profile are of Russian nationality"; however, this behavior indicator only identifies 8% of all Russian players in Eve Online. Given this scenario, does the stated example exemplify the desired outcome of Reynard?

The RW demographics of the entire base population of a VW may be unknown. To the extent that the demographics are known, the researcher should report on them. See BAA Section I.A (Program Overview).

Behavioral Indicators should be of the form: Within the context of particular VW(s) W, there is a X% probability, given a character exhibiting VW behavior Y, that they are Z demographic in RW.

05-01-09
028 Machine learning approaches, while able to provide accurate predictions, oftentimes employ black box solutions. For example, we might develop an algorithm that can predict whether a player is male in RL, but not be able to articulate how the algorithm does what it does. Given the emphasis of the Reynard program on finding mappings between VW behaviors and RL traits, are black box solutions out of scope? Black box approaches should not be employed. Algorithms may be employed in the analysis of data, but such algorithms should be logically and clearly explained. 05-01-09
029 Could you explain more about the criteria for validation and what data will NIST use in the validation process? See BAA Section I.A (Program Overview). IARPA intends to engage the National Institute of Standards and Technology (NIST) as a neutral third-party who will provide independent technical assessment and feedback to the Reynard Program Manager regarding research methods and data analysis techniques. This Assessment Team will be comprised of individuals with expertise in social science research methods and statistical analysis. The Assessment Team will accompany the Program Manager on all site visits and will attend all program reviews and program workshops. The Assessment Team will require access to research designs, data gathered for the research, and the analyses generated from the research. All materials will be retained by the Assessment Team for no longer than 12 months after the conclusion of the Period of Performance for the contract. The Assessment Team will employ primarily inspection methods to the materials, but might re-analyze data in some cases. 05-01-09
030 Do you envision a small number of projects going to large teams or a mixture of large and small projects with large and small teams? See BAA Section II.D.1 (Collaborative Efforts). There is no preference for large or small teams, or teams led by large organizations or small organizations. 05-01-09
031 Would you consider research in a VW that was persistent, massive, had goals, self-representation, but no graphics? E.g., a VW that enabled economic exchanges but without graphical representation of players. See BAA Section I.A (Program Overview) for the Virtual World Definition used for this Program. Proposals that study VWs that do not meet the definition in the BAA may be considered non-compliant. 05-01-09
032 MMOG or Internet cafe? Are both acceptable places to perform research? See BAA Section I.A (Program Overview). Both are acceptable, individually or in combination. 05-01-09
033 Can you say more about demographically heterogeneous VWs? Should studies include more than one VW? See BAA Section I.A (Program Overview). Proposers should consider study of behavioral indicators across a demographically heterogeneous set of VWs as opposed to a single demographically homogeneous VW. If VW behavior is influenced by RW characteristics, might some of these behavioral indicators be observable across different VWs? 05-01-09
034 Will Reynard participants require security clearances? See BAA Section VI.B.1 (Security). No. All Reynard research and deliverables will be UNCLASSIFIED. 05-01-09
035 Will there be any limitations on academic publication and presentation of the result of Reynard funded research? See BAA Section VI.B.6. (Publication Approval). 05-01-09
036 Heterogeneous environments could be different games built on the same gaming platform, right? See BAA Section I.A (Program Overview). The Reynard Program defines heterogeneity from the standpoint of the demographics of the user population, not the gaming platform per se. 05-01-09
037 My collaborators and I have been working on studying social interactions using dynamic facial expressions. It is not an immersive environment like virtual worlds, but does offer some additional experimental control and quantification that are difficult in virtual worlds. I have attached a short description of our approach, and would like your opinion on whether this is something that would be considered. See BAA Section I.A (Program Overview) for the Virtual World Definition used for this Program. IARPA will not review abstract or white paper submissions to the Reynard program. Only full proposals that are submitted following the process outlined in Section IV will be accepted. Research ideas not related to the Reynard Program scope may be submitted to the IARPA-wide BAA. Please read and follow submission instructions. 05-01-09
038 The eligibility information outlined in section III of the BAA disallows direct participation by Government-related organizations. Given this restriction, will IARPA accept a US Government scientist (for example, one employed by NASA or ARL) taking an advisory role on a team in an individual capacity? Such a person might act as an external scientific reviewer, and neither the individual or their organization would have access to any Reynard data or other resources. A government employee may not serve as a scientific advisor to a Reynard proposer. 05-01-09
039 Would you be interested in a white paper on the use of AI in a MMOG environment to evaluate group formation, dynamics, and tactics as a predictor of tactical team skills in the RW? See response to question #37. 05-29-09
040 May a participant be included in more than one proposal as either PI, CO-PI, or senior personnel? See response to question #11. 05-29-09
041 If a team consists of industry and academic partners, and none of the industry partners has its own IRB, can they use the IRB of their academic partner to acquire their Assurance of Compliance? If not, can you please suggest another process for acquiring this assurance by organizations that do not have their own IRBs? Industry-led teams may use the Institutional Review Board (IRB) of their academic partners. If a team doesn't have access to an academic IRB, they may use a certified commercial IRB, or local government IRB. 05-29-09
042 The cover sheet asks for "Team Member(s)/Organization" and "Name(s) of Each Team Member." If I provide "John Smith/ACME Inc." to fill the first field, what do I need to provide that is different to complete the second field? The first field is for the organization name and the second is for individual names. In your example, the first field would be filled with "ACME Inc" and the second field would be filled by John Smith. This format eliminates the need to list organization name for every member in the same organization. 05-29-09
043 1) Can you confirm whether the following is the correct interpretation of the statement about validation of RW characteristics on page 10 of the BAA: Self-report data is not sufficient. Self-report data has to be confirmed via some other source due to online deception issues etc.
2) If (1) is the correct interpretation, how strict is this requirement? Should teams only include RW characteristics that they are able to validate? Or is it ok to include high interest variables even if complete validation is not planned (due to prohibitive cost or impracticality)?
3) What would validation mean in terms of the inherently subjective variables such as ideology or worldview? Are self-report survey scales sufficient in these cases?
See BAA I.A Program Overview, pages 9-10. As discussed in the BAA, all research methods have inherent strengths and weaknesses. One technique for dealing with this is to use multiple research methods, but other techniques may also be used. Researchers should clearly articulate the strengths and weaknesses of their chosen methods, and how they intend to mitigate the risks associated with the weaknesses of particular research methods. 05-29-09
044 The BAA states on page 37, "The Reynard program is envisioned as a 3-year effort that will begin in the fourth quarter of Fiscal Year 2009." Does that mean a start date of 1 October, 2009, or 1 July 2009? See page 16 of the BAA. The exact start date of the program is undetermined and will be influenced by the length of time required for proposal review, selection, and contract negotiation. 05-29-09
045 Could you please offer some clarification of what is meant by the following:
"individual differences in the use of currency" (Page 6 of BAA)
and
"handling of VW currency" and "treatment of VW goods and currencies" (Page 8 of BAA)?
Individual differences in the use of VW currency might be found that could be attributable to demographic factors such as age, gender, education, etc.

"Handling" or "treatment" of VW goods and currencies might include economic behaviors such as buying, selling, trading, or saving, both in the VW and in external markets.
05-29-09
046 My proposal emphasizes the characteristics of user-generated avatars using platforms such as "Spore" and "Little Big Planet", corresponding to the "Avatars and Representation" and "Things Avatars Do" features stressed in the "Program Research Focus" section of the solicitation. However, "Spore" and "Little Big Planet" lack the "persistent", "shared", and "massive" characteristics described in the "Virtual World Definition" section. Is my focus on avatar customization instead of multi-user virtual character interaction adequately relevant to the Reynard program? See response to question #31. 05-29-09
047 Please clarify/confirm if Volume I, Section I, which consists of the cover sheet and transmittal letter, are included in the 30 page total proposal page limit? See BAA Section IV.A.2 Proposal Format. "Except for the cover sheet, transmittal letter, signed Academic Institution Acknowledgement Letter(s) if required, OCI waiver/certification, bibliography, and relevant papers, Volume I shall not exceed 30 pages. " As stated, the cover sheet and transmittal letter are not included in the 30 page limit. 05-29-09
048 The Reynard CFP (p. 25: Volume I, Section IV) asks that this form be completed:
https://www.dss.mil/GW/ShowBinary/DSS/isp/
foci/documents/sf328.pdf

I am coordinating a multi-university proposal. Does each sub-institution need to complete this form, or only my primary one?
See BAA Section IV.A.2.a Format of Volume I, Technical and Management Proposal {Limit of 30 pages}. As stated on page 25:
"It is highly recommended that offerors submit with their proposal a completed and signed Standard Form 328 (SF328), Certificate Pertaining to Foreign Interests, hereafter referred to as the Foreign Ownership & Controlling Interests (FOCI) document, for each entity that is part of their team, whether serving in the role of prime, subcontractor, or consultant at any tier of their team."
05-29-09
049 Can we include the CVs of all key personnel as an appendix to Volume I, Section IV (additional information)? Could this information be included without contributing to the page count of Volume I (similar to the three relevant technical papers)? Would this information be helpful to the reviewers?

See BAA section IV.A.2.a Format of Volume I, Technical and Management Proposal {Limit of 30 pages}:

"...offerors will provide brief biographical sketches of key personnel and significant contributors..."

These biographical sketches are included in the 30 page limit.

Volume I Section IV Additional Information is reserved for the bibliography, copies of not more than 3 relevant papers, and the completed FOCI form. These documents are not included in the 30 page limit.

06-05-09
050 The proposers' day documentation states that "development of new interface mechanisms, such as hardware peripherals that are alternatives to the keyboard and mouse" is out-of-scope. Does this rule out processing of audio data, coming from a player's headset, as shown on p. 16 ("understanding cultural differences")? Also, is machine translation considered "artificial intelligence", i.e., would the offering of a machine translation service be considered an out-of- scope enhancement of the game? See BAA section I.A Program Overview. The Reynard Program includes the study of verbal communication. Reynard researchers may collaterally develop tools to support their data collection and analysis as part of their research methodology. However, such tools are not the focus of the research. 06-05-09
051 We have foreign participants (entities) as part of our team; are they required to complete the Standard Form 328 (SF328)? Yes 06-05-09
052

We have a strong interest in this research area and have only recently identified potential team partners; therefore, the 16 June date for a quality proposal is not achievable. The BAA indicates, "IARPA may evaluate proposals received after this date for a period up to one year..."

Taking your funding profile into consideration as well as the time required for evaluating those proposals submitted on 16 June, do you have a recommendation for an appropriate time period to submit a proposal AFTER the 16 June date?

See BAA section IV.A.1 Proposal Information.

"Offerors are required to submit proposals by the time and date specified in section IV.B.1 in order to be considered during the initial round of selections. IARPA may evaluate proposals received after this date for a period up to one year from the date of initial posting on FedBizOpps. Selection remains contingent on availability of funds."

06-05-09
053 When a researcher is engaged with people via a virtual world environment do they need to follow the Human Use requirements in Section VI.B.5 for their research protocols? Yes 06-05-09
054 Could you clarify the eligibility of individuals affiliated with Federally Funded Research and Development Centers (FFRDCs) or University Affiliated Research Centers (UARCs), to propose to or participate as team members under this BAA.

Individuals who are current employees of or have appointments with FFRDCS/UARCs, are prohibited from proposing or serving as members of a proposal team.

Individuals who are or were engaged with an FFRDC/UARC in some other capacity (for example: as a consultant or as a former FFRDC/UARC employee) MAY be eligible to compete under this BAA. However, their ability to compete under this BAA is subject to the Organizational Conflict of Interest requirements in BAA Section II.A.1 (Procurement Integrity, Standards of Conduct, Ethical Considerations, and Organizational Conflicts of Interest (OCI).

06-05-09
055 If an Academic Institution member contributions are restricted to the IRB, is an Academic Acknowledgement Letter from the Institution required?

See BAA section III.B U.S. Academic Organizations.

If funds will be provided to the academic institution for its provision of an IRB, then an Academic Acknowledgment letter is required before IARPA can enter into any negotiations.

06-10-09
056 Do universities need to identify in their proposal, each graduate student who is a foreign national that may support Reynard?

See BAA section IV.A.2.a.F

"F. A clearly defined organization chart for the program team which includes the following required elements: (1) the organizational affiliation of each team member; (2) the teaming strategy among the team members, including team roles, (3) the unique and relevant capabilities of team members; (4) the tasks or contributions of team members; (5) the amount of effort to be expended by each person during each year; and (6) the country of citizenship of team members."

If graduate students are currently proposed as members of the team, they should be listed in the organization chart.

06-10-09
057

I have a few IRB-related questions about the BAA 09-05 submission (per pp. 33-34 of BAA):

1. Providing documentation of Federalwide Assurance (FWA): Does this mean including a letter within the proposal, or providing such documentation prior to the award being made?
2. Evidence of or a plan for IRB review: Does a letter from the Institutional Official, indicating that IRB review is pending, suffice?
3. "The protocol, separate from the proposal": By protocol, do you mean the protocol that will be submitted to the IRB, and not the Human Use section of the proposal, then? Do you want the protocol attached to the proposal?
4. Evidence of IRB approval: Although proof of IRB approval is not mandatory with the proposal submission, should this be submitted anytime after the submission? Or should we wait for it to be requested?

See BAA section VI.B.5. See also response to question #24.

1. FWA documentation must be provided prior to contract award.
2. Yes
3. The "protocol" is the documentation submitted to the IRB. If human studies are to be conducted in the first year of funding, proposers must submit evidence of or a plan for review of their protocol by an IRB, at the time of proposal submission. Offerors should read section VI.B.5 carefully and allot sufficient time for both their local IRB approval and the DOD review and approval.
4. Evidence of IRB approval should be submitted as soon as the approval is obtained. No IARPA funding can be used towards human subjects research until ALL approvals are granted.
06-10-09