Wargame

As discussed above, several data streams with biological and medical information are available to policymakers. Individually, these data streams provide varying contributions to addressing the issues of specificity, sensitivity and timeliness. In general, a review of the scientific literature suggests the analytical work done to evaluate their true utility to policymakers has been lacking. One recent research paper concluded: “Most evaluations of detection systems and some evaluations of diagnostic systems for bioterrorism responses are critically deficient.” 77 Indeed, the JASON study commented that one major exercise conducted in 2002—designed in part to test the technical capabilities of a set of biosensors—seemed “…to be little more than a demonstration of currently funded programs.” 78

In light of these shortcomings, this study was conducted as a “first-cut” evaluation of a system-of-systems approach. Rather than relying on just one data stream for information about a possible attack, this study has devised an integrated system with ten separate data streams and uses them in a modified wargame scenario to allow participants—who act as advisors to policymakers—to evaluate the value of the various reporting systems. Using a statistical analysis technique known as the analysis of variance (ANOVA) ,79 the results of the wargame allow us to recommend suggested combinations of data streams—a system-of-systems—that would provide the highest value data to policymakers.

The wargame is concerned only with the release of a biological agent (man-made or naturally occurring) and thus focuses on the use of the data to indicate an outbreak of disease. Thus, it assumes a system-of-systems operating as detect-to-treat; although, it attempts to provide knowledge of an incident at the earliest possible time.

A depiction of the system-of-systems is shown in figure 20. In this graphic the various data sources are collated and subjected to analysis using a variety of proprietary and public tools and results in several outputs that are provided to policymakers. The generic, and in some cases specific, data fusion requirements have been reviewed by several researchers and are discussed in the publications of Lober, et al. (2002) 80 and Tech, et al. (2002). 81

Figure 20 – A system-of-systems approach to biomonitoring.

All of the proposed systems have been subjected to individual critical analysis and have generated claims about the specific performance characteristics that they provide. Against this background of claims about system performance in terms of sensitivity, specificity, range of agents detected, cost, utility, etc., there is a possibility (some would say good probability) that any selection process will discard a promising technology on the basis of limited value to decisionmaking. Moreover, a competitive analysis does not easily allow for the fact that the solution may best result from a combination of sensor systems or system-of-systems. Competitive “down-selection” is also subject to considerable influence by the assessors – a system that reduces bias and that incorporates the rigor imposed by statistical analysis would appear to offer some advantages.

The wargame requires each respondent to use a pre-specified scoring system to assess the attributes of a particular system, or combination of systems. Two different scenarios are used to describe bioincidents—one with a known agent and one with an unknown agent. The respondents were randomly assigned to one of eight survey groups and asked about his or her strength of belief in statements concerning the sensor system under review.

The Wargame

The data provided to each of the players in the wargame is provided in Appendix A. For analysis, each of the ten components of the system-of-systems (see figure 20) was placed into one of three groups, based on their overall “theme.” The Business Group consisted of Medical Claims Reporting, Pharmacy OTC Sales, and Absenteeism Reporting. The Medical Group consisted of Medical Surveillance Reporting, Veterinary Surveillance, Agricultural Reporting, and Nurse Help Line Calls. The Positive ID Group included Laboratory Reporting, Sensor Reporting, and Prodromic Reporting.

Each group and combination of groups was evaluated with respect to utility, trustworthiness, and resource requirements. (See Appendix B for full discussion of the experimental design and statistical analysis.)

Table 4 presents a simplified version of the ANOVA results (see Appendix B for the complete analysis). In table 4 the groups are shown in rank order, as judged by the wargame respondents. These are compared against a baseline. If a group—or combination of groups—is not listed, then it was not statistically significant from the baseline.

Veterinary Surveillance
Scenario 1 (Known Agent)

• Experts find the combination of all groups—Business+Medical+Positive ID—to have the most utility; more data are better than less.
• The Positive ID Group is viewed as the most trustworthy set of data, i.e., a sensor report that can confirm the identity of the agent is valuable.
• The combination of Medical+Positive ID Groups is the most demanding of resources; it will cost the most to operate.
• Overall, the combination of data from all three groups is preferred. Again, more data are better. Data derived solely from the Medical Group is better than the baseline, but least preferred

Scenario 2 (Unknown Agent)

The unknown agent presents the more likely scenario. Whether naturally occurring or terrorist-induced, at least the early stages of a bioincident will include uncertainty as to the causative agent. In the event of an unknown agent attack, the experts conclude:

• Only Medical Group data will have any real utility.
• The combination of Business+Positive ID group reporting is seen as trustworthy. (This finding appears somewhat anomalous and will be further investigated in follow-on studies. It is likely that the manner in which the questions were asked in the study led to this finding.)
• Medical Group reporting was seen as requiring the most resources when dealing with an unknown agent. Given that Medical Group reporting is rated as having the most utility, it is likely that the most resources would be put into collecting the data.
• Overall, when faced with an unknown agent, experts prefer to have data from the Medical Group.

Known vs. Unknown Biological Agents

It was clear from the wargame that there was a considerable difference in the perceived value of surveillance systems when the biological agent is known rather than unknown and that these differences were statistically significant in the wargame. This is an expected result and is a consequence of the fact that most of the surveillance systems that are being developed and were featured in the wargame only detect or identify specific biological agents. Moreover, some systems have an even narrower spectrum of use in that they require the specific biological agents to conform to strict limits in terms of their behavior in the assay system. 82 If the particular agent does not conform, for example, due to changes in the cellular components expressed at the cell surface, then the system will not recognize the agent and will fail to provide a positive result. If, however, the agent under test is an unknown agent or the known agent has characteristics that are in some way anomalous and thus do not meet the system limits of specificity or even sensitivity, then the system may be deemed to be of limited value.

Even when the groups were combined, the results indicate that there is a considerable difference across systems when they are required to address anything other than a known biological agent. Thus, it is not surprising that the wargame confirms that we have far to go in achieving a system that is resource-feasible, trustworthy, and useful for an unknown agent or a known agent that does not conform to the expected pattern.

Conclusions

In assessing these results, it is important to note that the size and scope of this evaluation provide sufficient data to suggest that further and more comprehensive analysis is warranted. For example, by grouping system components together into three categories, it is difficult to know if one reporting component is accounting for all of the response, or if it is evenly divided. (For example, is it just absenteeism reporting, or is it OTC and medical claims?) Nevertheless, this preliminary look provides good guidance for designing the next round of investigation. The next round also requires a larger respondent base, to help increase the usefulness of the statistics.

Those reservations aside, this study supports conclusions that the Federal Government should:

• Reassess efforts currently underway that attempt to capture data from absenteeism reporting, OTC pharmacy sales and medical claims reporting. Their value-added may not be worth the cost.
• Increase efforts to capture medical data. These efforts would include, but not be limited to, capturing data from doctors’ offices and ER visits, as well as expanded veterinary and agricultural surveillance. In addition, increase data collection from medical website visits and nurse helplines. These data sources are valuable for early detection of both known and unknown agents.
• Reassess current plans to significantly increase the number of biosensors deployed as part of both the BioWatch 83 and Guardian 84 programs, in light of the limited value of sensors for detecting unknown agents. (See further discussion of this point immediately below.)

77. Dena M. Bravata, et al., “Evaluating Detection and Diagnostic Decision Support Systems for Bioterrorism Response,” Emerging Infectious Diseases, January 2004, Vol. 10, No. 1, 100-108: 100.
78. JASON, “Biodetection Architectures,” The MITRE Corporation, McLean, VA, February 2003: 2. See <http://www.fas.org/irp/agency/dod/jason/biodet.pdf>, accessed December 2003.
79. See <http://trochim.human.cornell.edu/tutorial/rehberg/popper.htm>, accessed March 2004.
80. William B. Lober, et al., “Rountable on Bioterrorism Detection: Information System-based Surveillance,” J Am Med Inform Assoc. 2002 Mar-Apr; 9(2): 105-15. See <http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pubmed&pubmedid=11861622>, accessed January 2004.
81. Tech et al., 2002.
82. Margaret E. Kosal, “The Basics of Chemical and Biological Weapons Detectors,” Center for Nonproliferation Studies, Monterey Institute of International Studies, Monterey, CA, November 24, 2003. See <http://cns.miis.edu/pubs/week/031124.htm>, accessed March 2004.
83. “Government Provides Details of Bioterror Sensors in Cities,” The Associated Press, Washington, DC, November 15, 2003.
84. Gerry J. Gilmore, “’Guardian’ Project to Bolster Force, Installation Security,” American Forces Press Service, Washington, DC, May 8, 2003. See <http://www.defenselink.mil/news/May2003/n05082003_200305084.html>, accessed March 2004.