Appendix B – Methodology & Statistical Analysis Survey Design We identified ten desirable characteristics of a warning system for biological agents and grouped these into three categories [Table B.1]. Survey respondents evaluated Warning Systems on these characteristics using a 1-to-5 Likert scale of agreement. Respondents were solicited via email and the scoring was conducted in a Microsoft Excel spreadsheet attachment. Experimental Design For each of the two scenarios, Known Agent and Unknown Agent, an experimental design was conducted. First, the authors drafted a list of nine potential attack indicators placed into three groups [table B.2] and a list of expert evaluators with various backgrounds. Rather than have experts score each Indicator Group one at a time (as in a typical survey), we implemented a full factorial design with the Indicator Groups as factors. That is, experts were randomly assigned to assess one of eight Systems, varying from a baseline of no indicators to a full system of all three Groups: Business, Medical, and Sensor indicators. This design allows us to assess the value of Group combinations, i.e. do Indicator Groups complement or duplicate each other’s information for first responders? Data Analysis The authors sent surveys to 132 experts and further instructed them to distribute the instrument to other experts. This yielded 49 respondents, each of which analyzed one Warning System for each Scenario. Composite scores for Utility, Trustworthiness, and Resource Requirements were created by averaging the characteristics for each. Averaging the three composite scores created an Overall score. Table B.3 shows the raw scores and a measure of the components’ consistency/reliability, Cronbach’s Alpha. Note that the reliability for Trustworthiness is marginal (.65), suggesting that measurement of this element needs refinement in future studies. Analysis of Variance (ANOVA) was performed on the four composite scores for each scenario. This statistical analysis can detect patterns in the eight combinations of Indicator Groups to isolate those that drive better scores. For example, examining the Utility Scores for the Unknown Agent in table B.3, we see that Systems that include Medical information score higher than those without Medical data do. These results are reflected in the ANOVA coefficients [table B.4] and the Systems that the models indicate as better than the baseline [table B.5]. Notes Fit statistics and coefficients for the Known Agent models are generally larger than those of the Unknown Agent models. This suggests that warning for Unknown Agents is more difficult [table B.4]. Additional warning indicators can be added to the Preferred Systems in table 5, but doing so will not provide responders & policymakers with statistically-significant value-added.
This study was not designed to isolate the individual effects of the components
of the three Indicator Groups (Medical, Business and Positive ID). This
will require additional study-- for example, a larger sample employing
a fractional factorial design with individual indicators as factors. Table B.1 – Desirable Characteristics of Warning Systems & Category Assignments
Table B.2 – Biological Attack Indicators & Group Assignments
Table B.3 – Means, Sample Sizes & Reliability Scores for Composite Scores Table B.4 – Simplified ANOVA Models: Coefficients and R2
Statistical Significance: * <.05; **<.01 Table B.5 – Systems Preferred over Baseline and Predicted Scores
Note: If an Indicator Group is not present in the listed System, there is no statistically significant value-added.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||