July 15, 2014 report
Researchers measure accuracy of forecasts in strategic intelligence
(Phys.org) —Two researchers have measured the accuracy of forecasts given by two teams within Canada's Intelligence Assessment Secretariat (IAS), a group that provides government leaders with intelligence information regarding the Middle East and Africa. David Mandel with Defence Research and Development and Alan Barnes, formerly with IAS report in a paper they've had published in Proceedings of the National Academy of Sciences on their findings and offer opinions on the overall quality of such intelligence gathering based on what they learned.
Intelligence gathering has been conducted by people for most of human history—knowing what friends and enemies alike are about to do plays a very large role in strategic planning. But, as the two researchers note, in modern times, very little information is available regarding the accuracy of strategic intelligence gathered by governments. To learn more, they focused their efforts on IAS, looking at 1,514 strategic intelligence forecasts (with particulars redacted for security purposes) that were made over the period March 2005 to December 2011. Comparing the forecasts with actual outcomes revealed the unit was right 76 percent of the time, which the researchers describe as "very good."
Not surprisingly, the researchers also found that more experienced analysts were better at their jobs (when making predictions based on judgment rather than informational value) than those with just a few years of experience—their work was more accurate though there were miscalibrations that the researchers attributed to assigning more uncertainty to some events than might have been necessary due to under-confidence that came about as a result of an air of caution. Such an environment has come about, the research pair note, from stated goals that strive for "informativeness" and an aversion to overstating what has been found. They noted also that most of the analysts had both on-the-ground experience in their area of study and post-graduate degrees, which suggests they were highly qualified for their positions.
The researchers conclude their analysis by offering their opinion on the quality of intelligence forecasting offered by the agency, suggesting they found "cause for tempered optimism" regarding the work done by the analysts in the IAS, but also suggest that forecasting could be made even more accurate by using postforecast transformations.
More information: Accuracy of forecasts in strategic intelligence, PNAS, 2014. www.pnas.org/content/early/201 … /1406138111.abstract
Abstract
The accuracy of 1,514 strategic intelligence forecasts abstracted from intelligence reports was assessed. The results show that both discrimination and calibration of forecasts was very good. Discrimination was better for senior (versus junior) analysts and for easier (versus harder) forecasts. Miscalibration was mainly due to underconfidence such that analysts assigned more uncertainty than needed given their high level of discrimination. Underconfidence was more pronounced for harder (versus easier) forecasts and for forecasts deemed more (versus less) important for policy decision making. Despite the observed underconfidence, there was a paucity of forecasts in the least informative 0.4–0.6 probability range. Recalibrating the forecasts substantially reduced underconfidence. The findings offer cause for tempered optimism about the accuracy of strategic intelligence forecasts and indicate that intelligence producers aim to promote informativeness while avoiding overstatement.
Journal information: Proceedings of the National Academy of Sciences
© 2014 Phys.org
