The limits of phenomenology

What's wrong with the science of science? Do the methods we use to explore new innovations in psychology, drug testing and engineering design really work? A new paper has troubling answers: Big Data isn't big enough to be useful. We face a problem with the empirical. We had best re-examine our confidence that conventional methods of scientific research can deliver real answers.

The paper, "The Limits of Phenomenology: From Behaviorism to Drug Testing and Engineering Design" by Yaneer Bar-Yam, president of the New England Complex Systems Institute, spells out why the only answer is a turn toward complex systems to work out the world's most critical challenges—managing social and political conflicts, testing drugs, and achieving better designed infrastructure included.

"Data by itself is not enough," said Bar-Yam, in discussing conventional systems of collecting information. "There are so many factors that play a role that no empirical observation is ever useful to tell us what will happen in the future. The same conditions never happen twice."

Bar-Yam's examples of where conventional scientific methods go astray include drug approvals, where safety data from trials appears to offer adequate support for positive conclusions and yet the drugs carry severe, even lethal, side effects. Drug testing even in using the accepted gold standard of double blind randomized control tests cannot determine negative interactions and side effects. No number of trials can determine them. In engineering, large projects such as US air traffic control modernization and British medical records systems, costing up to $6 billion and $10 billion, respectively, were held up because they could not pass tests.

Bar-Yam makes a case for complex models that can deliver greater understanding because complex models take into account more factors than meet the eye.

"Our results imply that we need to improve our ability to model complex interacting systems," he said. Big data—with all its power to amass information about everything and anything captured by the second, does not provide answers that are useful. In recent years, he noted, complexity scientists have unraveled the financial crisis, determined the causes of riots and revolutions and mapped the social networks we are part of. They managed to do this because complex systems science involves validated modeling.

Said Bar-Yam: "In order to represent the behavior of a system in response to a large set of possible conditions, it is necessary to have a representation of the system that identi?es the response to cases that have not been observed. A model enables inference to determine the behavior of a system across a range of conditions."

No empirical observation, he added, is ever useful as a direct measure of a future observation. "It is only through generalization motivated by some form of model/theory that we can use past information to address future circumstances."