The real odds: is success predicted by the 5 per cent chance failure rule in statistics?

摘要

Some day we may have access to medical software in addition to statistical software that offers a one-touch approach between robotic cardiac surgery if we need an artery bypass, have to repair or replace a leaky heart valve, remove a tumour, treat congenital heart conditions or carry out a biopsy. The challenge is to mimic a surgeon who needs many years of education and experience to perform these procedures credibly and the support team. Yet, in contrast to reporting research results with statistical errors which may lead to wrong conclusions that are refuted only a few studies several years later, when mistakes happen during such medical operations, unavoidably, evidence about them emerges substantially more rapidly and the impact may be more severe. But right now analysing data by suitable statistical methods and understanding and interpreting them on matters as complicated as brain research, the ageing process, chronic illnesses, drug delivery and its interactions, and other intervention effects requires a more time-consuming and prudent study than clicking buttons to run algorithms. What medicine and statistics have in common, although, is that it takes time and effort to gain a solid knowledge of the techniques before applying them fruitfully. When researchers evaluate and summarize data from their studies, statistical tests that are dichotomized as being statistically significant or not, P values, contributing or predicting factors, and significant associations, links, or patterns can be just a number of mouse clicks away. The t-test, chi-square test, or tests in more advanced models such as linear regression or generalized linear models that became prevalent during the last century unabatingly dominate the statistical landscape in various realms of the medical field, social sciences and economics. Yet, bulk printouts of P values often convey false impressions of data, analysis and results being simple and easily understood. They also harbour the danger of providing the wrong answers or not answering the right questions. Given obvious progress in technology and big data, one is tempted to gravitate towards online self-service portals for introductory statistical courses and freely available software. It is easily forgotten that 30 or 40 years ago, statistics mostly was in the hands of statisticians with 5-10 years of education in their field who, as part of their PhD curriculum, started their careers by consulting, say, on medical studies. They carried out the data analyses on mainframe computer servers, initially using punch cards, and were forced to write abstract programming codes in Pascal, Fortran, C++ and SAS to process the data. While progress and enhancements in interactive graphics unfold rapidly, have a profound effect on the planet, and are greeted with enthusiasm by most of us; carefully applying and presenting statistical concepts continues to require more time than most researchers in other areas, or people presented with statistics, have time for. It seems inevitable that the pastiche of statistical thinking and methodology, which have become the framework for the medical, soft, and hard sciences in the 20th century (Efron 1998), will increase its presence and influence in the 21st century as big data penetrate more widely and deeply into the scientific fields and daily lives. There is reason for hope that in the coming decades researchers will adopt some of the new ideas, methods, and algorithms that have been emerging from these efforts to draw value from big data – and let go of simple-minded P value reports. In what follows, I will discuss some of the sources of the problem of statistical misunderstanding and conclude with a few fallacies in presenting statistical summaries that researchers in other areas might be interested in and I have reasons to believe have not ceased to recur. In empirical and clinical studies, notoriously, the term ‘significant,’ when only appropriate as a reference to ‘statist