The Probability of the Paranormal: A Matter of Personal Whim?
by John Rudkin
In his reference to the testing of the claims of Natasha Demkina, (FT 208, April 2006), it is a pity that space limitations prevented Hierophant’s Apprentice from providing the justification that may have made his comments more convincing. The reader could be forgiven for thinking that, perhaps, he has the same problem in understanding and applying the relevant mathematics as was demonstrated by Hyman and Wiseman in their televised investigation of Natasha. Some comments on the application of probability statistics seem merited both in this specific case, and generally in the study of anomalistic phenomena.
At best, of course, a single, small matching test could provide only a provisional indication of the presence or absence of a paranormal ability, but in this case the design and conduct of the investigation was fundamentally flawed. As the Apprentice points out, Natasha’s task was made simpler than usual by the nature of the matching test. More importantly, as Hyman himself affirmed on the programme, Natasha claims to be able to tell people what is wrong with them; she does not claim to detect what had been wrong before successful treatment. Before the test began, she specifically pointed out that she could have difficulty with two cases in which there had been remedial surgery. On hearing this, unbelievably, the “world’s foremost sceptical scientists” crassly left the test as it was and and condescendingly told Natasha that she could get two matches wrong and still pass their test!
Hyman and Wiseman further revealed their lack of understanding of the basic mathematics of their matching challenge by making inaccurate statements about the statistical probabilities that were involved. During the TV programme we were told that the chance of achieving the target of 5 or more correct matches out of 7 was 1 in 250. (The correct figure is actually 11 in 2,520.) It was also stated that the chance odds of Natasha getting 4 right out of 7, as she did in the unfair test, were 1 in 50. This error is more serious in magnitude and nature: the correct odds of getting the result she actually achieved were 1 in 72 in a fair test – but that score must obviously be reconsidered in the light of the knowledge that two of the cases lay outside her claims. It is unfortunate that the inept conduct of the investigation rules out any meaningful quantified assessment of Natasha’s actual achievement, which, at the very least, appeared to merit a test that is properly designed, conducted and evaluated.
The post hoc mention of Bayesian thinking in arbitrarily assigning odds of 99:1 against there being a paranormal explanation is an interesting revelation. For those who are as uncertain about this – as the Apprentice seems to be – “Bayesian” means a method by which “the conditional probability of each of a set of possible causes for a given observed outcome can be computed from a knowledge of the probability of each cause and the conditional probability of the outcome of each cause” (Oxford Dictionary of English, second edition, 2003). Despite its apparent key significance, neither the CSICOP pair nor the Apprentice has attempted to explain the use of Bayesian thinking in the quantification of probabilities in this case. The adverb arbitrarily is more easily understood: it means “based on random chance or personal whim, rather than any reason or system” (Oxford Dictionary of English). That says a lot: Hyman and Wiseman wasted an unusual opportunity for a valuable scientific investigation and turned it into a subjective quantification and demonstration of their renowned and undoubted scepticism.
As Hyman interestingly remarked, many would have set the odds far higher than 99:1 against Natasha getting 5 out of 7 right – even in a fair test. For example, science writer Robert Matthews has stated that his arbitrarily-set odds for his highest level of scepticism (which he terms a “High Level of Scepticism”) are 999,999:1, or, in other words, his “Maximum p-value Needed to Justify Any Claim of Significance” at this level is 1 in a million. (Journal of Scientific Exploration, 1999, 13 ,1, 1, 1). Matthews’ attention has been drawn to the results of studies of the influence of human intention on the output of random-number generators by the Princeton Engineering Anomalies Research laboratory (Jahn RG & Dunne BJ, Journal of Scientific Exploration, 1997, 11, 3, 345). A meta-analysis of the huge data base generated by this work has shown that the probability of obtaining the results by chance is around 1 in ten-million-million, or ten-million times less than Matthews’ criterion for significance at his highest level of scepticism. It has been suggested to him, without response, that this gives very good grounds indeed for suspecting a causal relationship of a profound significance to our understanding of reality.
Probability criteria are widely and routinely used, and their results are accepted as meaningful, in many areas of science; a prime example is research into the efficacy of potential new drugs. However, the use of probability criteria to justify claims of significance in the investigation of anomalous phenomena is only marginally known and acknowledged by the scientific community. Why should this be? According to Jessica Utts, Professor in the Department of Statistics at the University of California, Davis: “The answer may have several aspects, but it surely does not lie in statistics”. (“The Significance of Statistics in Mind-Matter Research”, Journal of Scientific Exploration, 1999, 13, 4, 615).
Go to the next page below: