The Research of the Skeptics
by Chris Carter
” …the psi controversy is largely characterized by disputes between a group of researchers, the parapsychologists, and a group of critics who do not do experimental research to test psi claims or the viability of their counterhypotheses.”
– Charles Honorton, in “Rhetoric over Substance”
Charles Honorton, in his classic article ‘Rhetoric Over Substance’ noted an important difference between the psi controversy and more conventional scientific disputes. Controversies in science normally occur between groups of researchers who formulate hypotheses, design experiments, and then collect data in order to test their hypotheses. But as Honorton wrote, “In contrast, the psi controversy is largely characterized by disputes between a group of researchers, the parapsychologists, and a group of critics who do not do experimental research to test psi claims or the viability of their counterhypotheses.”
This lack of research may surprise anyone whose main source of information has been the skeptical literature. For instance, in 1983 the well-known skeptic Martin Gardner wrote:
“How can the public know that for fifty years skeptical psychologists have been trying their best to replicate classic psi experiments, and with notable unsuccess? It is this fact more than any other that has led to parapsychology’s perpetual stagnation. Positive evidence keeps coming from a tiny group of enthusiasts, while negative evidence keeps coming from a much larger group of skeptics.”
But as Honorton pointed out, “Gardner does not attempt to document this assertion, nor could he. It is pure fiction. Look for the skeptics’ experiments and see what you find.” For the most part, skeptics have simply criticized from the sidelines, and have produced no experimental research of their own.
One notable exception to this rule has been British psychologist Susan Blackmore. She began working on a PhD in parapsychology in the 1970’s, but has repeatedly claimed that she has failed to find any evidence for the existence of psi. For instance, she wrote in 1996: “When I decided to become a parapsychologist I had no idea it would mean 20 years of failing to find the paranormal.” Blackmore has made a career for herself as one of the world’s most well known skeptics of psi, and in 1988 was elected a Fellow of CSICOP.
In a number of publications Blackmore claims to have become increasingly skeptical of the existence of psi phenomena after “ten years of intensive research in parapsychology.” These claims led parapsychologist Rick Berger to critically examine the Blackmore experiments in great detail, and he found that “The claim of ‘ten years of psi research’ actually represents a series of hastily constructed, executed, and reported studies that were primarily conducted during a 2-year period.’” These consisted of a set of experiments conducted between October 1976 and December 1978 for her PhD dissertation.
Blackmore reported 29 experiments completed over this two-year period, of which 21 were eventually published as separate experiments in five parapsychology journal papers. Seven of these experiments produced statistically significant results. Although these experiments form the basis of Blackmore’s claim of “failing to find the paranormal”, the odds against 7 successes out of 21 happening by chance are over 20,000 to one!
So, how does Blackmore reconcile the fact of 7 successful experiments out of 21 with her often-repeated claim that her own research led her to become a skeptic? Simple: results from successful experiments were dismissed as due to flaws in the experiment, yet study quality was simply ignored when the results were nonsignificant. There are many design flaws that can lead to false positive results, but there are also many that can lead to false negatives, such as inadequate sample size (low statistical power), inappropriate sampling, and so forth. Berger writes “Blackmore’s database is replete with examples of such flaws”, and continues:
“Some skeptics, including Blackmore, argue that differing standards of experimental design can be held depending on study outcome: Significant positive outcomes must have tighter designs than the same study with a negative outcome. This post hoc determination of experimental criticism leads to the paradox exemplified by the Blackmore work: Had such work produced consistently positive outcomes, the results could all be dismissed as having arisen from design flaws… Negative conclusions based on flawed experiments must not be given more weight than positive conclusions based on the same flawed experiments.”
In other words, our decision to invoke study flaws to dismiss the results of an experiment should not be influenced by our preconceptions of what the result “should have been.” But this seems to have been exactly what Blackmore has done in order to justify her beliefs, as evidenced in the following remark of hers:
“Well, if you don’t find evidence of ESP, what can you say? Only that you have failed to find something which, according to science, shouldn’t have been there in the first place!”
As we shall see, this appeal to ‘science’ as a monolithic body of conclusions that tell you in advance what should and should not be the case is a rhetorical tactic often used by Blackmore. But at any rate, Berger finally concluded:
“Blackmore’s claims that her database shows no evidence of psi are unfounded, because the vast majority of her studies were carelessly designed, executed, and reported, and in Blackmore’s own assessment, individually flawed. As such, no conclusions should be drawn from this database…. Blackmore is extremely vocal in decrying psi research in her writings, on television and radio, and before the skeptical advocacy group CSICOP (the Committee for Scientific Investigation of Claims of the Paranormal), citing her own work as the basis for her strong convictions. … [She] has achieved a notable position in the skeptical community based on her conversion from believer to skeptic during her “ten years of negative research.” Her insistence to the contrary notwithstanding, I believe that my review of her psi research has achieved a constructive end by showing that her conversion from parapsychologist to CSICOP Fellow had no scientific basis in her own experimental work.”
The same journal issue also includes a response by Blackmore to Berger’s critique, in which Blackmore conceded “I agree that one cannot draw conclusions about the reality of psi based on these experiments.” Near the end of his critique Berger had written “During my aborted meta-analysis of Blackmore’s published work, I was struck by patterns in the data suggestive of the operation of psi…. Without a serious meta-analysis of the original unpublished source material, complete with weighting for flaws…the issue of whether the Blackmore experiments show evidence for psi cannot be resolved.” Presumably eager to nip this embarrassment in the bud, Blackmore hastened to say “I am glad to be able to agree with his final conclusion – ‘that drawing any conclusion, positive or negative, about the reality of psi that are based on the Blackmore psi experiments must be considered unwarranted.’”
It is interesting to examine Blackmore’s writings before and after Berger’s critique. Two years earlier, in an article for Skeptical Inquirer entitled “The Elusive Open Mind: Ten Years of Negative Research in Parapsychology”, she wrote:
“How could I weigh my own results against the results of other people, bearing in mind that mine tended to be negative ones while everyone else’s tended to be positive ones? I had to find some kind of balance here. At one extreme I could not just believe my own results and ignore everyone else’s…. At the other extreme I could not believe everyone else’s results and ignore my own. That would be even more pointless. There would have been no point in all those years of experiments if I didn’t take my own results seriously.” (Emphasis added.)
In another article written at about the same time she wrote:
“The other major challenge to the skeptic’s position is, of course, the fact that opposing positive evidence exists in the parapsychological literature. I couldn’t dismiss it all. This raises an interesting question: Just how much weight can you or should you give the results of your own experiments over those of other people? On the one hand, your own should carry more weight, since you know exactly how they were done… On the other hand, science is necessarily a collective enterprise…. So I couldn’t use my own failures as justifiable evidence that psi does not exist. I had to consider everyone else’s success.
“I asked myself a thousand times, as I ask the reader now: Is there a right conclusion?
“The only answer I can give, after ten years of intensive research in parapsychology, is that I don’t know.”
Although after Berger’s critique Blackmore was willing to concede in an academic journal that “I agree that one cannot draw conclusions about the reality of psi based on these experiments”, her writings in the popular press have not reflected this admission. Commenting on the ganzfeld experiments in a newspaper article in 1996, she wrote:
“My own conclusion is biased by my own personal experience. I tried my first ganzfeld experiment in 1978, when the procedure was new…. Of course the new auto-ganzfeld results are even better. Why should I doubt them because of events in the past? The problem is that my personal experience conflicts with the successes I read about in the literature and I cannot ignore either side. The only honest reaction is to say ‘I don’t know’.”
Wouldn’t a more honest reaction be for Blackmore to admit in the popular press that “one cannot draw conclusions about the reality of psi” based on her own experiments, and that a scientific opinion should be based only upon a critical evaluation of other peoples’ published works?
But perhaps this is asking too much. After all, Blackmore pursued a PhD in parapsychology in order to become a “famous parapsychologist”. Having failed to produce research supporting the psi hypothesis, she evidently decided to try to make a name for herself by attacking the psi hypothesis, which must at the time have seemed to be an easy target. Apparently, though, in a recent article she claims to have given up. “At last, I’ve done it. I’ve thrown in the towel”, she wrote.
“Come to think of it, I feel slightly sad. It was just over thirty years ago that I had the dramatic out-of-body experience that convinced me of the reality of psychic phenomena… Just of few years of careful experiments changed all that. I found no psychic phenomena… I became a sceptic. (Emphasis added.)
“So why didn’t I give up then? There are lots of bad reasons. Admitting you are wrong is always hard, even though it’s a skill every scientist needs to learn. And starting again as a baby in a new field is a daunting prospect. So is losing all the status and power of being an expert. I have to confess I enjoyed my hard-won knowledge.
“… None of it ever gets anywhere. That’s a good enough reason for leaving.
But perhaps the real reason is that I am just too tired – and tired above all of working to maintain an open mind. I couldn’t dismiss all those extraordinary claims out of hand. After all, they just might be true… ”
We’ll miss you, Susan.