A Follow-up to “The Girl with X-Ray Eyes”
How Not to Do an Experiment
by Guy Lyon Playfair

“A few researchers have claimed that they did have scientific proof for a paranormal claim Scrutiny by other scientists, however, showed that the ‘scientific proof’ had serious flaws. Furthermore, none of these claims could be independently replicated.”

Thus wrote Dr. Ray Hyman in the May/June issue of Skeptical Inquirer in the course of a seven-page article in which he seeks to justify CSICOP’s mishandling of Natasha Demkina, “The Girl With X-Ray Eyes”, or as his fellow inquisitor Andrew Skolnick would have it in his four-page contribution, “The Girl With Normal Eyes”.

Hyman does not tell us who these ‘few researchers’ were, so let me remind him of just one of them – the late Charles Honorton. To his credit, Hyman went to some lengths to collaborate with Honorton in tightening the controls for Ganzfeld experiments and assessing their significance.

This is what he wrote in 1991:

“Honorton’s experiments have produced intriguing results. If … independent laboratories can produce similar results with the same relationships and with the same attention to rigorous methodology, then parapsychology may indeed have finally captured its elusive quarry.” (Statistical Science 6, p. 392)

As Hyman must know as well as the entire parapsychology community knows, since 1991 several independent laboratories have produced similar results, with a total (by 1997) of 2,549 Ganzfeld sessions in Edinburgh, Amsterdam, Cornell University, the Rhine Research Center, Gothenburg and Utrecht.

The overall hit rate, where 25% would be expected by chance, was 33.9%. The probability of this result being due to chance is one in about one-million-billion.

Sounds significant?

To return to the stitching-up of Natasha Demkina, Hyman and Skolnick prove to be quite good critics of their own research methods. They reveal a succession of ‘serious flaws’. For a start, as they make clear, the CSICOP project was cobbled together in a considerable hurry:

“We had to conduct the test of Natasha’s claims to fit the constraints of a television program”, says Hyman, and “we had only a month to devise a protocol that would be acceptable to all parties.”

Skolnick had “the task of finding appropriate subjects, and coordinating the many details” and “had less than a week to accomplish all this.” Moreover, “he had to do this from Amherst, more than 350 miles from New York City” (where the test took place).

These are curious admissions from members of a supposedly scientific organisation.

Skolnick did not select his subjects very well, it seems. “We had great difficulty recruiting subjects for the test and had to settle for several people with characteristics that suggested their target conditions”.

In other words, anybody could see what was wrong with them just by lookng at them. Thus sounds like more than carelessness, as does the fact that two of the patients were found to have had exactly the same problem.

The ‘control’ patient, who had nothing wrong with him, was the youngest and fittest-looking of the group. So, says Hyman, “he was a good candidate for the person with no defects”. The oldest and least fit-looking “was an obvious choice for the person with staples in his chest”. As for the subject with part of a lung missing, she “might have given herself away through bodily reactions”.

Natasha and her interpreter turned up unexpectedly early for one of the sessions and so were able to see patients arriving and climbing the stairs. “This breach of protocol may have provided them clues about which subjects did or did not have the artificial hip.”

Altogether, according to Hyman, “our test included five subjects for whom external clues were available concerning their internal conditions”. This is a very strange admission, since (so I am informed by a reliable source) both the producer and one of the inquisitors tried to match the diagnoses as a control and got them all wrong.

Somehow, this detail was not mentioned in either the TV film or the Skeptical Inquirer articles.

One way and another, the testing of Natasha Demkina was a shambles. Had Hyman’s article been submitted to any of the peer-reviewed parapsychology journals it would have been marked (by this reviewer at least) Return to Sender.

In any case, the whole experiment was not designed to replicate what Natasha actually claims to do, which is diagnose people’s current complaints. In New York, she was asked instead to diagnose complaints that some patients no longer had. She did attempt this with Skolnick himself and was clearly not successful (if he knows anything about experimenter effects, he will know why).

However, this is not what she normally does for those who come to see her back home in Saransk, where as Skolnick notes with horror, she “reportedly” charges a whopping $13 per consultation – rather less than your average New York consultant, one suspects.

“We wanted to make the test as comfortahle and nonstressful as possible for Natasha as possible,” says Hyman, with a disingenuousness one would expect from a CIA consultant psychologist. It is clear that they did exactly the opposite.

Would Natasha not have been more comfortable and less stressed in her own home? Would not a genuine test of her claims have been to confront her with some conventionally diagnosed patients who actually had something wrong with them now, not several years ago? Or, even simpler, could they not film a random sample of her routine diagnoses and then have the patients checked out by normal means? Could they not have found any of those doctors in Saransk who are said to have testified that Natasha’s “X-ray” methods were as good as theirs if not better?

Incidentally, one wonders why neither Hyman nor Skolnick made any mention in their articles of the six patients Natasha had examined before attempting her diagnosis-matching, to the obvious satisfaction of at least five of them.

Looks like deliberate suppression of positive evidence to me.

Finally, Hyman tells us what serious researchers already know, that “any scientific hypothesis – especially a paranormal one – cannot be confirmed or disconfirmed by one test or one experiment. … Independent investigators must also replicate the findings before they gain scientific credibility.”

Indeed. This also applies to skeptical hypotheses. Had Hyman, Wiseman and Co. seen Natasha match all seven diagnoses correctly, they would have hastily assured us that “we cannot leap to conclusions after a single test”.

Yet after their single test, which they rated a failure despite Natasha scoring at over three times the level of chance statistical probability, they made it quite clear that they reckoned they had debunked the well-meaning and innocent-looking Russian teenager once and for all.

Why else did that bitchy narrator announce that “if she fails, her reputation could be ruined”?

“The Girl With X-ray Eyes” was screened several times on European and Asian channels but not, at least as of Spring 2005, in the USA.

“I am afraid,” Skolnick laments, “that Discovery Channel may consider the program too skeptical for the American audience.”

Having taken part myself in Discovery’s highly skeptical “Miracle Hunters” (shown in the USA on 28 April 2004) I suspect there are other reasons. It may be that following the barrage of adverse comments (including mine) that greeted the original screening, Discovery wisely decided to write the show off as an embarrassing flop.

Natasha Demkina did not fail. Given the unfamiliar and highly stressful surroundings and the demand to do something she had not done before, she did remarkably well.

It is CSICOP’s reputation, not hers, that has crumbled into ruins.

Go to the next page below: