The Double-Blind Gaze

How the Double-Blind Experimental Protocol Changed Science

by Steven Bratman

I AM A FORMER "HOLISTIC" M.D. WHO practiced alternative medicine for over a decade. What finally brought my practice to a halt was not lack of success but a close encounter with one of the most important and least noticed scientific breakthroughs of the 20th century--the randomized, double-blind, placebo-controlled study.

During my years of practice, many of my patients reported benefits, and I could often see those benefits with my own eyes. Use of medications declined, days off work decreased, objective measurements of urine flow and liver enzymes and blood pressure shifted toward the normal range, and injured joints recovered normal range of motion. Nonetheless, despite all these apparent benefits, I am now quite convinced that my eyes were wrong, and most of the treatments I provided were nothing more than elaborate placebos.

Readers of this journal will not find anything surprising in this statement. However, the deeply skeptical paradigm of the double-blind study casts doubt not only on alternative medicine, but also on much of conventional medicine, as well as established "facts" in many other fields. Under the double-blind gaze, much that is obvious disintegrates. Furthermore, the insights of the double-blind study suggest that, for a large category of phenomena, some seemingly simple questions may never be answered.

What Is A Double-Blind Study?

In a randomized double-blind, placebo-controlled trial of a medical treatment, some of the participants are given the treatment, others are given fake treatment (placebo), and neither the researchers nor the participants know which is which until the study ends (they are both "blind"). The assignment of participants to treatment or placebo is done randomly, perhaps by flipping a coin (hence, "randomized"). The actual identity of the treatment given to each participant is encoded, and the code is not revealed until the study is complete.

Why Double-Blind Studies? The experience of the last forty years has shown that, for most types of medical treatments, only a randomized double-blind, placebo-controlled study can properly answer the deceptively simple question: "Does Treatment A benefit Condition B?"

Commonsense says that it's easy to tell if a treatment works--simply try it. However, in this case, common sense is wrong. Unblinded observation allows one to draw valid conclusions only in the case of "high effect size treatments." A high effect-size treatment is one that, in nearly all subjects, causes effects that lie entirely outside the range of normal variation of that subject. To test the hypothesis "will a quart of hard liquor cause drunkenness?" one would not need a double-blind study. The behavior and physiology of a person who has consumed that much alcohol are sufficiently different from the behavior and physiology of a person who has not consumed alcohol, that simple observation would be sufficient to verify the hypothesis.

A number of medical treatments fall into the high effect-size category: for example, appendectomy for appendicitis, penicillin for streptococcal pneumonia, vitamin C for scurvy, anesthetics to produce unconsciousness, and defibrillation for restoring heart function. However, for many medical treatments, the subjective and objective signs of untreated individuals overlap considerably with those of treated individuals. People who have an ankle sprain may experience a reduction of symptoms when they take ibuprofen, for example, but not the elimination of symptoms; furthermore, ankle sprain symptoms change from moment to moment and are affected by state of mind. This type of fluid, inconsistent data turns out to be very difficult to appraise accurately. Only double-blind studies are up to the job.

Confounding Factors

Subtle influences called "confounding factors" can easily create the illusion of efficacy when an ineffective treatment is used. Consider the practice of "letting blood," a technique that endured for many centuries, and reached its heyday in the 17th and 18th centuries. The medical literature of Enlightenment-era Europe is full of testimonials to the marvelous effect of slitting a vein. Today, it's clear that bleeding is not helpful, and no doubt was responsible for killing a great many people.

Why did bloodletting survive so long? Not because the people who used it were stupid, dishonest, or unobservant--the greatest minds of the time were certain that letting blood was a medical necessity. The practice endured because they saw benefits from it. If one begins with the assumption that a treatment is helpful, one is highly likely to observe benefits by using it. Such misleading observations are ensured by the following confounding factors (among others):

• The Placebo Effect
• The Re-interpretation Effect
• Observer Bias
• Natural Course of the Illness
• Regression to the Mean
• The Study Effect (Hawthorne Effect)

The Placebo Effect

The term "placebo effect" properly refers to actual improvements in a condition brought about by use of a fake treatment passed off as a real one. Many conditions respond well to this approach. For example, if a person has an ankle sprain, and cannot move the ankle fully, use of a placebo treatment is almost certain to increase the measured range of motion of the ankle. Other conditions that also respond well to placebo treatment include musculoskeletal pain in general, (1) menopausal hot flashes, (2) and prostate enlargement. (3) It is often said that 30% of people respond to placebo, but in these conditions the apparent placebo response can reach as high as 70%.

Thus, if one gives a placebo treatment to 100 people with musculoskeletal pain, one may get 70 testimonials of benefit. They will be sincere, convincing testimonials too: no one believes it's the placebo effect when it happens to them. The placebo response doesn't feel fake, or weak, or superficial. It feels real.

Doctors, too, are fooled. Up until recently, orthopedic surgeons believed that "knee scraping" surgery (technically, arthroscopic surgical debridement) was quite effective, and hundreds of thousands of such surgeries were performed every year. If one asked a surgeon, "how do you know this treatment works?" the surgeon would very likely reply, "Because I can see that it works with my own eyes. I have patients who go into surgery unable to walk, and a month later, they're skipping rope."

After performing this surgery for decades, one surgeon decided to use the double-blind, placebo-controlled methodology to test whether it really worked. The results were shocking: Arthroscopic surgery for knee arthritis did indeed bring about dramatic and long-lasting results; however, so did fake surgery (anesthesia and an incision), and to the same extent. (4) Surgeons were shocked and chagrined to find that people given the fake surgery were so pleased with the results that they said they would happily recommend the treatment to others.

Grave doubts exist about the tree utility of many other surgeries, including, most notoriously, back fusion surgery. In general, surgery lags behind other branches of conventional medicine in the extent to which it incorporates modern standards of evidence. However, there are many commonly used drugs that lack meaningful support as well. For example, recently it has become clear that cough syrup containing either codeine or the "cough suppressant" dextromethorphan doesn't actually suppress coughs, despite decades (in the case of dextromethorphan) or centuries (in the case of codeine) of apparently effective use. (5)

Comparison to placebo treatment is thus essential; without such comparison, any random treatment is likely to appear effective. Few proponents of alternative medicine have grasped this basic though counterintuitive fact. It is quite common for an alternative medicine product or technique to be advocated based solely on research in which people with a problem are given a treatment, and lo and behold, they improve. But in the absence of double blind controls and a placebo group, such studies are meaningless. Any nonsense treatment should be able to produce apparent improvement in a fair number of people. Consider the following examples.

In a study of 321 people with low back pain, chiropractic manipulation was quite helpful, but no more helpful than giving patients an essentially useless educational booklet on low back pain. (6)

In a randomized, controlled trial of 67 people with hip pain, acupuncture produced significant benefits, but no greater benefits than placement of needles in random locations. (7)

In a double-blind, placebo-controlled study of 30 people with carpal tunnel syndrome, use of a static magnet produced dramatic and enduring benefits, but so did use of fake magnets. (8)

In a randomized, controlled trial of 177 people with neck pain, fake laser acupuncture proved to be more effective than massage. (9) If your neck hurts, go to the doctor and request fake laser acupuncture. It's tremendously effective.

Beyond the Placebo Effect

At least the placebo effect involves a real benefit. Many other illusions can create the impression of benefit although no benefit has occurred at all. These are often loosely referred to by the plural term "placebo effects." An article published in 2001 pointed out that it is very difficult to separate a true placebo effect from these other confounding factors. (10) The authors of that article concluded that the placebo effect doesn't exist at all, but this conclusion is probably too strong. Nonetheless, regardless of how the placebo effect issue is ultimately decided, double-blind, placebo-controlled studies are the only known way to eliminate the effects of the other confounding factors.

Even when a fake treatment doesn't actually improve symptoms, people may re-interpret their symptoms and experience them as less severe. For example, if you are given a drug and informed that it will make you cough less frequently, you will likely think that you are coughing less frequently, even if your actual rate of coughing as measured by a recording instrument doesn't change. In other words, you will re-interpret your symptoms to perceive them as less severe.

Observer bias is a similar phenomenon, but it affects doctors (and researchers) rather than patients. If doctors believe that they are giving a patient an effective drug, and they interview that patient, they are likely to observe improvements, even if the drug is ineffective. This was beautifully shown in a classic study that compared a new treatment for multiple sclerosis against placebo. (11)

This was a double-blind study, and therefore the physicians whose job it was to evaluate the results were kept in the dark about which study participants were receiving real and which were receiving fake treatment. These blinded observers failed to observe any greater benefit in the treatment group than the placebo group. In other words, the treatment did not work. However, researchers introduced an interesting wrinkle: they allowed a few physicians to know for certain which patients were receiving treatment (these observers were "unblinded"). The results were surprising. The unblinded physicians "observed" a great deal of benefit in the treated group as compared to the placebo group. In other words, the unblinded physicians observed the benefit they expected to see.

These results are appalling because of what they say about so-called "professional experience." Suppose a physician has tried two drugs for a certain condition and found by experience that drug A is more effective than drug B. Does this mean that drug A is actually more effective than drug B? Not at all. If the doctor has any reason to expect drug A to produce better results (e.g., memorably positive experiences with a few patients, recommendation from a respected colleague, impressive salesmanship on the part of a pharmaceutical company), the doctor is very likely to experience drug A as more effective than drug B, regardless of the actual comparative efficacy as tested in double-blind trials. The effect is so strong that doctors are likely to discount the results of studies that don't agree with what they "know" to be true.

Natural course of the illness is another confounding factor. Many diseases get better on their own. Because of this, any random treatment given at the beginning of such an illness will seem to help if no placebo control is available for comparison. Furthermore, a doctor using such a treatment is likely to experience an illusion of agency, the psychological sense of having altered the outcome of a situation by intervening. This illusion is compelling, and even highly objective people are fooled.

Regression to the mean is a statistical cousin of natural course, but applies to chronic conditions. Consider high blood pressure. Blood pressure levels wax and wane throughout the day and from week to week. Suppose that a person's average blood pressure is 140/90, but occasionally it gets as high as 160/105 or as low as 120/70. If such people are tested and found at the moment to have high blood pressure, they may be seen as needing treatment. However, if they happen to be nearer their average blood pressure, or lower than their average, they won't be seen as needing treatment. In other words, when patients have a fluctuating condition, doctors (and researchers) tend to enroll them in treatment at the moment of an unhealthy extreme. By the laws of statistics, after any sufficient interval, a fluctuating variable is more likely to be measured near its mean than at its extremes. Therefore, regardless of what treatment (if any) is used, blood pressure will appear to improve with time.

The study effect (also called the Hawthorne effect) refers to the fact that people enrolled in a study tend to take better care of themselves, and may improve for this reason, rather than for any specifics of the treatment under study. This is a surprisingly powerful influence, lf people enroll in a trial of a new drug for reducing cholesterol, and receive a placebo, their cholesterol levels are almost certain to fall significantly. Why? They begin to eat better, exercise more, etc.

Double-Blind Studies to the Rescue

M1 of these confounding factors are eliminated in a properly designed, randomized, double-blind, placebo-controlled trial. Provided the study is sufficiently large and designed correctly, if people receiving the real treatment improve to a substantially greater degree than those in the placebo group, one can state with some degree of certainty that the treatment is effective on its own, above and beyond its inevitable placebo effects and the influence of all the other confounding factors.

In the current medical environment, no new drug can be approved for treatment unless it passes a few sizeable double-blind studies. Some existing drugs, however, have never passed such tests (such as cough syrup), and practically no surgeries have been studied in a controlled fashion. Besides surgery, another large gap exists for what is called "off-label" use of approved drugs. Once a drug has been FDA-approved for a specific purpose, doctors are allowed to prescribe it for any purpose they like, and they frequently do. In many cases, these off-label uses are based merely on anecdote, professional experience, the experience of colleagues, or plausible reasoning: highly unreliable sources of information, as the insights of double-blind studies show.

Even with a treatment known to be effective, proof of benefit requires additional testing. For example, there is no doubt that suitable antibiotics can kill the bacteria that cause ear infections. But does antibiotic treatment shorten the duration of ear infections or prevent complications? Evidence from studies conducted decades after such treatment became ubiquitous suggests that it does not work, at least not significantly. (12)

Another area of conventional medicine that lacks formal examination is known as "behavioral suggestion"--use ice during the first 72 hours of a sprain or strain, stay in bed if you're a pregnant woman with signs of impending miscarriage, lay your baby face down to sleep, drink plenty of water to avoid headaches, etc. These common recommendations, and so many others, lack substantiation and are really no more than urban legends, dangerous legends in some cases: putting babies to sleep face down is now thought to have contributed to the epidemic of Sudden Infant Death Syndrome (SIDS).

However, if conventional medicine has its problems with poor evidence, alternative medicine is far worse (though probably not as dangerous). Excepting a handful of herbs and supplements, the great majority of well-designed studies of alternative therapies have failed to find benefit. The ordinary course of affairs is as follows:

• Treatment A is identified as having a long history of traditional use, or a plausible reason it might work. Widespread marketing of the treatment begins.
• Doctors and patients try treatment A and report benefits.
• A study is done in which patients are given treatment A for a period of time, and at the end of the treatment they report feeling better. The absence of a control group, however, makes the results meaningless.
• Another study compares treatment A to no treatment. Again, treatment A scores a success.
• A tiny, badly-designed double-blind study is done, and the treatment is "shown" to work well.
• Serious researchers become interested and perform a large scale, well-designed double-blind study, in which the treatment proves no better than placebo.

One might reasonably request providers of alternative therapies to test their treatments before purveying them, but this is not likely to happen. It is much easier to invent a treatment than to test it, and a manufacturer burned by a negative double-blind study can simply consider the studied product to have reached the end of its product cycle and start marketing a new one.

What's Wrong with the Placebo Effect?

At this point in the discussion, many readers may wonder what's wrong with the placebo effect. Why not use treatments that do no harm, even if they don't help in any specific way?

There is certainly much to be said for this approach. After all, for the sufferer, whatever alleviates suffering is a blessing. However, this can become a slippery slope. As soon as one decides that placebos are acceptable, one opens the door to every other superstition, fantasy, and fraud. I know a doctor who provides treatments at $2000 a pop with a machine that "alters DNA" to improve overall health and cure diseases. His patients are satisfied. Is there anything wrong with this? What makes it wrong, the price? Where should one draw the line?

The very existence of the placebo effect creates an ethical swamp at the center of all forms of medicine. If doctors lie to their patients, and say that a nonsensical treatment works, the patients are likely to experience benefit. If doctors lie to themselves and believe that the nonsensical treatment really works, they will be able to convey more certainty, and the patient (whether through the placebo effect or reinterpretation) will benefit even more. However, if the doctor tells the truth and admits that the treatment is fake, the patient won't benefit. Thus, in medicine, a lie is no longer a lie if it is told convincingly enough, and truth-telling leads to harm.

Why not use treatments that produce a tree benefit on top of the placebo effect; in other words, those that have passed double-blind placebo-controlled studies?

Limitations of Double-Blind Studies

Unfortunately, certain treatments cannot be tested in a double-blind trial. In some cases, reasonable work-arounds can be invented. Certain treatments, however, may be intrinsically impossible to test. In the latter circumstance, we may never know whether they really work. Examples of difficult-to-test treatments include acupuncture, surgery, and physical therapy.

The problem is that there is no way that practitioners can fail to know whether or not they are providing a real or fake treatment. In the case of surgery this may not matter, so long as the surgeon does not talk to the patient before or after anesthesia. (Any conversation is likely to subtly communicate whether the treatment was real or fake, and thus influence the outcome.) For treatments such as physical therapy and acupuncture, the best that can be done is to use "blinded observers," researchers unaware of whether the patient received real or fake treatment. Nonetheless, this is an imperfect solution. If an acupuncturist or physical therapist applies fake treatment, non-verbal communication of the false nature of the treatment might skew the results. It's better to compare real physical therapy against an impressive blindable placebo, such as an ultrasound machine that, unbeknownst to the therapist, has been disabled.

If it is difficult to study physical therapy or acupuncture, it may be impossible to study the holy grail of preventive medicine: dietary change. We all "know" that it is important to reduce intake of saturated fat and increase intake of cold-water fish, fiber, fruits and vegetables. However, the source of this common knowledge is highly questionable.

To conduct a double-blind study of a diet, one would have to put all the food in capsules, and that is simply not going to happen. Furthermore, when diet is involved, there are too many interlinked variables for a clear study design: if one wishes to evaluate the effect of reduced saturated fat intake, one has to decide how to make up for the missing calories, or one is also studying the effect of a reduced calorie diet. With what should one replace those calories? Simple carbohydrates? Complex carbohydrates? Protein? Polyunsaturated fat? Monounsaturated fat? Each of these options adds new variables to the experiment.

For these and other reasons, proper double-blind placebo-controlled studies of diet are not possible. When scientists wish to discover the health effects of diet, they must resort to far weaker forms of evidence. The most widely used form of research into the health effects of diet is the observational or epidemiological study. Unfortunately, such studies are highly unreliable, and may lead to conclusions that are the exact opposite of the truth.

The Observational Study

In an observational study, researchers don't actually give people any treatment. Instead, they simply observe a vast number of people. For example, in the Nurse's Health Study, almost 100,000 nurses have been extensively surveyed for many years, in an attempt to find connections between various lifestyle habits and illnesses. Researchers have found, for example, that nurses who consume more fruits and vegetables have less cancer. This finding appears to indicate that fruits and vegetables prevent cancer. However, this is not a reliable inference. All we know from such a study is that high intake of fruits and vegetables is associated with less cancer, not that it causes less cancer. People who eat more fruits and vegetables may have other healthy habits as well, and those could be the cause of the benefit, not the fruits and vegetables.

For a dramatic example of the difficulties, consider hormone replacement therapy (HRT). Up until 2001, it was a common belief among physicians that use of hormone replacement therapy was the single most important step an older woman could take to maintain a healthy heart. This belief was derived largely from observational studies; in such studies, women who used HRT had up to a 50% lower incidence of heart disease than women who didn't.

These robust results appeared to suggest that HRT prevents heart disease. However, as a few researchers pointed out, there are a number of reasons why such a conclusion might be insecure. Women who used hormone replacement therapy were typically in a higher socioeconomic class than women who didn't; they also tended to go to the doctor more often. Both of these factors are independently associated with less heart disease, therefore they (along with other, unrecognized factors) could create an illusion that hormone replacement therapy has a causal effect on heart disease.

Doctors, however, believed they had several good reasons to trust the results of these observational studies. Prior to menopause, women have far lower rates of heart disease than their male cohorts; from the time of menopause onward the rates of the two sexes converge. Since estrogen and progesterone levels fall precipitously in menopause, it seems logical that taking estrogen and progesterone should help prevent heart disease.

Furthermore, estrogen reduces cholesterol levels. Since we know that high cholesterol accelerates heart disease, this finding served as a "surrogate marker" to confirm estrogen's value. (Actually, the case for a causal connection between cholesterol and heart disease is more questionable than is widely believed; only one class of cholesterol lowering drugs, the statins, has been found to reduce heart disease rates. It is possible that the benefits of statins stem from other, unrecognized effects, and high cholesterol may be only a non-causal marker for an underlying causal factor.)

Estrogen also possesses antioxidant properties, and in the early and mid '90s, antioxidants were thought to protect the heart. (This hypothesis has now all but collapsed.) Finally, estrogen has favorable effects on the artery wall that should help prevent heart attacks.

Putting all this evidence together, physicians strongly endorsed HRT for postmenopausal women. In retrospect, this was inexcusable. A drug intended for healthy people should pass even higher--not lower--standards of evidence than a drug used for the treatment of disease. As early as the late 1980s, some women's health groups had pointed out the problem and demanded double-blind, placebo-controlled trials. But it was too obvious that HRT worked. Double-blind studies didn't seem necessary.

When at last, more than a decade later, such a study was performed, it turned out that HRT actually causes heart disease, rather than prevents it. (13) HRT also increases the risk of breast cancer, and, possibly, of Alzheimer's disease as well.

In other words, by placing trust in observational studies, plausible reasoning, and common sense, rather than waiting for the results of double-blind studies, doctors prescribed a treatment that killed many women. This is, in a very real sense, more unconscionable than the indiscriminate marketing of useless but relatively safe treatments by the alternative medicine industry.

"Healthy" Diet

With hormone replacement therapy, it was possible to design a double-blind placebo-controlled trial to verify (or falsify) the results of observational studies. However, since double-blind studies of diet are not possible, there is no similar avenue available to validate statements about the health effects of diet.

Some connections between lifestyle habits and disease act like high effect-size treatments, and show themselves beyond statistical noise. The best and most conclusive example of this is cigarette smoking. Cigarette smoking increases the rate of lung cancer so dramatically that it is obvious that the former causes the latter. However, when it comes to notions about the benefits of fruits and vegetables, fiber, green tea, or the moderate use of alcohol on the one hand, or the harm caused by saturated fat on the other, the presumed effects are of fairly low effect-size, and easily overlap statistical noise. The surprising truth is that we don't know that high consumption of fruits and vegetables is healthy, apart from the mere provision of vitamins. We also don't know that saturated fat is unhealthy.

This is more than surprising. Common sense tells us that saturated fat is bad for you. However, when one sets aside preconceptions and looks at the subject dispassionately, the case for cutting down on saturated fats fails. (14) In fact, it is significantly weaker than the evidence which led physicians to prescribe HRT. Many observational studies indicate that individuals who consume more saturated fat have a higher incidence of heart disease than those who avoid fatty foods. As with estrogen, however, these findings do not by themselves indicate that saturated fat increases heart disease risk. The apparent connection between the two numbers could be a statistical fluke, just as the apparent contribution of estrogen to heart health proved in retrospect to have been a statistical fluke.

Furthermore, while the supporting observational evidence for an estrogen-heart disease connection is quite consistent, the observational case regarding saturated fat is not. In France, for example, high consumption of saturated fats is not associated with excess risk of heart disease. To explain away this so-called "French Paradox," proponents of the saturated fat hypothesis have invoked red wine, olive oil consumption and lifestyle differences. However, Occam's razor suggests a simpler and more obvious explanation: that there's no paradox at all. Without the preconception that saturated fat causes heart disease, the French findings are simply evidence that disconfirms the saturated fat hypothesis. In fact, observational studies performed in other countries have also failed to find a straightforward connection between saturated fat and heart disease. lf observational evidence deserves to be looked at with suspicion, inconsistent observational evidence is entirely untrustworthy.

What about effects on cholesterol? It is true that increased intake of saturated fat raises levels of total cholesterol. Back in the early 1960s, when physicians relied upon total cholesterol as the only cholesterol measurement, this seemed a damning finding. However, current medical practice no longer focuses on total cholesterol alone, but includes measurements of LDL ("bad") cholesterol, HDL ("good") cholesterol, and triglycerides. Increased saturated fat intake does worsen levels of total and LDL cholesterol, but it improves levels of HDL cholesterol and triglycerides. As far as anyone can tell, the outcome is a wash. And, contrary to popular belief, dietary fat doesn't simply deposit on the artery wall. The connection between high cholesterol and hardening of the arteries, if there really is one, is far more subtle. (Furthermore, it is not clear that high fat intake automatically leads to weight gain. Studies on the Atkins diet have more or less disproved this widely accepted theory.)

Thus, reducing saturated fat might prevent heart disease, have no effect on heart disease, or even increase it. The same goes for all the other statistical associations widely reported in the press: for example, that green tea prevents cancer, that moderate alcohol use prevents heart disease, and that a high intake of fruits and vegetables prevents arthritis. For each of these assertions, the evidence at present is limited to inconsistent observational trials and questionable surrogate markers; any dietary suggestions made on the basis of such weak evidence are highly speculative, and could be dead wrong.

It is possible to test green tea in a double-blind study. However, as noted above, it is not possible to double-blind a study of diet. For this reason, it is difficult to make a case that a diet consisting of junk food plus a multivitamin is any less healthy than any other (provided one does not become obese); it is also difficult to conceive of how to properly make such a case. Nonetheless, medical authorities are inclined to give, and the public to piously accept, sweeping health rules that make verboten whole classes of enjoyable foods. This has resulted in a psychological habit, a kind of health puritanism, and it easily jumps chasms of unknowing to arrive at specious certainties.

We so much want to know the right thing to do to ward off the spectre of illness. However, the mere fact that it would be satisfying to know something doesn't make it knowable. The relationship between diet and health is confounded by thousands of low-effect-size, non-independent variables, most of whose effects we do not know, and have no way to know. It is therefore possible that the true nature of a healthy diet may remain unsettled until the day that we can reverse-engineer the body and understand its function from scratch.

Double-Blind Studies Outside of Medicine

The results of double-blind studies indicate that humans are poor observers of highly variable data like the effects of medical treatments. They also show that our attempts to deduce the effects of treatment by indirect reasoning are quite fallible, and can lead to conclusions that are dead wrong. It seems reasonable to expect that similar errors occur outside of medicine as well.

Physicists have learned that if they look at cloud chamber results expecting to find the track of a particular particle, they are likely to observe the track they wish to see. For this reason, blinded analysis techniques are now common among particle physicists. But what about other areas of hard science that do not use blinding? How many results in the scientific literature are based on observer bias?

If the hard sciences are problematic, one wonders whether there is any rational reason to favor any conclusions drawn by the soft sciences. Consider economics. An economy is a complex system with multiple interlocking variables and a great range of natural variation, much like a human body. Because it isn't possible to change just one variable in an economy, nor to try an experiment twice by restarting an identical society under new roles, economic analysis is really the equivalent of observational evidence (economic records) combined with plausible reasoning (an economic model). In medicine we have seen that observational evidence and plausible reasoning can lead to conclusions that are the direct opposite of the truth. How, then, should a rational person form an opinion on whether, for example, lowering (or raising) taxes on the rich will benefit (or harm) the economy as a whole, or affect a given class of people within that economy?

This is an area where the consequences of an action can markedly affect the lives of a great many people. When we cannot in any rational sense know the outcome of a change, why do we tend to have such strong opinions on what course of action is best.? Is it anything more than pure emotion, or raw prejudice dressed up after the fact with reasons?

Perhaps it is true that when it comes to political decisions, one has a duty to act on the best evidence available. If so, it is sobering to note that, as with hormone replacement therapy, the ultimate effects may be exactly the opposite of those we intend to achieve.

References

(1.) Solomon S. 2002. "A Review of Mechanisms of Response to Pain Therapy: Why Voodoo Works." Headache, 42:656-62.

(2.) MacLennan, A., S. Lester, V. Moore. 2001. "Oral Estrogen Replacement Therapy Versus Placebo for Hot Flashes: A Systematic Review." Climacteric, 4:58-74.

(3.) Nickel, J.C. 1998. "Placebo Therapy of Benign Prostatic Hyperptasia: A 25-Month Study." Canadian PROSPECT Study Group. Br J Urol., 81:383-387.

(4.) Moseley, J.B., K. O'Malley, N. J. Petersen, et al. 2002. "A Controlled Trial of Arthroscopic Surgery for Osteoarthritis of the Knee." N Engl J Med., 347:81-8.

(5.) Taylor, J.A., A. H. Novack, J. R. Almquist, et at. 1993. "Efficacy of Cough Suppressants in Children." J Pediatr., 122:799-802; Croughan-Minihane, M.S., D. B. Petitti, J. E. Rodnick. 1993. "Clinical Trial Examining Effectiveness of Throe Cough Syrups." J Am Board Fam Pract., 6:109-115; Schroeder, K., T. Fahey. 2001. "Over-the-counter Medications for Acute Cough in Children and Adults in Ambulatory Settings." Cochrane Database Syst Rev., CD001831; Paul, I.M., K. E. Yoder, K. R. Crowell, et al. 2004. "Effect of Dextromethorphan, Diphenhydramine, and Placebo on Nocturnal Cough and Sleep Quality for Coughing Children and Their Parents." Pediatrics, 114:e85-90.

(6.) Cherkin, D.C., R. A. Deyo, M. Battle, et al. 1998. "A Comparison of Physical Therapy, Chiropractic Manipulation, and Provision of an Educational Booklet for the Treatment of Patients with Low Back Pain." N Engl J Med., 339:1021-1029.

(7.) Fink, M., M. Karat, B. Wippermann, et al. 2001. "Non-Specific Effects of Traditional Chinese Acupuncture in Osteoarthritis of the Hip: A Randomized Controlled Trial." Complement Ther Med, 9:82-88.

(8.) Carter R., T. Hall, C. B. Aspy, et al. 2002. "Effectiveness of Magnet Therapy for Treatment of Wrist Pain Attributed to Carpal Tunnel Syndrome." J Fam Pract., 51:38-40.

(9.) Irnich D., N. Behrens, H. Molzen, et al. 2001. "Randomised Trial of Acupuncture Compared with Conventional Massage and Sham Laser Acupuncture for Treatment of Chronic Neck Pain." BMJ, 322:1-6.

(10.) Hrobjartsson A., P. C. Gotzsche. 2001. "Is the Placebo Powerless? An Analysis of Clinical Trials Comparing Placebo with No Treatment." N Engl J Med., 344:1594-1602.

(11.) Noseworthy, J. H., G. C. Ebers, M. K. Vandervoort, et al. 2001. "The Impact of Blinding on the Results of a Randomized, Placebo-Controlled Multiple Sclerosis Clinical Trial." Neurology, 57:S31-5.

(12.) Pappas, D.E., Owen Hendley. 2003. "J Otitis media. A Scholarly Review of the Evidence." Minerva Pediatr, 55:407-14.

(13.) Manson, J.E., J. Hsia, K. C. Johnson, et al. 2003. "Women's Health Initiative Investigators. Estrogen Plus Progestin and the Risk of Coronary Heart Disease." N Engl J Med., 349:523-34.

(14.) Taubes G. 2001. "The Soft Science of Dietary Fat." Science, 291:2536-45

COPYRIGHT 2005 Skeptics Society & Skeptic Magazine