Guide for Evaluating Therapy and Research Claims

A Guide to Analyzing New Therapeutic Methods for the Consumer

Many of us search the internet looking for anything that may help the person with aphasia. Sometimes we hear about something and then “Google” for information about it. We may hear about behavior therapies, drugs for stroke, or surgical procedures such as brain implants. The best thing to do first is to ask your doctor or your speech-language pathologist about unfamiliar methods. It is also good to do your own investigation.

How do we evaluate new therapeutic methods, especially those with attractive sites on the internet?

Do not stop at the home page. Search through the website. Evaluate the way the site is written. Is it reasoned and cautious? Or does it glow with superlatives and promises? Do not just look at the product’s website. Do a search of the product’s name. Look for independent reviews or blogs about the product. Wikipedia can be helpful as a starting point.

Be aware of terminology that is general. It may not pertain to your situation. Here are examples:

APHASIA: “This therapy helps people with aphasia.” What type of aphasia? Is it like my spouse’s aphasia? Positive results may have been obtained only for mild cases or only for those with good comprehension.

ACTIVITIES OF DAILY LIVING: “Our research showed that people recovered functions important for daily living.” Were positive results found for language? Positive results may have been obtained for walking, vision, or attention, but not necessarily for language.

BRAIN: “This therapy changes the brain.” This could refer to any part of the brain. For aphasia, we are most interested in what happens to the cortex in the left hemisphere. Changes in the brain may have occurred in areas that are not related to language.

Let us focus on drugs as an example. To be confident in drugs, first ask your doctor, and then ask yourself:

• What is the company producing the drug? Is it a familiar pharmaceutical company? Where is the company located?
• Was it approved in the United States by the Food and Drug Administration (FDA)?
• Who are the officers of the company? Who serves on its panel of advisors or consultants? Are any of the consultants affiliated with universities that you know?
• Promises are good enough to be true when they are supported by evidence. Does the site present evidence to support its claims? Investigate whether research was conducted by scientists who are independent of the company making the product. You may find this information in your search for other sites.
• Does the site list references for studies that appeared in medical journals? Professional journals publish research that has been approved by a panel of independent experts, called “peer review.”

When the site provides evidence and describes studies, we consider the following:

Some evidence may consist of individual clients telling their story about how a therapeutic procedure was helpful. These are called testimonials.

A testimonial has three limitations:

1. The website may report only the successes and ignore the failures.
2. The testimonial may not be accurate. The person could be reporting real progress caused by something else, or the report could be wishful thinking.
3. It may not apply to you. The chances that positive results apply to you are increased when the results are observed in more people, especially those with a similar problem.

A person’s earnest testimonial has value, because it can inspire researchers to design controlled studies of many people.

If the site describes a study, look for certain technical details. Studies of medical treatments are called “clinical trials.” Trials are conducted in a series of phases, gradually building information that should advise and protect the public.

Phase I: researchers test a new drug in small groups for its safety (such as looking for side effects).

Phase II: the drug is tested on a larger group mainly for its effectiveness.

Phase III: continues to pursue issues of Phases I and II. Now, there is a comparison to other treatments in the determination of effectiveness. One example would be to compare a drug to regular language therapy.

Phase IV: after a product is marketed, independent researchers continue to do studies in order to compare different populations and explore the safety of long-term use.

What does it take to prove effectiveness? A website may suggest that studies were double-blind randomized placebo-controlled trials. This implies that the study conformed to rules that minimize any bias that the experimenter might have. But what does it mean? Let us consider drug research.

Placebo-controlled: because people often report being helped by any kind of attention, good studies compare people who receive the drug with a control group of people who receive a sugar pill called a placebo. Similarly, a fake medical procedure might be called a “sham procedure.”

Double-blind: neither the research subject nor the doctor administering the drug knows who receives the drug and who receives the placebo.

Randomized: assignment of subjects to the therapeutic group or the control group is made without any special criteria.

A study with these design features may provide evidence that a person’s improvement was likely to have been caused by the drug or the therapeutic procedure.

The person’s improvement is likely to have occurred in the brain. You may hear or see claims that a therapeutic method reorganized or “rewired” the brain. If so, explore whether researchers actually looked at people’s brains to support such claims. Currently, scientists can take pictures of brain activity using sophisticated brain imaging tools such as functional magnetic resonance imaging (fMRI).

Just taking pictures is not enough to make a claim that changes in the brain were caused by the therapy or drug. The changes might have happened anyway. A control group that did not receive the therapy is important for suggesting that reorganization of the brain was caused by the therapy. There are studies that contain adequate controls and indicate that a language therapy changed the brain.

In general, scientists are very cautious about their conclusions. No study is perfect. Every study has flaws. A good scientist is honest about citing the limitations of his or her research.

Click here for a downloadable pdf of this guide.

Written by G. Albyn Davis, Ph.D., CCC-SLP (July 2012)