Evaluating Treatment Research: How do I know what to choose?

By Katy Dorendorf, Psy.D.

Over 230 different psychotherapeutic treatments have been proposed for the variety of developmental and psychological disorders that appear in children and adolescents, yet only a few can be labeled effective based on scientific research (Gold, Voracek, & Wigram, 2004). Moreover, information regarding the effectiveness of these treatments is often only presented in scholarly research journals. While professionals have access to these journals and specific training to evaluate the risks and benefits of treatments, parents and family members have little access to academic journals and limited training in the evaluation of good scientific research. This leaves families at a disadvantage for determining what treatments are actually based on good research and limits their ability to be advocates for their child’s care. So here are some of the buzz words the professionals know and some tips you can use, so you can begin to understand the basics of good clinical evidence and feel confident in making the right treatment choices for your family. Note: Buzz words are italicized for easy identification.

Professionals using scientifically based treatment methods evaluate the strength of different types of clinical evidence and research to make overall decisions about the benefits and risks of different treatment methods (APA, 2002). First professionals typically evaluate the quality of clinical evidence and research based on nothing other than mathematics, well statistics to be exact. Various elements are taken into account including the statistical strength of the research design, the amount of bias that is built in to the research design, and the size and similarity of those who participated in the research to the overall population to whom the treatment will be applied (APA, 2002; Research Advocacy Network, 2005; Robson, 2002). Therefore, when evaluating clinical evidence to determine if a treatment is promising and may someday meet the standards of evidence based practices put forth by professional health organizations such as the American Psychological Association, it is important to have a basic understanding of the ranking of research methods and designs used in clinical research.

Generally, meta-analysis is considered one of the strongest and least biased types of research, as it involves a comprehensive and inclusive statistical review of many well-designed research studies at once (Research Advocacy Network, 2005; Robson, 2002). Meta-analysis also provides strong evidence due to the increased amount of information and statistical power achieved when combining a variety of research subjects. So if you see a treatment is supported by one or more meta-analyses, you can be fairly confident the treatment is supported by science.

Additionally, randomized control trials or RCTs are considered to produce strong and unbiased clinical evidence, especially when the study is designed to be double-blind (i.e., both the researcher and the subject are unaware of the treatment method the individual is receiving; APA, 2002; Robson, 2002). However, single-blinded (i.e., the subject is unaware of the treatment method) and non-blinded RCTs can also produce strong evidence when double blinding is not possible due to limitations within the population or ethical concerns associated with the study (Research Advocacy Network, 2005). So if a treatment is showing support in two or more substantially sized RCTs it is usually on it’s way to being considered evidence based if it hasn’t already gotten there yet.

Quasi-experimental studies, such as nonrandomized experimental designs, controlled experimental designs, single-group experimental studies, pre-test and post-test studies, and cohort studies, can at times produce strong clinical evidence when RCTs are not able to be attempted (APA, 2002; Research Advocacy Network, 2005; Robson, 2002). However, these studies are often easily influenced by the beliefs of both the researcher and the subject. Additionally, quasi-experimental studies are often more easily impacted by the presence of confounding factors (i.e., variables the research cannot control that also impact the treatment results). So if you see research using a quasi-experimental method, the treatment may be effective, but it is important that the authors of talk about how their biases and the different confounding factors involved in the treatment could have impacted their results. If you do not see a section addressing these concerns it is likely you are looking at biased and unsatisfactory research.

Evidence from non-experimental studies such as comparative studies, correlational studies, surveys, and case-studies, falls among the lowest rankings when evaluating research strength as these designs are easily biased by researchers and subjects alike and do not allow for control over multiple confounding factors (APA, 2002; Robson, 2002). While non-experimental studies can start researchers on the right track they are not sufficient for determining if a treatment will be effective or safe for a larger population. Media outlets are notorious for reporting on the relationship between different behaviors and disorders or illnesses (i.e., vaccines and autism, cancer and aspartame, etc.). It is important to know when you see the word relationship authors are typically referring to correlational non-experimental studies. It is important to remember that “correlation does not equal causation,” but simply implies an association between the two events which can often times be indirect (Robson, 2002). This is a critical principal in the understanding of good research.

The least reliable and most biased evidence tends to come from case reports, clinical examples, and anecdotal reports (APA, 2002; Research Advocacy Network, 2005). Evidence gathered from those sources should be approached with caution and interpreted with the understanding they are likely to reflect the biases of the author and are typically not generalizable to the population as a whole. Many complementary and alternative methods of treatment are supported by this type of evidence. One way to spot treatments supported by this type of evidence is if a website or article provides many quotes from satisfied customers or families. These testimonials are often used to obscure the fact that very little scientific evidence is reported about the treatment.

So how do you spot a treatment that has limited research? Well it is simpler than you think. Here are a few tips you can use to sort out the good research from the bad:

• Look for a large amount of testimonials.
• Look for the word relationship and If the study is attempting to indicate one event is causing a disorder or difficulty it should also contain the words Randomized Controlled Trial, Meta-Analysis, or qualify as a quasi-experimental design.
• Look for research claims that are not attributed to a study. If someone is writing about research they should cite the research article where the information came from. If you cannot find attributions or links to the articles they are citing it is likely the research is not solid.
• Look for claims support by research that is not published in academic or professional journals. Anyone can do research and many companies do, but just because there is research on a product doesn’t mean it is good research. Academic and professional journals generally have requirements about the types of research they publish and so information published in these sources is more likely to meet the standards of high quality research.
• Ask your provider for copies of articles that support the treatment they are recommending. If the treatment is truly evidence based they should have access to articles that meet the above criteria for good research. If the provider is unwilling to provide you with such documentation it is likely they do not have good research supporting their claims.
• Look for images of brains. Brains have been found to increase the likelihood that individuals will believe claims about a product. Images of brains alone do not indicate that the research is bad, but it is important to take this as a cue to read the research carefully. For more information about how to protect yourself from this kind of marketing watch the following short TED talk by Dr. Molly Crockett (http://www.ted.com/talks/molly_crockett_beware_neuro_bunk?language=en)
• Trust your gut. If something sounds too good to be true probably is. Unfortunately, advertisers are often willing to take advantage of the distress that can often result after a diagnosis of a developmental or psychological disorder. Your judgement is your best weapon against bad research and those who would try to take advantage of you and your family.

References

American Psychological Association (2002). Criteria for evaluating treatment guidelines. American Psychologist, 57(12), 1052-1059. doi: 10.1037//0003-066X.57.12.1052

Crockett, M. (2012, November). Beware neuro-bunk [Video File]. Retrieved from http://www.ted.com/talks/molly_crockett_beware_neuro_bunk?language=en

Gold, C., Voracek, M., & Wigram, T. (2004). Effects of music therapy for children and adolescents with psychopathology: A meta-analysis. Journal of Child Psychology and Psychiatry, 45(6), 1054-1063.

Research Advocacy Network (2005). Skill builder shortcut: Level’s of evidence. Retrieved July 6, 2012 from https://ran.memberclicks.net/assets/Skillbuilders/shortcut%20sheet%20-%20levels%20of%20evidence%20w%20suny%20tutorial.pdf

Robson, Colin (2002). Real World Research (2^nd ed.). Malden, MA: Blackwell Publishing.

‍