Cochrane Consumer Network

What if there is no clear evidence one way or the other of the intervention under review?

Submitted by Catherine McIlwain on Thu, 14/07/2011 - 18:52

I would like to run past Consumers a problem I have had with comments that have turned up on the Internet that have been taken out of context. I personally have never put anything on the net. Also I have never seen the following subject raised in discussion in Consumer circles when review results are not clear-cut.

I am the lead person of the review Speleotherapy for Asthma, Cochrane Collaboration Review, Beamon, S., Falkenbach, A., Fainburg, G., Linde, K., published on CD Rom, Cochrane Library, 1999. Updated regularly. I questioned this matter with Airways Entity years ago and below are the relevant comments.

'When referring to speleotherapy reviews where there is no clear evidence to support the use of an intervention. It is a complex methodological issue - just because a review does not provide evidence to support the use of an intervention readers should not conclude that the intervention is necessarily unhelpful. One can make fairly strong conclusions from a meta-analysis that demonstrates a clear treatment effect, but we can never draw conclusions (in favour of either direction) from a meta-analysis that demonstrates no clear treatment effect.

The history of evidence based medicine is littered with examples of people wrongly concluding that an absence of evidence is evidence of no benefit ...’

This is not true as not proved either way.

‘... A review can give you a clear answer only if there are high quality clinical trials that present us with that evidence.’'

There was, and still is, a PAUCITY of speleotherapy systematic randomised controlled trials.

I don't think these problems are raised enough or explained and I would be interested in what other Consumers think.

* Speaking as a former chronic asthmatic I personally believe in speleotherapy treatment. I undertook such therapy in both Romania (1995) and Poland (1997), then was completely free of asthma for five years and only experience mild symptoms occasionally since (2011). I realise of course I am prejudiced !

Sylvia P. Beamon (Airways Entity)
sylvia@salisburyvillas.freeserve.co.uk

Please click "Add new comment" below to add your comments.
***Don't forget to click "Preview" then "Save" to submit your comments.

Page last updated: Thu 14th Jul 2011 20:18:41 CEST

Add new comment

27 Oct19:50

The Cochrane Empty Reviews Project

By Lindsay Shepard (not verified)

An interesting instance in which a systematic review may provide no clear evidence of effectiveness one way or the other, is when no studies meeting the inclusion criteria are identified. These so-called “empty reviews” might occur in new areas of study, in topics in which there is some urgency to find interventions that work but there is no high quality research, or in reviews with very specific population, setting, diagnostic, intervention, or methodological inclusion criteria. However, the same problem highlighted by this post appears to be applicable to empty reviews, wherein some reviewers and readers may mistakenly interpret no evidence as evidence of no effect. As such, this issue is currently being considered by The Cochrane Empty Reviews Project, funded by The Cochrane Opportunities Fund. The Empty Reviews Project has examined the incidence and basic characteristics of empty reviews and presented project findings at the 2010 Cochrane Colloquium in Keystone, Colorado and prepared a manuscript for publication. Presently, The Empty Reviews Project team is in the process of developing recommendations for the Cochrane Collaboration regarding guidance for the reporting of empty reviews. These recommendations will be based on a June 2011 consensus meeting of Cochrane collaborators and an October 2011 meeting and presentation at the Cochrane Colloquium in Madrid, Spain. For further information regarding The Cochrane Empty Reviews Project, please visit http://empty-reviews.org/, or email cochrane.empty.reviews@gmail.com. Paul Montgomery, PhD, University of Oxford, PI Sally Hopewell, PhD, UK Cochrane Centre, Co-PI Joanne Yaffe, PhD, University of Utah, Co-PI Lindsay Shepard, MSc, University of Utah, Research Associate

19 Jul11:19

Evidence of effectiveness / ineffectiveness

By Ben (not verified)

As a starting point (if this hasn't been done already), wouldn't it be helpful to: 1 Agree some criteria for classifying a review as one of the following: demonstrating effective treatments; demonstrating in-effective treatments; no clear results 2 Classifying all reviews into one of these categories and allowing the database to be filtered on these criteria

15 Jul18:21

No clear evidence one way or the other

By Lorraine Johnson (not verified)

The adage is the lack of proof is not proof of lack. The fact is that there are a number of reasons that there may not be sufficient evidence supporting efficicacy. There may not have been clinical trials. It may be a pharmaceutically disadvantaged research area. Most of our clinical trials come from pharma; if pharma is not interested because the market is not seen as desirable (not big enough or treatment not long enough or off patent medications), the research may not be done. There may be clinical trials that are underpowered (not large enough) to show clinically relelvant effects. There may be trials, but they may be measuring effects that were larger than those that would be clinically relevant. This happens when researchers attempt to offset small sample sizes with larger treatment effects that are not not the "minimal clinical treatment" effect. The efficacy here will only show up with minimally clinically relevant treatment effects using larger sample sizes--design short cuts that occur when funding or or other constraints (e.g. rare disease) make the use of the larger sample sizes necessary to would demonstrate efficacy impractical. When any of these is the case, the science is uncertain. When the science is uncertain patients still require care and denying care until sufficient research is done is not the appropriate response from a social justice or equity perspective; it is also not the way to make progress in a disease. Many innovations and advancements in medicine come from clinical care experience. In these situations, the default should be to clinical judgment and patient preferences. If we do not have the science, we should not constrain autonomy and choice. The physician should be able to utilize available treatment modalities to restore the patient to health based on whatever evidence is available (non-controlled trials, case studies, basic science principles).It would be great if Cochrane assessments made it clear that lack of proof should not be conflated with lack of efficacy.Lorraine Johnson, JD, MBACalifornia Lyme Disease Association

05 Aug18:12

No clear evidence

By Jim Gudgeon (not verified)

One's view of this issue is, of course, colored by one's experiences and "agenda". As both an evidence-based practitioner, consumer of healthcare services, and payer of both taxes and insurance premiums I have to say that when someone, whether clinician or patient, is asking others to (help) pay for a service that the rules established by those representing the payers must be followed. In most developed countries this means that unless and until the weight of evidence supports the belief that the service provides more benefit than harm and, in many but not all countries, there is reasonable value for the benefit (/harm). The general view in this setting is that the starting point is that new healthcare services are considered to be investigational until they are able, through acceptable evidence, to pass the criteria established above (which is pretty well understood). Of course, if the clinican wants to provide the service without charge or the patient is willing to pay for the service then they can set their own rules. While I agree with Lorraine's stated view that services/treatments that aren't linked with patents and big money often struggle with gaining sufficient evidence to "pass muster", it is also true that many such services do, when studies do get done, fail the evidence bar. In either case, patients/individuals retain the right to obtain the service, and accept responsibility to pay.

Search

Cochrane Consumer Blog

top of this page disclaimer The Cochrane Library The Cochrane Collaboration Admin