Power failure

 

powerfistIn a highly cited paper in 2005, John Ioannidis answered the question ‘Why most published research findings are false’  (PLoS Med. 2, e124). The answer, in one sentence, is ‘because of low statistical power and bias’. A current analysis in Nature Reviews Neuroscience ‘Power failure: why small sample size undermines the reliability of neuroscience’ (advance online publication, Ioannidis is a coauthor) now focuses on the neurosciences, and provides empirical evidence that in a wide variety of neuroscience fields (including imaging and animal modeling) exceedingly low statistical power and hence very low positive predictive values are the norm. This explains low reproducibility (e.g. special issue in Exp. Neurol. with (lack of) reproduction in spinal cord injury research, Exp Neurol. 2012 Feb;233(2):597-605) and inflated effect sizes. Besides this meta-analysis on power in neuroscience research, the article also contains a highly readable primer on the concepts of power, positive predictive value, type I and II error, as well as effect size. Must read.

 

Diversity outcross mice

mice

Most rodent models of disease (in stroke research, anyway) use young, healthy male, inbred mouse strains kept under specific pathogen free (SPF) conditions, restricted antigen exposure in their environment,  and on a diet optimized for maximum reproduction rates  (high in antioxidants, trace elements and other supplements, etc.). It is like studying cohorts of 12 year old male identical twins kept on an identical health diet in a single  sealed room, without any contact to the outside world  (the ‘plastic bubble’). What may be good for reproducible modeling, is potentially problematic for translational research, as patients often have comorbidities (e.g. hypertension and diabetes in stroke), already take various medicines, are elderly, and include females… Thus, external validity of the models often is low, at least partially explaining some of the failures when moving from promising new therapeutic strategies in rodents to real life patients in randomized clinical trials. Fortunately, external validity can be improved by studying comorbid animals at advanced age and of both genders. It is trickier in rodents to produce a mature immune system  that had contact with pathogens and multiple antigens. The answer to reduced genetic diversity  may be to use populations specifically developed to provide wide genetic variability, such as the diversity outbred population or the partially inbred collaborative cross strains developed by the Jackson Laboratory. However, in my field  (stroke research), which is particularly hit hard by the ‘translational roadblock’ I have not seen a single study making use of these strains.

Domestic cats kill billions of mice in US

cat

A new study estimates that domestic cats kill 1.4–3.7 billion birds and 6.9–20.7 billion mammals (mostly mice) annually in the United States. PETA posits that 100 million mice and rats are used in animal experiments per year in the US.

Cherrypick your h-index!

Scientometrics are increasingly used to evaluate scientists for positions, etc. Some while ago, citation numbers for individuals (and derived parameters, such as the h-index) could only be obtained via Thomson Reuters  ISI Web of Science. Then came Elsevier’s Scopus, and now we also have Google Scholar Citations. Most reseachers use them without much thinking about them, and quite often without referencing the specific source they used to obtain their personal metrics. However, the citation counts and h-indices calculated by these 3 services may be very different.

Continue reading →

Nature Neuroscience initiative to improve reporting

Nature Neuroscience is currently  undertaking an initiative to improve statistics and methods reporting.

Making methods clearer : Nature Neuroscience : Nature Publishing Group.

Following an initative of the NIH/NINDS (Landis et al.), Nature Neuroscience is testing a new scheme to improve reporting, and consequently to reduce bias and faulty statistics in work published in their journal. Authors have to fill out at very detailed checklist (stats checksheet nat neurosci source file ud) which is sent to  the reviewers. Other journals had checklists before, but they were pro forma, and mostly ignored. This checklist looks very bureaucratic, and authors will hate it, but it contains all the relevant questions (stats including power; bias including blinding and randomization; ethics; detailed reporting of strains, animal husbandry, material; design of functional imaging studies, etc.). It forces authors to think about these issues, and if they haven’t done their homework before designing the experiments they either have to cancel submission, or fake entries into the sheet. Which would put them in clear violation of good scientific practice, compared to just not reporting that they used an underpowered design without blinding and randomization (which is the current practice)….

Nature Neuroscience should be commended for this initiative. Other journals will hopefully adopt this policy, and reviewers take the time to study the checksheets in order to request clarification from the authors or pull the plug on publication of flawed papers.

(De)Personalized Medicine?

(De)Personalized Medicine.

In todays issue of Science Horwitz et al. correctly point out that the promises of personalized medicine will not be delivered without the integration of the -omic data with individual clinical, social, and environmental information. One of the examples they use to make their point is to ask what would have happened if the -omics based personalized treatment approach would have been applied to native American indians (who have a genetic predisposition to diabetes) at the time before they switched to a Western diet – exagerated risk assessment and overtreatment.  It sounds trivial, but it is indeed important to stress the relevance of gene-environment interactions. Horwitz et al argue that in most common diseases environmental factors dominate genetic ones. Unfortunately, the article does not tell us how to implement the ‘enriched approach’ to personalized medicine  they suggest.

Do fat guys live longer?

ImageA recent meta- analysis in JAMA by Flegal et al has created a major media frenzy and angry reactions in the public health research community by concluding: “Relative to normal weight, both obesity (all grades) and grades 2 and 3 obesity were associated with significantly higher all-cause mortality. Grade 1 obesity overall was not associated with higher mortality, and overweight was associated with significantly lower all-cause mortality.”

At a time when childhood underweight has been downgraded from #1 global risk factor in 1990 to #8 in 2010, while high body mass index (BMI) rose from #10 to #6 (Lancet. 2012 Dec 15;380(9859):2224-60. doi: 10.1016/S0140-6736(12)61766-8.) it is counterintuitive to learn from Men’s Health that fat guys live longer. Did the media misinterpret the JAMA study, or is Dr. Walter Willett (Harvard School of Public Health,  Chair, Nutrition Department) right :  “Stated politely, the paper is a pile of rubbish” and  “It is clear that the Flegal study is misleading and should be ignored by health professionals and the general public. ”

The Harvard School of Public health has staged a symposium on Feb. 20 (check out the videocast), asking “Does being overweight really reduce mortality?” The participants squash the JAMA study. Main arguments: The low BMI groups are biased by the sick and smokers, and high quality studies containing data from 6 million individuals were excluded.

Irrespective of potential weaknesses of the JAMA study: Its data, as well as previous studies, suggest that the nadir of the J-shaped BMI vs. mortality curve is farther to the right than most people think, close to a BMI of 25. Of note, no data exists to demonstrate that a dietary intervention to bring BMI to the nadir (fasting, or  ‘overeating’, depending from which side of the curve you come) reduces individual mortality!

Does negative publication bias exist?

Positively negative – are editors to blame for publication bias? | Discussions – F1000 Research.

In this interview Stephen Senn is correct in pointing out that the argument, recently voiced by Ben Goldacre in ‘Bad Pharma’ that there is no negative publication bias is flawed. “Senn explains […] that the flaw in the argument of ‘no editorial bias’ is the assumption that the papers submitted to any given journal are more or less of the same quality, regardless of whether they are positive or negative. What we’re apparently overlooking here are the papers that aren’t being submitted.”

 Read the full article at F1000research.com:

http://f1000research.com/articles/1-59/v1

Genomic responses in mouse models poorly mimic human inflammatory diseases

Genomic responses in mouse models poorly mimic human inflammatory diseases.

Fig. 1.

This PNAS paper has been featured in the lay press – from the New York Times  in the US to Der Spiegel in Germany. Its major conclusion is indeed disturbing:  ‘Here, we show that, although acute inflammatory stresses from different etiologies result in highly similar genomic responses in humans, the responses in corresponding mouse models correlate poorly with the human conditions and also, one another.’

It should be noted that the findings cannot be simply generalized to other models and settings. Importantly, the authors have studied genomic responses of blood cells , not mechanisms directly related to the inflammatory stimulus (burn, trauma, etc.). In addition, only the genomic response of blood cells was assessed, not translation of these genes into proteins. In the discussion the authors ignore the existing literature, where a number of studies in candidate approaches have shown congruent (qualitatively and sometimes even qualitatively) responses in patient material and the corresponding mouse models (compare for example http://jem.rupress.org/content/198/5/725.long and http://stroke.ahajournals.org/content/39/1/237.long.)

Nevertheless, particularly with respect to immune cells in the blood there are obvious and drastic differences between mouse and man – sending blood from a healthy 1 month old mouse to a clinical routine laboratory would return the diagnosis of an acute lympatic neoplasia. Laboratory rodents are raised in a pathogen free environemt (SPF), their immune system is ‘untrained’, lymphatic, and immature.

Thus, Seok et al. expose an important caveat in the interpretation of rodent studies. There is an urgent need for translational research to use biomarkers to expose similarities and dissimilarities in the pathobiology of rodents and humans, as well as to improve the predictiveness of extrapolating from mouse to man with respect to responses to novel theapies. In addition, we need to increase the external validity of our models, by using rodents which have the same comorbidities, age, and environmental exposure as our patients.