Negative studies published in medical journals of India do not give sufficient information regarding power/sample size calculation and confidence interval

Jaykaran¹, D Saxena², P Yadav¹, ND Kantharia¹,  
¹ Department of Pharmacology, Govt. Medical College, Surat, India
² Department of Community Medicine, Govt. Medical College, Surat, India

Correspondence Address:
Jaykaran
Department of Pharmacology, Govt. Medical College, Surat
India

Full Text

Sir,

Many studies published in medical journals are considered as negative studies as they fail to disprove the null hypothesis. These kinds of studies should be interpreted cautiously as chances of false interpretation are large. A reader can consider an intervention not statistically significant as compared to comparator (placebo or drug) on the basis of negative results but this difference may be because of either actual no difference or because of less sample size hence less power or by chance (Type 2 error). So a study should have an adequate sample size/ power so that negative results because of less sample size can be ruled out. One most important reason for these underpowered studies is: no calculation of sample size during early phase of the study. [1] Sample size/power calculation is very important during the design phase of the study and its method should be reported in sufficient detail so that readers can asses the validity of the study. [1] Sufficient information about the methodological parameters should be given in the published manuscript of negative studies so that readers can asses the validity of the study independently as a poorly conducted negative study may not be valid but may discourage the reader to do work in the same area. This is very important in the light of findings which show that many negative studies are underpowered to see moderate and large differences in outcome and many of them have incorrect statistics. [2],[3] In the case of negative studies, reporting of one more component is important and that is the confidence interval. Confidence interval gives a range in which the actual population value may lie and also gives an idea about the clinical significance. [4] Studies done for Western journals show that information about power/sample size and confidence interval is not reported adequately in published studies. [5],[6],[7],[8] But information is lacking about the studies published in Indian medical journals and specially negative studies, so this study was designed with the aim of surveying negative studies published in Indian medical journals to see whether or not adequate information is given regarding the sample size/power calculation and confidence interval.

PubMed indexed journals subscribed by the central library of our institute were taken into consideration. These journals were Annals of Indian Academy of Neurology (AIAN), Indian Journal of Orthopedics (IJOrtho), Indian Journal of Critical Care Medicine (IJCCM), Indian Journal of Dermatology, Venereology and Leprology (IJDVL), Indian Journal of Nephrology (IJN), Indian Journal of Dermatology (IJD), Indian Journal of Ophthalmology (IJO), Indian Journal of Urology (IJU), Indian Journal of Anesthesia (IJA), Indian Journal of Psychiatry (IJPsy), Indian Pediatrics (IP), Indian Journal of Medical Research (IJMR), Indian Journal of Medical Science (IJMS) and Indian Journal of Community Medicine (IJCM). All the original articles published in these journals in 2009 were evaluated by first author (JK) to sort out the negative studies from them. Studies were considered negative studies on the basis of these criteria- one primary outcome is not statistically significant, second most important outcome is not statistically significant (if primary outcome is not reported). Most important outcome was considered to be that outcome for which either sample size calculation was done or which came first in the abstract. [9] All negative studies were evaluated by first author (JK) for reporting of power, sample size and confidence interval. Second author (DS) reassessed all the negative studies (k = 0.85) and any discrepancies were resolved by consensus. Descriptive statistics in the form of frequency and percentage were used. Reporting of sample size in clinical trials and other studies were compared with Fisher's exact test.

Out of 276 articles published in these 14 journals in 2009, 52 (18.8%, 95% CI 14.6-23.8%) were negative studies [Table 1]. Out of 52 negative studies 39 (76.4%) were prospective studies, nine (17.6%) were clinical trials and four (7.6%) were retrospective studies. Out of 52 negative studies sample size calculation was reported in only 10 (19.2%, 95% CI 10.8- 1.9%). Out of these 10 articles all components of sample size calculation (Type 1 error, Type 2 error, power, delta) was reported in seven articles, only power was reported in the remaining three articles. Out of the seven articles reporting effect size (delta), rational for selection of particular effect size was mentioned in only 3 articles . Out of 52 negative studies confidence interval was reported in only three (5.7%, 95% CI 1.9-15.6%) studies. Average sample size was 54.3. Most of the studies had a sample size less than 40 per group [Table 2]. It was observed that reporting of sample size was more in clinical trials as compared to other studies (5/9 Vs. 5/43, P = 0.010) [Table 3]. Out of these 52 negative studies lower sample size/power was mentioned as a limitation in 14 (26.9%, 95% CI 16.7-40.2%) articles. {Table 1}{Table 2}{Table 3}

On the basis of this study it can be concluded that negative studies published in Indian Medical Journals are grossly deficient in the reporting of power/sample size calculation and confidence interval hence it is difficult to check the validity of these studies. These findings are similar to studies done for Western journals. [3],[4],[5],[6] In a study done by Moher et al., it was observed that sample size was calculated in only 32% negative studies which are more than observed in this study. [7]

It is observed that many of the medical journals are biased towards publication of positive studies. Almost 90% of published articles deal with positive results. A properly conducted negative study should be published so that a researcher planning to do similar work may get some idea about the probable results and hence can save resources. Most of the meta-analyses and systematic reviews are based on published literature so the conclusions reached in these reviews may get biased as most of the published studies are positive studies. [10],[11]

References