This post has been self-published on Youth Ki Awaaz by K. Just like them, anyone can publish on Youth Ki Awaaz.

WTF: ‘A Large Fraction Of Scientific Research That Gets Published, Is False’

More from K

By Kabir Sharma:

Almost every day we read about a scientific study linking something or the other we eat or do with increased chances of contracting an ailment. Often the next day we read the exact opposite has been proven by science. Which is true?
It turns out, probably neither. A large fraction of scientific research that gets published, is false.

This astonishing revelation is the result of Dr. John Ioannidis- Professor of Health Research and Policy at Stanford University – and team’s pioneering meta-research (research on research) on the truth of published scientific findings.
In his 2005 paper, strikingly titled ‘Why Most Published Research Findings Are False‘, Ioannidis- a sort of a hero in the scientific community now – found two major factors behind this. Non-conformance of scientific good practices, such as choosing nonrandomized (i.e. selective) and small sample sizes, statistical misinterpretation, etc; as well as bias. The very existence of bias, defined as “the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced“, is a most dangerous sign for science.

laboratory-385349_1920
What it means is that if a research team has the incentive to arrive at a particular result, and has enough leverage in their method, it is likely that it would be able to show what it wants to.

Though the paper has become hugely influential and popular among the scientific community it targeted, it surprisingly did not shock many scientists. It seems everyone already had an idea of what was going on.

Consensus Across Fields: Irreproducibility

Ioannidis’ work showed theoretically how, and how much the above factors influence published scientific papers, and focused on the biomedical sector (which has direct implications on healthcare). The findings have since stood consistent with various empirical studies. Three years ago, when drugs company Amgen tried to replicate the “landmark publications” in the field of cancer research, 47 out of 53 could not be replicated. Researchers at pharmaceutical companies have reported that their attempts to replicate the conclusions of peer-reviewed papers fail at rates upwards of 75%.

You can question some of the details of John’s calculations, but it’s hard to argue that the essential ideas aren’t absolutely correct,says Doug Altman, an Oxford University researcher who directs the Centre for Statistics in Medicine.

Over recent years Ioannidis’ findings have shown to hold true in multiple fields of enquiry as well; including psychology, economics, as well as the physicochemical and social sciences.

The major findings of the seminal research work are discussed below.

The Greater The Financial And Other Interests And Prejudices, The Less Likely The Findings Are To Be True.

For a large fraction of studies, the former factor is debilitating. Funders want results to be in line with what they want to project, and they seem to be getting what they want. It is always worth checking to see who funded a piece of research.

A very recent publication showed an astonishing 97% of drug trials back the firm that paid for them. A study finding chocolate to be “scientifically proven to help with fading concentration” was funded by Hershey. Tobacco companies have a long history of funding fraudulent health research — getting the WHO to call this “the most astonishing systematic corporate deceit of all time.” Today this legacy is carried forward by oil companies who are funding scientists denying global warming and dozens of groups sowing doubts about climate change.

Another issue is that scientists need publications to progress in their careers, and people like to see big, flashy results. Hence, there is much incentive to fudge studies. Such manipulation could be done, by “serendipitous inclusion or exclusion of certain patients or controls… commercially available data mining packages actually are proud of their ability to yield statistically significant results through data dredging.

The Hotter A Scientific Field, The Less Likely The Findings Are To Be True

Fast-moving fields of research with many competing teams working on the same problem see high pressure to come up with positive results, resulting in the publication of numerous false findings. And higher the number of teams, lower the probability of a published result being true!

And unfortunately, in many fields the siloed investigator writing grants promising to discover something worthy of the Nobel Prize is still the dominant paradigm.

The Smaller The Study, And Smaller The Impact, Less Likely The Findings Are To Be True

Small studies (taking a small set of people, or animals, or anything else) are far more likely to result in statistically significant results that are in fact a false positive, so they should be treated with caution.

Most effects of a big magnitude – like the link between smoking and lung cancer – have already been recognized. Studies today deal with much smaller effects, and consequently to find meaning with the small ‘signal to noise’ ratio, they need to be a lot more sensitive, which they aren’t.

Delving a little deeper, the extent of these issue becomes clearer.

Publication Bias

Scientific journals, the media, people in general- like to see positive results. Research teams reaching positive and flashier conclusions by fluke, or by coercive means, are thus more likely to be published than others showing negative results, perhaps through more robust analyses.

The problem starts at the funding stage, where only the flashiest proposals get funding, and then are bound to deliver. And indeed, for reasons of career, tenure and funding, it is important for scientists to publish prolifically, resulting in a huge amount of manipulation.

Isabelle Boutron, a professor of epidemiology at René Descartes University in Paris, points out that peer reviewers are influenced by trial results; a study showing they not only favored a paper showing a positive effect over a near-identical paper showing no effect, they also gave the positive paper higher scores for its scientific methods.

Harms Of Little Negative Research Getting Published

Wakefield’s study had concluded that the measles-mumps-rubella vaccine caused autism, and became a huge sensation when it came out. Later, a study definitively refuted it, but failed to get the same response from the public. Indeed, negative results rarely become popular, even if they get published. This gives a skewed representation of evidence. Due to this, studies continue to be referenced for years after they are debunked, and a lot of research builds on that which was wrong in the first place.

Besides, many scientists don’t like to step on each other’s toes. “They feel a lot of pressure not to contradict each other,” said Elizabeth Iorns of Science Exchange. “There’s a lot of evidence that if you do that, it’ll be negative for your career.

Problems In The Peer Review Process

Though scientists constantly praise the value of the peer-review process, researchers admit among themselves that biased, erroneous, and even blatantly fraudulent studies frequently slip through it. Nature, the epochal science journal, stated in a 2006 editorial, “Scientists understand that peer review per se provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth.” What’s worse, the peer-review process often pressures researchers to shy away from doing genuinely groundbreaking research, instead building on the findings of their colleagues (that is, their potential reviewers).

Further, reviewers are often unable to spend enough time to go through papers in all their depth before accepting them. Even worse, famous scientists may suppress via the peer review process the publication of findings that refute their findings, condemning their field to perpetuate false dogma. Empirical evidence on expert opinion shows that it is extremely unreliable.

Monetary Disaster

A series of articles in the Lancet noted that, in 2010, about $200 billion (a staggering 85% of the world’s spending on medical research) was squandered on studies that were flawed in their design, redundant, never published or poorly reported.

Way Ahead

Among suggestions to improve the state of affairs, it is a consensus that steps to ensure replicability of studies, by for example, improving data accessibility must be incorporated. An increasing number of journals have recently begun to emphasize this as well as the importance of negative results, including Nature and Science.

Also, investigators ought to be rewarded for performing good science rather than for submitting flamboyant grant proposals and papers making extravagant claims. Better peer review at multiple levels, including post-publication and promoting large scientific teams are other suggestions.

Ioannidis, now director of the new Meta-Research Innovation Centre at Stanford (METRICS) designed to perform research on research, still remains a great believer in the scientific method; and feels it is the greatest thing achieved by humans. “I am optimistic. I think that science is making progress. There’s no doubt about that. It’s just an issue of how much and how quickly.

You must be to comment.
  1. ItsJustMe

    I agree 100%. Great article and citation of facts. Very often statistical methods are used in areas where they should never be used. For instance cancer studies. Very often we find the so called research claiming say consuming a substance X will make someone 3 times more likely to have cancer. But what it actually says is hey I studied 100 people and 10 of them consumed substance X and 3 of them had cancer (30%). However out of the other 90 only 9 had cancer (10%). Thus people consuming substance X are 3 times more likely to have cancer.
    So as you can see you need zero expertise in cancer or its causation to come up with such papers. All you need to know is some basic methods of experimentation and ability to project the statistics you want to be projected. We are making the mistake of thinking that applied statistics is science and correlation is causation in many of these instances.

More from K

Similar Posts

By Simran Pavecha

By Himanshu Pandey

By Wasiq Agha

Wondering what to write about?

Here are some topics to get you started

Share your details to download the report.









We promise not to spam or send irrelevant information.

Share your details to download the report.









We promise not to spam or send irrelevant information.

An ambassador and trained facilitator under Eco Femme (a social enterprise working towards menstrual health in south India), Sanjina is also an active member of the MHM Collective- India and Menstrual Health Alliance- India. She has conducted Menstrual Health sessions in multiple government schools adopted by Rotary District 3240 as part of their WinS project in rural Bengal. She has also delivered training of trainers on SRHR, gender, sexuality and Menstruation for Tomorrow’s Foundation, Vikramshila Education Resource Society, Nirdhan trust and Micro Finance, Tollygunj Women In Need, Paint It Red in Kolkata.

Now as an MH Fellow with YKA, she’s expanding her impressive scope of work further by launching a campaign to facilitate the process of ensuring better menstrual health and SRH services for women residing in correctional homes in West Bengal. The campaign will entail an independent study to take stalk of the present conditions of MHM in correctional homes across the state and use its findings to build public support and political will to take the necessary action.

Saurabh has been associated with YKA as a user and has consistently been writing on the issue MHM and its intersectionality with other issues in the society. Now as an MHM Fellow with YKA, he’s launched the Right to Period campaign, which aims to ensure proper execution of MHM guidelines in Delhi’s schools.

The long-term aim of the campaign is to develop an open culture where menstruation is not treated as a taboo. The campaign also seeks to hold the schools accountable for their responsibilities as an important component in the implementation of MHM policies by making adequate sanitation infrastructure and knowledge of MHM available in school premises.

Read more about his campaign.

Harshita is a psychologist and works to support people with mental health issues, particularly adolescents who are survivors of violence. Associated with the Azadi Foundation in UP, Harshita became an MHM Fellow with YKA, with the aim of promoting better menstrual health.

Her campaign #MeriMarzi aims to promote menstrual health and wellness, hygiene and facilities for female sex workers in UP. She says, “Knowledge about natural body processes is a very basic human right. And for individuals whose occupation is providing sexual services, it becomes even more important.”

Meri Marzi aims to ensure sensitised, non-discriminatory health workers for the needs of female sex workers in the Suraksha Clinics under the UPSACS (Uttar Pradesh State AIDS Control Society) program by creating more dialogues and garnering public support for the cause of sex workers’ menstrual rights. The campaign will also ensure interventions with sex workers to clear misconceptions around overall hygiene management to ensure that results flow both ways.

Read more about her campaign.

MH Fellow Sabna comes with significant experience working with a range of development issues. A co-founder of Project Sakhi Saheli, which aims to combat period poverty and break menstrual taboos, Sabna has, in the past, worked on the issue of menstruation in urban slums of Delhi with women and adolescent girls. She and her team also released MenstraBook, with menstrastories and organised Menstra Tlk in the Delhi School of Social Work to create more conversations on menstruation.

With YKA MHM Fellow Vineet, Sabna launched Menstratalk, a campaign that aims to put an end to period poverty and smash menstrual taboos in society. As a start, the campaign aims to begin conversations on menstrual health with five hundred adolescents and youth in Delhi through offline platforms, and through this community mobilise support to create Period Friendly Institutions out of educational institutes in the city.

Read more about her campaign. 

A student from Delhi School of Social work, Vineet is a part of Project Sakhi Saheli, an initiative by the students of Delhi school of Social Work to create awareness on Menstrual Health and combat Period Poverty. Along with MHM Action Fellow Sabna, Vineet launched Menstratalk, a campaign that aims to put an end to period poverty and smash menstrual taboos in society.

As a start, the campaign aims to begin conversations on menstrual health with five hundred adolescents and youth in Delhi through offline platforms, and through this community mobilise support to create Period Friendly Institutions out of educational institutes in the city.

Find out more about the campaign here.

A native of Bhagalpur district – Bihar, Shalini Jha believes in equal rights for all genders and wants to work for a gender-equal and just society. In the past she’s had a year-long association as a community leader with Haiyya: Organise for Action’s Health Over Stigma campaign. She’s pursuing a Master’s in Literature with Ambedkar University, Delhi and as an MHM Fellow with YKA, recently launched ‘Project अल्हड़ (Alharh)’.

She says, “Bihar is ranked the lowest in India’s SDG Index 2019 for India. Hygienic and comfortable menstruation is a basic human right and sustainable development cannot be ensured if menstruators are deprived of their basic rights.” Project अल्हड़ (Alharh) aims to create a robust sensitised community in Bhagalpur to collectively spread awareness, break the taboo, debunk myths and initiate fearless conversations around menstruation. The campaign aims to reach at least 6000 adolescent girls from government and private schools in Baghalpur district in 2020.

Read more about the campaign here.

A psychologist and co-founder of a mental health NGO called Customize Cognition, Ritika forayed into the space of menstrual health and hygiene, sexual and reproductive healthcare and rights and gender equality as an MHM Fellow with YKA. She says, “The experience of working on MHM/SRHR and gender equality has been an enriching and eye-opening experience. I have learned what’s beneath the surface of the issue, be it awareness, lack of resources or disregard for trans men, who also menstruate.”

The Transmen-ses campaign aims to tackle the issue of silence and disregard for trans men’s menstruation needs, by mobilising gender sensitive health professionals and gender neutral restrooms in Lucknow.

Read more about the campaign here.

A Computer Science engineer by education, Nitisha started her career in the corporate sector, before realising she wanted to work in the development and social justice space. Since then, she has worked with Teach For India and Care India and is from the founding batch of Indian School of Development Management (ISDM), a one of its kind organisation creating leaders for the development sector through its experiential learning post graduate program.

As a Youth Ki Awaaz Menstrual Health Fellow, Nitisha has started Let’s Talk Period, a campaign to mobilise young people to switch to sustainable period products. She says, “80 lakh women in Delhi use non-biodegradable sanitary products, generate 3000 tonnes of menstrual waste, that takes 500-800 years to decompose; which in turn contributes to the health issues of all menstruators, increased burden of waste management on the city and harmful living environment for all citizens.

Let’s Talk Period aims to change this by

Find out more about her campaign here.

Share your details to download the report.









We promise not to spam or send irrelevant information.

A former Assistant Secretary with the Ministry of Women and Child Development in West Bengal for three months, Lakshmi Bhavya has been championing the cause of menstrual hygiene in her district. By associating herself with the Lalana Campaign, a holistic menstrual hygiene awareness campaign which is conducted by the Anahat NGO, Lakshmi has been slowly breaking taboos when it comes to periods and menstrual hygiene.

A Gender Rights Activist working with the tribal and marginalized communities in india, Srilekha is a PhD scholar working on understanding body and sexuality among tribal girls, to fill the gaps in research around indigenous women and their stories. Srilekha has worked extensively at the grassroots level with community based organisations, through several advocacy initiatives around Gender, Mental Health, Menstrual Hygiene and Sexual and Reproductive Health Rights (SRHR) for the indigenous in Jharkhand, over the last 6 years.

Srilekha has also contributed to sustainable livelihood projects and legal aid programs for survivors of sex trafficking. She has been conducting research based programs on maternal health, mental health, gender based violence, sex and sexuality. Her interest lies in conducting workshops for young people on life skills, feminism, gender and sexuality, trauma, resilience and interpersonal relationships.

A Guwahati-based college student pursuing her Masters in Tata Institute of Social Sciences, Bidisha started the #BleedwithDignity campaign on the technology platform Change.org, demanding that the Government of Assam install
biodegradable sanitary pad vending machines in all government schools across the state. Her petition on Change.org has already gathered support from over 90000 people and continues to grow.

Bidisha was selected in Change.org’s flagship program ‘She Creates Change’ having run successful online advocacy
campaigns, which were widely recognised. Through the #BleedwithDignity campaign; she organised and celebrated World Menstrual Hygiene Day, 2019 in Guwahati, Assam by hosting a wall mural by collaborating with local organisations. The initiative was widely covered by national and local media, and the mural was later inaugurated by the event’s chief guest Commissioner of Guwahati Municipal Corporation (GMC) Debeswar Malakar, IAS.

Sign up for the Youth Ki Awaaz Prime Ministerial Brief below