Scientific articles often appear definitive. They contain statistics, technical language, graphs, expert authors, and peer review. Yet publication does not guarantee that a finding is correct, universal, or independently repeatable.
Large replication projects in psychology and cancer biology have shown that many influential results become weaker, change substantially, or fail to meet the original statistical criteria when researchers repeat the experiments. However, the popular claim that “half of all scientific papers are wrong” is too broad.
There is no reliable evidence that exactly half of every scientific article in every discipline is false. The deeper problem is that a substantial share of published findings may be less certain, less generalisable, or less reproducible than readers assume.
What Is the Reproducibility Crisis?
The reproducibility crisis refers to widespread concern that researchers cannot reliably recreate many published scientific findings.
The National Academies distinguishes two related concepts:
- Reproducibility means obtaining the same computational result using the original data, code, methods, and analytical conditions.
- Replicability means obtaining compatible results in a new study that collects independent data to answer the same scientific question.
A paper may be computationally reproducible but not experimentally replicable. For example, another analyst may successfully reproduce every published calculation while a new experiment fails to find the reported effect.
Replication tests whether a result survives beyond its original dataset, laboratory, research team, and circumstances.
Where the “Half May Be Wrong” Claim Comes From
The alarming headline is partly based on major replication projects and surveys.
In 2015, the Open Science Collaboration repeated 100 experiments published in prominent psychology journals. While 97% of the original studies reported statistically significant results, only 36% of the replications did so. Replication effects were also approximately half the size of the original effects on average.
That does not prove that 64% of the original papers were completely false. Replications can differ in statistical power, participants, conditions, interpretation, and measurement. Different criteria also produce different estimates of replication success.
A 2016 Nature survey of more than 1,500 researchers found that over 70% had tried and failed to reproduce another scientist’s experiment, while more than half reported difficulty reproducing one of their own results. This was a survey of researchers’ experiences, not a representative audit proving that half of published science is wrong.
Cancer Biology Revealed Similar Difficulties
The Reproducibility Project: Cancer Biology attempted to repeat selected experiments from prominent preclinical cancer studies.
The project initially planned to examine 193 experiments from 53 papers, but practical problems—including missing methodological information and unavailable materials—meant that only 50 experiments from 23 papers were completed.
The replication effects were substantially smaller than the original findings. According to the Center for Open Science, only 46% of examined effects succeeded on more replication criteria than they failed, while positive original results were less likely to replicate successfully than original null findings.
This illustrates another part of the crisis: sometimes researchers cannot repeat a study because the published article does not contain enough operational detail.
Small Studies Produce Unstable Answers
Small participant groups and limited numbers of experiments create noisy estimates.
Imagine testing a treatment on 20 people. By chance, one group may contain more healthy participants, more severe cases, or more people likely to respond. The estimated treatment effect may therefore appear much larger than its true average effect.
Small studies also produce wide uncertainty intervals. When a surprisingly strong result is published, later research with larger samples often finds a smaller effect.
This does not necessarily mean that the original scientists acted improperly. Random variation can generate apparently impressive findings even when researchers follow their protocol honestly.
Publication Bias Distorts the Scientific Record
Journals have traditionally preferred novel, positive, statistically significant findings.
A study reporting that a new treatment works is more exciting than one reporting no meaningful difference. Researchers may also be less motivated to submit negative results, while editors and reviewers may consider them insufficiently interesting.
This creates the file-drawer problem: unsuccessful experiments remain unpublished, while the few positive results become visible.
Suppose 20 teams test an ineffective supplement. Nineteen find nothing, but one obtains a positive result by chance. If only that study is published, the literature gives a misleading impression that the supplement works.
The scientific record can become biased even when every published calculation is technically correct.
Flexible Analysis Can Manufacture Significance
Researchers frequently make legitimate analytical choices: which participants to exclude, which outcomes to prioritise, when to stop collecting data, and which statistical model to use.
Problems arise when many possible analyses are tried but only the successful one is reported. This practice is sometimes called p-hacking.
Related behaviour includes HARKing—forming a hypothesis after seeing the results but presenting it as though it had been predicted from the beginning.
These practices may occur consciously or unconsciously. Human beings are skilled at building convincing explanations after patterns have already appeared.
Preregistration helps by recording the hypothesis, primary outcome, sample size, and analysis plan before results are known.
Peer Review Is Not an Independent Replication
Peer review is valuable, but its purpose is often misunderstood.
Reviewers assess whether the research question is relevant, the methods appear reasonable, and the conclusions follow from the reported evidence. They normally do not repeat the experiment, inspect every raw data point, or independently rerun every analysis.
They may also lack access to complete datasets, laboratory materials, software, or detailed protocols.
Peer review is a quality filter, not a guarantee that a result is true.
Fraud can produce irreproducible research, but many replication problems result from ordinary statistical uncertainty, weak documentation, design limitations, or genuine differences between populations and environments.
A Failed Replication Does Not Automatically Disprove a Study
Scientific replication is not as simple as copying a recipe.
Participants may differ by country, age, culture, health status, or historical period. Laboratory equipment, animal strains, temperatures, reagents, and researcher expertise may also affect results.
A real phenomenon may operate only under specific conditions. Discovering those boundaries can improve scientific understanding.
The National Academies therefore warns that non-replication has several possible explanations and may sometimes reveal important variation rather than straightforward error.
The proper response is to compare protocols, inspect uncertainty, conduct further studies, and evaluate the entire body of evidence.
How Open Science Can Improve Reliability
Reform efforts increasingly promote:
- Preregistered hypotheses and analysis plans
- Larger and better-powered studies
- Open data and computer code
- Detailed experimental protocols
- Registered reports reviewed before results exist
- Publication of negative findings
- Independent replication
- Multi-laboratory collaborations
- Clear reporting of uncertainty
- Corrections and retractions when necessary
Registered reports are particularly useful because journals evaluate the importance of the question and quality of the method before researchers know the outcome. Publication therefore depends less on whether the result is positive.
Sharing materials also allows other researchers to identify coding errors, reproduce analyses, and test alternative explanations.
Expert Perspective
The National Academies concluded that reproducibility and replicability are essential to scientific progress, while rejecting simplistic claims that every unsuccessful replication proves misconduct or invalidity. It recommends stronger transparency, better statistical practice, access to data and code, and incentives for rigorous replication.
Large replication initiatives communicate a similar lesson. The Open Science Collaboration did not demonstrate that psychology as a whole was useless. It showed that confidence should depend on repeated evidence rather than the prestige of one journal or the excitement of one result.
Science becomes more trustworthy when it exposes its weaknesses, corrects mistakes, and rewards verification rather than novelty alone.
How Readers Should Evaluate Scientific Headlines
Do not ask only whether a study was published. Ask:
- Was the sample large enough?
- Was the study preregistered?
- Was there an appropriate control group?
- Were the data and methods available?
- Has the result been independently replicated?
- Is the effect practically important or merely statistically significant?
- Do systematic reviews support it?
- Are there conflicts of interest?
- Does the headline exaggerate the actual conclusion?
A single paper should usually be treated as one contribution to an evolving body of evidence.
Scientific knowledge is strongest when many independent methods and studies converge on the same answer.
Interesting Facts
- In the major psychology replication project, replicated effects were about half the magnitude of the original effects on average.
- A replication can fail even when the original and new researchers both work carefully.
- Statistical significance does not reveal whether an effect is large, useful, or clinically important.
- Researchers may be unable to reproduce an analysis when the original data or code is unavailable.
- Negative results can be scientifically valuable because they help prevent ineffective ideas from appearing more convincing than they are.
- Multi-laboratory studies test whether a finding survives across different researchers, locations, and participant groups.
- Reproducibility uses the original data, whereas replicability generally involves collecting new data.
- Scientific self-correction may be slow, but replication projects, corrections, and retractions demonstrate that correction mechanisms exist.
Glossary
- Reproducibility — Obtaining the same analytical result using the original data, code, and methods.
- Replicability — Obtaining compatible findings in a new study using independently collected data.
- Replication Crisis — Concern that many published scientific findings cannot be reliably repeated.
- Publication Bias — Preferential publication of positive or statistically significant findings.
- File-Drawer Problem — The disappearance of negative or inconclusive studies from the visible literature.
- P-Hacking — Trying multiple analyses or decisions until a statistically significant result appears.
- Preregistration — Publicly recording hypotheses and methods before analysing the results.
- Statistical Power — The probability that a study will detect a real effect of a specified size.
- Registered Report — A research article provisionally accepted based on its question and methods before results are known.
- Peer Review — Evaluation of research by other specialists before or after publication.
- Effect Size — A measurement of the magnitude of a difference or relationship.
- Systematic Review — A structured assessment of all relevant studies addressing a defined question.
