
Bilal was in the second year of his master's degree in Faisalabad when his supervisor handed him a paper to repeat. The paper, published in a good journal, claimed that a 10-minute writing exercise before an exam improved students' scores by 15 percent. Bilal ran the same exercise with 60 students in his own department. The improvement was 2 percent — small enough to be chance. His first thought was that he had done something wrong. His second thought, the right one, was that he had just done a replication study, and that the difference between 15 percent and 2 percent was exactly the kind of finding science needs to hear about.
A replication study repeats an earlier study's method to check whether its finding holds up. The key word is "repeat": you follow the original procedure as closely as you can — the same task, the same measures, the same analysis — and you see whether the same result appears. If it does, confidence in the finding grows. If it does not, science has learned something important, and the original finding has to be re-examined.
A replication is not a new study that happens to look at a similar topic, and it is not a second analysis of the same data. It needs new data, collected with the original method. It is also not a criticism of the original authors. Bilal's study counted as a replication because he used the same writing exercise, the same 10-minute timing, the same kind of exam, and the same analysis the original paper described. He changed only what he had to change: the students were Pakistani undergraduates rather than American ones, and he said so openly in his write-up.
Replication matters because a single study, however well done, can mislead. The clearest demonstration came from the Reproducibility Project, published in Science in 2015. A team of 270 researchers repeated 100 published psychology experiments as faithfully as they could. In the original studies, 97 out of 100 results had been statistically significant. In the replications, only 36 out of 100 reached significance, and the average effect size was about half the size of the original.
The same lesson appears in famous single cases. The "power pose" effect — the claim from a 2010 paper that standing in an expansive posture for two minutes raised confidence and changed hormone levels — was repeated by Ranehill and colleagues in 2015 with a larger sample and preregistered methods. The hormone effect did not appear, and the confidence effect was much smaller than originally reported. For a student, these numbers are good news: a replication is a publishable contribution precisely because so few findings have been checked.
Not every study deserves a replication. Pick one that passes four tests. First, the finding matters: it is cited often, it shapes teaching or practice, or it surprises people. Second, the finding is uncertain: the original sample was small, the effect was surprisingly large, or later work has questioned it. Third, the study can actually be repeated: the materials exist or can be rebuilt, the procedure is described clearly enough to follow, and you can recruit a similar sample. Fourth, you can do it well with the resources you have.
Bilal's choice passed all four. The writing-exercise paper had been cited more than 300 times and was used in study-skills workshops. Its original sample was 45 students — small enough that a 15 percent improvement could easily be a chance result. The materials were one page of instructions, which the authors shared by email within a week. And Bilal could recruit 60 students from his own department's first-year classes in a single term. Three warning signs should make you choose a different study: the procedure is not described in enough detail and the authors do not answer your email; the effect depends on equipment or a population you cannot access; or the finding has already been replicated several times with consistent results.
Before you collect any data, read the original paper the way a detective reads a case file: looking for every detail you will need to copy. Make a table with one row per detail: the exact instructions given to participants, the timing of each step, the materials used, the order of tasks, the exclusion rules, the sample size per group, the exact measures and their scoring, the statistical test, and the effect size with its confidence interval.
Bilal's table caught two details that would have changed his result. First, the original study had excluded 6 of its 45 students — 13 percent — because they did not follow the writing instructions, but the paper mentioned this only in a footnote. Bilal applied the same rule and excluded 8 of his 60, and he reported both numbers openly. Second, the original exam was a 20-question multiple-choice test given one week after the exercise; Bilal's department used a different exam format, so he built a 20-question test in the same style and noted the difference in his preregistration. If a detail is missing, write to the original authors before you start — most are glad to help, and keeping the exchange gives you evidence for reviewers.
Preregistration means writing down your plan — hypothesis, sample, materials, procedure, and analysis — in a public, time-stamped record before you collect data. The usual place is the Open Science Framework (osf.io), which is free. A preregistration protects you twice: it stops you from quietly changing the analysis after you see the results, and it stops reviewers from suspecting that you did.
Bilal's preregistration was one page, and it took him an afternoon. It said, in order: the hypothesis ("the writing exercise will improve exam scores by at least 10 percent, as in the original study"); the sample ("60 first-year undergraduates, 30 per group"); the materials ("the original one-page instructions, shared by the authors"); the procedure ("10-minute exercise one week before the exam, 20-question test, same exclusion rule as the original"); and the analysis ("independent-samples t-test, alpha 0.05, effect size with a 95 percent confidence interval"). Two rules make preregistration work: register before you collect any data, and if you must change the plan, change it openly in the final paper. For the methods section itself, our guide to how to write the methodology section of a research paper shows how to describe the procedure so a reader can repeat it.
The commonest mistake in replications is using the same sample size as the original study. Original studies are often underpowered, and effects in published papers tend to be overestimated, because only the large, lucky results get published. A replication with the same small sample has a poor chance of finding the effect even if it is real.
A practical rule, proposed by Uri Simonsohn, is to use about 2.5 times the original sample size: if the true effect is only as large as the original study could reliably detect, a replication needs roughly 2.5 times the participants to have 80 percent power to find it. Bilal applied the rule directly. The original study had 45 students, so 2.5 times that is about 113 — more than he could recruit in one term. He recruited 60, noted openly in his preregistration that this gave him power to detect only a large effect, and reported the limitation honestly. His result — a 2 percent improvement — was still informative, because with 60 students a 15 percent effect would very likely have shown up. Free tools such as G*Power will calculate the sample you need — run the calculation before you recruit, not after.
Faithful running means following the preregistered procedure and writing down every deviation, however small. Keep a running log from the first session: date, group, number of participants, anything unusual. Bilal's log recorded that on 12 March, three students arrived 15 minutes late and did the exercise in 8 minutes instead of 10; that on 19 March, the projector failed and he read the instructions aloud; and that two students asked whether the exercise was "a trick" and were told it was a study-skills task, exactly as the original script said.
Deviations are normal. What matters is that they are recorded and reported, not hidden. A reviewer who finds an unreported deviation loses trust in the whole study. A reviewer who finds a logged deviation with a sensible explanation trusts the author more, not less. Two practical habits help: use the same script every time, reading the instructions from a printed sheet rather than improvising, and blind yourself where you can — if the person scoring the exam does not know which students did the exercise, the scoring cannot be biased by expectation.
A different result is a finding, not a failure — but it needs careful handling. Bilal's replication found a 2 percent improvement where the original reported 15 percent. His effect size was d = 0.08, with a 95 percent confidence interval from −0.25 to 0.41. That interval includes zero and excludes the original effect size of d = 0.62, which is the key point: his study was large enough that the original effect, if real, should have appeared, and it did not.
Report the comparison directly, with numbers. State the original effect size and your effect size side by side, with confidence intervals for both. Say what your study could and could not detect. Then discuss the honest explanations in order of likelihood: the original effect may have been overestimated by chance; the population may differ; a small procedural difference may matter. Do not claim you have proved the original wrong — one replication rarely proves that. One more step matters: tell the original authors. Bilal emailed them his result before submission, with the draft attached. They replied with two useful points, both of which went into his discussion section.
Five mistakes damage replication studies more than any others. First, changing the procedure without saying so — a different task, a different timing, a different population — and still calling the study a direct replication. Every change must be declared. Second, collecting data before preregistering, which removes the protection preregistration is meant to give. Third, using the same small sample as the original and then claiming the result is decisive either way.
Fourth, analysing the data until something significant appears — trying different exclusions, different tests, different subgroups — and reporting only the version that worked. The preregistration is the defence against this: the analysis you registered is the analysis you report first, and any extra analysis is labelled as exploratory. Fifth, overstating the conclusion. One replication with 60 students does not prove a finding false, just as one original study with 45 students did not prove it true. Report the numbers, state the limits, and let the evidence accumulate.
| Type | What is repeated | What it tests | Main strength |
|---|---|---|---|
| Direct replication | Same methods, new sample | Whether the result reproduces | Strongest test of reproducibility |
| Conceptual replication | Same idea, different methods | Whether the idea generalises | Tests the theory, not the procedure |
| Systematic replication | Same study, one feature changed | Whether the result holds in a new setting | Shows the limits of the finding |
| Multi-site replication | Same protocol across sites | Whether the result holds across teams | Larger sample, less single-team bias |
These authoritative external resources explain the standards and tools mentioned in this guide:
What is a replication study?
A replication study is an independent attempt to repeat an earlier study's methods with fresh data, to see whether the same result appears again. It checks a finding rather than reporting a new discovery, and it is judged on fidelity to the original design and transparency of reporting.
Do journals publish replication studies?
Yes, and more journals welcome them than in the past. Some publish replications in regular issues, some have dedicated replication article types, and some use registered reports, where the plan is reviewed before data collection and publication is offered regardless of the result.
Should I preregister my replication study?
Yes. A preregistration records your hypothesis, sample, materials, procedure, and analysis before data collection, on a platform such as the Open Science Framework. It protects you against suspicion that the study was adjusted after the results were known. Register before you collect any data, and note any change to the plan openly in the final paper.
What if my replication gives a different result?
Report it fully and honestly. A differing result is a finding, not a failure, and journals that publish replications expect it. Discuss plausible explanations, including chance and real differences in setting or sample, and do not reanalyse the data until it matches.
How large should my replication sample be?
Large enough to give the original finding a fair test. Do not simply copy the original sample size, since a small original study may have exaggerated the effect. Plan the sample with a realistic effect estimate and explain your reasoning in the preregistration.
Should I contact the original authors first?
It is good practice. Ask for materials and clarification of any details the paper leaves unclear. Most authors are willing to help, and documenting the exchange shows reviewers that you worked to repeat the study faithfully. Keep copies of the messages, because reviewers may ask what the original authors confirmed.
Submit your manuscript ← Back to GAP Insights
Useful publishers, databases, funders, tools and institutions for researchers: