Choosing a statistical test feels harder than it should. Most guides start with a long list of test names, and the reader is left guessing which one fits their study. The choice becomes much simpler when it is made in the right order: first the research question, then the type of data, then the study design, and only then the test name.
This guide follows that order. It explains the common tests in plain language, shows what each one is for, and covers the checks and reporting habits that reviewers expect. It builds on our guides to writing a methodology section and writing results and discussion sections, and it pairs naturally with our guide to organising and storing research data.
Every statistical test answers a particular kind of question, so the question must come first. Are you comparing groups, looking for a relationship, or predicting an outcome? A test chosen before the question is clear will often answer a question nobody asked, and reviewers notice that quickly.
Write the question in one sentence before you open any statistics software. For example: does a new feeding supplement change average daily weight gain in calves, compared with the standard ration? That sentence already tells you that you are comparing groups on a measured outcome, which narrows the choice to a small family of tests. A vague question, such as whether diet and growth are connected, leaves every option open and usually means the study design is not yet settled.
It also helps to state what would count as a meaningful result, not only a statistically significant one. A tiny difference can be statistically significant with a very large sample, yet too small to matter in practice.
The second decision is the type of data you have. Numerical data measured on a scale, such as weight, temperature, or concentration, are handled differently from counts in categories, such as the number of animals that recovered, relapsed, or did not respond. Many wrong test choices come from treating categorical data as if it were numerical, or from averaging values that should only be counted.
Within numerical data, it matters whether the values are continuous, like body weight, or counts, like the number of parasites per sample. Count data often behave differently from measurements, especially when counts are small, and may need a different approach. Within categorical data, it matters whether categories are simply different, like breed or sex, or ordered, like a severity score from mild to severe. Ordered categories carry extra information that some tests can use and others ignore.
When the question is whether groups differ on a numerical outcome, the standard starting points are the t-test and analysis of variance, known as ANOVA. A t-test compares two groups, for example a treatment group and a control group. ANOVA compares three or more groups in a single analysis, which is preferable to running many separate t-tests, because repeated testing increases the chance of a false positive result.
Both approaches carry assumptions, most importantly that the data within each group are roughly normally distributed and that the groups have similar variability. These assumptions should be checked, not assumed, by looking at the distribution of the data and, where appropriate, using a formal test of normality. With small samples, plots of the data are often more informative than formal tests, which have little power to detect departures from normality when few observations are available.
When the assumptions clearly fail, non-parametric alternatives exist: the Mann-Whitney U test in place of a two-group t-test, and the Kruskal-Wallis test in place of ANOVA. These tests compare ranks rather than means, which makes them less sensitive to extreme values and skewed distributions. The trade-off is that they answer a slightly different question, about the distribution of values rather than the average, so the result should be described in those terms. Choosing a non-parametric test after looking at the data is acceptable when the choice follows the assumption checks, but switching tests repeatedly until a significant result appears is not good practice and should be avoided.
When the question is about a relationship rather than a group difference, the choice depends again on data type. For two numerical variables, correlation describes the strength and direction of their linear association. Pearson correlation is the usual choice when both variables are numerical and roughly normally distributed; Spearman rank correlation is the alternative when the relationship is monotonic but not linear, or when the data are ordered categories.
Regression goes further than correlation: it models how an outcome changes with one or more predictors, and it can adjust for other variables at the same time. Simple linear regression uses one predictor; multiple regression uses several. Regression output needs careful reading: the coefficient for a predictor describes its association with the outcome while holding the other predictors in the model constant, which is not the same as the simple association seen on its own. Confusing the two is one of the most common misreadings in published papers.
For two categorical variables, the chi-square test of independence asks whether the pattern of counts differs from what would be expected if the variables were unrelated. It needs adequate counts in each cell of the table; when counts are very small, an exact test is the safer choice. Whatever the test, a relationship found in observational data does not by itself prove that one variable causes the other, and the discussion should say so plainly rather than using causal language the design cannot support.
Study design changes the test even when the question looks the same. Measurements taken from the same subjects twice, such as before and after treatment, are paired: the two values from one subject are related, and the analysis should use that pairing. A paired t-test, or its non-parametric counterpart the Wilcoxon signed-rank test, compares the within-subject changes rather than treating the two sets of measurements as independent groups.
The same principle applies to repeated measurements over several time points, and to matched designs where each treated subject is paired with a similar control. Ignoring the pairing and analysing the data as if all observations were independent is a serious error: it discards information and can give a misleading result in either direction. Repeated-measures ANOVA and mixed-effects models are the usual approaches when there are more than two time points or a more complex structure, and they should be planned with the design, not added as an afterthought.
If the design includes pairing or matching, say so explicitly in the methodology section and name the test that respects it. Reviewers check this closely, because a mismatch between design and analysis undermines the whole result.
A test can only detect a difference if the study is large enough, which is why sample size should be planned before data collection. Power is the probability that the study will detect a real effect of a stated size; researchers commonly aim for high power, and a study with low power risks missing a real effect and producing an inconclusive result. Power depends on the size of the effect, the variability in the data, the sample size, and the significance level chosen.
An effect size expresses how large a difference or relationship is, in terms that do not depend on sample size. Reporting an effect size alongside the test result lets readers judge practical importance, which a p-value alone cannot do. A small p-value with a trivial effect size is not a strong finding, and a moderate effect size with a non-significant p-value in a small study is not proof of no effect; it may simply be an underpowered study. Both numbers, plus the uncertainty around them, belong in the results.
Where a formal sample-size calculation is possible, report its inputs: the effect size assumed, the variability expected, the power aimed for, and the significance level. Where it is not possible, for example in exploratory work or studies of rare conditions, say so and explain how the sample size was decided instead. Reviewers accept honest constraints; they do not accept a sample size that appears to have been chosen by convenience and never mentioned.
Every parametric test rests on assumptions, and the methodology section should show that they were checked. The usual checks are normality of the residuals or of the data within groups, similar variability across groups, independence of observations, and, for regression, linearity and the absence of extreme influence from single observations. Plots are the most useful first check: a histogram or quantile plot for distribution, a residual plot for regression, and a simple plot of the data by group will reveal problems that a single test statistic can hide.
When an assumption fails mildly, the test is often still reasonable, and many tests are robust to small departures, especially with larger samples. When it fails badly, the options are to transform the data, for example by working on a logarithmic scale where that makes scientific sense, or to switch to a non-parametric or robust method. Transformations change the scale of interpretation, so the results should be reported in a way the reader can understand, with back-transformed values where appropriate.
Whatever is done, the analysis should follow a plan decided in advance as far as possible, and any departure from that plan should be reported. Trying many tests and reporting only the one that gives a significant result is a form of selective reporting, and it invalidates the p-value. If the analysis plan changed, say what changed and why; transparency about analysis decisions is worth more than a tidy-looking result that cannot be trusted.
A result that cannot be checked cannot be trusted, so report enough detail for a reviewer to follow the analysis. For each test, name the test, state the test statistic with its degrees of freedom where applicable, give the exact p-value rather than only a threshold, and report the effect size with a confidence interval. The confidence interval matters because it shows the range of values compatible with the data; a wide interval means the estimate is imprecise, however the p-value looks.
Tables should carry the full detail and the text should draw attention to the key pattern, not repeat every number. Means should be accompanied by a measure of variability, and counts should be accompanied by the totals they come from, so that percentages can be checked.
Finally, keep the language precise. A significant result means the data are unlikely under the null hypothesis, not that the effect is large or important. A non-significant result means the data did not provide enough evidence against the null, not that the null is true. Writing results in these careful terms, and discussing limitations honestly, is what separates a credible paper from one that overclaims. Our guides to writing the limitations section and responding to reviewer comments show how to carry that honesty through revision.
| Research question | Data involved | Usual test | If assumptions fail |
|---|---|---|---|
| Compare two independent groups | Numerical outcome | Independent t-test | Mann-Whitney U test |
| Compare three or more groups | Numerical outcome | One-way ANOVA | Kruskal-Wallis test |
| Compare paired measurements | Numerical outcome, same subjects | Paired t-test | Wilcoxon signed-rank test |
| Relationship between two numerical variables | Two numerical variables | Pearson correlation or regression | Spearman rank correlation |
| Relationship between two categorical variables | Counts in categories | Chi-square test | Exact test for small counts |
| Predict an outcome from predictors | Outcome plus one or more predictors | Regression model | Robust or non-parametric model |
These authoritative external resources explain the standards and guidance mentioned in this article:
Do I need a statistician to choose a test?
Not always, but it helps. Standard designs with clear questions can be handled with this guide and a good textbook. If the design is complex, the sample is small, or the analysis will be challenged, consult a statistician before collecting data, not after.
What if my data are not normally distributed?
First check how badly the assumption fails and whether a plot shows a clear problem. Mild departures are often acceptable; severe skew or extreme values may call for a transformation or a non-parametric test. Report what you checked and what you did.
Can I use a t-test for three groups?
You can, but you should not run separate t-tests for every pair. Multiple testing increases the chance of a false positive. Use ANOVA first, then planned or adjusted pairwise comparisons if the overall test supports them.
What does a p-value actually tell me?
It tells you how unlikely the observed data, or more extreme data, would be if there were no effect. It does not tell you how large or important the effect is. Always report an effect size and a confidence interval alongside it.
Should I choose the test after seeing the results?
No. Choose the test from the question, data type and design, and check assumptions as part of the plan. Trying several tests and reporting only the significant one is selective reporting and invalidates the result.
What should I report for each test?
Name the test, give the test statistic with degrees of freedom where applicable, the exact p-value, the effect size and a confidence interval. State the sample size and any data excluded or transformed, so a reviewer can follow the analysis.