Last updated: 2026-10-06

U
Undergraduate level

Designing Experiments and Writing Critical Analysis

An experiment is a comparison with the conditions controlled. Many student projects say they run experiments when they have in fact run a single configuration, looked at the numbers, and described them. That can be a useful piece of work, but it cannot support a claim that one approach is better than another. A properly designed comparison can.

This page covers the design principles that make a comparison trustworthy, and the critical analysis that turns the results into a judgement. The collection and validation of the data itself is covered in Collecting Evidence, Evaluating It, and Validating the Result.next step: gathering and checking the data

What a Comparison Needs

Three principles from the design of experiments apply to most student comparisons. Montgomery's standard text sets them out in detail[1], and the summary here is for a project, not a laboratory.

  • A control. There must be a condition to compare against: the existing system, a simple baseline, or no intervention. Without one, a good result tells you nothing about the change.
  • Randomisation or fair allocation. Conditions should be assigned so that differences between them are not driven by something else. In a study with participants, random allocation is the usual way. In a software comparison, it means running the conditions in a varied or shuffled order, so that caching, warm-up, or time of day does not favour one of them.
  • Replication. A single run is an anecdote. Repeated runs show how much the result varies, which is what you need to judge whether a difference is larger than the noise.
Don't do this. Compare your new method, run once, against a baseline run last week on a different machine. Any difference you find may come from the machine, the week, or the order, and none of them is the method.

Choosing What to Vary

An experiment varies one thing at a time, or varies several things deliberately, in a planned pattern. The simplest design changes one factor and holds the rest fixed. It is easy to explain and often enough for a project. When several factors may interact, a planned design that varies them together can show the interactions, but it needs more runs and more care in the analysis. Start with the simpler design unless you have a clear reason to need the other.

Write down what is held constant as well as what varies. The list of constants is part of the method, and a reader needs it to judge whether the comparison is fair.constants define the baseline

Planning the Analysis

Choose the statistical analysis before running the experiment. For a comparison of two conditions, this usually means a measure of the difference, an estimate of its uncertainty, and a statement of what size of difference would matter. The supporting ideas are in Probability and Statistics for Computing. A confidence interval reported alongside a mean is far more informative than the mean alone.pre-registering analysis plan

Plan also for what you will do with unexpected results. A result that contradicts your expectation is still a result, and the plan should say how it will be investigated: whether the experiment was run correctly, whether the baseline was fair, and whether the measure captures the thing you care about.

Critical Analysis

Analysis describes and interprets results. Critical analysis goes further, by evaluating how far the evidence supports the conclusion and what would be needed to support a stronger one. The University of Reading's guidance on styles of writing describes critical writing as requiring you to view a topic from a variety of angles and to evaluate the evidence, and it notes that most academic work needs descriptive, analytical, and reflective elements together, with descriptive writing kept to a minimum[2].

In practice, a critical analysis section answers four questions:

  1. What does the evidence show? State the result plainly, with its uncertainty.
  2. Why might it be so? Offer the most plausible explanation, and say what evidence supports it.
  3. What else could explain it? Name at least one alternative, and say whether the data can rule it out.
  4. What would change the conclusion? State the result that would have led you to a different view, and whether a further study could obtain it.

The fourth question is the one most often missing. A conclusion that could not have been changed by any result is an assertion, however well it is written.

Limitations, Stated Usefully

Every project has limitations, and a limitations section that lists them without consequence is a weak one. Each limitation should be linked to a claim: the sample is small, so the result applies to this group; the machine is one configuration, so the timing does not generalise. A limitation that changes no claim is probably not a limitation worth stating, and one that changes a central claim should be dealt with in the conclusion.

References

  1. Montgomery, D. C. Design and Analysis of Experiments. John Wiley & Sons.
  2. University of Reading. (n.d.). Descriptive, analytical and reflective writing. Academic writing guides, LibGuides. https://libguides.reading.ac.uk/writing/stylesofwriting