Coaching mock-test scores: measure real improvement, not luck

The academic head's question

An entrance-exam coaching institute runs fortnightly mock tests. After Mock 1, students under a cutoff are put into extra doubt-clearing sessions. After Mock 2, the academic head sees that this group's average rose sharply while the batch average fell, and wants to expand the sessions to every centre. It is an expensive decision: faculty hours, rooms and weekend slots. The analyst's job is to say how much of that rise the sessions caused.

Data you will get

Three numbers to compute

• Raw gain for a group = mean(Mock 2 score − Mock 1 score)
• Estimated effect = raw gain of the remedial group − raw gain of an eligible comparison group
• Standardised score z = (score − that mock's mean) ÷ that mock's standard deviation, so gains are relative to the batch on each test

Use an uncertainty estimate for the effect. The simplest is the standard error of a difference in means: √(var₁/n₁ + var₂/n₂).

A simulation where we know the answer

The best way to trust a method is to run it on data where the true effect is known. Here every synthetic student has a fixed ability, each mock adds random noise, Mock 2 is five marks harder, and the remedial class truly adds 3 marks. Because sessions had room for only half the eligible students, the eligible list was shuffled and alternate students assigned; the rest were scheduled for the next cycle.

import random
from statistics import mean, pstdev, variance

random.seed(27)

# Synthetic coaching batch: 2,000 students, two mock tests out of 100 (hypothetical)
N = 2000
CUTOFF = 40            # students below this on mock 1 are eligible for remedial classes
TRUE_EFFECT = 3.0      # the remedial effect built into the simulation, in marks
MOCK2_SHIFT = -5.0     # mock 2 is harder than mock 1

students = []
for i in range(N):
    ability = random.gauss(55, 10)
    m1 = ability + random.gauss(0, 8)
    students.append({"id": i, "ability": ability, "m1": m1})

eligible = [s for s in students if s["m1"] < CUTOFF]
random.shuffle(eligible)
for k, s in enumerate(eligible):
    s["remedial"] = k % 2 == 0          # capacity for half: alternate allocation after shuffle
for s in students:
    s.setdefault("remedial", False)
    s["m2"] = (s["ability"] + MOCK2_SHIFT + random.gauss(0, 8)
               + (TRUE_EFFECT if s["remedial"] else 0))

def z(xs):
    mu, sd = mean(xs), pstdev(xs)
    return [(x - mu) / sd for x in xs]

for s, a, b in zip(students, z([s["m1"] for s in students]), z([s["m2"] for s in students])):
    s["z1"], s["z2"] = a, b

def gain(group, key1="m1", key2="m2"):
    return round(mean(s[key2] - s[key1] for s in group), 2)

rem = [s for s in students if s["remedial"]]
ctl = [s for s in eligible if not s["remedial"]]
rest = [s for s in students if s["m1"] >= CUTOFF]

print("eligible:", len(eligible), "remedial:", len(rem), "eligible, not yet remedial:", len(ctl))
print("raw gain, remedial:", gain(rem))
print("raw gain, eligible control:", gain(ctl))
print("raw gain, everyone else:", gain(rest))
print("estimated effect (remedial - control):", round(gain(rem) - gain(ctl), 2))
g_r = [s["m2"] - s["m1"] for s in rem]
g_c = [s["m2"] - s["m1"] for s in ctl]
se = (variance(g_r) / len(g_r) + variance(g_c) / len(g_c)) ** 0.5
print("standard error of the difference:", round(se, 2))
print("z gain, remedial:", gain(rem, "z1", "z2"), "| control:", gain(ctl, "z1", "z2"))

Output:

eligible: 255 remedial: 128 eligible, not yet remedial: 127
raw gain, remedial: 6.77
raw gain, eligible control: 3.38
raw gain, everyone else: -5.74
estimated effect (remedial - control): 3.39
standard error of the difference: 1.29
z gain, remedial: 0.85 | control: 0.58

Reading it line by line

The remedial group's 6.77-mark rise, on a test that was five marks harder, is what the academic head saw. Yet the 127 eligible students who had no extra sessions also rose, by 3.38 marks. Nothing was done for them. They were selected because Mock 1 went badly, and a bad day does not repeat on schedule. The rest of the batch fell by 5.74, slightly more than the difficulty shift, for the mirror-image reason: some of them scored above the cutoff on a lucky day.

The difference between the two eligible groups, 3.39 marks, is close to the 3 marks built into the simulation. The standard error of 1.29 says the true value could plausibly be anywhere from roughly 1 to 6 marks. That is the honest summary: the sessions very likely help, by a few marks, and a raw before-and-after would have doubled the claim.

Standardised scores make the same point without needing to know how much harder Mock 2 was: relative to the batch, the remedial group moved up 0.85 standard deviations and the comparison group 0.58.

Traps in coaching data

• Selecting on the first test. Any group chosen for a low or high score will drift back. A comparison group chosen by the same rule is the only reliable fix.
• Comparing with students just above the cutoff. They were selected differently and are not a like-for-like control, although a careful analysis around the cutoff can still be informative.
• Changing difficulty and syllabus. A harder paper lowers everyone. Compare within-test percentiles or z-scores, not raw marks, across mocks.
• Absentees and dropouts. If weaker remedial students stop taking mocks, the group's average rises without anyone improving. Report how many of each group sat both tests.
• Attendance versus assignment. Compare groups as assigned. Restricting to students who attended most sessions picks the motivated ones and exaggerates the effect.
• Too few students. One centre's 20 students cannot distinguish a 3-mark effect from noise. Pool across centres or cycles before concluding.

Showing it to the academic head

Use a single chart with three bars: remedial gain, comparison gain and the difference, with an error bar on the difference. Then one sentence: "The sessions add about 3 marks per student on the next mock, with a plausible range of 1 to 6; around half of the improvement we first saw would have happened anyway." Follow it with the recommendation that follows from the evidence, for example keeping a waitlisted comparison group in each expansion wave so the estimate keeps improving.

Do not rank faculty on raw gains of their remedial batches. Their students were chosen for bad luck, so every remedial teacher looks good.

Where to practise this

Simulating data with a known answer, and then checking that your method recovers it, is a habit the Data Science programme builds before any experiment design. The same selection logic hides in cold-chain excursion analysis whenever only flagged shipments get inspected. The free skill check will show how far your current profile covers this kind of work.