Likelihood versus probability in a fitted model

Probability fixes a model and asks about possible observations. Likelihood fixes the observations and compares candidate model parameters. The same expression can serve both purposes, but changing what is held fixed changes its interpretation.

Suppose an original teaching experiment records seven heads and three tails in ten independent tosses, with a constant but unknown head probability p. A likelihood calculation asks how well different values of p account for that observed outcome under these assumptions. It does not directly assign probabilities to those values of p.

Define the event before calculating

For one specified sequence containing seven heads and three tails, the probability is p**7 * (1-p)**3. For the count event of exactly seven heads in any order, multiply by the 120 possible arrangements.

At p=0.7, those probabilities are approximately 0.0022235661 and 0.266827932. Neither is a mistake: they describe different events. The combinatorial factor is constant with respect to p, so it changes the likelihood's height but not its maximizing parameter.

The NIST binomial reference documents the count model. Our ten-toss fixture and checks are included in the mathematics lab.

Reproduce the two calculations

Run this block from the lab directory. The helper calculates in log space before exponentiating and handles impossible boundary outcomes explicitly.

import numpy as np
from inference_core import coin_likelihood

sequence = coin_likelihood(.7)
count = coin_likelihood(.7, count_event=True)
assert np.isclose(sequence, .0022235661)
assert np.isclose(count, .266827932)
assert np.isclose(count / sequence, 120)
values = {p: coin_likelihood(p) for p in (.2,.5,.7)}
assert values[.7] > values[.5] > values[.2]
assert coin_likelihood(0)==0
print({'sequence_probability':sequence,'count_probability':count,
       'likelihood_comparison':values})

The comparison favors p=0.7 over the two other candidates. It does not mean that the probability of p being 0.7 is 0.2668. Treating the likelihood values as a discrete probability distribution over an arbitrarily chosen parameter grid would introduce an additional assumption.

Explain what a posterior adds

A Bayesian posterior combines a likelihood with a prior and normalizes over the parameter space. The prior is part of the specification; it cannot be silently supplied by calling likelihood a probability of the parameter.

For continuous observations, likelihoods can be built from densities rather than event probabilities. A density value can exceed one, so a large likelihood value is not automatically invalid. Its scale depends on the observation model and units. Comparing raw likelihood numbers across different datasets is usually not the question that parameter estimation answers.

Check the model assumptions before trusting the fit

If the coin mechanism changes halfway through the experiment, one constant p may conceal a change. If toss outcomes depend on earlier outcomes, the independent-product expression may be inappropriate. A mathematically correct optimization does not test those assumptions for you.

The ten observations also leave uncertainty about the underlying probability. The maximizing value is a point estimate, not a claim of certainty or evidence that a future sample must contain exactly seven heads.

Exercise: change the observation to six heads. Compare the specified-sequence and count-event likelihoods at p=0.6. Explain why their ratio changes while the maximizing p remains the observed head fraction. Submit the event definition alongside your output.

In NeuraPath's Data Science course, this distinction supports model fitting, classification losses and probabilistic reasoning. A useful explanation identifies the observed data, the unknown parameter and the assumptions before presenting a number.

Continue learning

This article is part of the Mathematics and statistical foundations sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Law of large numbers versus central limit theorem.
• Continue with Maximum likelihood estimation with a coin example.