Poisson regression for nonnegative event counts

Counts have structure that ordinary least squares can ignore. Tickets per hour, defects per batch and arrivals per interval cannot be negative. Their variance often changes with their expected level. Poisson regression models the logarithm of the expected count, so its fitted mean remains positive.

That does not make every count dataset Poisson. The observation window, repeated entities, excess zeros and dependence between events still need investigation.

Build a count experiment with an explicit grain

Our original fixture has 500 independent rows. Each row represents one equal one-hour exposure. A load index is sampled between -2 and 2, and the authored expected count is exp(0.5 + 0.8 * load_index). The observed target is a Poisson draw from that mean. The generator mean remains in the file for audit but is excluded from model inputs.

The split is fixed at 300 training, 100 validation and 100 test rows. An unpenalized PoissonRegressor uses only the load index. Its fitted intercept is 0.5797 and coefficient is 0.7863. These estimates are close to the authored rule in this sample; they are not business effects.

The scikit-learn PoissonRegressor documentation specifies the log link and Poisson deviance. The library's non-normal loss example also demonstrates frequency modelling with exposure as sample weight.

Compare against a baseline on the right loss

The baseline predicts the training mean, 2.72 events, for every row. On validation, mean Poisson deviance is 0.9485 for the model and 2.7087 for the baseline. On the untouched test partition, it is 0.9498 versus 2.8123.

All fitted means are positive. The validation set still contains 29 observed zeros and the test set 19. A positive expected count does not say every interval must contain an event.

from decision_cases import count_case

r = count_case()
assert r['results']['validation']['minimum_prediction'] > 0
assert r['results']['test']['zero_actual_rows'] == 19
assert r['results']['test']['model_deviance'] < r['results']['test']['baseline_deviance']
print(r)

Run this in the supervised-model lab. The fixture validator rejects duplicate row IDs, non-finite fields, negative or fractional counts, and unequal exposure. That last check is deliberate: the example cannot silently treat a ten-hour window like a one-hour window.

Know when the likelihood is too simple

A Poisson distribution connects its conditional mean and variance. Real counts can be more dispersed because customers, machines or days differ in unmeasured ways. They can also be under-dispersed, zero-inflated or temporally dependent. Diagnose residuals and variance by relevant segments instead of inferring suitability from a nonnegative target alone.

Exposure is part of the target definition. With unequal windows, model a rate with the appropriate exposure treatment, such as an offset supported by the chosen implementation or frequency with exposure weights under a documented contract. Do not merely add window length as an unconstrained feature and assume equivalence.

Repeated customers also invalidate the fixture's independence story. Use grouped or temporal assessment when the deployment unit demands it. A lower deviance in this synthetic example establishes one worked calculation, not universal superiority over tree models or negative-binomial alternatives.

Exercise: regenerate the fixture with exposures of 0.5, 1 and 4 hours while holding the event rate fixed. Define the target and weighting contract before fitting. Then test whether duplicating a four-hour observation into four one-hour observations changes the conclusion.

NeuraPath's Data Science course links distributional assumptions to data grain, loss functions and reproducible checks. That is the difference between recognizing a Poisson API and designing a count model responsibly.

Continue learning

This article is part of the Supervised learning methods sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Naive Bayes: test the independence assumption empirically.
• Continue with Quantile regression for uncertainty-aware delivery estimates.