Forecast intervals: measure coverage as well as width

A point forecast hides uncertainty. A forecast interval supplies lower and upper bounds for a stated nominal level, such as 90%. The label does not guarantee that 90% of future observations will fall inside. That depends on model assumptions, estimation and whether future conditions resemble the data used to fit the model.

Evaluate at least two properties: coverage, the share of actuals inside the interval, and width, how broad the intervals are. Coverage can be increased mechanically with unusably wide bounds, so neither measure is sufficient alone.

A structural-break stress test

The time-series lab selects a nonseasonal ARIMA order on validation, refits it on 120 development months, and requests 90% model-based intervals for 24 held-out months. The test period begins with an authored structural shift.

from timeseries_cases import interval_case

result = interval_case()
assert result["months"] == 24
assert result["covered_months"] == 14
assert result["observed_coverage"] < result["nominal_coverage"]
print(round(result["observed_coverage"], 3), round(result["mean_interval_width"], 2))

The result records undercoverage even though the mean interval spans more than 61 units. The model’s historical uncertainty did not adequately represent the new level. This is a fixture-specific demonstration, not a coverage estimate for another series.

Score by horizon and segment

Forecast uncertainty usually grows with horizon. Pooling all horizons can conceal narrow undercovered long-range intervals behind better short-range results. Report coverage and width at each business-relevant horizon. For multiple products, include high-volume, sparse and high-cost segments.

Twenty-four dependent monthly horizons provide a descriptive observed rate, not proof that the true coverage is exactly 58.3%. Rolling origins give more forecast cases, though overlapping targets remain dependent. Preserve the denominator and interval level with every reported rate.

Connect intervals to decisions

An inventory policy may care more about the upper tail than symmetric central coverage. Quantile forecasts or calibrated probabilistic methods can align with asymmetric understock and overstock costs. Backtest the resulting decision, not only the statistical interval.

Monitor realized coverage after release, but do not wait for a single monthly miss to refit blindly. Define a window, minimum sample and escalation rule. Investigate data quality, missed calendar events and regime change. Widening intervals may acknowledge uncertainty; it cannot repair a biased process by itself.

Record whether intervals are conditional on known future covariates and what scenarios they exclude. A 90% interval under an approved promotion calendar does not cover an unannounced competitor intervention by definition.

The Data Science course develops interval evaluation alongside point metrics, temporal validation and stakeholder handover.

Exercise

At each rolling origin, generate 80% and 90% intervals. Report coverage, mean width and interval score by horizon. Decide which level supports the planning cost and state whether any calibration uses validation only.

Continue learning

This article is part of the Forecasting and time-series analysis sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Exponential smoothing with interpretable components.
• Continue with Regression forecasting with known future covariates.

References: statsmodels ARIMA results and forecasting API and Forecasting: Principles and Practice on prediction intervals.