Audit the gaps in Delhi's air monitors before you trust an average

The question, stated so it can be wrong

"Are PM2.5 readings from Delhi's monitors missing completely at random, or are gaps more common on some days, stations or seasons, in a way that biases monthly means?"

That phrasing matters. "Clean the data" is not a question. A testable claim about missingness is, and it forces you to report a result even if the result is "no evidence of bias".

Where the files live

CPCB's CCR portal is at airquality.cpcb.gov.in/ccr. Two routes work:

• AQI Data Repository lists files by State, City and Station Name. It has frequencies of 15M, 1H, 8H and 24H, and data files by year and month. It is the easiest place to collect many station-months without re-running searches.
• All India CAQMS Data is the Advance Search, with date range, state, city, station, parameter and averaging interval (15M through 24H) behind a captcha.

For a gap audit, use hourly data (1H). Daily averages hide the hours that went missing inside a day. The "CAAQMS Stations List Delhi NCR" card gives you the station roster for Delhi. Column names differ between the repository files and the search export, so check the downloaded file's header and write one loader per format.

Why a hiring manager would care

Every analytics team has a version of this problem: sensor feeds, app telemetry, store-level sales with shops that did not report. A candidate who can say "18% of hours were missing, and missingness rose on the worst days, so the raw mean understates exposure" is showing judgment. Being fluent with a charting library does not show that. It is also a project where your conclusion can differ from other candidates' conclusions, which makes for a better interview.

Four kinds of gaps to separate

1. 1Structural gaps: the station did not exist yet, or was decommissioned. Exclude these from the denominator.
2. 2Outages: runs of consecutive missing hours, often maintenance or power.
3. 3Scattered drops: isolated missing hours.
4. 4Invalid values: zeros, negatives or constant repeats (a flatlined sensor reporting the same value for many hours). Treat these as missing, and count them separately so the rule is visible.

Keep the four kinds apart. An average that drops a planned month-long shutdown means something different from one that drops the worst smog hours.

Synthetic simulation: how gaps can bias a mean

The data below is synthetic. It builds a fake hourly series in which the chance of a reading going missing rises with pollution. That lets you see the bias mechanism before you look for it in real data.

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
hours = pd.date_range("2025-11-01", periods=24 * 30, freq="h")
true_pm = 150 + 60 * np.sin(np.arange(len(hours)) / 24 * 2 * np.pi) + rng.gamma(2, 25, len(hours))

# Missingness that depends on the value itself (missing not at random)
p_missing = np.clip(0.05 + (true_pm - true_pm.mean()) / 800, 0.01, 0.6)
observed = np.where(rng.random(len(hours)) < p_missing, np.nan, true_pm)

df = pd.DataFrame({"ts": hours, "true": true_pm, "obs": observed})
df["missing"] = df["obs"].isna()

print("share missing:", round(df["missing"].mean(), 3))
print("true mean:", round(df["true"].mean(), 1))
print("naive observed mean:", round(df["obs"].mean(), 1))

# Detect: do missing hours sit in higher-pollution neighbourhoods?
df["neighbour_mean"] = df["obs"].rolling(7, center=True, min_periods=2).mean()
compare = df.groupby("missing")["neighbour_mean"].mean().round(1)
print(compare)

# Outage runs: lengths of consecutive missing hours
run_id = (df["missing"] != df["missing"].shift()).cumsum()
runs = df[df["missing"]].groupby(run_id).size()
print("longest outage (hours):", int(runs.max()), " runs:", len(runs))

In the real data you do not have the true column. That is why the neighbour comparison matters: if the observed hours around a gap run higher than average, the gap probably hid high values too. It is indirect evidence, and your write-up should call it that.

Analysis plan

• Coverage table: valid hours ÷ expected hours per station per month, excluding structural gaps.
• Calendar heatmap: one row per station, one column per day, coloured by missing share. Outages show up as horizontal bars.
• Hour-of-day profile: is missingness higher at night, when fewer staff are on site?
• Missingness vs neighbourhood level: the comparison in the code above, done per station.
• Adjusted estimates: compare the naive mean with two alternatives. One is the mean of daily means, with a minimum number of hours per day. The other is a seasonal-hourly imputation. Report all three.
• Station agreement: where two stations are close, check whether one fills the other's gaps plausibly.

Deliverable specification

The limits section should say that you cannot prove why a reading is missing. You can only show patterns consistent with one explanation or another.

Interview questions to prepare for

• Explain MCAR, MAR and MNAR using your own stations as examples.
• Why did you use hourly rather than daily data for the audit?
• Would multiple imputation help here, and what would you condition on?
• If a regulator used the naive mean, in which direction would they be wrong, and by how much in your data?
• How would you alert on a flatlined sensor in a live pipeline?

For a lighter first project on the same portal, see the Noida and Ghaziabad PM2.5 comparison. The work of reasoning about missingness and imputation is covered in depth in our Data Science programme.