Validate an API response before loading it into a dataframe

A JSON response is not automatically a valid analytical table. An API may return an error envelope, omit required fields, change types or provide only one page of a larger result. Validate the response contract before allowing dataframe type inference to shape the data.

Separate transport success, schema validity and extract completeness. A successful HTTP status does not establish all three.

Define the envelope and record contract

The original simulated API module expects four page fields: items, next cursor, snapshot ID and total unique-item count. Each item requires an event ID, an allowed region and a nonnegative integer amount in paise.

The API is fictional and makes no network requests. Its strict field policy is deliberate: unexpected fields trigger review instead of being silently ignored. A real client may choose a different compatibility policy, but it should be explicit.

Amounts are integers, not strings containing currency symbols or floating-point approximations. Boolean values are rejected even though Python treats booleans as integer subclasses.

Validate the complete collection, then construct the frame

import pandas as pd
from api_extract import collect_pages,fixture_pages,validate_page,APIContractError

pages = fixture_pages()
result = collect_pages(pages.__getitem__)
frame = pd.DataFrame(result['items'],columns=['event_id','region','amount_paise']).astype({
    'event_id':'string','region':'string','amount_paise':'int64'})
assert frame['event_id'].tolist()==['E1','E2','E5']
assert frame['event_id'].is_unique
assert frame['amount_paise'].sum()==3500
assert str(frame['amount_paise'].dtype)=='int64'
bad_page = fixture_pages()[None]
bad_page['items'][0]['amount_paise']=True
try:
    validate_page(bad_page)
except APIContractError as error:
    assert str(error)=='invalid_item_amount'
else:
    raise AssertionError('boolean amount accepted')
print(frame.to_dict(orient='records'))

The example runs with the programme's recorded pandas environment and uses explicit dtypes after validation. It does not rely on pandas to decide whether a boolean, text value or missing field is a legitimate monetary amount.

The collector also verifies pagination, stable snapshot identity and the declared total before the dataframe is created. Loading only the first valid page would still produce an incomplete table.

Do not convert an error response into an empty dataset

A response such as an authentication error is not equivalent to a valid envelope with an empty item list and a declared total of zero. Treating both as an empty dataframe can turn access failure into a misleading “no activity” report.

Likewise, a missing items field should not default to an empty list merely to keep the pipeline running. Fail with a diagnostic that identifies the contract violation and preserves enough nonsecret context for investigation.

For a valid empty extract, construct the expected columns and dtypes explicitly. Otherwise downstream code may behave differently because an empty dataframe has no inferred columns or unexpected types.

Validate values as well as types

An integer amount can still be negative when the contract permits only nonnegative values. A string region can still be outside the allowed vocabulary. An event ID can be duplicated with conflicting content.

In this fixture, refunds are not negative payment amounts; they would require a separate explicitly modeled record type. Do not force a new business meaning into an existing field without revising the contract.

Timestamp, currency and unit fields need similar rules when present. Avoid accepting a numeric field named amount without knowing whether it represents major currency units, minor units or something else.

Plan for API changes without hiding them

Keep representative valid and invalid response fixtures in tests. A provider adding a field, changing pagination or returning a new status should trigger a deliberate compatibility decision.

Strict validation can interrupt reporting during a harmless schema extension; permissive validation can conceal a consequential change. Choose which fields are required, which unknown fields are tolerated and which changes require a version update according to the actual provider contract.

The lab's checks cover a controlled schema. They do not establish that a real provider's data are semantically correct or that its declared total is trustworthy.

Exercise: replace an amount with the string '1000', then with 1000.0. Verify that the strict contract rejects both. If the actual provider documents numeric strings, add a deliberate parser and tests rather than relying on incidental dataframe coercion.

NeuraPath's Data Analytics with Generative AI course connects API extraction with reliable pandas analysis. A dataframe should begin with validated records and an explicit meaning for every field.

Continue learning

This article is part of the Reliable reporting automation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Separate secrets from report configuration.
• Continue with Handle paginated APIs without missing the final page.