Retry a failed data extract without duplicating records

A retry should repeat the failed operation without repeating already committed effects. For a paginated read, retry the same page request, validate the successful response and add its records once. Do not append an incomplete attempt directly to the final reporting table.

Stable record identifiers and a clear commit boundary are essential. “Remove duplicates at the end” is insufficient if duplicate IDs contain conflicting values or if the source changes during extraction.

Use a bounded simulated API

The original API extraction module defines a fictional cursor API. It is a local teaching contract, not a client for any vendor. Three pages contain E1, E2 and E5, with E2 repeated identically across two pages and an empty final page.

Each page declares the same snapshot ID and expected unique-item count. Collection returns a result only after all pages validate and the unique count reconciles. A failed collection exposes no partial return value.

Retry the same cursor after a transient failure

from api_extract import collect_pages,fixture_pages

pages = fixture_pages()
calls,waits = [],[]
def fetch(cursor):
    calls.append(cursor)
    if cursor=='page2' and calls.count(cursor)==1:
        raise TimeoutError('simulated transient read failure')
    return pages[cursor]

result = collect_pages(fetch,sleep=waits.append)
assert calls==[None,'page2','page2','page3']
assert waits==[1]
assert result['raw_items']==4 and result['unique_items']==3
assert result['identical_replays']==1
assert [row['event_id'] for row in result['items']]==['E1','E2','E5']
assert sum(row['amount_paise'] for row in result['items'])==3500
print({'request_cursors':calls,'simulated_waits_seconds':waits,
       'unique_items':result['unique_items'],'amount_paise':3500})

The wait function records the delay rather than sleeping. No network request occurs. Page two is requested twice, but only its successful response contributes records. The deliberate cross-page replay of E2 is then collapsed by its stable ID and identical payload.

Reject conflicting identity evidence

If the second E2 has a different amount, the collector raises a conflict instead of choosing whichever copy happened to arrive last. That prevents transport order from becoming an undocumented business rule.

Some real APIs expose legitimate record updates with versions or update timestamps. In that case, define the version-selection rule and snapshot semantics explicitly. This teaching contract requires a stable snapshot, so conflicting payloads violate it.

Keep retries limited to suitable operations

The module retries selected transient HTTP failures and connection timeouts, with a bounded attempt count. It does not retry authentication failures as if waiting would repair credentials. For rate limiting, it respects a supplied delay or defers when that delay exceeds the local wait budget.

HTTP 429 and its optional Retry-After behavior are defined in RFC 6585. A deployed client must also follow the provider's actual retry and quota contract.

These examples concern reads. Retrying a payment, message send or other mutation may create duplicate effects unless the service supports an appropriate idempotency mechanism and the client uses it correctly. Do not generalize a read-retry loop to every API call.

Choose a commit and checkpoint policy

The lab holds the extract in memory and returns only a complete verified collection. If an attempt ultimately fails, a later invocation starts again. That is simple and bounded, but it is not a persistent checkpoint design for a huge extract.

A checkpointed workflow needs a source snapshot identifier, cursor state, committed record identity and a way to ensure the checkpoint and data agree. Advancing a cursor before its page is durably recorded can lose data; recording a page without advancing the cursor can replay it. Design recovery for both cases.

Once a complete extract is available, the report bundle workflow demonstrates a separate local output boundary. Neither component sends a report externally.

Exercise: make page two fail on every attempt. Verify that the retry count is bounded and no completed extract is returned. Then rerun successfully and confirm the final unique count and amount match the original reference.

NeuraPath's Data Analytics with Generative AI course connects Python error handling with reliable data preparation. A useful retry design preserves identity and completeness as carefully as it handles temporary failure.

Continue learning

This article is part of the Reliable reporting automation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Data freshness checks: stop a stale dashboard release.
• Continue with Build a reconciliation step into every automated report.