Respect API rate limits in a reporting pipeline

When an API asks a client to wait, retrying immediately can worsen the problem and consume the reporting window without making progress. Respect the provider's delay, bound the number of attempts and distinguish deferral from a completed extract.

HTTP 429 represents too many requests, and a response may include Retry-After guidance. The protocol definition is in RFC 6585; the provider's documentation supplies the applicable quota and operational details.

Test delay handling without real requests or sleeping

The original API lab accepts an injected fetch function, clock and sleep function. Tests can therefore prove which delay was requested without calling a live service or waiting in real time.

The implementation retries selected transient read failures. Its local exponential backoff begins at one second. A supplied Retry-After value can increase that delay, and a delay beyond the local wait budget causes deferral rather than an early retry.

from datetime import datetime,timezone
from api_extract import fetch_with_retry,HTTPFailure,RetryDeferred,retry_after_seconds

calls,waits = [],[]
def fetch(cursor):
    calls.append(cursor)
    if len(calls)==1:
        raise HTTPFailure(429,'2')
    return {'result':'simulated success'}

result = fetch_with_retry(fetch,'page2',sleep=waits.append)
assert result=={'result':'simulated success'}
assert calls==['page2','page2'] and waits==[2]
clock = datetime(2026,1,12,1,0,0,tzinfo=timezone.utc)
assert retry_after_seconds('Mon, 12 Jan 2026 01:00:07 GMT',clock)==7

def long_delay(cursor):
    raise HTTPFailure(429,'120')
deferred_waits = []
try:
    fetch_with_retry(long_delay,None,sleep=deferred_waits.append,max_wait_seconds=30)
except RetryDeferred:
    pass
else:
    raise AssertionError('long server delay was ignored')
assert deferred_waits==[]
print({'numeric_delay_seconds':waits,'http_date_delay_seconds':7,'long_delay_deferred':True})

The two-second instruction is honored. A 120-second instruction is not shortened to thirty seconds just because the local process has a smaller waiting budget. Instead, the caller receives an explicit deferral and can arrange a later attempt under its scheduling policy.

Keep the retry budget finite

The default teaching limit is three attempts, with waits only between attempts. After the final failed attempt, the error propagates. An infinite retry loop can occupy a worker indefinitely and conceal a persistent source problem.

Authentication and permission failures are not treated as transient rate limits. Repeatedly retrying the same unauthorized request is unlikely to repair access and may create additional operational noise.

Malformed delay headers also need an explicit policy. This lab rejects an invalid value rather than guessing. A real client should follow the provider's guidance while maintaining a safe bounded fallback where appropriate.

Coordinate beyond a single process

Several workers can each obey their own local limit and collectively exceed an account-level quota. A deployed system may need shared quota coordination, concurrency limits or a central extraction schedule.

Jitter can help avoid synchronized retry waves when many clients fail together. The teaching implementation uses deterministic backoff for transparent tests and does not implement fleet jitter or a distributed limiter. Those are separate additions to evaluate for the actual deployment.

Respect the scope of the quota: it may apply by account, endpoint, token, tenant or another provider-defined unit. Rotating identities to evade a limit is not a sound reporting strategy; align extraction volume and schedule with the authorized service contract.

Preserve completeness during deferral

A rate-limited page must not be replaced by an empty list and treated as the end of pagination. Keep the extraction incomplete, retain its diagnostic and resume or restart according to the source's snapshot and cursor rules.

The retry tutorial explains how stable IDs and a commit boundary prevent repeated reads from duplicating records. Rate-limit handling is one part of that larger correctness contract.

Exercise: simulate three consecutive 503 responses and verify the attempt and delay counts. Then simulate a 401 response and verify that it is surfaced after one attempt without any sleep call.

NeuraPath's Data Analytics with Generative AI course connects API automation with operational discipline. A useful rate-limit strategy waits when required, stops when appropriate and never disguises an incomplete extract as a successful report.

Continue learning

This article is part of the Reliable reporting automation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Handle paginated APIs without missing the final page.
• Continue with Design a human approval step for automated report distribution.