Build an ingestion error queue for failed documents

Silently skipping a failed document creates an incomplete knowledge base that can still look healthy. An error queue makes missing sources visible and separates transient retries from failures that need an owner.

Classify the failure

The RAG ingestion lab creates three synthetic failures.

from ingestion_cases import error_queue_case

result = error_queue_case()
assert result["retryable_ids"] == ["temporary-read"]
assert result["dead_letter_ids"] == ["bad-encryption", "unknown-encoding"]
assert result["blind_retry"] is False

Only a storage timeout is retryable. Encrypted content and an unknown encoding require resolution rather than repeated processing. The queue is an in-memory fixture, not a running broker.

Store actionable context

Each item needs source/version ID, stage, reason code, attempt count, first/last time, worker/configuration version, owner, next action and safe diagnostic details. Avoid copying sensitive document content into a broadly visible queue.

Use exponential backoff and a maximum for transient failures. Make stages idempotent so a retry cannot duplicate chunks. Move permanent or exhausted items into a review state and notify the accountable owner according to severity.

Connect errors to completeness

The source manifest should show expected, indexed, quarantined and failed documents. Prevent publication when required sources are missing. A pipeline-success rate can hide one critical policy, so report failures by owner/type and identify exact source gaps.

Test recovery: correct a source, replay the same item, verify prior partial artifacts are replaced, and record the successful manifest transition. Keep poison documents from blocking unrelated sources while preserving the overall release gate.

The Generative & Agentic AI course connects error queues to idempotency, observability and source-completeness checks.

Exercise

Inject timeout, encrypted file, malformed PDF, unknown encoding and oversized-page failures. Define retry policy and owner, recover two cases and prove an incomplete required corpus cannot become active.

Continue learning

This article is part of the RAG ingestion and document preparation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Choose an embedding model with task-specific evidence.
• Continue with Validate document encoding and language before indexing.

Reference: Apache Airflow best practices.