Write a data contract for a knowledge assistant

A knowledge assistant depends on more than a vector collection. Its data contract defines which source versions are eligible, how they are processed, who may retrieve them and what blocks publication.

Make the contract inspectable

The RAG ingestion lab creates a synthetic contract.

from ingestion_cases import contract_case

result = contract_case()
assert result["complete"] is True
assert result["approval"] == "pending"
assert result["live_service_claim"] is False
assert result["contract"]["grain"] == "one record per source version"

The record covers key, required fields, 210-day freshness, 24-hour deletion target, allowed languages, chunker/embedding versions and a publish gate requiring completeness plus authorization tests. Complete fields do not mean human approval or a live service.

Cover source and derived data

Define source grain, stable ID, owner, permission, format, language, version/effective interval, freshness and retention. For derived data, define extraction/chunking identity, metadata inheritance, embedding version, source spans and index destinations.

Add invariants: unique source-version keys, no chunk without lineage, no active source outside its effective interval, no missing access class, and no vector whose dimension/configuration conflicts with its collection. State how malformed, encrypted, stale and contradictory sources are handled.

Define change and deletion behaviour

Classify additive metadata, new source versions, changed access, schema breaks and semantic changes. Name affected consumers and reevaluation requirements. Document deletion across raw files, chunks, lexical/vector indexes, caches, old index versions and backups under policy.

The publication gate should combine manifest completeness, extraction sample review, authorization tests, labelled retrieval results and owner approval. Record who can waive which condition; critical privacy or authorization gates should fail closed.

The Generative & Agentic AI course uses this contract as the bridge from ingestion to retrieval evaluation, LLMOps and capstone operations.

Exercise

Write a contract for a two-language internal assistant. Inject a missing owner, stale version, access downgrade and deletion request. Show each failure, affected index and exact condition required before publication resumes.

Continue learning

This article is part of the RAG ingestion and document preparation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the preceding task in Handle contradictory document versions explicitly.
• Return to the cluster foundation in Build a RAG document inventory with source ownership.

Reference: OpenLineage specification.