Portfolio project: a cited policy assistant with abstention

A useful RAG portfolio should demonstrate the document lifecycle and evaluation, not only a chat screen. A cited policy assistant is a bounded project when its sources, permissions, version logic and abstention behaviour are explicit.

Verify the local contract fixture

The retrieval evaluation lab defines six expected outcomes.

from retrieval_cases import portfolio_case

result = portfolio_case()
assert result["cases"] == 6
assert result["contract_correct"] == 6
assert result["all_contract_cases_pass"] is True
assert result["service_deployed"] is False
assert result["model_executed"] is False
assert result["human_review"] == "pending"

Five answerable questions map to exact synthetic source IDs; the meal-allowance question abstains. This checks an expected-output contract without generation or deployment.

Build the project in layers

1. 1Author or obtain permitted documents and create a source/version/access manifest.
2. 2Extract pages/tables, create source-linked chunks and verify completeness.
3. 3Build BM25 and label question-to-evidence pairs plus hard negatives.
4. 4Add dense/hybrid retrieval only after a fair baseline comparison.
5. 5Generate structured claims with citations and an abstention response.
6. 6Evaluate retrieval, support, correctness, citation eligibility and unanswerable cases separately.

Use synthetic policies for a public portfolio unless you have explicit permission. Label them clearly. Add version-sensitive questions, an authorization filter and a deletion/reindex demonstration.

Present evidence and limits

Include one-command reproduction, environment, manifests, eval set, per-case results, traces, release report and operating runbook. Demonstrate one correct answer, one stale hard negative, one unsupported question and one failed input. State whether model calls, deployment and load tests actually ran.

The Generative & Agentic AI course aligns this project with its RAG deliverable and later agent/LLMOps modules.

Exercise

Expand the local corpus to 20 invented documents and 80 labelled questions. Add a real permitted embedding/model run, keep its configuration/results separate from fixtures and submit the release report for human review.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the preceding task in Design a RAG release report for a reviewer.
• Return to the cluster foundation in Build a keyword retrieval baseline before vector search.

Reference: NIST Generative AI Profile.