Why a RAG demo isn't a portfolio, and the audit that fixes it
In this article (11 sections)
- What the reviewer sees in the first two minutes
- The eight-question audit
- Upgrading each low score
- Questions 1 and 2: build the evaluation first
- Question 3: add unanswerable questions
- Question 4: check citations by hand
- Question 5: make one real decision
- Question 6: write the failure table
- Question 7: describe the corpus
- Reframe the README around the audit
- Where this leaves your other projects
Related programme: the Generative & Agentic AI course (3 months).
A RAG demo that answers questions over a PDF shows that you can connect a document loader, an embedding model and a chat model — which a tutorial also shows. It becomes portfolio evidence only when it answers the questions a reviewer actually has: does it work on cases you did not pick, does it know when it doesn't know, and can you explain why it fails when it fails. The audit below turns a demo into that evidence.
What the reviewer sees in the first two minutes
Picture someone screening forty candidates for an AI engineering role. They open your repository and see a README with a screenshot of a chat window and a correct answer. They have seen that screenshot many times this week. They look for three things, quickly:
- 1Is there a test set, and a number?
- 2Are there failures written down?
- 3Is there any decision the candidate made that the tutorial didn't?
If all three are missing, the demo reads as "completed a walkthrough", no matter how polished the interface is.
The eight-question audit
Score your project honestly: 0 = absent, 1 = partial, 2 = done and documented.
| # | Audit question | What a 2 looks like |
|---|---|---|
| 1 | Does it have a question set you did not tune on? | At least 30 questions with expected answers, kept separate from those you used while building |
| 2 | Does it report retrieval quality separately from answer quality? | "Correct source in top 3: 26/30. Correct answer: 22/30." |
| 3 | Does it refuse questions the documents don't answer? | A set of unanswerable questions with a refusal rate reported |
| 4 | Do citations point to the passage that supports the claim? | Sampled and checked by hand, with the result stated |
| 5 | Did you make and justify a design choice? | "Chunked by section heading, not fixed length, because policies reference section numbers; compared both on the question set" |
| 6 | Are failures listed and diagnosed? | Five or more failures with the pipeline stage responsible |
| 7 | Is the corpus described honestly? | Source, licence or synthetic generation method, size, and what it does not cover |
| 8 | Can someone else run it? | One command, pinned dependencies, no secret keys committed |
A typical tutorial-derived demo scores 2 on question 8 at best and 0 on most others. A total of 12 or more is a project worth leading with.
Upgrading each low score
Questions 1 and 2: build the evaluation first
Write thirty questions before you change anything else. For each, record the expected answer and the document section that contains it. Then measure retrieval and generation separately. When the answer is wrong, you now know whether the right passage was never retrieved or was retrieved and misused — two different fixes.
Question 3: add unanswerable questions
Write ten questions that sound related but that your documents do not answer. A system that confidently answers them is the failure hiring managers worry about most in production. Reporting "refused 8 of 10" shows you tested for it.
Question 4: check citations by hand
Pick twenty answers at random and verify that each cited passage supports the sentence it is attached to. Models sometimes cite a real passage that says something slightly different. Report the count.
Question 5: make one real decision
Choose one variable — chunking method, number of passages retrieved, or whether to rewrite the user's query — and compare two options on your question set. It does not matter which wins. It matters that you decided from evidence.
Question 6: write the failure table
| Question | What happened | Stage | Fix tried |
|---|---|---|---|
| "What is the notice period in probation?" | Returned the post-probation period | Retrieval: both sections scored similarly | Added section titles to chunks; now correct |
| "Is Diwali a fixed holiday?" | Answered from a general knowledge guess | Generation: no refusal when source silent | Added refusal instruction; still fails 1 in 3 runs |
The second row, with a fix that only partly works, is more credible than a table where everything was solved.
Question 7: describe the corpus
State whether documents are synthetic, public domain or licensed, how many there are, and what topics they omit. If you generated them, say how. This protects you in an interview when someone asks, "would this work on our documents?" — you can answer precisely what it was tested on.
Reframe the README around the audit
Restructure your README in this order: the user and their problem, the corpus, the evaluation results table, the design decision and its comparison, the failure table, how to run it. Move the chat screenshot to the bottom. Reviewers who scroll that far will already be interested.
Where this leaves your other projects
The same audit logic applies to any AI project: evaluation set, separated metrics, refusal, a justified decision, diagnosed failures. If you want fully specified alternatives to a document chatbot, see the three LLM portfolio pieces with acceptance criteria. You can also run your resume through the free skill check to see how your listed skills read against a target role. Evaluated retrieval projects are a core part of the Generative & Agentic AI add-on.
Pankit Kumar has 10 years in Data Science & AI, building and shipping production systems in regulated pharma and clinical environments. He is a freelance trainer at Boston Institute of Analytics, AnalytixLabs and Scaler, and has taught this material to thousands of working professionals.
This article is part of our Generative & Agentic AI programme — 3 months. Add practical GenAI, retrieval and agent-building skills to your existing toolkit.
Explore Generative & Agentic AI