System design interview questions for AI applications

The prompt

"Design an assistant that answers employee questions using internal documents. Twelve business units, strict separation of their documents, 50,000 employees, answers expected within a few seconds."

Q1. Walk me through the request path and where time goes.

Authenticate the user and resolve their permissions; rewrite the query if needed; retrieve candidate chunks filtered by permission; rerank; build the prompt; call the model with streaming; validate and post-process; log. The model call dominates. A useful move is to write the budget explicitly:

budget_ms = {
    "auth_and_permissions": 30,
    "query_embedding": 40,
    "vector_search_filtered": 60,
    "rerank_top_30": 150,
    "model_time_to_first_token": 900,
    "post_processing": 20,
}
first_token = sum(budget_ms.values())
print(f"time to first token about {first_token} ms")
print(f"model share: {budget_ms['model_time_to_first_token'] / first_token:.0%}")

With these hypothetical figures, first token arrives at about 1.2 seconds and the model is three-quarters of it. Streaming makes the answer feel fast even when full generation takes longer. State that the numbers are placeholders to be measured.

Being assessed: that you budget latency by stage and know where optimisation pays.

Q2. How do you enforce business-unit separation?

Filter at retrieval, not after generation. Each chunk carries tenant and access-group metadata; the retriever applies a mandatory filter derived from the authenticated user, enforced in the retrieval service rather than passed by the client. For the strictest units, use separate indexes. Test it: an automated suite that queries as users of each unit and asserts no foreign chunk IDs are ever returned.

Q3. Can you cache responses?

Partially. Exact-match caching of the final answer is safe only when the question, the permission set and the document versions are identical, so the cache key must include all three. Caching embeddings and retrieval results for frequent queries is simpler. Semantic caching — reusing answers for similar questions — risks returning an answer to a subtly different question and needs a strict threshold plus evaluation. Invalidate on document updates.

Q4. The model provider has an outage or slows sharply. What happens?

Set timeouts per stage. On timeout or error, retry once with jitter, then fall back to a secondary model or provider if one is approved, and finally to a degraded mode: return the top retrieved passages with links and a message that generated answers are unavailable. Circuit breakers stop the whole fleet hammering a failing provider. Test fallbacks regularly, because an untested fallback fails when needed.

Q5. Estimate peak load and cost.

employees = 50_000
daily_active_share = 0.20
questions_per_active_user = 4
peak_hour_share = 0.15          # share of the day's queries in the busiest hour

daily_queries = employees * daily_active_share * questions_per_active_user
peak_qps = daily_queries * peak_hour_share / 3600
print(f"{daily_queries:,.0f} queries/day, peak about {peak_qps:.1f} queries/s")

tokens_per_query = 3_000        # prompt + output, hypothetical
cost_per_million_tokens = 200   # placeholder rupees; use the provider's current price
print(f"model cost about Rs {daily_queries * tokens_per_query * cost_per_million_tokens / 1e6:,.0f} per day")

About 40,000 queries a day and 1.7 per second at peak — modest for the web tier, so the real constraints are provider rate limits and cost. The biggest cost levers are prompt size (fewer, better chunks), routing simple questions to a smaller model, and caching.

Q6. How do you choose between a large and a small model per request?

Route by task: classification, query rewriting and short factual answers often work on a smaller model; complex synthesis goes to the larger one. Decide the routing rule from evaluation results by question type, not intuition, and monitor quality by route.

Q7. How do you evaluate quality in production?

Offline: a versioned test set per business unit with reference answers and expected refusals, run on every prompt, retrieval or model change, with release gates. Online: user feedback, escalation to human help desks, citation clicks, and weekly sampled human review. Track retrieval recall separately from answer faithfulness so regressions can be located.

Q8. What do you log, and what don't you?

Log request IDs, user and tenant IDs, retrieved chunk IDs, model and prompt version, latency per stage, token counts and validation outcomes. Treat question text and answers as potentially sensitive: apply the company's retention rules, restrict access, and avoid sending logs to systems outside the approved boundary.

Q9. An employee reports the assistant quoted an outdated HR policy. Trace the fix.

Find the request by ID; check which chunk was retrieved and its document version. Likely causes: the old version was never removed from the index, the ingestion job failed silently, or both versions were retrieved and the model chose the old one. Fixes: version-aware ingestion that retires superseded chunks, effective-date metadata with a filter, an ingestion health check that alerts on failures, and a test case added for this policy.

Q10. What would you build first?

A thin end-to-end path for one business unit: ingestion, permission-filtered retrieval, generation with citations, logging and an evaluation set. Measure, then add reranking, caching and routing where the numbers justify them. Senior interviewers listen for this sequencing — the evaluation loop before optimisations.

What senior candidates do differently

They state assumptions and put numbers on them, enforce security in the service layer rather than the prompt, design the degraded mode as carefully as the happy path, and keep evaluation in the release process.

The model-side questions are in the ML system design round plan and the RAG bank. Production architecture for AI systems, including multi-tenant retrieval and evaluation pipelines, is covered in FDE for Professionals.