LLM and generative AI interview questions, a mock round

Opening

Interviewer: We want an assistant that answers staff questions about claim procedures. Before design, some basics.

1. How generation works

Interviewer: In one minute, how does an LLM produce an answer?

Candidate: It splits text into tokens, then repeatedly predicts a probability distribution over the next token given everything so far, picks one according to the sampling settings, appends it and repeats until a stop condition. It has no separate lookup of facts; everything comes from patterns learned in training plus whatever we put in the prompt.

Score note: strong. Named next-token prediction and the consequence — no built-in fact store — which sets up the hallucination question.

2. Temperature and sampling

Interviewer: What does temperature do? Show me.

Candidate: It divides the model's scores before softmax. Low temperature sharpens the distribution toward the top token; high temperature flattens it.

import math

logits = {"approved": 3.0, "pending": 2.0, "rejected": 0.5}   # toy next-token scores

def softmax(scores, temperature):
    scaled = {k: v / temperature for k, v in scores.items()}
    total = sum(math.exp(v) for v in scaled.values())
    return {k: round(math.exp(v) / total, 3) for k, v in scaled.items()}

for t in (0.2, 1.0, 2.0):
    print(f"temperature {t}: {softmax(logits, t)}")

Candidate: At 0.2 almost all probability sits on "approved"; at 2.0 "rejected" becomes plausible. For a procedures assistant I'd keep temperature low, because we want consistent, grounded answers rather than variety. Top-p limits sampling to the smallest set of tokens covering probability p, which trims the unlikely tail.

Score note: strong. Tied the setting to the use case. A weaker answer says "temperature controls creativity" and stops.

3. Hallucination

Interviewer: Why do models make things up, and what would you do about it here?

Candidate: The model is trained to produce likely text, not to check truth, so when the answer isn't well represented in training or the prompt it still produces something fluent. For claim procedures I'd put the relevant procedure passages into the prompt through retrieval, instruct it to answer only from them and say when they don't cover the question, and ask it to cite which passage it used so a reviewer can check. Then measure how often answers are unsupported on a test set.

Score note: covered cause, mitigation and measurement. Did not claim retrieval eliminates hallucination — good.

4. Context window

Interviewer: Why not paste all 400 pages of procedures into every prompt?

Candidate: It may not fit the context window; even if it fits, cost and latency scale with input tokens on every request, and models can miss details buried in very long inputs. Retrieving the few most relevant passages is cheaper and usually more accurate. I'd test that claim on our documents rather than assume it.

5. Fine-tuning versus retrieval

Interviewer: A colleague says we should fine-tune on the procedures instead. Your view?

Candidate: They solve different problems. Retrieval supplies facts that change — procedures get revised, and with retrieval we update a document, not a model. Fine-tuning changes behaviour: format, tone, classification of claim types, domain phrasing. It is poor at reliably memorising specific facts and harder to update. Here I'd start with retrieval and good prompts, and consider fine-tuning later if we need a consistent output style that prompting can't achieve.

Score note: this is the question that separates candidates. Clear distinction, a default choice and a condition for changing it.

6. Cost estimate

Interviewer: Rough monthly cost?

Candidate: I'd estimate from tokens. Placeholder prices — I'd replace them with the provider's current price sheet:

requests_per_day = 3_000
input_tokens = 2_500      # question + retrieved passages + instructions
output_tokens = 300
price_in_per_million = 100.0    # placeholder rupees per million input tokens
price_out_per_million = 400.0   # placeholder rupees per million output tokens

daily = requests_per_day * (input_tokens * price_in_per_million
                            + output_tokens * price_out_per_million) / 1_000_000
print(f"daily Rs {daily:,.0f}, 30-day Rs {daily * 30:,.0f}")
print(f"input share of cost: {input_tokens * price_in_per_million / (input_tokens * price_in_per_million + output_tokens * price_out_per_million):.0%}")

Candidate: With these placeholders it's about Rs 1,110 a day, and two-thirds of it is input tokens — so trimming retrieved context saves more than shortening answers. The structure of the estimate matters more than the placeholder prices.

Score note: excellent. Explicitly labelled the prices as placeholders instead of quoting remembered numbers.

7. Evaluation

Interviewer: How do you know the assistant is good enough to launch?

Candidate: Build a test set of real staff questions with reference answers written by the claims team, including questions the documents don't answer. Score each output on correctness, whether it is supported by the retrieved passages, and whether it correctly declines when it should. Use an LLM grader to scale, but check its judgements against human ratings on a sample first. Set launch thresholds with the claims team, and keep the set for regression testing after every prompt or model change.

8. Safety and data

Interviewer: Anything that worries you?

Candidate: Claim files contain personal and health information. I'd check which data may be sent to an external API under our agreements and applicable law, mask identifiers where possible, log prompts securely with retention limits, and restrict retrieval so staff only get passages they are permitted to see. Also prompt injection if the assistant ever reads customer-submitted documents.

Debrief

Interviewer's summary (hypothetical): hire recommendation. Reasons: consistent trade-off thinking, labelled assumptions in the cost estimate, and evaluation designed before launch. Area to probe next round: depth on retrieval evaluation.

Using this transcript

Answer each question aloud yourself before reading the candidate's version, then compare. The retrieval part continues in the RAG interview bank, and prompt engineering questions cover output control in more depth. These topics form the core of the Generative and Agentic AI programme.