Data scientist to AI engineer: what to relearn and what to drop

What transfers without change

Start with what you already have, because it is more than many AI engineers do:

• Scepticism about a good-looking number. You know leakage, overfitting and cherry-picked examples. LLM demos are full of all three.
• Designing an evaluation set. Choosing representative cases, including hard ones, and keeping them separate from what you tune on.
• Error analysis. Reading failures one by one and grouping them. This is the core of improving an LLM application.
• Communicating uncertainty. Explaining that a system is right most of the time but not always, and what that means for a decision.

Six instincts to retrain

1. "Improve the model" becomes "improve the system around it"

As a data scientist, a bad result often meant feature work or a different algorithm. With an LLM you usually do not control the weights. The levers are retrieval quality, the instructions, the tools the model can call, input validation and how the output is checked. Your first question becomes: "which step in the pipeline produced this failure?"

2. One metric becomes many small checks

A classification model might be judged on F1. An assistant that answers policy questions needs separate checks: did it cite a real source, did it refuse when it should, is the number it quoted in the source, is the format parseable. You track each one.

3. Deterministic evaluation becomes repeated evaluation

The same prompt can produce different outputs. Run evaluations more than once, compare pass rates, and treat small differences as noise until repeated.

4. Notebook code becomes service code

A notebook that works once is not deliverable. AI engineering expects functions with clear inputs, tests, logging, timeouts and retries, configuration outside the code, and a way to deploy. This is often the biggest relearning.

5. Offline scoring becomes online monitoring

In many data science roles, a model was evaluated before release and checked occasionally. An AI application needs you to log inputs and outputs (with privacy controls), watch for failures in production, and feed real failures back into the test set.

6. Accuracy becomes accuracy plus cost plus latency

Every call has a price and a response time. A design that is slightly more accurate but twice as slow may be wrong for a support agent waiting on a customer. You will make these trade-offs weekly.

What you can mostly drop, for now

• Hyperparameter search as a daily activity.
• Building custom architectures for text tasks that existing models already handle.
• Long offline experiments before anyone sees a working version.

These skills are not wasted; fine-tuning and custom models still matter in some teams. They just stop being the default first move.

The same instinct, written the engineering way

Here is a small evaluation harness for a hypothetical policy assistant. The "model" is faked with a lookup so the code runs anywhere; in real work you would call your application instead. The point is the structure: named cases, independent checks, a pass rate per check.

# Synthetic evaluation harness: each case has checks that pass or fail independently.
CASES = [
    {"id": "leave-01", "q": "How many casual leaves per year?", "must_contain": "8", "source": "HR-POL-3", "should_refuse": False},
    {"id": "leave-02", "q": "Can I carry forward casual leave?", "must_contain": "not", "source": "HR-POL-3", "should_refuse": False},
    {"id": "salary-01", "q": "What is my manager's salary?", "must_contain": "", "source": "", "should_refuse": True},
]

def fake_assistant(question):
    answers = {
        "How many casual leaves per year?": ("You get 8 casual leaves per year.", "HR-POL-3"),
        "Can I carry forward casual leave?": ("Casual leave can be carried forward.", "HR-POL-3"),
        "What is my manager's salary?": ("I can't share other employees' pay details.", ""),
    }
    return answers[question]

def run(cases):
    results = {"content": [], "citation": [], "refusal": []}
    for c in cases:
        text, cited = fake_assistant(c["q"])
        refused = "can't" in text.lower()
        results["refusal"].append(refused == c["should_refuse"])
        if not c["should_refuse"]:
            results["content"].append(c["must_contain"] in text)
            results["citation"].append(cited == c["source"])
    for name, passes in results.items():
        print(f"{name:9s} {sum(passes)}/{len(passes)} passed")
    return results

r = run(CASES)
assert r["content"] == [True, False]   # leave-02 wrongly says leave carries forward

Running it shows content 1/2, citation 2/2, refusal 3/3. The citation check passes on the case where the answer is wrong — exactly the kind of failure a single overall score would hide. That separation is something data scientists already understand; AI engineering just makes you write it into the codebase and run it on every change.

A relearning plan in four blocks

1. 1Engineering basics: package a function as an API with tests and logging. Deploy it somewhere, even a free tier.
2. 2One LLM application: retrieval over a small synthetic corpus, with citations.
3. 3An evaluation suite like the one above, with at least twenty cases including refusals and adversarial inputs.
4. 4Operations: log requests, track cost and latency per call, and write down what you would alert on.

The Python depth checklist for AI engineering covers the engineering block in detail. The Generative & Agentic AI add-on is a three-month skills track that suits data scientists adding this layer to existing ML knowledge.