Content writer to GenAI: turn your style guide into an eval rubric

The documents on your desk that already do evaluation work

You have been running evaluations for years without calling them that. Every editor's note on a draft is a label. Every line in a brand style guide is a rule someone expected you to follow consistently. Before you learn anything new, collect these:

• The style guide or brand voice document — banned words, sentence-length preferences, heading conventions, how the brand writes numbers and dates.
• Content briefs — target reader, primary keyword, required sections, word range, call to action. A brief is a specification, and a specification is what an evaluation checks against.
• Tracked-changes history — your edits on other people's drafts, or your editor's edits on yours. Each accepted change is a before/after pair with an implicit reason.
• SEO checklists — meta description length, heading order, internal link count.

Keep the originals private. For the portfolio, rewrite the guide as a fictional brand so nothing confidential leaves your employer.

Checklist: turning each style-guide line into a rubric item

Work through your style guide line by line and sort every rule with this checklist. The point is to separate what a script can check from what still needs a human or a second model.

1. 1Is the rule countable? "Sentences under 25 words", "no more than one exclamation mark" — a script checks these exactly. Mark them automatic.
2. 2Is it a closed list? Banned words, required disclaimers, the product's exact spelling. Also automatic, with a word list.
3. 3Does it depend on meaning? "Don't overpromise results", "match the reader's level". Mark these judgement. They need a human grader or a carefully calibrated model judge.
4. 4Can you write a failing example? If you cannot write one sentence that clearly breaks the rule, the rule is too vague to grade. Rewrite it until you can.
5. 5Does it come from the brief rather than the guide? Word range and required sections change per piece. Store those as per-prompt fields, not global rules.
6. 6What is the severity? A banned competitor mention is a hard fail; a slightly long sentence is a minor deduction. Assign pass/fail or a weight.
7. 7Who decided this rule? Note whether it came from legal, brand or SEO. In an interview, knowing why a rule exists matters more than the rule.

Most writers find that roughly a third of their guide is automatic and the rest is judgement. That ratio is itself a useful finding to report.

The worked example: scoring outputs against automatic rules

Here is a minimal version on synthetic outputs for a fictional brand. It checks banned words, sentence length and whether a required call to action appears.

import re

BANNED = {"revolutionary", "unbeatable", "best-in-class"}
MAX_SENTENCE_WORDS = 25
REQUIRED_CTA = "book a free consultation"

# Synthetic model outputs for three prompts
outputs = {
    "p01": "Our planners are revolutionary. Book a free consultation today.",
    "p02": ("This checklist helps first-time buyers compare home loans by "
            "tenure, processing charges, prepayment terms and the documents "
            "each lender asks for before it will sanction and disburse your "
            "loan. Book a free consultation."),
    "p03": "Compare three savings options side by side before you decide.",
}

def check(text):
    issues = []
    words = set(re.findall(r"[a-z\-]+", text.lower()))
    hits = sorted(BANNED & words)
    if hits:
        issues.append("banned: " + ", ".join(hits))
    for s in re.split(r"(?<=[.!?])\s+", text.strip()):
        n = len(s.split())
        if n > MAX_SENTENCE_WORDS:
            issues.append(f"long sentence ({n} words)")
    if REQUIRED_CTA not in text.lower():
        issues.append("missing CTA")
    return issues

for pid, text in outputs.items():
    result = check(text)
    print(pid, "PASS" if not result else "FAIL", result)

Running it prints:

p01 FAIL ['banned: revolutionary']
p02 FAIL ['long sentence (29 words)']
p03 FAIL ['missing CTA']

Three prompts, three different failure types. Your real project needs twenty or more prompts, and it needs a second column: your own manual grade for each judgement rule.

Where the automatic checks disagree with you

The valuable part of the project is not the script. It is the table where you compare the script's verdict and your own reading. Expect cases like these:

Write two or three paragraphs explaining each disagreement and what you would change — loosen the length rule for list sentences, add a judgement rule for implied promises. This is the reasoning an evaluation role needs.

The gap you actually have to close

Writers rarely lack judgement. The gaps are usually these three:

• Python basics for data handling — reading a CSV of outputs, looping, writing results back. Not algorithms.
• Measuring agreement — when two graders (you and a model judge, or you and a colleague) label the same outputs, you need to report how often they agree rather than trusting either one.
• How model behaviour changes — why the same prompt gives different outputs, what temperature does, why retrieval changes answers. The Generative & Agentic AI add-on covers these alongside retrieval and agent workflows.

Evaluation and content-quality titles to aim for

Search listings for titles such as AI content specialist, prompt and evaluation specialist, or content quality analyst (AI). These sit closest to what you already do: owning output quality. Titles with "engineer" in them usually expect production software skills you will not have yet. Treat the first role as the bridge, and read each job description for whether it mentions evaluation, rubrics or annotation guidelines.

Pitching the move from bylines to rubrics

For five years my job was making sure every piece matched a style guide and a brief. When our team started drafting with a language model, I noticed the edits I was making were the same few every time, so I wrote them down as rules. About a third turned out to be checkable by a script, and I built that script. The rest needed judgement, so I graded those by hand and compared my grades with the script's. The interesting results were where we disagreed — for example, the script passed outputs that implied a result the product can't promise. I want a role where writing those rules, and finding where they break, is the main job.

Notice what that answer does: it names a real artefact, a number you can defend, and a failure you found. Before an application goes out, run your CV through the free skill check to see which evaluation and Python terms it currently fails to show.