AI Grading Lab
FLYER internal · AI grading — writing / audio / video · 28.08.2026

Free AI grading costs less than one teacher's lunch. Prove it, then ship it.

Companion to the 6-month roadmap: an editable financial model (every assumption below is a live input) and the Phase-0 test kit — grading prompts, golden-set template, eval protocol, and a runnable API script — so we can validate quality and cost before writing any product code.

One essay, graded & corrected
with batch + caching
All-in monthly at current inputs
AI + infra + team
6-month program total
mostly people, not API
01

The money model

Change anything on the left — unit costs, monthly bill, and the budget below recompute instantly. Defaults: Claude Haiku 4.5 ($1 / $5 per MTok, Aug 2026), Batch API −50%, cached rubric ≈ 0.1× input price, self-hosted Whisper for speech-to-text. Your edits are saved in this browser.

Scenario

Model price ($ / MTok)

Cost levers

Batch API (−50%)
Cache rubric (0.1× input)
Video: analyze keyframes (v2)

Team & infra

Advanced token assumptions
Writing · 250-word essay
Speaking · 2-min audio
Video · 3-min clip

Monthly variable cost

Per active learner / month
Average variable cost. A learner maxing the free quota (3/day, 30 days) costs at most — the quota caps tail risk.
B2B2C absorbs it
Price “unlimited AI grading” into the school LMS package at ₫5,000/learner/mo — schools pay, learners never do. Margin shown vs average variable cost.

Variable cost stays flat while users scale

Monthly cost (USD) vs active learners, current mix & levers. Steps = added GPU servers. Hover to inspect.
02

Six-month program budget

People are ~85% of the program; the API is a rounding error. AI spend ramps with rollout (2% → 10% → 25% → 45% → 70% → 100% of the steady-state bill at your current inputs); the GPU server starts in month 3 with Speaking.

Budget by month USD, at current inputs

MonthPeopleInfraAI APITotal $₫M
03

Phase-0 test kit

Everything needed to run the validation this week: copy the prompt, grade the sample essay, compare against a teacher, repeat ×100. Nothing here requires product engineering.

T1Writing grading promptcopy → paste → test in Console or API

The system prompt below is the cached prefix — rubric, rules and band anchors stay byte-identical across calls so you pay ~0.1× for them. Only the student's essay travels per request. Fill the three {{placeholders}}, paste 2–3 teacher-graded sample essays where marked, and keep the JSON contract untouched (the app UI will render from it).

You are FLYER's English writing examiner for young learners in Vietnam.
Grade ONE student submission against the rubric below. Accurate, consistent, kind.

TARGET EXAM: {{LEVEL}}            // "Cambridge A2 Flyers" | "KET" | "IELTS Writing Task 2"
TASK PROMPT: {{TASK_PROMPT}}
STUDENT AGE BAND: {{AGE_BAND}}    // e.g. "8–11"

RUBRIC — score each criterion 0–5:
1. Content       — answers the task; ideas developed for the level
2. Organisation  — ordering, paragraphing, linking words
3. Language      — grammar + vocabulary range and accuracy for the level
4. Mechanics     — spelling, punctuation, capitalisation

BAND ANCHORS (keep byte-identical across calls so they cache):
<<PASTE 2–3 TEACHER-GRADED SAMPLE ESSAYS PER BAND HERE>>

RULES:
- Quote errors EXACTLY as written. Never invent an error.
- Max 8 errors, ranked by learning value; merge repeats of the same mistake.
- praise_vi: warm, specific, in Vietnamese, 2 sentences max — praise BEFORE correction.
- Corrections in English; each "why_vi" is ONE short Vietnamese sentence.
- Off-task, empty, not English, or unsafe content → set "flag", leave scores null.
- Output ONLY the JSON object. No markdown, no commentary.

OUTPUT JSON:
{
  "flag": null | "off_task" | "empty" | "not_english" | "safety",
  "scores": {"content":0-5, "organisation":0-5, "language":0-5, "mechanics":0-5},
  "overall_band": <number on the rubric scale>,
  "praise_vi": "…",
  "errors": [{"quote":"…","type":"grammar|vocab|spelling|punctuation|linking","fix":"…","why_vi":"…"}],
  "rewrite_examples": ["<1–2 improved sentences>"],
  "next_steps_vi": ["<2 concrete practice tips>"]
}

User message per call — nothing else:

STUDENT SUBMISSION:
{{ESSAY}}

Sample essay to test with (Flyers level, seeded with typical errors — a good grader should find the tense errors, "more better", and the missing linking words, and still praise the ideas):

My favourite holiday

Last summer I go to Da Nang with my family. We stay in a hotel near the beach.
Every morning me and my brother swimming in the sea, it was very fun. The food
is more better than home, I eat seafood everyday. One day we visit Ba Na Hills,
the bridge with two big hand is very beautiful. I take many photo there.
I want come back next year because I love this place very much.
T2Speaking grading prompttranscript + timing metrics in, timestamped feedback out

Speaking reuses the same shape. Whisper produces the transcript with word timestamps; fluency numbers are computed in code, free (words/min, pauses ≥ 1.5s, filler count) and passed in as data — the LLM interprets, it doesn't measure. Word-level pronunciation flags come from comparing the transcript against the target text (read-aloud tasks) — phoneme scoring is a paid v2 add-on, not needed for launch.

You are FLYER's English speaking examiner for young learners in Vietnam.
Grade ONE recorded answer using the transcript and measured metrics below.

TARGET EXAM: {{LEVEL}}         // "Movers Speaking" | "IELTS Speaking Part 2" …
TASK PROMPT: {{TASK_PROMPT}}

RUBRIC — score each criterion 0–5:
1. Pronunciation — intelligibility; words clearly misread/omitted (word level)
2. Fluency       — use ONLY the measured metrics; do not re-estimate from text
3. Vocabulary & Grammar — range and accuracy for the level
4. Content       — answers the task; ideas developed for the level

BAND ANCHORS: <<PASTE 2–3 GRADED SAMPLE TRANSCRIPTS PER BAND>>

RULES:
- Every error must carry the [start,end] seconds of where it was said,
  copied from the transcript timestamps (the app plays that clip back).
- ASR text may itself contain recognition mistakes: if a phrase is implausible
  for a child but grammatical, treat it as unclear pronunciation, not grammar.
- praise_vi first, Vietnamese, specific. Output ONLY the JSON.

OUTPUT JSON:
{
  "flag": null | "empty" | "not_english" | "safety",
  "scores": {"pronunciation":0-5,"fluency":0-5,"vocab_grammar":0-5,"content":0-5},
  "overall_band": <number>,
  "praise_vi": "…",
  "errors": [{"quote":"…","span_sec":[12.4,14.1],"type":"pronunciation|grammar|vocab|fluency","fix":"…","why_vi":"…"}],
  "model_answer": "<3–4 sentence answer at target band, child-appropriate>",
  "next_steps_vi": ["…","…"]
}

User message — transcript with timestamps + metrics JSON:

TRANSCRIPT (word → seconds):
{{TRANSCRIPT_WITH_TIMESTAMPS}}

MEASURED METRICS (computed, trust these):
{"duration_sec": 118, "words_per_min": 74, "pauses_over_1500ms": 6,
 "longest_pause_sec": 4.2, "filler_count": 9, "self_corrections": 3}
T3Golden-set templatethe dataset that decides ship / no-ship

Target: 300 writing + 100 speaking submissions, each graded blind by two teachers. Spread ~20% per band so the AI is tested where it's hardest (middle bands), across Flyers / KET / IELTS. Where the two teachers disagree by more than 1 band, a third adjudicates. This CSV is the whole contract — one row per submission:

id,skill,level,task_id,task_prompt,submission_ref,grader1,g1_content,g1_org,g1_lang,g1_mech,g1_overall,grader2,g2_content,g2_org,g2_lang,g2_mech,g2_overall,adjudicated_overall,notes
W-0001,writing,flyers,FT-12,"My favourite holiday",essays/W-0001.txt,Ms.Trang,4,3,3,4,3.5,Mr.Nam,4,3,4,4,3.5,3.5,
W-0002,writing,ielts_t2,IE-03,"Some people think…",essays/W-0002.txt,Ms.Trang,5,6,5,6,5.5,Mr.Nam,6,6,5,6,6.0,5.5,"graders split on TR"
S-0001,speaking,movers,MS-07,"Describe this picture",audio/S-0001.mp3,Ms.Trang,4,4,3,4,4.0,Mr.Nam,4,3,3,4,3.5,4.0,
  • Collection order: pull real anonymized submissions from the LMS first; only commission new ones for bands with gaps.
  • Blind means blind: graders never see each other's scores or the AI's.
  • Teacher–teacher agreement is the ceiling. Record it — the AI only has to approach human consistency, not exceed it.
T4Eval protocol & ship gateshow we decide the AI is good enough

Run the prompt (fixed version, logged) over the golden set, join AI scores against adjudicated teacher scores, compute four numbers. Re-run the same script monthly on a 1% random sample of live gradings to catch drift.

  • Gate 1 — agreement: ≥ 85% of submissions within ±0.5 band of the teacher score (writing); ≥ 80% for speaking.
  • Gate 2 — no disasters: zero submissions off by more than 1.5 bands.
  • Gate 3 — human ceiling: AI–teacher agreement within 5 points of teacher–teacher agreement on the same set.
  • Gate 4 — feedback quality (sampled): teachers rate 50 random feedback outputs; ≥ 90% "would show to my student as-is or with tiny edits", zero factually wrong corrections.
  • Also report: per-criterion mean absolute error (shows whether e.g. Organisation is systematically off), and cost + latency per graded submission.

If a gate fails: fix in this order — add/replace band-anchor examples (cheapest), tighten rubric wording, escalate model tier for the failing level only. Re-run the full set after every prompt change; never eyeball a few examples and call it fixed.

T5Runnable API test scriptPython · grade the sample essay end-to-end

Minimal harness: cached rubric prefix, one call per essay, JSON out. Grade the golden set with claude-haiku-4-5 (the economics above), and re-grade disagreements with claude-opus-5 to see what the calibration tier buys. On the second call, check usage.cache_read_input_tokens > 0 — if it's 0, the prefix isn't byte-stable and you're paying full price. For >100 essays switch to the Batch API (−50%, results within an hour).

# pip install anthropic   ·   export ANTHROPIC_API_KEY=…
import anthropic, json, pathlib

MODEL_GRADE = "claude-haiku-4-5"   # mass tier — the unit costs in this page
MODEL_CALIB = "claude-opus-5"      # calibration / appeals tier

RUBRIC_PROMPT = pathlib.Path("writing_rubric_prompt.txt").read_text()  # = T1, byte-stable

client = anthropic.Anthropic()

def grade(essay: str, model: str = MODEL_GRADE) -> dict:
    msg = client.messages.create(
        model=model,
        max_tokens=2000,
        system=[{
            "type": "text",
            "text": RUBRIC_PROMPT,
            "cache_control": {"type": "ephemeral"},   # rubric+anchors cached ≈0.1×
        }],
        messages=[{"role": "user", "content": f"STUDENT SUBMISSION:\n{essay}"}],
    )
    if msg.stop_reason == "refusal":                  # safety decline → route to teacher
        return {"flag": "safety", "scores": None}
    raw = msg.content[-1].text
    out = json.loads(raw[raw.index("{"): raw.rindex("}") + 1])
    out["_usage"] = {"in": msg.usage.input_tokens,
                     "cached": msg.usage.cache_read_input_tokens,
                     "out": msg.usage.output_tokens}
    return out

if __name__ == "__main__":
    essay = pathlib.Path("sample_essay.txt").read_text()
    print(json.dumps(grade(essay), ensure_ascii=False, indent=2))
    print(json.dumps(grade(essay)["_usage"], indent=2))  # cached > 0 on this 2nd call

Production hardening (later, not for the pilot): validate against a JSON schema and retry once on mismatch; log prompt_version, model, tokens, cost per grading; content-hash dedupe so a resubmitted identical essay returns the stored result.

T6Pilot design — 2 weeks, 3 classeswhat we measure with real teachers
  • Who: 2–3 classes at one partner school + a handful of B2C learners; teachers keep grading normally (AI runs in shadow the first week, visible the second).
  • Teacher metrics: minutes saved per submission (self-reported + timed sample); NPS ≥ 8/10; % of AI gradings edited before showing students.
  • Learner metrics: % who resubmit after feedback (the learning loop — the real goal); time from submit to feedback (< 3 min sync, < 60 min batch).
  • Cost metrics: actual $ per graded submission vs this model's prediction; cache hit rate; % needing the calibration tier.
  • Exit decision: all four T4 gates green + teacher NPS met → build Phase-1 product UI. Any red → another prompt iteration, not more features.
04

Cost governance & safety rails

Set these up in week 1 — they're what makes “free for every learner” safe to promise.

Spending caps

  • Daily billing alert + hard kill-switch env flag (grading pauses, submissions queue).
  • Free quota per learner per day (worst-case cost per abuser shown in the model above).
  • Dedupe: identical resubmission → cached result, no API call.
  • Weekly dashboard: cost / submission, cache hit rate, batch share.

Child data

  • No PII in prompts — student ID only, never name/school/age beyond age-band.
  • Audio/video: parental consent on enable, visible to the class teacher only, auto-delete after 90 days.
  • Moderation pass on submissions and on generated feedback before display.

Trust

  • Every grading labeled “AI-graded”; one-tap teacher override, edits logged for calibration.
  • Appeal path: re-grade with the calibration tier; two results > 1 band apart → human.
  • Suspected AI-written homework → flag quietly to the teacher, never auto-punish.

Vendor risk

  • All LLM calls behind one internal module — model/price changes are a config edit.
  • ASR self-hosted (open-source Whisper) → immune to per-minute pricing.
  • Re-check this page's price assumptions quarterly; the model above makes that a 2-minute exercise.