Prompt Engineering & Structured Output

4.1 Explicit criteria beat vague guidance

The exam rewards prompts that state categorical, checkable criteriaover vague adjectives. "Review this code for quality issues" and "check this document for problems" sound reasonable, but they hand the model an ill-defined bar — every response invents its own definition of "quality" or "problem," and no two runs grade the same way.

Compare the two versions of a code-review prompt used at scale (500 PRs a week, every PR reviewed the same way):

The distinction matters most exactly where the exam likes to test it: a prompt running at scale. Reviewing 1 PR, a vague prompt might work fine because a human reads the output and fills the gaps. Reviewing 500 PRs a week with no human reading every response, vague instructions produce inconsistent, unverifiable output — while explicit criteria produce reproducible, gradeable output that a downstream system (or a human doing spot-checks) can actually act on.

4.2 Few-shot examples: show what's hard to say

Some tasks have a desired output format or edge-case handling that is hard to state in words but easy to show. When that is true, 2-4 input/output example pairs placed in the prompt before the real input anchor the model's behavior far more reliably than a longer written description alone.

Support-ticket classification: description-only vs few-shot

A prompt that lists categories in prose — "classify as billing, technical, or account" — under-specifies exactly the tickets that matter most: the ambiguous ones.

# Description-only (under-specifies edge cases)
"Classify the following support ticket as one of: billing,
technical, or account. Respond with only the category name."

Ticket: "My payment method got reset when I changed my email."
-> unclear which category this should be; the model guesses

# Few-shot (anchors the exact edge-case handling)
Ticket: "I was charged twice for my subscription this month."
Category: billing

Ticket: "The app crashes every time I try to upload a photo."
Category: technical

Ticket: "My payment method got reset when I changed my email
address, and now I can't tell if my subscription is still active."
Category: account
# (edge case: touches billing AND technical, but the root cause
# is an account-settings change - the example teaches the model
# to classify by ROOT CAUSE, not by which keyword appears)

Ticket: "{{real_ticket}}"
Category:

The third example pair is doing the real work: it is deliberately an ambiguous ticket, and it shows the model exactly how to resolve the ambiguity (classify by root cause, not by surface keyword). No amount of additional prose describing the "account" category would teach that resolution rule as reliably as one worked example.

4.3 tool_use for guaranteed structure

Asking Claude to "please respond in JSON" in plain text is not reliable structured output — it is a request the model interprets like any other instruction, and nothing on the API side validates the response against a shape. The response can come back with extra prose around the JSON, a dropped comma, or schema drift on a field name.

Giving Claude a tool definition with a JSON input_schema— even when the "tool" isn't really an external action, just a way to force schema-shaped output — is far more reliable than asking for JSON in plain text. One precision worth knowing for the exam: plain tool_use alone is very reliable but does not strictly guarantee conformance — the model can still infer a value for a missing required field. Adding strict: true to the tool definition (Strict tool use) is what turns that into a real guarantee, validating the call arguments against the schema before you ever see them.

Tool schema for invoice-field extraction

{
  "name": "record_invoice_fields",
  "description": "Record the extracted fields from a vendor invoice.",
  "input_schema": {
    "type": "object",
    "properties": {
      "invoice_number": { "type": "string" },
      "vendor_name":    { "type": "string" },
      "amount":         { "type": "number" },
      "due_date":       { "type": "string", "format": "date" },
      "po_number":      { "type": ["string", "null"] }
    },
    "required": ["invoice_number", "vendor_name", "amount", "due_date", "po_number"]
  }
}

Claude responds with a tool_use block whose input already matches this shape — no free text to strip, no brackets to hunt for. Your code reads tool_use.input.invoice_number directly.

4.4 Nullable fields prevent hallucination

A schema-shaped answer solves one problem and creates another if you are not careful: if a schema marks a field as required but the source document simply does not contain that data, the model will often invent a plausible-looking value to satisfy the schema — for example, fabricating a PO number that looks right but does not exist on the invoice.

The schema-level fix has two parts, and both matter:

1. Mark optional fields nullable in the schema itself — "type": ["string", "null"] — so the model has a valid, schema-conformant way to say "this isn't here" instead of being forced to produce something.
2. Explicitly instruct the null path— "If the PO number is not printed on the invoice, return null. Do not guess or infer a value." — and validate downstream: flag or reject any record where a nullable field is unexpectedly populated with a low-confidence-looking value, or route null-heavy records for human review.

This is a direct continuation of the structured-output problem in 4.3: getting a schema-shaped answer is necessary but not sufficient — the schema also has to make the true absence of data representable, or the model will paper over the gap.

4.5 Validation-retry loops and multi-pass review

Once you have structured output, the production pattern is to run it through a programmatic validator— a schema check, plus business-rule checks like "due_date must be after invoice_date." If validation fails, the architecture retries the extraction call with the validation error appended to the prompt, so the model can see exactly what was wrong and self-correct — up to a bounded retry count.

That bound is not optional. It is the same loop-bounding principle from Domain 1: an agentic loop that is allowed to retry forever is a production incident waiting to happen, whether the loop is calling tools or re-attempting a structured extraction. A typical bound is 3 attempts, after which the record is flagged for human review — never silently dropped, and never retried indefinitely.

def extract_with_validation(document, max_retries=3):
    error_context = ""
    for attempt in range(max_retries):
        result = extract(document, prior_error=error_context)
        errors = validate(result)          # schema check + business rules
        if not errors:
            return result                  # accept
        error_context = f"Previous attempt failed: {errors}. Fix these issues."
    flag_for_human_review(document, result, errors)  # bounded - never loop forever
    return None

Multi-pass review: a cheap draft, a stronger check

A related production pattern is multi-pass review: a cheap or fast model pass drafts the extraction or output, then a second pass — the same or a stronger model, reviewing against explicit criteria — checks the draft. This catches errors a single pass misses, at a cost premium that is worth paying for high-stakes output (financial figures, medical data, anything that feeds a downstream decision without further human review).

4.6 Message Batches API — when async wins

The Message Batches API lets you submit many independent requests together and get results back within a 24-hour processing window, at roughly 50% lower cost than real-time calls. It is built for large, deadline-tolerant, non-interactive workloads — the textbook case is a nightly extraction job over 20,000 archived documents, where nothing downstream is blocked on the job finishing before the next morning.

This maps directly onto Domain 1's anti-pattern #3: using the Batches API for a blocking, user-facing, real-time flow is wrongbecause it has no real-time SLA — a user waiting on a live chat response cannot wait up to 24 hours for the reply. The mistake runs in both directions and the exam tests both: routing a live conversation through Batches is the classic wrong answer, and looping the real-time API over 20,000 documents just because "it's simpler to code" throws away a real cost saving for no architectural benefit.