Skip to main content

​
Zero-Shot Verifiers

Zero-shot verifiers allow you to use Synth’s advanced evaluation logic (including MapReduce and RLM architectures) immediately, without providing a training dataset. These are ideal for rapid prototyping of custom judges.

​
Built-in Verifier IDs

When calling the /graphs/completions endpoint, you can use these reserved IDs to route to built-in verifier graphs:
Graph IDBest ForMax Tokens
zero_shot_verifierAutomatic routing based on trace sizeVaries
zero_shot_verifier_rubric_singleStandard traces (1-2 messages)50,000
zero_shot_verifier_rubric_mapreduceLong traces (10+ messages)500,000
zero_shot_verifier_rubric_rlmMassive contexts / tool-heavy traces1,000,000+

​
Request Structure

Zero-shot verifiers accept a session_trace and an optional rubric. 
curl -X POST https://api.synth-ai.com/graphs/completions \
  -H "Authorization: Bearer $SYNTH_API_KEY" \
  -d '{
    "job_id": "zero_shot_verifier",
    "session_trace": {
      "session_id": "test_session_123",
      "session_time_steps": [...]
    },
    "rubric": {
      "outcome": {
        "criteria": [
          {"name": "Accuracy", "description": "Is it correct?", "weight": 1.0}
        ]
      }
    }
  }'

​
Few-Shot In-Context Learning

You can improve zero-shot performance by providing calibration_examples directly in the inference request: 
{
  "job_id": "zero_shot_verifier",
  "session_trace": { ... },
  "calibration_examples": [
    {
      "input": { "trace": { ... } },
      "output": { "score": 1.0, "reasoning": "Excellent response" }
    }
  ]
}