Skip to main content
Rubrics define how traces are scored by hosted verifiers. Each rubric contains criteria with weights that determine the final reward.

​
Structure

from synth_ai.data.rubrics import Rubric, Criterion

rubric = Rubric(
    version="1.0",
    goal_text="Classify banking intents accurately",
    criteria=[
        Criterion(id="accuracy", description="Correct intent label", weight=1.0, required=True),
        Criterion(id="confidence", description="High confidence score", weight=0.5),
    ],
    aggregation="weighted_sum",
)
FieldTypeDescription
versionstrRubric version identifier
goal_textstrHuman-readable goal description
criterialist[Criterion]Scoring criteria
aggregationstrHow to combine scores: sum, weighted_sum, custom, inherit

​
Criterion

FieldTypeDescription
idstrUnique criterion identifier
descriptionstrWhat the verifier evaluates
weightfloatScore multiplier (default: 1.0)
requiredboolIf true, failure on this criterion fails the whole rubric

​
Using with Task Apps

Pass rubrics to your task app config: 
from synth_ai.sdk import TaskAppConfig
from synth_ai.data.rubrics import Rubric, Criterion

outcome_rubric = Rubric(
    version="1.0",
    goal_text="Classify banking intents accurately",
    criteria=[
        Criterion(id="accuracy", description="Correct intent label", weight=1.0, required=True),
    ],
    aggregation="weighted_sum",
)

config = TaskAppConfig(
    name="banking77",
    # ... other config
)
The task app’s /info endpoint exposes these rubrics to the backend.

​
Rubric Sources

Hosted verifiers can fetch rubrics from:
  1. Task app /info endpoint — Rubrics bundled in TaskAppConfig
  2. Task app /task_info?seed=N — Seed-specific rubrics
  3. Backend synth_verifier_id — Pre-registered rubrics or VerifierGraphs in Synth AI’s backend
Configure in your TOML: 
[verifier]
enabled = true
reward_source = "verifier"
synth_verifier_id = "banking77-v1"  # Use backend rubric or graph

​
JSON Example

{
  "version": "1.0",
  "goal_text": "Classify customer banking queries into correct intent categories",
  "criteria": [
    {
      "id": "correct_intent",
      "description": "The predicted intent matches the ground truth label",
      "weight": 1.0,
      "required": true
    },
    {
      "id": "reasoning_quality",
      "description": "Clear reasoning before the final answer",
      "weight": 0.3,
      "required": false
    }
  ],
  "aggregation": "weighted_sum"
}