Skip to main content
This cookbook shows how to train a policy graph for intent classification using the Banking77 dataset - 77 unique banking-related customer intents.

​
The Task

Classify customer messages into one of 77 banking intents: 
Input: "When will my card arrive?"
Output: "card_arrival"

Input: "My card payment was declined."
Output: "declined_card_payment"

​
Dataset Format

The Banking77 Graph Evolve dataset follows this structure: 
{
  "version": "1.0",
  "metadata": {
    "name": "banking77-classification",
    "description": "Intent classification for banking customer queries",
    "task_description": "Classify the customer message into one of 77 banking intents"
  },
  "tasks": [
    {
      "id": "banking_001",
      "input": {
        "customer_message": "When will my card arrive?"
      }
    },
    {
      "id": "banking_002",
      "input": {
        "customer_message": "My card payment was declined."
      }
    },
    {
      "id": "banking_003",
      "input": {
        "customer_message": "I need to change my PIN"
      }
    }
  ],
  "gold_outputs": [
    {
      "task_id": "banking_001",
      "output": {
        "intent": "card_arrival"
      }
    },
    {
      "task_id": "banking_002",
      "output": {
        "intent": "declined_card_payment"
      }
    },
    {
      "task_id": "banking_003",
      "output": {
        "intent": "change_pin"
      }
    }
  ],
  "verifier_config": {
    "mode": "rubric"
  }
}

​
Training the Graph

import os
from synth_ai.sdk import GraphOptimizationJob

job = GraphOptimizationJob.from_dataset(
    dataset="banking77_dataset.json",
    policy_model="gpt-4o-mini",
    rollout_budget=200,
    api_key=os.environ["SYNTH_API_KEY"],
)

result = job.submit()
print(f"Job ID: {result.graphgen_job_id}")

# Stream progress
final_status = job.stream_until_complete(timeout=3600.0)
print(f"Best accuracy: {final_status.best_score:.2%}")

​
Available Intent Labels

The 77 Banking77 intents include:
CategoryExample Intents
Cardcard_arrival, card_linking, card_not_working, card_swallowed, declined_card_payment
Accountactivate_my_card, age_limit, apple_pay_or_google_pay, atm_support
Transferbalance_not_updated_after_bank_transfer, beneficiary_not_allowed, cancel_transfer
PINchange_pin, compromised_card, contactless_not_working
…77 total intents

​
Running Inference

Once trained, classify new messages: 
job = GraphOptimizationJob.from_existing("graphgen_abc123", api_key=api_key)

result = job.run_inference({
    "customer_message": "How do I add money to my account?"
})

print(result["output"]["intent"])  # "top_up"

​
Batch Classification

import asyncio

messages = [
    {"customer_message": "Cancel my card please"},
    {"customer_message": "Where is my refund?"},
    {"customer_message": "I lost my card"},
]

results = asyncio.run(job.run_batch(messages))
for msg, res in zip(messages, results):
    print(f"{msg['customer_message'][:30]}... → {res['output']['intent']}")

​
Evaluation Metrics

Banking77 uses exact match scoring:
  • Score = 1.0 if predicted intent exactly matches gold intent
  • Score = 0.0 otherwise
The verifier normalizes intent names to snake_case before comparison.

​
Configuration Tips

​
For Higher Accuracy

job = GraphOptimizationJob.from_dataset(
    dataset="banking77_dataset.json",
    policy_model="gpt-4o",       # Stronger model
    rollout_budget=500,          # More optimization budget
    target_llm_calls=3,          # Allow chain-of-thought
)

​
For Lower Cost

job = GraphOptimizationJob.from_dataset(
    dataset="banking77_dataset.json",
    policy_model="gpt-4o-mini",
    rollout_budget=100,
    target_llm_calls=1,          # Single-pass classification
)

​
Downloading the Graph

Export your trained graph for inspection or local use: 
graph_yaml = job.download_prompt()
print(graph_yaml)
Example output: 
graph_type: policy
nodes:
  - id: classifier
    type: llm
    model: gpt-4o-mini

​
Ready to get started?

Get Started

Sign up and start optimizing your prompts today.

Schedule Demo

See Synth in action with a personalized walkthrough.