Skip to main content
Smoke-tests a Task App by emulating a trainer rollout using GPT-5-Nano. This command posts a minimal RL rollout to the task app and validates that the response contains the fields required by the RL trainer.

​
Usage

uvx synth-ai smoke
The smoke command is designed for rapid iteration on your task app. It runs a lightweight rollout to verify:
  • Task app is reachable and responding
  • Rollout endpoints return valid data
  • Inference URL routing works correctly
  • Trace correlation IDs are properly propagated
You can add a [smoke] section to your RL config TOML file to configure smoke testing. This enables auto-start features and sets test defaults, making smoke tests as simple as: 
uvx synth-ai smoke --config my-rl-config.toml
Important: The [smoke] section is only used by the smoke CLI command and is completely ignored by the RL trainer. It will not affect your training jobs.

​
Auto-Start Features

The [smoke] section can automatically start required services in the background:
  • ✅ Task App Server: Auto-starts your task app with proper port management
  • ✅ sqld Server: Auto-starts sqld for tracing with configurable ports
  • ✅ Automatic Cleanup: Stops all background services when smoke test completes

​
Full Configuration Example

# my-rl-config.toml
type = "rl"

[smoke]
# === AUTO-START: Task App ===
task_app_name = "grpo-crafter"  # Name from 'synth-ai task-app list'
task_app_port = 8765
task_app_env_file = ".env"  # Optional: path to .env file
task_app_force = true  # Kill existing process on this port

# === AUTO-START: sqld (for tracing) ===
sqld_auto_start = true
sqld_db_path = "./traces/local.db"
sqld_hrana_port = 8080  # Hrana WebSocket port
sqld_http_port = 8081  # HTTP API port

# === TEST PARAMETERS ===
env_name = "crafter"
policy_name = "crafter-react"
max_steps = 10
policy = "gpt-5-nano"  # mock, gpt-5-nano, openai, groq
model = "gpt-5-nano"
mock_backend = "synthetic"  # synthetic (no API calls) or openai (requires API key)
mock_port = 0  # 0 = auto-assign
return_trace = true
use_mock = true

# === REST OF RL CONFIG (ignored by smoke command) ===
[algorithm]
type = "online"
method = "policy_gradient"
variety = "gspo"

[policy]
model_name = "Qwen/Qwen3-4B"
trainer_mode = "full"
# ... rest of your RL config ...

​
Quick Start Examples

1. Full Auto (Zero Setup) 
# Automatically starts task app + sqld, runs test, cleans up
uvx synth-ai smoke --config my-rl-config.toml
2. Manual Task App + Auto sqld 
# Config without task_app_name (only auto-starts sqld)
[smoke]
sqld_auto_start = true
# ... test parameters ...
3. Override Any Setting 
# Override max_steps from config
uvx synth-ai smoke --config my-rl-config.toml --max-steps 5
CLI arguments always override values from the [smoke] section.

​
[smoke] Section Reference

All fields in the [smoke] section are optional. CLI arguments override TOML values.

​
Auto-Start: Task App

Configure automatic task app startup:
FieldTypeDefaultDescription
task_app_namestrNoneTask app name (from synth-ai task-app list)
task_app_portint8765Port to run task app on
task_app_env_filestrNonePath to .env file (e.g., “.env”)
task_app_forceboolfalseKill existing process on port before starting
Example: 
[smoke]
task_app_name = "grpo-crafter"
task_app_port = 8765
task_app_env_file = ".env"
task_app_force = true

​
Auto-Start: sqld (Tracing)

Configure automatic sqld server startup:
FieldTypeDefaultDescription
sqld_auto_startboolfalseAuto-start sqld server
sqld_db_pathstr”./traces/local.db”Database file path
sqld_hrana_portint8080Hrana WebSocket port
sqld_http_portint8081HTTP API port
Example: 
[smoke]
sqld_auto_start = true
sqld_db_path = "./traces/local.db"
sqld_hrana_port = 8080
sqld_http_port = 8081

​
Test Parameters

Configure smoke test behavior:
FieldTypeDefaultDescription
task_urlstrNoneTask app URL (overridden by auto-start)
env_namestrNoneEnvironment name
policy_namestr”react”Policy name
max_stepsint3Number of agent/env step pairs
policystrNoneInference preset: mock, gpt-5-nano, openai, groq
modelstr”gpt-5-nano”Model ID for inference
mock_backendstr”synthetic”Mock backend: synthetic or openai
mock_portint0Mock server port (0 = auto)
return_traceboolfalseRequest v3 trace in response
use_mockbooltrueUse local mock inference server
Example: 
[smoke]
env_name = "crafter"
policy_name = "crafter-react"
max_steps = 10
policy = "gpt-5-nano"
model = "gpt-5-nano"
mock_backend = "synthetic"
return_trace = true

​
CLI Options

​
Core Options

  • --url URL - Task app base URL (default: $TASK_APP_URL or http://localhost:8765)
  • --api-key KEY - Environment API key for X-API-Key header (default: $ENVIRONMENT_API_KEY)
  • --env-name NAME - Environment name to roll out (auto-detected if possible)
  • --policy-name NAME - Policy name to pass to task app (default: react)
  • --model MODEL - Model ID to route in inference payload (default: gpt-5-nano)
  • --max-steps N - Number of agent/env step pairs (default: 3)

​
Inference Options

  • --policy PRESET - Inference route preset: mock, gpt-5-nano, openai, or groq
  • --inference-url URL - Override inference URL (default: Synth API chat completions)

​
Mock Server Options

  • --use-mock / --no-mock - Use local mock inference server (default: true)
  • --mock-backend BACKEND - Mock backend: synthetic (deterministic) or openai (passthrough) (default: synthetic)
  • --mock-port PORT - Port for local mock inference server, 0 = auto-assign (default: 0)

​
Rollout Options

  • --rollouts N - Number of rollouts, using seeds 0..N-1 (default: 1)
  • --batch-size N - Alias for --rollouts
  • --group-size N - Completions per seed to emulate GRPO grouping (default: 1)
  • --parallel N - Run this many rollouts concurrently to emulate a train step (0 = sequential, default: 0)

​
Configuration Options

  • --config PATH - RL TOML config to derive URL/env/model and load [smoke] section
  • --env-file PATH - Path to .env file to load before running
  • --return-trace - Request v3 trace in response if supported

​
Examples

​
Basic smoke test

uvx synth-ai smoke --url http://localhost:8001 --env-name crafter

​
Using a config file

uvx synth-ai smoke --config configs/rl.toml

​
Test with multiple rollouts in parallel

uvx synth-ai smoke --config configs/rl.toml --rollouts 4 --parallel 4

​
Use real OpenAI instead of mock

uvx synth-ai smoke --config configs/rl.toml --mock-backend openai

​
Override config settings via CLI

# Config has max_steps=10, override to 5 for faster testing
uvx synth-ai smoke --config configs/rl.toml --max-steps 5

​
Smoke Section Reference

All fields in the [smoke] section are optional. CLI arguments take precedence over TOML values.
FieldTypeDescriptionDefault
task_urlstringTask app base URL$TASK_APP_URL or http://localhost:8765
env_namestringEnvironment nameAuto-detected
policy_namestringPolicy namereact
max_stepsintNumber of steps3
policystringInference preset: mock, gpt-5-nano, openai, groqNone
modelstringModel IDgpt-5-nano
mock_backendstringMock backend: synthetic or openaisynthetic
mock_portintMock server port (0 = auto)0
return_traceboolRequest v3 tracefalse
use_mockboolUse local mock servertrue

​
Notes

  • The smoke command automatically starts a local libSQL/sqld instance for tracing if needed
  • When use_mock=true, a local FastAPI mock server emulates GPT-5-Nano
  • In synthetic mode, the mock returns deterministic tool calls for reproducible testing
  • In openai mode, the mock proxies requests to OpenAI (requires $OPENAI_API_KEY)
  • The [smoke] section is validated by Pydantic but has no effect on training jobs