Run the Notebook
1. Install dependencies
The first cell installssynth-ai, datasets, and cloudflared.
2. Configure API keys
The setup cell auto-generates a Synth AI demo API key. Optionally setOPENAI_API_KEY in Colab secrets to enable the baseline vs optimized preview (Step 4).
3. Define the classifier
The notebook loads the Banking77 dataset and defines a classification function usinggpt-4o-mini with function calling.
4. Preview baseline vs optimized
Compares a simple baseline prompt (~78% accuracy) against an optimized prompt (~92% accuracy) on 50 test samples.5. Start the Local API
Launches a FastAPI server with a Cloudflare tunnel so Synth AI can reach the notebook.6. Run GEPA optimization
Submits a prompt learning job that evolves prompts over 3 generations, evaluating each on 50 samples.7. Evaluate on held-out data
Runs a formal eval comparing baseline vs optimized prompts on 50 held-out samples (indices 100-149).8. Cleanup
Terminates the Cloudflare tunnel.Results
- Baseline accuracy: ~78%
- Optimized accuracy: ~92%
- Improvement: +14%