Phoenix Evals
Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.
Quick Reference
Task
Files
Setup
setup-python , setup-typescript
Decide what to evaluate
evaluators-overview
Choose a judge model
fundamentals-model-selection
Use pre-built evaluators
evaluators-pre-built
Build code evaluator
evaluators-code-python , evaluators-code-typescript
Build LLM evaluator
evaluators-llm-python , evaluators-llm-typescript , evaluators-custom-templates
Batch evaluate DataFrame
evaluate-dataframe-python
Run experiment
experiments-running-python , experiments-running-typescript
Create dataset
experiments-datasets-python , experiments-datasets-typescript
Generate synthetic data
experiments-synthetic-python , experiments-synthetic-typescript
Validate evaluator accuracy
validation , validation-evaluators-python , validation-evaluators-typescript
Sample traces for review
observe-sampling-python , observe-sampling-typescript
Analyze errors
error-analysis , error-analysis-multi-turn , axial-coding
RAG evals
evaluators-rag
Avoid common mistakes
common-mistakes-python , fundamentals-anti-patterns
Production
production-overview , production-guardrails , production-continuous
Workflows
Starting Fresh:
observe-tracing-setup → error-analysis → axial-coding → evaluators-overview
Building Evaluator:
fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}
RAG Systems:
evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)
Production:
production-overview → production-guardrails → production-continuous
Reference Categories
Prefix
Description
fundamentals-*Types, scores, anti-patterns
observe-*Tracing, sampling
error-analysis-*Finding failures
axial-coding-*Categorizing failures
evaluators-*Code, LLM, RAG evaluators
experiments-*Datasets, running experiments
validation-*Validating evaluator accuracy against human labels
production-*CI/CD, monitoring
Key Principles
Principle
Action
Error analysis first
Can't automate what you haven't observed
Custom > generic
Build from your failures
Code first
Deterministic before LLM
Validate judges
>80% TPR/TNR
Binary > Likert
Pass/fail, not 1-5
Summary Phoenix Evals is a comprehensive reference guide for building evaluators for AI/LLM applications using the Arize Phoenix platform. It covers code-based evaluators, LLM-as-judge evaluators, experiments, datasets, error analysis, production monitoring, and validation workflows. The skill is a structured reference library with 34 markdown files organized by task and workflow, providing Python and TypeScript examples throughout.
Detected Capabilities documentation reading code pattern examples python and typescript reference api guidance workflow design best practices documentation
Trigger Keywords Phrases that MCP clients use to match this skill to user intent.
build evaluators llm as judge evaluate outputs error analysis validation experiments production monitoring evaluate dataframe
Referenced Domains External domains referenced in skill content, detected by static analysis.
app.phoenix.arize.com
Use Cases Build code-based evaluators for deterministic output validation Create LLM-as-judge evaluators for subjective quality criteria Run systematic experiments across datasets with multiple evaluators Perform error analysis on production traces to discover failure modes Validate evaluator accuracy against human-labeled golden datasets Monitor production systems with continuous evaluation and alerting Create and manage evaluation datasets from production traces Generate synthetic test data for edge case coverage Quality Notes Comprehensive reference with 34 well-organized files covering the full evaluator lifecycle Clear separation between Python and TypeScript implementations with parallel examples Strong emphasis on best practices and anti-patterns with specific gotchas for common mistakes Well-structured quick-start patterns for code evaluators, LLM evaluators, and experiments Workflows provided for multiple scenarios (starting fresh, building evaluator, RAG systems, production) Extensive use of tables and code examples for clarity Key principles section establishes philosophy (error analysis first, code before LLM, validate judges) Guidance on validation metrics (TPR/TNR targets >80%) and golden dataset construction Production guidance distinguishes guardrails (blocking) from evaluators (measuring) Common mistakes section directly addresses legacy API usage and migration issues Good coverage of edge cases and error handling patterns Model: claude-haiku-4-5-20251001 Analyzed: Jun 26, 2026
Reviews Add this skill to your library to leave a review.
No reviews yet
Be the first to share your experience.