--- name: eval-framework description: Professional LLM-as-Judge evaluation framework for deterministic, high-correctness automated assessment. Used by autoresearch loops to measure skill quality. triggers: - eval - evaluate - assessment - quality check - autoresearch eval - binary eval - golden example --- # Professional Eval Framework Deterministic evaluation system using LLM-as-Judge with binary questions, multiple runs, and ground truth comparison. ## 5 Core Principles ### 1. Binary Questions Only Never ask for scores (1-10). Always yes/no. ``` BAD: "Rate the quality of this response from 1-10" GOOD: "Is this response written in the correct language? YES or NO" ``` Binary questions produce consistent results across runs. Numeric scores drift. ### 2. Multiple Runs with Majority Voting Run each binary question N times (default: 5). Take majority vote. ``` Run 1: YES Run 2: YES Run 3: NO Run 4: YES Run 5: YES Result: YES (4/5 majority) ``` This filters random noise and raises accuracy to 90%+. ### 3. Dimension Decomposition Break "is it good?" into specific, measurable dimensions: ``` Instead of: "Is this a good response?" Use: - Is the response in the correct language? - Does it answer the user's question? - Is the tone natural (not robotic)? - Is it free of AI vocabulary (delve, crucial, vibrant, etc.)? - Is it concise (no unnecessary filler)? ``` Each dimension is a separate binary eval. Final score = dimensions passed / total dimensions. ### 4. Ground Truth Comparison Store known-good outputs as "golden examples" in `golden/` directory. ``` Eval question: "Is the NEW output equal to or better than the GOLDEN example? YES or NO" ``` This turns subjective quality into objective comparison against a verified baseline. ### 5. Strong Model as Judge Evals ALWAYS run on the strongest available model (Claude Opus), regardless of what model the skill under test uses. | Skill Model | Judge Model | |-------------|-------------| | Haiku | Opus | | Sonnet | Opus | | Gemini | Opus | | Any | Always Opus | The judge must be smarter than the defendant. ## Eval Suite Structure ``` eval-suite/ ├── dimensions.json # List of binary questions per skill ├── golden/ # Golden example outputs (ground truth) │ ├── tts_hebrew_01.json │ ├── voice_natural_01.json │ └── ... ├── results/ # Historical eval results │ └── 2026-03-14_tts.json └── runner.sh # Eval execution script ``` ### dimensions.json Format ```json { "skill": "text-to-speech", "runs_per_question": 5, "judge_model": "claude-opus-4-6", "dimensions": [ { "id": "correct_language", "question": "Is the audio output in the same language as the input text? Answer only YES or NO.", "weight": 1 }, { "id": "natural_tone", "question": "Does the speech sound natural and human-like, not robotic? Answer only YES or NO.", "weight": 1 }, { "id": "no_artifacts", "question": "Is the audio free of cuts, glitches, or distortion? Answer only YES or NO.", "weight": 1 }, { "id": "correct_pace", "question": "Is the speaking pace appropriate - not too fast, not too slow? Answer only YES or NO.", "weight": 1 }, { "id": "ai_detection", "question": "Would a listener believe this was recorded by a real person? Answer only YES or NO.", "weight": 1 } ] } ``` ## Scoring ``` Score = (dimensions_passed / total_dimensions) * 100 Example: - correct_language: YES (5/5) → PASS - natural_tone: YES (4/5) → PASS - no_artifacts: NO (2/5) → FAIL - correct_pace: YES (5/5) → PASS - ai_detection: YES (3/5) → PASS Score: 4/5 = 80% ``` ## Autoresearch Integration The eval framework plugs into the autoresearch loop: ``` 1. BASELINE: Run eval suite on current skill → Score A 2. MUTATE: Claude creates a variation of the skill 3. EVALUATE: Run eval suite on variation → Score B 4. COMPARE: If Score B > Score A → promote variation 5. REPEAT: Go to step 2 ``` ### Promotion Rules - New version must score STRICTLY HIGHER than current - Minimum 3 eval runs on the new version to confirm stability - Golden examples updated only when a human approves the new output - All results logged to `results/` for trend analysis ## Binary Question Templates ### For Text Output - "Is the response in {language}? YES or NO" - "Does the response directly answer the question asked? YES or NO" - "Is the response free of AI vocabulary (delve, crucial, vibrant, tapestry, landscape, pivotal, showcase, foster)? YES or NO" - "Is the response concise without unnecessary filler? YES or NO" - "Does the response contain actionable information? YES or NO" ### For Code Output - "Does the code execute without errors? YES or NO" - "Does the code produce the expected output? YES or NO" - "Is the code free of security vulnerabilities (injection, XSS, etc.)? YES or NO" - "Does the code follow the project's existing patterns? YES or NO" ### For Voice/Audio Output - "Is the pronunciation correct? YES or NO" - "Is the tone appropriate for the context? YES or NO" - "Is the pacing natural? YES or NO" - "Is the audio free of artifacts? YES or NO" ### For UI/Visual Output - "Does the page load without JavaScript errors? YES or NO" - "Are all interactive elements clickable? YES or NO" - "Does the layout match the design spec? YES or NO" - "Is the page responsive on mobile? YES or NO" ## Usage When creating evals for a new skill: 1. Identify 3-7 binary dimensions that define quality 2. Write each as a clear YES/NO question 3. Create 2-3 golden examples as ground truth 4. Set runs_per_question to 5 (default) 5. Set judge to claude-opus-4-6 6. Run baseline eval and record score 7. Begin autoresearch loop