LifeOS
← All 17 principles

Principle № 5 · Technical

Spec / Test / Evals First

Define success criteria before building anything

Spec / Test / Evals First

Overview

Define expected behavior before writing implementation. If you can’t specify it, you can’t test it. If you can’t test it, you can’t trust it.

This is test-driven development (TDD) extended to AI systems. For traditional code, you write tests first. For AI components, you define evaluations (evals) first. Both serve the same purpose: defining success before attempting to achieve it.

Evaluations measure whether AI components actually work. Without evals, you’re guessing. With evals, you’re measuring.

Why This Matters

Specifications prevent scope creep - When you define what “done” looks like upfront, you stop building when you get there. Without specifications, projects drift endlessly.

Tests enable refactoring - You can’t safely improve code without tests. Change something, run tests, verify nothing broke. No tests = fear of change.

Evals enable iteration - AI output quality varies. Evals let you measure: “Is this output good enough?” Without evals, quality assessment is subjective and inconsistent.

Specifications enable delegation - “Build X that does Y” is delegable. “Build something good” is not. Clear specs let others (including AI) implement correctly.

Implementation

LifeOS applies this across multiple levels:

Development Methodology - Engineer agent follows TDD religiously. Tests written first, implementation second, refactoring third. Red → Green → Refactor.

Skill Structure - Every skill’s SKILL.md contains:

  • Problem statement (specification)
  • Solution description (expected behavior)
  • Workflows (test cases)
  • Success criteria (evaluation)

Contract Tests - API specifications and interfaces defined before implementation. Integration tests verify real-world user journeys.

Evals Skill - Dedicated skill for LLM-as-Judge evaluation framework. Template-based judge prompts, multi-model panels, statistical rigor.

CI/CD Integration - Tests run automatically. Failed tests block deployment. No “we’ll test later” culture.

Examples

Example 1: Traditional Code (TDD)

// 1. Write test first (RED phase)
describe('UserAuth', () => {
  it('should reject invalid email formats', () => {
    expect(validateEmail('not-an-email')).toBe(false);
  });
});

// 2. Implement minimal code (GREEN phase)
function validateEmail(email: string): boolean {
  return email.includes('@');
}

// 3. Refactor for completeness
function validateEmail(email: string): boolean {
  return /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(email);
}

Example 2: AI Component (Eval-First)

// 1. Define evaluation criteria
const evalCriteria = {
  accuracy: "Does the summary capture all key points?",
  brevity: "Is it under 200 words?",
  readability: "Flesch reading score > 60?"
};

// 2. Implement AI component
async function generateSummary(text: string): Promise<string> {
  // Implementation
}

// 3. Eval validation
const result = await generateSummary(testText);
assert(evalCriteria.accuracy(result) >= 0.9);
assert(result.split(' ').length < 200);
assert(fleschScore(result) > 60);

Example 3: Skill Development

  • Before building Research skill: Define what “good research” means
  • Specification: “Multi-source analysis with source attribution, bias detection, and confidence scoring”
  • Test: Feed known-answer questions, verify output quality
  • Eval: LLM-as-Judge assesses completeness, accuracy, source quality