feat: Add pointwise evaluation mode with pytest integration

[benjibc]

Summary

This PR introduces a new pointwise evaluation mode that provides an elegant API for writing LLM evaluation functions with full pytest integration.

Key Changes

🎯 New Pointwise Mode

• Added mode='pointwise' parameter to @evaluation_test decorator
• Enables row-by-row evaluation where users define only the core evaluation logic
• Framework handles all infrastructure concerns (models, datasets, rollouts, aggregation)

📊 Clean Architecture

• Separation of Concerns: Evaluation logic separate from test configuration
• Full Parameterization: Models, datasets, thresholds all configured in decorator
• Pytest Integration: Run as regular pytest tests with full reporting and CI/CD

📚 Documentation Overhaul

• Completely rewrote README.md with clean, modern structure
• Added Mermaid diagram showing parameterized evaluation components
• Included both pointwise and batch mode examples
• Aggressive cleanup of verbose old examples

🧪 Working Example

• Added comprehensive word_count evaluation example using pointwise mode
• Includes haikus dependency and dataset adapter
• Demonstrates the elegant new pattern with working test

Before vs After

Old Pattern (Deprecated):

@reward_function(id="word_count", requirements=["haikus==0.3.8"])
def evaluate(messages: List[Message], **kwargs) -> EvaluateResult:
    # Mix of evaluation logic AND infrastructure concerns

New Pattern (Recommended):

@evaluation_test(
    input_dataset=["data/sample.jsonl"],
    model=["gpt-4o-mini"],
    input_params=[{"temperature": 0.0}],
    threshold_of_success=0.8,
    rollout_processor=default_single_turn_rollout_processor,
    mode="pointwise",  # 🎯 Key innovation
)
def test_word_count_evaluate(messages: List[Message], **kwargs) -> EvaluateResult:
    # ONLY core evaluation logic - everything else parameterized!

Testing

• ✅ All tests pass
• ✅ Word count example runs successfully with pointwise mode
• ✅ Backward compatibility maintained for batch mode
• ✅ Full pytest integration working

Migration Path

• Old @reward_function pattern still works but is deprecated
• New pytest-based approach provides better testing, CI/CD integration, and flexibility
• Migration guide included for existing users

This provides a much more elegant and maintainable approach to LLM evaluation functions.

[dphuang2]

Makes sense

[dphuang2]

im going to hijack this