Skip to main content

Tactus

Examples / Evaluations

Evaluations

This chapter explores Tactus evaluations - quantitative assessments of AI workflow performance across datasets.

Examples in This Chapter

Simple Eval

A basic evaluation demonstrating core concepts without requiring LLM API calls. This example shows:

  • Defining inline datasets with test cases
  • Evaluation syntax with Evaluation({...})
  • Expected output validation
  • Success criteria based on exact output matching
  • Running evaluations with tactus eval

Has Specs

Success Rate

Demonstrates success rate calculations and aggregated metrics. This example shows:

  • Running multiple test cases in a single evaluation
  • Calculating success rate across dataset
  • Pass/fail criteria for individual test cases
  • Aggregated statistics and reporting
  • Using success rate to measure overall quality

Has Specs · Requires API Keys

Thresholds

Shows how to set minimum acceptable thresholds for metrics. This example demonstrates:

  • Defining threshold requirements (e.g., success_rate >= 0.95)
  • Evaluation failure when thresholds aren't met
  • Multiple threshold configurations
  • Using thresholds for quality gates in CI/CD
  • Balancing strictness with practical tolerance

Has Specs · Requires API Keys

Want to contribute?

The examples repository is open source. Add your own examples or improve existing ones.

View on GitHub