evalora
A Python evals library that runs offline with zero config — install, write a check, see results land in `.evalora/`. No account, no signup, no API key required to get started.
Multiple scoring modes out of the box: normalized exact match, semantic similarity via embeddings, AI judge panels with majority vote, and custom scorers that return 0.0–1.0.
Seven built-in templates (`rag-quality`, `chatbot`, `code-gen`, `summarization`, `safety-audit`, …) and integrations with LangChain, LlamaIndex, CrewAI, and OpenAI Agents.
Production niceties baked in — pytest fixtures, cost guard, resume-on-crash checkpointing, pairwise A/B with position-bias randomization, webhooks, and trend detection that fails CI on regression.