How I Built and Evaluated My Own AI System

Most AI work focuses on the model. In practice, the harder problem is everything around it.

A basic system looks like this:

That works for demos. It doesn’t hold up as a system.

Outputs change between runs.
Some responses look correct but aren’t.
Structured formats break.

The core issue isn’t generation. It’s the lack of a reliable way to measure results.

To fix that, I built a lightweight evaluation layer directly into the system.

Before, the workflow looked like this:

from openai import OpenAI

client = OpenAI()

def ask(prompt: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "user", "content": prompt}
        ],
    )
    return response.choices[0].message.content

result = ask("What is the capital of France?")
print(result)

from openai import OpenAI

client = OpenAI()

def ask(prompt: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "user", "content": prompt}
        ],
    )
    return response.choices[0].message.content

result = ask("What is the capital of France?")
print(result)

After adding Evalora, the same flow becomes testable:

from openai import OpenAI
from evalora import Evalora

ev = Evalora()
run = ev.run(name="math test")

client = ev.wrap(OpenAI(), run=run)

def ask(prompt: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "user", "content": prompt}
        ],
    )
    return response.choices[0].message.content

run.check(ask, "What is 2+2?", expected="4")
run.done()

from openai import OpenAI
from evalora import Evalora

ev = Evalora()
run = ev.run(name="math test")

client = ev.wrap(OpenAI(), run=run)

def ask(prompt: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "user", "content": prompt}
        ],
    )
    return response.choices[0].message.content

run.check(ask, "What is 2+2?", expected="4")
run.done()

Now the system doesn’t just generate outputs. It evaluates them.

Each run produces a score based on checks like exact matching, semantic similarity, AI-based judging, and validation rules.

This changes the development process.

Before: test once and move on.

After: run evaluations, compare results, and iterate with feedback.

AI systems don’t fail because they can’t generate responses.
They fail because there’s no clear signal of whether those responses are correct.

Evaluation is what makes that signal visible.

Help build the future of evaluation: evalora