AI Engineer · advanced
Agent Evals and Trace Observability Harness
A regression-testing and tracing system for an AI agent — every run captured as a trace with tool calls and reasoning steps, scored against a golden dataset, and gated in CI so a prompt change that breaks behavior fails the pull request.
Project overview
Lack of trace-level visibility and quality measurement is the most cited reason agent rollouts stall in production. Teams that can see what an agent did and prove a change did not regress it are the ones that get to ship — and that skill is scarce.
Real-world use case
A company runs a support agent that can issue refunds. Before any prompt or tool change reaches production, the harness replays 200 real conversations and blocks the deploy if refund accuracy drops.
Tools and technologies
- Python
- LangSmith or Langfuse
- DeepEval
- OpenTelemetry
- pytest
- GitHub Actions
Skills you can demonstrate
LLM Evaluation, Agent Tracing, Test Design, CI/CD Gating, Observability Engineering
Build roadmap
- Instrument the agent
Take an existing agent (or the one from cs-5) and add tracing. Every model call and tool call must appear as a span with inputs and outputs. - Build the golden dataset
Collect 50–200 cases with expected outcomes. Include adversarial and out-of-scope inputs. Store it as versioned JSONL in the repo. - Write the evaluators
Implement exact-match, LLM-judge, and trajectory evaluators. Calibrate the judge on 20 hand-scored cases before trusting it. - Run and baseline
Run the suite against the current agent and record pass rates per evaluator. This baseline is what every future change is compared to. - Gate CI
Add a GitHub Actions job that runs the suite on every PR and fails if any pass rate drops more than a set margin. Intentionally break a prompt to prove the gate fires. - Add the observability layer
Export traces to LangSmith or Langfuse and build a view that links a failed eval case straight to its trace. Write up one bug you found this way.
Ways to extend the project
Add trajectory-level assertions — not just "was the final answer right" but "did the agent call the lookup tool before the refund tool" — and alert when the tool-call order drifts.
How to showcase your work
- GitHub: The agent, the eval suite, the golden dataset, and a CI badge. Include a failing-run screenshot so reviewers see the gate actually works.
- LinkedIn: Show a trace waterfall of one agent run with a tool call highlighted, next to the eval summary that scored it.
- Resume: Frame it as reliability engineering: "Built an agent evaluation and tracing harness gating deploys on a 200-case golden set."