AI Engineer · intermediate

Cost-Aware LLM Model Router

A gateway that classifies every incoming request by difficulty and sends the easy ones to a cheap, fast model and the hard ones to a frontier model — then proves the savings with a dashboard built from a real query log.

Project overview

Every team shipping AI features is under pressure to cut model spend without hurting quality, and routing is the first lever they reach for. Measuring the trade-off on a real log, instead of guessing, is exactly the judgment a senior AI engineer is paid for.

Real-world use case

A customer-support product handles 50,000 chats a month. Most are simple lookups; a few need careful reasoning. Routing cuts the bill by more than half with no measurable drop in satisfaction.

Tools and technologies

• Python
• FastAPI
• LiteLLM
• Claude Haiku + Claude Opus
• PostgreSQL
• Streamlit

Skills you can demonstrate

LLM Cost Optimization, Request Classification, API Gateway Design, Evaluation, Observability

Build roadmap

1. Build the pass-through gateway
   Stand up a FastAPI endpoint that forwards chat requests to one model through LiteLLM and logs tokens, latency, and cost to Postgres. No routing yet — just measurement.
2. Create a labeled request set
   Generate or collect 300 requests across difficulty levels. Label them. This is your test set for every decision that follows.
3. Implement the classifier
   Start with the simplest approach that works — often a small embedding model plus logistic regression. Log the predicted difficulty for every request.
4. Route and compare
   Send easy requests to the cheap model and hard ones to the frontier model. Run the full test set through both the router and a frontier-only baseline.
5. Score quality with an LLM judge
   Use a rubric-driven judge to score every answer 1–5. Confirm the router’s scores match the baseline within your tolerance; if not, tune the threshold.
6. Ship the dashboard
   Build a Streamlit page showing cost, latency, and quality over time, split by model. This chart is the project — make it clean.

Ways to extend the project

Add a feedback loop — when the cheap model’s answer is rated poorly, automatically re-run the request on the frontier model and use the pair to retrain the classifier.

How to showcase your work

• GitHub: The gateway service, the classifier, the eval script, and a dashboard screenshot showing cost per day before and after routing.
• LinkedIn: Post the one chart that matters: total cost with and without routing, with the quality score line flat across both.
• Resume: Quantify it: "Reduced LLM inference cost by X% across N requests while holding answer quality within 2 points."