AI Engineer · beginner

AI Experiment Lab: Compare Prompts and Models on Real Job Posts

A small, repeatable experiment that runs several prompts and models against the same set of job descriptions, checks whether each one correctly spots real entry-level roles, and reports which did best on accuracy, speed, and cost. Inspired by the experiment loop in Andrej Karpathy's autoresearch: change one thing, measure it, keep what works.

Project overview

Teams that use AI at work do not pick a prompt or a model by feel. They test options against the same task, measure, and keep the winner. This project proves you can run that loop: define a task, build a small labeled set, compare fairly, and explain a result with numbers. That is the skill behind every "we improved accuracy by 12%" line on an AI engineer's resume, and it needs no training from scratch.

Real-world use case

A college career office receives hundreds of job posts a week and wants to flag the ones that are genuinely entry-level before sharing them with students. Before anyone trusts an AI to do that, the lab shows which prompt and model gets it right most often, how fast, and at what cost.

Tools and technologies

Skills you can demonstrate

Experiment Design, Prompt Engineering, Evaluation Metrics, Model Comparison, Cost and Latency Measurement, Data Labeling, Python

Build roadmap

  1. Build a small labeled set
    Collect 50 to 80 real job descriptions from public postings across a few fields. For each one, decide by hand whether it is truly entry-level and record yes or no in a CSV with a one-line reason. This is your ground truth, and nothing you measure later is better than it.
  2. Write the baseline runner
    A Python script that reads the CSV, sends each description to one model with one plain prompt ("Is this an entry-level job? Answer yes or no."), and saves the answer, response time, and token counts. Include a dry-run mode with canned answers so you can test the plumbing for free.
  3. Score the results
    Compare each answer to your hand label and compute accuracy, precision, and recall. Print a small table. Then open the results side by side with the descriptions and mark each one right or wrong yourself to confirm the script scored it correctly.
  4. Add prompt variants, one change at a time
    Write three more prompts: one with a clear definition of entry-level, one with two labeled examples, and one that asks for a reason before the answer. Run each against the same model and set. Change only the prompt between runs so you know what caused any difference.
  5. Add a second and third model
    Run the best prompt against a cheaper hosted model and a free local one through Ollama. Record time and cost for each. Build the cost estimate from your logged token counts and the provider price table, and write down the date you checked prices.
  6. Build the results page and write it up
    A one-page Streamlit app that shows the results table, a chart of accuracy against cost, and a review view where you can click any job and see every model's answer next to your label. Write a README with the winner, the surprise, and what you would test next.

Ways to extend the project

Follow-up challenges, in order of difficulty: (1) Add a second task, such as extracting required years of experience, and see whether the same prompt still wins. (2) Let the lab run a prompt against itself by asking the model to rewrite the prompt, test the rewrite, and keep it only if the score improves, the way autoresearch keeps a change only when the metric goes up. (3) Add a third judge: have a second model grade the answers and measure how often it agrees with your hand labels. (4) Swap in a tiny local model through Ollama and document exactly where it falls short.

How to showcase your work

Open this project in BuildersBenchBrowse AI Engineer projects