Original Reddit post

Most teams ship prompts the same way they used to ship CSS in 2008. Tweak, eyeball a few outputs, push to prod, wait for users to complain, repeat. Prompts are production code. They deserve the same testing infrastructure your Python does. That’s why I built PromptLabs. How the loop works, in five steps:

  1. You provide the input. Either an intent (“classify customer support emails as billing, technical, account, or other”) or an existing production prompt plus the failure modes you’ve been seeing.
  2. EvalGen writes your test suite. It picks 5 to 8 categories of inputs that will exercise the prompt (happy path, edge cases, adversarial), fires one parallel LLM call per category, and dedupes the result. So you get real coverage, not 50 reworded copies of the same easy case. The same call also writes the scoring rubric. Then it splits the test set into train and holdout. The holdout never leaks into optimization.
  3. Runner executes the prompt across every target model in parallel. Choosing between Sonnet 4.6, GPT-5, and Gemini 3? All three run at once on the same eval set. Results in minutes, cost per eval plotted on the same chart.
  4. Judge scores every output, criterion by criterion. LLM-as-judge with reasoning attached, so you can see exactly why a score is what it is.
  5. Optimizer proposes a diff, not a regeneration. It looks at where the prompt failed, then returns specific line edits (insert this clause after line 3, delete this sentence, reword this paragraph). You read it like a pull request. The new version is scored on the holdout set. The loop checks for convergence or overfitting, and either accepts the result or loops back to step 3 with the new prompt. The accepted prompt is served over HTTP. Your production code fetches the latest version at request time, so you can iterate without redeploying. Three things that make this different from tools you’ve probably tried: The eval set is real, not theater. Stratified by category with parallel generation and dedup, so you get coverage of edge cases instead of fifty rewordings of the happy path. Most tools either skip eval generation entirely, or give you one LLM call that quietly produces 40 near-duplicates. Train and holdout stay separate, and the loop enforces it. The trajectory chart shows the gap widening the moment you start overfitting, and the loop halts itself when it does. The “best version” pick uses a lower confidence bound so a lucky high-variance run can’t game the leaderboard. Most “optimizer” tools you’ve seen don’t even have a holdout set. The Optimizer evolves your prompt, it doesn’t replace it. A diff is reviewable. You can accept some edits and reject others. The domain knowledge you spent six months baking into your prompt isn’t thrown out every iteration. DSPy-style frameworks regenerate; this one refines. If you’ve been gluing promptfoo + dspy + langfuse together to do what should be one workflow, this is one tool that does the whole thing. If you’re treating prompts like config strings instead of like the production code they are, you’re leaving accuracy on the table and inviting silent regressions you wont see until they hurt. MIT, local, your keys. https://github.com/temm1e-labs/promptlabs submitted by /u/No_Skill_8393

Originally posted by u/No_Skill_8393 on r/ClaudeCode