LLM-as-a-judge solves a pretty obvious problem. Once an AI system is producing thousands of outputs, humans can’t realistically review everything, while exact-match metrics don’t tell you much about things like groundedness, relevance, or instruction following. So the basic logic makes sense: more outputs → less human coverage → automated evaluation → LLM judge The part we’re still careful with is how much we trust the score itself. It’s useful as a signal, but we wouldn’t treat a 4.3/5 as some objective measure of quality. In practice, we prefer to split evaluation based on what we’re trying to measure: Exact / deterministic requirement → schema validation, exact match, required fields, tests Open-ended quality requirement → LLM judge High-risk, ambiguous, or disputed result → human review For straightforward checks, we still keep it deterministic. If the output needs to match a schema, contain a required field, or return an exact value, we just test that directly. We bring in an LLM judge when the question is more subjective: groundedness, relevance, instruction following, completeness, that kind of thing. The next part is making that judge somewhat trustworthy. Our rough setup looks more like this: generator output → separate judge model → specific rubric → structured score + explanation → human review for uncertain / important cases We generally avoid having the same model generate and grade its own output. Separating the roles doesn’t remove bias, but it avoids the fairly obvious problem of a model favoring the same wording and patterns it just produced. The rubric is another big part of this. Something like “rate helpfulness from 1–5” gives you a number, but not necessarily a useful evaluation. We’d rather break “helpfulness” into concrete checks: Did it answer the actual question? Are the claims supported by the context? Did it follow the constraints? Is anything important missing? This also makes score changes easier to interpret, because you can see which part of the rubric moved instead of just looking at one overall number. Even then, there are some annoying failure modes: answer order → position bias longer response → possible verbosity bias small rubric change → score distribution changes judge model update → baseline moves same judge used repeatedly → generator starts learning its preferences That last one is especially interesting to us. If you optimize a generator against the same judge for long enough, you can end up with: judge prefers X → generator learns X → judge score improves → actual user experience… maybe improves Maybe being the important part. A model can learn that the judge likes longer answers, a particular structure, certain phrasing, or very explicit explanations. Your evaluation graph starts moving up while the system itself may simply be getting better at pleasing the evaluator. So we treat judge performance as something that also needs to be evaluated. For important workflows, we still want a human-rated reference set and periodically compare: human ratings ↔ judge ratings If agreement starts dropping after a model update, prompt change, or rubric change, that’s a signal to investigate the evaluation layer itself rather than immediately assuming the product got worse. Same with multiple judges. If: Judge A: 5/5 Judge B: 5/5 Judge C: 1/5 we’re probably more interested in why they disagree than in forcing those scores into a single average. So for us, LLM-as-a-judge is useful as part of the evaluation setup, but it still needs to be checked against humans and monitored over time.
Curious how people here are doing this in production. Are you comfortable using an LLM judge as an actual release gate, or is it mainly a filter for deciding what humans should inspect? Also interested in how people are detecting judge drift specifically, because once these systems have been running for a while, it can be hard to tell whether the product changed or the judge did. submitted by /u/Innowise_
Originally posted by u/Innowise_ on r/ArtificialInteligence
You must log in or # to comment.