Prompt gets tweaked, output looks fine on a quick check, it ships. No versioning, no regression tests, no real evaluation beyond someone’s gut feel. Weeks later something’s off and nobody can point to what changed or when, because nothing was ever actually measured in the first place. This is the norm right now for a huge share of LLM features being shipped, model selection by intuition, “evals” that are just a handful of manual spot checks, retrieval that was never benchmarked, and cost problems that show up as a surprise invoice instead of something caught early. There’s a hands-on masterclass on Sep 12 built around applying real engineering rigor to this: prompts treated as versioned code with regression tests, an eval harness combining deterministic checks and LLM-as-judge, statistically sound model comparisons using bootstrap confidence intervals and paired significance testing rather than “it feels better,” evaluated RAG with proper retrieval metrics, agents with guardrails and fallbacks that degrade gracefully instead of compounding errors, and full production observability, tracing, cost, latency. Led by Bruno Gonçalves, PhD, founder of Data For Science, previously a Data Science Fellow at NYU’s Center for Data Science, who trains engineers at Fortune 500 companies on this exact discipline. Link for more details submitted by /u/camerongreen95
Originally posted by u/camerongreen95 on r/ArtificialInteligence