I tried a simple way to evaluate an LLM controller: log its forecast before advancing the sim, then score its actions against a baseline. At sim time 65s it predicted the ambulance would arrive at 100s. At 95s, after the delay was visible, it revised the prediction to 112s. The actual arrival was 112.2s, while a basic queue rule got there in 107.6s with the same arrivals, routes and safety constraints. The late estimate was close, but the earlier estimate is the one I’d compare across controllers if the goal is planning ahead. This was one Godot run. My recorded experiment: https://www.youtube.com/watch?v=YY41_8_H-O8 submitted by /u/ViceGodIm
Originally posted by u/ViceGodIm on r/ArtificialInteligence
You must log in or # to comment.