I have been doing some work with local models and continued pretraining (CPT), specifically around teaching a small model (qwen 3.5 4B) a new domain. Here are some of my findings around creating evals for measuring the model’s ability to internalize the knowledge: The model outputs travel legs that make up an n-transfer subway journey, so I decided to model the llm response based on a strict json schema. To improve the model’s ability to reliably output the schema, I fine tuned the model post CPT training. The fine tuning is done through Unsloth SFT (alpaca format). In addition to exact matches, I also measure partial matches (e.g. prediction a partial subway route) and semantic matches. Overall this method seems to work well. More info here in case you are interested in more details: https://www.teachmecoolstuff.com/viewarticle/creating-evals-to-validate-llm-training submitted by /u/funJS
Originally posted by u/funJS on r/ArtificialInteligence
You must log in or # to comment.