every tutorial i followed spent maybe one paragraph on chunking and moved on. figured it was straightforward. it wasn’t. fixed size chunking splits on token count, not on where a thought actually ends. so you retrieve a chunk that’s about the right thing but the sentence with the actual answer got cut into the next chunk that didn’t make the retrieval cutoff. model gets half the context and wings the rest. spent weeks thinking it was an embedding problem. the thing that finally helped wasn’t changing anything, just actually reading what was coming back for the queries that were failing. the answer was almost always in there somewhere, just split in the wrong place. vector search also just doesn’t work for exact identifiers and i found this out the hard way. someone queries a specific version number or product code, semantic search returns stuff that’s “close” and close is wrong. BM25 alongside vectors fixed it, but i’d never seen it mentioned in any of the intro material i’d gone through. stale index is the other one. updated a document, forgot to re-index, confidently wrong answers for two days before i figured out what happened. not a hard problem but nobody warns you about it. submitted by /u/SilverConsistent9222
Originally posted by u/SilverConsistent9222 on r/ArtificialInteligence