I’m building a platform where an LLM has to reference a large number of existing nodes. For example, when generating a DAG, it needs to know about many previously defined nodes and correctly reference them while constructing the graph. I’m trying to figure out the best way to provide this large amount of context while optimizing for latency, cost, and reasoning quality. Is context caching a good solution when most of the context remains the same across requests? Alternatively, would a Retrieval-Augmented Generation (RAG) setup with a vector database be a better choice? My concern is that the model may need to reference a large number of nodes, not just retrieve a handful of semantically similar ones. How do people handle situations where an LLM needs access to a very large amount of structured context? I would really appreciate any information, guidance, recommendations, experiences, or resources. Thank you so much! submitted by /u/Firm-Track3617
Originally posted by u/Firm-Track3617 on r/ArtificialInteligence
You must log in or # to comment.