Original Reddit post

I’ve seen a few comparisons of different models recently and how they perform at coding. A common recurring test seems to be how they handle so-called “large code bases”. As a software developer, I’m wondering: Does one really need to fully understand a large code base in order to work with it? I usually do, after some time, but never all at once, and I’ve seen a lot of human developers be quite productive despite not understanding everything at once all the time. The mental context window you need to work with a code base likely depends heavily on how it is structured. If it is messy, with dependencies all over the place, then you probably do need a lot of context. If not, then only local context should do. I see code bases like databases. An indexed query in a database should have a cost of roughly O(log N) where N is the size of the table. At least that’s the complexity you get with all kinds of binary trees (I have no idea how actual databases work, but I guess they don’t run on magic). This means that complexity (the number of rows you have to look at, or “context window”) doesn’t grow linearly with the size of the data. Also, this is a rather pessimistic analogy. Code is not an indexed table (you can index it in various ways, but searching in indexes is not understanding). when you work on one part of a code base, chances are that 95% of the code is not relevant to your work at all, so asymptotic context window size would be closer to O(1) with any log N term being due to residual messy code and dependencies that shouldn’t be there, rather than something inherent to the “algorithm”. Finding the right place in the code to touch can usually be done with mechanical (non-AI) tools, like regex search. Coding agents are in fact quite good at “outsourcing” thinking about code to mechanical tools, such as the compiler. Just like a human developer would. I have seen GPT run the compiler to get the size of a data structure when I asked it. Personally, I would have just calculated it in my head, as writing the code to have the compiler do it for me would have taken longer. But the LLM can “type” much faster than me, so it ran the dumb mechanical tool to do the math and rather than consuming context tokens to do it “manually”. Many human developers also use the compiler to test if their ideas are sound or which direction to go next. At least I do. Because we all have limited “context windows”. So why do we judge models on performance on large code bases? Because most code bases are messy? Because people vibe code and don’t know how to keep their code clean, structured and modular? Because of untyped / uncompiled languages (JavaScript, Python, …) where the only reliable way to get feedback on whether your code is correct is running it? If a lesser model struggles with your large project, then perhaps so would humans? submitted by /u/EC36339

Originally posted by u/EC36339 on r/ArtificialInteligence