I’m a software engineer and AI enthusiast who wants to get involved with AI research, but I don’t have the full requisite math, ML coding chops, or compute needed to do typical research. I’m writing this post because I assume there are many others in my boat, and i think i have a meaningful research problem with a shape that allows people like this to make progress. I explain the problem and why it’s tractable by people like this at length in the google doc linked in the comment of this post, but in essence: I believe there’s a chance there’s some mathematical rule that allows you to cheaply imbue the in-context understanding a model gains directly into its weights. IF a rule like this existed, then checking if you’ve found it requires very little compute. The core loop requires running the input token forward passes of a model large enough to learn in context (for reference, a 1 billion parameter model can do this and runs on a mac book pro), apply this rule (which, by the hypothesized construction of where in the solution space we’re looking, is computationally cheap), then quiz the model without the context on what it demonstrably knew in context / run regression benchmarks to make sure the application of the rule didn’t damage the model’s other / previously written capabilities. Although checking if you’ve found this rule is computationally cheap, proposing, implementing, and evaluating candidate rules is very difficult. It requires diverse mathematical and machine learning expertise, along with the scientific rigor to guide the search process. Up until now, there were very few people with access to those abilities. However, this is changing with modern frontier models. OpenAI and Anthropic both have soon to be released models capable of valuable mathematical work (re the erdos unit distance problem solved by the internal OpenAI model and Mythos). My proposal is to form a research community of “citizen scientists” to make progress on this problem. It’s possible the solution doesn’t exist, or is so incredibly complicated that modern frontier models have no hope of solving it. But, my argument is that for the first time, the solution is plausibly within reach of model capabilities. This, in combination with the immense upside of LLMs being able to cheaply learn from experience, makes researching it very high expected value. Participating in this community would involve sharing results, progress, benchmarks, and research insights. It would be fully open source. To productively contribute, rough requirements are: a 200 tier AI subscription a computer ~ as capable as a mac book pro M3 chip / willingness to pay 10 bucks a day for the cloud compute, A working knowledge of how LLMs function and the field of AI / cognitive science. Excitement about the problem submitted by /u/Independent-Soft2330
Originally posted by u/Independent-Soft2330 on r/ArtificialInteligence