long term i want to build a model that does something llms are bad at: look at a pile of scientific knowledge and point at what’s missing. like “something belongs here and it should look like this”, instead of writing a nice sounding answer. im betting on JEPAs (joint embedding predictive architectures, the lecun/meta line: I-JEPA, V-JEPA). a jepa hides part of its input and predicts a summary of the hidden part instead of the next word. hiding pieces and guessing what fits is basically the “find the missing piece” game, so it felt like the right place to start. first test: can it rediscover mendeleev? in 1869 mendeleev left gaps in his periodic table and predicted gallium, scandium and germanium years before anyone found them. so i trained a small jepa (~118k params) only on chemistry known in 1869: the 47 elements from hydrogen to barium, with valence, metal or not, and state at room temp. then i asked it where blanks would make the sequence make the most sense.
- i wrote down the pass criteria and locked the answer key with a hash before writing any model code, so i couldnt fool myself later
- no pretrained model anywhere bc those already know the future result: it failed my own bar.
- without atomic weights it ranked the scandium gap 7th-14th and the gallium/germanium gap 9th-11th out of 46 possible spots (chance is ~23rd). some signal but not stable across machines
- totally missed technetium’s gap (25th-26th)
- never figured out that 2 elements were missing between zinc and arsenic
- with atomic weights in, it did worse than the dumbest baseline (just ranking the biggest jumps in weight) lol next i wanna try the same thing on physics before 1905: train only on what was known and see if it points at the holes relativity and quantum later filled. what id love opinions on: is “how much does adding a blank reduce the model’s surprise” a sound way to detect gaps, or would you score it differently? with data this small is a jepa just the wrong tool? would a plain masked model do as well? know any good historical test cases where the answer is known but leakage risk is low? bigger picture: is finding what’s missing actually a different skill from what llms do, or just next token prediction in disguise? fyi im an independent researcher, not an ml person by career, and i build w ai coding agents. harsh feedback welcome, id rather hear now if the idea is broken submitted by /u/WarningSure1670
Originally posted by u/WarningSure1670 on r/ArtificialInteligence