We have benchmarks (fixed questions, hidden answers, they saturate), competition problems (solved, so they can be memorized), and open-problem lists (no evidence, no clear success criterion). With Claude and Sol, I’ve tried something in between: a graded ladder of AI puzzles from one research area, aperiodic order. What makes it a different tool:
- One ladder, not a pile: “human level” (pen and paper) up to “ASI level” (open). The labels are playful, but each rung really is harder than the last.
- Solved rungs for training and checking, with exact answers a program can verify, and easy to vary.
- Open rungs with everything a solver needs: strong numerical evidence, routes already known to fail, files, and a success criterion a machine can check.
- One rule: only a checkable proof counts. Floats are evidence, not proof.
- Credit: solvers of open rungs are named, with agent and model, in the next version of the paper. It’s a way for researchers to hand the frontier of their field to AI agents (and humans) in a form they can actually attack, and for anyone to see whether a model can tell a guess from a proof. First field test: Grok, asked in public ( https://x.com/grok/status/2114103836583264721 ), claimed a proof for an open rung that was only a numerical estimate, then retracted it. It got the human-level rung wrong at first, fixed it after one hint, and proved the expert rung on the first try. On MathOverflow, an expert is already testing constructions for the top rung. Set 1: huggingface.co/datasets/jarekduda/autocrystal-ai-puzzles Top rung on MathOverflow: mathoverflow.net/questions/516372 Would you use this format for your own field’s open problems? submitted by /u/jarekduda
Originally posted by u/jarekduda on r/ArtificialInteligence
You must log in or # to comment.