I’m working on a research project that has verification as its main goal (rather than building a better LLM). Specifically, the problem I am trying to solve is: how can we verify if an answer generated by an AI model is sufficiently trustworthy for a given application, for example, finance, law, engineering, medicine, research, etc. Most of the current work is focused on improving the model’s ability to predict, but I’m more interested in verifying the predictions rather than making better predictions. So, here’s my question: What does it mean for a claim to be true, justified, and trustworthy mathematically? I’m open to philosophical thoughts, but I’m more interested in mathematical than philosophical considerations. Some questions I’m trying to ask myself: Can you define trust as a mathematical function rather than a set of heuristics that estimate confidence? Is there a mathematical relationship between truth, evidence, proof, constraints, and trust? Should trust be defined using probability theory, information theory, formal logic, graph theory, topology, category theory, optimization, or something else? Can all claims be represented as some object that has evidence, assumptions, constraints, and derivations? Are there any works that try to design a proof of correctness for answers generated by AI models rather than estimating their confidence? How would you define the difference between: a true claim, a justified claim, and a trustworthy claim if you were to design the Trust Engine? One approach I thought of was to think of verification as a constraint satisfaction problem where a claim has to satisfy certain mathematical/logical/evidential constraints to be considered trustworthy. Another approach is to think of trust as a type of convergence to truth as more evidence becomes available, but I’m not sure if this is the right way to think about it. I’m looking for recommendations for papers, books, formal methods, mathematical frameworks, theories, directions for investigation, and criticism of the ideas presented above. I’m most interested in the thoughts of people working in formal methods, theorem proving, mathematical logic, knowledge representation, verification, optimization, information theory, and trustworthy AI. Let me know how you would approach this problem from first principles. submitted by /u/MuhammadMujtaba21
Originally posted by u/MuhammadMujtaba21 on r/ArtificialInteligence