I’m a type 1 diabetic and since March I’ve been working on training my own transformer model to predict my blood sugar: It’s an encoder-only transformer with variable-width context size (8 - 24 hours) that uses past blood glucose (BG), and past + future carbs & insulin to condition its predictions for future BG. Both input and output BG values live in kovatchev risk space reparameterized to [40, 400] range. I pretrained the model on the outputs of my custom simulator and then fine-tuned the model on three real-world datasets (ohiot1dm, azt1d, shanghait1dm), and on my own blood sugar data. Here’s the link to the source code (MIT-licensed) + weights + evaluation data: Github Project Link I’ve posted about this model to another subreddit before where it got called fat, so let me emphasize that I’ve trained multiple models ranging in size from less than 40K parameters to ~17M parameters. It natively supports what-if predictions, i.e. you can ask the model how will eating 20 grams of carbs with GI = N affect my blood sugar K minutes from now giving everything else that I’ve done in the past 8 - 24 hours? It also predicts time by looking at the context (which contains past blood glucose and meal & bolus information). Here are some evaluation metrics: median line (RMSE/MAE in mg/dL, MARD in %) Pooled, 3 real cohorts (n=1958 windows) PH RMSE MAE MARD 30 16.68 11.36 8.06 60 22.77 15.65 10.91 120 32.76 22.76 16.09 OhioT1DM (263 windows, 6 patients) PH RMSE MAE MARD 30 19.41 13.16 8.37 60 26.39 18.19 11.51 120 40.77 30.21 18.55 AZT1D (1350 windows, 24 patients) PH RMSE MAE MARD 30 16.90 11.56 8.26 60 21.56 14.96 10.43 120 27.66 19.43 13.65 ShanghaiT1DM (345 windows, 12 patients) PH RMSE MAE MARD 30 13.19 9.18 7.04 60 24.35 16.42 12.32 120 42.79 30.13 23.75 T1DMSIM, synthetic (1440 windows, 30 patients) PH RMSE MAE MARD 30 16.77 12.51 8.22 60 28.73 21.56 14.66 120 45.89 36.28 25.86 I used DILATE loss for the median line and pinball loss for uncertainty bands. The two are combined using Kendall-Gal weighting. I used Defazio’s AdamC schedule-aware weight-decay correction to keep gradients stable toward the end of the training. The architecture itself is inspired by BERT and modern LLMs: it uses bidirectional attention heads, Muon optimizer, RoPE, QK-norm, SwigGLU FFNs, and autoregressive rolling for >2 hour predictions. I also have built a custom app for myself to use & test the model on my own data. There are two variants: fp32 version which runs on the CPU (~30 ms average inference) and an fp16 version that runs on the GPU using Vulkan (which turned out to be not that useful since GPU warm-up takes a lot of time). I have also tried to use the Mediatek NPU on my phone but last time I tried it required buying a license. There are exporter scripts in the repo that allow exporting the model for fp32 CPU and fp16 Vulkan-based inference. Let me know what you think! The video above is the gui.py script running the model in what-if mode. submitted by /u/0xdeadf1sh
Originally posted by u/0xdeadf1sh on r/ArtificialInteligence
You must log in or # to comment.