Here’s a new rank estimation method I’ve been working on. It’s basically an upgraded Principal Component Analysis (PCA) built on information theory instead of linear variance. It’s robust to mixed data types, highly non-linear generative processes, low signal to noise ratios, and sparsity (more variables than samples). It’s especially useful if you need to find the exact rank of a dataset to explicitly size a neural bottleneck (like an autoencoder). I just open-sourced the code and put up the preprint. I’d love to hear what you guys think or if you end up testing it on your own data! GitHub (R Code): https://github.com/tjleestjohn/Entropic-Scree Preprint: https://doi.org/10.5281/zenodo.22028087 submitted by /u/Chocolate_Milk_Son
Originally posted by u/Chocolate_Milk_Son on r/ArtificialInteligence