Hey r/ArtificialInteligence! Visualizing the loss landscape of a neural network is notoriously tricky since we can’t naturally comprehend million-dimensional spaces. To generate the 3D surface plots of deep learning model’s loss landscape, I tried the methodology from Li et al . and verified the things mentioned in the 2018 Li et al. paper about short cuts like those that existi in resnet smoothen the loss landscape, loss when visualized during train mode with dropout show up as spikes, and that certain model architecture choices result in smoother/rougher loss landscapes. A known limitation of these dimensionality reductions is that 2D/3D projections can sometimes create geometric surfaces that don’t exist in the true high-dimensional space. I’d love to hear from anyone who studies optimization theory and how much stock do you actually put into these visual analysis when analysing model generalization or debugging. I built a small, interactive browser experiment https://www.hackerstreak.com/articles/visualize-loss-landscape/ to help build better intuitions for this. It maps these spaces and lets us actually visualize the terrain for those model architectures mentioned in the paper. submitted by /u/Hackerstreak
Originally posted by u/Hackerstreak on r/ArtificialInteligence
You must log in or # to comment.