Session: 04 - 01 Emerging Techniques & Technology

Paper Number: 107747

107747 - Better Understanding Physics Informed Neural Network Convergence Through Visualization and Nonconvex Optimization 

Physics Informed Neural Networks (PINNs) refer to a specific category of neural networks that integrate domain-specific physics knowledge with the pattern-recognition capability of neural networks to model physical processes. In nondestructive evaluation (NDE), PINNs have been used to model ultrasonic wave propagation and full waveform inversion as a meshless alternative to traditional FWI. Most PINNs achieve this by incorporating partial differential equations (PDEs) as a term in their loss function. However, such PDE loss terms can introduce additional failure modes to the training process of the PINN. In particular, these equations alter the shape of the loss function (or “loss landscape”), which can cause significant delays or even failure to converge. This casts doubt on the validity of a converged PINN solution. To diagnose and alleviate this failure mode for PINNs, we propose combining two techniques: loss landscape visualization and nonconvex optimization. We use loss landscape visualization as a tool to capture the high-level characteristics of the loss landscape such that difficult areas can be identified. Then we employ a nonconvex optimizer which utilizes the local second-order information about the loss landscape to find the direction of most-negative curvature, allowing training to continue without delay. This talk presents a brief theory of PINNs, nonconvex optimization, and loss landscape visualization. Then we present the above method of recovering a failed PINN training in the form of a case study of a 1-D convection model. Though a naïve choice of the optimizer is shown to fail to converge to a physically plausible solution, evidence is shown that a nonconvex optimizer like SpiderBoost converges, and the behavior of the two optimizers is investigated. Finally, the nonconvex optimizers is used to recover a failed-to-converge PINN without the need for a full retraining.  The application of this technique in NDE can increase the reliability of PINNs and facilitate their wider adoption in the field.

Presenting Author: Jiaze He University of Alabama

Presenting Author Biography: Dr. Jiaze He is an assistant professor in the Department of Aerospace Engineering and Mechanics at the University of Alabama. Before arrival, he was a postdoctoral research associate in the Theoretical & Computational Seismology group at Princeton University and a research scholar at NASA LaRC, during which he also served as an adjunct assistant professor in MAE at NCSU from 2016 to 2018.
His core research area is in advanced diagnostics. His Ph.D. (2015) and M.S. (2013) are in Mechanical Engineering; B.S. (2011) is in Theoretical and Applied Mechanics, during which he visited the Department of Materials Science and Engineering (NCSU) and School of Civil Engineering (HIT). His research interests include ultrasound imaging, full waveform inversion, nondestructive evaluation, structural health monitoring, machine learning, and medical imaging.