Investigating the Role of Overparameterization While Solving the Pendulum with DeepONets

Machine learning methods have made substantial advances in various aspects of
physics. In particular multiple deep-learning methods have emerged as efficient
ways of numerically solving differential equations arising commonly in physics.
DeepONets [22] are one of the most prominent ideas in this theme which entails an
optimization over a space of inner-products of neural nets. In this work we study
the training dynamics of DeepONets for solving the pendulum to bring to light
some intriguing properties of it. We demonstrate that contrary to usual expectations,
test error here has its first local minima at the interpolation threshold i.e when
model size ≈ training data size. Secondly, as opposed to the average end-point error,
the best test error over iterations has better dependence on model size, as in it shows
only a very mild double-descent. Lastly, we show evidence that triple-descent [1]
is unlikely to occur for DeepONets