The Division of Mathematical Sciences (DMS) at the National Science Foundations (NSF) has awarded Zaid Harchaoui, Associate Professor of Statistics, and his interdisciplinary team $1.1million for their research on “Scaling Laws of Deep Learning”. The research project builds the mathematical and scientific foundations of deep learning by characterizing the fundamental quantities and general laws that govern the empirical phenomena observed by applied scientists and engineers. Deep learning is a paradigm in machine learning and artificial intelligence where statistical models are learned from data by designing networks of parameterized modules and training them on big datasets using optimization algorithms. It has widespread effects on science and society, from autonomous vehicles to online commerce and social media. Scientific research increasingly relies on deep learning for data-driven scientific discovery.
The research program addresses the concept of scaling laws. On the practical side, scaling laws greatly simplify parameter setting for large experiments and model transfer to new domains. On the theoretical side, scaling laws shed light on empirical phenomena and unify them with mathematical concision. The team builds upon recent advances in optimal transport, empirical processes, nonparametric statistics, information theory, and complexity theory, and grounds its work in empirical observations made in large experiments in natural language processing and computer vision, among other applied domains. The research outcomes provide practical guidelines for scientists and engineers who employ deep learning to tackle challenging problems, and also constitute fundamental advances in the core areas of mathematical, statistical, and computer sciences.