Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, V. Mirrokni
Nested Learning (NL) redefines learning as multi-level nested optimization problems, presenting expressive learning algorithms and a continual learning module (Hope) that surpass the limitations of traditional deep learning.
Modern deep learning, especially language models, faces fundamental challenges and unanswered questions regarding continual learning, self-improvement, and effective solution discovery. Existing architectures struggle to absorb new knowledge or adapt over long periods due to fixed learning rules and limited memory structures.
NL represents a model as a set of nested, multi-level, and/or parallel optimization problems, each with its own context flow. It reveals that existing learning methods compress context flows from data, and in-context learning naturally emerges in large models. Three core contributions: (1) Gradient-based optimizers like Adam and SGD with Momentum are shown to be associative memory modules that compress gradient information; more expressive optimizers with deep memory are presented. (2) Self-Modifying Learning Module: a sequence model that learns its own update algorithm. (3) Continuum Memory System: a new formulation generalizing traditional short/long-term memory. Combining these, the Hope module shows promising results in language modeling, knowledge incorporation, few-shot generalization, continual learning, and long-context reasoning.
The NL paradigm offers a new philosophy for designing learning algorithms, enabling higher-order in-context learning and effective continual learning. The Hope module demonstrates superior performance across various tasks, validating the practical value of NL.