Stephen Wolfram Readings: What’s Really Going On in Machine Learning? Some Minimal Models

The speaker begins by expressing curiosity about the fundamental workings of machine learning, particularly the lack of a clear understanding of why neural networks function effectively. They mention a recent breakthrough in understanding this mystery, which was unexpectedly linked to the study of biological evolution. The talk aims to delve into the essence of machine learning by examining minimal models that are more easily visualized and understood.

The speaker discusses the complexity of neural networks, noting that while much is known about constructing them for various tasks, the underlying principles remain elusive. They highlight the simplicity of the basic structure of neural networks and the difficulty in visualizing their trained state. The speaker questions which aspects of the setup are essential and which are mere historical artifacts. The goal is to strip down the complexity and explore minimal models that can still capture the phenomena of machine learning.

The speaker shares their surprise at discovering that very simple models can reproduce machine learning phenomena. These minimal models allow for easier visualization and understanding of the essential processes. The speaker challenges the idea that machine learning systems build structured mechanisms, instead suggesting that they sample from the complexity of the computational universe, finding solutions that overlap with desired outcomes.

The concept of computational irreducibility is introduced as a key factor in the richness of the computational universe and the success of machine learning. The speaker explains how computational irreducibility leads to the effectiveness of training processes in machine learning systems. They also discuss the implications of computational irreducibility for the possibility of a general narrative explanation of machine learning, suggesting that such a science may not be possible.

The speaker draws parallels between machine learning and biological evolution, noting that both involve adaptive processes. They describe a simple model of biological evolution that captures essential features and aligns with the phenomena of machine learning. The talk suggests that the core of machine learning and biological evolution are connected through computational irreducibility.

The speaker discusses the practical implications of understanding the foundations of machine learning, suggesting that it could lead to more efficient and general machine learning methods. They also touch on the theoretical insights gained from studying minimal models, such as the potential for a new kind of science that is fundamentally computational.

The speaker revisits traditional neural networks, using a fully connected multi-layer perceptron as an example. They demonstrate how these networks can compute functions and how their internal behavior changes with different inputs. The discussion highlights the complexity of understanding what happens inside these networks and the challenges in visualizing their training process.

The speaker delves into the training process of neural networks, explaining how weights are adjusted to minimize loss and how this process involves randomness. They show examples of different networks that have learned the same function and discuss the variability in the training process. The talk also touches on the adaptability of networks to different parameter settings and activation functions.

The speaker introduces mesh neural networks as a simplification of traditional neural networks, where each neuron receives input from only two others. They demonstrate that these simplified networks can still effectively compute functions and are more easily visualized. The talk explores the training process for mesh networks and their ability to reproduce functions with fewer connections and weights.

The speaker explores the idea of making machine learning systems completely discrete, drawing on the example of a three-color cellular automaton. They discuss how discrete adaptive processes can lead to the evolution of rules that achieve specific goals, such as generating patterns with a certain lifetime. The talk highlights the surprising effectiveness of discrete systems in machine learning.

The speaker discusses the relationship between adaptive evolution and rule arrays, drawing parallels with biological evolution. They explore how rule arrays can be used to represent discrete systems and how these can be trained through simple mutation and selection processes. The talk emphasizes the computational irreducibility of these systems and their potential for machine learning.

The speaker introduces multi-way mutation graphs as a way to visualize the evolution of machine learning systems. They discuss how these graphs can represent different strategies for problem-solving and how they relate to the concept of branchial space. The talk also touches on the optimization of learning processes and the potential for more efficient methods than random mutation.

The speaker discusses the concept of change maps, which show how the value of a system changes with mutations. They explore the idea of using these maps to guide the learning process towards more efficient solutions. The talk also covers the limitations of change maps and the potential for combining them with other methods to improve learning outcomes.

The speaker reflects on the broad capabilities of machine learning, noting that it can, in principle, learn any function. They discuss the representability of functions by different systems and the learnability of these functions through adaptive evolution. The talk highlights the importance of computational irreducibility in enabling machine learning to find solutions.

The speaker concludes by discussing the role of the computational universe in machine learning. They emphasize that machine learning harnesses the complexity of this universe to find solutions that align with objectives. The talk suggests that machine learning's success relies on the diversity and richness of computational processes, rather than on building structured mechanisms.