Matthieu Wyart, a statistical physicist, discusses the abstraction levels in AI learning with Tim Scarfe. The conversation explores how deep networks can uncover hidden hierarchies in data, which shallow models often miss, and the implications for AI's ability to learn and create.
Key points
Matthieu Wyart introduces himself and his interest in whether AI should predict at a low token level or at higher abstraction levels. 0:23
Wyart explains Chomsky's poverty of stimulus argument and how deep architectures have an implicit bias to build coarse-grained variables. 0:53
The discussion touches on how deep networks can learn abstractions faster due to their introspective learning from latent spaces. 1:36
Wyart draws parallels between physical systems and AI, using examples like sand and stock markets to explain complex systems. 3:03
The conversation explores the idea of universal learning algorithms and the constraints in AI training processes. 10:00
Wyart argues that deep architectures can solve tasks by understanding hidden hierarchies in data, contrary to the belief that they lack high-level abstraction capabilities. 26:53
The discussion highlights the importance of predicting in latent spaces for more sample-efficient learning. 53:00
Wyart discusses the curse of dimensionality and how deep networks overcome it by discovering hierarchical structures in data. 49:01
The conversation touches on the potential for AI to achieve transformative creativity by discovering new subspaces and constraints. 39:03
Wyart emphasizes the need for new training methods that predict in latent spaces for more efficient learning. 57:00
Quotes
If you never do mistakes, maybe it's a sign that you're staying a bit on the beaten path in science.
The reason why deep architecture can solve those tasks is precisely because they understand just like the physicists understood about pressure velocity field.
To build abstraction you need to bring configuration.