At a recent event, Ilya Sutskever, co-founder and chief scientist at OpenAI, spoke about his long-held belief that large neural networks will be key to achieving artificial general intelligence (AGI).
Sutskever explained that to believe neural nets can mimic the capabilities of the human brain, you need two convictions:
1. The human brain is large and capable, while smaller animal brains are less capable.
2. Artificial neurons are similar enough to biological neurons that they can approximate the information processing of the brain.
Given these two premises, Sutskever has believed that scaling up neural nets will lead to emergent general intelligence, like that seen in humans. This early conviction motivated his groundbreaking work training larger neural networks during his PhD with Geoff Hinton.
When asked what most surprised him as models scaled up, Sutskever remarked that the biggest surprise is that neural networks work at all. Early on, their capabilities were extremely limited. Now we can train models that continually get better as they get bigger, validating the core premise that artificial and biological neurons are comparable.
Discussing the path to AGI, Sutskever noted that while Transformers are not the complete solution, they point the way forward. With sufficient scale, even older architectures like LSTMs could achieve strong capability. The algorithm matters less than the scale.
Sutskever then outlined his vision and concerns around superintelligent AI that goes beyond human-level AGI. With great power comes great responsibility. To steer superintelligent AI toward benefitting humanity, we need to solve the technical alignment problem, guard against misuse by bad actors, and ensure the technology remains robust over time.
