Welcome to AI Decoded, Fast Company’s weekly newsletter that breaks down the most important news in the world of AI. You can sign up to receive this newsletter every week via email here.
Are the biggest AI labs betting on the wrong horse?
Big AI companies are betting nearly all of their R&D and capital expenditure on the idea that pre-trained transformer models can deliver AI with human-level general intelligence. This approach relies heavily on backpropagation, the standard algorithm used to train deep neural networks.
Ben Goertzel, who coined the term “AGI” with his 2005 book Artificial General Intelligence (co-written with DeepMind founder Shane Legg), is skeptical. “The commercial AI industry is just betting everything on copying GPT [generative pre-trained transformers] in various permutations, which in my view is a waste of resources because all these LLMs are kind of doing about the same thing.”
“When something works, everyone wants to double and triple down on what worked,” he says. But this concentration of resources around a single paradigm may be risky. Transformer models require billions of dollars in compute to train, along with enormous ongoing computational resources to operate. So far, major AI labs have continued to see intelligence gains from adding more compute and training data. But as models grow larger, those gains are becoming increasingly expensive, raising the possibility that the returns may eventually no longer justify the cost. And because the financial stakes are so high, labs have little room to invest seriously in fundamentally different approaches.
Goertzel argues that scale alone is not enough without the right underlying algorithms. In his view, a major limitation of transformer models is that they cannot continually learn from new experiences and update their internal parameters in real time the way humans do. Instead, they revert to their baseline parameters with each new interaction, without meaningfully learning from prior exchanges.
Researchers at Google DeepMind, Microsoft, and Ilya Sutskever’s Safe Superintelligence are exploring alternative neural network architectures that may enable continual learning, Goertzel says. “DeepMind has incredible diversity within their AI team” and possesses a “deep bench” of experience with alternate AI paradigms, he says.
The result is an AI landscape in which massive compute resources are largely devoted to refining existing methods rather than pursuing fundamentally different architectures that may be better suited to the kind of human-level generalization required for true AGI. Goertzel remains optimistic that AGI could emerge within the next few years, but he believes it will likely require moving beyond simply scaling current LLMs.
