The Evolution and Re-emergence of Neural Networks in AI

Key Insight

In the 1990s and early 2000s, Artificial Intelligence was characterized by a prevailing skepticism towards neural networks, with most research focusing on rule-based systems. Projects like Cyc, launched in the mid-1980s by Doug Lenat, aimed to create a 'common sense machine' by programming millions of rules, which ultimately highlighted the 'herculean nature' and 'Sisyphean' challenge of hand-coding intelligence, with Lenat estimating it would take 2000 human years of labor. Rule-based AI achieved notable successes, such as the ALICE chatbot in 1995, which impressed with its 'eerily human' scripted responses, and IBM Deep Blue's victory over world chess champion Garry Kasparov in 1997, a high point for hard-coded computing that, in hindsight, was seen as a 'death knell' for that approach.

Despite widespread academic hostility and the challenging timing of AI's 'first deep winter' in the late 1970s, Geoff Hinton, a descendant of George Boole, maintained an unwavering belief in neural networks, describing them as 'completely obvious.' His breakthrough came with a postgraduate fellowship at UCSD, and later at Carnegie Mellon, where he and a colleague developed the Boltzmann machine, a multilayered network representing a 'giant leap forward' beyond earlier single-layer perceptrons, as it began to more closely resemble the workings of the human brain by learning from data. Faced with deep-seated prejudice against neural nets, Hinton and fellow pioneers like Yoshua Bengio and Andrew Ng, rebranded their work with terms such as 'machine learning' and 'connectionism,' eventually popularizing 'deep learning,' a term coined by Rina Dechter in 1986, to overcome the academic resistance.

Before 2010, neural networks showed early promise in various applications, even if they were dismissed as 'baby steps' by investors. Carnegie Mellon's AI lab, for instance, developed an autonomous Chevy that successfully navigated 125 miles from Pittsburgh to Erie in 1995 without human intervention by learning from visual data. Yann LeCun, an acolyte of Hinton, created a program modeled on the visual cortex, trained on U.S. Postal Service envelopes, leading to a device sold to banks for reading handwritten checks by the mid-1990s. Fei-Fei Li's ImageNet, trained on nearly nine million labeled images, outperformed rule-based computer vision, and Terry Sejnowski's NETtalk learned to read full sentences aloud within a week. However, these advancements often failed to attract investor interest due to previous 'AI winters' and unfulfilled expectations, with even true believers like LeCun and Hinton questioning whether sufficient computing power would ever exist to make large neural nets truly useful, a question answered as the 2010s marked a turning point with powerful Nvidia GPUs and exponential data from the internet, ushering in the 'Golden Decade' for AI.