Evolution and Commercialization of OpenAI's GPT Models

Key Insight

GPT-2 was developed by training a 1.5 billion-parameter transformer on WebText, a dataset curated from 8 million Reddit upvoted links, an innovative method for sourcing high-quality content. This model demonstrated advanced capabilities, including writing essays, fan fiction, news articles, and even performing translation, suggesting nascent general intelligence behaviors. Internally, significant concerns arose about the potential for malicious applications, such as text deepfakes, prompting the decision not to release the full GPT-2 code but rather a smaller version. This cautious approach, driven by fears highlighted in a paper on 'The Malicious Use of Artificial Intelligence,' paradoxically led to public ridicule and negative media attention.

Despite initial public reception, GPT-2 was a technological breakthrough, outperforming existing systems and utilizing ten times more data than its predecessor. OpenAI then aimed to scale up by another order of magnitude for GPT-3. This ambition sparked internal discord, notably a conflict regarding project leadership which exposed a company rift. A key scientific breakthrough, detailed in a paper on 'scaling laws,' demonstrated that large language model performance consistently improves with increases in data, computational power, and neural network size. This insight was deemed crucial for fundraising, with its potential impact described as the 'most important discovery of the twenty-first century.'

GPT-3's training involved vast datasets, including over 1 trillion words from Common Crawl, Wikipedia, an updated WebText, and two book corpora, one of which (Books2, with approximately 300000 ebooks) faced legal challenges for its likely origin in illegal 'shadow libraries.' Featuring 175 billion parameters, more than one hundred times GPT-2, it could generate convincing poems, news articles, and computer code through 'few-shot' learning, requiring minimal examples. To secure funding, OpenAI developed an API prototype for GPT-3. Early efforts to attract users by approaching hundreds of startups in San Francisco were largely unsuccessful. However, Nick Walton, creator of AI Dungeon (a GPT-2 based fantasy game with over 1 million users in six weeks), became a crucial early tester. The OpenAI API launched on June 11, 2020, justified as both a financial necessity and a means to gather user data for safety, with its release restricted to a private beta.