Early AI Model Development and Safety Challenges

Key Insight

OpenAI encountered significant safety and ethical challenges with its early models. During the GPT-3 beta test, instances of illicit content, specifically fantasies involving sex with children, emerged on AI Dungeon, a text-based fantasy game using the model. This led to a content moderation system that users found intrusive and clumsy, flagging innocuous phrases like '8-year-old laptop,' causing subscription cancellations from thousands of users paying 10 to 50 dollars monthly. A key challenge was that the AI itself, trained on vast internet data, sometimes generated the problematic content.

The cost of running GPT-3 proved prohibitive for AI Dungeon, which despite thousands of users, found the AI's operational cost too high to sustain a business, requiring a GPU cluster equivalent to a Tesla's price. Separately, OpenAI developed DALL-E in early 2021, an image generator, initially producing simple cartoon-like images. Subsequent models focused on improving behavior; InstructGPT (January 2022) used reinforcement learning from human feedback (RLHF) to teach the AI human-desired conduct, responding politely even to biased questions, and was later made the default API model.

Further efforts to refine models included WebGPT, a project to teach models to use a web browser for fact-checking and prevent hallucinations, particularly as transformer models operate as 'black boxes.' This project concluded in late 2021, shifting focus to conversational models as an alignment tool. By GPT-3.5, an upgrade released in late 2022, the 'aligned' model significantly improved sales, indicating customer preference for models that reliably executed desired tasks. The goal was for models to do 'what we want, not saying false things or worse, deliberately trying to trick us.'