AI

November 30, 2022 · ChatGPT arrives

On Wednesday November 30th, 2022, ChatGPT was announced, and the world has not been the same since.

Before ChatGPT, for most of us, AI was a term of Science Fiction or something that could play a good game of Chess.

And then ChatGPT came along and suddenly, anyone – anywhere – could talk with it (in text), ask questions, get useful answers, and it understood you and could follow your chain of thought.

It also knew a lot about every topic under the sun!  It could write, create stories, tell jokes, write in the voice of a favorite author, write in different languages, translate, and even write computer code!

And with that, the revolution began.

A revolution, but one word at a time!^†

When one asks ChatGPT a question, the answer appears gradually as if someone were typing it.  But that isn’t just a display trick.

Ask ChatGPT:

What is a typo?

And it does not come up with the answer and then display it.

Instead, its task is far more limited, it asks itself:

What is the most likely next word?

It displays the chosen word as the beginning of an answer.

What is a typo?
A

It has not found a full answer; it has only found the first word of an answer.

It then repeats the process, over and over, each time asking: What is the most likely next word?

What is a typo?
A typo

And then again:

What is a typo?
A typo is

and again:

What is a typo?
A typo is a

and again:

What is a typo?
A typo is a mistake

This continues until the most likely next word isn’t a word at all, but is an indication that the answer is complete.  Then it stops.

What is a typo?
A typo is a mistake made while typing text, usually caused by a slip of the fingers on a keyboard or screen.

That this approach produces great results is a surprise to everyone involved.

To add variety, it actually doesn’t always pick the most likely next word, but randomly selects one of the most likely next words.  You can ask the same question twice, exactly the same way, and you will likely get slightly different results each time.  This is why.

This isn’t just how ChatGPT works.  Other well-known Large Language Models (LLMs), including Claude from Anthropic, Gemini from Google, Llama from Meta, and DeepSeek, use this same basic approach.

Context Window and Memory

The question and its growing answer make up the context window.  When the answer stops being generated, and one asks a new question, say "Give me some examples.", the original typo question, its answer, and the new question are all taken in with the goal of once again finding the next likely word, and then the next, and the next, until the next stop.

Through this ever growing context window, the AI understands new questions in the context of the previous questions and answers.  This creates a form of memory where each new word generated not only is in response to the words of the immediate question, but is informed by the words of the entire series of questions and answers that came before.

This ability to draw on the preceding words of the question, and the words of the earlier questions and answers, comes from the AI’s attention mechanism.

Remarkably, this too operates at the word level, influencing the understanding of the words of the question in the context of the other words in the question and in prior questions and answers.

Generative Pre-trained Transformer

GPT, from ChatGPT, stands for:

Generative means it produces text, building a response one word at a time; technically, one token at a time, where a token can be a word, part of a word, or punctuation.

Pre-trained means it first learns patterns from large amounts of text, before post-training adapts it for tasks such as answering questions and following instructions.

Transformer is the neural-network architecture it uses; its attention mechanism helps it relate different parts of the text, such as connecting a pronoun with the person to whom it refers, or connecting a follow-up question to an earlier topic.

ChatGPT is an application allowing one to interact with an AI model in a conversational style.  The original version used the GPT-3.5 model which is an example of a Large Language Model (LLM).  LLMs are trained on vast amounts of text so that they can process and generate language.  Most other LLMs also share the generative, pre-trained, transformer architecture even if they do not use it in their names.

How much text was it trained on?

The original ChatGPT was based upon the GPT-3.5 model.  In a night-before-the-launch decision, the name was changed from “Chat with GPT 3.5” to simply “ChatGPT”.  The 3.5 model was based upon GPT 3.0 with additions for coding.  OpenAI hasn’t disclosed GPT-3.5’s exact training data, but GPT-3 had been trained on:

Around 225 billion words of text (300 billion tokens):

- 60% from a crawl of billions of web pages

- 22% from WebText2, a high-quality dataset covering highly upvoted content from Reddit.

- 16% from Book1 & Book2, two long-form Internet book repositories

- 3% from Wikipedia

The increment to GPT-3.5 included an undisclosed amount of high-quality computer code from open sources such as GitHub which heavily improved the model’s logic, reasoning, and programming capabilities.

Learn more: OpenAI's launch announcement · GPT-3 training data (pages 8–9) · What are tokens?

1966 · The illusion of understanding

Fifty six years before ChatGPT, there was ELIZA.

ELIZA illustrates how engaging text conversations can be and how easily the illusion of understanding can be produced.

ELIZA is your psychotherapist, try it out:

Pattern:  I am [something]
Reply:    How long have you been [something]?

Pattern:  My [something]
Reply:    Your [something]?
          Change “me” to “you.”

You:      I am tired.
ELIZA:    How long have you been tired?
You:      For weeks. My work keeps me awake.
ELIZA:    Your work keeps you awake?

Browser implementation: elizabot.js by Norbert Landsteiner (2005), based on Weizenbaum's ELIZA and DOCTOR rules.  This port includes some later additions.  Read about restoring the original ELIZA.

1955–1956 · Giving a field its name

In 1955, a summer research project at Dartmouth College was proposed.  The title was:

A PROPOSAL FOR THE
DARTMOUTH SUMMER RESEARCH PROJECT
ON ARTIFICIAL INTELLIGENCE

This is where the new field got its name.  The meeting itself took place in the summer of 1956, in Hanover, New Hampshire.

The Logic Theorist

At the 1956 meeting, the Logic Theorist was introduced.  In its early tests, it was able to prove 38 of 52 theorms drawn from the important book Principia Mathematica.  This is an early example of Symbolic AI, which represents information using symbols and applies explicit rules to reason about them.

The new field of AI was off to a strong start.

Logic Theorist used heuristics — rules of thumb that guided its search toward promising proofs.  Its successor, GPS (General Problem Solver), generalized this into means-ends analysis which is still important today.

Sources: McCarthy's account of Dartmouth · Newell, Shaw, and Simon's 1957 report · Stanford's history of AI.

Means–ends analysis

Means-ends analysis involves choosing actions that brings one closer to a goal (end), then setting subgoals (means) when needed.

Example:

1. Goal: Print the document.
2. Notice a problem: The printer is out of paper.
3. Subgoal: Load paper into the printer.
4. Complete the goal: Print the document.

In order to reach the goal, subgoals must be determined and resolved first.

Learn more: Newell, Shaw, and Simon's GPS report · Peter Norvig's GPS explanation · How Large Reasoning Models Solve Problems: A Mechanistic Study on Tower of Hanoi.

To appreciate the initial advancements in Artificial Intelligence, it is worth remembering just how new, and few, computers were in the 1950s.  The first commercial electronic computer was only just delivered in 1951.  Across the entire decade, there were only a relative handful sold:

In today’s dollars, the IBM 650, the inexpensive system on the list, would cost around $2.5 million and occupy as much space as a small industrial kitchen.  The other systems were in the $10 million to $25 million range.  And even the most advanced, the IBM 704, had processing power and memory capacity that a smart-card chip on a credit card in your wallet can match or exceed.

Programming Languages

There were hardly any programming languages in the early 1950s.  Most computers were programmed using their native machine languages or labeled versions of machine language called assembly language.

New programming languages arrived toward the end of the decade:

• Fortran (Formula Translation), released in 1957, made scientific and mathematical calculations easier to express.
• Cobol (Common Business-Oriented Language), designed in 1959, aimed at readable business programs that could run on different manufacturers' computers.
• Lisp (List Processing), developed by John McCarthy (from the Dartmouth Summer project) beginning in 1958, made symbolic expressions and lists central to programming.

Sources: IBM's Fortran history · Cobol manual and history · McCarthy's history of Lisp.

Just as we drew inspiration from birds for flight, we have drawn inspiration from the human brain for AI's neural networks (Artificial Neural Networks).

The biological neuron

The artificial neuron

An adult human brain contains some 86 billion neurons.  Cortical neurons typically have around 10,000 incoming synaptic connections and a similar number of outgoing connections.  Altogether, there are hundreds of trillions of synaptic connections in the brain.

From this biological inspiration, artificial neural networks (ANNs) have borrowed these ideas:

• Neurons receive and combine signals
• Connections are of various strengths
• Learning changes those strengths
• Output is determined by a threshold or function of combined inputs.

Neurons made of Arithmetic: The Perceptron

In 1957, Frank Rosenblatt introduced a neural network he called the perceptron with three key features:

Weights.  Each input is multiplied by a weight representing the strength of its connection.  The network learns by adjusting these weights.

Learning Rule.  The perceptron’s learning rule provided a systematic way to adjust weights using training examples, giving researchers a concrete, practical method for building learning systems.

Threshold.  Each neuron adds its weighted inputs and produces an on/off output depending on whether the sum reaches a threshold.  This threshold decision is a nonlinear activation function.

Single-Layer Perceptron

For a single-layer perceptron, it was shown mathematically that the learning rule would find weights that correctly classified all training data, for certain kinds of training data.

There was much excitement in the 1960s about what could be done with SLPs and what kind of pre or post processing could work around their limitations.

For instance, in “SLP Perceptron: Demo” that follows, we use an SLP with ten neurons to read handwritten digits.  After training on 10,000 examples, it is quite effective — give it a try!

Part of its effectiveness is that before your drawing is presented for recognition, it is scaled and centered to have a similar size and position to the training images.  If you turn off that scaling and centering (which you can do in the demo), it becomes much less effective.

Multi-Layer Perceptron

The perceptron lives on today in the multi-layer perceptron (MLP).  An MLP combines several trainable layers, with each layer feeding its results to the next.  The on/off thresholds have been replaced with a more general activation function, allowing MLPs to learn patterns that an SLP cannot.

The development of backpropagation provided a practical way to adjust weights across many layers, not just a single layer as with the original perceptron learning rule.

MLPs are a type of feed-forward network and a mainstay of modern AI, used extensively in LLMs and other deep learning systems.  Understanding the SLP and then the MLP teaches one a great deal about modern AI.

Sources: NIH's introduction to neurons · Synaptic development and learning · Artificial neurons and activation functions · The human brain’s neuron count · Cortical synaptic connections · Rosenblatt's Principles of Neurodynamics · Learning representations by back-propagating errors · Feed-forward networks in the Transformer.

Early success in the field of AI led to a great deal of initial enthusiasm and research funding.  This even spilled over to popular culture, where movies about the dangers of AI appeared, such as 2001: A Space Odyssey (1968) and Colossus: The Forbin Project (1970).

But by the late 1960s and early 1970s it was becoming clear that these early successes were hard to build upon and many techniques did not scale up.  One important critique came in the 1969 book Perceptrons by Marvin Minsky and Seymour Papert, offering proofs of the limitations of single-layer and certain other perceptrons.

By the mid-1970s, enthusiasm and research funding had declined significantly.  This period is called the AI Winter, or the first AI Winter.  A second began in the late 1980s, as the promise of expert systems also failed to materialize.

1986 · Teaching the hidden layers

While single-layer perceptron neural networks were limited, multi-layer perceptrons offered greater promise.  But how does one train them?

For single-layer perceptrons, the learning rule gives a clear method to adjust the weights as successes and failures against the training data are seen.  But with multiple layers, where each neuron connects to every neuron in the next layer, how does one know what to adjust?

Single-Layer Perceptron

Multi-Layer Perceptron

This is the credit-assignment problem: when an answer is wrong, which connections should change, and by how much?

Learning from the error

Backpropagation techniques popularized in 1986 provided for MLPs what the Learning Rule had provided for SLPs.

Backpropagation uses the chain rule from calculus to determine how each connection weight influences the output, working backward from the output through the layers to guide adjustments to the weights.

Source: Rumelhart, Hinton, and Williams (1986).