Untitled

---

title: "Language, Probability, and Cognition – the technological equation in the history of humanity’s language modeling" description: "Explore the grand evolutionary journey of language models, where language, probability, and cognition intertwine throughout the history of technology: from simple statistical N-gram models, to neural networks and RNNs, then breakthroughs with LSTM, Seq2Seq, and Attention; from the Transformer revolution paving the way for BERT, GPT, and countless variants, to the era of LLMs with capabilities for creativity, reasoning, and multimodality. This is not just a technical story, but also a philosophical epic about how machines reflect human thought and hint at the vision of artificial general intelligence." date: "2025-09-21" author: "Vu Hoang" category: "LLMs" tags: [ "natural-language-processing", "large-language-models", "transformers", "statistics", "mathematics", ] image: "/images/blog/2025-09-21-evolution-of-language-models/hero.jpg" slug: "evolution-of-language-models" published: "published"

Language is not merely a means of communication; it is the first building block, the binding glue, the ever-flowing stream that shapes human intelligence and civilization. Since the earliest times, the ability to express, store, and transmit knowledge through language has distinguished humans from all other species, enabling us to build complex cultures, sciences, and societies.

1. The Voice of Intelligence – Why do humans need machines that can understand and speak?

As we enter the digital era—an era where information explodes and computers become extensions of the human mind—a fundamental and challenging question arises:

-> How can a computer, a purely logical machine, understand, interpret, and even generate the rich, nuanced, and ever-changing natural language of humans?

This is not just a technical problem; it is a deep aspiration to extend human capabilities through technology. From this aspiration, the concept of the Language Model (LM) was born. At its core, an LM is a mathematical or machine learning system designed to estimate the probability of a sequence of words (e.g., a sentence, a paragraph), or to predict the most likely next word given a context.

Why do we need this probability? Because understanding language often reduces to predicting what is most likely to occur. For example:

• In machine translation, we need to find the most probable target-language sequence corresponding to a source sentence.
• In speech recognition, we determine the most probable word sequence that matches the observed audio.
• In predictive typing, we want the machine to suggest the most suitable next word.
• In information retrieval, we want systems to understand query intent and return the most relevant results.

Each step in the evolution of LMs is not just a technological innovation, but an effort to overcome inherent limitations and open new horizons for human–computer interaction.

---

2. The Statistical Era: N-gram – Early foundations from discrete language (1980s – 2000s)

2.1. Motivation: Timely and pragmatic

In the 1980s and 1990s, computational resources were extremely limited. Researchers in Natural Language Processing (NLP) chose a pragmatic and effective approach: statistics. The central idea was that language, despite its complexity, could be modeled through probabilistic patterns hidden in large datasets. If we could count how often words and sequences co-occur, we could "predict" language behavior.

The initial problem was how to estimate $P(w_1, w_2, ..., w_T)$, the probability of a word sequence. This is difficult because the space of possible sequences is infinite. The solution was to apply the Markov assumption, which states that the probability of a word depends only on a limited number of preceding words.

2.2. N-gram Model: The power of simplicity

With the Markov assumption, the probability of a sequence is approximated as the product of conditional probabilities:

$$P(w_1, w_2, ..., w_T) \approx \prod_{t=1}^T P(w_t | w_{t-n+1}, ..., w_{t-1})$$

Where:

• $w_t$: the $t$-th word in the sequence.
• $T$: the length of the sequence.
• $n$: the number of words in the context (n-gram window).
• $P(w_t \mid w_{t-n+1}, ..., w_{t-1})$: the probability of word $w_t$ given the previous $n-1$ words.

N-gram illustration: from Unigram to Trigram in probabilistic language modeling.

Here, N is the number of context words considered to predict the next word.

• Unigram (N=1): $P(w_t)$ – Each word is treated independently, ignoring context entirely. (Very simple, low usefulness)
• Bigram (N=2): $P(w_t | w_{t-1})$ – The probability of $w_t$ depends only on the previous word.
• Trigram (N=3): $P(w_t | w_{t-2}, w_{t-1})$ – The probability depends on the two preceding words.

Example: To predict the next word in "I eat rice with...", a Bigram model only looks at "with" to predict (e.g., "chopsticks", "spoon"). A Trigram model considers "rice with" for a more accurate prediction.

The probabilities are estimated by counting the frequency of n-grams in a large training dataset.

2.3. Strengths and weaknesses: The limits of short-sightedness

Advantages:

• Simple and easy to implement: Requires minimal computational resources and expertise.
• Surprisingly effective: Works well for early applications like speech recognition or spelling correction where local context suffices.
• Interpretable: Probabilities of word pairs or triples are easy to inspect.

Disadvantages and driving issues:

• Short and local context: N-grams are severely "short-sighted." They cannot capture long-range semantic or syntactic relationships.
• Sparsity problem: As N increases, the number of possible n-grams grows exponentially, leading to many unseen combinations (zero-frequency problem).
• Vocabulary explosion: Each word is treated independently, with no semantic relationships.
• No semantic understanding: N-grams only count frequencies; they do not learn meaning.

These limitations created a need for more flexible language modeling capable of capturing longer context and deeper meaning.

---

3. Neural Language Models – The first wave of deep learning in language (Early 2000s)

3.1. Motivation: From discrete symbols to numerical meaning

N-grams treat words as independent symbols. However, humans understand that words like "car" and "automobile" are semantically related. This suggests that words can exist in a semantic space where distance reflects meaning similarity.

To enable machines to capture this, we need a way to represent words in a continuous space where similar words are close mathematically.

Vector space representing word meanings.

3.2. Bengio et al.’s breakthrough (2003)

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin introduced “A Neural Probabilistic Language Model” in 2003, laying the foundation for deep learning in NLP.

Key innovations:

• Word Embeddings: Each word is mapped to a dense vector in a low-dimensional space.
• Neural Network (MLP): A feedforward network predicts the next word using embeddings of context words.

Significance:

• Distributed representation: Word features are spread across dimensions.
• Semantic understanding: Embeddings capture relationships (e.g., "king" - "man" + "woman" ≈ "queen").
• Reduced sparsity: Even unseen n-grams can be inferred via embeddings.

3.3. Limitations

• Fixed context window.
• High computational cost (softmax over large vocabulary).
• Limited sequence modeling capability.

---

4. Recurrent Neural Network (RNN): Sequential memory (1990s – 2010)

4.1. Motivation

Language is sequential, and earlier words influence later ones. Models needed memory across time.

4.2. Mechanism

RNN introduces recurrence:

$$h_t = f(W_{xh}x_t + W_{hh}h_{t-1} + b_h)$$ $$y_t = W_{hy}h_t + b_y$$

RNN architecture with sequential inputs and hidden state loop.

4.3. Strengths and weaknesses

Advantages:

• Handles variable-length sequences.
• Captures sequential dependencies.
• Shares parameters across time.

Disadvantages:

• Vanishing/exploding gradients.
• Sequential computation (slow).
• Information compression bottleneck.

---

5. LSTM & GRU: Long-term memory with gates (Late 1990s – 2010)

5.1. Motivation

Need mechanisms to selectively remember and forget.

5.2. LSTM (1997)

Introduces gates and cell state:

• Cell state ($C_t$)
• Input, forget, output gates

LSTM cell with gating mechanisms.

5.3. GRU (2014)

Simplified LSTM with update and reset gates.

Advantages:

• Better long-term dependency learning.
• Strong performance in NLP tasks.
• Faster (GRU).

Limitations:

• Still sequential.
• Hard to scale for very long sequences.
• Bottleneck in Seq2Seq.

---

6. Seq2Seq: Sequence transformation breakthrough (Early 2010s)

6.1. Motivation

Transform one sequence into another.

6.2. Encoder-Decoder

• Encoder → context vector
• Decoder → output sequence

Basic Seq2Seq Encoder-Decoder architecture.

6.3. Bottleneck problem

Single fixed context vector → information loss.

---

7. Attention Mechanism: A revolution of focus (Mid 2010s)

7.1. Motivation

Allow flexible focus on relevant input parts.

7.2. Mechanism

$Attention(Q, K, V) = softmax(\frac{QK^T}{\sqrt{d_k}})V$

Attention allows focusing on relevant parts of input.

7.3. Impact

• Solves bottleneck.
• Improves translation.
• Enables interpretability.
• Foundation for Transformer.

---

8. Transformer: "Attention Is All You Need" (2017)

8.1. Motivation

Remove sequential dependency.

8.2. Architecture

• Self-Attention
• Multi-Head Attention
• Positional Encoding
• Feedforward layers

Transformer architecture with self-attention.

8.3. Pros & Cons

Advantages:

• Parallel computation.
• Captures long-range dependencies.
• Scalable.

Disadvantages:

• Quadratic complexity.
• High resource demand.

---

9. Transformer-based Models Era (2018 – Present)

9.1. BERT

• Encoder-only
• MLM, NSP
• Focus: understanding

9.2. GPT

• Decoder-only
• Autoregressive
• Focus: generation

9.3. Other models

T5, BART, RoBERTa, etc.

9.4. Beyond NLP

• ViT
• Multimodal models

---

10. LLM Era: Creative and general intelligence (2020s – Present)

10.1. Characteristics

• Massive scale
• Huge data
• Emergent abilities

10.2. Major players

• GPT
• LLaMA
• Claude
• Gemini

10.3. Impact

Transforming education, healthcare, programming, creativity.

---

11. Future: Toward AGI

1. Efficiency
2. Multimodality
3. External knowledge integration
4. Safety & ethics
5. Toward AGI

---

A grand journey, an infinite future

The evolution of language models is an epic story of relentless human effort to overcome limitations and create breakthroughs.

From N-gram → NNLM → RNN → LSTM/GRU → Seq2Seq → Attention → Transformer → LLMs.

Today, LLMs are no longer just predictors of words; they are becoming powerful "cognitive companions." The future will redefine how we learn, work, communicate, and even understand intelligence itself.