🔄 RNN — Recurrent Neural Networks

Real-world scenario: You're reading the sentence "The animal didn't cross the street because it was too tired." What does "it" refer to? You know immediately — "the animal." You didn't re-read the sentence from the start; your brain carried a running summary forward as you read. RNNs were the first neural networks designed to do exactly this — maintain a "memory" while processing a sequence one step at a time.

🌟 Why RNNs Were Needed

Before RNNs, neural networks took fixed-size inputs and produced fixed-size outputs. A feedforward network can't process a sentence of variable length, and it has no concept of order — it treats every input element independently.

Language is fundamentally sequential. The meaning of a word depends on what came before it. RNNs introduced the concept of processing a sequence step by step, passing a summary of everything seen so far (the hidden state) from one step to the next.

Sequential Processing

An RNN processes a sequence one element at a time. At each time step t, it reads the current input x_t and the previous hidden state h_{t-1}, and produces a new hidden state h_t:

h_t = tanh(W_h · h_{t-1} + W_x · x_t + b)

import torch
import torch.nn as nn

rnn = nn.RNNCell(input_size=4, hidden_size=8)

# Simulate processing a 5-token sequence
h = torch.zeros(1, 8)           # initial hidden state
sequence = torch.randn(5, 1, 4) # 5 tokens, batch=1, embed_dim=4

for t, x in enumerate(sequence):
    h = rnn(x, h)
    print(f"Step {t}: hidden state shape = {h.shape}")
# Each step: hidden state shape = torch.Size([1, 8])

The hidden state at each step encodes a compressed summary of everything the RNN has seen so far.

Hidden State

The hidden state h_t is the RNN's "working memory." It's a fixed-size vector that gets updated at every time step. After processing the entire sequence, the final hidden state h_T is meant to capture the "meaning" of the whole input.

In practice, this is both RNNs' strength and their weakness. The hidden state is fixed-size no matter how long the sequence is. A sentence of 5 words and a paragraph of 500 words both compress into the same size vector. For long sequences, early information gets overwritten — the hidden state at step 500 has mostly "forgotten" what was at step 1.

import torch
import torch.nn as nn

# Process a sequence of 100 steps — early info gets compressed/lost
rnn = nn.RNN(input_size=16, hidden_size=32, batch_first=True)
x = torch.randn(1, 100, 16)    # batch=1, seq_len=100, features=16
output, h_final = rnn(x)

print(f"All hidden states: {output.shape}")   # (1, 100, 32)
print(f"Final hidden state: {h_final.shape}") # (1, 1, 32) — just the last step

Memory

RNNs have two kinds of "memory":

1. Within a sequence: the hidden state carries information forward step by step
2. Across training: the weight matrices W_h and W_x are updated by gradient descent and encode patterns learned from all training examples

The within-sequence memory is what makes RNNs useful for language tasks. But it's short-term memory — not long-term memory. The further back in the sequence something occurred, the less influence it has on the current hidden state.

# Demonstrating memory decay visually:
rnn_cell = nn.RNNCell(input_size=1, hidden_size=1)
nn.init.constant_(rnn_cell.weight_hh, 0.5)  # small recurrent weight → fast forgetting
nn.init.constant_(rnn_cell.weight_ih, 1.0)
nn.init.constant_(rnn_cell.bias_hh, 0.0)
nn.init.constant_(rnn_cell.bias_ih, 0.0)

# Inject a "signal" at step 0, then zeros
h = torch.zeros(1, 1)
inputs = [torch.tensor([[1.0]])] + [torch.tensor([[0.0]])] * 9

for t, x in enumerate(inputs):
    h = rnn_cell(x, h)
    print(f"Step {t}: h = {h.item():.4f}")
# Step 0: h = 0.7616   (tanh of the signal)
# Step 1: h = 0.3793
# Step 2: h = 0.1849
# ...  signal decays rapidly
# Step 9: h = 0.0009   (almost gone)

Forward Pass

A complete RNN forward pass processes the entire sequence and returns either:

• All hidden states (one per time step) — for tasks like translation where you need per-token context
• Only the final hidden state — for tasks like sentiment classification where you need a single summary

import torch
import torch.nn as nn

rnn = nn.RNN(
    input_size=16,
    hidden_size=64,
    num_layers=2,         # stacked RNN layers
    batch_first=True,
    dropout=0.2,
)

# Input: batch of 32 sentences, each 20 tokens, each token embedded as 16-dim vector
x = torch.randn(32, 20, 16)
all_hidden, final_hidden = rnn(x)

print(f"All hidden states: {all_hidden.shape}")    # (32, 20, 64)
print(f"Final hidden state: {final_hidden.shape}") # (2, 32, 64) — 2 layers

# For classification, use only the last time step:
logits = nn.Linear(64, 2)(all_hidden[:, -1, :])    # binary classification

Vanishing Gradients

This is the critical failure of vanilla RNNs. During backpropagation, gradients must flow backwards through every time step. At each step, the gradient is multiplied by the weight matrix W_h. If the values in W_h are less than 1, the gradient shrinks exponentially as it travels back through time.

After 50 steps, the gradient at step 1 might be reduced by a factor of (0.9)^50 ≈ 0.005. The network can no longer learn from information that occurred far back in the sequence. This is the vanishing gradient problem.

Gradient at step t = Gradient at step T × ∏(dh_i/dh_{i-1})  for i=T..t
                   = Gradient at step T × (W_h)^(T-t)

If |W_h| < 1 → (W_h)^(T-t) → 0 as (T-t) increases

The reverse is also possible: if values in W_h are greater than 1, gradients grow exponentially (exploding gradients), causing training instability. Gradient clipping addresses the explosion problem; the vanishing problem required a fundamentally different architecture — the LSTM.

Long-Term Dependency Problem

The vanishing gradient problem manifests as an inability to learn long-term dependencies — relationships between elements that are far apart in a sequence.

Example:

"The books on the table near the window ___ old."

The verb must agree with "books" (plural → "are"), but "books" is 7 tokens before the blank. By the time the RNN reaches the blank, its hidden state has been updated 7 times and the gradient connecting the blank to "books" is nearly zero. The RNN tends to match the verb to the nearest noun ("window") instead.

This is precisely the problem that LSTM gates (forget, input, output) were designed to solve — by letting the network explicitly choose what to remember and what to forget, rather than letting gradients determine it implicitly.