Intermediate Course

Transformer Architecture Intermediate

49 lessons across 7 chapters. Every lesson is standalone — start anywhere.

49 lessons 7 chapters
Beginner Intermediate Advanced
Start Intermediate Course — Lesson 1 →
1 Scaling Laws 7 lessons
1
What scaling laws discovered
2
Compute, data, parameters: the three variables
3
The Chinchilla law: optimal training ratio
4
Why bigger is not always better
5
Emergent capabilities at scale
6
Return on investment of scale
7
Limits of Scaling
2 Modern Architecture Improvements 7 lessons
1
Grouped Query Attention: fewer KV heads
2
Sliding window attention: local context
3
RoPE: rotary positional embeddings
4
ALiBi: Attention with Linear Biases
5
Flash Attention: memory-efficient attention
6
Why these changes matter for long context
7
The compute-memory tradeoff
3 Mixture of Experts Architecture 7 lessons
1
What MoE is: routing to specialized sub-networks
2
Expert layers: structure and size
3
Router mechanism: token routing
4
Top-K routing: selecting active experts
5
Total vs active parameters
6
Why MoE achieves quality at lower compute
7
MoE models: Mixtral, Llama 4, Qwen3
4 Context Window Architecture 7 lessons
1
How context window is determined
2
Quadratic attention cost: the scaling wall
3
Techniques to extend context: RoPE scaling
4
KV cache: what it stores
5
KV cache memory requirements
6
Why 1M+ context is architecturally hard
7
Practical vs theoretical context limits
5 Training vs Inference Architecture 7 lessons
1
Training: all tokens processed in parallel
2
Autoregressive inference: one token at a time
3
KV cache: avoiding recomputation
4
Prefill vs decode phases
5
Batch effects on inference throughput
6
Why inference is memory-bandwidth bound
7
Speculative decoding intuition
6 Architecture Comparisons 7 lessons
1
GPT architecture: decoder stack
2
BERT architecture: encoder stack
3
T5 architecture: encoder-decoder
4
Llama architecture: modern decoder
5
Mistral architectural innovations
6
What makes architectures converge
7
Architecture choices that survive scaling
7 Common Misconceptions 7 lessons
1
Transformers do not have memory: correction
2
Bigger context equals better understanding: nuance
3
Attention equals the model understands: correction
4
Parameters equal intelligence: nuance
5
All tokens treated equally: correction
6
Training equals memorization: the nuance
7
Architecture vs training data: what matters more