Text to Tokens - The Foundation
Deep dive into tokenization: why models can't read text directly, subword algorithms like BPE, practical patterns, and the pitfalls that cause production failures
ai-engineering Intermediate
AI Engineering Fundamentals
Deep dive into tokenization: why models can't read text directly, subword algorithms like BPE, practical patterns, and the pitfalls that cause production failures
8 articles
108 min total
Articles in this series
- 1
- 2
Tokens to Embeddings - Vectors That Capture Meaning
Deep dive into embeddings: why one-hot encoding fails, how meaning emerges from training, measuring similarity, and the difference between token and sentence embeddings
- 3
Embeddings to Attention - Relating Tokens to Each Other
Deep dive into attention mechanisms: why transformers replaced RNNs, scaled dot-product attention, multi-head attention, and how context length affects performance
- 4
Attention to Generation - Producing Text Token by Token
Deep dive into text generation: the generation pipeline, temperature and sampling, decoding strategies, and why deterministic generation doesn't exist
- 5
Generation to Retrieval - Grounding LLMs in Facts
Deep dive into retrieval: why pure generation hallucinates, vector similarity search, dense vs sparse retrieval, chunking strategies, and multi-stage retrieval with reranking
- 6
Retrieval to RAG - The Complete Pipeline
Deep dive into RAG: prompt construction, reranking, failure modes, the debugging decision tree, and how to diagnose when things go wrong
- 7
RAG to Agents - From Retrieval to Action
Deep dive into AI agents: the agent loop, tools, ReAct pattern, memory systems, when agents are wrong, and agent failure modes you'll encounter in production
- 8
Agents to Evaluation - Measuring What Matters
Deep dive into agent evaluation: the three dimensions (task completion, process quality, safety), evaluation strategies, building test suites, and production monitoring