AI Learning Roadmap

Track 01

Non-technical audience

You don't need to become an engineer. You need enough fluency to read what AI tools give you, ask for what you want, and put agents to work.

01 · Languages

Languages

01 PythonThe one language that gets you everywhere in AI. Readable, forgiving, and what every tool speaks.

02 · Tools

Tools

01 Learn JSON / XML / HTML structureThe shape of inputs and outputs. Once you can read these, you can read what any tool or API hands back.
02 Learn API requests with PostmanSend a request, read a response, no code required.
- API keys / tokens vs public keys: what each one is, which one you must keep secret.
03 Use AI everywhereEach model has a lane. Use more than one.
- Claude→coding
- Gemini→deep research
- ChatGPT→verification / review
- Grok→faster iteration, use with Grok Bot
- Cursor→auto mode, access to multiple models at once

03 · Concepts

Concepts

01 LLMLarge language models. Learn to shape them instead of just prompting them.
- Write your own skill: package a repeatable instruction so the model does it the same way every time.
- Set your own context: give the chat the background, files, and rules it needs before you ask.
02 AgentsModels that take actions, not just answer.
- Spin up background agents to automate a task or a whole workflow while you do something else.

04 · Courses

Courses

01 DeepLearning.AIShort, practical courses on LLMs, agents, and prompting.
02 Andrew Ng's Machine LearningThe classic path into classical machine learning.
03 Stanford CS 106AProgramming methodology. A gentle, well-taught intro to writing code.
04 Harvard CS50Introduction to computer science. Free, rigorous, and famous for a reason.

Track 03

LLM expert

For people who build, train or serve models. Each topic maps to a component in the LLM Atlas, where you can rotate the whole system, follow a lifecycle mode and run the simulations.

01 · Data & tokenization

Data & tokenization

01 Corpora & governanceWhere training data comes from, what you are allowed to use, and how provenance is tracked.
- Licensing, opt-outs, takedowns
- Mixture weighting: which sources get upsampled and why.
02 Filtering, deduplication, contaminationMost of the quality of a base model is decided here.
- Quality filters: heuristic rules and classifiers.
- Exact + fuzzy dedup: hashing, MinHash / LSH.
- PII and secret scrubbing
- Benchmark contamination checks: n-gram overlap against evals.
03 TokenizersString → ids. Not a neural network, but it sets the cost of everything downstream.
- BPE vs unigram / SentencePiece, vocabulary size, byte fallback.
- Tokenizer decode vs generation decode: two different things called “decode”.
- Atlas: encode → prefill → decode walkthrough
04 Packing & shardingTurning documents into fixed-length training sequences that many loaders can read in parallel.

02 · Transformer internals

Transformer internals

01 Embeddings & positionToken embeddings, RoPE, and how context length extension works.
02 AttentionQ/K/V projections, causal masking, and why the number of KV heads matters for serving.
- MHA vs GQA vs MQA
- FlashAttention: exact attention with less memory traffic, not a different objective.
- Atlas: decoder block explorer
03 Residual stream, norms, MLPPre-norm RMSNorm, SwiGLU, and the shared vector every layer reads from and adds to.
04 LM head & logitsTraining uses every position; decoding only needs the last one.

03 · Mixture of experts

Mixture of experts

01 RoutingRouter logits, top-k selection, permutation, dispatch, weighted combine.
- Total vs active parameters: capacity grows with experts, per-token compute with k.
- Atlas: MoE router simulator
02 Load balancingCapacity factors, auxiliary losses, bias adjustment, and what happens to dropped tokens.
03 Expert parallelismExperts live on different devices; all-to-all communication is the cost.
- EP vs TP: whole experts vs sliced matrices.
- Placement and hot-expert replication

04 · Pre-training at scale

Pre-training at scale

01 One training stepForward, shifted labels, cross-entropy, backward, gradient collectives, AdamW.
- Global batch, microbatch, gradient accumulation
- Mixed precision: BF16 / FP16 / FP8 and the ~16 bytes per parameter that come with AdamW.
- Gradient clipping, warmup, weight decay
- Atlas: training-step calculator
02 Parallelism strategiesEach one partitions a different dimension; real systems compose several.
- DP batch · TP tensors · PP depth · CP sequence · EP experts · FSDP / ZeRO optimizer state
- Atlas: 8-GPU parallelism explorer
03 Memory & efficiencyActivation checkpointing, MFU, data-loader stalls, network bottlenecks.
04 Checkpointing & recoverySharded, asynchronous saves that include optimizer state and loader position.

05 · Post-training

Post-training

01 SFT & the annotation pipelineTaxonomy, rubrics, calibration, agreement, adjudication, chat template, loss masking.
- Quality vs quantity, model-assisted and synthetic demonstrations.
- Catastrophic forgetting and replay mixing.
- Atlas: SFT pipeline
02 LoRA & QLoRAW′ = W + (α/r)·B·A. Cheap to train and store; not automatically faster to serve.
03 Preference optimizationSame pairs, two paths: reward model + PPO-style RLHF, or DPO-family losses.
- Reference policy and KL regularization
- Reward hacking, verbosity bias, judge bias, sycophancy, over-refusal
04 Safety, red teaming, evaluationHeld-out human evals, LLM-as-judge with its biases, contamination control.

06 · Inference engine

Inference engine

01 Prefill vs decodeOne parallel pass that sets TTFT, then a bandwidth-bound loop that sets inter-token latency.
- Chunked prefill: keep decodes responsive while long prompts arrive.
- Atlas: batching visualizer
02 KV cache & paged memory2 × layers × tokens × KV heads × head dim × bytes, per sequence.
- PagedAttention-style block tables, fragmentation, eviction, offloading.
- KV-cache quantization (distinct from weight quantization).
- Atlas: KV cache calculator
03 Prefix cachingCross-request reuse of prompt-prefix KV blocks. Cuts prefill; does not touch decode.
- Atlas: prefix cache simulator
04 Continuous batchingBatch membership changes at every decode iteration. Not the same as dynamic batching.
05 Decoding policies & speculative decodingGreedy, sampling, beam, structured; then draft + verify for cheaper tokens.
- Speedup depends on draft cost, candidate length, acceptance rate, batch size.
- Atlas: speculative decoding visualizer
06 QuantizationBF16 → FP8 → INT8 → INT4. Smaller weights, not proportionally faster.
- Weight-only vs weight + activation, calibration, GPTQ / AWQ style, QAT.
- Atlas: weight memory calculator

07 · Serving & observability

Serving & observability

01 The serving planeGateway → model router → queue / admission → worker → stream → telemetry.
02 Parallel & disaggregated inferenceTP / PP / EP at serving time, and separate prefill and decode pools with KV transfer in between.
03 Fleet operationsMulti-LoRA, token-aware autoscaling, backpressure, canary releases, CUDA Graphs and compiled kernels.
04 Metrics that matterQueue time, TTFT, TPOT / ITL, throughput, KV utilization, prefix hit rate, batch occupancy, cost per task.
- GPU utilization only means something next to memory bandwidth, kernel occupancy and communication.
- Atlas: metrics glossary

08 · Primary sources

Primary sources

01 ArchitectureThe papers behind the block.
02 Training & post-training
- ZeRO · Megatron Core parallelism guide
- InstructGPT (RLHF) · DPO
- LoRA · QLoRA
03 Inference & serving

Have feedback?

This list is intentionally short and will change. Tell me what's missing, what's wrong, or what helped you most.

Send feedback →

Non-technical audience

Languages

Tools

Concepts

Courses

Technical audience

Must-know tools

Concepts

LLM expert

Data & tokenization

Transformer internals

Mixture of experts

Pre-training at scale

Post-training

Inference engine

Serving & observability

Primary sources

Have feedback?