Problem library
ML building blocks, one implementation at a time.
Practice the functions, model classes, and training loops inside production ML systems.
Stable Softmax
Convert logits into probabilities without overflow or underflow.
Masked Softmax
Normalize only valid logits while assigning masked positions zero probability.
Sinusoidal Positional Encoding
Build deterministic position vectors from paired sine and cosine frequencies.
Layer Normalization
Implement LayerNorm and compare its Pre-LN and Post-LN placement in a residual block.
SwiGLU Feed-Forward Block
Gate an up projection with stable SiLU before projecting back to model width.
Scaled Dot-Product Attention
Compute stable masked attention from query, key, and value matrices.
BPE Merge
Apply one deterministic byte-pair encoding merge across a token sequence.
BPE Tokenizer
Tokenize Unicode text with ranked BPE merges and encode the final symbols as vocabulary IDs.
Multi-Head Attention
Project, split, attend, concatenate, and mix multiple attention heads.
Grouped-Query Attention
Build a reusable layer that shares projected key-value heads across contiguous query-head groups.
Rotary Positional Encoding
Rotate paired query or key features with position-dependent frequencies.
KV Cache Update
Write newly projected keys and values into a reusable autoregressive cache.
Greedy Decoding
Select the highest-logit token at each generation step until EOS or a length limit.
Temperature Sampling
Sample autoregressive tokens from temperature-scaled logits with deterministic random draws.
Top-K Sampling
Sample autoregressive tokens after restricting each temperature-scaled distribution to its deterministic top-k set.
Top-P Sampling
Sample autoregressive tokens from the smallest deterministic nucleus that reaches a probability threshold.
PyTorch Causal Mask
Build a batched boolean decoder self-attention mask from causal order and key padding.
PyTorch Pairwise Distances
Use singleton dimensions and broadcasting to compute every row-pair distance.
PyTorch Batched Embedding Lookup
Look up token embeddings while preserving batch and sequence axes.
PyTorch Final Logits Selection
Select each padded sequence's last valid vocabulary-logit row for autoregressive decoding.
MMoE Task Routing
Mix shared expert outputs with an independent softmax gate for each task.
Sparse MoE Routing
Route each token to a capacity-limited top-k subset of experts.