Model architecture
| # | Dimension | Choices and foundational papers |
|---|---|---|
| 1 | Core architecture | Transformer, Mamba / SSM, RWKV, Hybrid (Jamba) |
| 2 | Model topology | Dense, Sparse MoE, Switch MoE |
| 3 | Sequence mixing | Full attention, Sliding window, Sparse attention, Linear attention, Selective recurrence |
| 4 | Attention heads | MHA, MQA, GQA, MLA |
| 5 | Positional encoding | RoPE, ALiBi, Sinusoidal, learned absolute, relative, none |
| 6 | Normalization | LayerNorm, RMSNorm, QK-Norm |
| 7 | FFN / activation | GELU, SwiGLU / GEGLU, Squared ReLU |
| 8 | Model geometry | Layers, hidden size, FFN width, attention heads, head dimension, expert count |
| 9 | Parameter sharing | Tied embeddings, Cross-layer sharing, independent parameters |
| 10 | Context / memory | KV cache, Windowed context, Recurrent state, Hybrid memory |
Training and model development
| # | Dimension | Choices and foundational papers |
|---|---|---|
| 11 | Training objective | Autoregressive / seq2seq, Masked LM, Text-to-text / span corruption, Diffusion LM |
| 12 | Training stage | Base / pretrained, Instruction / RLHF, DPO, reasoning-tuned |
| 13 | Tokenization | Subword BPE, Unigram LM, SentencePiece, byte-level |
| 14 | Modality | Text-only, Vision-language (LLaVA), Audio-language (Whisper), multimodal |
Numerical representation
| # | Dimension | Choices and foundational papers |
|---|---|---|
| 15 | Numerical precision | FP32, FP16, BF16, FP8, INT8, INT4 (GPTQ), AWQ, FP4 |
Inference-specific extensions
These are not additional neural network architecture dimensions. They concern how a model is executed and served.
| Dimension | Techniques and papers |
|---|---|
| Attention kernels | FlashAttention |
| KV cache management | PagedAttention / vLLM |
| Decoding optimization | Speculative decoding |
| Parameter-efficient tuning | LoRA, QLoRA |
Notes
- Links are to introducing or representative papers, not necessarily the earliest historical use of each concept.
- Dense, model geometry, and numerical formats are generally choices rather than distinct architectural inventions.
- SentencePiece is a tokenizer framework, not a segmentation algorithm. It supports both BPE and Unigram models.
- QK-Norm is applied to attention queries and keys; it is not simply an interchangeable replacement for LayerNorm or RMSNorm.
- KV caching is an inference mechanism, not a trainable layer.
- FlashAttention is an exact attention implementation, not another attention-head architecture like MHA or GQA.
- FP8, INT8, INT4 and FP4 are formats or precision categories; GPTQ and AWQ are quantization methods.
- SLMs generally use the same architectural dimensions as larger LLMs.