Model architecture

# Dimension Choices and foundational papers
1 Core architecture Transformer, Mamba / SSM, RWKV, Hybrid (Jamba)
2 Model topology Dense, Sparse MoE, Switch MoE
3 Sequence mixing Full attention, Sliding window, Sparse attention, Linear attention, Selective recurrence
4 Attention heads MHA, MQA, GQA, MLA
5 Positional encoding RoPE, ALiBi, Sinusoidal, learned absolute, relative, none
6 Normalization LayerNorm, RMSNorm, QK-Norm
7 FFN / activation GELU, SwiGLU / GEGLU, Squared ReLU
8 Model geometry Layers, hidden size, FFN width, attention heads, head dimension, expert count
9 Parameter sharing Tied embeddings, Cross-layer sharing, independent parameters
10 Context / memory KV cache, Windowed context, Recurrent state, Hybrid memory

Training and model development

# Dimension Choices and foundational papers
11 Training objective Autoregressive / seq2seq, Masked LM, Text-to-text / span corruption, Diffusion LM
12 Training stage Base / pretrained, Instruction / RLHF, DPO, reasoning-tuned
13 Tokenization Subword BPE, Unigram LM, SentencePiece, byte-level
14 Modality Text-only, Vision-language (LLaVA), Audio-language (Whisper), multimodal

Numerical representation

# Dimension Choices and foundational papers
15 Numerical precision FP32, FP16, BF16, FP8, INT8, INT4 (GPTQ), AWQ, FP4

Inference-specific extensions

These are not additional neural network architecture dimensions. They concern how a model is executed and served.

Dimension Techniques and papers
Attention kernels FlashAttention
KV cache management PagedAttention / vLLM
Decoding optimization Speculative decoding
Parameter-efficient tuning LoRA, QLoRA

Notes

  • Links are to introducing or representative papers, not necessarily the earliest historical use of each concept.
  • Dense, model geometry, and numerical formats are generally choices rather than distinct architectural inventions.
  • SentencePiece is a tokenizer framework, not a segmentation algorithm. It supports both BPE and Unigram models.
  • QK-Norm is applied to attention queries and keys; it is not simply an interchangeable replacement for LayerNorm or RMSNorm.
  • KV caching is an inference mechanism, not a trainable layer.
  • FlashAttention is an exact attention implementation, not another attention-head architecture like MHA or GQA.
  • FP8, INT8, INT4 and FP4 are formats or precision categories; GPTQ and AWQ are quantization methods.
  • SLMs generally use the same architectural dimensions as larger LLMs.