Model · gpt2-xl
GPT-2 XL (1.5B)
Developer: OpenAI
Availability: available · checked 2026-09-24 · source
Raw record: /data/models/gpt2-xl.json
Fields
- id
- gpt2-xl
- identifiers
- huggingface
- openai-community/gpt2-xlgpt2-xl
- developer
- OpenAI
- release_date
- 2019-11recorded · source
- weights_status
- open
- availability
- available · checked 2026-09-24 · source
- license
- Modified MIT Licenserecorded · source
- architecture
- family
- decoder_only
- n_layers
- 48recorded · source
- hidden_size
- 1600recorded · source
- n_heads
- 25recorded · source
- vocab_size
- 50257recorded · source
- positional_encoding
- learned_absoluterecorded · sourcenote: GPT-2 paper says the model 'largely follows the details of the OpenAI GPT model'; it is the GPT-1 paper that states 'learned position embeddings'. Code: wpe table in modeling_gpt2.py.
- context_length
- 1024recorded · source
- normalization
- LayerNorm, pre-norm (at the input of each sub-block) plus a final LayerNormrecorded · source
- activation
- GELU (tanh approximation, config 'gelu_new')recorded · sourcenote: GELU choice stated in the GPT-1 paper, which GPT-2 follows.
- ffn_hidden_size
- 6400recorded · sourcenote: Config n_inner is null; code defaults to 4 x hidden size.
- training_data
- WebText: ~8M documents scraped from outbound links on Reddit posts with karma >= 3. Not publicly released as a dataset dump.partial · source
- techniques
- transformer-decoderbyte-pair-encoding
- primary_sources
- record_history
- date:2026-09-24 · change:created from primary sources (paper, GitHub repo, HF config.json) · by:wilson-pruitt + claude ·date:2026-09-24 · change:positional-encoding source corrected to GPT-1 paper; normalization, activation, ffn width added (narrative/exercises/01-gpt2-xl-vs-mistral-7b.md) · by:wilson-pruitt + claude ·date:2026-09-24 · change:availability checked and recorded · by:wilson-pruitt + claude ·
Parents
Training data
Children
Design
- ← same_architecture_retrained GPT-3 (175B, 2020 paper) declared source Paper: 'the same model and architecture as GPT-2', except alternating dense and locally banded sparse attention. gpt2-xl stands in for the GPT-2 family.