Model · gpt-neo-2-7b
GPT-Neo 2.7B
Developer: EleutherAI
Availability: available · checked 2026-09-24 · source
Raw record: /data/models/gpt-neo-2-7b.json
Fields
- id
- gpt-neo-2-7b
- identifiers
- huggingface
- EleutherAI/gpt-neo-2.7B
- developer
- EleutherAI
- release_date
- 2021-03-21recorded · sourcenote: GitHub release v1.0 'Release Pretrained Models'.
- weights_status
- open
- availability
- available · checked 2026-09-24 · source
- license
- MITrecorded · source
- architecture
- family
- decoder_only
- note
- Card: 'designed using EleutherAI's replication of the GPT-3 architecture.' Alternates global and local (256-token window) attention layers (config attention_types, window_size).
- n_layers
- 32recorded · source
- hidden_size
- 2560recorded · source
- n_heads
- 20recorded · source
- vocab_size
- 50257recorded · source
- positional_encoding
- learned_absoluterecorded · sourcenote: Code: wpe = nn.Embedding(max_position_embeddings, ...).
- context_length
- 2048recorded · source
- training_data
- The Pile (EleutherAI's ~800GB curated English corpus); 420B tokens over 400,000 steps.recorded · source
- techniques
- transformer-decoderbyte-pair-encoding
- primary_sources
- record_history
- date:2026-09-24 · change:created from primary sources (Phase 1 seed, batch 2) · by:wilson-pruitt + claude ·date:2026-09-24 · change:availability checked and recorded · by:wilson-pruitt + claude ·
Parents
Training data
Design
- same_architecture_retrained → GPT-3 (175B, 2020 paper) declared source Card: 'designed using EleutherAI's replication of the GPT-3 architecture.' Like GPT-3, alternates dense (global) and local attention layers.
Children
No edges recorded.
Read in
No station on the reading path has touched this record yet.