Model · pythia-6-9b
Pythia 6.9B
Developer: EleutherAI
Availability: available · checked 2026-09-24 · source
Raw record: /data/models/pythia-6-9b.json
Fields
- id
- pythia-6-9b
- identifiers
- huggingface
- EleutherAI/pythia-6.9b
- developer
- EleutherAI
- release_date
- 2023-04-03partial · sourcenote: arXiv v1 date. Card: an earlier version of the suite was released first and all models were renamed in January 2023.
- weights_status
- open
- availability
- available · checked 2026-09-24 · source
- license
- Apache 2.0recorded · source
- architecture
- family
- decoder_only
- note
- Paper sec. 2.3: architecture 'largely follow[s] Brown et al. (2020)' (GPT-3) with deviations from recent practice incl. GPT-NeoX-20B; uses the GPT-NeoX-20B tokenizer and the GPT-NeoX library. One of 16 models (8 sizes x Pile / deduplicated Pile) trained on identical data in identical order, with 154 public checkpoints each.
- n_layers
- 32recorded · source
- hidden_size
- 4096recorded · source
- n_heads
- 32recorded · source
- vocab_size
- 50432recorded · source
- positional_encoding
- rotary (RoPE), partial: 25% of dimsrecorded · sourcenote: Paper sec. 2.3: 'We use rotary embeddings'; the 25% fraction is config rotary_pct 0.25.
- context_length
- 2048recorded · source
- training_data
- The Pile (EleutherAI's ~800GB curated English corpus), 300B tokens; a twin model was trained on a near-deduplicated Pile (~207B tokens).recorded · source
- techniques
- transformer-decoderrotary-position-embeddingparallel-attention-ffn
- primary_sources
- record_history
- date:2026-09-24 · change:created from primary sources (Phase 1 seed, batch 2) · by:wilson-pruitt + claude ·date:2026-09-24 · change:availability checked and recorded · by:wilson-pruitt + claude ·
Parents
Training data
- trained_on → The Pile declared source 300B tokens; a twin model was trained on the deduplicated Pile.
Design
- design_follows → GPT-3 (175B, 2020 paper) declared source Paper: 'Our model architecture and hyperparameters largely follow Brown et al. (2020)'.
Children
No edges recorded.
Read in
No station on the reading path has touched this record yet.