Model · pythia-12b
pythia-12b
Developer: EleutherAI
Availability: available · checked 2026-09-24 · source
Raw record: /data/models/pythia-12b.json
Fields
- id
- pythia-12b
- identifiers
- huggingface
- EleutherAI/pythia-12b
- developer
- EleutherAI
- release_date
- 2023-02-28partial · sourcenote: HF repo creation date; may predate or postdate public release.
- weights_status
- open
- availability
- available · checked 2026-09-24 · source
- license
- apache-2.0partial · sourcenote: From HF card metadata (uploader-declared); confirm against the license text.
- architecture
- family
- decoder_only
- note
- family read from config 'architectures': ['GPTNeoXForCausalLM'].
- n_layers
- 36recorded · source
- hidden_size
- 5120recorded · source
- n_heads
- 40recorded · source
- vocab_size
- 50688recorded · source
- context_length
- 2048recorded · source
- positional_encoding
- rotary (RoPE), partial: 25% of dimsrecorded · sourcenote: Paper sec 2.3: 'We use rotary embeddings'; fraction is config rotary_pct 0.25. Same as pythia-6-9b.
- training_data
- The Pile (300B tokens). One of 8 sizes trained on identical data in identical order (Pythia paper).recorded · source
- techniques
- primary_sources
- record_history
- date:2026-09-24 · change:ingested as candidate from HF (EleutherAI/pythia-12b@bb1e3e710cdf6b524461d543cfb5ba773f0a81b6) · by:ingest_hf.py ·date:2026-09-24 · change:preparer: developer, positional encoding set; trained_on accepted · by:claude (preparer, Sonnet 5) ·date:2026-09-24 · change:preparer cleanup: removed 3 stale flag(s), filled training_data · by:claude (preparer, Sonnet 5) ·date:2026-09-24 · change:reviewed and promoted from staging (1 edge(s) accepted) · by:Wilson Pruitt ·
Parents
Training data
- trained_on → The Pile declared_by_uploader source HF card metadata datasets: EleutherAI/pile. Often incomplete; confirm which training stage used it.
Children
Weights descend
- ← fine_tuned_from dolly-v2-12b declared source Blog: 'a 12B parameter language model based on the EleutherAI pythia model family', 'based on EleutherAI's pythia-12b'.
Read in
No station on the reading path has touched this record yet.