Model · opt-6-7b
OPT 6.7B
Developer: Meta AI
Availability: available · checked 2026-09-24 · source
Raw record: /data/models/opt-6-7b.json
Fields
- id
- opt-6-7b
- identifiers
- huggingface
- facebook/opt-6.7b
- developer
- Meta AI
- release_date
- 2022-05-02partial · sourcenote: arXiv v1 date of the paper.
- weights_status
- open
- availability
- available · checked 2026-09-24 · source
- license
- OPT-175B License Agreement: non-commercial research use only.recorded · sourcenote: metaseq OPT README: 'The use of OPT model weights is subject to the Model License.'
- architecture
- family
- decoder_only
- note
- Paper: trained 'to roughly match the performance and sizes of the GPT-3 class of models'; hyperparameters 'largely follow Brown et al. (2020)'. ReLU activation. The HF model card for this repo was written by Hugging Face, not Meta; Meta's own card is Appendix D of the paper.
- n_layers
- 32recorded · source
- hidden_size
- 4096recorded · source
- n_heads
- 32recorded · source
- vocab_size
- 50272recorded · sourcenote: Paper: GPT-2 byte-level BPE tokenizer.
- positional_encoding
- learned_absoluterecorded · sourcenote: Code: OPTLearnedPositionalEmbedding.
- context_length
- 2048recorded · source
- training_data
- ~180B tokens: RoBERTa corpus subsets (BookCorpus, Stories, updated CCNews), a subset of the Pile, and PushShift.io Reddit; tokenized with the GPT-2 BPE.recorded · source
- techniques
- transformer-decoderbyte-pair-encoding
- primary_sources
- record_history
- date:2026-09-24 · change:created from primary sources (Phase 1 seed, batch 2) · by:wilson-pruitt + claude ·date:2026-09-24 · change:availability checked and recorded · by:wilson-pruitt + claude ·
Parents
Training data
- trained_on → The Pile declared source A subset of the Pile, alongside RoBERTa corpus data and PushShift.io Reddit (not recorded as datasets).
Design
- design_follows → GPT-3 (175B, 2020 paper) declared source Paper: trained 'to roughly match the performance and sizes of the GPT-3 class of models'; hyperparameters 'largely follow Brown et al. (2020)'.
Children
No edges recorded.
Read in
No station on the reading path has touched this record yet.