Model · tinyllama-1-1b-intermediate-step-1431k-3t
TinyLlama-1.1B (intermediate-step-1431k-3T)
Developer: TinyLlama project (Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, Wei Lu; StatNLP Research Group, Singapore University of Technology and Design; paper authors)
Availability: available · checked 2026-09-24 · source
Raw record: /data/models/tinyllama-1-1b-intermediate-step-1431k-3t.json
Fields
- id
- tinyllama-1-1b-intermediate-step-1431k-3t
- identifiers
- huggingface
- TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
- developer
- TinyLlama project (Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, Wei Lu; StatNLP Research Group, Singapore University of Technology and Design; paper authors)
- release_date
- 2023-12-28partial · sourcenote: HF repo creation date (2023-12-28) for this checkpoint. Card: training started 2023-09-01. Paper (arXiv 2401.02385) dated January 2024 (v2 read: 4 Jun 2024).
- weights_status
- open
- availability
- available · checked 2026-09-24 · source
- license
- Apache-2.0recorded · sourcenote: Card metadata `license: apache-2.0`. License file text not separately read.
- architecture
- family
- decoder_only
- note
- family read from config 'architectures': ['LlamaForCausalLM'].
- n_layers
- 22recorded · source
- hidden_size
- 2048recorded · source
- n_heads
- 32recorded · source
- vocab_size
- 32000recorded · source
- context_length
- 2048recorded · source
- n_kv_heads
- 4recorded · source
- positional_encoding
- rotary (RoPE)recorded · sourcenote: Paper: 'We use Rotary Positional Embedding (RoPE)'. Paper also lists RMSNorm (pre-norm), SwiGLU, grouped-query attention (32 query heads, 4 key-value heads: matches config n_kv_heads 4), FlashAttention-2.
- training_data
- Up to 3 trillion tokens: SlimPajama (with its GitHub subset removed) plus code from the StarCoder training data (starcoderdata); about three epochs.recorded · sourcenote: Paper: 'primary sources: SlimPajama and the training data of StarCoder'; 'we remove the GitHub subset from SlimPajama and only sample code-related data from the StarCoder training dataset to avoid duplication'; 'approximately three epochs, cumulatively processing 3 trillion tokens'.
- techniques
- primary_sources
- record_history
- date:2026-09-24 · change:ingested as candidate from HF (TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T@59f6f375b26bde864a6ca194a9a3044570490064) · by:ingest_hf.py ·date:2026-09-24 · change:preparer: developer, license, positional encoding (paper), training data, 3 edges (same_architecture_retrained llama-2-7b; trained_on slimpajama, starcoderdata); sources: card, arXiv 2401.02385 · by:claude (preparer, Sonnet 5) ·date:2026-09-24 · change:reviewed and promoted from staging (3 edge(s) accepted) · by:Wilson Pruitt ·
Parents
Training data
- trained_on → SlimPajama declared source Card metadata `datasets: cerebras/SlimPajama-627B`; paper names SlimPajama as a primary source (GitHub subset removed). The `slimpajama` record has availability `removed` (HF API 401) — the corpus is recorded, not necessarily obtainable.
- trained_on → StarCoderData declared source Card metadata `datasets: bigcode/starcoderdata`; paper: 'the training data of StarCoder', code-related samples only. `starcoderdata` is gated.
Design
- same_architecture_retrained → Llama 2 7B declared source Card: 'We adopted exactly the same architecture and tokenizer as Llama 2.' Paper: 'Following the same architecture and tokenizer as Llama 2, we name our model TinyLlama.' Relation chosen from those exact words (method.md: 'same' -> same_architecture_retrained). Parent id llama-2-7b stands for the Llama 2 design; the developer names Llama 2 generally, not a size. TENSION: the paper's own hyperparameters use grouped-query attention, which the llama-2-7b record notes is not used at 7B; and the size is 1.1B. If the reviewer reads 'exactly the same' as too strong, design_follows is the alternative.
Children
No edges recorded.
Read in
No station on the reading path has touched this record yet.