# Genesis 1B: Supervised Fine-Tuning - Eval Baseline and SFT Setup - Kroonen AI > Pre-SFT evaluation baseline on Genesis 1B (step 40,000) across MMLU, ARC, HellaSwag, PIQA, Winogrande, and Lambada. SFT data mixture and training configuration for the next phase. Canonical: https://www.kroonen.ai/blog/genesis-sft/ [Lab notebook](https://www.kroonen.ai/blog/) / [Genesis 1B](https://www.kroonen.ai/blog/#genesis) / Entry 007 Working draft SFT Entry 007 # Genesis 1B: Supervised Fine-Tuning Pre-SFT evaluation baselines, a 510K-example data mixture, and the training configuration for the next phase. Author Robin Kroonen Published Apr 7, 2026 Reading time 5 min read Entry 007 Program Genesis 1B Format Working draft Phase SFT Subjects Genesis / SFT / fine-tuning / evaluation [Program index →](https://www.kroonen.ai/blog/#genesis) Table of Contents 1. [Pre-SFT Evaluation Baseline](#baseline) 2. [What These Numbers Mean](#what-it-means) 3. [SFT Data Mixture](#data-mixture) 4. [Training Configuration](#training-config) ## Pre-SFT Evaluation Baseline Before starting supervised fine-tuning, the pretrained Genesis 1B checkpoint at step 40,000 was evaluated using [EleutherAI's lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) across six standard benchmarks. These are the **before** numbers: the raw pretrained base, no instruction tuning, no RLHF. Eval command CUDA\_VISIBLE\_DEVICES=0 lm\_eval \\ --model hf \\ --model\_args pretrained=/path/to/genesis-1b-export,dtype=bfloat16 \\ --tasks hellaswag,piqa,winogrande,arc\_easy,arc\_challenge,lambada\_openai,mmlu \\ --device cuda:0 \\ --batch\_size auto:2 \\ --output\_path ./eval-results | Task | Metric | Value | Random baseline | | --- | --- | --- | --- | | ARC-Challenge | acc\_norm | 0.2594 | 0.25 (4-choice) | | ARC-Easy | acc\_norm | 0.2525 | 0.25 (4-choice) | | HellaSwag | acc\_norm | 0.2604 | 0.25 (4-choice) | | **PIQA** | acc | **0.5359** | 0.50 (binary) | | Winogrande | acc | 0.4878 | 0.50 (binary) | | Lambada (OpenAI) | acc | 0.0000 | N/A | | Lambada (OpenAI) | perplexity | 3,360,048 | N/A | | MMLU (overall) | acc | 0.2567 | 0.25 (4-choice) | | \- Humanities | acc | 0.2455 | | | \- STEM | acc | 0.2750 | | | \- Social Sciences | acc | 0.2658 | | | \- Other | acc | 0.2459 | | ## What These Numbers Mean Most tasks land at ~25% - exactly random chance for 4-option multiple choice. The model hasn't been taught to reason through MCQ format. It generates text; it doesn't answer questions. Two tasks show genuine signal above random: **PIQA at 53.6%** (physical intuition, binary choice) and **Winogrande at 48.8%** (commonsense coreference, binary choice). For a base model with no instruction tuning, this is expected. The model has absorbed real-world knowledge from the pretraining corpus, it just does not know how to surface it in a structured task format yet. **Lambada is a complete failure**: perplexity of 3.36M, accuracy 0%. Lambada tests long-range context prediction on narrative text. This is a known weakness of models trained on short context (2048 tokens) with a large vocabulary mismatch on the final-word prediction task. SFT will not fix Lambada. That requires more pretraining tokens at longer context. These are the _before_ numbers. The same eval will run again after SFT on the fine-tuned checkpoint for a direct comparison. ## SFT Data Mixture The SFT dataset is assembled from two sources, tokenized into ChatML format with per-token loss masking (only assistant turns are trained on). ### Constitutional data (synthetic, Kroonen AI) 10,000 examples generated with Claude Haiku using a constitutional principles framework. Covers reasoning, safety, personality, and instruction following, designed specifically for Genesis. This is the personality layer. ### External instruction datasets - **SmolTalk (Magpie Ultra)** - 100,000 examples. High-quality synthetic instruction data from Hugging Face. - **OpenHermes 2.5** - 200,000 examples sampled. General-purpose instruction following. - **Tulu 3 SFT mixture** - 100,000 examples. Diverse instruction types from AI2. - **MetaMathQA** - 50,000 examples. Mathematical reasoning. | Source | Examples | Purpose | | --- | --- | --- | | Constitutional (Haiku) | 10,000 | Personality, principles | | SmolTalk (Magpie Ultra) | 100,000 | Instruction following | | OpenHermes 2.5 | 200,000 | General instruction | | Tulu 3 SFT mixture | 100,000 | Diverse tasks | | MetaMathQA | 50,000 | Mathematical reasoning | | **Total** | **510,577** | | All datasets are converted to the [archived ChatML format specification](https://github.com/openai/openai-python/blob/release-v0.28.1/chatml.md) with four new special tokens appended to the vocabulary: `<|im_start|>` (49152), `<|im_end|>` (49153), `` (49154), `` (49155). New vocab size: 49,156. The model's embedding matrix is resized before fine-tuning, with new token embeddings initialized as the mean of existing embeddings. Total tokenized: **365.6M tokens**, of which **267.6M (73.2%) are trainable** assistant tokens. System prompts, user turns, and special tokens are masked out. The model only learns to produce assistant responses. ## Training Configuration
Base checkpointGenesis 1B step_040000 (pretrained)
GPUs2× RTX 4090 (PCIe, no NVLink)
Batch size2 per GPU
Gradient accumulation8 steps
Effective batch32 sequences / ~65,536 tokens per step
Learning rate2e-5 to 2e-6 (cosine decay, 5x lower than pretrain peak)
Warmup200 steps
OptimizerAdamW (β1=0.9, β2=0.95, wd=0.01)
Max steps15,955 (~1 epoch over 510,577 sequences)
Sequence length2048
LossCross-entropy on assistant tokens only (masked)
Scriptsft_v1.py
Gradient accumulation was reduced from 32 (pretrain) to 8 for SFT. With a diverse 510K-example dataset, tighter update loops produce better instruction following than large batches. The learning rate is set at 2e-5 peak, a 5x reduction from the pretrain peak of 1e-4, conservative enough to preserve pretrained weights while teaching new behavior. 1 epoch over the full mixture is the target. Overfitting on SFT data is a real risk; repeating the dataset multiple times tends to degrade diversity and reward-hack the loss. After 15,955 steps, the model will be exported to HuggingFace format and evaluated on the same benchmarks as the baseline above. Notebook / Reading path ## Continue in Genesis 1B [Complete index →](https://www.kroonen.ai/blog/) [ Previous entry 006 / 8 min read ### Genesis 1B: Run 2 Complete at 80,000 Steps Read entry →](https://www.kroonen.ai/blog/genesis-training-progress/)[ Previous entry 005 / 5 min read ### Genesis 1B, Run 2: Nearly 3× Throughput, Same Hardware Read entry →](https://www.kroonen.ai/blog/genesis-architecture-v2/) [← Lab notebook](https://www.kroonen.ai/blog/) [Back to top ↑](#main-content)