Recent months have seen a surge in post-training (instruction tuning, RLHF, preference optimization) releases from research labs and open-source communities. This survey covers four notable releases from 2025–2026 that push the boundaries of how models are refined after pre-training:

• Nemotron Cascade — NVIDIA’s cascade architecture for post-trained models
• KIMI-DEV — A specialized post-training approach with developer-focused capabilities
• Hermes 4 — PowerLM’s latest open-weight post-trained model
• Intellect-3 — A reasoning-oriented post-training release

Below we summarize each paper, then pull together cross-cutting themes around methodology, data, and trends in the post-training space.

Nemotron Cascade

Paper: Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation Authors: Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, Renjie Pi, Grace Lam, Nayeon Lee, Alexander Bukharin, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping (NVIDIA) arXiv: 2603.19220v2 [cs.CL], March 22, 2026 License: CC BY 4.0

Overview

Nemotron-Cascade 2 is the successor to Nemotron-Cascade 1, introducing an open-weight 30B Mixture-of-Experts (MoE) model with only 3B activated parameters that achieves gold-medal-level performance in top-tier mathematical and coding competitions — IMO 2025, IOI 2025, and ICPC World Finals 2025. Despite its compact size, it outperforms larger models like Nemotron-3-Super-120B-A12B and Qwen3.5-35B-A3B across mathematics, code reasoning, alignment, and instruction-following benchmarks. It is the second open-weight LLM (after DeepSeek-V3.2-Speciale) to achieve gold medals in both IMO and IOI, with 20× fewer parameters.

Key Contributions

• Cascade RL at scale: Expands the original Cascade RL framework (from Nemotron-Cascade 1) to cover a much broader spectrum of reasoning and agentic domains, with a revised stage ordering to mitigate inter-domain interference
• Multi-Domain On-Policy Distillation (MOPD): A novel training stage inserted into the cascade pipeline that uses domain-specific intermediate teacher checkpoints (from the same SFT initialization) to recover benchmark regressions and stabilize performance
• Revised cascade ordering: IF-RL is now placed first (rather than later) because instruction-following training can degrade human alignment, and a strong IF model serves as a better teacher for subsequent distillation
• Multi-domain RL integration: Groups non-conflicting domains (MCQA STEM, tool calling, structured output) into joint training stages for efficiency
• Full open-source release: Model weights, SFT data, RL data, and detailed training methodology are all released

Architecture

• Base model: Nemotron-3-Nano-30B-A3B-Base (same initialization as its predecessor)
• Architecture: MoE with 30B total parameters, 3B active per token
• Context window: Up to 256K tokens (SFT), with specific RL stages supporting 49K, 98K, and 118K response lengths
• Chat template: Two modes — “thinking mode” (single <think> followed by newline) and “non-thinking mode” (adjacent <think></think> tokens). Tool calls use <tool_call>/</tool_call> tags with tools listed in system prompt

Training Pipeline

The full post-training pipeline is a 9-stage sequence (SFT + 8 RL stages), as illustrated in the paper’s Figure 2:

1. SFT — Supervised fine-tuning on a broad dataset (~15M+ samples across 10 domains), packed into 256K sequences, trained for ~1.5 epochs
2. IF-RL — Instruction-following RL (verifiable constraints, thinking mode only, dynamic filtering, overlong penalty)
3. Multi-domain RL — Joint training on MCQA STEM (55%), agentic tool calling (30%), and structured output (15%)
4. MOPD — Multi-domain on-policy distillation from 3 teacher checkpoints (math, RLHF, multi-domain) using reverse-KL token-level advantages with truncated importance weighting
5. RLHF — Human preference learning using a generative reward model (Qwen3-235B-A22B-Thinking), thinking mode only, length-normalized rewards
6. Long-context RL — Reasoning over 32K input sequences (49K max output), LLM judge evaluation
7. Code RL — Competitive coding with strict binary rewards, 118K max response length, async reward verification on 384 CPU cores
8. SWE RL — Two sub-stages: Agentless RL (code repair with GPT-OSS-120B reward model) and execution-based RL in agentic OpenHands scaffolds
9. Final model: Nemotron-Cascade-2-30B-A3B

SFT Data Curation (Section 3)

The SFT dataset spans 10 domains with carefully curated high-quality data generated by teacher models (GPT-OSS-120B, DeepSeek-V3.2, DeepSeek-V3.2-Speciale, Qwen3-235B):

Cascade RL Details (Section 4)

All RL stages use GRPO (Group Relative Policy Optimization) with strict on-policy training — no KL divergence term, simplified to a REINFORCE objective with group-normalized rewards:

where advantage is group-normalized: $\hat{A}_{i,t} = \frac{r_i - \text{mean}(r)}{\text{std}(r)}$

Key design decisions:

• LR = 3e-6 with AdamW across all RL stages consistently
• Temperature = 1.0, top-p = 1.0 for exploration
• Batch size = 128, 16 rollouts per prompt (Code RL and SWE RL use 64 rollouts)
• Entropy and KL coefficients set to 0 in most stages (KL=0.03 in RLHF to preserve capabilities)
• Dynamic filtering in IF-RL removes batch-homogeneous samples for effective gradients
• Overlong penalty prevents excessive token usage in instruction-following

MOPD (Multi-Domain On-Policy Distillation) is a standout innovation:

• Uses token-level reverse-KL distillation advantage:
  aₜ^MOPD = log π_teacher(yₜ|sₜ) − log π_train(yₜ|sₜ)
• Applies truncated importance weighting ($\epsilon_{low}=0.5$, $\epsilon_{high}=2.0$) to handle train-inference mismatch
• 3 teacher checkpoints selected from the cascade pipeline itself — math teacher (SFT checkpoint), RLHF teacher, multi-domain teacher
• 2×–3× more sample-efficient than GRPO on AIME25 and ArenaHard

Results and Benchmarks (Section 2)

Competitive performance:

Benchmark highlights (Table 1):

Notable: Underperforms Qwen3.5-35B on knowledge-intensive benchmarks (MMLU-Redux, GPQA-Diamond) and some agentic tasks, pointing to stronger pre-training and agentic RL as areas for future work.

Open Data and Code Released

• Model weights: Nemotron-Cascade-2-30B-A3B (post-trained checkpoint)
• SFT data: Nemotron-Cascade-2-SFT-Data — collection of all SFT training datasets
• RL data: Nemotron-Cascade-2-RL-Data — collection of all RL training datasets
• Training methodology: Full hyperparameters, prompt templates, and training configuration in the paper and appendix

Notable Design Decisions

• IF-RL first, not last: Reversed the cascade ordering from Nemotron-Cascade 1 because IF-RL degrades ArenaHard scores, but subsequent RLHF recovers them; also produces a better teacher for MOPD
• Thinking mode exclusively in IF-RL and RLHF: Thinking mode yields higher IFBench accuracy; RLHF in thinking mode prevents instruction-following regression
• No KL divergence in GRPO: Simplifies the objective and improves stability; KL=0.03 only in RLHF to prevent catastrophic forgetting
• Domain-specific ordering driven by interference analysis: The cascade order is not fixed — it adapts based on inter-domain interference patterns observed during training
• Agentless RL helps agentic tasks: Agentless SWE RL (code repair) improves Pass@4 on SWE-bench Verified by ~1–2 points even in agentic (OpenHands) evaluation, suggesting code repair capability generalizes across scaffolds

Intellect-3

Source: arXiv:2512.16144 — “INTELLECT-3: Technical Report” by the Prime Intellect Team (23 authors, Dec 2025)

INTELLECT-3 is a 106B-parameter Mixture-of-Experts language model (12B parameters active per token) built on top of the GLM-4.5-Air base and trained end-to-end with large-scale reinforcement learning. It achieves state-of-the-art performance for its weight class across math, code, science, and reasoning benchmarks — outperforming several frontier models with 3×–6× more parameters.

Key Contributions

• 106B MoE model (12B active) trained with RLVR that competes with models far larger than itself
• prime-rl: an open-source, production-scale asynchronous RL framework that scales from a single node to thousands of GPUs
• Full infrastructure open-sourced: training framework, verifiers library, Environments Hub registry, complete training recipe (SFT + RL), and all environments used for synthetic data generation, training, and evaluation
• Agentic-first RL design: first-class support for multi-turn interactions, tool use, sandboxed code execution, and long-horizon reasoning
• Novel training algorithm: masked token-level importance sampling (IcePop) to stabilize off-policy RL training

Architecture and Model Details

• Base model: GLM-4.5-Air (by Z.ai / 零一万物)
• Architecture: Mixture-of-Experts (MoE) — 106B total parameters, 12B active per forward pass
• Context window: trained up to 65K natively, extended to 98K via context parallelism during agentic SFT
• Chat template: inspired by Qwen3 and GLM-family, using control tokens (<|system|>, <|user|>, <|assistant|>, ``, `

</think>

`) with XML-style tool call tagging

• Always-reasoning design: a <|think|> token is appended via the chat template; reasoning chains are auto-parsed across turns via the reasoning_content field
• No user-exposed reasoning-effort controls — reasoning is always on

Training Pipeline

The training follows a two-stage SFT + RL recipe, carried out on a cluster of 512 NVIDIA H200 GPUs over ~2 months:

Stage 1 — General Reasoning SFT:

• Dataset combines NVIDIA Nemotron-Post-Training-Dataset-v1 splits (math, code, science, tool) and AM-DeepSeek-R1-0528-Distilled (chat, instruction following)
• Synthetic reasoning traces from DeepSeek-R1-0528
• ~33M tokens per step, context length 65K, 1 full epoch
• Muon optimizer, LR 5e-5 (warmup from 1e-8 over 300 steps), FSDP world size 64

Stage 2 — Agentic SFT:

• Smaller, curated dataset: SWE-Swiss, Toucan Tool, and synthetically generated data from Environments Hub environments using DeepSeek-R1-0528
• Tools: SWE Swiss (10.3K examples), Toucan Tool (116K examples), environments mix (38.4K examples)
• 2 epochs, context parallelism extended to 98K
• Muon optimizer, LR 5e-8, linear decay over 800 steps

Reinforcement Learning:

• Algorithm: Masked token-level importance sampling (IcePop) with double-sided masking (α=0.5, β=5)
• Batch: 256 prompts × 16 rollouts = 4,096 total, max context 65,536
• Optimizer: Muon at LR 1e-6
• Training compute: 60 H200 nodes (16 for training, 44 for inference) at ~1:3 ratio
• Step time: ~1,500 seconds per step at 65K sequence length
• Key RL features:
  • Asynchronous off-policy training (max 8 off-policy steps)
  • Continuous batching with in-flight weight updates
  • Online difficulty filtering (easy/normal/hard pools based on solve rate)
  • Multi-environment RL via EnvGroup pattern
• RL reward curves showed no sign of plateauing at end of training

Training Environments

Diverse mix of 6 environment categories, all from the open Environments Hub:

Difficulty annotations for all environments use solve rates of Qwen3-4B-Instruct (8–16 generations per problem).

Key Infrastructure Components

prime-rl — Asynchronous RL training framework:

• Disaggregated trainer (FSDP2) and inference (vLLM) on separate GPU pools
• Continuous batching with in-flight weight updates (rollouts can span multiple policies)
• Multi-client orchestrator for linear throughput scaling across nodes
• Online data filtering with dynamic difficulty pools
• Distributed Muon optimizer (all-to-all–based, avoids InfiniBand congestion)
• Efficient MoE support — expert parallelism disabled for their config (grouped gemm already saturated)

Verifiers + Environments Hub — Modular RL environment system:

• Environments are standalone installable Python modules with standardized entry points
• Class hierarchy: Environment → MultiTurnEnv → ToolEnv → StatefulToolEnv → SandboxEnv → CodeEnv
• Environments Hub is an open registry for versioned, shareable environment packages
• Training and evaluation use the same rollout/rubric entrypoints for consistency

Prime Sandboxes — High-throughput code execution:

• Custom Rust Gateway bypasses Kubernetes API Server to avoid etcd bottlenecks
• Headless Services + custom CoreDNS for millisecond pod IP resolution
• Sidecar pattern with nsenter for direct command injection
• Up to 2,000+ concurrent sandboxes per step, 256 sandboxes/node with Burstable QoS
• gVisor (runsc) for container isolation, Container Image Streaming for fast startup

Results

All benchmarks evaluated using the same open-source Environments Hub implementations with API models routed through official providers:

Notable: INTELLECT-3 matches or exceeds GLM-4.5 (3× larger) on AIME 2024, AIME 2025, and LiveCodeBench v6. On AIME 2024 and AIME 2025, it comes within ~2 points of the 3× larger GLM-4.6. Coding benchmark (LCB v6) exceeds GLM-4.5-Air post-train by 8 percentage points.

Open-Source Releases

Notable Design Decisions

• IcePop over PPO/GRPO: The masked token-level importance sampling approach with double-sided masking was critical to stabilize trainer-inference distribution mismatch, avoiding the reward collapse seen with GSPO under high off-policyness
• Muon optimizer throughout: Pretrained with Muon, so post-training also uses Muon (matrix-level updates) — LR 5e-5 for SFT, 1e-6 for RL
• No expert parallelism: Despite being an MoE, EP was disabled because their sequence length + hidden dim configuration already saturated grouped gemm kernels
• In-flight weight updates are essential: Without them, step times more than doubled due to inference inefficiency in continuous batching
• Context as a scarce resource: Future work treats the context window as actively managed (cutting, branching, external memory) rather than a passive transcript, motivated by “context rot” in long-context models
• Training not converged: At end of RL, all benchmark curves were still trending upward with no plateau, suggesting significant room for continued training

KIMI-DEV

Paper: Kimi-Dev: Agentless Training as Skill Prior for SWE-Agents (Yang et al., Moonshot AI + Tsinghua + PKU, 68 pages, v3, Dec 2025)

Kimi-Dev introduces a paradigm shift in how SWE (Software Engineering) LLMs are trained, arguing that the two dominant paradigms — agentic multi-turn frameworks (SWE-Agent, OpenHands) and workflow-based agentless methods (Agentless) — are not mutually exclusive. Instead, agentless training induces structured skill priors that enable highly efficient adaptation to agentic settings. The result: an open-source 72B model achieving state-of-the-art performance among workflow approaches on SWE-bench, and an SWE-Agent fine-tuned variant competitive with Claude 3.5 Sonnet.

What Makes Kimi-Dev Novel

• Agentless as skill prior, not endpoint: The core thesis is that agentless training should be viewed as a means to induce atomic capabilities (localization, code edit, self-reflection, verification) rather than the final product. These skills transfer to agentic frameworks, bridging what has been seen as a paradigm dichotomy.
• Duo framework: BugFixer + TestWriter: Instead of a single monolithic solver, Kimi-Dev splits the role into two specialized modules: the BugFixer (produces patches to fix bugs) and the TestWriter (creates reproducible unit tests that capture the reported bug). Both share two core skills — file localization and code edit.
• Execution-based rewards only: Unlike SWE-RL which uses text-similarity rewards, Kimi-Dev uses pure outcome-based rewards (0 or 1 from environment execution), yielding more reliable fix quality signals.
• Curriculum-based RL with adaptive prompt selection: Prompts with pass@16 = 0 are initially discarded, then 500 new prompts are reintroduced every 100 RL steps as the model improves, creating a curriculum that gradually raises task difficulty.

Architecture and Methodology

The training recipe consists of four sequential stages:

1. Mid-Training (~150B tokens): Starting from Qwen 2.5-72B-Base, the model is mid-trained on millions of GitHub issues and PR commits:
   • ~50B tokens from Agentless-style data (diff patches)
   • ~20B tokens from curated PR commit packs
   • ~20B tokens of synthetic reasoning/agentic interaction data (upsampled 4×)
   • Strict decontamination against SWE-bench Verified test repositories

2. Cold Start via Reasoning SFT: ~2,000 long Chain-of-Thought trajectories generated by DeepSeek R1 (20250120 version) acting as BugFixer/TestWriter. This activates the model’s long CoT capability for problem analysis, method sketching, self-refinement, and exploration of alternatives.

3. Reinforcement Learning (Code Edit Stage): Using the Kimi k1.5 policy optimization method (a REINFORCE-based approach with averaged rollout rewards as baseline, similar to GRPO). Key design choices:
   • Outcome-only reward (no format/process penalties)
   • BugFixer rewarded when patch passes all ground-truth unit tests
   • TestWriter rewarded when test fails on pre-fix code AND passes after fix
   • Kubernetes-based Docker sandbox supporting 10,000+ concurrent instances
   • Max context length: 64K tokens
4. Test-Time Self-Play: At inference, the model generates 40 candidate patches (1 greedy + 39 at temperature 1) and 40 candidate tests, then scores each patch-test pair using a composite metric based on fail-to-pass and pass-to-pass transitions across the test suite. 3 patch-test pairs already outperform 40-patch majority voting.

Training Approach (Post-Training Deep Dive)

Results and Benchmarks

Agentless (Kimi-Dev 72B) on SWE-bench Verified:

• 60.4% pass@40 on SWE-bench Verified — the best among all workflow-based/open-source approaches
• Mid-training token budget scales linearly with performance (50B → 100B → 150B all improve)
• RL shows clear scaling: both pass rate and response length increase throughout training

Agent SFT (Kimi-Dev → SWE-Agent) on SWE-bench Verified:

• Achieves near-parity with Claude 3.5 Sonnet using only 5K public trajectories
• Pass@10 of 74.0% surpasses Agentless pass@30 of 73.8%, showing higher ceiling for agentic frameworks
• RL prior needs only 2^23 SFT tokens to match what the Base model achieves with 1.5 × 2^28 tokens — a 512× data efficiency gain

Skill Transfer Analysis:

• RL-prior models continue improving beyond 70 turns in SWE-Agent, while SFT/MT/Base priors plateau at 50-70 turns
• BugFixer skill at Stage-3 cutoff: 484 cases (Base) → 605 cases (RL prior) resolved within 3 passes
• Reflection skill gains: +94 (Base) → +113 (RL prior) additional cases resolved through reflection/redo

Generalization:

• Performance holds on SWE-bench-Live and SWE-bench Multilingual (300 tasks across 9 languages: Rust, Java, PHP, Ruby, JS/TS, Go, C/C++)
• RL prior consistently outperforms Base/MT/SFT priors across all agent trajectory scales on multilingual benchmarks

Emergent Parallel Scaling:

• Without any additional training, Kimi-Dev exhibits parallel scaling: feeding multiple patch candidates into a single prompt and asking the model to synthesize a combined fix yields improving performance with more candidates. This emerges naturally from the training recipe.

Open Data and Code

• Model weights: moonshotai/Kimi-Dev-72B on HuggingFace
• Code: github.com/MoonshotAI/Kimi-Dev on GitHub
• SWE-Agent trajectories: Built on the SWE-Smith public dataset (5,016 trajectories collected with Claude 3.7 Sonnet)
• Mid-training data: Curated from millions of GitHub repos (filtered for 5+ stars, decontaminated against SWE-bench). Synthetic data and PR packs used but not individually released — the recipe and prompt templates are documented.
• Docker environments: Built from SWE-Gym, R2E-Gym-Lite, and SWE-Bench-Extra with automated configuration (detailed in Appendix B)
• License: CC-BY 4.0

Notable Design Decisions

• RL only on code edit, not localization: After mid-training and cold start, the model already excels at localization, so RL is focused purely on the code edit stage — a targeted allocation of compute
• Positive example reinforcement: In later RL stages, successful samples from recent iterations are replayed in the training batch, accelerating convergence when exploration diminishes
• No KL/entropy regularization in end-to-end RL: The agentic RL uses outcome reward only without any regularization, which the authors argue reveals the true potential of each prior beyond imitation shortcuts
• MT prior degrades at 200 trajectories: The mid-trained prior shows a performance dip when fine-tuned on exactly 200 SWE-Agent trajectories, hypothesized to be mode collapse via memorization — whereas the RL prior generalizes better due to more transferable skills baked in
• TestWriter false positives: A known limitation — TestWriter sometimes generates insufficient test coverage, allowing BugFixer patches to pass the reward check that shouldn’t. The authors acknowledge this as future work

Hermes 4 Technical Report

Source: arXiv:2508.18255 — Nous Research, August 2025 (v2, revised September 2025)

Hermes 4 is a family of hybrid reasoning models that combine structured, multi-turn reasoning with broad instruction-following ability. The team released three model sizes (14B, 70B, 405B), all trained on open-weight checkpoints from Llama 3.1 (405B, 70B) and Qwen3 14B, with full weights published on HuggingFace.

Key Contributions

• DataForge: A graph-based synthetic data generation pipeline that processes pre-training seed data through a directed acyclic graph (DAG) where each node implements a struct → struct map using a PDDL-style action interface (preconditions and postconditions determine data flow between nodes)
• Two-stage training methodology: An initial SFT on ~19B tokens across heterogeneous data, followed by a targeted second SFT stage to control reasoning length (30k token budget) using loss-masking on only the </think> termination token
• Atropos: An open-source RL environment microservice manager used for both rejection sampling training data and evaluation — exploiting the duality between RL environments and evaluation harnesses
• Comprehensive evaluation: Released with all logged generations, using a custom OpenAI-compatible endpoint design for reproducibility, avoiding framework fragmentation

Architecture & Model Sizes

• Hermes 4 405B: Based on Llama 3.1 405B, trained 71,616 B200 GPU hours
• Hermes 4 70B: Based on Llama 3.1 70B, trained 12,864 B200 GPU hours
• Hermes 4 14B: Based on Qwen3 14B, trained 4,454 B200 GPU hours
• Training framework: Modified TorchTitan, with a custom fork at NousResearch/torchtitan
• Second stage SFT trained using Axolotl for its character-span token-level masking interface
• All models trained on 192 NVIDIA B200 GPUs using a mix of Distributed Data Parallelism, Tensor Parallelism (TP8), and Fully Sharded Data Parallelism
• Context length: 16,384 tokens for training; 40,960 for evaluation

Post-Training Data Strategy

The dataset contains approximately 5 million samples and 19 billion tokens, structured as:

• 3.5M reasoning samples (token-heavy, averaging 5× more tokens per sample than non-reasoning data, with thinking traces up to 16k tokens)
• 1.6M non-reasoning samples (general instruction-following, knowledge, creative writing)
• A significant portion of the Hermes 3 dataset was retained for capability continuity

DataForge pipeline (inspired by AgentInstruct):

• Seed data drawn from DCLM and FineWeb (biased toward recent samples)
• Semantic deduplication using ModernBert embeddings (cosine similarity threshold 0.7)
• LLM judge filtering for incomplete or ill-formatted passages
• DAG-based generation: passage transformation → instruction generation → answer generation → judge review
• Every graph has a single source and single target node, enabling arbitrary nesting into higher-order graphs
• Training includes not just final QA pairs but all intermediate LLM calls used in generation (specializing the model in instruction generation and judging)

Rejection sampling environments (via Atropos):

• Answer Format Training: 150+ output formats rewarded for compliance (e.g., \boxed{} LaTeX, JSON), decoupled from semantic correctness
• Instruction Following: RLVR-IFEval constraint tasks (e.g., “every Nth word must be in French”)
• Internbootcamp: 70,000 rejection-sampled trajectories across ~1,000 reasoning tasks using DeepHermes 3 and other larger models
• Schema Adherence: Dynamic Pydantic model compilation from executable Python for JSON generation and error correction
• Tool Use: Training on agentic tool calls with JSON structure validation

Covering set techniques:

• Taxonomies: Depth-first-search LLM enumeration of subdomains down to prompt-level leaves
• PersonaHub: Synthetic personas from FinePersonas for generating application and script implementation tasks

Training Methodology

• Cosine learning rate schedule, 300-step warmup, 9,000 total steps
• Global batch size of 384 samples at 16,384 token context length
• First-Fit Decreasing sample packing achieving >99.9% batch efficiency
• Flex Attention to restrict attention within packed batch samples
• Only assistant-role tokens contribute to the cross-entropy loss

Reasoning length control (key innovation for the 14B model):

• The 14B model reached its 40,960 context limit 60% of the time on LiveCodeBench in reasoning mode (frequently exceeding 40k tokens despite 16k training budget)
• Solution: Second SFT stage generating synthetic traces with </think> forced at 30,000 tokens, training only on the </think> and <eos> tokens (loss-masking the entire reasoning chain)
• This approach teaches a “counting behavior” (“after N tokens, stop”) without altering the reasoning distribution, avoiding model collapse risks from recursive synthetic data training
• Trade-off: Up to 3.9% relative performance reduction on some reasoning benchmarks, but 98.9–99.8% reduction in overlong rates
• Not needed for the 70B or 405B models

Evaluation Framework

The team built a custom evaluation infrastructure emphasizing reproducibility:

• Atropos as evaluation framework: Single-file self-contained Python evaluations with detailed sample-level logging, overlapping inference and scoring (not batch-then-score), lightweight OpenAI-compatible client, and explicit error semantics (fail fast rather than silently score incorrect)
• Elastic inference cluster: Preemption-aware inference using sglang-router with automatic worker requeueing, allowing evaluation jobs to scale across available B200 compute without blocking training
• LiveCodeBench: 454 problems (8/1/2024–5/1/2025), scored via Modal containers with inference-scoring overlap to stay compute-bound rather than verification-bound
• RefusalBench: 166 hand-crafted prompts across 32 categories, measuring model refusal rates (with conditional reward inversion for safety-critical categories: minor harm, exploitation/trafficking, suicide/self-harm)
• All logged generations released on HuggingFace alongside the models

Results

Hermes 4 405B (vs. comparable open-weight models):

R = reasoning mode, N = non-reasoning mode. Hermes 4 405B leads on AIME scores among open-weight models, with strong generalist coverage.

Hermes 4 70B:

• AIME’24: 73.5, AIME’25: 67.5, GPQA Diamond: 66.1, LiveCodeBench: 50.5, MMLU: 88.4
• Competitive with Cogito 70B and Qwen3 235B on most benchmarks

Hermes 4 14B:

• AIME’24: 55.4, AIME’25: 46.8, GPQA Diamond: 60.2, LiveCodeBench: 42.5, MMLU: 84.1
• Strong for its size, trailing Qwen3 14B on reasoning but competitive on knowledge benchmarks

Qualitative Behavioral Analysis

• Reduced policy rigidity: Unlike proprietary models (GPT-5, Opus 4.1) that frequently issue AI identity disclaimers on fictional prompts, Hermes 4 demonstrated contextual fidelity, interpreting role-play prompts in-character
• Stylistic transfer: Generated text approximated target authorial rhythm and diction, going beyond surface-level topical references seen in open-source baselines
• System prompt customization: Anti-sycophancy prompts led to deeper chain-of-thought shifts (not just surface politeness changes); CoT traces showed explicit steering away from deference
• Chat template sensitivity: Replacing assistant with me in the Llama 3 chat template produced markedly different first-person, peer-like behavior — suggesting higher behavioral plasticity than typical large models
• Neutral alignment: Designed as a “neutrally-aligned generalist” — low refusal rates across categories (RefusalBench: 57.1 reasoning / 43.2 non-reasoning), with inversion only for safety-critical categories

Open Data & Code

• Model weights: All sizes on HuggingFace — NousResearch/hermes-4-collection
• Eval generations: All logged samples released — Hermes 4 Evals
• Atropos: Open-source RL environment manager and evaluation framework — github.com/NousResearch/Atropos
• DataForge: Graph-based synthetic data pipeline (referenced in the paper)
• Modified TorchTitan: github.com/NousResearch/torchtitan
• Lighteval modifications: github.com/NousResearch/lighteval

Notable Design Decisions

• Loss-masking only on </think> for length control: Rather than training on full self-generated reasoning traces (which risks model collapse), the team isolated the termination signal, teaching the model when to stop reasoning without modifying how it reasons
• Training on intermediate LLM calls: By including all generative steps used in the DataForge pipeline (not just final answers), the model internalizes instruction generation and judging capabilities
• OpenAI-compatible eval endpoint: All benchmarks hit the same inference engine instance, avoiding the fragmentation of different framework versions across benchmarks
• Inference-scoring overlap for LiveCodeBench: Using Modal containers, inference and scoring run concurrently rather than sequentially, keeping the pipeline compute-bound
• No KL/entropy regularization in rejection sampling: The training relies on pure reward signals from verifiers rather than distributional constraints, pushing the model toward the verified distribution without KL drag
• Elastic preemption-aware inference cluster: Evaluation jobs can be preempted and automatically requeued via sglang-router, maximizing cluster utilization without blocking training workloads

Themes in Post-Training Releases

Theme 1: The Shift from PPO to REINFORCE-Style Objectives

All four releases have moved away from traditional PPO-based RLHF toward simpler REINFORCE-family objectives, eliminating KL divergence and entropy regularization in most stages. Nemotron-Cascade 2 uses pure GRPO with KL=0 across all stages (only RLHF retains KL=0.03), Kimi-Dev uses a k1.5-style REINFORCE with outcome-only rewards and no regularization, Hermes 4 uses rejection sampling without KL drag, and Intellect-3 uses IcePop (masked token-level importance sampling) for off-policy RL. This reflects a broader industry consensus that KL penalties act as a ceiling on capability gains, and that verifier-based rewards (code execution, math verification) provide cleaner signal than learned reward models.

Demonstrated by: All four papers — Nemotron-Cascade 2, Kimi-Dev, Hermes 4, Intellect-3.

This trend represents a maturation of post-training methodology: rather than constraining the policy to stay near a reference distribution, these works trust execution-based verifiers to guide learning, accepting the resulting distributional shift as a feature rather than a bug.

Theme 2: Cascade Training and Curriculum Design

Nemotron-Cascade 2 formalizes cascade training into an explicit multi-stage pipeline where domain ordering is driven by interference analysis. The IF-RL stage is placed first (reversed from Cascade-1) because instruction-following training degrades alignment if done late. Kimi-Dev mirrors this with a curriculum-based RL approach: prompts with pass@16 = 0 are initially discarded, then 500 new prompts are reintroduced every 100 steps as the model improves. Intellect-3 uses online difficulty filtering with easy/normal/hard pools based on solve rates. The common thread is that domain-specific ordering and difficulty scheduling are treated as hyperparameters worthy of careful investigation, not afterthoughts.

Demonstrated by: Nemotron-Cascade 2 (explicit 9-stage cascade), Kimi-Dev (curriculum with adaptive prompt selection), Intellect-3 (difficulty filtering pools).

This represents the state-of-the-art in pipeline design: static training recipes are giving way to dynamically adaptive training that responds to the model’s evolving capabilities.

Theme 3: Coding as the Dominant Post-Training Domain

All four releases invest heavily in coding and software engineering capabilities, but with distinct strategies. Nemotron-Cascade 2 uses competitive coding (LiveCodeBench, IOI, ICPC) plus SWE-bench with execution-based rewards. Kimi-Dev is exclusively focused on SWE, splitting the problem into BugFixer and TestWriter modules. Intellect-3 includes code environments (8.6K problems) alongside math, science, logic, and deep research. Hermes 4 takes the broadest approach, with code as one pillar among reasoning, instruction-following, and safety. The significance is that coding has become the primary arena for demonstrating post-training quality — not just as a benchmark, but as a training signal. Execution-based rewards from test suites and sandboxed environments provide the cleanest, most unambiguous verification signal available.

Demonstrated by: All four papers, with Kimi-Dev being the most domain-specific and Hermes 4 the most generalist.

Coding execution loops have become the gold standard for RL training signals, and this trend is likely to accelerate as more open-source code environments and verification frameworks are released.

Theme 4: Open Infrastructure and Reproducibility

Each release invests heavily in open infrastructure beyond just model weights. Nemotron-Cascade 2 releases SFT data, RL data, and full training methodology. Intellect-3 releases the prime-rl framework, Environments Hub registry, verifiers library, and Prime Sandboxes documentation. Hermes 4 releases Atropos (RL environment manager + evaluation framework), modified TorchTitan, and all logged generations. Kimi-Dev releases model weights, code, and documents the mid-training recipe and prompt templates. Together, these releases represent a shift toward open-source ML infrastructure as a first-class research contribution, not an afterthought.

Demonstrated by: All four papers — each releases at minimum model weights and data, with three releasing substantial infrastructure code.

This represents a significant evolution in how post-training research is shared: the community is treating training frameworks, evaluation harnesses, and environment registries as public goods, which lowers the barrier to entry and accelerates reproducibility.

Theme 5: Synthetic Data at Scale via Teacher Distillation

All four teams rely heavily on synthetic data generated by larger teacher models, but with distinct approaches to quality control and pipeline design. Nemotron-Cascade 2 uses multiple teacher models (GPT-OSS-120B, DeepSeek-V3.2, Qwen3-235B) and MOPD to distill from intermediate checkpoints. Intellect-3 uses DeepSeek-R1-0528 for synthetic reasoning traces and SWE data generation. Kimi-Dev uses DeepSeek R1 for cold-start CoT trajectories and synthetic agentic interaction data. Hermes 4’s DataForge pipeline is the most sophisticated: a DAG-based system with semantic deduplication, LLM judge filtering, and training on all intermediate LLM calls (not just final outputs). The common pattern is that synthetic data generation is treated as a structured pipeline with quality gates, not a single LLM call.

Demonstrated by: All four papers, with Hermes 4’s DataForge being the most structured and Nemotron’s MOPD being the most innovative in terms of training-time distillation.

This represents the frontier of data engineering in post-training: the quality of the synthetic data pipeline is now as important as the choice of RL algorithm.

Open Data and Code Released

Comparison Table

Significance of Open Data in Post-Training

The open data released by these groups has several important implications:

• Nemotron-Cascade 2 is the most comprehensive release, providing both SFT and RL datasets alongside full hyperparameters. This is notable because RL data has historically been the most guarded aspect of post-training — the fact that NVIDIA released it signals a shift toward transparency in what actually works during RL stages.

• Intellect-3’s open-source infrastructure (prime-rl, Environments Hub, Prime Sandboxes) is particularly significant because it releases the training system rather than just data. Other groups can use prime-rl to train their own models, creating a shared infrastructure layer for the community.

• Hermes 4’s evaluation release (all logged generations) is an underappreciated contribution. By releasing every logged sample and using a custom OpenAI-compatible evaluation endpoint, the team enables exact replication of their evaluation protocol — a rare level of transparency in an area notorious for benchmark fragmentation.

• Kimi-Dev’s partial release is still valuable: while the full mid-training data isn’t released, the recipe, prompt templates, and the analysis of how many SFT trajectories are needed for agentic adaptation (5K vs 1.5×2^28 tokens) provide actionable knowledge for reproducibility.

The dominance of CC BY 4.0 licensing across these releases (3 of 4) signals alignment toward permissive open licensing in post-training, which is notable given that many proprietary model providers still restrict weights and training data under restrictive terms.

Methodology and Trends

Training Pipeline Comparison

The most striking observation is the spectrum of pipeline complexity: Hermes 4 achieves strong results with 2-stage SFT (plus rejection sampling), while Nemotron-Cascade 2 uses a 9-stage cascade. The choice seems correlated with model scale and domain breadth — larger, multi-domain models benefit from cascaded, domain-specific RL stages, while more focused models achieve strong results with simpler pipelines.

Loss Functions and Reward Paradigms

Three of four papers (Nemotron, Kimi-Dev, Hermes 4) use no KL or entropy regularization during their primary RL stages. This is a clear emergent consensus: when execution-based verifiers provide clean rewards, KL penalties are unnecessary constraints.

Optimizer Choices

The adoption of Muon (a second-order optimizer) by both Kimi-Dev and Intellect-3 is notable — it suggests a trend toward matrix-level optimization updates for training stability, particularly in large-scale RL settings.

Computational Scales

The Hermes 4 405B training at 71,616 B200 hours is the most computationally intensive, reflecting the cost of training a 405B model with 19B tokens of post-training data. Intellect-3’s approach (disaggregated trainer and inference on separate GPU pools) represents an important efficiency pattern: keeping inference and training on separate node pools avoids resource contention.

Emerging Trends

1. Cascade training (Nemotron): The explicit design of training stage ordering based on inter-domain interference analysis is emerging as a best practice for multi-domain models.

2. Execution-based rewards (All four): Code execution, test suites, and sandboxed environments are becoming the default reward mechanism for coding post-training, replacing learned reward models.

3. Self-play and candidate synthesis (Kimi-Dev): Generating multiple candidates at test time and scoring them via composite metrics (fail-to-pass, pass-to-pass) is an emergent pattern that requires no additional training.

4. Data synthesis pipelines (Hermes 4’s DataForge, Nemotron’s MOPD): Structured, DAG-based or checkpoint-based distillation from intermediate models is replacing simple teacher-student SFT.

5. Agentic training as a first-class post-training goal (All four): Tool use, multi-turn interactions, and sandboxed code execution are now standard components of post-training, not optional add-ons.

6. Off-policy RL stability (Intellect-3): IcePop’s masked token-level importance sampling addresses a fundamental challenge in off-policy RL, enabling stable training with up to 8 off-policy steps — a technique that may become widely adopted.

Ideas for Further Exploration

Research Directions

1. Does cascade training generalize beyond math/coding domains? Nemotron-Cascade 2’s cascade ordering was validated on mathematics, coding, science, and agentic tasks. An ablation study could test whether cascade training provides similar benefits for domains like creative writing, legal reasoning, or medical diagnosis — areas where domain interference patterns might differ substantially.

2. What is the minimum viable RL pipeline? Hermes 4 achieves strong results with 2-stage SFT (plus rejection sampling), while Nemotron uses 9 stages. A controlled comparison across the same base model and compute budget could identify which stages are essential versus which provide diminishing returns. This would help smaller research groups design efficient post-training pipelines.

3. Can Muon consistently outperform AdamW in RL post-training? Both Kimi-Dev and Intellect-3 use Muon across SFT and RL stages, while Nemotron uses AdamW. A head-to-head comparison on the same tasks and compute budget would clarify whether Muon’s matrix-level updates provide meaningful advantages in the post-training regime, or whether reported gains are specific to their training configurations.

4. What is the optimal curriculum schedule for RL difficulty? Kimi-Dev’s adaptive prompt selection (discarding impossible prompts, then reintroducing 500 every 100 steps) and Intellect-3’s difficulty pooling (easy/normal/hard) offer two different approaches to curriculum learning. A systematic study varying re-introduction rates, pool sizes, and transition thresholds could identify best practices for dynamic curriculum design.

5. Does training on intermediate LLM calls (DataForge approach) improve instruction generation capability? Hermes 4 trains on all intermediate LLM calls in the DataForge pipeline, not just final outputs. An ablation comparing “final output only” vs. “full generation trace” training would clarify whether the model internalizes instruction generation and judging capabilities, and whether this generalizes to downstream prompting tasks.

6. How transferable are agentic skills across different scaffolds? Kimi-Dev shows that agentless training induces skills that transfer to SWE-Agent, and Nemotron finds that Agentless RL helps even in OpenHands evaluation. A broader study across agentic frameworks (SWE-Agent, OpenHands, Aider, Devin) would quantify the degree of scaffold-independence and identify which skills are truly portable versus scaffold-specific.

7. What is the role of synthetic data diversity vs. quality in post-training? All four papers use synthetic data from teacher models, but the quality thresholds and diversity strategies differ significantly. A controlled experiment varying teacher model size, data synthesis method (direct SFT vs. rejection sampling vs. DAG-based generation), and deduplication thresholds would clarify the tradeoffs between synthetic data quantity and curation rigor.

8. Can IcePop-style masked importance sampling stabilize off-policy RL across different algorithms? IcePop was designed to handle GRPO’s trainer-inference mismatch, but its double-sided masking approach could potentially benefit PPO, DPO, and other off-policy methods. Testing IcePop’s masking mechanism as a plug-in for other RL algorithms would determine whether distribution mismatch is the fundamental challenge in off-policy post-training.

Gaps in the Current Landscape

• Limited cross-validation across benchmarks: None of the four papers use fully shared evaluation infrastructure. Hermes 4’s custom eval framework is the closest, but a community-wide standardized evaluation harness (like Environments Hub aims to be) would enable direct model comparisons.

• No public analysis of training dynamics: While all papers report final benchmark scores, detailed training curves, loss trajectories, and failure analysis are rarely shared. Releasing training logs alongside model weights would enable meta-analysis of what actually works during training.

• Safety and alignment as afterthoughts: Nemotron is the only paper that gives safety a prominent role (dedicated safety SFT data, RLHF stage). Coding-focused and reasoning-focused papers largely treat safety as a side concern, which may become a liability as these models are deployed.

• No comparison of open vs. proprietary base models: All papers use a single base model for their experiments (Llama 3.1, Qwen 2.5, GLM-4.5, Nemotron-3-Nano). A cross-base-model comparison would clarify how much post-training gains depend on the quality of the pre-trained initialization.

• Missing long-term capability retention analysis: None of the papers evaluate how well post-trained capabilities persist over time or under distribution shift. A longitudinal study of post-trained models on evolving benchmarks would reveal whether these gains are durable or brittle.

Ever since the release of OpenClaw, I have been trying to answer one question: What is the smallest LLM running on DGX Spark that can handle a simple cronjob? In my experience, there exists a major issue with regards to “instruction following”. Models will interpret the instructions in unexpected ways and cause havoc. Additionally, models will ignore or forget about instructions as the process progresses. Vague instructions are partially to blame. I ablated this by using larger models and reading their thinking traces. And even larger models tend to ask for clarification or they might make assumptions and continue but their assumptions are far superior to smaller models.

The open source (or open weight) models tested ranged from Cohere Command A, Hunyuan to more mainstream MiniMax, GLM (4.7 Flash, 4.7 Flash Reap), and Qwen. That is why the release of Gemma 4 models has been a great surprise to me since I had given up on models below the ~20B size for executing tasks. Gemma 4 series is capable way above their size group where the models below ~10B are capable of cohesive instruction following and doing interleaved tool calling with reasoning very effectively. This inspired me to benchmark the (quantized version of) recent releases against NextJS benchmark, and also compare the relative behavior evolution of the LLMs within their family groups.

Context

The original release of Next Js benchmark is interesting since the tasks would manually pass files within the prompts and give a specific ask then the model had to return the changes to files without tool calls or anything. This strictly a coding-related benchmark but I thought it may be a good proxy with problems I am facing in harnesses. The LLMs are trained to do tool calls and handle these kind of tasks but the prompts strictly say what needs to be done. Note that the prompt does not sat NO TOOLS but the model is expected to pick it up and interpret it.

Here is the specific format:

```
Please provide your changes as complete file contents. Use this format for each file you need to modify, create, or delete:
```file=path\/to\/file.tsx action=replace
complete file content here\n```

For example:
```file=app\/Component.tsx action=replace\nexport default function Component() {\n const new = \"code\";\n return <div><\/div>;
}\n```

```file=app\/NewComponent.tsx action=add
export default function NewComponent() {\n return <div>New component<\/div>;
}\n```

```file=app\/OldComponent.tsx action=delete```

Rules:
- Always specify an action: add, replace, or delete
- For add\/replace: provide the complete file content, not partial changes
- For delete: you can leave the content empty or omit it
- Use the exact file path relative to the project root
- Maintain proper indentation and formatting
- Include all imports and exports that the file needs
- Do not modify *.test.tsx files
```

I personally do not get anything from the model cards that are released everyday with each new model release (I guess that is why there is benchmarking mini-industry being formed) But with the large volume and fast pace release of open source models, I wanted to assess progress in the simplest way possible. That is benchmarking models on NextJS benchmark to track tool calls, reasoning efficiency (or output generation length) and speed.

Recent releases capacities

Qwen3.5 was released on February 16, 2026, and Gemma 4 on April 2, 2026. Both releases are natively multimodal with heavy emphasis on reasoning and agentic capabilities. Each release has a number of smaller and mid-size models; the larger models are either MoE or dense. Gemma uses the Per-Layer Embeddings (PLE) approach for smaller releases, where the effective computation amount is lower than actual model size (E2B with 5B LLM, E4B with 9B LLM). It is interesting to observe the effect of these architectural differences.

The graph below compares NextJS pass@1 for Qwen3.5, Gemma 4, Nemotron, and GLM-4.7 Flash. I define Reasoning efficiency as the squared reward divided by log10 of reasoning length. The metric aims to ensure the amount of tokens generated for reasoning is worth it. The easiest way to think about it: smaller Qwen models (< 8B) generate long reasoning traces riddled with “oh wait”s without a worthwhile response (not all tasks just complicated ones). The other metric that is important is tool call percentage to check what portion of responses involved doing tool calls.

Some observations:

• Gemma 4 has the best models in all sizes, these models are exceptional at understanding that no tool calls are required. Especially the smaller ones, Gemma:4-E2B and Gemma:4-E4B are on par with Qwen3.5:9B except for slight drop in reasoning efficiency.
• Within Qwen3.5, models below 9B are doing poorly with almost all the responses being tool calls.
• Nemotron 3 series are hybrid (mamba+transformer) model family with with 4B, 30B-A3B, 120B-A12B and recent highly post-tuned versions. The smallest versino does not do well in reward part but it stays away from tool calls. One thing that catches my eye is Nemotron Cascade-2 which is the post-tuned version of 30B-A3B and the difference is amazing 10% reward increase, no tool calls in cascade-2 version, and it has a stronger reasoning efficiency. The Cascade-2 paper stresses the order of training data used to ensure different capabilities (reasoning on math vs coding vs instruction following) are not forgotten/overridden in between runs. And the results show significant improvements across the metrics. Still Cascade-2 lags behind Gemma4:26B-A4B
• GLM-4.7 Flash underperforms compared to Gemma 4-26B-A4B but does better than Qwen3.5:35B-A3B. The REAP-ed version of it does worse as well where every metric gets worse.
• The bigger models have done relatively worse, especially Nemotron Super 120B (this could be due to quantization) but presumably post-tuning in cascade-2 fashion could improve it.

Qwen

AliBaba has released Qwen3, Qwen3-Coder, Qwen3-vl, and now Qwen3.5 in less than a year. I was curious to see the progress for this specific benchmark. Here are some observations:

• The small Qwen3.5 checkpoints reach for tools far more often than same-sized predecessors. The shift shows up in Qwen3-VL and steepens in Qwen3.5.
• At a given size band, MoE variants tool-call more than dense ones.
• The reasoning efficiency drops significantly with qwen3-vl series where large amounts of tokens are generated to no avail plummeting the efficiency metric. It is important to note that for the Qwen3-vl:2B, max tokens is set since the model ran an exceptionally long time without it. But reasoning efficiency in Qwen3.5 is phenomenal, looking at the reasoning length, we see a major decline while the rewards are higher than previous generations as well.
• My main gripe is tool sprawl on small models. They are fine-tuned to call tools, but spurious reads and directory listings are enough to wreck a harness. This is something I had noticed when using extremely small models, they keep reading files and listing directory files to no avail.

Overall, the trajectory still feels promising since more accurate and leaner token generation is driving the cost of model usage down.

Gemma

Comparing Gemma family will be slightly different since Gemma 3 is not a thinking series so I have added the instruct version of Gemma 4 series for better comparison. The PLE architecture was present in Gemma 3n models alongside regular dense models but Gemma 4 smaller models only use the PLE format.

• Within Gemma 3 series, the 3n outperform the dense models (2B and 4B) by a large margin, Gemma 4s E2B and E4B have also improved significantly with a slight rise in tool call for the 4EB which is not ideal but when run in thinking mode the E4B does not make any tool calls.
• The MoE model does more tool calls than dense models but overall the performance is not bad, and its reasoning efficiency is better than the dense.
• Thinking variants improve exceptionally in smaller sizes but their main contribution to large ones is to reduce tool calls.

Unnecessary Commands

The benchmark does not require tool calls, yet the tasks are still “edit this code,” which is exactly the topic harnesses training optimizes for. That makes it a good stress test for restraint: does the model stay inside the stated action space, or does it reach for shell/file tools anyway?

The graph below shows command distributions across all runs.

One observation is the usage of NPX and NPM which I think are the most non-sensical commands. Editing folder structure by removing is a breaking change which will degrade the model response. Navigation to other repos are useless as well since everything needed is already provided. But doing find, ls or cat are somewhat more acceptable still redundant. Now imagine these models will make these kind of mistakes wasting context and compute across multiple steps or completely derail the process within harness.

A more detailed view shows that release families do have a different distributions of tool calls, some observations:

• Gemma 4 has the most tame cmds where there are just for inspection.
• The Qwen3 series and subseries (Qwen3-VL, Qwen3.5) all show a diverse range of CMDs pinpointing a very bad understanding of how to handle the issue. The main uptick is within the smaller models making them fragile to handle tool calling and reasoning.
• Gemma 4 Instruct version does worse than Gemma 4 thinking but still way better than Qwen releases.

Conclusion

There is value in understanding the ins and outs of benchmarks, potentially by rewriting them in other libraries (like OpenEnv or verifiers). These benchmarks are intended as proxies for real-world use cases in specific categories. As one goes through the benchmarks, the shortcomings of benchmarks will be highlighted, like for instance most of the questions within the Next.js eval are extremely simple and I would argue getting 30% correct is not an accomplishment for the models. There are numerous other similar benchmarks that papers show going from 0.10 to 0.4 but if one actually goes through the tasks, one might realize that anything below 40% is useless in that benchmark. The point is that most of the model card benchmarks are smoke and mirrors except for some which are genuinely difficult or open-ended in such a way that is not easy to saturate. And obviously the benchmark ecosystem is an evolving space with new benchmarks coming out on an hourly basis.

Additionally PLE seems to be the differentiating factor in Gemma models, this opens up new research avenues around further optimization of PLEs: maybe it does not have to be “per layer” so it can be scaled to the 20-30B range. One major usecase that I anticipate to take over for LLMs is embedding them in all apps and have them guide or handle operations, rendering every day apps behave like cursor. New architectures are being developed just to address these issues which falls under the intersection of verbatim instruction following, creativity, and interleaved reasoning with other modes like tool calling.

Writing-zero is an approach focused on increasing performance on non-verifiable tasks such as creativity. The paper uses Bootstrapped Relative Policy Optimization (BRPO) which is inspired by GRPO but the main difference is that it uses LLM-as-Judge or a generative reward model (GenRM) to assigning reward per rollouts. Additionally there are no normalization within GRPO.

The main idea is to utilize high quality data to develop a GenRM that specializes in choosing between options. The paper claims that using GenRM the model gets better at picking in between options in tasks that it was not directly trained on.

Data Synthesis

As part of their cold-start SFT, the researchers utilized high quality dataset where the size of the dataset was doubled by changing the order at which two responses are passed within the prompt. I have created a similar dataset using different LLMs and the LitBench dataset. Snippet below shows the process of alternating response to avoid position bias.

prompt_1 = '''
You are a helpful assistant
...
responses:
...
[Response_A]
[Response_B]
....
'''

prompt_2 = '''
You are a helpful assistant
...
responses:
...
[Response_B]
[Response_A]
....
'''

Each prompt is repeated twice (once rejected first, once accepted response first), if the LLM is able to choose the correct response each time then we keep that reasoning trace.

Self-Critique principle generation (SCP)

The paper builds on the work of Liu et al. (2025) where self-critique principle generation is utilized. It is a prompting technique asking the LLM to come up with a rubric and score each response against the rubric. The prompt looks similar to this:

.....
### 1. Develop Specific Evaluation Criteria for Literary Creativity
Based on the literary nature of the content and focusing specifically on creativity as the primary evaluation metric, refer to the **General Evaluation Criteria** to develop **Specific Evaluation Criteria** tailored to the current question. 
The specific criteria should supplement or adjust for the particular need and complement the general criteria. 
The specific criteria should include: 
- Dimension descriptions 
- A 1-10 scoring scale

### 3. Comparative Evaluation Using both the **General Evaluation Criteria** and the **Specific Evaluation Criteria**, conduct a comparative analysis of the two assistants’ responses for each dimension.

### 4. Weight Allocation Based on the general and specific evaluation criteria, allocate weights for each specific item. Ensure the total weight sums to 100%. Note: If any response scores low on the harmfulness criterion, increase the weight of harmfulness.

### 5. Scoring Method Score each evaluation dimension separately on a scale of 1 to 10, where 1 means completely unsatisfactory and 10 means fully satisfactory. After scoring, calculate the weighted average score for each response based on the weights of each dimension, resulting in a comprehensive score between 1 and 10 for each response.
————
.....

Training

The prime-rl stack breaks the process into these components:

1. Orchestrator: Generates rollouts using the verifiers environment passed and computes advantage.
2. Train: Forward and backward pass is handled once orchestrator gets the reward and logprobs.
3. Inference: Hosts the model to be trained or the judge model used for training.

And separate .toml files need to be passed for each in cmd:

uv run rl \
  --trainer @ examples/writing_zero/GenRM/rl/train.toml \
  --orchestrator @ examples/writing_zero/GenRM/rl/orch.toml \
  --inference @ examples/writing_zero/GenRM/rl/infer.toml \
  --model.name ... \
  --wandb.project ... \
  --wandb.name ...

GenRM

1. Cold-start fine tunning: This step is a simple SFT using the synthesized data.
2. GRPO: The LitBench environment is utilized so the model can apply the same reasoning trace as in synthesized data on larger corpora of data.

Bootstrapped Relative Policy Optimization (BRPO)

Here is the simple explanation of BRPO:

1. Generate rollouts for prompts
2. Put them into group_sizes
3. within each group, at random pick one as reference
4. Use GenRM to do pair-wise comparison within the group, if reference gets a bigger score then return 1 else return -1
5. Use raw 1 and -1 values for updating the weights, NO NORMALIZATION

So the main difference with GRPO is the use of GenRM and there being no normalization.

Implementation

Within the Prime Intellect stack, in order to apply BRPO, we need an environment in which we go through all the steps so the orchestrator can return the advantages (which in this case are the same as rewards)

For that, w0-brpo is implemented where it generates all rollouts, breaks into groups, choose on at random then assigns reward based on GenRM. Here are some details about the environment:

• Rubric override: The rubric is where the grouping happens and a reference answer is chosen at random and added to its state. Then within the reward function, reward is generated via the pair-wise comparison.

class BootstrapRubric(vf.Rubric):
    def __init__(self, group_size: int = 2, **kwargs):
        super().__init__(**kwargs)
        self.group_size = group_size
    
    async def score_group(self, states: list[State], **kwargs):
        index_to_group = {}
        for i, state in enumerate(states):
            index_to_group.setdefault(state['example_id'], []).append(i)

        for indices in index_to_group.values():
            for inner_idx in range(0, len(indices), self.group_size):
                sub_states = [states[i]['completion'][0]['content'] for i in indices[inner_idx:inner_idx + self.group_size]]
                ref_answer = random.choice(sub_states)
                for i in indices[inner_idx:inner_idx + self.group_size]:
                    states[i]['ref_answer'] = ref_answer

• Reward Function: Uses state to access the reward function and does the pair-wise comparison.

   def make_rf():
        async def brpo_reward_function(prompt, completion, state: List[State]):
            answer = completion[0]['content']
            ref_answer = state['ref_answer']
              
            if answer == ref_answer:
                return 1
              
            loaded_prompt = load_dataset('dmnsh/W0_GenRM', name='prompt_template_generation', split='SPC').filter(lambda ex: ex['dataset'] == state['info']['db'])[0]
              
            spc = SPC(
                system_intro=loaded_prompt['sys_intro'],
                instructions=loaded_prompt['instructions'],
                general_eval_guide=loaded_prompt['general_eval_guide'],
                dialogue_content=loaded_prompt['response_content'],
                output_requirements=loaded_prompt['output_requirements']
            )
            final_prompt = spc({
                'prompt': prompt[0]['content'],
                'response_1': answer,
                'response_2': ref_answer
            })
                      
            judge_response = await maybe_await(
                judge_client.chat.completions.create,
                model=judge_model,
                messages=[{"content": final_prompt, "role": "user"}],
            )
            judge_response = str(judge_response.choices[0].message.content)
            match = re.search(r'\\boxed\{([\d.]+),\s*([\d.]+)\}', judge_response)
                          
            if not match:
                state['judgments'] = {"alt": ref_answer, "score_A": -1, "score_B": -1}
                return -1
              
            score_A, score_B = float(match.group(1)), float(match.group(2))
            state['judgments'] = {"alt": ref_answer, "score_A": score_A, "score_B": score_B}
            return 1 if score_A > score_B else -1
          
        return brpo_reward_function
  

Now another details that matters is that when passing advantage keyword inside orch.toml file, we need to pass it as ‘None’ since the default behavior subtracts the mean.

....
batch_size = 1024
rollouts_per_example = 16

advantage='None'
....
[environment]
id = "dmnsh001/w0-brpo"
....

Evaluation results

Minimum training yields the same results as the paper suggested wherein BRPO on creative tasks does increase models ability on selecting better when given multiple choice questions (RewardBench2). This is evident by 2% increase (4.18% of original score) in RewardBench and 0.148 increase within WritingBench

Further improvement

The synthesized data can be more diverse in terms of topics and RL training parameters can be further optimized. Also a larger model can be utilized.

Context

MoE was first introduced in 2017 and within the last few years there has been steady rise in deploying this algorithm.

This work is expands on previous works

• Sparse Upcycling (Komatsuzaki et al., 2023)
• Brain-Train-Mix (BTX) (Sukhbaatar et al., 2024)
• Branch-Train-Merge (BTM) (Li et al., 2022)

Novel approach utilized in Nexus is using a dynamic router within MoE

• Usual MoE routers have fixed number of experts
• Usual MoE only route using the tokens
• Nexus router uses domain embeddings and expert embedding for routing hence new MoEs could be added.

Sparse Upcycling

• Training mixture-of-experts from dense checkpoints
• Take a dense model and make it MoE by only training the router
• Keep the transformer bit intact, copy the FFN part, add multiples of it with a router

Branch-Train-Mix (BTX)

• Training mixture-of-experts from dense checkpoints
• Take a dense model and make it MoE by only training the router
• Keep the transformer bit intact, copy the FFN part, add multiples of it with a router
• Brain-Train-Merge: Averages everything and no routers.

Nexus

Domain embeddings are utilized with SwiGLU to generate the expert embeddings which later on determine which router should be used given the tokens. This approach is a kin to:

1. Mahabadi et al., 2021 (Parameter-efficient multi-task fine-tuning)
2. Üstün et al., 2022 (Hyper-X)

$e_i = P_r(d_i)$ (Domain to Expert Embeddings)

$= W_2 \cdot \text{SwiGLU}(W_1 \cdot d_i)$

Note: Similar to DeepSeek V3, it keeps a shared FFN.

• Mixing expert lms into a mixture-of-experts lm
• Merges already expert LLMs
• Takes their FFNs and averages transformer bits to create a shared body

Summary Table

Expermintal setup

• Two model sizes: 470M, 2.8B
• Five different categories
• Trained an expert then added it to study properties

Results for Upcycling

Result: Expert selection

Ablations

1. Effects of load balance weights

1. Altering data training composition
2. Effectiveness of domain embedding

embed_effectiveness</img>

Similarity to Hierarchical Reasoning?

Similar to HRM, this approach involves a low-level and high-level a operatives where the high-level drafts out the overall strtucture and planning, and multiple low-level modules evolve the answers then those answers are utilized to refine the original draft/plan. And this continues multiple times.

Ppaer Notes

Inspiration from human research: cycles of searching, reasoning, and revision. Humans do not write one word at a time sequentially; there is a process of planning out, writing each part then doing revisions.

Previous works focuses test-time scaling

• CoT
• Best-of-n sampling
• Monte Carlo Tree Search
• Debate mechanism
• Self-refinement loops

The approaches are not cohesive when it comes to drafting, searching and refining

A: Denoising with retrieval

• Initial research report drafted by LLM - De-noised using external information

  B: Self-Evolution

• Each individual component also goes through its own optimization process
• Encourage the exploration of diverse knowledge

Previous approaches (HuggingFace OpenDR, GPT Researcher, Open Deep Researcher) employ a linear or parallelized process of planning/search/generation -> loss of global context and miss critical dependencies during the research process.

Draft Diffusion

Draft-centric approach remains cohesive, provides a dynamic guide for the research direction; mitigating information loss.

• Stage 1
  • Dedicated LLM generates a structured plan
  • Outlining key areas
  • Initial scaffolding
  • Guide subsequent information-gathering process.
  • Passes each action to agent steps
• Stage 2
  • Answer and Question sub-agents
  • One generates question related to the topic the other one answers it
• Stage 3: synthesis

Componet-wise evolution

1. Produce multiple responses to one search query
2. Each asnwers is assesed in an LLM-as-judge for feedback
3. With the feedback each answer adjusts
4. Merged multiple answers into one

Evalution

This is a multi-step and agent environemnt and it is not straightforward to evaluate. Here is the collected metrics:

• high-quality human judgement annotations
• calibrate LLM-as-judge calibrated with humna preference
• calibrated LLM-as-judge as the final evaluator

Metrics

Helpfullness and Comprehensiveness

• Helpfullness
  1. Satisfying use intent
  2. ease of understanding
  3. accuracy
  4. appropiate language
• Comprehensiveness: Absence of missing key information

Side-by-side quality comparison

AKA pair-wise evaluation with these options between A and B: 1) Much Better If A is both more helpful and more comprehensive than B 2) Better If A is more helpful than B and equally comprehensive as B, or if A is more comprehensive than B and equally helpful as B 3) Slightly Better If A is more helpful but less comprehensive than B 4) About The Same If none of the above conditions are met. The same logic applies when B is better than A.

LLM-as-Judge

two benchmarks used: LongForm Research and DeepConsult

Previous approaches did not use human raters to calibrate the LLM-as-judge

Ablation Studies

Advanced LLMs perform poorly without search tools

• Gemini2.5-pro on full HLE: 20%, on HLE-search: 8%
• Looking at table below, it is evident that all the loss of the model (from 20% to 8%) was due to lack of search

Self-evolution working better

Graph below shows how self-evolution picks up more key points cumulatively as each search step goes on.

Conclusion

• Report generation as a diffusion process
• Preliminary draft is the research direction
• Demonstrates superior performance for long report generation

Paper: https://www.alphaxiv.org/abs/2506.21734

Context

Abstract Reasoning Corpus (ARC)

This paper aims at solving ARC using minimal amount of computation with a new archtecture.

Thinking Fast, Thking Slow:

• Lots of research papers refer to this book
• The book talks divides brain’s thinking process into slow and fast
  • Fast: Fast, Automatic, Intuitive
  • Slow: Deloberate, effortful reasoning

Introduction

Previous reasoning models use CoT (Chain-of-Thought), downsides:

• Brittle task decomposition: Single misstep can break the chain
• Extensive data requirements: Large training data
• High latency: Generates loads of token; slowing down inference

WHat are they offering: Latenet Reasoning

• Language is for human communication and ideas/thoughts are compressed effectively without translating back to language
• The model operates within its own hidden state

Inspired by human brain

• High-level (slow) part and low-level (fast) part interact
• The low-level exectures ideas based on the global knowdlege stored in High-level
• High-level gets feedback from low-level and adjusts
• Hierarchy and multi-stage

Datasets

Result

• Models being compared are general purpose
• Not tested on ARC-3

Method

Data + Augmentations

1000 From each (ARC, Sudoku, Maze) + ==≥ 3,831,994

• ARC: transition/rotations/flips/color permutations
• Soduku: used band and digits permuations
• Maze: No change, raw data used

H-L combo

• H-level updates after T steps of L-level
  • When L-level reeaches local equilibrium
• H-level’s world-view changes; L-level resets -≥ new computation path
• N (number of H update) x L (number of l updates) increase the reasoning depth

Memory footprint

Backpropagation Through Time (BPTT) saves model parameters at each time step and backpropagates.

• Biological implausibility: Human brain does not do this

This approach is O(1): Constant memory; uses one-step gradient approximation::uses the first and last state of H and L level, intermediary steps as constants.

• Aligns with idea of local rule in brian
• Brain does not utilize all previous computations for learning

More Detailed Result

ARC-AGI Performance Comparison (HRM vs. Baselines)

🔍 Key Insights:

• HRM is in a different league: Despite using no pretraining, no CoT, and 1,000× fewer parameters, it outperforms all CoT-based models on ARC-AGI-1.
• Efficiency: HRM achieves 40.3% on ARC-AGI-1 using only 960 training examples and 27M parameters — a level of data and parameter efficiency unmatched by any other model.
• CoT limitations: Even the strongest CoT models struggle with ARC-AGI-2, which requires compositional, multi-step abstraction — suggesting CoT has fundamental limits on novel reasoning.
• Architecture > Scale: HRM proves that better internal reasoning design can beat scaling alone — a shift from “bigger is better” to “deeper, structured computation wins.”

This hierarchical view underscores HRM’s significance: it achieves superior reasoning not by being larger or more data-hungry, but by thinking differently — deeply, internally, and adaptively.

Related

Resembles these:

• VAEs, Diffusion
• Intuitive physics understanding emerges from self-supervised pretraining on natural videos
• A Path Towards Autonomous Machine Intelligence

Result

Mixture of Depths

Mixture of Depths (MoD) is an optimization technique for transformer models that allows different tokens to use varying amounts of computation by selectively routing tokens through layers. Instead of processing every token through every layer, MoD uses learned routing scores to determine which tokens receive full self-attention computation and which can skip layers with just residual connections. This adaptive approach exploits the natural variation in token complexity—simple tokens like “the” or “and” require less processing than semantically rich tokens—achieving 2-3x inference speedup with minimal quality degradation by dynamically allocating computational resources where they’re most needed.

Reference: Raposo, D., Ritter, S., Santoro, A., Wayne, G., Weber, T., Botvinick, M., van Hasselt, H., & Song, F. (2024). Mixture-of-Depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258.

<div align=”center” style=”margin-top: 20px; padding: 15px;; border-radius: 8px>

MoD Token Filtering Process 🔵 (Active tokens) 🔴 (Elimintaed Tokens)

The Mixture of Depths (MoD) algorithm progressively reduces the token set through recursive filtering, retaining only the most semantically relevant tokens for downstream processing.

</div>

Mixture of Recursions

Mixture of Recursions (MoR) extends the efficiency principles of Mixture of Depths by introducing hierarchical token filtering through recursive processing stages. While MoD focuses on layer-wise token routing within a single forward pass, MoR applies token-level selection across multiple recursive iterations, progressively refining the active token set at each stage. Inspired by both MoD’s adaptive computation allocation and token-level pruning techniques, MoR creates a multi-stage filtering pipeline where tokens compete for computational resources across recursive depths rather than transformer layers. This hierarchical approach allows for more aggressive token reduction while maintaining semantic coherence, as each recursive stage can focus on increasingly refined subsets of the most relevant tokens, potentially achieving even greater efficiency gains than traditional depth-based routing methods.

Note: This appears to be a novel approach building on established MoD principles - specific citation pending publication.

<div align=”center” style=”margin-top: 20px; padding: 15px;; border-radius: 8px>

MoR Token Filtering Process 🔵 (Active tokens) 🔴 (Elimintaed Tokens)

The Mixture of Ranks (MoR) algorithm progressively reduces the token set through recursive filtering, retaining only the most semantically relevant tokens for downstream processing.

</div>

Caching Mechanisms: MoD vs MoR

The caching mechanism is a critical optimization in both MoD and MoR approaches, but they differ significantly in their implementation and efficiency. In MoD, caching occurs at the layer level where skipped tokens maintain their previous layer representations, requiring cache invalidation and updates as tokens move through different routing decisions across layers. MoR introduces a more sophisticated hierarchical caching system that leverages the recursive nature of its filtering process, allowing for persistent token state across recursive iterations and enabling more efficient cache reuse patterns.

Computational Efficiency Analysis

Traditional MoD Approach (3 × 5×5 grids):

• Memory Operations: 3 × (5² = 25) = 75 cache entries
• Attention Complexity: 3 × O(25²) = 3 × 625 = 1,875 operations
• Cache Invalidations: High frequency due to layer-wise routing changes
• Total Memory Footprint: 75 × d_model bytes (constant across layers)

MoR Hierarchical Approach (5×5 → 3×3 → 2×2):

• Memory Operations: 25 + 9 + 4 = 38 cache entries (49% reduction)
• Attention Complexity: O(25²) + O(9²) + O(4²) = 625 + 81 + 16 = 722 operations (62% reduction)
• Cache Invalidations: Minimal due to hierarchical persistence
• Total Memory Footprint: Progressive reduction from 25d to 4d model bytes

Intermediary Mapping Matrices

MoR maintains efficient token relationships through sparse mapping matrices between recursion levels:

First Mapping Matrix (25→9):

M₁ ∈ ℝ^(9×25) where M₁[i,j] = {
  1 if token j maps to position i in reduced grid
  0 otherwise
}

• Sparsity: 9/25 = 36% non-zero entries
• Storage: 9 indices instead of 225 full matrix entries

Second Mapping Matrix (9→4):

M₂ ∈ ℝ^(4×9) where M₂[i,j] = {
  1 if token j maps to position i in final grid
  0 otherwise
}

• Sparsity: 4/9 = 44% non-zero entries
• Storage: 4 indices instead of 36 full matrix entries

Efficiency Gains:

Memory Efficiency:

• Traditional: 75 cache entries across 3 layers
• MoR: 38 cache entries with progressive reduction
• Savings: 49% memory reduction

Computational Efficiency:

• Traditional: 1,875 attention operations
• MoR: 722 attention operations
• Savings: 62% computational reduction

Cache Management:

• Traditional: O(n) invalidations per layer transition
• MoR: O(log n) mapping updates per recursion
• Savings: Logarithmic vs linear cache management overhead

The hierarchical structure enables compound efficiency gains where both memory and computation scale favorably, while sparse mapping matrices maintain token relationships with minimal overhead.

In recent years, the field of artificial intelligence has witnessed remarkable advancements, particularly in the development of Mixture of Experts (MoE) models. These models, which leverage the power of multiple specialized sub-models or “experts” to handle different aspects of a task, have gained significant traction due to their ability to scale efficiently and improve performance across a wide range of applications. Unlike traditional monolithic models, MoE architectures dynamically route inputs to the most relevant experts, enabling more efficient use of computational resources and often achieving superior results.

Amid this growing interest in MoE models, DeepSeek has emerged as a notable player, with its recent releases of the DeepSeek R1 and V3 models capturing the attention of the AI community. These models represent a significant leap forward in terms of both architecture and performance. DeepSeek R1 introduced a novel approach to expert routing and model scaling, while the subsequent DeepSeek V3 further refined these techniques, incorporating advanced training methodologies and optimization strategies. The result is a family of models that not only push the boundaries of what is possible with MoE architectures but also demonstrate remarkable efficiency and adaptability across diverse tasks.

The surge in interest around DeepSeek R1 and V3 underscores the potential of MoE models to address some of the most pressing challenges in AI, from scaling to generalization. As the field continues to evolve, these models are likely to play a pivotal role in shaping the future of AI research and applications, offering a glimpse into the next generation of intelligent systems.

Setup

I am using the Digit Recognizer competition on Kaggle to get a public score for the models. The goal is to start from a minimal base model and add MOE for the linear part and the CNN part and collectively comparing them. The implementation code is also here.

Sanity Check

To establish a baseline for our MoE experiments, I implemented a simple CNN architecture. This baseline model serves as our reference point to evaluate the effectiveness of the MoE implementations. The architecture consists of:

• Two convolutional layers with ReLU activation
• Max pooling layers
• Two fully connected layers
• Dropout for regularization

After training for 100 epochs, this basic CNN achieves an accuracy of 0.98407 on the MNIST test set. This performance metric will serve as our benchmark for comparing the various MoE implementations that follow.

Routing Collapse

One common challenge with MoE architectures is “routing collapse”. The “route” refers to the selection process of which expert to use for a given input where the model falls into a pattern of only using a small subset of experts. This happens because:

1. Early in training, some experts may perform slightly better by chance
2. These better-performing experts get selected more frequently
3. With more practice, these experts improve further, creating a feedback loop
4. Other experts become neglected and never improve

Load Balancing Solutions

To prevent routing collapse, we implement three types of losses that were introduced in various MoE research:

1. Diversity Loss: Encourages the gating network to use all experts by maximizing the entropy of expert selection probabilities Shazeer et al., “Outrageously Large Neural Networks” (2017)

2. Importance Loss: Ensures each expert handles a similar total amount of input across the batch by penalizing deviations from the mean usage Lepikhin et al., “GShard: Scaling Giant Models with Conditional Computation” (2020)

3. Overflow Loss: Prevents individual experts from being overloaded by penalizing usage above a specified capacity threshold Fedus et al., “Switch Transformers” (2021)

These losses are combined with the main classification loss during training to ensure balanced expert utilization. The combination of these techniques has proven effective in large-scale models like GShard and Switch Transformers.

Shared Layer concept

A key innovation in modern MoE architectures is the concept of shared layers. Unlike traditional MoE models where each expert operates independently, shared layers introduce common components that are used across all experts. This approach, pioneered in DeepSeek’s architecture, helps to:

1. Reduce model parameters by sharing common features across experts
2. Improve training stability by providing a consistent base representation
3. Enhance knowledge transfer between experts through shared components

DeepSeek’s implementation of shared layers, as described in their technical report, demonstrates that this approach can significantly improve model efficiency while maintaining or even improving performance. The shared layers act as a common foundation that all experts build upon, allowing for more specialized expert networks to focus on their specific tasks while leveraging shared knowledge.

DeepSeek Technical Report

MOE on classification

Without shared expert layer and 30 epoch training with only one Linear MoE: | Model | Hidden Size | Training accuracy | Public Score | Score Difference | | ——— | ———– | —————– | ———— | —————- | | MOE(1,5) | 64 | 98.86 | 97.98 | 0.88 | | MOE(2,5) | 64 | 99.28 | 98.21 | 1.07 | | MOE(3,5) | 64 | 99.63 | 98.76 | 0.87 | | MOE(6,10) | 16 | 99.06 | 97.99 | 1.07 |

Neural networks have been able to accomplish tasks that previously seemed impossible. Self-driving cars and AI chatbots rely on these networks. However, neural networks are black box algorithms meaning it is difficult to reason about what happens inside of the network, and why it outputs what it does. For instance, the output of a tree-based model can be traced but not a neural network’s. Engineers at Google experimented with different layers of the network to understand the inner workings of each layer. The goal of the experiments was to analyze which features within an image are detected within a layer. They realized that layers at different depths will have unique sensitivities: “If we choose higher-level layers, which identify more sophisticated features in images, complex features or even whole objects tend to emerge” [1]. Pretrained ResNet50 is the feature extractor used for the original DeepDream algorithm. During the experimentation, the algorithm aims to exaggerate the detected features in each layer. In other words, the algorithm optimizes the image so the chosen layers within the neural network can better see the features they are seeing. The exaggeration reaches a point where the model starts to see things in the images that are not there. In other words: it is imaging or hallucinating. ResNet50 is trained on animal images which explains why the extracted features will start creating various shapes in the image of a dog. Later the engineers realized this could be seen as artwork and published their code [1].

This is a personal project in which I have tried to understand the DeepDream generative algorithm a bit better. I am using the kaggle dataset [2] which is a dataset containing various famous paintings in different styles. I have run the algorithms with different parameters and tried to answer some of the questions: What is captured? Which original figures are more likely to be picked given different weights? The main goal for me is to better understand Generative AI and inner workings of deep learning algorithms. The code for this algorithm is hosted on kaggle [3] and it is influenced by [4].

Overall architecture of Deep Learning

Deep learning is the process of training deep neural networks to learn patterns of data. Neural networks are inspired by the workings of our brains, and each layer is similar to a neuron. Each neuron has at least one input and at least one output. A neural network is made up of the stacked neurons where the input of a neuron is the output of a preceding one. A neural network with too many layers is called “deep”. The first layers are referred to as shallow layers but as the number of preceding layers for a layer increases, they are referred to as deeper layers.

Algorithm

The goal of the algorithm is to exaggerate what the layers are seeing in the picture to be able to analyze their different properties. Hence, in the training process the input image is being updated based on the gradients of feature extractors. Instead of updating the weights of the models, the image is updated and after each iteration, the features detected by layers are imposed upon the image. In other words, the image is updated to resemble what the layers are extracting.

DeepDream algorithm follows a generative approach: there is no labeled data, and the model's output is similar to the input. Layers from ResNet50 are used to extract features: mixed4, mixed5, mixed6, and mixed7 (name of the layers). In each iteration of training, a forward pass through the network will yield certain features, and the loss is calculated based on the extracted features from each layer. At the end of the iteration the input image is updated using the gradients for that loss. This process changes the input image so the shapes identified by each layer becomes more illusive after each iteration. This process informs what the network layers are actually picking up. Given that we are using multiple layers, there is going to be a correlation between what different layers are seeing given that each is influencing the input as the model iterates.

An important component is the associated weight for each of the feature layers. Each layer has a different weight. This will cause the layer with a higher weight to influence the shapes within the image more than other layers. This weighting mechanism is at the heart of analysis. Changing the weights, and comparing the results we can reason about the features being captured at different levels.

Loss function

The loss function used for the algorithm is the average of the outputs from the feature layers. This metric signifies the sensitivity of the feature layers to the image. And when updating the model using this metric, these sensitivities are optimized so that the features can be seen more clearly.

Model sees a shape that is not present but nevertheless the loss function encourages to keep seeing it more clearly. The weights used for experimentation come in handy here since the loss for each layer is weighted differently. Some layer’s features are encouraged more than the other ones.

for weight, feature_layer in set(weights, feature_layers):
    features = feature_layer(image)
	feature_layer_loss =  mean_across_channels(squared(features))
	overall_loss += weight * feature_layer_loss

Experiments

Within a deep neural network shallow layers detect the general geometric shapes and features such horizontal lines. And deeper layers are capable of identifying more complex and abstract shapes such as faces or eyes.

The most basic experiment is to choose a set of layers, keep these layers constant but change the weights. This will illustrate the difference between features captured in shallow and deep layers. Four sets of weights are chosen, each focuses on certain aspect of the analysis:

1. Set 1 is the benchmark, it is a moderate weighting.
2. Set 2 focuses on the deeper layer more than the shallower ones.
3. Set 3 is the opposite of set 2.
4. Set 4 is a combination in which all the weights are high, this set is used so it can be compared with the first set.

Visual differences between the weightings in each iteration

The weights that emphasize more abstract geometric shapes begin to detect them early on, leading to subtle differences between generated images. If an image is initially altered based on these basic shapes, the model may perceive or “hallucinate” abstract forms differently than if it were focused on finer details from the start. This early influence of shape-based weighting can create unique visual interpretations that diverge significantly as the model continues to refine the image.

Analyzing the Gif below helps visualize the “Algorithmic Logic” behind DeepDream. The painting is of waves crashing into rocks which in turn creates geometric shapes blending with different colourings. Taking a closer look: the model starts seeing Dog eyes within the rocks. It is as if the model is determined to see a “Dog” somewhere and the geometric shapes within the rocks is the perfect place to see one.

The Gif above used weight set 4 meaning that, abstract features were heavily subsidized in seeing a dog. How would it differ if the weighting favored simpler details? Gif below shows the process for weight set 1. The dog is not seen in the rocks immediately, in fact it is not seen in the rocks the way it used to. Since focusing on simple geometric features (straight lines or curves), the model blurs the sky more heavily and as it modifies the sand (bottom left), it starts to see small dog heads.

These two examples show how changing the weight shifts the model’s logic and sequentially affects what the model sees.

Similar effects across different images

DeepDream algorithm is using ResNet which is trained on pictures of dogs and cats. The algorithm optimizes the image in a way that transforms the shapes to what the ResNet layers perceive the image to contain. At each iteration certain features in the image are changed to look more like what ResNet is trained on. As an example: a plain part of the image such as sky will tend to become more distorted as the model progresses. I believe this happens because shallow layers are sensitive to subtle color changes and strokes. These subtleties will be sharpened to show the gap and it looks as if the model is on a drug and can see the sky properly. The difference between this part of the image is mostly similar across different weightings given that there is not much in that part of the image.

Comparing weightings

Figure below shows the difference the weighting of the layers causes. When the deeper layers have a higher weight, the model tries to see abstract features from the first iteration. This will cause the output to mainly pick up strokes and distort those (Set 2). Whereas a good balance of shallow and deep layers produces an image where the shapes are gradually turned into face and eye of a dog, in set 1, the colors changes are depicted similar to dog ears and the top of the hair, multiple eyes are hallucinated. In Set 3, there are only eyes present within the picture but not face structure or ears, this shows that when the focus is only on shallow layers, only basic features are detected and in turn imagined. Whereas other sets create a full facial structure. In the image, only one eye is painted but all the models are imaging the other eye and what it might look like. This is very interesting given that this behavior is similar to the outputs of diffusion models.

Possible applications

If we have a small dataset, and want to augment data, can we utilize the model’s ability to hallucinate to generate data with positive class? For the leaf disease prediction: what if the benign images were given to the model so that it could hallucinate the diseased versions of them?

It is adding dogs to images, why can it not add other things and be used for the augmentation process. For instance, can we augment X-ray images by imaging early stage tumors? This has proven to be difficult since the model is not deterministic and navigate the latent space, a more discrete approach with a specialized pre-trained network might be doable.

There are certain fields where the size of good quality data is small and makes it difficult to provide deep learning solutions. Medical field is one of these fields. Given a large amount of CT scans, a deep model will get a better accuracy however due to various logistic reasons, CT scan datasets are not as large to yield high results. Now what if the generative AI similar to deep dream could hallucinate and imagine what a normal CT scan would look like if there was a tumor present? It is seeing dogs where there are no dogs, in theory it could see a tumor in a benign CT scan. But again the problem is that the feature extractor base of the model should be trained on a large dataset of tumors to be able to imagine it, and as mentioned: the datasets in this field are not large enough.

References

1. https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html
2. https://www.kaggle.com/datasets/ikarus777/best-artworks-of-all-time
3. https://www.kaggle.com/code/damoonshahhosseini/dlwp-12-2
4. https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/chapter12_part02_deep-dream.ipynb