Introduction

Ever since the release of OpenClaw, I have been trying to answer one question: What is the smallest LLM running on DGX Spark that can handle a simple cronjob? In my experience, there exists a major issue with regards to “instruction following”. Models will interpret the instructions in unexpected ways and cause havoc. Additionally, models will ignore or forget about instructions as the process progresses. Vague instructions are partially to blame. I ablated this by using larger models and reading their thinking traces. And even larger models tend to ask for clarification or they might make assumptions and continue but their assumptions are far superior to smaller models.

The open source (or open weight) models tested ranged from Cohere Command A, Hunyuan to more mainstream MiniMax, GLM (4.7 Flash, 4.7 Flash Reap), and Qwen. That is why the release of Gemma 4 models has been a great surprise to me since I had given up on models below the ~20B size for executing tasks. Gemma 4 series is capable way above their size group where the models below ~10B are capable of cohesive instruction following and doing interleaved tool calling with reasoning very effectively. This inspired me to benchmark the (quantized version of) recent releases against NextJS benchmark, and also compare the relative behavior evolution of the LLMs within their family groups.

Context

The original release of Next Js benchmark is interesting since the tasks would manually pass files within the prompts and give a specific ask then the model had to return the changes to files without tool calls or anything. This strictly a coding-related benchmark but I thought it may be a good proxy with problems I am facing in harnesses. The LLMs are trained to do tool calls and handle these kind of tasks but the prompts strictly say what needs to be done. Note that the prompt does not sat NO TOOLS but the model is expected to pick it up and interpret it.

Here is the specific format:

```
Please provide your changes as complete file contents. Use this format for each file you need to modify, create, or delete:
```file=path\/to\/file.tsx action=replace
complete file content here\n```

For example:
```file=app\/Component.tsx action=replace\nexport default function Component() {\n const new = \"code\";\n return <div><\/div>;
}\n```

```file=app\/NewComponent.tsx action=add
export default function NewComponent() {\n return <div>New component<\/div>;
}\n```

```file=app\/OldComponent.tsx action=delete```

Rules:
- Always specify an action: add, replace, or delete
- For add\/replace: provide the complete file content, not partial changes
- For delete: you can leave the content empty or omit it
- Use the exact file path relative to the project root
- Maintain proper indentation and formatting
- Include all imports and exports that the file needs
- Do not modify *.test.tsx files
```

I personally do not get anything from the model cards that are released everyday with each new model release (I guess that is why there is benchmarking mini-industry being formed) But with the large volume and fast pace release of open source models, I wanted to assess progress in the simplest way possible. That is benchmarking models on NextJS benchmark to track tool calls, reasoning efficiency (or output generation length) and speed.

Recent releases capacities

Qwen3.5 was released on February 16, 2026, and Gemma 4 on April 2, 2026. Both releases are natively multimodal with heavy emphasis on reasoning and agentic capabilities. Each release has a number of smaller and mid-size models; the larger models are either MoE or dense. Gemma uses the Per-Layer Embeddings (PLE) approach for smaller releases, where the effective computation amount is lower than actual model size (E2B with 5B LLM, E4B with 9B LLM). It is interesting to observe the effect of these architectural differences.

The graph below compares NextJS pass@1 for Qwen3.5, Gemma 4, Nemotron, and GLM-4.7 Flash. I define Reasoning efficiency as the squared reward divided by log10 of reasoning length. The metric aims to ensure the amount of tokens generated for reasoning is worth it. The easiest way to think about it: smaller Qwen models (< 8B) generate long reasoning traces riddled with “oh wait”s without a worthwhile response (not all tasks just complicated ones). The other metric that is important is tool call percentage to check what portion of responses involved doing tool calls.

Overall benchmark metrics — Overall metrics

Some observations:

Gemma 4 has the best models in all sizes, these models are exceptional at understanding that no tool calls are required. Especially the smaller ones, Gemma:4-E2B and Gemma:4-E4B are on par with Qwen3.5:9B except for slight drop in reasoning efficiency.
Within Qwen3.5, models below 9B are doing poorly with almost all the responses being tool calls.
Nemotron 3 series are hybrid (mamba+transformer) model family with with 4B, 30B-A3B, 120B-A12B and recent highly post-tuned versions. The smallest versino does not do well in reward part but it stays away from tool calls. One thing that catches my eye is Nemotron Cascade-2 which is the post-tuned version of 30B-A3B and the difference is amazing 10% reward increase, no tool calls in cascade-2 version, and it has a stronger reasoning efficiency. The Cascade-2 paper stresses the order of training data used to ensure different capabilities (reasoning on math vs coding vs instruction following) are not forgotten/overridden in between runs. And the results show significant improvements across the metrics. Still Cascade-2 lags behind Gemma4:26B-A4B
GLM-4.7 Flash underperforms compared to Gemma 4-26B-A4B but does better than Qwen3.5:35B-A3B. The REAP-ed version of it does worse as well where every metric gets worse.
The bigger models have done relatively worse, especially Nemotron Super 120B (this could be due to quantization) but presumably post-tuning in cascade-2 fashion could improve it.

Qwen

AliBaba has released Qwen3, Qwen3-Coder, Qwen3-vl, and now Qwen3.5 in less than a year. I was curious to see the progress for this specific benchmark. Here are some observations:

The small Qwen3.5 checkpoints reach for tools far more often than same-sized predecessors. The shift shows up in Qwen3-VL and steepens in Qwen3.5.
At a given size band, MoE variants tool-call more than dense ones.
The reasoning efficiency drops significantly with qwen3-vl series where large amounts of tokens are generated to no avail plummeting the efficiency metric. It is important to note that for the Qwen3-vl:2B, max tokens is set since the model ran an exceptionally long time without it. But reasoning efficiency in Qwen3.5 is phenomenal, looking at the reasoning length, we see a major decline while the rewards are higher than previous generations as well.
My main gripe is tool sprawl on small models. They are fine-tuned to call tools, but spurious reads and directory listings are enough to wreck a harness. This is something I had noticed when using extremely small models, they keep reading files and listing directory files to no avail.

Overall, the trajectory still feels promising since more accurate and leaner token generation is driving the cost of model usage down.

Qwen release progression — Qwen's progression

Gemma

Comparing Gemma family will be slightly different since Gemma 3 is not a thinking series so I have added the instruct version of Gemma 4 series for better comparison. The PLE architecture was present in Gemma 3n models alongside regular dense models but Gemma 4 smaller models only use the PLE format.

Within Gemma 3 series, the 3n outperform the dense models (2B and 4B) by a large margin, Gemma 4s E2B and E4B have also improved significantly with a slight rise in tool call for the 4EB which is not ideal but when run in thinking mode the E4B does not make any tool calls.
The MoE model does more tool calls than dense models but overall the performance is not bad, and its reasoning efficiency is better than the dense.
Thinking variants improve exceptionally in smaller sizes but their main contribution to large ones is to reduce tool calls.

Gemma release progression — Gemma's progression

Unnecessary Commands

The benchmark does not require tool calls, yet the tasks are still “edit this code,” which is exactly the topic harnesses training optimizes for. That makes it a good stress test for restraint: does the model stay inside the stated action space, or does it reach for shell/file tools anyway?

The graph below shows command distributions across all runs.

One observation is the usage of NPX and NPM which I think are the most non-sensical commands. Editing folder structure by removing is a breaking change which will degrade the model response. Navigation to other repos are useless as well since everything needed is already provided. But doing find, ls or cat are somewhat more acceptable still redundant. Now imagine these models will make these kind of mistakes wasting context and compute across multiple steps or completely derail the process within harness.

Command breakdown by model family — Command breakdown by family

A more detailed view shows that release families do have a different distributions of tool calls, some observations:

Gemma 4 has the most tame cmds where there are just for inspection.
The Qwen3 series and subseries (Qwen3-VL, Qwen3.5) all show a diverse range of CMDs pinpointing a very bad understanding of how to handle the issue. The main uptick is within the smaller models making them fragile to handle tool calling and reasoning.
Gemma 4 Instruct version does worse than Gemma 4 thinking but still way better than Qwen releases.

Conclusion

There is value in understanding the ins and outs of benchmarks, potentially by rewriting them in other libraries (like OpenEnv or verifiers). These benchmarks are intended as proxies for real-world use cases in specific categories. As one goes through the benchmarks, the shortcomings of benchmarks will be highlighted, like for instance most of the questions within the Next.js eval are extremely simple and I would argue getting 30% correct is not an accomplishment for the models. There are numerous other similar benchmarks that papers show going from 0.10 to 0.4 but if one actually goes through the tasks, one might realize that anything below 40% is useless in that benchmark. The point is that most of the model card benchmarks are smoke and mirrors except for some which are genuinely difficult or open-ended in such a way that is not easy to saturate. And obviously the benchmark ecosystem is an evolving space with new benchmarks coming out on an hourly basis.

Additionally PLE seems to be the differentiating factor in Gemma models, this opens up new research avenues around further optimization of PLEs: maybe it does not have to be “per layer” so it can be scaled to the 20-30B range. One major usecase that I anticipate to take over for LLMs is embedding them in all apps and have them guide or handle operations, rendering every day apps behave like cursor. New architectures are being developed just to address these issues which falls under the intersection of verbatim instruction following, creativity, and interleaved reasoning with other modes like tool calling.