Introduction

Writing-zero is an approach focused on increasing performance on non-verifiable tasks such as creativity. The paper uses Bootstrapped Relative Policy Optimization (BRPO) which is inspired by GRPO but the main difference is that it uses LLM-as-Judge or a generative reward model (GenRM) to assigning reward per rollouts. Additionally there are no normalization within GRPO.

The main idea is to utilize high quality data to develop a GenRM that specializes in choosing between options. The paper claims that using GenRM the model gets better at picking in between options in tasks that it was not directly trained on.

Data Synthesis

As part of their cold-start SFT, the researchers utilized high quality dataset where the size of the dataset was doubled by changing the order at which two responses are passed within the prompt. I have created a similar dataset using different LLMs and the LitBench dataset. Snippet below shows the process of alternating response to avoid position bias.


prompt_1 = '''
You are a helpful assistant
...
responses:
...
[Response_A]
[Response_B]
....
'''

prompt_2 = '''
You are a helpful assistant
...
responses:
...
[Response_B]
[Response_A]
....
'''

Each prompt is repeated twice (once rejected first, once accepted response first), if the LLM is able to choose the correct response each time then we keep that reasoning trace.

Self-Critique principle generation (SCP)

The paper builds on the work of Liu et al. (2025) where self-critique principle generation is utilized. It is a prompting technique asking the LLM to come up with a rubric and score each response against the rubric. The prompt looks similar to this:

.....
### 1. Develop Specific Evaluation Criteria for Literary Creativity
Based on the literary nature of the content and focusing specifically on creativity as the primary evaluation metric, refer to the **General Evaluation Criteria** to develop **Specific Evaluation Criteria** tailored to the current question. 
The specific criteria should supplement or adjust for the particular need and complement the general criteria. 
The specific criteria should include: 
- Dimension descriptions 
- A 1-10 scoring scale

### 3. Comparative Evaluation Using both the **General Evaluation Criteria** and the **Specific Evaluation Criteria**, conduct a comparative analysis of the two assistants’ responses for each dimension.

### 4. Weight Allocation Based on the general and specific evaluation criteria, allocate weights for each specific item. Ensure the total weight sums to 100%. Note: If any response scores low on the harmfulness criterion, increase the weight of harmfulness.

### 5. Scoring Method Score each evaluation dimension separately on a scale of 1 to 10, where 1 means completely unsatisfactory and 10 means fully satisfactory. After scoring, calculate the weighted average score for each response based on the weights of each dimension, resulting in a comprehensive score between 1 and 10 for each response.
————
.....

Training

The prime-rl stack breaks the process into these components:

  1. Orchestrator: Generates rollouts using the verifiers environment passed and computes advantage.
  2. Train: Forward and backward pass is handled once orchestrator gets the reward and logprobs.
  3. Inference: Hosts the model to be trained or the judge model used for training.

And separate .toml files need to be passed for each in cmd:

uv run rl \
  --trainer @ examples/writing_zero/GenRM/rl/train.toml \
  --orchestrator @ examples/writing_zero/GenRM/rl/orch.toml \
  --inference @ examples/writing_zero/GenRM/rl/infer.toml \
  --model.name ... \
  --wandb.project ... \
  --wandb.name ...

GenRM

  1. Cold-start fine tunning: This step is a simple SFT using the synthesized data.
  2. GRPO: The LitBench environment is utilized so the model can apply the same reasoning trace as in synthesized data on larger corpora of data.

Bootstrapped Relative Policy Optimization (BRPO)

Here is the simple explanation of BRPO:

  1. Generate rollouts for prompts
  2. Put them into group_sizes
  3. within each group, at random pick one as reference
  4. Use GenRM to do pair-wise comparison within the group, if reference gets a bigger score then return 1 else return -1
  5. Use raw 1 and -1 values for updating the weights, NO NORMALIZATION

So the main difference with GRPO is the use of GenRM and there being no normalization.

Difference between BRPO and GRPO; from the Writing-zero paper

Implementation

Within the Prime Intellect stack, in order to apply BRPO, we need an environment in which we go through all the steps so the orchestrator can return the advantages (which in this case are the same as rewards)

For that, w0-brpo is implemented where it generates all rollouts, breaks into groups, choose on at random then assigns reward based on GenRM. Here are some details about the environment:

  • Rubric override: The rubric is where the grouping happens and a reference answer is chosen at random and added to its state. Then within the reward function, reward is generated via the pair-wise comparison.
class BootstrapRubric(vf.Rubric):
    def __init__(self, group_size: int = 2, **kwargs):
        super().__init__(**kwargs)
        self.group_size = group_size
    
    async def score_group(self, states: list[State], **kwargs):
        index_to_group = {}
        for i, state in enumerate(states):
            index_to_group.setdefault(state['example_id'], []).append(i)

        for indices in index_to_group.values():
            for inner_idx in range(0, len(indices), self.group_size):
                sub_states = [states[i]['completion'][0]['content'] for i in indices[inner_idx:inner_idx + self.group_size]]
                ref_answer = random.choice(sub_states)
                for i in indices[inner_idx:inner_idx + self.group_size]:
                    states[i]['ref_answer'] = ref_answer
  • Reward Function: Uses state to access the reward function and does the pair-wise comparison. 
     def make_rf():
      async def brpo_reward_function(prompt, completion, state: List[State]):
          answer = completion[0]['content']
          ref_answer = state['ref_answer']
            
          if answer == ref_answer:
              return 1
            
          loaded_prompt = load_dataset('dmnsh/W0_GenRM', name='prompt_template_generation', split='SPC').filter(lambda ex: ex['dataset'] == state['info']['db'])[0]
            
          spc = SPC(
              system_intro=loaded_prompt['sys_intro'],
              instructions=loaded_prompt['instructions'],
              general_eval_guide=loaded_prompt['general_eval_guide'],
              dialogue_content=loaded_prompt['response_content'],
              output_requirements=loaded_prompt['output_requirements']
          )
          final_prompt = spc({
              'prompt': prompt[0]['content'],
              'response_1': answer,
              'response_2': ref_answer
          })
                    
          judge_response = await maybe_await(
              judge_client.chat.completions.create,
              model=judge_model,
              messages=[{"content": final_prompt, "role": "user"}],
          )
          judge_response = str(judge_response.choices[0].message.content)
          match = re.search(r'\\boxed\{([\d.]+),\s*([\d.]+)\}', judge_response)
                        
          if not match:
              state['judgments'] = {"alt": ref_answer, "score_A": -1, "score_B": -1}
              return -1
            
          score_A, score_B = float(match.group(1)), float(match.group(2))
          state['judgments'] = {"alt": ref_answer, "score_A": score_A, "score_B": score_B}
          return 1 if score_A > score_B else -1
        
      return brpo_reward_function

Now another details that matters is that when passing advantage keyword inside orch.toml file, we need to pass it as ‘None’ since the default behavior subtracts the mean.

....
batch_size = 1024
rollouts_per_example = 16

advantage='None'
....
[environment]
id = "dmnsh001/w0-brpo"
....

Evaluation results

Minimum training yields the same results as the paper suggested wherein BRPO on creative tasks does increase models ability on selecting better when given multiple choice questions (RewardBench2). This is evident by 2% increase (4.18% of original score) in RewardBench and 0.148 increase within WritingBench

Benchmark\Model PrimeIntellect/Qwen3-4B dmnsh/Qwen3-4b-W0-BRPO Change (%)
RewardBench2 0.478 0.498 +4.18%
Writing Bench 6.864 7.012 +2.16%

Further improvement

The synthesized data can be more diverse in terms of topics and RL training parameters can be further optimized. Also a larger model can be utilized.