These are notes for this paper.

Similarity to Hierarchical Reasoning?

Similar to HRM, this approach involves a low-level and high-level a operatives where the high-level drafts out the overall strtucture and planning, and multiple low-level modules evolve the answers then those answers are utilized to refine the original draft/plan. And this continues multiple times.

Ppaer Notes

Inspiration from human research: cycles of searching, reasoning, and revision. Humans do not write one word at a time sequentially; there is a process of planning out, writing each part then doing revisions.

Human Process

Previous works focuses test-time scaling

  • CoT
  • Best-of-n sampling
  • Monte Carlo Tree Search
  • Debate mechanism
  • Self-refinement loops

The approaches are not cohesive when it comes to drafting, searching and refining

A: Denoising with retrieval

  • Initial research report drafted by LLM - De-noised using external information

    B: Self-Evolution

  • Each individual component also goes through its own optimization process
  • Encourage the exploration of diverse knowledge

Previous approaches (HuggingFace OpenDR, GPT Researcher, Open Deep Researcher) employ a linear or parallelized process of planning/search/generation -> loss of global context and miss critical dependencies during the research process.

Comparison of other linear approaches

Draft Diffusion

Draft-centric approach remains cohesive, provides a dynamic guide for the research direction; mitigating information loss.

  • Stage 1
    • Dedicated LLM generates a structured plan
    • Outlining key areas
    • Initial scaffolding
    • Guide subsequent information-gathering process.
    • Passes each action to agent steps
  • Stage 2
    • Answer and Question sub-agents
    • One generates question related to the topic the other one answers it
  • Stage 3: synthesis
Planning aspect

Componet-wise evolution

Evolution within each component
  1. Produce multiple responses to one search query
  2. Each asnwers is assesed in an LLM-as-judge for feedback
  3. With the feedback each answer adjusts
  4. Merged multiple answers into one

Evalution

This is a multi-step and agent environemnt and it is not straightforward to evaluate. Here is the collected metrics:

  • high-quality human judgement annotations
  • calibrate LLM-as-judge calibrated with humna preference
  • calibrated LLM-as-judge as the final evaluator

Metrics

Helpfullness and Comprehensiveness

  • Helpfullness
    1. Satisfying use intent
    2. ease of understanding
    3. accuracy
    4. appropiate language
  • Comprehensiveness: Absence of missing key information

Side-by-side quality comparison

AKA pair-wise evaluation with these options between A and B: 1) Much Better If A is both more helpful and more comprehensive than B 2) Better If A is more helpful than B and equally comprehensive as B, or if A is more comprehensive than B and equally helpful as B 3) Slightly Better If A is more helpful but less comprehensive than B 4) About The Same If none of the above conditions are met. The same logic applies when B is better than A.

LLM-as-Judge

two benchmarks used: LongForm Research and DeepConsult

Previous approaches did not use human raters to calibrate the LLM-as-judge

Ablation Studies

Advanced LLMs perform poorly without search tools

  • Gemini2.5-pro on full HLE: 20%, on HLE-search: 8%
  • Looking at table below, it is evident that all the loss of the model (from 20% to 8%) was due to lack of search

Self-evolution working better

Graph below shows how self-evolution picks up more key points cumulatively as each search step goes on.

Conclusion

  • Report generation as a diffusion process
  • Preliminary draft is the research direction
  • Demonstrates superior performance for long report generation