Evaluation Agent
Efficient and Promptable Evaluation Framework for Visual Generative Models

Fan Zhang1*, Shulin Tian2*, Ziqi Huang2*, Yu Qiao1†, Ziwei Liu2†
(* equal contributions, † corresponding authors)
1 Shanghai Artificial Intelligence Laboratory    2 S-Lab, Nanyang Technological University   
Paper Video GitHub
Teaser.

An Example of Evaluation Agent. Existing evaluation methods typically assess visual generative models by extensively sampling from a fixed benchmark. In contrast, our Evaluation Agent framework requires only a small number of sampled images or videos, tailored to the user's specific evaluation request. Additionally, it goes beyond providing a simple numerical score by offering detailed explanations to the evaluation conclusions.

Video

Abstract

Recent advancements in visual generative models have enabled high-quality image and video generation, opening diverse applications. However, evaluating these models often demands sampling hundreds or thousands of images or videos, making the process computationally expensive, especially for diffusion-based models with inherently slow sampling. Moreover, existing evaluation methods rely on rigid pipelines that overlook specific user needs and provide numerical results without clear explanations. In contrast, humans can quickly form impressions of a model's capabilities by observing only a few samples. To mimic this, we propose the Evaluation Agent framework, which employs human-like strategies for efficient, dynamic, multi-round evaluations using only a few samples per round, while offering detailed, user-tailored analyses. It offers four key advantages: 1) efficiency, 2) promptable evaluation tailored to diverse user needs, 3) explainability beyond single numerical scores, and 4) scalability across various models and tools. Experiments show that Evaluation Agent reduces evaluation time to 10% of traditional methods while delivering comparable results. The Evaluation Agent framework is fully open-sourced to advance research in visual generative models and their efficient evaluation.

Pipeline of Evaluation Agent Framework

Overview of Evaluation Agent Framework. This framework leverages LLM-powered agents for efficient and flexible visual model assessments. As shown, it consists of two stages: (a) the Proposal Stage, where user queries are decomposed into sub-aspects, and prompts are generated, and (b) the Execution Stage, where visual content is generated and evaluated using an Evaluation Toolkit. The two stages interact iteratively to dynamically assess models based on user queries.

Comparing with Other T2I/T2V Benchmarks

Benchmark Analysis Customized Queries Supported Models # Required Samples Open Evaluation Request Support Dynamic Evaluation Open Tool-Use
FID / FVD ❌ ❌ T2I / T2V 2,048 ❌ (Fixed-Form) ❌ ❌
T2I-CompBench ❌ ❌ T2I 18,000 ❌ (Pre-Defined) ❌ ❌
VBench ❌ ❌ T2V 4,730 ❌ (Pre-Defined) ❌ ❌
Evaluation Agent (Ours) ✔ ✔ T2I & T2V 400 ✔ (Open-Ended) ✔ ✔

Comparison of the Evaluation Agent Framework with Traditional T2I and T2V Benchmarks. The Evaluation Agent framework supports customized user queries in natural language and works with both T2I and T2V models. Unlike traditional benchmarks, it dynamically updates the evaluation process using multiple tools, providing comprehensive and explainable results with detailed textual analysis.

Experimental Results of the Evaluation Agent

Compare Time Consumption on the VBench

Models VBench (Total Cost) ↓ VBench (Avg. Cost per Dimension) ↓ Evaluation Agent (Ours) ↓
Latte-1 2557 min, 4355 samples 170 min, 290 samples 15 min, 25 samples
ModelScope 1160 min, 4355 samples 77 min, 290 samples 6 min, 23 samples
VideoCrafter-0.9 1459 min, 4355 samples 97 min, 290 samples 8 min, 24 samples
VideoCrafter-2 4261 min, 4355 samples 284 min, 290 samples 23 min, 23 samples

Time Cost Comparison Across Models for VBench Dimensions. This table compares the evaluation time of four different models using the original VBench pipelines versus the Evaluation Agent. The Evaluation Agent significantly reduces the overall evaluation time.


Compare Time Consumption on the T2I-CompBench

Models T2I-Comp (Total Cost) ↓ T2I-Comp (Avg. Cost per Dimension) ↓ Evaluation Agent (Ours) ↓
SD-1.4 563 min, 12000 samples 141 min, 3000 samples 5 min, 26 samples
SD-2.1 782 min, 12000 samples 195 min, 3000 samples 5 min, 26 samples
SDXL 1543 min, 12000 samples 386 min, 3000 samples 8 min, 26 samples
SD-3.0 1410 min, 12000 samples 352 min, 3000 samples 7 min, 25 samples

Time Cost Comparison Across Models for T2I-CompBench Dimensions. This table compares the evaluation costs for assessing four models across T2I-CompBench dimensions using both the original T2I-CompBench pipelines and our Evaluation Agent. The Evaluation Agent achieves a substantial reduction in evaluation time compared to the traditional pipelines.


Performance across VBench Dimensions with Different Base Models

Performance across VBench Dimensions for Different Base Models. This visualization highlights the performance of all backbone models, including GPT-4o and Claude models, providing a comprehensive comparison in each dimension for different backbone models. Hatched portions indicate predictions within the exact range, and solid portions within an error margin of one range.


Statistics of the Open-Ended User Query Dataset

Data Distribution of Open-Ended User Query Dataset. We analyze the constructed open-ended user query dataset from three aspects: General/Specific, Ability, and Specific Domain. The results indicate that our dataset exhibits a relatively balanced distribution across these dimensions. The three graphs give an overview of the distributions and types of our curated open-ended user queries dataset. Left: the distribution of query types, which are categorized as General or Specific. Middle: the distribution of the ability types, which are categorized as Prompt Following, Visual Quality, Creativity, Knowledge and Others. Right: the distribution of the content categories, which are categorized as History and Culture, Film and Entertainment, Science and Education, Fashion, Medical, Game Design, Architecture and Interior Design, Law.

Evaluation Cases for Open-Ended User Queries

(Display of Comprehensive Evaluation Results for Open-Ended User Queries)

BibTeX

If you find our work useful, please consider citing our paper:

@article{zhang2024evaluationagent,
      title = {Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models},
      author = {Zhang, Fan and Tian, Shulin and Huang, Ziqi and Qiao, Yu and Liu, Ziwei},
      journal={arXiv preprint arXiv:2412.09645},
      year = {2024}
  }