StructEval
The benchmark, measured
Compare structured output quality across models. Explore scores by release date or model size, with the latest completed result for each model.
Performance explorer
| # | Model | T · Gen | T · Conv | V · Gen | V · Conv | Results | |||
|---|---|---|---|---|---|---|---|---|---|
| Loading results… | |||||||||
All models share one overall ranking; filtering preserves their overall ranks. Green rows indicate Open Source; blue rows indicate Closed Source, based on model weight availability. Scores are percentages (higher is better). T = text-only; V = visually renderable; Gen = generation; Conv = conversion. Parameters are total model parameters, with active MoE parameters shown separately. Undisclosed sizes are omitted from the size chart.
As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce StructEval, a comprehensive benchmark for evaluating LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: (1) generation tasks, producing structured output from natural language prompts, and (2) conversion tasks, translating between structured formats.
Our benchmark encompasses 18 formats and 44 task types, with metrics for format adherence and structural correctness. The leaderboard includes the latest available result for each model, with new models added as evaluations are completed.
StructEval comprises 2,035 examples covering 44 unique structure generation tasks across 18 structured output formats. The dataset is organized into two main subsets:
Evaluates text-only structured outputs
Evaluates visually rendered outputs
Please output JSON code. Task: Summarize metadata about a fictional scientific article. Feature Requirements: 1. Top-level field "title" is a string 2. Field "authors" is a list of exactly two items 3. Each author has "name" and "affiliation" 4. Field "publication.year" is an integer 5. Field "keywords" is a list of strings
titleauthors[0].nameauthors[1].affiliationpublication.yearkeywords[2]Please output HTML code. Task: Design a webpage for a travel itinerary. Feature Requirements: • Centered <h1> with "Trip Summary" • Use a <table> with 3 rows and 2 columns • Apply class "highlight" to second row • Add <button> labeled "Export PDF"
Our evaluation framework employs four core metrics:
Binary metric (0 or 1) indicating whether the generated code can be successfully loaded or rendered without syntax errors
Verifies structural correctness (existence of required keys, relationships between keys, etc.) using dot-path rules. Calculated as the percentage of dot-path rules satisfied by the generated output format.
Evaluates presence of desired keywords using exact string matching. Calculated as the percentage of keywords found in the raw generated output code.
Assesses visual correctness of rendered content through question-answer pairs. Calculated as the percentage of Q&A pairs satisfied by the rendered output.
HTML, React, SVG, LaTeX, Mermaid, etc.
Final Score = (0.2 × Render Score) +
(0.1 × Keyword Matching) +
(0.7 × VQA Score)
JSON, XML, YAML, CSV, TOML
Final Score = (0.2 × Render Score) +
(0.8 × Syntax Score)
@misc{yang2025structeval,
title={StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs},
author={Jialin Yang and Dongfu Jiang and Lipeng He and Sherman Siu and Yuxuan Zhang and Disen Liao and Zhuofeng Li and Huaye Zeng and Yiming Jia and Haozhe Wang and Benjamin Schneider and Chi Ruan and Wentao Ma and Zhiheng Lyu and Yifei Wang and Yi Lu and Quy Duc Do and Ziyan Jiang and Ping Nie and Wenhu Chen},
year={2025},
eprint={2505.20139},
archivePrefix={arXiv},
primaryClass={cs.SE},
doi={10.48550/arXiv.2505.20139}
}