Evaluating financial ai agents using professional rubrics

Evaluating artificial intelligence agents in specialized professional domains requires assessment criteria that reflect actual workplace standards. Traditional evaluation methods typically derive rubrics directly from task prompts or model outputs. However, this approach often overlooks the implicit norms and tacit standards that are only visible in finished professional deliverables.

To address this gap, researchers have introduced FinProBench alongside a reusable pipeline known as Role-Grounded Rubric Construction. This framework operates across four distinct stages: deliverable collection, competency extraction, rubric synthesis, and validation. By analyzing work produced by human practitioners, the resulting rubrics capture nuanced quality levels that transfer across related tasks within the same professional role.

Methodology and Role Classification

The research categorizes 57 occupations by deliverable genre, dividing them into 30 prior-rich conventional roles and 27 prior-sparse role-specialized roles. This classification helps determine when standard model priors are sufficient and when specialized grounding is necessary.

Experimental results indicate that prompt-based evaluation nearly matches grounded rubrics for conventional roles, achieving 89.2 percent compared to 90.7 percent. However, for specialized roles requiring domain-specific expertise, the role-grounded approach substantially outperforms prompt-only methods, scoring 99.1 percent against 78.0 percent.

Benchmark Composition

FinProBench is constructed from 1,723 curated deliverables spanning 57 occupations, 8 financial sub-industries, and 161 deliverable types. The initial evaluation set releases 20 complete tasks covering 20 roles across 7 sub-industries.

When tested using heterogeneous language model judges and role-level rubrics, human-authored deliverables rank first on average with a score of 73.7 out of 100. Evaluated systems scored closely behind, ranging from 69.6 to 70.3, with overlapping confidence intervals indicating complementary strengths.

Furthermore, reusing rubrics at the role level significantly reduces operational overhead. The pipeline decreases estimated per-task construction effort by 6.7 times compared to authoring individual rubrics from scratch for every new task.