Addressing the limitations of contemporary artificial intelligence benchmarking

As artificial intelligence systems become increasingly integrated into daily operations across various sectors, traditional benchmarking methods are struggling to keep pace. Legacy evaluation frameworks, many of which were designed years ago, often rely on publicly available academic tests. This creates an environment where developers can inadvertently or intentionally optimize their models to pass specific exams without necessarily improving their broad functional capabilities.

The shift toward task-specific evaluation

Founded in 2024, the startup Vals aims to address these shortcomings by moving away from abstract knowledge tests. Instead of measuring whether a model can pass a generalized examination, the company evaluates artificial intelligence systems on their ability to execute complex, multi-step workflows tailored to specific industries such as law, finance, and software engineering.

To prevent developers from training models directly on test materials, these evaluation sets are kept private. This approach is designed to simulate authentic workplace environments, testing not only whether an artificial intelligence model can produce human-quality output, but also identifying potential negative outcomes or safety risks.

Expanding into sensitive domains

As the scope of artificial intelligence applications widens, evaluation metrics are expanding to cover more specialized and sensitive areas. Current testing domains now include:

• Cybersecurity and biosecurity protocols
• Mental health application assessments
• Recursive self-improvement mechanisms
• Legal frameworks such as the Geneva Convention