San Francisco – November 06, 2026 -- ExpertCrowd, Toptal's AI model training platform, and Y Combinator-backed HUD have released ExpertCrowd-SWE, a software engineering benchmark that evaluates frontier-grade large language models (LLMs) and coding agents.
Toptal engineers wrote the tasks and grading rubrics
Engineers from Toptal's network of domain experts authored the benchmark's tasks and grading rubrics. HUD's infrastructure then hosted the task-run environments and verified the benchmarks through its internal quality assurance pipelines.
HUD and Toptal AI research teams reviewed the methodology and results. The teams include more than a dozen PhDs and researchers with academic or professional experience at Harvard, EPFL, the University of Liège, DeepMind, Qwen and ByteDance.
Partners target gap between lab tests and professional practice
The partners built the benchmark around tasks that reflect professional practice, with grading criteria meant to define what successful execution means. ExpertCrowd contributes expert-led task development, while HUD supplies infrastructure for running complex reinforcement learning (RL) environments.
Toptal founder and CEO Taso Du Val said the collaboration is "setting a higher standard for real-world performance." HUD founder and CEO Jay Ram said ExpertCrowd used HUD's tooling to turn real-world workflows into tasks, reference outputs and evaluation frameworks that test leading AI models.
ExpertCrowd draws on 30,000 experts across 140+ countries
ExpertCrowd offers RLHF, RL data generation, data labeling and model evaluation. Its talent network includes more than 30,000 specialized domain experts in over 140 countries. That pool covers more than 2,000 AI and big data specialists, 12,000 professionals with advanced degrees and 1,000 PhD-level AI specialists.
HUD provides infrastructure for building RL environments and is used by startups, Fortune 500 companies and frontier labs. Its DataVendor marketplace lets AI labs license training and evaluation data directly from the companies that created it.
ExpertCrowd-SWE is the first release of an ongoing partnership
Toptal and HUD have additional initiatives in development, with further announcements to follow. The companies plan to bring more specialized human expertise into the tasks, environments and evaluations that support the next generation of AI.