AISmellBench: A Reusable Benchmark of AI Code Smells in ML and LLM-Based Systems

Artificial Intelligence (AI) systems are increasingly implemented through Machine Learning (ML) pipelines and Large Language Model (LLM) integrations, where design and implementation choices are scattered across data preprocessing code, model configuration, training scripts, provider APIs, and prompt/output logic. Poorly designed choices may form AI code smells, namely recurring implementation patterns that can undermine reproducibility, robustness, maintainability, and trustworthy behavior. Yet, evaluating AI code smell detection approaches remains difficult due to the lack of reusable benchmarks with validated ground truth. We present AISmellBench, a benchmark for AI code smells built through a reproducible pipeline combining GitHub repository mining, eligibility filtering, tool-assisted identification of potential code smell instances, and manual annotation with adjudicated validation artifacts. AISmellBench brings together 1,221 AI-based Python repositories (888 ML, 333 LLM-based) and contains 1,265 annotated Python files with 3,714 validated AI code smell instances across 31 smell types. The benchmark preserves full provenance from each validated instance to its source analyser and file, integrates prior manually reviewed annotations, supports precision and recall analyses, and exposes tool disagreement points. It ships with construction and evaluation artifacts, a reviewer guide, and a lightweight local dashboard. Demonstration video: https://youtu.be/xepUUGSHaiU. Dataset: https://doi.org/10.5281/zenodo.20085308.