DiversityLens
A demographic dataset auditing framework and pipeline that exposed hidden distributional structures across more than 1.6 million image and video frames.
The Challenge
Modern computer vision algorithms are frequently benchmarked on standard public datasets under the assumption that these collections fairly represent real-world diversity. In practice, training and evaluation sets harbor profound, unquantified demographic skews in age, gender presentation, ethnicity, and lighting condition.
Manual inspection cannot scale to millions of images and multi-gigabyte video sequences. DiversityLens was created to turn demographic auditing into an automated, systematic, and reproducible engineering discipline.
Pipeline Architecture
Delivered as an end-to-end Python package, DiversityLens manages the complete life cycle of large-scale dataset verification:
- Discovery & Intelligent Sampling: High-throughput recursive file discovery and deterministic keyframe extraction for large video corpuses, preventing redundant computation while capturing representative temporal variations.
- Multi-Scale Detection & Alignment: Robust face detection handling challenging real-world variations including extreme poses, partial occlusions, and severe lighting imbalances.
- Cross-Backend Attribute Inference: Evaluates attributes across multiple distinct inference engines and measures predictive divergence against curated ground-truth to avoid measurement bias.
- Structured Aggregation & Reporting: Generates multidimensional demographic profiles, statistical disparity metrics, and automated interactive visual dashboards for instant exploration.
Research Software Rigor
DiversityLens was built not as a disposable script, but as production-grade research software designed for ongoing audit campaigns:
- Comprehensive CLI tooling with structured logging and runtime profiling.
- Extensive automated unit and integration tests via
pytest. - CI/CD automation with GitHub Actions ensuring cross-platform stability.
- Documented APIs and reproducible containerized environments for researcher adoption.
Publication & Thesis
Conducted as Master’s thesis research in collaboration with the DFKI Robotics Innovation Center (German Research Center for Artificial Intelligence) and Fachhochschule Dortmund.
V. Ates et al., “DiversityLens: A Large-Scale Demographic Auditing Framework for Visual AI Systems,” to appear in Proceedings of the 10th International Conference on Computer-Human Interaction Research and Applications (CHIRA 2026).