Empirically comparing different machine learning algorithms is a core component of the field’s progress, but the rapid pace of advancement presents a barrier to machine learning research. This project addresses this barrier by expanding on a public repository of ML testbed datasets to improve the ease and speed of replicating results.

Machine learning (ML) is one of the fastest growing research areas within computer science, enabling a multitude of new applications that span medicine, the sciences, engineering, business, and more. A cornerstone in the steady progress of ML over the past 30 years has been regular and continuous systematic empirical comparisons of different ML algorithms on testbed ML datasets. However, as the numbers of ML research papers, algorithms, and datasets all continue to grow rapidly, ML researchers are faced with information overload, making it near impossible to keep track of the latest performance advancements in a systematic manner. In turn, this creates significant inefficiencies for researchers, slowing down the pace of advances in new ML research and applications.

This project will address these issues by building upon the success of the existing University of California - Irvine (UCI) Machine Learning Repository, a well-known and widely-used online public repository of ML testbed datasets that ML researchers use to evaluate and track progress in ML algorithm development. This project will involve building the next version of the Repository that will provide rich metadata for ML datasets, linking datasets to research papers and automatically extracting metadata and performance data in leaderboard style. The new Repository will also provide systematic support for reproducible science by allowing users to readily validate empirical ML results on testbed datasets. This project will lead to research advances in two aspects. The first aspect will be the development of new methods and algorithms for information extraction of metadata from the scientific literature. The second aspect will result from improvements in the way ML researchers carry out their experimental work.

Providing tools to support broader, more systematic, and more reproducible evaluations of ML algorithms, will lead to ML advances that are more robust, better calibrated, and more likely to operate well when used in real-world environments. Given that ML algorithms are now being applied to prediction problems in areas as diverse as climate science, judicial decisions, and personalized medicine, the advances in scientific reproducibility from this project, in terms of systematic evaluation of ML algorithms, have potentially far-reaching societal and scientific benefits.