Open Benchmarks for Cheminformatics

Yesterday's post on cheminformatics benchmarking generated a number of interesting comments, both here and on a similar article posted to Egon Willighagen's blog.

One thing the discussion highlights is the need for a suite of benchmarks specifically aimed at comparing the performance of diverse cheminformatics toolkits under controlled conditions. Toward this end, Egon has set up a GitHub project called cheminfbenchmark, with my own fork of it appearing here.

If you wanted to create a fair and balanced benchmark suite, how would you do it? What tests would you include, how would you run them, how would you select the test data, and how would you report the results?