After a brief period of quiet, things have started moving with incorporation of the Japex benchmarking framework. If you're not familiar with Japex, it does for benchmarking what Ant does for building. One of the things that makes Japex especially useful is its integration of HTML reports in which graphics are generated by JFreeChart.
Two cheminformatics libraries will be compared first: CDK and MX. If there's another library you'd like to see in the comparison, feel free to ask. Java libraries are best, but there are always ways around that. The only requirement is that it needs to be freely distributable.
See also: Open Benchmarks for Cheminformatics.