Depth-First: Tag opendata

Forty-Eight Free QSAR Datasets (and More)

Rich Apodaca — Thu, 06 Dec 2007 10:20:00 +0000

Whether you're a medicinal chemist or an informatician, QSAR datasets can be very helpful in understanding complex biological phenomena. These datasets typically consist of a hundred or fewer compounds associated with a specific parameter such as intestinal absorption, volume of distribution, blood-brain barrier penetration, or activity at one or more biological targets. Most of them are published as part of a paper appearing in a peer-reviewed journal.

Unlike chemistry databases, which typically combine a search engine to a dataset of thousands or millions of compounds with a user interface, the QSAR dataset is much more focused and raw. You need to supply your own data viewer, report generator, and query tool.

The Internet hosts a bewildering assortment of QSAR datasets tucked into various nooks and crannies. The problem is finding them. One useful resource is cheminformatics.org, which hosts a page linking to forty-four datasets.

Recently, Shaillay Kumar Dogra, Scientific Editor of QSARWorld, wrote in to let me know about the site's offering of forty-eight free QSAR datasets. Each dataset is linked to the primary literature and is available in four formats, including SD File. In contrast to many datasets, those at QSARWorld are manually curated. QSARWorld is also actively seeking new datasets to convert into machine-readable form; if you find one, write to them to have it added in the collection.

Systematic efforts to collect, curate, and distribute raw data from the primary literature are long overdue. QSARWorld offers an intriguing model for doing so. Although some non-scientific issues, such as intellectual property rights, don't appear to have been addressed yet by QSARWorld, the site's offering of machine-readable raw data offers plenty of food for thought to anyone working with QSAR.

What's your favorite dataset resource?

Image Credit: B.G. Lewandowski

How to Find Chemical Information on the Internet: Why Open Source, Open Access, and Open Data Matter

Rich Apodaca — Fri, 26 Jan 2007 16:21:00 +0000

The Web may be the most effective information-delivery platform ever created. Unfortunately, a variety of barriers, both technical and cultural, restrict the use of the Web for chemistry. In the last few years, three powerful forces for change have emerged: Open Source; Open Access; and Open Data. Most of what's written on these subjects takes a theoretical angle that makes it difficult to visualize real benefits. In this article, I'll discuss these ideas from a much more practical perspective.

A Thought Experiment

Try this simple thought experiment: using only a browser and the free Internet, find all Web pages pages that have anything scientifically-relevant to say about your favorite molecule. How would you do it?

It's Trivial

If you were lucky enough to have chosen a molecule with a trivial name such as 'caffeine', you could just try Google. Google's first result would link you to the Caffeine Wikipedia article. Wikipedia is an evolving phenomenon that, according to some critics, will never have a place in scientific research. It may not be ready now, but reading the meticulously annotated and cross-referenced entry for caffeine should make anyone who would say "never" at least a little nervous. Many of the citations in Wikipedia's caffeine article point to the primary scientific literature through PubMed.

The remainder of Google's top-50 results are general audience items unlikely to interest a scientist keeping his or her nose to the grindstone: companies that sell caffeinated products; a variety of FAQs; self-help medical articles; and of course, this one. We shouldn't be surprised. In the eyes of a massive search engine like Google, chemistry is just one of many niche markets.

Adding terms to our Google search might produce more targeted results. For example, what if we wanted to find a proton NMR spectrum of caffeine? We could type "caffeine proton nmr" into Google. The first result links, indirectly, to an article in the subscription-only Russian Journal of Physical Chemistry A. This does us no good because we have no subscription, limited funds, and no access to the journal at a library. The second link is a direct hit: the proton NMR spectrum of caffeine in water-formic acid. Significantly, the information is contained in a peer-reviewed article (DOI) published by the Japanese Open Access journal, Analytical Sciences. The fact that Analytical Sciences is an Open Access journal has made a world of difference in our search.

Although this might seem like the perfect solution, recall that the goal of the experiment was to locate all scientifically-relevant online content relating to the molecule. The technique we just used is most likely to succeed when we want specific information about molecules with a single trivial name. Even then, many resources may not cite a trivial name at all.

The Real World

Our options are even more limited when it comes to comprehensively text-searching even the simplest molecules lacking a widely-used trivial name. For example, consider the molecule represented by the systematic name '3-phenyl-2-methylpropene.'

If we were using a proprietary system such those offered by Chemical Abstracts Services (CAS), we could simply enter the structure into a client and read off our results. This works because CAS isn't matching text when a query is submitted. Instead, it's matching molecular structures that have previously been encoded by both humans and machines.

The minute we step out of the orderly system created by CAS and into the chaos of the Internet, we confront a thorny problem. In practice, there are only two widely-used methods to convert a molecular structure diagram into a form that can be text-searched, and each has major limitations:

IUPAC Nomenclature This method has the advantage of being Open. It suffers from the complexity of its encoding rules, resulting in a variety of nonstandard implementations. As a result, it's possible to find multiple phrasings of the same IUPAC name, reducing its use as a unique identifier.
CAS Numbers This method replaces a standard encoding system with a central registration authority. The advantages are that the representation of the identifier itself is unambiguous. Conversely, the meaning of a CAS Number can only be known by referring to a registration authority. Unfortunately, the current business model for CAS is based on restricting information flow, rather than promoting it.

Search by IUPAC Name

Let's try searching Google for IUPAC nomenclature. Entering '3-phenyl-2-methylpropene' produces a results page containing three unique entries. One of them links to a database run from the University of Hamburg. I gather the purpose of this database, which I was unaware of before doing this search, is to link to Landolt-Börnstein Online, a collection of numerical data. Interestingly, a search for data on a single compound turned into the discovery of yet another free chemistry database.

The remaining hits from the Google search linked to two pages (pdf, pdf) from ACS journals. The ACS routinely makes the first page of its articles a free download. It's interesting to note that these were the only ACS hits returned by Google.

We've exhausted the possibilities with our chosen IUPAC name and Google. But before moving on to searching by CAS Number, we need to solve a problem. How do we get the CAS number for our compound?

Finding a CAS Number: A PubChem Detour

Concisely summarizing what PubChem does can be difficult because different users will emphasize different aspects of its design. For our purposes, PubChem probably contains a Web page describing the compound we've been researching, and on that page may be a CAS number.

Submitting our molecule to PubChem's search page produces one result. Fortunately, this page lists our compound's CAS number: 3290-53-7.

Search by CAS Number

Submitting our CAS number to Google produces sixteen results. The first two link to the Landolt-Börnstein pages. The next result links to a product listing page for a chemical supplier.

The fourth result links to a far more interesting page - the Organic Syntheses website. Fortunately for us, Organic Syntheses makes its contents freely available online. Following the link takes us to a preparation in which one of the reagents can be substituted with our molecule of interest. Further down in this page, we can see that this molecule has been further cross-referenced. Two procedures are listed, one of which is new to us. Following the link, we find a complete synthetic procedure with full characterization and primary literature citations. Jackpot.

Organic Syntheses permits free public access, but is it Open Access? Many would say not, due to the fact that it retains full copyright to its works and doesn't permit free redistribution. The distinction mainly matters to those seeking to create Open Data repositories based on the contents of periodicals such as Organic Syntheses. To an end user, however, the distinction matters little in the short run.

The remaining results from our Google search are interesting, but mainly consist of chemical supplier catalogs. It should, however, be noted that all of the results returned by a Google search of our CAS number were relevant to our molecule of interest.

InChI

In an effort to overcome the limitations of CAS Registry Numbers and IUPAC systematic nomenclature as unique molecular identifiers, a new system has recently been introduced, the IUPAC International Chemical Identifier (InChI). In contrast to a CAS number, an InChI can be assigned independently of a central authority. Like systematic nomenclature, an InChI can be decoded to a molecular representation. Unlike IUPAC systematic nomenclature, an InChI is generated by a computer algorithm far too complicated for human use. The developers of the InChI software have released their work under an Open Source license, promoting its widespread use by ensuring that services like PubChem will have no difficulties integrating InChI with their software infrastructures. Unlike either CAS Numbers or IUPAC names, InChIs are not yet in widespread use, a point which currently limits their utility.

The PubChem page for our search molecule listed an InChI, as do all PubChem Compound Summary pages. As shown by Peter Murray-Rust and others, it is perfectly feasible to use Google to search for InChIs. Let's try.

Submitting our InChI query to Google gives no results. Leaving off the leading 'InChI=' text, as briefly mentioned here, also results in no hits. This tells us that Google has found no instances of our InChI, and that Google still does not crawl PubChem Compound Summary pages.

Use a Free Database

Numerous free chemistry databases are now running on the Internet. For example, a recent article highlighted thirty-two of them. Would one of them be useful to our search? We need to ask ourselves if we really want to perform more than thirty individual searches. What if we were looking for data on several molecules? Nothing would prevent us from doing this in theory, but in practice, this is too much work.

What we'd really like is to submit a structure query to a single service that will query all of these free databases for us. While such services do exist in name, their breadth is restricted. A more comprehensive solution would be very helpful indeed.

Conclusions

The Web's convenience and ubiquity have prompted many calls for greater Web accessibility to public chemical information. As hinted at by the examples in this article, Open Source, Open Data, and Open Access are three interrelated forces that can make this vision a reality. Open Access journals lower the economic barriers to compiling Open Data sources. Making these Open Data sources useful to scientists in a cost-effective way requires Open Source software. The availability of good Open Source software stimulates the creative combination of Open Data sources. And so on.

A lot needs to be done before this positive feedback loop can replace the status quo. But even with the chaotic, balkanized system that now exists, the benefits are plain to see. With even a small amount of coordination among Open Source software developers, Open Data providers, and scientific publishers, the most amazing things could happen.

Making the Case: Topological Maximum Cross Correlation

Rich Apodaca — Tue, 23 Jan 2007 16:01:00 +0000

... For the Gasteiger partial charges, we took maximum values for positive and negative charges from the “fragmentlike” subset of the ZINC database, consisting of 49 134[sic] molecules, carrying out the calculation with Open Babel 2.0.0.

...

... All structure handling, atom typing, and descriptor calculation was carried out using the open source Java library JOELib.

...

Source code (in Java) to generate the TMACC descriptors is freely available from our Web site under the GNU General Public License at http://comp.chem.nottingham.ac.uk/download/tmacc/index.html.

-James Melvile and Jonathan Hirst, J. Chem. Inf. Model.

Science happens when the experiments and conclusions of your fellow scientists can be freely questioned and independently verified. For example, readers of the cited paper may have questions about the assumptions in the TMACC method, or how to implement it. Questions may be raised about the suitability of the data set used and how others would perform. Readers may even have questions about how to extend TMACC to areas not considered by the authors.

By basing their work on open source software and open data, and by releasing their reference implementation as open source, Melvile and Hirst raise their work to the level of science. The questions that any reasonable scientist would have about the work described in the paper can be answered at any desired level of detail because all source code and all test data are freely available.

Why don't all authors adopt the same approach? Why doesn't a flagship journal such as J. Chem. Inf. Model. require it of all manuscript submissions? As far back as 1984, John Figueras was making this case. Thankfully, Melville and Hirst are taking the message seriously.

From Famine to Feast: A Bumper Crop of Free Chemistry Databases

Rich Apodaca — Fri, 05 Jan 2007 14:53:00 +0000

"Until PubChem came on the scene, the state of chemoinformatics compared to bioinformatics was 20 years behind," says Christopher Lipinski, who formulated the eponymous rule-of-five criteria for drug bioavailability.

-Monya Baker, Nature Reviews Drug Discovery

The number of free chemistry databases on the Web just keeps growing. A recent Depth-First article discussed twelve of them. It turns out that Chembiogrid from Indiana University maintains a list of forty free chemistry databases, most of which are alive and well.

As this trend continues, the need for database standards will become painfully obvious. Not only will interoperable infrastructure technologies and user interface standards need to be developed, but thorny intellectual property issues including access, chain of title, and digital rights will need to be resolved. However, the most immediate need is much more down-to-earth: to involve chemists with the growing number of free alternatives to the chemical information monopoly they've come to rely on.

98219-070105-899330-68

Open Source and Open Data: Why We Should Eat Our Own Dogfood

Rich Apodaca — Wed, 03 Jan 2007 15:57:00 +0000

The National Institutes of Health (NIH) has decided to use OpenEye Scientific Software's cheminformatics toolkits to provide key infrastructure for PubChem, a database of small organic molecules containing chemical structure and biological activities information. PubChem is being developed by the National Center for Biotechnology Information (NCBI) as part of the Molecular Libraries and Imaging component of the NIH Roadmap Initiative. "I am excited to see our software built into PubChem," says Roger Sayle OpenEye's Vice President of Bioinformatics. "It's gratifying that our software will be part of such a useful public resource."

...

Along with the recent decision by the Research Collaboratory for Structural Bioinformatics (RCSB) to use OpenEye's cheminformatics toolkits to curate and depict the Protein Data Bank (PDB) ligand dictionary, the NIH's decision is a clear indication of the speed and robustness of OpenEye's technology for large and diverse sets of chemical structures and data. "It has been beneficial working with the NCBI for their project," says Sayle. "Their data includes enough unusual chemistry to make it a nice validation of the software beyond our regular test sets."

-OpenEye Scientific Software Press Release - October 12, 2004

Why did PubChem, the granddaddy of all open chemistry databases, choose a closed, proprietary toolkit for its software infrastructure? A recent Depth-First article highlighted twelve free chemistry databases. Of those for which information is available, many have chosen the same path as PubChem. Why is this?

A huge opportunity is being wasted every time this happens. We, the authors of Open Source software packages, could be working with the architects of Open Data systems to solve their problems in ways that vendors of closed systems can't. We could be using these Open Data systems as real-world proving grounds for our software, fixing bugs that would have never been detected otherwise, and pushing our systems to the limit. We could be identifying new and exciting uses for our software as the organizations we work with repeatedly ask "what if." Sadly, none of this is happening. A great deal more needs to be done by the Open Source community to persuade the Open Data community to at least try their software. The worst that can happen is that we begin to understand the appeal of closed, proprietary products.

One bright spot is NMRShiftDB, which uses the Open Source Chemistry Development Kit for its infrastructure. This is a fine example of Open Source software powering an Open Data source in chemistry. More examples of this kind of Open Source/Open Data symbiosis would go a long way toward making the case.

Eating your own dogfood is an effective way to break into new markets and develop truly competitive products. After all, if those with closely-aligned goals won't use what you have to offer, who else will?

Hacking Molbank: Creating a Graphical Table of Contents

Rich Apodaca — Mon, 11 Dec 2006 15:00:00 +0000

Molbank is an Open Access collection of single-compound articles on synthetic chemistry. Previous articles on Depth-First have highlighted Molbank's practice of including machine-readable molecular representations of its content, and its very liberal policy on mirroring and robots. In this article, we'll take advantage of both of these features to build something that was left out of Molbank: a graphical table of contents.

The Graphical Table of Contents (GTOC)

The Molbank Graphical Table of Contents (Molbank GTOC) is available online. It consists of a single Web page containing a grid of color 2-D chemical structures representing the contents of Molbank. Each structure is hyperlinked into the Molbank site itself. Clicking on the structure takes you to the complete synthetic procedure and characterization data.

Prerequisites, Downloading, and Running

To run this project, you'll need Ruby CDK. A recent article described the small amount of system configuration required for Ruby CDK on Linux. Another article showed how to install Ruby CDK on Windows.

The complete source code for this project can be downloaded from RubyForge. A subdirectory called demo contains the pre-built final result.

After unpacking the molbank-0.1.0 archive, the demo application can be run:

$ cd molbank-0.0.1
$ ruby test.rb

Problems, We've Got Problems

Several problems were uncovered while building the Molbank GTOC. This is to be expected with any data produced "in the wild" rather than within the safety of an Ivory Tower. Here are the main categories:

Blank Images The entry for M52 is blank. Checking the underlying molfile reveals four instances of bond stereo flags set to "6," a problem common to many of the blank images in the GTOC. According to the Molfile specification, a value of 6 indicates "Down, double bonds," whatever that means. Given that the molecules shown in M52 only have one possible stereo bond, and that the Molfile specification relies on 2-D coordinates to encode double-bond geometry, an encoding inconsistency or incorrect stereo interpretation may be the cause.
Images Containing an "R" Atom Label Entry M53 shows an "R" group at what should be the carbonyl carbon. The underlying molfile contains several less-common entries in the properties block, a common feature of images containing "R" in the GTOC.
Molfile not Found Entry M95 has no associated Molfile because it simply reports errata for other articles. M253-M259, on the other hand, lack molfiles because the articles were "withdrawn before publication." M347 describes a cyclodextrin for which, understandably, no molfile was provided. There are also a couple of cases in which a link to a molfile is provided, but is not available, such as M352.
Broken Molfiles The Molfile for M162 encodes its line endings as two carriage returns and a newline, giving rise to the appearance of blank lines after data lines. This is something the Molfile specification strictly forbids. Apparently, the underlying CDK molfile reader can only handle one carriage return and a newline. Perhaps the extra return was introduced as the file was copied into and out of text editors on various operating systems in preparation for uploading it to Molbank. Another common problem was binary files being used for molfiles, such as with M402. These files don't appear to be compressed with either Zip or GZip and their nature is currently unknown.
Bogus Molfiles For reasons I still can't understand, the Molfile for M407 encodes ethylene. So do several other Molbank molfiles. Other common dummy molfiles include toluene, benzene, and ethane.

After cataloging the problems that exist with the Molbank dataset and the software used to mine it, two interesting questions come into focus:

What can be done to help Molbank fix the most obvious problems in their molfiles and would they accept these improvements?
How can "real" datasets like Molbank help developers build better cheminformatics software? (a graphical Molfile Debugger Utility would come in handy...)

Clearly, the connection between Open Access, Open Source, and Open Data is very strong and runs very deep.

Behind the Scenes

The Ruby Molbank GTOC generator works by connecting to the www.mdpi.net server to get its data in real-time. Internally, the software creates a map of the Molbank website so that the molfile (and URL) for any article can be retrieved on demand. Each readable molfile is used to create a 2-D image using Ruby CDK. As a final step, the index.html page is generated, linking the 2-D images to a specific URL for a Molbank article. This file is produced with eRuby using a previously-described technique.

Conclusions

Building a Graphical Table of Contents for Molbank is not that difficult given the power of Ruby, and Molbank's forward-thinking attitude toward mirroring and robots. In working on this project, several problems were uncovered, both with Molbank's data, and the software used to mine it.

In some ways, the software described here and its output are less interesting than the larger questions they raise:

How do scientific journals best serve not only their readers, but developers who want to provide new ways to use the journal?
How far does copyright extend in scientific publications? For example, are molfiles copyrightable? If so, at what level of detail are they not? If atom coordinates or some other kind of non-essential information is left out, does that change anything?
In what other practical ways could the connection between Open Source, Open Data, and Open Access be explored?

These and many related questions are waiting just around the corner. As Open Access becomes more viable, both technically and commercially, look to Open Source and Open Data to provide the synergies that will unlock its true potential.

Molbank and the Convergence of Open Access, Open Data, and Open Source in Chemistry

Rich Apodaca — Thu, 30 Nov 2006 15:01:00 +0000

Molbank, published by Molecuar Diversity Preservation International, is one of the oldest of a handful of Open Access journals in chemistry. Although its longevity is a remarkable accomplishment in itself, there is much more to Molbank than meets eye. Just below the surface is a feature so revolutionary, yet simple, that chemistry publishers years from now will wonder why they didn't implement it sooner.

A Molbank article consists of a short monograph on a single compound, or possibly two. This may strike some scientists as a strange way to publish results, and it is unusual. On the other hand, this system offers vast potential to capture useful, but "unpublishable" findings that would otherwise be lost. Back when scientists actually read hardcopy journals, such a system would never have been feasible. Today, with hard drive space measured in terabytes, fiber optics cables crisscrossing the planet, Internet connectivity for almost everyone, and servers that can be had for virtually nothing, this system not only looks perfectly feasible, but preferable in many ways to the status quo.

Here's the revolutionary part: each article that Molbank publishes is accompanied by a publicly-available, machine-readable file encoding the structure of the article's subject molecule. That's it. There's nothing tricky or high-tech about it. In fact, the practice is about as low-tech as you could imagine. The file format in which structures are encoded, molfile, dates back at least fifteen years, and nearly every piece of chemistry software - both end-user and developer tools - can handle it. What makes Molbank's practice revolutionary is that not a single chemistry journal, Open Access or subscription-based, currently does this.

Why does the simple inclusion of a publicly-available molfile encoding molecular structures in a paper matter so much? This is where the second two entities of the trinity named in this article's title come into play: Open Source and Open Data. By providing a mechanism for a computer to decipher the chemistry in a paper, Molbank has opened the door to a host of highly-productive integration activities that nobody outside of Chemical Abstract Service has even been able to contemplate, let alone prepare for.

This article is the first in a series aimed at exploring the wide-open space that Molbank has created. Rather than arguing my point with words, I'll actually build working demonstrations of what is now easily within reach. At the same time, I'll document my work on this blog. I'm not sure where all of this will end up, but I do hope to shine some light on a vital, although currently obscure, component of the Open Access debate.

Readily Available, Without Infringements or Restrictions

Rich Apodaca — Wed, 23 Aug 2006 05:47:00 +0000

...If we consider that one of the purposes of publication is to offer testable data, then it would seem that a minimum requirement would be that where computer programs and their results are presented, the author will make source code available on request. ACS could render good service by undertaking the distribution of such requested code. Furthermore, I would make it a condition for publication that such source code be provided. If the scientist is unwilling to disclose his code because he wishes to engage in a commercial venture, then I suggest that he be invited to take out a paid advertisement in the journal and be denied the privilege of publication to promote his product.

-John Figueras J. Chem. Inf. Comput. Sci. 1984, 24, 276

Science moves forward only insofar as observations can be validated and put to use by a third party. Chemical informatics is no different from any other field in this respect. Yet publications of the type Mr. Figueras opposed can still be found in 2006. Why is this?

At issue isn't just software. The ACS has recently spoken out on the necessity of open data sets. As a condition for publication, any data reported in a manuscript must now either appear in Supplementary Material or be “readily available, without infringements or restrictions.” Although this is a positive development, the wait continues for an equivalent statement on the availability of source code.

Open software systems and open data packages are most useful when they can be readily found by others and used together. In an effort to work on this problem, several individuals, including myself, formed The Blue Obelisk group. Through this group and others like it, like-minded researches can begin to reap the benefits of openness enjoyed by other fields.