Depth-First: Tag mashup

User-Created Compound Monographs on Chempedia.net: Open Sourcing the Collation and Indexing of Chemical Information

Rich Apodaca — Thu, 17 Apr 2008 17:50:00 +0000

Printed encyclopedias of chemical information like the Merck Index suffer from the problem of becoming obsolete on publication. When new compounds are discovered, or when the information about a compound changes, those changes can take many months or years to appear in print form due to the high cost of publication. It doesn't have to be that way. This article introduces a new feature to the free online chemical encyclopedia Chempedia that lets working scientists update is contents via Wikipedia.

About Chempedia.net

A recent article introduced Chempdia, the free online chemical encyclopedia. This service is built on two of the largest free and open repositories of chemical information in existence: Wikipedia and PubChem. PubChem supplies low-level chemical information such as connection tables, and Wikipedia supplies free-text descriptions of the properties and uses of certain molecules.

Which Molecules?

Currently, Chempedia.net only includes compound monographs for about 1,000 of its over 300,000 molecules. These monographs were located by a manual process in which the titles for all Wikipedia articles were downloaded in alphabetized form; this process clustered titles that represented IUPAC nomenclature due to its use of leading numbers and symbols. IUPAC nomenclature titles were extracted, and then a script was written to extract the chemical information from these titles and combine it with that from PubChem.

This method, although useful for getting a service running, is clearly flawed. The biggest problem is in how to discover new compound monographs.

Why Not Put Users in Control?

Chempedia users themselves are in the best position to know when an existing Wikipedia compound monograph should appear in Chempedia but doesn't, when an existing monograph needs to be updated, or when a new monograph is written and needs to be linked.

How can the process be automated?

As a partial answer to this question, users now have the ability to notify Chempedia of any changes to a Wikipedia compound monograph, and to have those changes immediately reflected in the next viewing of a Chempedia compound monograph.

An Example

As an example, let's take anandamide, a compound I've had some experience with during my time as a medicinal chemist. Although the Chempedia entry for ananandamide exists, there is (or as of today - was) no link to the Wikipedia compound monograph. Let's create one.

At the top of Chempedia's main menu, you'll see a link titled 'Update'. Choosing this link leads to a form that will ask for two pieces of information: (1) the title of the Wikipedia article to which you want Chempedia to link - in this case 'anandamide'; and (2) reCaptcha text to keep robots from making mischief.

Submitting this information is all that's needed to create a new or updated link from Chempedia to Wikipedia. Chempedia handles the rest.

Conclusions

Wikipedia is a vast source of free, high-quality, semi-structured chemical information just waiting to have good chemically-aware interfaces applied to it. Chempedia.net is an attempt to do just that, but it's a bit more as well. Although it may appear that Chempedia is the major beneficiary in this relationship, Wikipedia also benefits. When chemists have a tool that allows them to query and visualize Wikipedia using their native language (the chemical structure) they're in a better position to both use and contribute to Wikipedia itself - something I've started to do.

This positive feedback effect is the real value of exposing Web services. The question is: who in cheminformatics is willing and able to take the risk to discover this simple principle and its benefits?

Hacking ChemSpider: Query by SMILES and InChI with Ruby

Rich Apodaca — Mon, 17 Sep 2007 08:19:00 +0000

Slowly but surely, cheminformatics Web APIs are starting to appear. What's the big deal, you may ask? By exposing Web APIs, service providers enable third parties to develop new applications that "mash up" functionality from two or more sites, or which take the original service in directions its founders never considered.

By way of Antony Williams' blog, I came across the announcement for the ChemSpider Web API. What can this API do for Web developers? To find out, let's write a small Ruby library.

The Library

Our library will accept a SMILES string or InChI identifier and returns a URL pointing to the corresponding ChemSpider compound summary page. Like previous Web API demos, this one uses the powerful Ruby library Mechanize, leading to very concise code:

require 'rubygems'
require 'mechanize'

module ChemSpider
  def url_for_inchi inchi
    agent = WWW::Mechanize.new
    page= agent.get "http://www.chemspider.com/inchi.asmx/InChIToCSID?inchi=#{inchi}"
    csid = (Hpricot(page.body)/"string").innerHTML

    csid == "" ? nil : "http://www.chemspider.com/RecordView.aspx?id=#{csid}"
  end

  def url_for_smiles smiles
    agent = WWW::Mechanize.new
    page= agent.get "http://www.chemspider.com/inchi.asmx/SMILESToInChI?smiles=#{smiles}"
    inchi = (Hpricot(page.body)/"string").innerHTML

    raise "Invalid SMILES: #{smiles}" if inchi == ""

    url_for_inchi inchi
  end
end

The url_for_inchi method directly uses the ChemSpider API to query by InChI. The url_for_smiles method first uses the ChemSpider API to convert a SMILES string to an InChI identifier, and then calls the url_for_inchi method.

Two points are worth noting. First, although for convenience the InChI identifier isn't escaped before being appended to the API URL, strictly speaking it should be. Second, both methods invoke the underlying Mechanize library Hpricot to parse the raw XML returned by ChemSpider.

Testing

Saving the above code to a file called chemspider.rb, we can get the URL to ChemSpider's benzene page from its InChI identifier via interactive Ruby (irb):

$ irb
irb(main):001:0> require 'chemspider'
=> true
irb(main):002:0> include ChemSpider
=> Object
irb(main):003:0> url_for_inchi "InChI=1/C6H6/c1-2-4-6-5-3-1/h1-6H"
=> "http://www.chemspider.com/RecordView.aspx?id=236"

We can work with SMILES strings just as easily as with InChIs:

$ irb
irb(main):001:0> require 'chemspider'
=> true
irb(main):002:0> include ChemSpider
=> Object
irb(main):003:0> url_for_smiles 'c1ccccc1'
=> "http://www.chemspider.com/RecordView.aspx?id=236"

Both the InChI and the SMILES string yield a URL pointing to the same Chemspider page for benzene.

Conclusions

Like most chemical databases, ChemSpider uses a compound summary page as a way of organizing the available resources for a given molecule. With a method in hand for accessing these pages based on arbitrary SMILES or InChIs, we can begin to think of manipulating ChemSpider independently of its current user interface. But that's a story for another time.

From IUPAC Nomenclature to 2-D Structures With OPSIN

Rich Apodaca — Tue, 17 Oct 2006 13:57:00 +0000

A previous article introduced OPSIN, an Open Source Java library for decoding IUPAC chemical nomenclature. In this tutorial, you'll see how OPSIN can, when interfaced with freely-available chemical informatics software, generate 2-D structure diagrams from IUPAC names.

Prerequisites

This tutorial requires Ruby CDK (RCDK), which in turn requires Ruby, Java, and the Ruby Java Bridge. Tutorials detailing the installation of RCDK on both Windows and Linux platforms are available.

In addition, you'll need a copy of the standalone jarfile opsin-big-0.1.0.jar. Future versions of RCDK will integrate the OPSIN jarfile, making this step unnecessary.

Outlining the Problem and a Solution

We'd like to create a simple Ruby class with a method that accepts an IUPAC chemical name as input and produces a PNG image of the corresponding molecule as output. OPSIN accepts IUPAC names as input, but it only produces Chemical Markup Language (CML) as output. The CML output lacks 2-D coordinates, and OPSIN itself has no 2-D rendering capabilities.

We'll use RCDK to augment OPSIN's capabilities. Thanks to CDK's built-in CML support, RCDK can read CML and generate an AtomContainer representation. RCDK also supports the assignment of 2-D coordinates to an AtomContainer via CDK's StructureDiagramGenerator. To produce the PNG image, we'll use the 2-D rendering capability made possible through Structure-CDK, which is a built-in component of RCDK.

A Simple Ruby Library

Create a working directory and copy opsin-big-0.1.0.jar into it. Next, create a file called depictor.rb containing the following Ruby code:

require 'rubygems'
require_gem 'rcdk'
require 'rcdk'

Java::Classpath.add('opsin-big-0.1.0.jar')

require 'util'

# A simple IUPAC->2-D structure convertor.
class Depictor
  @@StringReader = import 'java.io.StringReader'
  @@NameToStructure = import 'uk.ac.cam.ch.wwmm.opsin.NameToStructure'
  @@CMLReader = import 'org.openscience.cdk.io.CMLReader'
  @@ChemFile = import 'org.openscience.cdk.ChemFile'

  def initialize
    @nts = @@NameToStructure.new
    @cml_reader = @@CMLReader.new
  end

  # Writes a <tt>width</tt> by <tt>height</tt> PNG to
  # <tt>filename</tt> for the molecule described by
  # <tt>iupac_name</tt>.
  def depict_png(iupac_name, filename, width, height)
    cml = @nts.parseToCML(iupac_name)

    throw("Can't parse name: #{iupac_name}") unless cml

    molfile = cml_to_molfile(cml)

    RCDK::Util::Image.molfile_to_png(molfile, filename, width, height)
  end

  private

  def cml_to_molfile(cml)
    string_reader = StringReader.new(cml.toXML)

    @cml_reader.setReader(string_reader)

    chem_file = @cml_reader.read(@@ChemFile.new)
    molecule = chem_file.getChemSequence(0).getChemModel(0).getSetOfMolecules.getMolecule(0)

    molecule = RCDK::Util::XY.coordinate_molecule(molecule)

    RCDK::Util::Lang.get_molfile(molecule)
  end
end

Testing, Testing

A short test will demonstrate the capabilities of the Depictor library. Add the following to a file called test.rb in your working directory (or enter it interactively with irb):

require 'depictor'

depictor = Depictor.new
name = '3,3-dimethyl-7-oxo-6-[(2-phenylacetyl)amino]-4-thia-1-azabicyclo[3.2.0]heptane-2-carboxylic acid' #Penicillin G

depictor.depict_png(name, 'out.png', 300, 300)

Running this test produces a 300x300 PNG image of Penicillin G, named out.png, in your working directory:

As you can see, this simple library and test code has:

correctly parsed the rather complex IUPAC name (3,3-dimethyl-7-oxo-6-[(2-phenylacetyl)amino]-4-thia-1-azabicyclo[3.2.0]heptane-2- carboxylic acid) to a valid CML representation
converted this representation to a CDK AtomContainer
assigned 2-D coordinates
rendered a PNG image in color

Notice how the thiaazabicyclo[3.2.0] system, complete with properly-placed substitutents, was flawlessly identified and parsed.

If you entered the above test code interactively via IRB, you may have noticed a multi-second delay in instantiating Depictor. This latency results from a sluggish NameToStructure constructor in OPSIN. A similar delay also occurs in OPSIN's pure-Java unit tests. Once Depictor is instantiated, however, image generation occurs relatively quickly.

The unususal orientation of the beta-lactam carbonyl group is determined by CDK's StructureDiagramGenerator. The source of this behavior will be explored in a future article.

More Examples

To illustrate some of the capabilities of the OPSIN-RCDK combination, a few more examples are provided below.

One of OPSIN's more surprising features is how well it handles heterocycles. For example, the IUPAC name for caffeine (1,3,7-trimethylpurine-2,6-dione) is translated to:

As another example, consider the tetrazole (1-[2-hydroxy-3-propyl-4-[3-(2H-tetrazol-5-yl)propoxy]phenyl]ethanone):

Highly substituted benzene rings and carboxylic acids are also translated accurately, as in 3-acetamido-5-(acetyl-methyl-amino)-2,4,6-triiodo-benzoic acid (Metrizoate):

How about a hairy-looking macrocycle name with multiple levels of morpheme nesting (3,6-diamino-N-[[15-amino-11-(2-amino-3,4,5,6-tetrahydropyrimidin-4-yl)-8- [(carbamoylamino)methylidene]-2-(hydroxymethyl)-3,6,9,12,16-pentaoxo- 1,4,7,10,13-pentazacyclohexadec-5-yl]methyl]hexanamide)? Not a problem:

Limitations

In my tests of the OPSIN library, one structure appeared to be incorrectly parsed - N-(5-chloro-2-methyl-phenyl)-2-methoxy-N-(2-oxooxazolidin-3-yl)acetamide:

There are actually two problems with the output. First, an oxygen atom and a methyl group are overlapping near the top of the diargram. This cosmetic issue is related to CDK's StructureDiagramGenerator. Second, the oxazolidine nitrogen atom is misplaced by OPSIN. The correct 2-D image of this molecule, obtained from PubChem, is shown below:

Conclusions

It's not common to find an early-development Open Source project with the sophistication of OPSIN. The smooth handling of nested morphemes, aromatic heterocycles, macrocycles, and a good fraction of what I threw at it leads me to belive that a well-designed and extensible nomenclature parsing engine lies at OPSIN's core. More on that later, though.

What could you do with a powerful Open Source IUPAC nomenclature parser? The answer to that one question could fill a three-volume series. Suffice it to say that OPSIN, in combination with other Open Source software, offers virtually limitless potential for indexing, collecting, repackaging, reprocessing, and mashing up vast amounts of chemical information. Because of its Open Source license, OPSIN can be extended and otherwise modified to fit your particular needs. Future articles will highlight some of the possibilities.

Mashups for Fun and Profit

Rich Apodaca — Sat, 23 Sep 2006 16:27:00 +0000

ProgrammableWeb offers one-stop shopping for all things mashup-related. If you've ever wanted to try your hand at Web programming, this site makes an excellent first stop. Be sure to check out the listing of over 1,000 mashup sites indexed by category and API.

The move toward open, Web-based chemical information resources is fully underway. The genie has been let out of the bottle, and there's no putting him back. This is bad news for large, established chemical information players. Their business models based on restricting information flow will be irreversibly disrupted. It's good news for tens of thousands of researchers who will be able to exploit chemical information in ways unimaginable today. Leading the way will be mashups that creatively tie diverse Web resources together, and dynamic programming languages like Ruby that make doing so easy.

Are you ready for the future?

Hacking PubChem: Entrez Programming Utilities

Rich Apodaca — Sat, 23 Sep 2006 01:22:00 +0000

A recent article poses the question of how to balance the rights of owners of open chemical information resources against those of their users, while promoting an innovative environment for third-party developers. Although PubChem was the focus, the discussion could apply to any other chemical information resource. A reasonable approach would be to provide two separate entry points: one for Web browsers and another for various types of semi-autonomous software used in hacking and mashups.

Peter Corbett writes to point out that the Entrez Programming Utilities can be used to query PubChem and other databases under the NIH/NCI/NCBI umbrella. A separate developer server processes requests, and the terms of its use are fairly well stated. Future articles will explore the possibility of building some simple Ruby APIs for this developer PubChem entry point.

Hacking PubChem: Why The Open Access Fight is Just the Beginning

Rich Apodaca — Fri, 22 Sep 2006 13:58:00 +0000

Like no other medium, the Internet tests our basic beliefs about the rights of resource owners and resource users. As the Internet increasingly becomes home to scientific publication mechanisms that have no counterpart in the physical world, a larger question looms: what separates fair use of these services from abuse?

Depth-First hosts a series of articles, with possibly many more to follow, on programatically accessing open chemical information databases:

The availability of open chemical information resources like PubChem and NMRShiftDB is a very recent phenomenon, and desperately overdue. One premise of this blog is that chemical informatics is at the start of a renaissance; the chemical information revolution that started in the 1950's is now set to continue after a long period of stagnation. Large, open data sources, and open software that mines it, will fuel this transformation, just as they have in bioinformatics.

The interaction of non-browser software with public databases, although rich in potential payoffs, can also lead to a great deal of damage. PubChem contains millions of structure-searchable compounds. Setting the wrong kinds of programs loose on this site could cause service interruptions ranging from the annoying to the severe.

There is no standard mechanism for website owners to spell out acceptable use policies to non-browser software. The closest thing we have to a standard is the Robots Exclusion Protocol. This protocol defines acceptable behaviors for a robot, which according to one definition consist of: "... a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced." Other definitions are in use. The one thing these definitions seem to have in common is the concept of scale: the more comprehensive and indiscriminate the program is in its interactions with a website, the more like a robot, and less like a browser, it becomes.

Site owners specify their robots policy in a file called robots.txt hosted on their servers. The PubChem robots.txt file currently includes the following policies:

User-agent: *
Disallow: /substance/PcsSrv.cgi
Disallow: /summary/summary.cgi
Disallow: /assay/assay.cgi
Disallow: /image/imgsrv.fcgi
Disallow: /image/smi2gif.fcgi
Disallow: /image/smi2gif.cgi
Disallow: /image/structurefly.cgi
Disallow: /search/NbrQsrv.cgi
Disallow: /search/PreQSrv.cgi

Here, User-agent refers to the name of the robot, which is set as a wildcard, meaning any robot. The Disallow lines refer to resources off-limits to robots.

One of these disallowed resources, /search/PreQSrv.cgi is explicitly used in the PubChem SMILES query article.

Is a person who runs software of the type I describe in these articles violating PubChem's use policy? The best answer I can give is, "it depends." I think it would be hard for reasonable people to suggest that using the software as described in the tutorials, with their deliberately limited scope, for research purposes, and with no intent to do damage, represents abuse.

On the other hand, I can see how reasonable people could argue that a website operating as a comprehensive front-end to PubChem using the techniques described in these articles could be considered abuse. I know I might consider it abuse if I ran PubChem, depending on why I was running the service.

If I wanted to stimulate innovation in the area of open database mining, I might actually encourage front ends and similar third-party PubChem services. I might set aside servers specifically dedicated to this kind of activity. I might even develop an Open Source PubChem Web-API to help developers get started. Unfortunately, NIH's intentions are not exactly clear on this point.

Looking at the NCBI's Copyright and Disclaimers page, the only document that to my knowledge states any kind of use policy, is not especially illuminating:

Conditions of Use

This site is maintained by the U.S. Government and is protected by various provisions of Title 18 of the U.S. Code. Violations of Title 18 are subject to criminal prosecution in a federal court. For site security purposes, as well as to ensure that this service remains available to all users, we use software programs to monitor traffic and to identify unauthorized attempts to upload or change information or otherwise cause damage. In the event of authorized law enforcement investigations and pursuant to any required legal process, information from these sources may be used to help identify an individual.

We are left with the critical, but unanswered question: "What represents an unauthorized use of PubChem?"

The document cited above also raises the truly bizarre possibility of PubChem not actually being capable of granting rights to redistribute what is contained on its servers:

This site also contains resources such as PubMed Central, Bookshelf, OMIM, and PubChem which incorporate material contributed or licensed by individuals, companies, or organizations that may be protected by U.S. and foreign copyright laws. ...

But this is a subject for another day.

Getting back to accessing PubChem data, one very far-sighted thing the NIH has done is to make the entire dataset freely downloadable in three different file formats. Rather than mine the PubChem website itself, you could download the data to your machine, letting the software you write access it locally. The sheer size of this dataset creates problems of its own. Future articles will describe some approaches to solving them.

Regardless of your views on the use and abuse of chemical information resources like PubChem, it's clear that getting open resources on the Web is only the first in a long series of controversial steps that will ultimately transform both the practice and culture of research.

Hacking PubChem with Ruby

Rich Apodaca — Wed, 30 Aug 2006 02:29:00 +0000

PubChem is an increasingly popular, free-access, online molecular database operated by the National Institutes of Health. Web services are a hot topic, with sites such as Flickr, Google, and eBay offering developers the tools to build rich content through "mashups" of several web APIs. Although there is no formal PubChem API, it's possible to roll your own. As a demonstration, this article will show how structural information can be retrieved from PubChem using some simple Ruby code. The inspiration for this article came from the PubChem module that is part of Chemruby.

The only thing you'll need for this tutorial is Ruby, preferably version 1.8.2 or higher. Create a directory called pubchem and make it your working directory. Then create a file called pubchem.rb containing the following code:

require 'net/http'

# A very simple PubChem Web API.
class PubChem

  # Returns a molfile (as a String) for the molecule with PubChem
  # CID matching compound_id.
  def self.get_molfile(compound_id)
    molfile = nil
    path = '/summary/summary.cgi?cid=' + compound_id + '&disopt=DisplaySDF'

    Net::HTTP.start('pubchem.ncbi.nlm.nih.gov') do |http|
      response = http.get(path)
      molfile = response.body
    end

    molfile
  end

  # Writes a PNG image, for the molecule with PubChem
  # CID matching compound_id, to the file specified by filename.
  def self.write_image(compound_id, filename)
    path = '/image/imgsrv.fcgi?t=l&cid=' + compound_id

    Net::HTTP.start('pubchem.ncbi.nlm.nih.gov') do |http|
      response = http.get(path)
      image = response.body

      File.open(filename, "w") do |file|
        file << image
      end
    end
  end
end

PubChem references each of its compounds by a unique integer identifier, the PubChem CID. This is very handy because retrieving PubChem resources is as simple as encoding a URL containing the CID of interest. The class above illustrates how this system can be used to get a molfile and write a PNG image using just a few lines of Ruby. Using the PubChem class is simplicity itself. To get the molfile for Levonorgestrel (Plan B), which has the CID 13109:

require 'pubchem'

molfile = PubChem::get_molfile('13109') #=> returns the molfile for Levonorgestrel as a String

To write the 2-D structure diagram of Levonorgestrel as a PNG:

require 'pubchem'

PubChem::write_png('13109', 'image.png') #=> writes a PNG image of Levonorgestrel

This code saves the image below to your working directory as image.png. The above two code fragments can either be saved as a file and executed by the Ruby interpreter:

$ ruby filename.rb

or it they be entered interactively in your console with irb:

$ irb
irb(main):001:0>

As you can see, there's not much to building a PubChem API in Ruby. The same principles discussed here should apply in any programming language. Future articles in this series will show how to build more complex PubChem APIs and integrate them with other software packages and web services.