Depth-First: Tag pubchem

PubChem WTF #1

Rich Apodaca — Sat, 11 Oct 2008 03:35:00 +0000

In preparation for the first beta release of ChemPhoto, the chemical structure imaging application, I've been performing a lot of tests with PubChem SD files. It turns out that having a tool that can be used to quickly browse through tens of thousands of PubChem molecules turns up some very strange beasts, including the one depicted above.

If you're still curious as to what this PubChem record is actually referring to, this tool is quite useful.

Recombining Compressed PubChem SD Files with Open Babel

Rich Apodaca — Wed, 01 Oct 2008 01:25:00 +0000

While testing ChemPhoto, it became necessary to test the chemical structure imaging application with SD Files containing several hundred thousand records. Although it's tempting to meet this need by constructing "dummy" files with the same record or small set of records repeated, tests are always far more illuminating when real data is used.

PubChem is an excellent source of large molecular datasets, and the entire database can be downloaded by FTP. Because of PubChem's massive size, what's downloadable consists of files broken up into groups of about 25,000 in gzipped SD File format (*.sdf.gz). Although this is an excellent resource, it creates a problem: how can you conveniently recombine this set of compressed SD Files into a single SD File?

You might think about writing some "quick" code in your language of choice. Fortunately, Open Babel gets the job done - without any of the coding or debugging.

The following command will create a single SD File from all of the compressed SD Files in a given directory, while also stripping explicit hydrogens and removing all fields except PUBCHEM_COMPOUND_CID.

babel *.sdf.gz pubchem.sdf -d --delete PUBCHEM_COMPOUND_CANONICALIZED,PUBCHEM_CACTVS_COMPLEXITY,PUBCHEM_CACTVS_HBOND_ACCEPTOR,PUBCHEM_CACTVS_HBOND_DONOR,PUBCHEM_CACTVS_ROTATABLE_BOND,PUBCHEM_CACTVS_SUBSKEYS,PUBCHEM_IUPAC_OPENEYE_NAME,PUBCHEM_IUPAC_CAS_NAME,PUBCHEM_IUPAC_NAME,PUBCHEM_IUPAC_SYSTEMATIC_NAME,PUBCHEM_IUPAC_TRADITIONAL_NAME,PUBCHEM_NIST_INCHI,PUBCHEM_EXACT_MASS,PUBCHEM_MOLECULAR_FORMULA,PUBCHEM_MOLECULAR_WEIGHT,PUBCHEM_OPENEYE_CAN_SMILES,PUBCHEM_OPENEYE_ISO_SMILES,PUBCHEM_CACTVS_TPSA,PUBCHEM_MONOISOTOPIC_WEIGHT,PUBCHEM_TOTAL_CHARGE,PUBCHEM_HEAVY_ATOM_COUNT,PUBCHEM_ATOM_DEF_STEREO_COUNT,PUBCHEM_ATOM_UDEF_STEREO_COUNT,PUBCHEM_BOND_DEF_STEREO_COUNT,PUBCHEM_BOND_UDEF_STEREO_COUNT,PUBCHEM_ISOTOPIC_ATOM_COUNT,PUBCHEM_COMPONENT_COUNT,PUBCHEM_CACTVS_TAUTO_COUNT,PUBCHEM_BONDANNOTATIONS,PUBCHEM_CACTVS_XLOGP

865543 molecules converted
7 info messages 15372962 audit log messages

Apparently, there is no way to tell babel to keep just a particular field in an SD File - they need to be removed individually.

Still, not bad for a few seconds on the command line.

Imaging Chemical Structures with ChemPhoto: WYSIWYG Drawing Settings

Rich Apodaca — Thu, 11 Sep 2008 15:44:00 +0000

Depending on the audience and medium, chemical structures can be presented in a variety of styles. Chemical structure imaging applications should make it easy to visually and/or numerically arrive at the best appearance. ChemPhoto makes it easy to get exactly the right look for your structures through what-you-see-is-what-you-get (WYSIWYG) drawing settings.

The screenshots below illustrate the three main categories of drawing settings in ChemPhoto: Atoms; Bonds; and Images. As each setting is manipulated, the entire view is updated in real-time to reflect the changes. A set of changes can be rolled back by pressing the "Cancel" button, making it easy to undo unwanted modifications.

Turquoise Theme with Atoms Tab

Console Theme with Bonds Tab

Blueprint Theme with Images Tab

To try an alpha version of ChemPhoto for yourself, drop me a line.

ChemPhoto Screenshots: Appearance of Structures and Browsing Large Collections

Rich Apodaca — Wed, 10 Sep 2008 14:55:00 +0000

Chemical structure imaging software solves the problem of how to easily create large numbers of readable chemical structures in a variety of formats automatically. ChemPhoto was recently introduced as what appears to be the first chemical structure imaging application. With development of the ChemPhoto user interface now in full-swing, it's possible to show some screenshots.

Below are two screenshots in which around 25,000 structures from PubChem have been loaded.

If you're interested in trying an alpha version ChemPhoto, feel free to drop me a line.

Small Molecule 3D Coordinates From PubChem

Rich Apodaca — Fri, 23 May 2008 10:53:00 +0000

The PubChem team has quietly introduced a new feature - 3D coordinates for many of the small molecules in its compound collection. To my knowledge, these coordinates are only currently available via FTP. From the README:

The data contained here consists of a theoretical 3D description of PubChem Compound records computed using the MMFF94s force field without coulombic terms, including MMFF charges. Each provided theoretical 3D conformer is not a stationary point on the hyper-potential surface (i.e., is not at a minimum energy). Rather, the theoretical 3D description is a low energy conformer selected from a conformer model (a theoretical description of the conformational flexibility of a chemical structure consisting of multiple 3D representations or poses sampled using an RMSD {root mean squared distance} threshold) describing energetically-accessible and (potentially) biologically relevant coformations of a chemical structure.

Not every PubChem Compound record will have a theoretical 3D description. Structures considered too large (containing more than 50 non-hydrogen atoms) or too flexible (containing more than 15 rotatable bonds) are excluded. Furthermore, chemical structures containing elements other than H, C, N, O, F, P, S, Cl, Br, and I are also excluded.

Generation of theoretical 3D descriptions of small molecules is computationally intensive. As such, some PubChem Compound records may be added at a later time.

(A few open source packages for generating 3D conformers are also available.)

Recently, Geoff Hutchison wrote in to suggest that a potentially useful new feature of Chempedia could be the ability to directly obtain 3D coordinates for a molecule of interest.

One very economical way to do that would be to use PubChem's 3D dataset. It would also be trivial to display these coordinates as a resizable Jmol applet, in analogy to Chempedia's recently-added 2D molecule resizing feature.

Of course, there are many other potential uses for the PubChem conformer dataset, especially when applied to Web applications.

Chempedia.net: Mashing Up PubChem and Wikipedia

Rich Apodaca — Fri, 04 Apr 2008 10:06:00 +0000

PubChem and Wikipedia represent two of the largest open repositories of chemical information in the world. And they complement each other very nicely. PubChem contains mainly low-level chemical structure information whereas Wikipedia contains free-text descriptions of chemical compounds in the form of compound monographs.

Both services offer permission and access to copy and reuse their contents. But neither service is, by itself, nearly as useful as it could be.

Why not mash them up?

To explore that question my company, Metamolecular, LLC has launched Chempedia.

To my knowledge, Chempedia represents the first publicly-facing database of compounds to incorporate Wikipedia's collection of organic compound monographs. And it's one of the few cheminformatics services to make use of free-text descriptions generated by individual chemists.

Chempedia has been somewhat selective about the compounds it includes. To date, it has spidered over 2,500 monographs, combining them with over 300,000 of the most interesting compounds from PubChem. Not every Chempedia.net molecule has a monograph, but now there's a tool that can actually make that absence apparent.

Chempedia is both an experiment and a service. It's immediately useful for anyone in the business of making or doing things with organic molecules. It's created several unexpected moments of "Oh, that's actually a useful molecule!" It also will serve as a platform to test some of the ideas discussed in Depth-First over the last year or so on the advantages of the Web for collaboration in chemistry.

Stay tuned for more details about how Chempedia was created and some of its applications in chemistry.

Five Open Tools for 2D Structure Layout (aka Structure Diagram Generation)

Rich Apodaca — Wed, 26 Mar 2008 09:11:00 +0000

Given a molecular representation without 2D coordinates, how would you display a human-readable view?

This problem can arise in many situations, one of the most common of which is the parsing of line notations such as IUPAC nomenclature, SMILES, or InChI.

And then there are the cases when you have 2D coordinates, but they're not very aesthetically pleasing. Maybe the coordinates were created by people either in a hurry or working with low quality editors, or maybe they were generated as distorted 2D projections of 3D coordinates. Whatever the reason, simply having 2D coordinates may not be the same as having good 2D coordinates.

Last year, a Depth-First article discussed the Structure Diagram Generation (SDG) problem and how it can be solved with Open Source software. Given that nearly a year has passed, it seemed appropriate to revisit the topic.

The good news is that there are at least four independent Open Source implementations of SDG algorithms, and one potential open database approach. They are, in no particular order:

MCDL Written in Java, the emphasis of this software appears to be facilitating the use of Modular Chemical Descriptor Language. Unfortunately, no new releases of this intriguing software package have been made in the last year.
Chemistry Development Kit (CDK) This useful package handles about 70-80% of a typical assortment of chemical structures well. The large amount of activity on the CDK project in general makes this a particularly good SDG system to contribute to, especially in the areas of refactoring and handling special cases. See also Christoph Steinbeck's overview of CDK's layout system.
BKChem A 2D structure editor written in Python. Give it an InChI and it will display the structure, courtesy of SDG. The system worked remarkably well with the molecules I tested. BKChem has also been reported to work in batch mode.
RDKit Written in Python and C++, this package is the newest of the bunch. Although I haven't had much luck compiling RDKit, it still looks quite promising. Any chance of switching to make as a build system?
PubChem PubChem? Maybe. With a database of small molecules now numbering well over ten million, there's a good chance that the molecule for which you need to assign coordinates is already in PubChem. And if it's in PubChem, 2D coordinates have already been assigned. Use an InChI as a hash key, and voila - instant SDG without much software. Given the novelty of large, publicly-available databases of small molecules such as PubChem, this approach may have a great deal of untapped potential.

SDG is one of those issues that can stay off the radar for some only to become an instant, nagging problem with no clear way out. The tools cited here offer an excellent place to begin working toward a comprehensive solution.

Create Your Own PubChem Datasets: Exporting Results As SD Files

Rich Apodaca — Tue, 13 Nov 2007 16:43:00 +0000

Recently, I needed to create a subset of the PubChem database in Structure Data File (SD File) format. Although it's far from obvious how to do this, the capability does exist. In this article, I'll give a step-by-step procedure for creating custom datasets in SD File format from arbitrary PubChem structure queries.

Create and Execute the Query

Let's say we want to create a dataset in SD File format containing all N-Boc-protected piperidines registered in PubChem.

From the main PubChem site, choose the Structure Search link. Then click the "Sketch" button.

Next, draw your molecule in the 2D structure editor:

Then click the "Done" button.

Before starting the query (by clicking the "Search" button), be sure to select the "Substructure" option under "Search Type."

Exporting the Results

You should now be looking at a screen containing the first few hits of a 7700+ hitset. But how do we export these results in SD Format?

Next to a field labeled "Display", you'll see a drop-down box containing several different options. Choose the one labeled "PubChem Download."

You'll be redirected to a download page from which you can select output formats, including SDF, or SD File. You can also select a compression type (datasets of even 2000 records can be quite large uncompressed). For this example, we'll select SDF format with GZip compression.

Clicking on the "Download" button takes us to a status page that eventually informs us when our download has been processed. You should then get a "Save File" dialog or something similar. If not, you should see a link to the compressed SD file.

Downloading the results file completes the process.

PubChem for Newbies

Rich Apodaca — Wed, 26 Sep 2007 08:42:00 +0000

PubChem is arguably the most important free repository of information about small molecules on the planet. Although its size is staggering (over 10 million unique compounds), what makes PubChem important is its completely open approach to chemical information. Never before in the history of chemistry has so much information been made available, free to anyone who cares to use it.

Despite PubChem's pioneering approach, many factors make the service difficult to learn and navigate. Most notably, its practical yet bewildering integration into NIH's other far-flung database activities serve as highly effective camouflage for the treasures that lie beneath.

With this in mind, I thought it would be useful to collect all Depth-First articles on PubChem into one place. They have been broken down into categories, although many articles contain elements useful to anyone interested in PubChem.

For Chemists

Five Ways to Use PubChem Right Now Food for thought and grounds for further research.
Simple CAS Number Lookup with PubChem Could this be PubChem's killer app?
How to Find Chemical Information on the Internet: Why Open Source, Open Access, and Open Data Matter PubChem is also part of the process.

For Hackers

Hacking PubChem: Visually Inspect Results for CAS Number and Keyword Searches Radically change the look of PubChem.
Hacking PubChem: Convert CAS Numbers into PubChem CIDs with Ruby Could this be PubChem's killer app?
Hacking PubChem: Learning to Speak PUG I hope you like angle brackets.
Java Interface to PubChem's Power User Gateway Taming those angle brackets with Java.
The Best API May Be No API at All: PubChem and PDB Designing the obvious.
Hacking PubChem: Power User Gateway The preferred method to program PubChem.
Open Source and Open Data: Why We Should Eat Our Own Dogfood Open Source software and open databases should go hand-in-hand, but not in this case.
The Problem With Ferrocene Big trouble ahead.
Hacking PubChem: Direct Access With FTP Cheminformatics' most radical idea in the last 25 years - download all of PubChem onto your hard drive.
Hacking PubChem: Entrez Programming Utilities One of the preferred methods to access PubChem programatically.
Hacking PubChem: Query By Smiles Don't try this at home.
Hacking PubChem with Ruby One of my first attempts to use Ruby for cheminformatics.

For Everyone

PubChem is a Platform Just like the Apple ][ - hardly perfect, but good enough and infinitely hackable.
Thirty-Two Free Chemistry Databases PubChem in its context.
Hacking PubChem: Why the Open Access Fight is Just the Beginning Just getting an open database is not the end of the story.
Hacking PubChem: Free Speech or Free Beer? It depends on what the meaning of the word 'is' is.
Toward and Open Worldwide Chemical Information Network Back in 1965, Walter M. Carlson was way ahead of his time.
Changes My first blog post - inspired, in part, by the launch of PubChem.

If you've got a favorite PubChem resource you'd like to share, please feel free to leave a comment.

Image Service Credit: txt2pic.com

Hacking PubChem: Visually Inspect Results for CAS Number and Keyword Searches

Rich Apodaca — Tue, 25 Sep 2007 10:55:00 +0000

A recent article described how PubChem could be used to quickly search for CAS numbers. Although useful, the approach is limited in that only an array of PubChem CIDs was returned. What would be really useful would be a simple way to create a report with entries hyperlinked into the PubChem site itself to aid in visual inspection. In this tutorial, we'll see how an HTML template and a few extra lines of code can do just that.

The Template

Ruby supports a number of HTML templating mechanisms. In this example, we'll use an ERB template resurrected from the Molbank graphical table of contents tutorial:

<html>
  <head>
    <title>
      <%= "PubChem Search for #{term}" %>
    </title>
  </head>
  <body>
    <h1>
      <%= "Search: #{term}" %>
    </h1>
    <table>
      <tr>
      <% col = 0 %>
      <% cids.each do |cid| %>
        <td>
          <% image = "http://pubchem.ncbi.nlm.nih.gov/image/imgsrv.fcgi?cid=#{cid}" %>
          <% summary = "http://pubchem.ncbi.nlm.nih.gov/summary/summary.cgi?cid=#{cid}" %>
          <a href="<%= summary %>">
            <img src="<%= image %>" border="2"></img>
          </a>
          <center>
            <span style="font-size: 8px">
              <a href="<%= summary %>"><%= "CID-#{cid}" %></a>
            </span>
          </center>
        </td>
        <% col += 1 %>
        <% if col > 5 %>
          <% col = 0 %>
          </tr>
          <tr>
        <% end %>
      <%end %>
      </tr>
    </table>
  </body>
</html>

The above template uses a search term and an array of CIDs to build a table of results. Each cell in the table contains a color 2D image and the CID, both hyperlinked into PubChem itself.

Saving this library to a file called template.rhtml is all we need to do.

The Library

The library is a modification of the one shown in the previous article in this series:

require 'rubygems'
require 'mechanize'
require 'erb'

module PubChemTerms
  def report term
    cids = get_cids term
    erb = ERB.new(IO.read("template.rhtml"))

    File.open "output.html", 'w+' do |file|
      file << erb.result(binding)
    end
  end

  def get_cids term
    agent = WWW::Mechanize.new
    page = agent.get "http://www.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pccompound&retmax=100&term=#{term}"

    (page.parser/"id").collect {|id| id.innerHTML}
  end
end

The method report accepts a search term and uses our template to render a report.

Testing

By saving the above library in a file called pubchem.rb, we can search by keyword via interactive ruby (irb):

$ irb
irb(main):001:0> require 'pubchem'
=> true
irb(main):002:0> include PubChemTerms
=> Object
irb(main):003:0> report 'esomeprazole'
=> #

This produces a file called output.html that can be viewed with any browser:

As in the original version of the library, we can also query by CAS number:

$ irb
irb(main):001:0> require 'pubchem'
=> true
irb(main):002:0> include PubChemTerms
=> Object
irb(main):003:0> report '119141-88-7'
=> #

Conclusions

The simple approach outlined here could be extended in many ways. For example, we could easily retrieve molfiles based on keyword or CAS number search. We could pipe queries together or work with query lists. We could blend in ChemSpider data. We could even build a simple Web application (with Rails) that returned customized reports. Mixing in Ruby CDK or Ruby Open Babel offers still more possibilities.

Increasingly, the most important question in cheminformatics is not "What can we build?", but rather "What should we build?" Success in this new world requires a much deeper understanding of how cheminformatics software is being used by real chemists and where it's not.