Depth-First: Tag sdfgz

Recombining Compressed PubChem SD Files with Open Babel

Rich Apodaca — Wed, 01 Oct 2008 01:25:00 +0000

While testing ChemPhoto, it became necessary to test the chemical structure imaging application with SD Files containing several hundred thousand records. Although it's tempting to meet this need by constructing "dummy" files with the same record or small set of records repeated, tests are always far more illuminating when real data is used.

PubChem is an excellent source of large molecular datasets, and the entire database can be downloaded by FTP. Because of PubChem's massive size, what's downloadable consists of files broken up into groups of about 25,000 in gzipped SD File format (*.sdf.gz). Although this is an excellent resource, it creates a problem: how can you conveniently recombine this set of compressed SD Files into a single SD File?

You might think about writing some "quick" code in your language of choice. Fortunately, Open Babel gets the job done - without any of the coding or debugging.

The following command will create a single SD File from all of the compressed SD Files in a given directory, while also stripping explicit hydrogens and removing all fields except PUBCHEM_COMPOUND_CID.

babel *.sdf.gz pubchem.sdf -d --delete PUBCHEM_COMPOUND_CANONICALIZED,PUBCHEM_CACTVS_COMPLEXITY,PUBCHEM_CACTVS_HBOND_ACCEPTOR,PUBCHEM_CACTVS_HBOND_DONOR,PUBCHEM_CACTVS_ROTATABLE_BOND,PUBCHEM_CACTVS_SUBSKEYS,PUBCHEM_IUPAC_OPENEYE_NAME,PUBCHEM_IUPAC_CAS_NAME,PUBCHEM_IUPAC_NAME,PUBCHEM_IUPAC_SYSTEMATIC_NAME,PUBCHEM_IUPAC_TRADITIONAL_NAME,PUBCHEM_NIST_INCHI,PUBCHEM_EXACT_MASS,PUBCHEM_MOLECULAR_FORMULA,PUBCHEM_MOLECULAR_WEIGHT,PUBCHEM_OPENEYE_CAN_SMILES,PUBCHEM_OPENEYE_ISO_SMILES,PUBCHEM_CACTVS_TPSA,PUBCHEM_MONOISOTOPIC_WEIGHT,PUBCHEM_TOTAL_CHARGE,PUBCHEM_HEAVY_ATOM_COUNT,PUBCHEM_ATOM_DEF_STEREO_COUNT,PUBCHEM_ATOM_UDEF_STEREO_COUNT,PUBCHEM_BOND_DEF_STEREO_COUNT,PUBCHEM_BOND_UDEF_STEREO_COUNT,PUBCHEM_ISOTOPIC_ATOM_COUNT,PUBCHEM_COMPONENT_COUNT,PUBCHEM_CACTVS_TAUTO_COUNT,PUBCHEM_BONDANNOTATIONS,PUBCHEM_CACTVS_XLOGP

865543 molecules converted
7 info messages 15372962 audit log messages

Apparently, there is no way to tell babel to keep just a particular field in an SD File - they need to be removed individually.

Still, not bad for a few seconds on the command line.

Cheminformatics for the Web: Convert SD Files to HTML with Ruby CDK

Rich Apodaca — Mon, 13 Nov 2006 15:23:00 +0000

The Structure Data File (SDF) format is the de facto standard for cheminformatics data exchange. One of the problems that arises when working with SD Files, especially large ones like those distributed by PubChem, is "seeing" the structures they contain. Although commercial software packages are available for doing so, they are generally closed, unreasonably expensive, or overly complex. This article describes a simple solution to the SDF visualization problem that uses Open Source tools controlled from the elegant and agile Ruby programming language.

Cut to the Chase

This page shows the output produced by the software. You'll see a neatly arranged grid of colorful 2-D chemical structures in a Web page that was generated directly from a PubChem SDFGZ file. Each structure has a number below it, the PubChem Compound ID (CID). Both the structure and CID are hyperlinked to the Compound Summary page on PubChem. A partial screenshot is provided below.

Prerequisites

For this tutorial, you'll need Ruby CDK (RCDK). A recent article described the small amount of system configuration required for RCDK on Linux. Another article showed how to install RCDK on Windows.

Download the Software

The software described in this article can be downloaded here. Inflate this file and make it your working directory. You should see a 14 MB SDFGZ file, a RHTML template, and three Ruby files.

Ripping PubChem SD Files

The software is designed to work with PubChem SDFGZ files. The SDFGZ format simply results from the application of the gzip compression algorithm to an ordinary SD file.

Ripping the example SDFGZ file is just a matter of running test.rb:

$ ruby test.rb

You'll see some output indicating that various CIDs are being processed. On completion, the software has created a directory called rip containing a HTML file and an images directory.

The Little Engine That Could: CDK's StructureDiagramGenerator

If you've ever worked with PubChem's SD Files, you'll no doubt have noticed that the molfile section encodes all hydrogen atoms, which is not general practice. Rendering these hydrogens results in a very cluttered image.

To solve this problem, the software creates its graphics from the PUBCHEM_OPENEYE_CAN_SMILES field encoded by the SDFGZ file. This SMILES string is converted into a molecular representation and coordinates are assigned by CDK's StructureDiagramGenerator.

When an image can't be rendered in this way, it is left out. This was done for CIDs 18, 115, 147, 148, 222, and 223, for example. There are three common themes in these missing structures: metals, phosphorous, and molecules with a single heavy atom. The problem may, in fact, lie in the underlying Structure-CDK software, rather than with CDK. Stay tuned for more on this.

PubChem for Debugging

In developing this SD File Ripper program, I realized that it could be used as a powerful debugging tool. Notice how the missing structures (and their SMILES strings) can easily be examined via PubChem by clicking the empty cell. The alternative would have been for the program to spit out a list of SMILES that didn't process properly and to then try to construct a mental image of what this string represents. With PubChem, we do away with this tedium altogether.

I doubt the creators of PubChem envisioned this application of their work. Surely it's but one of many still to be discovered.

Another Cool Thing About Ruby: eRuby Templates

Our SDF Ripper program creates HTML output, something for which Ruby is well-suited through its eRuby ERB library. Among other uses, ERB enables Ruby code to be embedded within HTML. This inside-out scripting capability resembles that of other templating languages such as PHP, ASP, and JSP (ERB is used extensively by the Ruby on Rails web application framework). The file template.rhtml contains the rippers's ERB template. The separation of program logic from presentation makes it very simple to customize the appearance of the output.

Room to Grow

Our SDF Ripper only works with SDFGZ files from PubChem. The program is short enough that it should be simple to adapt it for your specific needs. It would not be much work at all, for example, to create an HTML table containing all fields encoded by the SDFGZ file. Similarly, adding support for non-compressed SD files is straightforward. If JavaScript is your medium, the possibilities become even more interesting. How about a pop-up menu showing an enlarged structure and data summary, a la Netflix when the user mouses over an image?

Paging is a technique that divides large Web pages into smaller pages linked to one another. For example, Google's search results are divided into groups of ten by default. Adding paging support to the software described here would also not be difficult, and would enable the convenient browsing of much larger datasets.

Other Software That Does This

I am aware of no product, commercial or otherwise, that performs the SDF to HTML conversion in the way shown here. SciTegic does offer an HTML table component as part of its Pipeline Pilot framework, but as far as I know, no standalone version is available.

Rajarshi Guha, among his many other interesting projects, has written a Java SDF to PDF convertor that uses CDK.

Conclusions

This article has demonstrated how the combination of RCDK and Ruby makes short work of converting the contents of an SD file into a Web-ready format. As usual, we've only scratched the surface of what's easily within reach. Watch for future articles to build on the concepts outlined here.

Hacking PubChem: Direct Access with FTP

Rich Apodaca — Fri, 29 Sep 2006 01:59:00 +0000

A previous article in the Hacking PubChem series pointed out that the entire PubChem database can be downloaded via FTP. This article shows how simple tools written in Ruby can be used to efficiently process the massive amount of data on PubChem's FTP-server.

Prerequisites

The only software you'll need for this tutorial is Ruby.

Organization of PubChem's FTP-Server

PubChem is a big database. To deal with its size, the FTP-server spreads its contents over about 950 files. Each file contains a contiguous range of Compound Identification Numbers (CIDs), which appears to be set at 10,000 [Now 25,000, see below]. In some of the files I've examined, the actual number of compounds in a given block was less than 10,000. The root directory containing the files can be accessed here.

Compression Saves the Day

For storage and transmission efficiency, PubChem's SDF files are compressed using the GZip algorithm, giving files that typically range in size from five to seven megabytes. Compression ratios for the files I've examined are about 10:1. I'm calling these files "SDFGZ" files, and they have the extension *.sdf.gz.

A back of the envelope calculation, based on 950 files with an average size of 6 MB and a compression ratio of 10:1, gives an approximate storage requirement of 57 GB for the uncompressed PubChem database. Although storing this much data is feasible with today's hardware, there are many better uses for storage space. This is especially true if only a few fields of the PubChem database are of interest.

Setting Up

You'll need to download some SDFGZ data. This tutorial uses the file containing CIDs 9540001-9550000. [Note: PubChem recently increased the number of compounds in each sdfgz file to 25,000. This means that the link to the file no longer works. Instead, choose a file from here.] Put this file in your working directory.

A Short Library

Create a file called sdfgz.rb containing the following code:

require 'zlib'

# A simple splitter for *.sdf.gz files available
# from PubChem's FTP-server.
class SDFGZSplitter
  @@stop = "$$$$\n"
  @@blank = ""

  # Configures this SDFGZSplitter using the <tt>IO</tt>
  # object <tt>io</tt>.
  def initialize(io)
    @gzip = Zlib::GzipReader.new(io)
  end

  # Yield a sequence of SDFile records.
  def each_record
    record = get_record

    while record != @@blank
      yield record
      record = get_record
    end
  end

  # Gets the next record, or an empty string if
  # none is available.
  def get_record
    line = read_line
    record = [line]

    while !(@@stop.eql?(line) || nil == line)
      line = read_line
      record << line
    end

    record.join
  end

  private

  # Reads the next line in the SDFGZ file.
  def read_line
    begin
      line = @gzip.readline
    rescue EOFError
      return nil
    end

    line
  end
end

# Utility class for getting data out of a SDFile record.
class Extractor
  # Gets the data from <tt>record</tt> associated with
  # <tt>key</tt>.
  def self.extract_data(record, key)
    record.match(/> <#{key}>\n(.+)\n/)
    $1
  end

  # Gets the molfile for <tt>record</tt>.
  def self.extract_molfile(record)
    record.match(/M  END$/).pre_match + "M  END\n"
  end
end

The SDFGZSplitter class uses Ruby's built-in GZip library to read SDFGZ files without inflating them. The method each_record is a Ruby iterator, one of the strangely cool things that makes Ruby the language it is. The iterator's job is to allow retrieval of each SDFGZ record individually, until all records have been retrieved.

Using the Library

As a test for the sdfgz library, lets scrape all PubChem CIDs and InChI identifiers from an SDFGZ file, and place the result into a new CSV file. Create the following code, either in a file to be run by ruby or in a terminal session using irb:

require 'sdfgz'

file = File.new('Compound_09540001_09550000.sdf.gz')
splitter = SDFGZSplitter.new(file)

puts "parsing..."

File.open('dictionary.csv', 'w+') do |file|
  splitter.each_record do |record|
    cid = Extractor.extract_data(record, 'PUBCHEM_COMPOUND_CID')
    inchi = Extractor.extract_data(record, 'PUBCHEM_NIST_INCHI')

    file << "#{cid},\"#{inchi}\"\n"
  end 
end

Running this test creates a (rather large) file called dictionary.csv in your working directory. Its contents consist of the following truncated output:

9540001,"InChI=1/C20H22N2O4/c1-13-7-5-10-16(14(13)2)22-20(26)15-8-3-4-9-17(15)21-18(23)11-6-12-19(24)25/h3-5,7-10H,6,11-12H2,1-2H3,(H,21,23)(H,22,26)(H,24,25)/p-1/fC20H21N2O4/h21-22H/q-1"
9540002,"InChI=1/C20H22N2O4/c1-13-7-5-10-16(14(13)2)22-20(26)15-8-3-4-9-17(15)21-18(23)11-6-12-19(24)25/h3-5,7-10H,6,11-12H2,1-2H3,(H,21,23)(H,22,26)(H,24,25)/f/h21-22,24H"
9540003,"InChI=1/C19H20N2O5/c1-26-16-8-3-7-15(12-16)21-19(25)13-5-2-6-14(11-13)20-17(22)9-4-10-18(23)24/h2-3,5-8,11-12H,4,9-10H2,1H3,(H,20,22)(H,21,25)(H,23,24)/p-1/fC19H19N2O5/h20-21H/q-1"
9540004,"InChI=1/C19H20N2O5/c1-26-16-8-3-7-15(12-16)21-19(25)13-5-2-6-14(11-13)20-17(22)9-4-10-18(23)24/h2-3,5-8,11-12H,4,9-10H2,1H3,(H,20,22)(H,21,25)(H,23,24)/f/h20-21,23H"

...

Many customizations of the above code are possible. For example, it would not be difficult to programatically log into the PubChem FTP-server, download a file, and process it as shown. By parsing the SDFGZ filename, a program could even know which file contained a given CID. Because the SDFGZSplitter constructor takes a Ruby IO object, it's also feasible to process PubChem's SDFGZ files directly from the FTP-server, without downloading them beforehand. But that's a subject for another day.

Summing Up

The PubChem FTP-server is a treasure trove of useful data that's available free of charge. Using simple tools like those discussed here, it's possible to generate a virtually infinite variety of customized views of this valuable resource. Many creative, and novel, applications are possible by combining the capabilities shown here with those of Open Source chemical informatics software, such as RCDK, and other Open data sources, such as NMRShiftDB.