Depth-First: Tag commandline

Fast Substructure Search Using Open Source Tools Part 4: Creating Fingerprints from Chemical Structures

Rich Apodaca — Wed, 15 Oct 2008 14:42:00 +0000

The previous articles in this series have detailed the steps needed to build a working fingerprint screening system using nothing more than the open source tools MySQL, Ruby, and ActiveRecord. With this system we can create, read, update, and destroy fingerprints in persistent storage. Although the system meets all of the requirements of a fingerprint screening system, it isn't a substructure search system - yet. For that, we need a way to convert chemical structure representations into fingerprints. This article describes a very simple method for doing so.

All Articles in this Series:

Part 1: Fingerprints and Databases
Part 2: Fingerprint Screen With SQL
Part 3: A CRUD API for Fingerprints in Ruby
Part 4: Creating Fingerprints from Chemical Structures
Part 5: Relating Molecules to Fingerprints with SQL
Part 6: Modelling a One-To-Many Relationship Between Fingerprints and Compounds in Ruby

A Ruby Fingerprinter in Eight Lines

Let's create a Fingerprinter class that's capable of converting a SMILES string into a Fingerprint that can be stored and queried. The Ruby code below makes use of Open Babel's babel command-line utility:

require 'fingerprint'

class Fingerprinter  
  def fingerprint_smiles smiles
    raw = %x[echo '#{smiles}' | babel -ismi -ofpt 2>/dev/null]
    bytes = raw.gsub(/>.*?\n/, '').gsub(/\n/, '').split

    Fingerprint.new.fill_bytes{|i| "#{bytes[2*i]}#{bytes[2*i+1]}".hex}
  end
end

This class takes advantage of Ruby's ability to interface directly with the command line through the %x operator in a way similar to that previously described for the cInChI command line tool. The babel output is then converted into a form suitable for use with our previously-defined Fingerprint class.

Although quite easy to implement, this approach may not work in every situation. For example, the fingerprint_smiles method opens the possibility that a malicious user could attempt to execute arbitrary shell commands by creating a mis-formed SMILES string. Windows users may need to adapt the code. But for trusted SMILES on Unix machines, this implementation works well and can be used in many different programming environments.

Testing the Fingerprinter

We can test the Fingerprinter through interactive Ruby (irb):

$ irb
irb(main):001:0> require 'lib/fingerprinter'
=> true
irb(main):002:0> fp=Fingerprinter.new
=> #<Fingerprinter:0xb7498038>
irb(main):003:0> f=fp.fingerprint_smiles 'c1ccccc1'
=> #<Fingerprint id: nil, byte0: 0, byte1: 512, byte2: 0, byte3: 0, byte4: 2112, byte5: 32768, byte6: 0, byte7: 0, byte8: 0, byte9: 0, byte10: 134217728, byte11: 0, byte12: 0, byte13: 0, byte14: 131072, byte15: 0, hex: nil>
irb(main):004:0> f.cardinality
=> 6
irb(main):005:0> f.bitstring
=> "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000100000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000"

As we previously saw, any Fingerprint we create can be stored and later retrieved from a MySQL database. If we've already stored the fingerprint for benzene it can be found with the following:

$ irb
irb(main):001:0> require 'lib/fingerprinter'
=> true
irb(main):002:0> fp=Fingerprinter.new
=> #<Fingerprinter:0xb74ae284>
irb(main):003:0> f=fp.fingerprint_smiles 'c1ccccc1'
=> #<Fingerprint id: nil, byte0: 0, byte1: 512, byte2: 0, byte3: 0, byte4: 2112, byte5: 32768, byte6: 0, byte7: 0, byte8: 0, byte9: 0, byte10: 134217728, byte11: 0, byte12: 0, byte13: 0, byte14: 131072, byte15: 0, hex: nil>
irb(main):004:0> Fingerprint.find_by_fingerprint f
=> #<Fingerprint id: 12687, byte0: 0, byte1: 512, byte2: 0, byte3: 0, byte4: 2112, byte5: 32768, byte6: 0, byte7: 0, byte8: 0, byte9: 0, byte10: 134217728, byte11: 0, byte12: 0, byte13: 0, byte14: 131072, byte15: 0, hex: "000000000000000000000000000002000000000000000000000...">

Conclusions

We now have the ability to create, store, and query fingerprints created from arbitrary SMILES strings. If there were a 1:1 relationship between molecules and fingerprints, we'd be nearly done. But things are not quite that simple. The next article in this series will show how to relate molecules to fingerprints.

Image Credit: adrenalin

Recombining Compressed PubChem SD Files with Open Babel

Rich Apodaca — Wed, 01 Oct 2008 01:25:00 +0000

While testing ChemPhoto, it became necessary to test the chemical structure imaging application with SD Files containing several hundred thousand records. Although it's tempting to meet this need by constructing "dummy" files with the same record or small set of records repeated, tests are always far more illuminating when real data is used.

PubChem is an excellent source of large molecular datasets, and the entire database can be downloaded by FTP. Because of PubChem's massive size, what's downloadable consists of files broken up into groups of about 25,000 in gzipped SD File format (*.sdf.gz). Although this is an excellent resource, it creates a problem: how can you conveniently recombine this set of compressed SD Files into a single SD File?

You might think about writing some "quick" code in your language of choice. Fortunately, Open Babel gets the job done - without any of the coding or debugging.

The following command will create a single SD File from all of the compressed SD Files in a given directory, while also stripping explicit hydrogens and removing all fields except PUBCHEM_COMPOUND_CID.

babel *.sdf.gz pubchem.sdf -d --delete PUBCHEM_COMPOUND_CANONICALIZED,PUBCHEM_CACTVS_COMPLEXITY,PUBCHEM_CACTVS_HBOND_ACCEPTOR,PUBCHEM_CACTVS_HBOND_DONOR,PUBCHEM_CACTVS_ROTATABLE_BOND,PUBCHEM_CACTVS_SUBSKEYS,PUBCHEM_IUPAC_OPENEYE_NAME,PUBCHEM_IUPAC_CAS_NAME,PUBCHEM_IUPAC_NAME,PUBCHEM_IUPAC_SYSTEMATIC_NAME,PUBCHEM_IUPAC_TRADITIONAL_NAME,PUBCHEM_NIST_INCHI,PUBCHEM_EXACT_MASS,PUBCHEM_MOLECULAR_FORMULA,PUBCHEM_MOLECULAR_WEIGHT,PUBCHEM_OPENEYE_CAN_SMILES,PUBCHEM_OPENEYE_ISO_SMILES,PUBCHEM_CACTVS_TPSA,PUBCHEM_MONOISOTOPIC_WEIGHT,PUBCHEM_TOTAL_CHARGE,PUBCHEM_HEAVY_ATOM_COUNT,PUBCHEM_ATOM_DEF_STEREO_COUNT,PUBCHEM_ATOM_UDEF_STEREO_COUNT,PUBCHEM_BOND_DEF_STEREO_COUNT,PUBCHEM_BOND_UDEF_STEREO_COUNT,PUBCHEM_ISOTOPIC_ATOM_COUNT,PUBCHEM_COMPONENT_COUNT,PUBCHEM_CACTVS_TAUTO_COUNT,PUBCHEM_BONDANNOTATIONS,PUBCHEM_CACTVS_XLOGP

865543 molecules converted
7 info messages 15372962 audit log messages

Apparently, there is no way to tell babel to keep just a particular field in an SD File - they need to be removed individually.

Still, not bad for a few seconds on the command line.

Rethinking the Command Line for Chemistry

Rich Apodaca — Tue, 27 Mar 2007 12:30:00 +0000

A recent article discussed the renaissance of the command line. Particularly on the Web, command line interfaces have become so advanced, that most of us don't even realize we're using them. Consider the Google search box, which is nothing more than one of the most powerful command line interfaces ever developed.

A service called YubNub takes this idea one step further. YubNub is a meta command line interface for the Web. The following YubNub command will do a Flickr search for benzene.

If this were all YubNub did, it would be merely interesting. What makes YubNub remarkable is that you can create your own commands that other people can use. I recently added the "ginchi" command to query Google for an InChI. Now you can try it out:

By itself this isn't particularly useful because you can just go to Google and query the InChI directly. However, it's not too hard to imagine several commands like ginchi that could be added. Some would use Google, others would use other services. How about something that searches Mitch Garcia's chemistry journal Yahoo pipe? It would be very convenient to have all of those commands accessible from the same Web page.

Command line interfaces can be phenomenally useful for both beginning and advanced users. The hardest part to get right is not what the user sees as they type, but what happens after they hit the enter key.

Line notations are the perfect match for command line interfaces. The widespread use of SMILES and the precision of InChI offer many possibilities for innovative chemistry Web services.

Customize InChI Output with Rino

Rich Apodaca — Mon, 19 Mar 2007 10:30:00 +0000

Rino is a toolkit for working with the IUPAC International Chemical Identifier (InChI) in Ruby. Because it's based on the IUPAC/NIST InChI toolkit, Rino can be configured using a variety of useful options. This article summarizes those options and provides an illustrative example.

Complete List of InChI Command Line Options

The following is a complete summary of the IUPAC/NIST InChI toolkit command line options:

SNon Exclude stereo (Default: Include Absolute stereo)
SRel Relative stereo
SRac Racemic stereo
SUCF Use Chiral Flag: On means Absolute stereo, Off - Relative
SUU Include omitted unknown/undefined stereo
NEWPS Narrow end of wedge points to stereocenter (default: both)
SPXYZ Include Phosphines Stereochemistry
SAsXYZ Include Arsines Stereochemistry
RecMet Include reconnected metals results
FixedH Mobile H Perception Off (Default: On)
AuxNone Omit auxiliary information (default: Include)
NoADP Disable Aggressive Deprotonation (for testing only)
Compress Compressed output
DoNotAddH Don't add H according to usual valences: all H are explicit
Wnumber Set time-out per structure in seconds; W0 means unlimited
SDF:DataHeader Read from the input SDfile the ID under this DataHeader
NoLabels Omit structure number, DataHeader and ID from InChI output
Tabbed Separate structure number, InChI, and AuxIndo with tabs
OutputSDF Convert InChI created with default aux. info to SDfile
InChI2InChI Convert InChI string into InChI string for validation purposes
SdfAtomsDT Output Hydrogen Isotopes to SDfile as Atoms D and T
STDIO Use standard input/output streams
FB (or FixSp3Bug) Fix bug leading to missing or undefined sp3 parity
WarnOnEmptyStructure Warn and produce empty InChI for empty structure

A Test

The following code displays the InChI for benzoic acid with and without mobile hydrogen atom perception. It requires both Rino and Ruby CDK. The latter library is used to convert a SMILES string into a molfile for use by Rino.

require 'rubygems'
require_gem 'rcdk'
require_gem 'rino'
require 'rcdk/util'

molfile=RCDK::Util::Lang.smiles_to_molfile 'c1ccccc1C(=O)O' # benzoic acid
reader = Rino::MolfileReader.new
inchi = reader.read(molfile)

puts "Without mobile hydrogen perception:\n#{inchi}\n\n"

reader.options << '-FixedH'
inchi = reader.read(molfile)

puts "With mobile hydrogen perception:\n#{inchi}"

The -FixedH flag used by the reader the second time tells Rino to identify mobile hydrogens in the InChI output. Some InChI authors use this form of InChI and others don't. PubChem is an example of a large InChI author that does use mobile hydrogen perception, as their entry for benzoic acid demonstrates. To perform an exact match of your InChIs with theirs, the -FixedH flag must be set.

Running the Test

Running the test code produces the following output:

Without mobile hydrogen perception:
InChI=1/C7H6O2/c8-7(9)6-4-2-1-3-5-6/h1-5H,(H,8,9)

With mobile hydrogen perception:
InChI=1/C7H6O2/c8-7(9)6-4-2-1-3-5-6/h1-5H,(H,8,9)/f/h8H

Conclusions

When matching InChIs generated by other authors, it's best to adopt their processing conventions. Rino makes it conventient to do so through its full support for the standard IUPAC/NIST command line options.

Do You Use the Command Line?

Rich Apodaca — Thu, 15 Mar 2007 11:18:00 +0000

In the run to abandon command line interfaces for the GUI, we've left behind the versatility of language.

...

[Imagine] using a drop-down menu to select the one web site you want to go to out of the 100 million web sites in existence. Ludicrous! How do we actually surf to a site? By typing an address into the address bar. When we want to go to the mail "application", we type in "gmail.com"; when we want to open a news "application", we type in "nytimes.com". On the old unix command lines, we would type type "pine" and "rn". See a similarity? The address bar is just a primitive command line. A command line that your grandmother can—and does—use.

-Aza Raskin, Get Humanized

The command line is alive and well. It's simply become so sophisticated that most of us don't realize we're using it. Whether we're entering a URL into a browser address bar, taking advantage of autocomplete to look up a co-worker's name in an address book, or using Google to search the Web, the command line is hard at work. Most people wouldn't want it any other way.

To an end user, a command line is nothing more than a box to enter text. The magic happens when this text is processed. Aza Raskin's company Humanized uses this simple idea to build text-driven applications that save time and effort.

What would happen if the same thinking were applied to chemical informatics?

Image credit: Bartholomule - Flickr