Depth-First: Tag merckindex

Building Chempedia: Learning About Contributors

Rich Apodaca — Wed, 02 Jul 2008 11:50:00 -0400

Chempedia is a free online chemical encyclopedia similar in concept to the Merck Index, but radically different in implementation. One key difference: the Merck Index is compiled by a small number of paid professionals while Chempedia is compiled by thousands of unpaid volunteers. Although this distinction raises a host of intriguing questions, one of the most basic revolves around what can be said about these volunteers in the aggregate. This article, the first in a series, explores this issue with some statistics compiled from Chempedia.

Learning About Contributors

Chempedia works in part by aggregating content from Wikipedia dealing with single molecular entities, or "Compound Monographs." This content is created by the now famous process of individuals taking upon themselves the responsibility of fixing what's broken in Wikipedia. (Some take it upon themselves to break what's working, but that's another topic.)

Chempedia associates each of its Compound Monographs with the last Wikipedia user to edit it. The current interface to these relationships is available on the Chempedia contributors page.

The interface to this page is currently limited. The analyses reported here were made for the most part by querying the Chempedia database directly.

Each contributor is linked to a contributor summary page containing links to that user's Wikipedia homepage and talk page, as well as a complete listing of all active contributions. For example, you can view the contributor page for one of Chempedia's most active contributors, Arcadian.

The data model is also limited. Because Chempedia only records the last Contributor to edit a Monograph, when another Contributor edits a Monograph, the link between the previous Contributor is lost. As a result, many Contributors have no associated Monographs.

How Many Monographs?

Chempedia currently hosts 6,308 Compound Monographs.

How Many Contributors?

Chempedia currently lists 2,516 Contributors. Of these, 1,046, or 42% are associated with one or more Monographs, meaning that they were the last to edit. The remainder are associated with no Monographs for which they were the last to edit.

Here is a list of the top 20 Contributors and the number of Monographs they were the last to edit:

anonymous	1022
DOI bot	904
Edgar181	378
Fvasconcellos	170
Meodipt	151
Arcadian	144
Chem-awb	133
Chowbok	122
Rifleman 82	114
SmackBot	105
Thijs!bot	99
ChemNerd	85
Puppy8800	80
DumZiBoT	78
Axiosaurus	63
Chempedia	63
Carlo Banez	55
Benjah-bmm27	52
OKBot	51
Cacycle	50

These Contributors represent 1.9% of all active Contributors and collectively are responsible for being the last to edit 62% of all Monographs. Although not performed here, a histogram plotting number of contributions would be expected to follow a power law.

'Anonymous' is an aggregation of all users who edited a Monograph without a Wikipedia account. 16% of all Monographs were last edited by an anonymous user. Leaving out the aggregated 'anonymous' users indicates that roughly half of all Monographs were last edited by the top 19 Contributors.

What is a Contributor?

Although it's difficult to say a lot about individual Contributors, most appear to have some training in science, although that training may not have involved chemistry or biology. Still others (for example, SJP) appear to have been drawn to contribute to a Monograph based on their nonscientific experience with the title compound or in an effort to fight vandalism or otherwise improve the nonscientific content of the Monograph. The ability of services like Wikipedia (and by extension Chempedia) to provide a platform for those without formal training in a particular area to make useful contributions is without question one of its most useful (and controversial) features.

Some Contributors are not even human, but rather robots designed to improve the quality of Wikipedia articles in general. For example, SmackBot performs an array of tedious quality control jobs such as fixing bad checksum ISBNs (CAS Numbers, anyone?) and capitalization errors.

Conclusions

Wikipedia's collaboration model has made the creation of a free and continuously-updated chemical encyclopedia feasible. Applying chemistry-specific user interfaces and data models exposes this hidden treasure. Although it's tempting to think of this process as mainly being the work of a handful of trained scientists, the numbers suggest a much broader base of contributors. Future articles will explore this idea.

Building Chempedia: Start Simple, Then Iterate

Rich Apodaca — Tue, 13 May 2008 11:38:00 -0400

As a medium for building software, the Web offers unparalleled adaptability. With nothing to download or install, users of Web applications automatically see the newest version - always. This may sound like a small thing, and technically it is. But it dramatically increases the effectiveness with which software can be created. The previous article in this series introduced Chempedia, the free Chemical encyclopedia and cheminformatics Web application. This article will discuss the process by which Chempedia will become a better service over time.

Iterative Web Application Development

Chempedia, like all actively-developed software, is a work in progress. It will be built in stages starting with the addition of new features, followed by a round of user feedback, bug fixing, and stabilization. This will then be followed by the next major iteration, and so on.

This iterative design style is ideally suited for Web applications. Because the barrier to pushing out new versions is essentially non-existent, a Web application can evolve at a much more rapid rate than other kinds of software. Indeed, the first version of a Web application need only work well enough to prove a point.

One of the keys to iterative Web development is a technology framework designed to facilitate it. Chempedia is being developed with Ruby on Rails, a tool that enables Web developers to take full advantage of the iterative development style the Web makes possible.

Another key element of iterative Web development is users willing to explore the system and offer criticism. Evolution succeeds only when the environment stresses an ecosystem; the same is true in Web application development.

Chempedia will take full advantage of the evolutionary nature of Web application development. As features are added and (hopefully) use of the service grows, Chempedia will evolve in ways that are impossible to predict today.

What's Wrong With Chempedia?

If you happened to take a look at Chempedia last week (that version is now no longer visible), you probably noticed many, many things that needed improvement. Some concerns were in the areas of:

Navigation. Navigation works best when the right granularity of options is achieved. Chempedia's navigation system grouped both closely-related and dissimilar actions at the same level.
Metaphor. The initial idea behind Chempedia was to see what happened when PubChem's chemical structures were mashed up with Wikiepia articles, using CAS numbers as the common link. The site design reflected this, with no clear organizing principle other than mashup. However, after the initial demonstration of the success of this approach, it became clear that Chempedia was strikingly similar in both form and function to the Merck Index. Perhaps this should be used as a clue in deriving a better organizing principle.
Wikipedia integration. The old Chempedia site didn't make it nearly as convenient as is should be to create or edit compound monographs. Because Chempedia serves as a chemically-aware front-end for Wikipedia, the easier it is to get to Wikipedia from Chempedia, the better.

What Changed?

During the process of trying to fix Chempedia's problems, it became clear that a major redesign was in order. This consisted of:

Creating a landing page oriented toward search. Using the Merck Index as a metaphor suggested that Chempedia's landing page should be designed around search, not browsing - as it was originally designed.
Emphasizing compound monographs, not compounds. Chempedia's central organizing principle is now the Compound Monograph. One way this is seen is in the new URL structure, which makes it very easy to see where a Chempedia link is about to take you. For example, consider the URL for benzene. Another way this can be seen is in the inclusion of Compound Monographs lacking a chemical structure.
Designing a streamlined menu system. The main menu system has been broken down into just three main categories: Search; Browse; and Create. These headings refer to actions on Compound Monographs, again in line with their importance as an organizing principle.
Promoting better integration with Wikipedia. After experimenting with a few implementation possibilities, it is now possible to edit Wikipedia articles directly from the Chempedia site, thanks to the use of inline frame. Once again, this capability is tied to the Compound Monograph, from which editing and updating links are accessible.
Striving for comprehensive Wikipedia coverage. Wikipedia had far more compound monographs than could be found on Chempedia, 6,411 of them, to be precise. Chempedia now contains all of them, regardless of whether a chemical structure can be found based on a CAS number in PubChem. This includes inorganics, organometallics, polymers, mixtures, and polypeptides.

Miles to Go Yet

Chempedia is far from being finished. For example, you'll notice many instances in which a Compound Monograph is truncated. This arises from difficulties in parsing Wikipedia's Wikitext format (more on this later).

Ultimately, the full text of each Wikipedia article will be present on Chempedia rather than just the first introductory paragraph. But it will take a significant amount of work to ensure that each article's Wikitext entry can be parsed faithfully.

Chempedia allows search by CAS number, PubChem CID and exact title. Full-text searching is not yet implemented, nor is autocomplete search, both of which would greatly enhance the usability of the service.

Exact structure searching is made possible by the ChemWriter editor in combination with SHA-1 hashed InChIs. Substructure search and query atom search will ultimately be added, but for an encyclopedia containing relatively few molecules, most of which having trivial names, this isn't yet seen as being critical.

You'll notice many Monographs on Chempedia that have no structure information. Behind the scenes, Chempedia uses the 350,000+ CAS numbers now contained in the PubChem database to associate a chemical structure with a Wikipedia article. In the future, these associations will be made by Chempedia and Wikipedia users, which will allow every Chempedia small-molecule Monograph to have a structure associated with it. (It will also create a rather large, publicly-curated, open database of CAS numbers linked to chemical structures, but that's a story for another time).

Your Feedback is Essential

Finally, many of the changes made in this iteration were the result of conversions with chemists and developers. If you see something on Chempedia that just doesn't work for you, please don't be shy about saying so. Feedback is an essential ingredient in making Chempedia the best service it can be.

Building Chempedia: Indexing Wikipedia's 6,411 Compound Monographs

Rich Apodaca — Mon, 28 Apr 2008 18:22:00 -0400

The Merck Index is one of chemistry's most useful reference works. Organized like an encyclopedia, each entry, or "Compound Monograph," describes a single compound complete with chemical structure, CAS Number, IUPAC name, trivial names, physical properties, and leading primary literature references describing uses. Unlike other chemistry databases, the Merck Index focuses on only those compounds with important industrial, biological, medical, or technical applications.

What's Wrong with the Merck Index?

Wonderful product though it may be, the Merck Index has some limitations. For starters, online versions are not free. The disadvantages of this access model go well beyond a simple price barrier; it prevents the very thing the Web was designed to promote: linking. Another limitation is the time it takes for new versions to appear, which is typically measured in years. Still another limitation is in the cost of adding entries for niche compounds that may not be suitable for a general audience, a major barrier to exposing chemistry's long tail.

What's Chempedia?

If we wanted to create a free, online service that worked like the Merck Index but which took full advantage of today's powerful collaboration and information technology tools, how could we go about doing so?

This article, the first in a series, discusses Chempedia, a free, structure-oriented online encyclopedia of useful chemical compounds designed to answer this question.

Background

The following articles may be useful in understanding Chempedia's approach and underlying technology:

Where to Begin?

One of the first problems we'd face in building a free Web-based version of the Merck Index is where to get the compound monographs.

It turns out that Wikipedia (yes, Wikipedia) hosts a growing collection of compound monographs that, when viewed together, bear a striking resemblance to the Merck Index. And the effort is becoming increasingly organized with respect to content and data provenance.

Why not start here?

The Task at Hand

To get an idea of just how Wikipedia's collection of compound monographs compares to the Merck Index, it helps to know: (1) how to find Wikipedia compound monographs; and (2) the range of information available for each entry.

This tutorial will describe a simple method to index Wikipedia's compound monographs using nothing but free tools and data. Subsequent articles will discuss qualitative aspects of Wikipedia's compound monographs and the challenges involved in organizing them into a chemically-aware service.

Indexing Wikipedia's Compound Monographs

We can index Wikipedia compound monographs via a simple procedure.

Most compound monographs employ one of four precompiled Wikpedia templates: Chembox (deprecated); Chembox new; Drugbox; and Explosivebox. As an example of what these templates look like, see the right-hand box on Wikipedia's entry on modafinil. To index Wikipedia's compound monographs, all we need to do is find the titles of all articles using one of these four templates.

To get started, we'll need a local copy of Wikipedia. The complete set of all Wikipedia articles, as of March 12, 2008 can be downloaded here. This data dump is updated periodically, so you may have access to a more recent version.

The Wikipedia dump, which contains the full text of every article in Wikipedia, consists of a 3.5 GB file in BZip2 format. Fortunately, we won't need to inflate it to index its chemical content.

The following code will scan the raw Wikipedia dump and produce a list of all compound monograph titles:

title = ""
log = File.new 'monographs.txt', "w"

while((line = STDIN.gets))
  line.match /<title>(.*)<\/title>/

  if $1
    title = $1

    next
  end

  if line.match /\{\{(chembox|drugbox|explosivebox)/i
    unless title == "" || title.match(/:/)
      puts title
      log.puts title
      log.flush

      title = ""
    end
  end
end

log.close

Saving this code into a file called filter.rb, we can run it by piping the output of bzcat on the raw dump file:

$ bzcat <path_to_dump>/enwiki-20080312-pages-articles.xml.bz2 | ruby filter.rb

Alphabetizing the output file gives a complete listing of Wikipedia's compound monograph titles (all 6,411 of them), which for convenience can be downloaded here.

We can construct a URL to each Wikipedia compound monograph by prepending the title with http://wikipedia.org/wiki/. In other words, our program's output can be used both as a list of chemical names and as a hash of chemical names to Wikipedia URLs. And with the URL in hand, all kinds of interesting things can be done.

Limitations

Although easy to carry out, the procedure described here has some limitations:

Monographs added after March 12, 2008 are not visible.
Monographs that don't use the chembox, chembox new, drugbox, or explosivebox templates are not visible.
A very small number of articles erroneously use the chembox template, for example this one.

Chempedia Redesign

Currently, Chempedia doesn't include all 6,411 monographs but rather a subset created by a much less comprehensive indexing method. As part of a major redesign of the site, all Wikipedia compound monographs will be available on Chempedia, which should result in a much more useful service.

Conclusions

Wikipedia is fast becoming a major storehouse of chemical information with tantalizing potential for creating powerful new services for chemists. More to the point for cheminformatics, the entire Wikipedia dataset can be downloaded and reprocessed free of charge; Wikipedia is one of those rare cheminformatics datasets that is both free as in speech and free as in beer.

As this article has shown, some simple programming is all it takes to begin doing useful things with Wikipedia's chemical content. Future articles will discuss some of the possibilities.

User-Created Compound Monographs on Chempedia.net: Open Sourcing the Collation and Indexing of Chemical Information

Rich Apodaca — Thu, 17 Apr 2008 17:50:00 -0400

Printed encyclopedias of chemical information like the Merck Index suffer from the problem of becoming obsolete on publication. When new compounds are discovered, or when the information about a compound changes, those changes can take many months or years to appear in print form due to the high cost of publication. It doesn't have to be that way. This article introduces a new feature to the free online chemical encyclopedia Chempedia that lets working scientists update is contents via Wikipedia.

About Chempedia.net

A recent article introduced Chempdia, the free online chemical encyclopedia. This service is built on two of the largest free and open repositories of chemical information in existence: Wikipedia and PubChem. PubChem supplies low-level chemical information such as connection tables, and Wikipedia supplies free-text descriptions of the properties and uses of certain molecules.

Which Molecules?

Currently, Chempedia.net only includes compound monographs for about 1,000 of its over 300,000 molecules. These monographs were located by a manual process in which the titles for all Wikipedia articles were downloaded in alphabetized form; this process clustered titles that represented IUPAC nomenclature due to its use of leading numbers and symbols. IUPAC nomenclature titles were extracted, and then a script was written to extract the chemical information from these titles and combine it with that from PubChem.

This method, although useful for getting a service running, is clearly flawed. The biggest problem is in how to discover new compound monographs.

Why Not Put Users in Control?

Chempedia users themselves are in the best position to know when an existing Wikipedia compound monograph should appear in Chempedia but doesn't, when an existing monograph needs to be updated, or when a new monograph is written and needs to be linked.

How can the process be automated?

As a partial answer to this question, users now have the ability to notify Chempedia of any changes to a Wikipedia compound monograph, and to have those changes immediately reflected in the next viewing of a Chempedia compound monograph.

An Example

As an example, let's take anandamide, a compound I've had some experience with during my time as a medicinal chemist. Although the Chempedia entry for ananandamide exists, there is (or as of today - was) no link to the Wikipedia compound monograph. Let's create one.

At the top of Chempedia's main menu, you'll see a link titled 'Update'. Choosing this link leads to a form that will ask for two pieces of information: (1) the title of the Wikipedia article to which you want Chempedia to link - in this case 'anandamide'; and (2) reCaptcha text to keep robots from making mischief.

Submitting this information is all that's needed to create a new or updated link from Chempedia to Wikipedia. Chempedia handles the rest.

Conclusions

Wikipedia is a vast source of free, high-quality, semi-structured chemical information just waiting to have good chemically-aware interfaces applied to it. Chempedia.net is an attempt to do just that, but it's a bit more as well. Although it may appear that Chempedia is the major beneficiary in this relationship, Wikipedia also benefits. When chemists have a tool that allows them to query and visualize Wikipedia using their native language (the chemical structure) they're in a better position to both use and contribute to Wikipedia itself - something I've started to do.

This positive feedback effect is the real value of exposing Web services. The question is: who in cheminformatics is willing and able to take the risk to discover this simple principle and its benefits?