The Horrifying Future of Scientific Communication

May 08, 2013

Technological change has a way of making fools out of experts and experts out of fools. One of the reasons that evaluating the future impact of new technologies is so difficult is that those with the greatest potential often start out looking - and acting - like junk. They can't initially do the job nearly as well as the technologies they end up replacing. They are written off as toys.

But a small number of these toy technologies have two remarkable qualities: (1) they are much cheaper or easier to use; and (2) they improve at a rate much faster than the competition. Over time, their performance is more than enough for even the most demanding uses, at which point the new technology dominates. Disruptive Innovation is the process of cheap, low-quality products ultimately displacing expensive, high-quality products, typically ruining the prospects of organizations caught on the losing side.

Examples of Disruptive Innovation are found throughout technical fields and can take the form of products (personal computer vs. smartphone), services (snail mail vs. email), ways of delivering goods and services (Blockbuster vs. Netflix), and standards (Flash vs. HTML5).

Disrupting the Business of Scientific Communication

Although it's possible that scientific communication is somehow immune to the forces of disruption found just about everywhere technology exists, it seems unlikely. Still, it hasn't happened yet.

One explanation could be the unusually high value placed on quality by both consumers (readers) and producers (authors). For example, consider the consistency with which scientists in any discipline can rank the relative face-value quality of an article based merely on the title of the journal it appears in. These perceptions are reinforced by a feedback mechanism in which scientists themselves rank the quality of their own work and select a journal for manuscript submission accordingly.

Clearly, electronic journal distribution marked a major turning point. But this advance represents a sustaining innovation: the quality of an existing product improved, allowing market leaders to grow their businesses by doing more or less what they've always done. One confirmation is that scientific journals are getting more expensive every year, not less, despite the fact that few print journals remain.

Although it's impossible to say what specific form a true disruptive innovation in scientific communication will take technically, it may be recognized through its qualities and effects:

  1. It will be much cheaper or easier to use.
  2. It will be demonstrably worse in one or more ways.
  3. Its low cost and/or ease of use will attract large numbers of new participants from the fringes of science or scientific disciplines.
  4. It may be criticized as a poor replacement for scientific journals, and possibly dangerous.
  5. A path to rapidly improve the things it does poorly or obviate their importance is already available or will be soon.
  6. It may be created from off-the-shelf components.

It seems counterintuitive: to discover the future of scientific communication today, begin by sifting through the cheapest, simplest, lowest-quality replacements currently available. Be sure the options would appeal to large numbers of scientists and sci-curious left on the sidelines by the current system. Then narrow the list to those approaches with the greatest potential for rapid, sustained improvement.

Open Access

Open Access publishers are sometimes accused of lowering publication standards, thereby lowering the quality articles being published. A recent example comes courtesy of Derek Lowe in a critique of a paper linking the active ingredient in a widely-used pesticide to "pretty much every chronic illness in humans." The last paragraph contains this perspective on the Open Access journal publisher, MDPI:

But really, all you need to know is that MDPI is the same family of "journals" that published the (in)famous Andrulis "Gyres are the key to everything!" paper. And then made all kinds of implausible noises about layers of peer review afterwards. No, this is one of the real problems with sleazy "open-access" journals. They give the whole idea of open-access publishing a black eye, and they open the floodgates to whatever ridiculous crap comes in, which then gets "peer reviewed" and "published" in an "actual scientific journal", where it can fool the credulous and mislead the uninformed.

Given that the quality of MDPI and other Open Access publishers is apparently lower than others, could we be looking at an early-stage marketplace disruption? Let's take stock:

  1. Much cheaper or easier to use? MDPI makes money by assessing authors an Article Processing Charge ranging from gratis to 1800 Swiss francs (~US $1900). In exchange, articles are available to readers without subscription cost. Although the charge to authors is not very much less than other publishers offering OA distribution, the cost to readers is much less. There's little apparent difference in ease of publication, although ease of reading is vastly improved. Conclusion: maybe.
  2. Demonstrably worse? Authors of papers appearing in MDPI journals can expect none of the prestige that authors of Science papers enjoy. Imprimatur matters. Likewise, readers of MDPI journal articles will likely approach any new article with either no knowledge of the publisher or a negative impression. Conclusion: yes.
  3. Attracts fringe elements? The paper criticized by Lowe was authored not by trained research toxicologists, but by an "Independent Scientist and Consultant" and a computer scientist with a bachelor degree in biophysics. Conclusion: yes.
  4. Criticized as dangerous? Lowe isn't alone in his criticism of MDPI. If anything, there seems to be an inherent bias against Open Access for the way it opens lines of communication previously closed off by editorial policy. Conclusion: yes.
  5. Path for improvement? If there is a way for upstart Open Access journals to improve and sustainably increase quality while keeping costs under control, it's not obvious at this point. Conclusion: no.
  6. Built with off-the-shelf components? I'm not familiar with how MDPI has built its platform. It wouldn't be hard to replicate the technical side given the availability of turnkey services such as Scholastica. Building an editorial board and network of peer-reviewers, on the other hand, would be a challenge. Conclusion: no.

Being worse and a little cheaper isn't enough for disruptive innovation. Many early-stage products start out looking cheap and ineffective. A steep, sustainable trajectory for improvement separates those that will remake industries.

A major problem with MDPI and other Open Access publishers is that they retain most of the practices and associated costs of established publishers: editorial boards and pre-publication peer review. These costs take the form of both money and time to publication. For this reason, it's not that much less expensive or difficult to publish in MDPI than, say, ACS Publications. In some ways, MDPI and some other Open Access Publishers have simply created a somewhat less expensive product missing one critical feature: brand-name recognition.

Open Access may turn out to be the ultimate sustaining innovation for established publishers (although I've suggested otherwise in the past). By transferring publication costs from reader to author, publishers can maintain current profit margins, offer a far more accessible product to readers, and continue to attract high-quality manuscripts by leveraging brand-names built in the previous print era. Whether publishers willingly adapt or are forced by government mandate, the result will be the same.

Defenders of Scientific Integrity

The traditional mission of scientific journal publisher was threefold: (1) to assure quality; (2) convert manuscripts into publication-ready format; and (3) to physically distribute printed articles.

The rise of desktop publishing long ago rendered Part (2) of the publisher mission obsolete.

The Internet has in turn rendered Part (3) of the publisher's mission obsolete. The often-cited need to ensure continued electronic distribution of the scientific record is doubtful at best given non-restrictive licensing coupled with ever-contracting prices of Web servers, bandwidth, and permanent storage.

It's hardly surprising that subscriber-pays scientific publishers have focused on the one service they now offer that has not become dirt cheap: quality assurance.

All of this has led some to reconsider the publisher value proposition. They argue that journals actually do a lousy job at protecting science from plagiarism, fraudulent claims, erroneous conclusions, and unrepeatable procedures. As evidence, they point to numerous cases described on sites such as Retraction Watch. Even worse, the more successful and established journals operate under a shroud of semi-secrecy resulting from high access fees.

According to Michael Eisen, co-founder of PLOS, science suffers from lack of honest debate around scientific communication and an inability to adapt to a world in which the rules have changed:

So, while it is a nice idea to imagine peer review as defender of scientific integrity – it isn’t. Flaws in a paper are far more often uncovered after the paper is published than in peer review. And yet, because we have a system that places so much emphasis on where a paper is published, we have no effective way to annotate previously published papers that turn out to be wrong.

Could it be that our faith in journal publishers and particularly in pre-publication peer review is misguided? Have we scientists and the public at large been lulled into a false sense of security every time we see the name of a flagship journal on an article header or on a CV? Could there be far more effective ways to defend science from honest mistakes, plagiarists, and crackpots with axes to grind?

Gazing into the Future

For an idea of what a future without scientific journal publishers could look like, consider these services:

  • arXive, a pre-publication article archive for sciences and humanities
  • Figshare, a repository allowing deposition of raw, unreviewed research results
  • Hacker News, a news service without moderators
  • Stack Exchange, a family of self-governing Q&A sites
  • Disqus, a simple way to add online discussion to any Web page

What all of these highly-successful, self-sustaining, and in the case of Stack Exchange, profitable, services have in common is that they provide the technical underpinnings necessary to make post-publication peer review work.

Even so, the idea of a scientific paper as something of value may be losing relevance. Consider how many papers you've actually read, all the way through, compared to the number of papers you've skimmed for experimental details or quick conclusions. Supporting Information, combined with data mining tools, could eliminate most of the need for manuscripts in the first place. Ranking and feedback mechanisms could neutralize most of the value derived pre-publication review, just like Google made it possible to productively navigate a Web littered with far more garbage than gold.

Conclusions

Disruptive Innovation is a process by which previously expensive products or service become radically cheaper or easier to use. Scientific communication is an extremely inefficient and costly process that in many ways appears ripe for disruption. Although Open Access might seem to fit the bill, the more likely outcome is that existing publishers will simply adopt OA publishing practices.

Real disruptive innovation will slash the costs of creating and using scientific communication products and services. Pre-publication peer review and editorial boards will likely be the first of many casualties. In the process, scientific communication will become accessible to entirely new groups of authors and readers currently being shut out.

It goes without saying that this future scares the hell out of publishers. But it will no doubt scare the hell out of many scientists as well.

Planes, Trains, and Organic Syntheses

April 23, 2013

Riders of commuter trains are accustomed to using maps like those produced by the London Underground:

These maps are capable of condensing large amounts of information into a useful package. Transit map representations, known more generally as graph or network representations, are especially good at revealing the big picture, although their applications are numerous and can be found these days just about anywhere you find large datasets.

All Aboard

Organic synthesis has proven itself an excellent fit for graph modeling, depiction, and analysis approaches. For example, in 2012 a team led by Bartosz Grzybowski showed how graph analysis of the entire set of known reactions could be used to predict then-unknown one-pot reactions. This research was used in building a software system called Chematica:

The Places You'll Go

More recently, a study from John Proudfoot at Boehringer Ingelheim described a graph-based visualization system for multistep organic syntheses. Focusing on a single target, strychnine, Proudfoot asks what can be learned by depicting synthetic routes uniformly, as networks.

As a relatively simple case, consider Woodward's synthesis:

This representation makes it easy to spot some important overall features of the synthesis: (1) it's highly linear; (2) advanced intermediate "XLV" was mainly produced from multistep degradation of morphine itself ("relay synthesis"); and (3) melting point determination was used extensively as a characterization method.

What can be learned by comparing synthetic routes from multiple labs? Between 1954 and 2012, several total syntheses of strychnine were reported. One of the more useful visualizations lays out each synthesis such that common intermediates share common nodes (circles) and are joined by a dark black line:

Each team took a significantly different initial approach to strychnine, leading to a handful of common intermediates near the completion of each synthesis. If this were the transit map of a city, visitors would likely notice a distinct trend toward suburban sprawl.

Clearly, graph-centered perspectives can lead to fundamentally new insights not readily available with other methods of data analysis. A few possibilities are discussed in the paper.

It's not too hard to imagine how similar analyses applied to other molecular targets (possibly incorporating biosynthetic routes, patent data, or other information) might yield many fruitful avenues for exploration. I wonder how many important discoveries in organic synthesis, hiding in plain sight, might be uncovered through a focused program of graph visualization and automated analysis.

Why the Subway Smells

Proudfoot's study made extensive use Cytoscape, an open source software platform for graph visualization. As a result of Cytoscape's biology focus, relatively little tooling exists for chemistry applications such as those described in the Proudfoot paper. As noted in the conclusion:

Although one goal of this exercise was to develop a process for network representations of reaction sequences that is less time consuming than those reported earlier, we cannot categorically state that this has been achieved. The layouts shown above required sustantial modification of the defaults available in Cytoscape in order to achieve representations reflective of the style of standard reaction sequences.

Probably far more time consuming, however, was the manual compilation of reaction data from the primary literature. These data exist as semi-structured tables and narratives often scattered between supporting information sections and the main paper.

One of the more generally useful outcomes of the study was the identification of a set of reaction properties that, when combined for each step in synthetic sequence, could be used to generate network diagrams. These properties are defined in the raw dataset.

Still, I shudder to think at the time it must have taken to compile and check that dataset in the first place. In what can only be described as an idea whose time has come, Proudfoot hints at what might be possible if chemistry could standardize its reporting of reaction scheme data:

Because reaction maps such as those represented here require a highly curated basis set, perhaps a standard format for reporting reaction information should be adopted. For example, a format analogous to Table 1 with the addition of fields for reactant and product structure (INCHI or SMILES representation) applied in a consistent way could allow the automated combination of reaction information across different sources without the need for manual data extraction and extensive curation.

Amen.

Free Access to ACS Publications and Why You Can't Have It (Yet)

April 17, 2013

Although not widely-known, the American Chemical Society permits unlimited, free article downloads across its vast journal lineup without a subscription thanks to the Articles on Request program (AoR). Of course there is a catch, explained below, but the official ACS Publications policy is that a significant portion of its collection can in principle be downloaded by anyone for free.

The only problem is that despite the longstanding official position of ACS Publications, its network hasn't been granting access to content eligible for free download under the AoR program. Read on for more about my ongoing efforts to get to the bottom of this issue.

That's Right - Free Access to ACS Publications

As early as 2008, ACS Publications has had a policy in place of allowing unlimited free access to certain journal content. As reported by ACS LiveWire in 2008:

For example, ACS Articles on Request enables authors to freely distribute their articles in the forum of their choosing. Dana Roth, Chemistry Librarian at Caltech, describes the program in a nutshell: “In a sense, the ACS is providing 'open access' by providing authors with a unique URL that allows a specific number of (free) downloads during the first year after publication and an unlimited number of downloads after the first year.” Authors receive this unique link once their article is accepted for publication. They are free to post it where they like (personal homepages, institutional websites, etc.), and distribute it via e-mail as broadly as they wish. The first 50 downloads from users at non-subscribing institutions, made within 12 months of the article's publication, can be accessed at no charge. After 12 months, ALL downloads via that link are free. [emphasis added]

Here's the Catch

Two requirements must be met to download articles through AoR:

  1. A special AoR link needs to have been published by the corresponding author. An example of such a link is: http://pubs.acs.org/articlesonrequest/AOR-MhutKByEQwq6x9Ih7w52. Notice how this link differs from the normal link format (http://pubs.acs.org/doi/abs/10.1021/jp2047085). Also notice how it's not possible to guess the AoR link from the normal link.
  2. The article must have been published 12 months ago or more. The offical cap during the first 12 months is 50 downloads, although it's not clear whether this limit is actually enforced. In other words, depending on the number of remaining downloads in the initial 12-month period, it may be possible to download any ACS paper at any time, provided that an AoR link is known.

First Problem: Lack of AoR Documentation for Authors

Although ACS does provide a mechanism to download publications for free, the need for special AoR links makes the feature exceedingly difficult to test in practice. Two months ago a reader reminded me about the unlimited download feature, so I decided to test it.

Remembering that I had served as the corresponding author on a J. Med. Chem. paper in 2003, I wanted to locate the corresponding AoR link to test it. My write-up describes the remarkable lack of author-focused documentation I ran into. I'm still not sure if it's even possible for me to recover the AoR link to my papers. However, based on some newer information, I will be making another attempt and describing what I find here.

Whatever else may have been neglected in implementing AoR, better clarity around exactly how corresponding authors obtain AoR links to older papers is desperately needed.

Second Problem: AoR Links are Broken

Although I was unable to locate the AoR link to my own paper, Geoff Hutchison was kind enough to supply a few of his own as was Daniel Lowe. More recently, Alain Borel posted one of his AoR links as well.

A previous post summarized the AoR links I had at the time and my failure in each case to retrieve the promised article. This is a complete list of the AoR links to articles more than 12 months old that I have so far collected:

Following any of these link at the time of writing resulted in the same problem described previously: no article can be downloaded. The one exception is the last; I currently have a subscription to the journal it appears in and so am not able to verify whether the AoR feature works with it or not.

ACS members may have an option to download an AoR paper using the "Universal Access Benefit". This kind of access is to my knowledge not "unlimited" because each member is only allotted 25 such accesses in one year. ACS documentation clearly states the "value" of this benefit ($875), so it's unreasonable to consider access to AoR content through this route as "free".

ACS Publications Response

In response to my last post about the ongoing issues surrounding AoR access, Jonathan Morgan, Director of Digital Strategy & Platform Development at ACS Publications commented:

We have determined that access to ACS Journals content via these unique URLs (which as you may know are distributed to all Corresponding Authors at the time of web publication and made available thereafter on each author’s profile page in the ACS Paragon Plus manuscript submission and peer review system) were affected by a routine software upgrade to our journals delivery platform that was performed in mid-March. While the configuration of the links themselves were not altered, a software code bug in that recent upgrade has been impacting our platform system's ability to deliver content via those links. A code fix has been identified and is currently being validated by our IT staff. Restoring the linking functionality is a high priority for us, and our estimate for completion is on or before next Friday, April 5.

My tests of the AoR links showed no change on April 6. Given that last week was the national ACS Meeting, I postponed following up with ACS Publications until April 15, at which point I emailed Jonathan Morgan and some other contacts I had made at ACS Publications in trying to find a solution to the problem.

I received a response from Brian Crawford, President of the ACS Publications Division on the same day:

I am sorry that it has taken longer than we anticipated to fully resolve this situation. We are working together with our software vendor to address the problem, which is indeed related to the upgrade that was implemented in March. When the fix is in place, we will confirm to you, and will plan also to make a broader announcement to inform other of our readers who might have experienced similar difficulties.

ACS has confirmed that the broken AoR links issue discussed here is in fact the result of an ACS software problem. The cause and solution were identified quickly and prompt resolution was estimated. Several days after that date, ACS stated that the problem is ongoing with no estimate of when it will be resolved.

Other Attempts

In the last few posts about AoR I've encouraged readers to try the service and let me know what they found. A few have graciously obliged.

One confirmation came from @onesleepynerd:

As stated previously, download via Universal Access is not the behavior that should be seen, so onesleepynerd's observation would appear to be confirmation.

Another confirmation came from Daniel Lowe, who unfortunately used a link to a paper in a journal to which I subscribe. For this reason I can't test the link:

I just tried the AoR link on my site ( http://pubs.acs.org/articleson... ) and it also didn't work when I tested it with a colleague's ACS account. Looks like it's broken.

However, more recently Alain Borel reported successfully viewing an AoR link for which he was corresponding author. My attempts to reproduce this observation using the same browser and operating system Alain used were unsuccessful. It should also be noted that the ACS has confirmed the issue I reported now twice. The reason for Alain's differing results aren't yet clear.

The Elephant in the Room

Given everything discussed on the topic to date, a reasonable question might be: Did the AoR unlimited download feature for content older than 12 months ever work in the first place?

It's a little surprising that an issue first described on March 6 was attributed to a software update that occurred in "mid-March". Having written a bit of software myself, I know that it can be frustratingly difficult to pinpoint the exact cause of a bug - and that initial diagnoses are sometimes incorrect. This state of affairs can continue for some time until the rabbit hole has been fully mapped - stretching the completion time beyond all initial expectations.

The scarcity of good author-focused documentation on AoR coupled with the need to use specially-coded, non-guessable links to actually test the feature leads me to wonder how many people have taken advantage of the AoR unlimited download feature in the last five years and whether or not it actually worked.

This is where you can help.

If you've served as the corresponding author for an ACS Publication and have published or given an AoR link to a colleague, were you able to get the AoR unlimited free download feature to work and if so, when?

A Third Failed Test of ACS Articles on Request and How to Help

March 27, 2013

A previous post detailed my failed attempt to test a rather interesting if not well-known program offered by ACS to corresponding publication authors: "free access" to final published articles after 12 months under the Articles on Request (AoR) program.

In that post I made a mistake that I'll attempt to correct here. Although it doesn't affect my conclusion - that the AoR unlimited "free access" provision doesn't seem to work after 12 months - my mistake does affect those who are trying to repeat my observations.

My mistake was not checking the publication date for the AoR article links I tested. The articles were less than one year old and so are still under the 50-download quota, which is not the part of AoR that's under discussion.

The AoR links that need to be tested instead are for papers more than one year old. From Geoff Hutchison's page, the only candidates are:

Using the first link as an example, on Chrome 26.0.1410.40 beta under OS X Lion I was redirected to the following page when logged in with my ACS ID:

ACS AOR Paywall

Clicking on the "Hi-Res PDF" link reloaded the paywall page. Logging out of my ACS account and trying the link again sent me to a login screen. After logging in I was directed to another paywall page.

Running the same test with Internet Explorer 9 under Windows Vista redirected to a different page, but with a paywall firmly in place nevertheless:

ACS AOR Paywall IE

There are three simple things you could do to help me get to the bottom of this mystery:

  • Click on one of the three J. Phys. Chem. C links above from a computer without access to that journal's content.
  • Let me know if you were able to get to the full article content and whether you had to use an ACS ID to get it.
  • Send me an AoR link to a paper published more than one year ago in a journal other than J. Phys. Chem. C to test.

The ACS Articles on Request "free access" provision didn't work as advertised in my hands for papers published more than 12 months ago. I'm sorry for any confusion resulting from my previous post and hope that the information contained here offers a better basis for discussion.

A Second Failed Test of ACS Articles on Request

March 26, 2013

Please see the revised post, A Third Failed Test of ACS Articles on Request and How to Help. Although the introduction, conclusions and comments section of the post below may be useful, the links and discussion around them should be disregarded.

fordA recent post described the lack of public author-oriented documentation for the ACS Articles on Request (AoR) program. For the unfamiliar, AoR is supposed to provide free access to any article published in an ACS Publication:

... the ACS Articles on Request service allows an author's article to be accessed without restriction 12 months after web publication. ACS authors may e-mail or post the Articles on Request URL on their website in distributing up to 50 free e-prints of their final published articles to interested colleagues. Under this service, the access restriction will be lifted at 12 months, allowing free access to such articles via those same author-directed links.

The author may distribute a link to the final version of the article at his or her own discretion. The link directs readers to the PDF version of the article on the ACS website. Those users who already have subscription access privileges obtain seamless IP-based access. Such access is not metered.

The very existence of this program appears to not be widely known. ACS has been staunchly opposed to almost all attempts to loosen its grip on community-created content over the years, so AoR sounds like a fine idea.

The only problem is it doesn't seem to work as advertised. Consider this comment from Benjamin at Pipeline:

Do you manage to get full text through this link ? http://pubs.acs.org/articlesonrequest/AOR-ARCeAEVJYDQ9wxxS3R7C

If yes, the link is in ACS Paragon under 'ACS Articles on Request' in 'recently published papers'.

Keep me posted, if it works, you can just put a link on a public webpage and 50 clicks is quite a lot in my opinion.

Let's try:

Failed attempt to read AoR article

The link worked perfectly, directing me to a valid ACS publication title page. You can see I'm logged in with my ACS ID. But instead of access to the PDF, the usual paywall is shown.

In the comments section to my original post, Geoff Hutchison was kind enough to supply some AoD links on his online bio. Let's try item #34 on the list, a 2012 paper in J. Phys. Chem. C:

Another failed attempt to read AoR article

Nothing but paywall. It's the same story with all of the AoR links - save one, a 2013 publication in J. Phys. Chem. Lett. However, this paper was published less that one year ago and so has not entered the 'without restriction' access phase.

To summarize, my tests show that the ACS Articles on Request service appears to be broken, badly. In a test from two different journals with articles more than 12 months old the 'access restriction' was in full force. This result stands in stark contrast to the ACS promises to authors on its website.

If you've tried these AoR links or others, I'd be very interested in knowing the outcome.