A New Model for Interdisciplinary Peer Review
In June 2005 I wrote out some ideas on improving interdisciplinary peer review, in the form of a mock proposal for creating an interdisciplinary journal (referred to as “New Biologist” in the text below). I later condensed this into a very brief piece in Nature Online’s peer review focus. Recently, Russ Altman asked me to post the longer, detailed proposal so he could refer to it on his blog. Here it is.
Goals
New Biologist will create a home for high-impact interdisciplinary research in the biological sciences, which is ill-served by existing publishing and peer review models. It will do so by focusing on solving the problems of interdisciplinary peer review, and of targeting (and in some cases creating) interdisciplinary audiences. We will provide a new channel for creative, path-breaking work that crosses boundaries and does not easily fit any single discipline. We will pioneer new models of interdisciplinary peer review, and the creation of “virtual journals” that can serve as community centers for new audiences forming at the intersection of multiple disciplines. For all of these goals, we will make use of new technologies for rapid interaction and community building based on the Internet.
The Market Need: Interdisciplinary Publishing
What is Interdisciplinary Publishing?
When a single scientific study creates a new combination of multiple fields, it’s interdisciplinary. Common patterns include applying the methods of one field to the data and goals of another field, or thinking about a problem in a new way by drawing on concepts from several fields.
Such “boundary crossing” is nothing new. Indeed, historically most existing “fields” were originally created by such recombination (for example, molecular biology arose in part from an influx of physicists to the study of genetics). However, the key, early phases when the work is perceived as “crossing boundaries” are especially difficult. Initially only a handful of pioneers are willing or able to cross the lines of existing disciplines, to assemble all of the tools needed for a brand-new field. The majority of researchers lack the diverse knowledge to assemble all these tools, or the spirit of innovation and experimentation required to make it a success. Typically, the innovators face an uphill battle for understanding and acceptance of their new ideas. Over time, as more researchers learn the new approach and obtain successful results, it becomes a recognized field, with its own audience, leaders, standards of quality, and publication channels (such as journals and conferences). Once it becomes a “field” with its own power structure and norms, most of its publications no longer are truly interdisciplinary (even though they still use the same combination of math and biology, say, that the original papers developed), in that they do not cross any new boundaries (they remain on the home turf carved out by the early leaders of the field).
I choose to define “interdisciplinary publishing” by this criterion of crossing new boundaries. What matters is not the name that you give to what you do (“this is math, and that’s biology”) but rather the entrepreneurial spirit of trying a new combination because it might yield discoveries. The distinction is simple: whereas the majority of researchers are defined by their field (and follow the limited patterns of their field), occasionally someone steps beyond the boundaries of the field—and in so doing often creates the foundation of a future field.
A necessary corollary of such interdisciplinary work is creativity. Genuine creativity (even if it stays within a single field) often faces similar difficulties as boundary-crossing work. Often they go hand in hand. New Biologist will provide a channel for both interdisciplinary work and creative work that tend to be blocked by traditional peer review.
The challenges of interdisciplinary publishing are not limited simply to problems of peer review (the resistance of the establishment). Equally important is the problem of finding an audience. Since existing journals follow the boundaries of existing disciplines, there is no journal that will reach the appropriate audience for an interdisciplinary paper. Typically such an audience consists of a subset of people from many fields, who are interested in the specific intersection of fields that the paper brings together. Publishing in a standard journal is a poor fit on many levels: since the real interest of the paper goes outside the field, readers of the journal will mostly not be interested; most of the audience who would be interested don’t read this journal; the paper’s creativity will be buried in a sea of un-creative papers in the journal, so people who want to read creative work again will have difficulty finding it. Interdisciplinary publishing must solve the problem of finding the widely scattered audience for each paper, at the intersections of different fields, and make it easy for such communities to form around a new discovery and even nucleate the creation of a “new field”.
The Need for Interdisciplinary Publishing
Throughout the marketplace, there are many forces pushing the growth of interdisciplinary research. Many of the most important new fields in science are interdisciplinary almost by definition (e.g. bioinformatics and –omics style biology). The growing importance of computation and massive datasets in many areas of science inevitably creates a need for hybrid approaches combining mathematical, computational, and experimental approaches.
This need has been recognized at the highest level by funding agencies such as NIH. NIH’s Roadmap Initiatives stress the need for specific types of interdisciplinary research (such as “computational biology”). And NIH’s “Teams of the Future” directly mandates that individual research projects be interdisciplinary, by combining experts from several areas in each research team. This movement away from single-investigator projects to multi-disciplinary projects extends beyond NIH, and reflects a recognition (at least in theory) that the most valuable discoveries (and new fields) are likely to come from interdisciplinary research.
Universities are also driving this trend, by directing resources into multidisciplinary initiatives (such as BioX at Stanford, or the CNSI here at UCLA). Even in traditional departments (such as the UCLA Biochemistry division of which I am a member), there is a clear sense that new faculty hires should bring in “new approaches” that cross disciplinary boundaries.
An obvious question is: where are all these “teams of the future” going to publish? To the extent that their work simply fits an existing field, they can publish in an existing journal. But if they go outside these boundaries, not only will new journals have to be created, but also new models of interdisciplinary peer review and publishing. The existing peer review process used universally by all journals just doesn’t work for interdisciplinary papers. These problems are soluble, and are going to be the foundation for an important new category of publishing: interdisciplinary journals.
The Interdisciplinary Peer Review Problem
Sometimes bad things happen to good papers. But if a paper is interdisciplinary, it’s more likely to encounter these classic obstacles:
Multiple audiences, multiple misunderstandings
• The paper is assigned referees from very different fields, including fields other than the authors’.
• The referees make contradictory criticisms and demands reflecting the divergent expectations and assumptions of their different fields. If you satisfy Referee A’s demands, you aggravate Referee B, etc.
Crossing disciplinary boundaries makes referees err on the side of conservativism
• Referees outside its work-area / target audience give little consideration to the potential value of publishing the work (i.e. impact; since it isn’t visible to them), and instead focus on validation questions (often highly speculative, arising from their lack of familiarity with its area).
• Referees outside its work-area lack knowledge of its existing literature, and give little consideration to whether its work is demonstrably an advance over previous work. Instead of addressing this question—the true gold standard for all publication decisions—they often retreat to a theoretical question of “confidence”: is there something they can criticize as “not completely proved”?
The “Creativity Tax”
If an approach is truly novel (new methodology; new questions; or a new combination of methods and questions), all of the above problems just get worse. And you add some new obstacles:
• Peer review in the absence of any true “peers”: peer review works best when reviewers are themselves experienced at doing exactly what the paper describes. But if its work is novel, by definition there is no one else who has already done it, who could act as a peer reviewer.
• The challenger’s “burden of proof”: novel approaches, by definition, challenge existing assumptions. This fundamentally shifts the burden of proof in most reviewers’ minds. Research that follows the assumptions of an existing field and is reviewed by referees who share those assumptions can be evaluated against the standard tests used by well-known papers in the field. Such research will be treated as “innocent until proven guilty”, in that it can just be checked for a clear violation of the short list of well-known tests. By contrast, research that challenges existing assumptions starts out with such a “violation” and must justify this heresy, typically by providing statistical “proof” that it is necessary. In other words, reviewers treat such work as “guilty until proven innocent”. For all the reasons outlined in the previous section, the concept of “proof” can be very slippery in interdisciplinary peer review.
• The multi-step problem: most published research consists of incremental steps taken one at a time. Most authors would agree that it is poor strategy to combine multiple “steps” in one scientific paper, since this increases the likelihood that at least one aspect will draw criticism. Creative breakthroughs, by contrast, often require multiple steps. (For example, moving from Euclidean geometry to hyperbolic geometry requires rethinking many conventional expectations). Unless all of these steps are articulated together, the picture will not only be incomplete, but actually may make no sense at all, and provide no useful new capabilities.
However, the insightful (or lucky) researcher who discovers the complete combination of steps necessary for this breakthrough, now faces a very difficult publishing problem. Getting reviewers to understand (and accept) many new concepts, each of which challenges existing assumptions, will be difficult in a single paper. On the other hand, breaking these steps into separate papers would also face an uphill battle, because the individual steps only produce valuable results when combined.
In terms of the educational process necessary for building a new field, the right approach is to publish a broad overview that explains the new principles simply, illustrated by examples of how the complete theory provides useful new capabilities. Readers intrigued by these new ideas and results would then be motivated to study subsequent in-depth papers deriving each individual step in a rigorous way.
However, this path is almost completely blocked in conventional peer review by the “guilty until proven innocent” problem: nothing can be published until everything is “proved”. This catch-22 is fundamental: each referee feels he must be confident about everything in a paper, but given the referee’s lack of one or more relevant areas of knowledge there is no way he could understand (let alone feel confidence about) everything in the paper.
Principles for Interdisciplinary Peer Review
Like many scientists who have experienced the deficiencies of the existing peer review system, I have generally felt that it is “the worst system except all the others”. However, I recently have seen some specific principles that can greatly improve interdisciplinary peer review:
Informal Conversation
I gave a talk at a recent conference about some work that combined several fields, and received skeptical questions from several audience members. I had a conversation with two of these “critics”, which continued by email after the conference. Whereas formal peer review of this work by a journal yielded only vague and simplistic criticisms, this conversation gradually identified specific problems that could be tested, and led to a much greater depth of understanding of the issues, based on the combined expertise of both “sides”. Indeed this conversation itself turned into a publication. I have had this same experience many times: in cases where interdisciplinary peer review was officious, simplistic and driven mainly by mutual misunderstanding, informal conversation with the same critics was substantive, penetrated deeper into the important questions by combining the expertise of both “sides”, and produced a larger understanding than either side could have reached alone.
Standard peer review works best when the reviewers have done the same kind of work as described in the paper, can understand it without having to ask basic questions, and thus can directly evaluate it. In interdisciplinary peer review, by contrast, a paper might combine three different fields, and each referee might only understand a third of the paper. In this case the referee cannot possibly understand the paper (let alone evaluate it) without a conversation, in which the referee asks questions about parts he doesn’t understand, or parts that seem to go against his assumptions, understands the authors’ answers, asks more questions etc. Only when this conversation is complete can peer review occur: it would be premature to render judgment on material one does not fully understand. In informal conversation, most people instinctively follow this “golden rule”. They don’t make up their minds before a conversation. Instead they ask questions: “I didn’t understand what you meant here, could you explain that?”, or “Your proposal of XYZ appears to violate the W principle, how do you account for that?” Only through this extended conversation will they gradually conclude they understand the authors’ point of view, and decide whether it is sound. Interdisciplinary peer review requires this gradual synthesis of the different types of expertise of the authors and referees, so that everyone arrives at an overall understanding of the work, and then can evaluate it.
Unfortunately, many aspects of standard peer review get in the way of this natural human process, in ways that are devastating for interdisciplinary work. The existing system casts the referee as the Ultimate Authority (prosecutor, judge and jury rolled into one), instead of what he really is: a peer, who knows no more (and quite possibly much less) than the authors do about their subject. This artificial role discourages the referee from admitting he lacks basic knowledge of one or more fields in the paper, and from asking the questions he needs to understand. Most seriously, by requiring the referee to render a judgment before he can ask any questions, it forces him to stake a position based on his ignorance (often driven mainly by his discomfort with work outside his field), instead of pursuing the necessary conversation and keeping his mind open. This position, once stated, drives the dynamic away from informal “conversation” (where the goal is mutual understanding) towards a legalistic pattern of “attack and defend” (where the goal is the opposite of synthesis: to assert one point of view by rejecting other points of view). In normal peer review, synthesis may not be needed (because each referee does sufficiently similar work to understand it fully), but for interdisciplinary work this is a fatal flaw.
In the Internet age there is no reason why peer review cannot incorporate more “conversation” between authors and referees, prior to the referees’ evaluations.
Separate Evaluation of Impact (Audience) from Technical Assessment
Standard peer review depends on two or three referees who each evaluate both the work’s interest and technical soundness. This makes no sense for an interdisciplinary paper, whose results (interest) typically lie in a different field than its methodology (technical soundness). A potential reviewer from the paper’s target audience (interest) will likely feel he cannot review the paper because he can’t evaluate its technical soundness. This immediately creates a bias against interdisciplinary papers relative to normal papers (whose intended audience and technical reviewers are one and the same). By contrast, reviewers who are comfortable with assessing the paper’s methodology often have little knowledge about the area of its results. Since they aren’t the paper’s intended audience, they are likely to criticize the paper as lacking significance (e.g. this work is of no interest to me) or lacking novelty (e.g. the methodology by itself is not very novel).
An obvious solution is to separate the evaluation of Impact (novelty and interest to the target audience), from technical assessment, by using two different sets of referees. Actually, this is analogous to how some journals conduct an initial, in-house screen to select papers that seem to have high impact, prior to sending them for outside review. The difference is that now we do not consider the methodology reviewers to be “experts” on the paper’s significance and interest. A corollary is that we’ll need more than 2-3 referees, which makes sense for interdisciplinary peer review: you cannot give a broad paper a narrow review, by people from just one area.
Subdivide Results and Interpretations
A common problem in interdisciplinary peer review is the “all or nothing” catch-22: “you can’t publish any of your data, because referee 3 doesn’t like one of your interpretations”. But if no experimentalists are ever to be permitted to see the data, and at least hear about the interpretations as speculative hypotheses, how can they ever be tested experimentally? While this might seem like an odd thing to get killed for (after all, it would be easy to ask the authors just to attach qualifications to that specific interpretation), interdisciplinary papers get rejected all the time for reasons like this.
To reduce this problem, the review process should explicitly separate results (data) from interpretations (hypotheses that the authors propose based on their data). It may be appropriate to ask authors to give a list of their individual results and interpretations at submission, and ask referees to indicate which result or interpretation they are questioning.
New Ideas Merit Consideration by the Community
New ideas are normally treated as “guilty until proven innocent”, while incremental work is treated as “innocent until proven guilty”. This seems backwards to me, at least for interdisciplinary work. Work in a mature field, that lacks real novelty, should be held to a higher standard (a grown man should be able to defend himself in a fight). Genuinely new ideas are few and far between, and are valuable contributions even if they are not entirely proven by data at inception (a newborn baby cannot be held to the same expectations as a grown man). Work that lacks new ideas should meet the Confident criterion; whereas new ideas should meet the Preponderance of Evidence criterion.
Three Phase Peer Review Process
To address these problems, New Biologist will pioneer a three phase peer review process:
1. Audience / Impact: can we find an audience for whom this paper has very high interest and novelty? Unlike standard peer review, we can’t just send the paper to one field. The authors should identify referees from multiple areas whom they think might have high interest. This phase should be very rapid and “easy” on referees: unless just seeing the title arouses strong interest in some definable target audience, reject. This phase should rapidly eliminate the vast majority of papers, which are incremental and not fundamentally innovative.
2. Questions & Answers: we identify technical assessment referees and instruct them to ask all the questions they have, but withhold judgment until they’ve heard all the answers. That is, both uncertainties and concerns must be raised as questions, not snap judgments, to begin a conversation with the authors aimed at mutual understanding based on the combined expertise of all parties.
3. Technical Assessment: once referees have answers to all their questions, we finally ask them to give a technical judgment of whether the work is an advance, based on explicit comparison with previous literature.
I. Audience / Impact
Based on author and editor supplied email addresses, search for an audience that expresses high interest in this paper, using “web advertising” style metrics like “click-through” rates.
• Send the referee an email: “Because the following manuscript appears to be of interest to your own work, the editor invites you to see whether you find it interesting and novel. If the title looks interesting to you, please click this link to see the abstract and / or full paper. We are not seeking a detailed review or technical assessment now, and viewing the paper in no way obligates you to give any review. <<TITLE>>”
“Otherwise, if this paper is not of real interest to you, please click the following link now so we will not bother you with any more emails. <<LINK>>
“Because we must complete this initial screen within three days, we ask that you click one of these links as soon as possible (ideally, just take a quick look at the paper now to see if you can decide it’s not of strong interest to you), and by <<DAY>> at the latest. If a paper doesn’t receive sufficient interest during this screen, our policy is to reject it. If you don’t have time for this, please click the following link <<LOOKS INTRIGUING BUT I JUST DON’T HAVE TIME>>
“If this paper overlaps your own work closely enough to create a competitive conflict of interest, click here <<CONFLICT>>
“We would be very grateful if you could suggest others who might find this work interesting (within your own group or elsewhere), even if you do assess it. To forward this invitation to a colleague, please click this link. <<FORWARD ME>>”
The title is a hyperlink to a view of the abstract on the journal review center website. The abstract then has link to the intro, results, etc. as separate pages, each of which click <<NEXT>>. Individual pages of the paper have figures which link to large versions of the figures. Measure click-through rates at every stage to see whether that individual will follow all the links; how much time they spend reading each item etc. At each stage (including initial email) give a one-click EXIT link “Not interested in this paper”. Lack of referee interest is itself a review. On each screen during this process, allow referees to type questions they want to ask the authors. This allows them to “get them off their chest” immediately, keep an open mind even about points they’re uncertain about, and get them into the “Q&A” mindset that we want.
• Categorize each referee user by keywords, tags, publications, self-classification. Search multiple audience categories to find an area of high interest. If none found, REJECT. This process should be complete within just a few days.
• For referees who have actually read the paper, now ask a one minute questionnaire covering interest, ideas, results, referee expertise. (see above; each one is just a multiple choice).
• If the paper does not meet high interest / novelty criteria for some identified audience, REJECT.
II. Q & A
The purpose of this phase is to allow referees to ask all the questions they have in a non-judgmental way—a conversation with the authors, and with the other referees—before they even enter the Technical Assessment phase. This need not be very formal (actually, a web forum / wiki seems like a fine model), but should distinguish clearly several types of questions
• False positive: Might result / interpretation X not be statistically significant, or biased? Indicate a specific test for the hypothetical problem
• False negative: is it possible your analysis missed some additional results due to problem Y? Indicate a specific test for the hypothetical problem.
• Overlap: how does your work overlap previous study X <<citation>>, and in what ways is it distinct?
• Clarification / elaboration: I didn’t understand X. Please explain.
• Addition: I suggest that idea X is relevant to your paper <<citation>>. Could that be a useful addition?
Each referee can post as many questions as he wants, and also can “second” other referees’ questions. Authors can immediately answer individual questions, by text or by adding new data / analyses. Referees can ask new questions about these responses and data. It might be useful to organize a real-time discussion, for example via instant messaging. Such discussion is important for synthesis (combining the expertise of all the referees and the authors) and for definitive clarification. It should leave no important question unanswered. No paper should be rejected just because “referee 2 didn’t realize method Y is well established in this field” (referee 1 could have told him so), or because “referee 3 missed the fact that the authors did perform test X on page 14” (the authors will tell him).
Each referee indicates when he has no more questions.
At the beginning of Q&A, we need to give referees clear instructions on Q&A and Technical Assessment. Just knowing our expectations will probably change how they approach the review process in positive ways. These expectations can be communicated both in terms of goals (e.g. we want referees to withhold judgment until all their questions have been answered) and in terms of process (knowing that they’re going to have to supply citations if they want to claim the paper’s “not novel” may make them a bit more careful in what they say).
I think that every question and answer during this process should be saved and visible to all parties. Referees should see other referees’ questions, as well as the authors’ responses. The goal of this process is synthesis, which is best served by bringing everyone’s expertise and perceptions together, in search of a consensus.
III. Technical Assessment
Referees who have direct experience with work similar to that reported in the paper (see Referee expertise, above) now are asked to fill out the very simple Technical Assessment question (see below).
Based on the combined reviews, the editor can accept the paper if it meets the appropriate criteria, or return the referee criticisms to the authors. There are several possible scenarios:
• ACCEPT: paper of high interest, good technical assessment.
• REJECT: if the paper is not of sufficiently high interest, and the technical assessment is negative.
• APPEAL: if the authors can persuade the editor that the referee criticisms are not supported by the evidence, and that the technical assessment really meets the appropriate criteria.
• REPORT: in contrast to a standard article (data + biological hypothesis), in certain cases of high interest and high controversy just about the biological hypothesis, it may be appropriate to resubmit the paper as a “data report” that mainly presents the data, with little interpretation. This way at least the data can be published. The editor can offer the authors the option to resubmit as a Report.
• FORUM: in certain cases of high interest and high controversy, it may be appropriate to publish the paper accompanied by a separate letter stating the criticisms of the referees. This gives the authors the chance to “stake their reputation” on their work if they truly believe in it, and at the same time give the community the full benefit both of the new idea / results and the critiques that must be tested to establish its validity. The editor can offer the authors the option to publish as a Forum paper, and to the referee(s) to publish a Forum critique. This is analogous to the common practice of publishing Letters to the Editor that strongly criticize a paper published in that journal.
Criteria for Interdisciplinary Peer Review
Phase I: Audience / Impact
Interest: (Evaluate broad vs. narrow by comparing responses from different referees.)
o High: I or someone I know will base new work on this paper.
o Medium: I will cite this paper in my next paper in this area.
o Low: I think this paper will be of interest to <<names>>…
o Not of interest in my area
New ideas: concepts (of any kind, including methodological) the paper introduces
o Deep: we will be thinking about this for years.
o Original: this work has given me new ideas, changed how I think about X.
o Incremental: this work includes a novel step of the kind I’d expect many researchers in this field to make in the normal course of their research <<example citations>.
o No: ideas previously published <<citations>>.
New results: the data themselves
o Exciting, surprising: no one else has been looking for this (or has been able to do it despite many efforts). This opens up new directions.
o Interesting, novel: these results represent a significant departure from previous work.
o No: results previously published <<citations>>
New interpretations: the hypotheses that the authors propose based on their data
o Exciting, surprising: no one else has been looking for this. This opens up new directions.
o Interesting, novel: these hypotheses represent a significant departure from previous work.
o No: hypotheses previously published <<citations>>
Referee Expertise: I have published similar analysis <<citations>>
o Using same / very similar / similar / different methodology
o On same / very similar / similar / different dataset
o Addressing same / very similar / similar / different biological question.
Phase III: Technical Assessment
Technical Validity:
o Confident: p<0.05 for null hypotheses
o Preponderance of Evidence: supported by most of the evidence, but not proven beyond a reasonable doubt. That means: the data are clearly not explained by old models; the new model explains the data successfully; but it is not yet possible to disprove every other possible alternative new model. Cite specific p value calculation or contradictory evidence <<citations>>
o False Positive: result contradicted by preponderance of evidence <<citations>>
o False Negative: result / interpretation is incomplete (in a misleading way), based on preponderance of evidence <<citations>>
o Not an advance: already shown by published papers with similar or stronger evidence <<citations>>.
o Outside of my specific expertise / experience:
Referees should be instructed to report different levels of confidence separately for different results. In other words, if they feel more confident about some results than others, they need to report those separately. (The interface should make it easy to report one confidence for the whole paper with one click, but equally easy to select a subset of the results for the first assessment, then automatically be asked about the remaining results / interpretations on a second screen, etc.).
Journal Content
Article Types
• ARTICLE: a standard paper, selected for high impact and novelty.
• REPORT: in contrast to a standard article (data + biological hypothesis), in certain cases of high interest and high controversy just about the biological hypothesis, it may be appropriate to resubmit the paper as a “data report” that mainly presents the data, with little interpretation. This way at least the data can be published.
• FORUM: in certain cases of high interest and high controversy, it may be appropriate to publish the paper accompanied by a separate letter stating the criticisms of the referees. This gives the authors the chance to “stake their reputation” on their work if they truly believe in it, and at the same time give the community the full benefit both of the new idea / results and the critiques that must be tested to establish its validity. The editor can offer the authors the option to publish as a Forum paper, and to the referee(s) to publish a Forum critique. This is analogous to the common practice of publishing Letters to the Editor that strongly criticize a paper published in that journal.
• NEWS & VIEWS: to arouse interest in a particular target audience, News & Views should be published for many papers in New Biologist. This would be invited, often from one of the referees identified in the audience with highest interest for the paper.
• REVIEW: the creation of a new field requires comprehensive summations and synthesis of the disparate elements whose intersection is the nucleus of the new field. New Biologist should publish review articles that perform this function – essentially, the establishment of a new field. This should be strongly linked to the Virtual Journals concept.
• TUTORIAL: a similar goal, but with a focus on a specific concept or methodology rather than a recent literature survey.
• OPINION: exposing an unsolved problem or question in a balanced but speculative way.
• THIS WEEK IN…: it may be useful as part of a Virtual Journal to solicit regular commentary on a field (like John Baez’s This Week’s Finds in Mathematical Physics). This would function more like a blog than like standard article submission. Individual “articles” would not be peer reviewed. Instead the entire series would be reviewed regularly on the basis of reader interest measurements and review criteria: of high interest to a broad set of readers within the defined area (typically a virtual journal); consistently useful and accurate.
Field Focus
New Biologist will be interdisciplinary and thus cannot restrict itself to a single topic. However, particularly for its initial launch, it needs some topics that will drive its success. These topics should have a strong “pressure buildup” of interdisciplinary research that is being held back by standard peer review, and that will be of high impact and interest. The journal’s editorial board must have strong expertise in these areas, so they can efficiently select referees, make independent evaluations, and avoid being “gamed” by authors. Possible topics include:
• Bioinformatics and –omics research in general. The bioinformatics publishing landscape is new and largely immature. Despite the phenomenal growth of the field, no high impact journal emerged as the clear leader. (see Competition)
One major theme is what I call “discovery bioinformatics”—using bioinformatics to answer a biological question. This kind of interdisciplinary research is of great interest and importance to biologists, but fares poorly at existing journals: biology journals have the right audience but are uncomfortable with the methodology; bioinformatics journals have the wrong audience and little interest in answering biological questions. This category is a great vehicle for New Biologist.
• Transcriptome: despite the growth of research on alternative splicing and post-transcriptional regulatory mechanisms such as RNAi, there is no journal that fits the character of this field (high throughput experiments plus bioinformatics). NAR is probably the best journal covering this area, but NAR is moving to cut significantly the fraction of papers it takes, while the field is growing rapidly. That’s a formula for frustration for interdisciplinary authors.
• Using evolution (comparative genomics etc.) to answer biological questions. This is another kind of “discovery bioinformatics” that’s likely to grow in importance as people start to make good use of all the genome data that’s available. Again, there’s a problem of audience vs. methodology.
Virtual Journals
Job One: Helping Interdisciplinary Papers Find Their Audience
Normal academia is organized around units (such as a department or journal) that each correspond to one audience (field). This is an over-simplification of course, but serves as the basis for most academic communication. I.e. a paper gets read by its proper audience by virtue of being published in a journal that matches that audience.
This creates a conundrum for an interdisciplinary paper: how can it reach its true audience? As outlined above, journals matching this audience may recognize its impact but will be perplexed by (and possibly hostile to) its methodology. By contrast, journals that focus on related methodology are unlikely to appreciate its results, and even if they publish it, won’t reach the desired audience. Catch-22! Even worse, the authors may have only a vague idea of their audience. The actual group of researchers who are able to immediately appreciate the value of the paper is likely to be a specific subset of that general audience (often younger investigators, or those with wider methodological knowledge). How to find this audience? A normal journal does little or nothing to help answer this question, either during the review process (to see if the paper has high interest to some audience), or once it’s actually published (by bringing it to the attention of that audience).
Creating a Virtual Journal for a new audience
An obvious way to solve this problem is RSS-style: to re-aggregate papers published in many fields on a single site that can become the “community center” for people who are interested in the new area. Individuals “join” the virtual journal and then can add papers they think are relevant; comment on papers that have been posted. These papers can come from any journal, not just New Biologist. Hosting the virtual journal provides many benefits besides the traffic it attracts. For example you get to measure interest levels on all the click-throughs, so you can develop rigorous ways of measuring impact of your papers vs. those of other journals. But the bottom line is you’re seizing the high ground in the war to create new fields. This process is happening all the time, but existing journals are doing nothing to facilitate it or extract benefit from it. This is a natural activity for an interdisciplinary journal: basically, it’s the sensible way to give the papers you publish a real follow-up, a place where they can build long-term audience. It synergizes with the core activity—publishing interdisciplinary papers. It also synergizes with the other catgories: publishing reviews, News & Views, etc. And it is so easy to do.
Marketing
Attracting Interdisciplinary Authors
One fundamental advantage for this journal is that its customers are essentially a captive audience—interdisciplinary researchers who are being trashed by standard peer review at other journals. We don’t have to compete with Science or even NAR for good papers; they will send them (screaming) to us. We can advertise this to authors very directly (“Sometimes bad things happen to good papers.”). We can suggest they send us their previous reviews even if “bad”. Indeed if the reviews give the paper a positive technical assessment, then all we have to do is do a quick interest screen.
<<HOW WE’RE GOING TO ADVERTISE TO AUTHORS>>
Motivating Referees
All peer review depends on the willingness of unpaid referees to do the journal’s real work. Interdisciplinary peer review is even harder (and requires more work) than normal peer review; that’s one reason why regular journals are unable to do it. Thus a major question is how New Biologist will motivate referees to devote the time necessary for this process. There are two basic strategies: to pre-qualify so that you only ask referees to review papers in which they have a high level of interest; and to make their involvement easy and rewarding.
Pre-qualifying
Submissions are first filtered for impact, in a fast and efficient way. No more struggling to find referees: if no one is interested in the paper, that means we weren’t able to find an audience for the paper, and we’re done. The vast majority of papers (incremental, non-innovative) will be filtered out by the simple fact that no one felt a burning need to read them. We don’t want those kinds of papers—other journals can publish them. We won’t ask our referees to put time into such papers; that cuts the total burden tremendously. Moreover, we’ll only ask a referee to review a paper that they personally are highly interested in. That’s the foundation of our system.
Our major need, then, is methodologically savvy referees who have open minds. One of our best ways to find such people is among our authors. It would make sense to cultivate this connection, and build a loyal group of interdisciplinary researchers who believe strongly in our approach because it has helped them personally. The best place to start is with an editorial board that exemplifies this interdisciplinary spirit.
Rewarding Referees
One principle that may be useful is to give referees both tangible and intangible products in return for their participation. For example, if a paper is published, you could invite them to write a News and Views. This gives them an authorship (tangible) and also sends a signal that they are a leader in the field (writing the intro to a potentially important paper in the field—intangible). If have negative views of a paper the journal has decided to publish, you could invite them to write a Forum “counterpoint”. Again they get an authorship and a prominent place to air their views.
Targeting Interdisciplinary Audiences
The main task of an interdisciplinary journal is to help each paper find its audience, by searching through a variety of possible audiences quickly, measuring their response automatically, and zeroing in on them wherever they may be. This is the number one task for phase I (assessing impact = finding an audience), and the number one task for publishing (bringing the paper to the attention of an audience that exists only as an intersection of several fields, and which does not all read any one journal).
My feeling is that there exists fertile ground for building interdisciplinary audiences around specific topics, and straightforward pathways for achieving this.
• Targeted email (based on relevant citations) for attracting readers to the initial publication of a paper would be easy to do. I.e. “Dear Dr. Smith, we are writing to alert you that the following paper just published in New Biologist cites one of your papers, and appears to be relevant to your work. We invite you to access the paper free online at <<LINK>>. If this paper is of interest to you, we invite you to join our XYZ Virtual Journal focusing on new papers in this area from many sources, and to consider citing this paper in your future work.”
• Greatly expanding the role of News & Views in targeting a specific audience for each paper would also be easy to do.
• Building Virtual Journals when you see a strong audience response around a nucleus of papers, would also be easy to do.
Once you solve the interdisciplinary peer review problem, I think, you release an enormous amount of pent-up, unsatisfied demand. People want these papers, they want to create new interdisciplinary fields, they want to get together with like-minded folks who can teach them new things. If you can release the high-impact boundary-crossing papers from purgatory, following up with the simple marketing ideas above will be a highly leveraged activity.
Competitive Analysis
The competitive landscape for New Biologist is defined above all by two facts:
• With a few exceptions (addressed below), nearly all journals define themselves around a specific discipline. They do not want to be interdisciplinary. And even if an editor decided to become interdisciplinary, he would have to fight the established powerbase in his journal to change the journal’s mission.
• All journals use a standard, universally agreed peer review system that is fundamentally incompatible with interdisciplinary work. Interdisciplinary publishing would require not just minor tweaks, but basic changes to the peer review process (like those I’ve proposed). There are both political and technical barriers to making such changes at an established journal. Only an editor with complete freedom of action, opposed by no pre-existing powerbase, and ample time for building a new, experimental peer review system could make such changes. And that’s exactly what you can’t get at an existing journal.
These facts translate to a favorable market situation now and in the near-term future: there currently is no journal in this space, and significant barriers exist to rapid entry of existing journals into this space.
Competitors
Journals such as Nature and Science give the illusion of interdisciplinary publishing that can reach an interdisciplinary audience. However, the vast majority of papers published in these journals are not interdisciplinary. Instead, you simply find a physics paper next to an evolution paper, next to a molecular biology paper, that happen to be published in the same journal. This publishing model can be best described as “hot topics”: the editors try to guess which fields are hot, and then publish a selection of “top papers” from these fields.
“Top paper” means unanimous acclaim from two or three referees from that field. This tends to disenfranchise papers that are interdisciplinary; they require many more referees than this (from multiple disciplines), and are far more likely to get negative comments than single-field papers (when each referee only understands only a third of the paper the likelihood of referee confusion, misunderstanding or hostility is high). The standard of “unanimity” strongly favors single-field papers.
Finally, Nature’s model of interdisciplinary publishing (“everyone should read this paper”) is fundamentally wrong. Good interdisciplinary papers are not much more likely to be of universal interest than good single-field papers: because they solve a specific problem at the intersection of several fields, they are of interest to a subset of people at that intersection of fields. The purpose of interdisciplinary publishing is to connect the paper to this widely scattered audience: to find them, and bring them together, quite possibly as the nucleus for creating a new “field”. Nature’s model is carpet-bombing; by contrast, what’s really needed is somewhere between Google’s search engine and Yahoo’s Community sites. It’s true that genuinely earth-shattering papers are usually interdisciplinary (e.g. the double helix), but the converse is a poor (and very limiting) model for interdisciplinary publishing as a whole.
Nature, with its family of journals, could launch a new journal targeting our space. They already have the beginnings of a tagging-based community site. The question is how completely and quickly they could adapt to the new peer review process. Without revolutionizing peer review, I think talk of the importance of “interdisciplinary research” is just that—talk.