Article - Issue 14, November 2002
Risk, adventure and the tyranny of peer review
John O’Reilly FREng
Peer review was introduced in around 1690 as a means of vetting contributions to the Royal Society (of London). It has served science well, a widely held view being that while it may not be perfect it is nonetheless far better than anything else we have been able to devise. And yet, paradoxically, it may be a ‘dead hand’ – a suggestion sometimes captured in the phrase ‘the tyranny of the majority’ or more provocatively, perhaps, ‘the tyranny of peer review’. Here I explore the suggestion that peer review, or at least conventional peer review as commonly operated by Research Councils and similar funding bodies, may be unduly risk-averse.
In science, peer review is used for a number of purposes, including:
to assess the (relative) merits of research proposals, as a basis for deciding the allocation of research funds
to assess the validity and merits of papers reporting research outcomes and to judge suitability for publication in peer reviewed, archival journals and the like.
But is peer review risk-averse? We might define high-risk research as:
research activity where there is a considerable amount of uncertainty about the success, because completely new questions are being asked, new methods are used or ‘new’ people are involved.1
While not everyone would accept all aspects of this definition, nor accept it as comprehensive, it provides a starting point. It has also been suggested that concentrating on the mainstream is in itself risky – so paradoxically we may need high-risk research to reduce risk!
Some illustrative examples of ‘difficult’ cases may aid development of the argument.
Einstein’s attempts at developing a ‘unified’ theory.
Einstein maintained that he was able to commit himself to this over so many years because, in his position, he could afford to do so: others needed to obtain results, while he could afford the risk of these not coming.
Maiman’s attempts to publish the first experimental demonstration of the laser were initially unsuccessful. While ruby had been identified by Townes and Schawlow as a potential candidate material, I have heard it said that a subsequent study within Bell Laboratories had indicated that it did not ‘in fact’ provide a good basis for laser action. Whatever the circumstances, Maiman’s paper was initially rejected by Physics Review Letters – without being sent out for external review – and so the first publication was in Nature.2
High temperature superconductors
How would a research proposal suggesting such a dramatic advance as that of Bednorz and Muller 3 have fared in peer review? Contrast that with the scramble to replicate the findings when it emerged and the flurry of activity exploring other materials via research proposals readily then supported in peer review. Less than two years after the paper was published the authors received the Nobel Prize for Physics for this work. In the context of the present discussion, it is interesting to note that the citation includes the phrase:
There was much excitement some years ago … but perhaps we won’t dwell on that!
The World Wide Web
The original paper by Tim Berners-Lee was rejected as not sufficiently significant!
Sonoluminescence from bubbles
There has been the suggestion that fusion occurs. While a recent paper now indicates otherwise,4 it is interesting to speculate how a proposal for fusion research in this area might have fared in peer review.
On the science fiction agenda since H.G. Wells introduced it, this has recently attracted significant public attention with at least some apparent degree of scientific support.5 How might a research council deal with such a proposal?
This is currently a contentious topic. While it is certainly a radical idea, understandably one which many find challenging to the point of simple refutation, there are well established physical principles around which to structure arguments. And here, indeed, there is certainly an ongoing, healthy, open, scientific debate.6
Hopefully this list, while but a partial, random sample, suffices to give credence to the argument that we may need to pay special, perhaps different, attention to certain types of research proposal that one might designate as:
swimming against the tide
flying in the face of conventional wisdom
A prime example of the last of these, in its time, was the idea that the Earth rotates around the Sun. There is room, surely, for proposals characterised by something along the lines::
I think there is perhaps a 20% (say) chance of this being a success (or proving correct, etc.), but if it is then the implications are very great indeed.
This then translates into ‘high-risk, high (potential) return’, a sound basis for inclusion in an investment portfolio. Since research councils invest in research it is right to explore how to incorporate such research as a fraction of the total research investment portfolio.
From risk to opportunity
We are familiar with risk management in business, in commercial projects, etc. Businesses have ‘risk registers’ and regularly review these. But I want to advocate an alternative, complementary but markedly more positive attitude to risk. I stress that I am not arguing against risk assessment as an important aspect of project management, but having identified the high-risk areas what then do we do? We should be looking for opportunities. It is often in the areas of highest risk that the greatest opportunities are to be found – opportunities, that is, to impact the success, timeliness, cost-effectiveness, etc. of a project. So, when engaged in risk management, I am alert to possible opportunities, referred to positively as ‘opportunity management’.7 I suppose this has something in common with the idea of ‘lateral thinking’, albeit perhaps stated rather differently. It may not be mainstream in project management circles but I have found it effective.
And where does this link to our discussion? The assertion:
the areas of highest risk are often where the greatest opportunities are to be found
leads to something that is well established in the financial investment arena as:
high-risk, high-return investments.
And this, I suggest, affords legitimacy to research proposals characterised by:
high-risk/uncertainty in the proposal but high impact if successful.
So far so good, perhaps, but there remains the thorny issue of ‘the tyranny of peer review’ to address!
An ‘adventurous research’ fund
It would be nice to have confidence that the established system would nurture the kind of proposals I am alluding to, but the evidence – admittedly often only anecdotal – is not encouraging in this respect. Accordingly the UK’s Engineering & Physical Sciences Research Council (EPSRC) has set up a modest fund for especially adventurous (multidisciplinary) research proposals. For my part I wanted to call this a ‘risk’ fund but my finance colleagues advised that our masters in the Office of Science & Technology and Her Majesty’s Treasury might not like the implication that we were taking risks – gambling if you will – with public funds! So we have used the term ‘adventurous’, which is nowhere near as good, since research by its very nature is an adventure. I much prefer the ‘investment portfolio’ perspective and we view this issue in that light.
But what of the fund – and what of peer review? While it is an incomplete judgment I conjecture that it is with proposals that don’t fit easily within established disciplines that the peer review ‘establishment’ has greatest difficulty. After all, excellence in peer review requires excellent peer reviewers. Just who are the ‘peers’ appropriate to judge anti-establishment, or rather extra-establishment, research? EPSRC is currently engaged in what can be viewed as a small-scale experiment – at the level of between 1% and 2% of the total available research funds. To this end we have:
set up a separate ‘adventurous’ interdisciplinary research fund
issued a call for outline proposals
adopted a ‘free-form’ arrangement for reviewers to respond rather than a tightly structured referees’ report form, eschewing ‘tick-boxes’ and the like that might engender conservatism in the refereeing process
established a special, separate panel to screen submissions so that these ‘adventurous’ proposals compete with one another but not with conventional proposals
suggested that referees, to some extent at least, suspend disbelief, looking in particular for high potential return or impact as a factor to justify research that might otherwise be considered too speculative, too uncertain, too risky!
The fund we have established I view as a minimum investment level; if the review of proposals identifies a wealth of potential ‘rich pickings’ then I shall encourage the managers of our established programmes to join in and provide additional co-funding. We have been awaiting with interest – bated breath even – the response to our call for ‘adventurous, multidisciplinary research’, confident in the belief that:
it is often in the areas of highest risk that the greatest opportunities are to be found.
I am greatly heartened to note that more than 600 outline proposals have been received, a very positive indicator for the spirit of adventure amongst engineering and physical sciences researchers in the UK. Whilst EPSRC has set up this fund and adopted distinctive peer-review arrangements, the hope for funding agencies must surely be that over time such special measures will become unnecessary – that peer review will be willing and able to embrace adventurous, high-risk, high potential impact proposals. A quotation8 over the entrance to the club house of the Yankees in New York captures the sentiment of this:
Do not follow where the path may lead; go instead where there is no path and leave a trail.
Or more pointedly:
New horizons are never discovered by following old roads.
Everyone who goes somewhere new leaves a trail that others can follow (highways were tracks once upon a time) but time erases the traces if the adventure remains a secret: a warning of the suppressive danger of undue conservatism in peer review!
What I have said here should not be interpreted as an attack on peer review as such. Rather it is a caution: care must be exercised to ensure that the peer review process used for assessing research proposals is not unduly conservative or risk-averse – and is not thought to be so by researchers. If it is it may fail to support the development of a balanced research investment portfolio, including an appropriate proportion of ‘high-risk, high-return’ activities. Even if it is merely thought to be so it may have the same effect by discouraging the inclusion of ‘high-risk, high-return’ activities in the research proposals submitted for consideration.
In assembling this contribution I have drawn freely on discussions with colleagues in my own and similar organisations, which brings to mind the quotation:
Copying from one source is plagiarism, copying from many is research
In this sense at least this paper has been widely researched!
Prof John O’Reilly FREng
Chief Executive, Engineering And Physical Sciences Research Council