Square One

I am presently conducting a systematic review and meta-analysis that involves a detailed and critical quality appraisal of studies which has made me realise all over again how impoverished the narrative of the classical scientific paper really is, and that indeed like Francis Crick wrote in his 1994 book The Astonishing Hypothesis, “There is no form of prose more difficult to understand and more tedious to read than the average scientific paper.”

The average scientific paper is a work of fiction: often seemingly perfect, compact, well cut, crisp and concise and perhaps deceptive and unreal in its seductive perfection, just like a movie or novel.

In an example that illustrates the disjunction of scientific papers from the reality of the scientific process, Richard Dawkins, former Professor of Public Understanding of Science at Oxford University tells the story of his experience in 1974 when he was appointed as UK editor of Animal Behaviour in his contribution to Leaders of Animal Behaviour: The Second Generation (2009), a volume of invited autobiographical chapters by ethologists:

My particular bugbear was the formulaic scientific paper with its standard headings: Introduction, Methods, Results, Discussion. The rubric’s limitations were especially glaring when – as was common – the author had done a series of experiments, each one prompting the next. I tried to persuade authors that that the proper sequence of the paper was: Question 1; Methods 1; Results 1; Discussion 1; leading to Question 2; Methods 2; Results 2; Discussion 2 leading to Question 3…and so on. You’d be amazed about how many people arranged their paper in the following way: Introduction; Methods 1, Methods 2, Methods 3, methods4…Results 1, Results 2, Results 3, Results 4…; Discussion. Could anything be obviously calculated to confuse and bore?

Need I point out that this is true, and taken for granted as normal for many papers in the general scientific literature, including medicine? It also shows how difficult it is to change a status quo.

However, I have a proposal:

For any movie I particularly enjoy or find intriguing for any particular reason, I want to see a second screening, just like reading a novel the second time. I also want to see the making or listen to an audio commentary by directors and possibly the actors. I want to observe, even if partly, the creative process.

Seeing the process of making the movie or listening to the directors and actors almost demystify the process however doesn’t quite undo it; rather, for me, it deepens and enriches understanding and appreciation of the work.

Just like the movie commentary, I imagine fellow researchers and the lay public would benefit from having a behind the curtain exposure to the workings of the mind of the scientist in action, the process of arriving at the research question, what each author did, how each person became an author, the moments of revelation, the debates, of choice between this analysis or that, this mode of presentation of results or the other: an exposure of science without much of its makeup.

For every publication, every analysis, there should be some sort of author commentary, chatty maybe, contemplative or argumentative, published separately, or recorded as an audio or video podcast, not necessarily for the public, but understandable by an intelligent non-scientific audience without compromising the scientific message.

The internet has revolutionised the amount of space available for publication and so we can’t make the same excuse again about limited space. The limits imposed by space might have been the reason for the present state and structure of the scientific paper, but we can begin to undo its sterile style and language.

It may even be an opportunity for journals that publish these commentaries online to make some revenue from them, and also a very good avenue through which scientists can begin to engage with the public in a more direct way, without the influence of the non-scientific media. I reckon it would also further enhance the standing of scientists, and a more honest engagement with peers and the public.

I imagine something like this:

We couldn’t have done it otherwise. It wouldn’t have made much sense if we did. Most other groups have used a Cox proportional hazards model to assess predictors of time to remission but we decided on deeper reflection and after much argument, mostly between SJG and RD – the two clinicians in the group – that what really matters to patients is not how long it takes for them to achieve seizure remission but how long they spend in remission. So we divided the patients into those who had spent the more than 1 year in continuous remission and those with less than one year in continuous remission and decided to look at the factors that may predict each outcome in a logistic model. The result, apart from being less equivocal than in previous studies is apparently also more useful although we doubt that we have contributed any much further to what was already known.

This post also appears on BMJ Blogs here

Rather curiously, as I was going to give a Work-in-Progress presentation on a Systematic Review and Meta-Analysis of prognosis studies in epilepsy, I looked at this week’s edition of the BMJ, a regular weekly practice, and in it were three articles discussing the quality of prognosis studies, with broader implications for observational research generally. The first, Ten steps towards improving prognosis research (free full text) by seasoned researchers of prognosis – Harry Hemingway, Richard Riley and Doug Altman – does just what its title says in what Richard Lehman described on his blog as “rather angry in tone, but not angry enough for my taste.”

In a linked editorial, Sørensen and Rothman in their interestingly titled The prognosis for research disagree with Hemingway et al on their suggestion that there should be a register for prognosis studies, and observational research generally as it is presently required by law for clinical trials:

We suspect that historians and philosophers of science would recoil at the notion that advance registration of all scientific studies in a publicly accessible database would produce better science. How much room would this policy leave for exploration, serendipity, or pursuit of unpopular theories?

If the rules precluded easy registration, that might create an undesirable drag on the end of the research spectrum that constitutes the quirky, brilliant work that is not enterprise driven. Moreover, registration would not prevent publication bias among the many studies conducted with secondary data, because researchers could still selectively register study ideas after the data have been explored.

They also disagree with Hemingway et al on their suggestion of developing guidelines for reporting prognosis research:

Reporting guidelines do have advantages, but the disadvantages are generally overlooked. On the positive side, guidelines increase uniformity and can improve the average quality of reporting. But guidelines also promote rigidity and can enshrine misconceptions, because they are merely compiled from the consensus of a few opinion leaders and form a common denominator of current beliefs. If all science throughout human history had been filtered through reporting guidelines, we suspect we would live in a very different world, one in which the science had lagged far behind what actually has been achieved.

They end their article by placing the responsibility and the blame for the quality of prognosis research at the feet of journal editors:

Consider the crucial role of the gatekeepers of published research. Any published research, including the low quality work … has survived the scrutiny of peer reviewers and of the ultimate gatekeepers, journal editors. Perhaps the priority should be continuing education efforts focused on journal editors.

Then comes the third article this week on the same topic as BMJ editors Elizabeth Loder, Trish Groves, and Domhnall MacAuley respond to Sørensen and Rothman in another editorial, Registration of observational studies. They defend the need for protocol driven observational studies:

At present, consumers of observational research cannot easily distinguish hypothesis driven studies from exploratory, post hoc data analyses. Researchers do not routinely disclose the number of additional analyses performed. Nor is there any satisfactory way to know whether the research questions or methods of statistical analysis diverged from those initially planned.

We agree that exploratory observational research is important. Many new ideas arise from unexpected findings in observational research, and many researchers learn their skills from examining available datasets. However, that is not the sort of research the BMJ usually aims to publish…

I thought that was a rather weak argument though, and while they are quiet about the need to train journal editors, they go ahead to state a series of not necessarily insurmountable hurdles to get your observational study published in the BMJ:

We will now ask authors of papers reporting observational studies submitted to the BMJ to tell us more about the origins, motivations, and data interrogation methods of that work.

We will be asking authors to report in their papers a clear statement of whether the study hypothesis arose before or after inspection of the data…

We will ask to see study protocols if they exist; and we will add to the papers’ abstracts their registration details, if they have been registered…

The Systematic Review and Meta-Analysis I’m presently conducting though shows that apart from the thorny issue of inconsistent definitions of disease state and classification, and the conceptualisation of outcome measures – which none of these hurdles addresses or could possibly address – there seems to be relatively better quality of prognosis research at least in epilepsy than these articles generally suggest, which may at least partly be due to the fact that clinical trials in epilepsy are rather tricky.

“Something has just happened that will almost certainly end the tyranny of impact factors and may well mark another step towards the extinction of most scientific journals,” writes Richard Smith triumphantly in his latest blog post on BMJ. He continues:

It was the appearance of something called rather clunkily “Article-Level Metrics.” These are a variety of scores and other bits of information attached to each article in the publications of the Public Library of Science. They shift attention from journals to articles, particularly for the academic bean counters anxious to find a convenient and low cost way of ranking academics.

Richard Smith concludes, after explaining that Article-Level Metrics works by tracking each article’s online usage including citations from scholarly literature, social bookmarks, comments left by readers, notes left within articles, blog posts, and ratings, saying:

Increasingly governments and research funders are interested not just in the number of times an article is cited in other publications (an incestuous and self serving measure) but on the impact they have in the real world, the changes they lead to.

So that’s why article level metrics might doom the impact factor, but why might they signal an end to many journals? It’s because they lead to articles rather than journals being what matters, and the articles can then be published quickly on databases rather than in journals…

The edifice of journals is beginning to crack—and not before time.

The full post that does justice to how the Article-Level Metrics works is here.

The Public Library of Science gives a background explanation of the Article-Level Metrics here and here, where Mark Patterson was wise enough to remind us that:

It’s also important to emphasize that online usage should not be seen as an absolute indicator of quality for any given article, and such data must be interpreted with caution.

There is an example of how the Article-Level Metrics statistics and graph look here.

This is an article from Nature about how the recession has dampened donor enthusiasm for scientific research in Africa and here is an excerpt about Nigeria:

Countries that don’t depend on aid are also struggling. In Nigeria, the drop in demand for oil and gas, exacerbated by a stricken banking sector, means that private donations — a major source of funding for Nigerian universities — are slowing. “In the past, a conference like this would have a lot of Nigerians coming, supported by industry grants. We don’t find many today,” says Oye Ibidapo-Obe, president of the Nigerian Academy of Science in Lagos. Nigeria’s government won’t pick up the slack left by the drop in private investments, Ibidapo-Obe adds. “Research is not seen as the major driver of the economy.”

I am not sure where Nature got these assertions from, or why Oye Ibidapo-Obe said what he is quoted to have said, but this certainly reeks of falsehood. Someone is either making up stories to mislead Nature, or Nature itself is doing the  embellishment. Nigerian banking sector, and industry grants being a major source of funding for Nigerian universities? This is just so false.

Member journals of the International Committee of Medical Journal Editors (ICMJE) have simultaneously published an editorial proposing a uniform format for requesting and reporting competing interest information from authors:

We ask authors to disclose four types of information. Firstly, their associations with commercial entities that provided support for the work reported in the submitted manuscript (the time frame for disclosure in this section of the form is the lifespan of the work being reported). Secondly, their associations with commercial entities that could be viewed as having an interest in the general area of the submitted manuscript (the time frame for disclosure in this section is the 36 months before submission of the manuscript). Thirdly, any similar financial associations involving their spouse or their children under 18 years of age. Fourthly, non-financial associations that may be relevant to the submitted manuscript.

The new form is here and there are instructions and examples to help authors provide the information here.

Nobel Laureate Tim Hunt, and other scientists E. O. Wilson, Larry Brilliant, Steven Pinker, Rodney Brooks, Peter Diamandis, David King, Lynn Margulis, Peter Raven and Frans de Waal want two new Nobel prizes for Global Environment and Public Health created. They also suggest expanding or adding to the Physiology or Medicine Prize “to recognise contributions from across the life sciences.” They publish an open letter to the Nobel Prize Committee in this week’s New Scientist:

We appreciate that the foundation is bound by Nobel’s will. But we also note that the foundation has shown flexibility in the past, the creation of the economics prize in 1968 being one example. With that in mind, we would like to recommend two changes that we feel would align the prizes with current challenges:

1. The creation of Nobel prizes for the Global Environment and Public Health. The new prizes would focus on applications of science rather than basic research. As with the existing peace prize, organisations would be eligible. The environment prize would recognise successes in promoting sustainability, mitigating climate change or reducing biodiversity losses. The public health prize would recognise improvements in global health, such as the reduction or eradication of disease. (We present these lists as examples; they are not intended to be complete).

2. The expansion of, or an addition to, the prize for physiology or medicine to recognise contributions from across the life sciences. Fields that are currently excluded, such as ecology, would become eligible. More emphasis would be placed on the rapidly expanding field of neuroscience. This could be achieved by expanding the existing prize for medicine or physiology or by the addition of new prizes for fundamental biology (including ecology, genetics and cellular, molecular and evolutionary biology) and behavioral science (including psychology and neuroscience).

Over the past century, progress in the basic sciences has transformed our world and our understanding of it. By recognizing the men and women that drove that progress, the Nobel prizes have made the public aware of the enormous contribution that science has made. Different forms of science and technology will transform our world during this century. We feel that these suggestions will enable the prizes to appropriately recognise future achievements, and to remain influential for another hundred years.

While I consider this an unnecessary venture, the near equivalent of asking for a separate Nobel Literature Prize for Africa, I think it is at least worthy of some discussion.

My take on it is that the Nobel Committee has often demonstrated flexibility, albeit inconsistently in deciding who wins the award and to how disciplinary boundaries are mapped. Take Nobel Peace Prize to Wangari Maathai, Al Gore and IPCC or even Muhammad Yunus and the Grameen Bank for example. Then again, the 1973 Prize in Physiology or Medicine that went to  Karl von Frisch, Konrad Lorenz and Niko Tinbergen for work done in ethology.

We must accept, even if grudgingly that the Nobel Prize is only a prize, and that no prize can possibly be representative of the world’s intellectual output. There’s no need to push it any further.

As we eagerly await this year’s Nobel Prize announcements in a few days, Annals of Improbable Research has once again thrilled the world’s scientific and intellectual community to its junior and supposedly less serious sibling, the Ig® Nobel Prizes, awarded for improbable research that makes you laugh and then think. This year, the awards are:

PUBLIC HEALTH
For inventing a brassiere that, in an emergency, can be quickly converted into a pair of gas masks, one for the brassiere wearer and one to be given to some needy bystander.

Elena N. Bodnar, Raphael C. Lee, and Sandra Marijan of Chicago, Illinois, USA

(Reference: U.S. patent # 7255627, granted August 14, 2007 for a “Garment Device Convertible to One or More Facemasks.”)

BIOLOGY
For demonstrating that kitchen refuse can be reduced more than 90% in mass by using bacteria extracted from the feces of giant pandas.

Fumiaki Taguchi, Song Guofu, and Zhang Guanglei of Kitasato University Graduate School of Medical Sciences in Sagamihara, Japan

(Reference: “Microbial Treatment of Kitchen Refuse With Enzyme-Producing Thermophilic Bacteria From Giant Panda Feces,” Fumiaki Taguchia, Song Guofua, and Zhang Guanglei, Seibutsu-kogaku Kaishi, vol. 79, no 12, 2001, pp. 463-9. [and abstracted in Journal of Bioscience and Bioengineering, vol. 92, no. 6, 2001, p. 602.]
Reference: “Microbial Treatment of Food-Production Waste with Thermopile Enzyme-Producing Bacterial Flora from a Giant Panda” [in Japanese], Fumiaki Taguchi, Song Guofu, Yasunori Sugai, Hiroyasu Kudo and Akira Koikeda, Journal of the Japan Society of Waste Management Experts, vol. 14, no. 2, 2003, pp. , 76-82.)

PEACE
For determining — by experiment — whether it is better to be smashed over the head with a full bottle of beer or with an empty bottle.

Stephan Bolliger, Steffen Ross, Lars Oesterhelweg, Michael Thali and Beat Kneubuehl of the University of Bern, Switzerland

(Reference:”Are Full or Empty Beer Bottles Sturdier and Does Their Fracture-Threshold Suffice to Break the Human Skull?” Stephan A. Bolliger, Steffen Ross, Lars Oesterhelweg, Michael J. Thali and Beat P. Kneubuehl, Journal of Forensic and Legal Medicine, vol. 16, no. 3, April 2009, pp. 138-42. DOI:10.1016/j.jflm.2008.07.013.)

MEDICINE
For investigating a possible cause of arthritis of the fingers, by diligently cracking the knuckles of his left hand — but never cracking the knuckles of his right hand — every day for more than sixty (60) years.

Donald L. Unger, of Thousand Oaks, California, USA

(Reference: “Does Knuckle Cracking Lead to Arthritis of the Fingers?”, Donald L. Unger, Arthritis and Rheumatism, vol. 41, no. 5, 1998, pp. 949-50.)

PHYSICS
For analytically determining why pregnant women don’t tip over.

Katherine K. Whitcome of the University of Cincinnati, USA, Daniel E. Lieberman of Harvard University, USA, and Liza J. Shapiro of the University of Texas, USA

(Reference: “Fetal Load and the Evolution of Lumbar Lordosis in Bipedal Hominins,” Katherine K. Whitcome, Liza J. Shapiro & Daniel E. Lieberman, Nature, vol. 450, 1075-1078 (December 13, 2007). DOI:10.1038/nature06342.)

CHEMISTRY
For creating diamonds from liquid — specifically from tequila.

Javier Morales, Miguel Apátiga, and Victor M. Castaño of Universidad Nacional Autónoma de Mexico

(Reference: “Growth of Diamond Films from Tequila,” Javier Morales, Miguel Apatiga and Victor M. Castano, 2008, arXiv:0806.1485.)

LITERATURE
Ireland’s police service (An Garda Siochana), for writing and presenting more than fifty traffic tickets to the most frequent driving offender in the country — Prawo Jazdy — whose name in Polish means “Driving License”.
[Karolina Lewestam, a Polish citizen and holder of a Polish driver's license, speaking on behalf of all her fellow Polish licensed drivers, was present to expressed her good wishes to the Irish police service.]

MATHEMATICS
For giving people a simple, everyday way to cope with a wide range of numbers — from very small to very big — by having his bank print bank notes with denominations ranging from one cent ($.01) to one hundred trillion dollars ($100,000,000,000,000).

Gideon Gono, governor of Zimbabwe’s Reserve Bank

(Reference: Zimbabwe’s Casino Economy — Extraordinary Measures for Extraordinary Challenges, Gideon Gono, ZPH Publishers, Harare, 2008, ISBN 978-079-743-679-4.)

ECONOMICS
The directors, executives, and auditors of four Icelandic banks — Kaupthing Bank, Landsbanki, Glitnir Bank, and Central Bank of Iceland — for demonstrating that tiny banks can be rapidly transformed into huge banks, and vice versa — and for demonstrating that similar things can be done to an entire national economy.