Square One

I couldn’t help posting this ballad by Richard Lehman from this weeks edition of his incisive summary of the “big five” general medical journals in BMJ’s doc2doc blogs:

A Ballad for Obama

When Massachusetts blew it
We bit our lips and hid,
We said he couldn’t do it,
But then he went and did.

The lobby men seemed cosy
On profits from the sick,
But out strode Dame Pelosi
And gave her troops some stick:

At sight of such a muster
Republicans turned white:
They dropped their filibuster
And lost the will to fight.

All hail, Barack and Nancy!
The world shares your delight:
Americans may fancy
Health is now theirs by right.

“…and nothing is gained by calling it by a nicer name,” says Tony Judt, historian and professor at New York University as conclusion to his reflections on living with a motor neuron disorder which has made him “effectively quadriplegic.”

Here is an excerpt from the last paragraph of the article, Night:

I suppose I should be at least mildly satisfied to know that I have found within myself the sort of survival mechanism that most normal people only read about in accounts of natural disasters or isolation cells. And it is true that this disease has its enabling dimension: thanks to my inability to take notes or prepare them, my memory – already quite good – has improved considerably, with the help of techniques adapted from the “memory palace” so intriguingly depicted by Jonathan Spence. But the satisfactions of compensation are notoriously fleeting. There is no saving grace in being confined to an iron suit, cold and unforgiving. The pleasures of mental agility are much overstated, inevitably – as it now appears to me – by those not exclusively dependent upon them. Much the same can be said of well-meaning encouragements to find nonphysical compensations for physical inadequacy. That way lies futility.

The full text of Night is here (free pdf)

Tony Judt presently writes a series of short reflections for The New York Review of Books. Night is first in the series.

I am presently conducting a systematic review and meta-analysis that involves a detailed and critical quality appraisal of studies which has made me realise all over again how impoverished the narrative of the classical scientific paper really is, and that indeed like Francis Crick wrote in his 1994 book The Astonishing Hypothesis, “There is no form of prose more difficult to understand and more tedious to read than the average scientific paper.”

The average scientific paper is a work of fiction: often seemingly perfect, compact, well cut, crisp and concise and perhaps deceptive and unreal in its seductive perfection, just like a movie or novel.

In an example that illustrates the disjunction of scientific papers from the reality of the scientific process, Richard Dawkins, former Professor of Public Understanding of Science at Oxford University tells the story of his experience in 1974 when he was appointed as UK editor of Animal Behaviour in his contribution to Leaders of Animal Behaviour: The Second Generation (2009), a volume of invited autobiographical chapters by ethologists:

My particular bugbear was the formulaic scientific paper with its standard headings: Introduction, Methods, Results, Discussion. The rubric’s limitations were especially glaring when – as was common – the author had done a series of experiments, each one prompting the next. I tried to persuade authors that that the proper sequence of the paper was: Question 1; Methods 1; Results 1; Discussion 1; leading to Question 2; Methods 2; Results 2; Discussion 2 leading to Question 3…and so on. You’d be amazed about how many people arranged their paper in the following way: Introduction; Methods 1, Methods 2, Methods 3, methods4…Results 1, Results 2, Results 3, Results 4…; Discussion. Could anything be obviously calculated to confuse and bore?

Need I point out that this is true, and taken for granted as normal for many papers in the general scientific literature, including medicine? It also shows how difficult it is to change a status quo.

However, I have a proposal:

For any movie I particularly enjoy or find intriguing for any particular reason, I want to see a second screening, just like reading a novel the second time. I also want to see the making or listen to an audio commentary by directors and possibly the actors. I want to observe, even if partly, the creative process.

Seeing the process of making the movie or listening to the directors and actors almost demystify the process however doesn’t quite undo it; rather, for me, it deepens and enriches understanding and appreciation of the work.

Just like the movie commentary, I imagine fellow researchers and the lay public would benefit from having a behind the curtain exposure to the workings of the mind of the scientist in action, the process of arriving at the research question, what each author did, how each person became an author, the moments of revelation, the debates, of choice between this analysis or that, this mode of presentation of results or the other: an exposure of science without much of its makeup.

For every publication, every analysis, there should be some sort of author commentary, chatty maybe, contemplative or argumentative, published separately, or recorded as an audio or video podcast, not necessarily for the public, but understandable by an intelligent non-scientific audience without compromising the scientific message.

The internet has revolutionised the amount of space available for publication and so we can’t make the same excuse again about limited space. The limits imposed by space might have been the reason for the present state and structure of the scientific paper, but we can begin to undo its sterile style and language.

It may even be an opportunity for journals that publish these commentaries online to make some revenue from them, and also a very good avenue through which scientists can begin to engage with the public in a more direct way, without the influence of the non-scientific media. I reckon it would also further enhance the standing of scientists, and a more honest engagement with peers and the public.

I imagine something like this:

We couldn’t have done it otherwise. It wouldn’t have made much sense if we did. Most other groups have used a Cox proportional hazards model to assess predictors of time to remission but we decided on deeper reflection and after much argument, mostly between SJG and RD – the two clinicians in the group – that what really matters to patients is not how long it takes for them to achieve seizure remission but how long they spend in remission. So we divided the patients into those who had spent the more than 1 year in continuous remission and those with less than one year in continuous remission and decided to look at the factors that may predict each outcome in a logistic model. The result, apart from being less equivocal than in previous studies is apparently also more useful although we doubt that we have contributed any much further to what was already known.

This post also appears on BMJ Blogs here

I thought I’d have you listen in on this IM conversation I had with a friend from medical school, Simon Adebola, about science, science illiteracy and biomedical science in Nigeria/Africa. Simon blogs at iInitiative.

Simon Adebola: So tell me, what is new in the nebulous world of cells transmitters and neurobiology?

Seye Abimbola: Nebulous world?

Simon Adebola: Just teasing. But wait, let’s see how well you can guard your territory. Imagine I thought it was nebulous and even went a bit further to say that science could be tricky and the analysis dodgy.

Seye Abimbola: …and you’d be perfectly right. That is indeed the true nature of science and the bravado and hubris of science in its more modern history is a loss and the way science has fed public imagination with promises of its powers is also unfortunate… That said, it is still the only way we know by which we can grasp the mysteries of  the natural world, hence the need for constant doubt and skepticism, from the makers and the consumers of science alike.

Simon Adebola: Wait a minute, you remind me of this Oxford Prof Jerome Ravetz. He wrote on post normal science, citing much of what you just stated above. It could be that much of what we call hard facts, especially in modern science is not as factual as we tend to want to make our journal editors, peers and larger public believe.

Seye Abimbola: Journal editors and peers are often conniving partners in the business… and unfortunately, the scientifically illiterate public and newspaper editors just take it in, and spread it… and it backfires some times, with the recent example of Climate Science. Climate science had an agenda and I am suspicious of any science with an agenda and unfortunately that is what much of science is today.

Simon Adebola: Well, all writing, I was taught, has an agenda, and that virtually spoilt films and entertainment for me because I then acquired a magnifying lens and sometimes it descends much lowers to an agenda for money. Science like religion has proven not to sit comfortably with the kind of scrutiny it has gotten. They both would rather prefer to be seen as being infallible and yet no enterprise with humans at the helm should be seen as such

Seye Abimbola: It is troubling how money and agenda drives a lot of research, including medical research and how unfortunately no one beyond the club is even able to really scrutinize. When I was at the BMJ (British Medical Journal) I had a different impression of how science worked. There was the image of science in its most perfect, ideal sense, and although it showed that there was a lot of crap science and studies going on, it didn’t quite ring home that it was a given in “the holy of holies of intellectual objectivity” (Wole Soyinka).

Simon Adebola: Being a strong believer in objectivity and experimentation (I find it truly fascinating) I wonder what the scientific community can do to regain its credibility.

Seye Abimbola: I don’t think it will happen unless we redefine our index of academic credit and the way science is funded – number of papers in peer reviewed journals is a bad idea and funding according to result – often number of papers or positive result – is killing science. It forces scientists to want to say something, when there isn’t anything to say, creates publication bias, unnecessary data analysis et cetera.

Simon Adebola: Sometimes it is like the case of a serially abused individual. Concurrently ignored and used by those they hoped would care about them – politicians and to a lesser extent industry. Over at Cuba (Forum 2009, Global Forum for Health Research) there was this palpable inferiority complex in the research community, a complex not devoid of pride, seemingly crying to be heard by policy makers. As they say in Yoruba, it is a thief who knows how to trace the footprints of another thief on a rock. Once the politicians/policy people see through the credibility flaws, they just would rather use, rather than trust the research community. What would you recommend to improve the assimilation of science into policy?

Seye Abimbola: There’s a lot that is wrong about how science is presently done and how it feeds into policy. I’ve been thinking a lot about policy these days…Ultimately what we need to do is improve scientific literacy. I wouldn’t mind suggesting a model that has scientists, not necessarily practicing, as policy makers in science/medicine…

Simon Adebola: …building a bridge sort of.

Seye Abimbola: Yes, because it’s so easy for scientists to stand on the other end of the divide and send in dumbed down, over-edited, information that lack the nuances, and the element of doubt that comes with science…I’m not happy about the example of Al Gore who has been the most public face for climate change for a long time…It would be a different scenario entirely if he is re-echoing what scientists in the field are saying to the public. However, scientists in the field are the ones trying to re-echo what he is saying by making their data agree.

Simon Adebola: No one is comfortable with the ‘everything is caused by climate change’ line. It gets rancid after a while, with science making the claim on both sides. Ten years ago, science predicted that due to climate change some parts of the world experience drops in snow, for example I heard they said British children would not know what snow was. Now science is proving to us that due to climate change, there would be fiercer snow storms. That breeds the reaction you get when you discover the movie you are watching does not have a plot you want to turn it off, but again you want to see if its plotlessness, is the ingenuity of the director in display, so you hang on watching, hoping it would eventually make sense, somehow.

Seye Abimbola: Again, this is because scientists are not committed to saying the truth the way it is…

Simon Adebola: …and that is the context in which post normal science explores its stance. “Post-Normal Science is a concept developed by Silvio Funtowicz and Jerome Ravetz, attempting to characterise a methodology of inquiry that is appropriate for cases where “facts are uncertain, values in dispute, stakes high and decisions urgent”. It is primarily seen in the context of the debate over global warming and other similar, long-term issues where we possess less information than we would like.” (Wikipedia)

Seye Abimbola: …and again it boils down to scientists feeling a need for that sort of misplaced recognition…

Simon Adebola: …true, opening them up to near destructive abuse. I guess each side just has to make peace with its roles. Oxford would never be Hollywood, or Washington DC, or the Super bowl. Hollywood with its fortune, sports with its fame, and Washington with its power wielding capabilities. The strength of science like you have said would continue to lie in its innovativeness and simplicity once other interests start driving it, that inferiority complex bites in, and self destruction could result. For now we observe the movie, hoping there is a plot. Those profiting off this, increase the hype, the noise, silence the naysayers and hope to bank as much as they can, such that win or lose, at least they have made enough to reward their efforts.

Seye Abimbola: I’m wondering what is there for science in Nigeria… There’s a lot that never happened, despite enormous early promise in Nigeria.

Simon Adebola: There is hope. New minds, fresh minds, need to be trained. We need a reorientation. Science as you know has flourished even when repressed. Galileo, Einstein. It is the commitment that we should hope does not dwindle. The value is in service that would drive a pursuit of excellence, creativity, and better ways of doing things…

Seye Abimbola: In medicine, if we look back to the days of Osuntokun et cetera, they somehow did not, and I suspect due to a lot happening on the political front in Nigeria, manage to build that critical mass that could help sustain scientific productivity. Those guys did and published a lot of great work, good, world class studies and it just didn’t trickle down the generations…and I’m wondering, what can we do? How do we ensure that fresh minds are trained?

Simon Adebola: I hope there can be mega research institutes that will represent a focus on excellent research, openness to innovation, and economically sustainable models where research and innovation lead to productivity and development. I also think scholarships and studentships focused on solving the actual needs in the continent are a crucial need – these should come first. It is just that the selfishness can be acute and sometimes crippling, but we can’t deny the need to keep building capacity.

Seye Abimbola: We are presently finishing up the Build AfReCa! (African Research Capacity) paper for the journal Science. Build AfReCa! Is a very young network of young scientists, mostly Africans in the Diaspora, mostly students trying to work towards improving research capacity in Africa…

Simon Adebola: We need more and more of that, aggressively driving knowledge growth.

Seye Abimbola: We put out a survey in the last quarter of last year to assess the needs of young scientists from Africa and why they might not work in Africa and what might make them want to work in Africa, and their general geographical spread. At this stage, it’s essentially advocacy, creating a voice, an image, some advocacy for the need for funding, coordinated funding for young scientists in sub-Saharan Africa, funded to do great work on the ground in Africa.

Simon Adebola: I think that is crucial and greater seriousness with African journals. We need the equivalent of The Lancet, BMJ and NEJM (New England Journal of Medicine) on the continent…In fact one could talk to some of these journals to help grow stronger journals with greater visibility on the continent.

Seye Abimbola: We will need to work with the model like PLoS (Public Library of Science). It would be nice to have a PLoS Africa…. PLoS is absolutely open access, and online with a good Impact Factor…The tricky bit is that it will be online, but again, internet access in Africa is getting better by the day…so, that can be done.

Simon Adebola: …and daily digests can be sent by email or even SMS gateways alerting of papers of interest…

Seye Abimbola: …the first place to go when looking for good studies from Africa.

Simon Adebola: I am sure we can get funding for that…The Library is online, you register and select your interest. Each time a paper of interest to you appears, based on your selection, you get an SMS with basic info on the paper.

Seye Abimbola: The journal will need an editorial team, a peer reviewer bank, et cereta.

Simon Adebola: This is the kind of aid they should be interested in giving Africa, not more money for corrupt leaders…

Seye Abimbola: Good. Maybe we should put a proposal together…

Simon Adebola: I think we should…once we have the back end defined well, and teams in place… and even though it costs, we can start with donor funding and once we have a critical base of users, we can work on different models to make it work. This would make research awareness go up greatly…

Seye Abimbola:  Thanks. It’s been a great conversation, and I’m tempted to blog excerpts from the conversation on NT.org (Nigerians Talking Science – An IM Conversation).

Simon Adebola: Thanks. Please feel free to do that. It’s been a huge pleasure on my part.

Richard Smith reviews Nigel Crisp‘s  Turning the World Upside Down: The Search for Global Health in the 21st Century in this week’s BMJ. The striking comment in the review was an explanation for health workforce brain drain in Africa:

His case for “turning the world upside down” might begin with the stark facts that Africa has 25% of the world’s burden of disease but only 3% of its healthcare resources and 1% of health workers. North America, in contrast, has 3% of the disease burden but 25% of healthcare resources and 30% of health workers.

Rich countries are plundering health workers from poor countries, and one reason that’s happening is that rich countries have exported their outdated health systems and ways of thinking—meaning that health workers in poor countries are trained inappropriately and feel more comfortable in rich settings.

It has just occurred to me that developing countries might indeed be in the best position to redefine the health worker for the 21st century in the light of America’s super-expensive over-doctored system, and the sheer effect of grappling with chronic illnesses on health systems globally.

The full review is here.

Rather curiously, as I was going to give a Work-in-Progress presentation on a Systematic Review and Meta-Analysis of prognosis studies in epilepsy, I looked at this week’s edition of the BMJ, a regular weekly practice, and in it were three articles discussing the quality of prognosis studies, with broader implications for observational research generally. The first, Ten steps towards improving prognosis research (free full text) by seasoned researchers of prognosis – Harry Hemingway, Richard Riley and Doug Altman – does just what its title says in what Richard Lehman described on his blog as “rather angry in tone, but not angry enough for my taste.”

In a linked editorial, Sørensen and Rothman in their interestingly titled The prognosis for research disagree with Hemingway et al on their suggestion that there should be a register for prognosis studies, and observational research generally as it is presently required by law for clinical trials:

We suspect that historians and philosophers of science would recoil at the notion that advance registration of all scientific studies in a publicly accessible database would produce better science. How much room would this policy leave for exploration, serendipity, or pursuit of unpopular theories?

If the rules precluded easy registration, that might create an undesirable drag on the end of the research spectrum that constitutes the quirky, brilliant work that is not enterprise driven. Moreover, registration would not prevent publication bias among the many studies conducted with secondary data, because researchers could still selectively register study ideas after the data have been explored.

They also disagree with Hemingway et al on their suggestion of developing guidelines for reporting prognosis research:

Reporting guidelines do have advantages, but the disadvantages are generally overlooked. On the positive side, guidelines increase uniformity and can improve the average quality of reporting. But guidelines also promote rigidity and can enshrine misconceptions, because they are merely compiled from the consensus of a few opinion leaders and form a common denominator of current beliefs. If all science throughout human history had been filtered through reporting guidelines, we suspect we would live in a very different world, one in which the science had lagged far behind what actually has been achieved.

They end their article by placing the responsibility and the blame for the quality of prognosis research at the feet of journal editors:

Consider the crucial role of the gatekeepers of published research. Any published research, including the low quality work … has survived the scrutiny of peer reviewers and of the ultimate gatekeepers, journal editors. Perhaps the priority should be continuing education efforts focused on journal editors.

Then comes the third article this week on the same topic as BMJ editors Elizabeth Loder, Trish Groves, and Domhnall MacAuley respond to Sørensen and Rothman in another editorial, Registration of observational studies. They defend the need for protocol driven observational studies:

At present, consumers of observational research cannot easily distinguish hypothesis driven studies from exploratory, post hoc data analyses. Researchers do not routinely disclose the number of additional analyses performed. Nor is there any satisfactory way to know whether the research questions or methods of statistical analysis diverged from those initially planned.

We agree that exploratory observational research is important. Many new ideas arise from unexpected findings in observational research, and many researchers learn their skills from examining available datasets. However, that is not the sort of research the BMJ usually aims to publish…

I thought that was a rather weak argument though, and while they are quiet about the need to train journal editors, they go ahead to state a series of not necessarily insurmountable hurdles to get your observational study published in the BMJ:

We will now ask authors of papers reporting observational studies submitted to the BMJ to tell us more about the origins, motivations, and data interrogation methods of that work.

We will be asking authors to report in their papers a clear statement of whether the study hypothesis arose before or after inspection of the data…

We will ask to see study protocols if they exist; and we will add to the papers’ abstracts their registration details, if they have been registered…

The Systematic Review and Meta-Analysis I’m presently conducting though shows that apart from the thorny issue of inconsistent definitions of disease state and classification, and the conceptualisation of outcome measures – which none of these hurdles addresses or could possibly address – there seems to be relatively better quality of prognosis research at least in epilepsy than these articles generally suggest, which may at least partly be due to the fact that clinical trials in epilepsy are rather tricky.

Richard Dawkins lends his well recognised voice to the homeopathy debate presently going on in the UK by proposing a study:

1. Take a large, predetermined number of patients, preferably who have presented themselves to homeopathic clinics and been rated suitable for treatment by homeopathic practitioners. They needn’t all be suffering from the same complaint, although it will increase the resolving power of the experiment if they are. Every patient should be examined before the experiment begins, by homeopathic practitioners, the best the profession can come up with, who should write a report on the patient. For every patient, the practitioners should agree upon a prescription of what they consider the ideal homeopathic treatment. The prescriptions for the different patients need not be the same. Every patient is written a prescription of an ideal homeopathic remedy, personally chosen, individually tailored to that individual and for the relevant complain – so nobody can come along afterwards and allege that the treatment was not sufficiently ‘holistic’, or did not take sufficient account of individual requirements.

2. Randomly assign half the patients to the experimental group, and half to the control group. It is vitally important that nobody involved in the experiment should have any way of discovering which patients are experimental and which control: not the homeopathic practitioners, nor the patients, nor the nurses taking care of them, nor anybody involved in writing down the data. The choice should be determined at random by a computer, unknown to any human, and stored securely in the computer.

3. For every one of the prescriptions written down for individual patients, professional homeopathic technicians (the best in the business) should make up the medication identically for the experimental and control cases, with an identical regime of succussion (successive dilution and shaking) with the single exception that the procedure for preparing the experimental doses begins with the purported active ingredient, while the control doses begin with the same volume of water. Apart from that, both must be made by the same regime of successive dilution and shaking. At all stages, the procedures should be carried out by fully trained and experienced homeopathic technicians, exactly as they normally would, but without knowledge of whether they are shaking the experimental or control dose on any one occasion.

4. At the end of the succussion regime, the technicians bottle up the medications, and make them into pills or whatever would be the normal procedure. Then, as determined by the randomising procedure above, each patient is given either the experimental version of his/her own personal prescription or the control version of his/her own personal prescription. Still neither the patient nor anybody else knows which dose is experimental and which control. Treatment proceeds for as long as the homeopathic practitioner has prescribed.

5. At the end of this time, all patients are re-examined by the same practitioners who examined them before the experiment, and judgment is written down as to whether the patient has improved, got worse, or stayed the same. That judgment, once written down, is securely sealed so that it cannot be tampered with after the codes are broken.

6. The computer codes are now broken, and the results analysed by statisticians who are told only that this set of patients belong to ‘Group A’, and that set of patients belong to ‘Group B’. If there is any statistically significant difference between the groups, the identities of ‘Group A’ and ‘Group B’ may now be divulged. My shirt is on there being no difference. Indeed, if there is a significant difference, and it is a repeatably verifiable effect, I will eat my shirt.

It’s just another Double-Blind placebo-Controlled Randomised Trial (DBCRT). That’s what we need to prove the evidence for as many medical interventions as are amenable to this design, and homeopathy absolutely is. The full article is here.