Some quick context:
- A few weeks ago Oxfam published a report (although is keen to point out that there were many non-Oxam parties involved) claiming that health coverage under Ghana's NHIS is much, much lower than claimed by the NHIS, and that a majority of Ghanians could be paying for a system that only covers 18% of the population.
- The Guardian picked it up (and, perhaps, exaggerated it a little to fit their tone).
- Amanda Glassman at the CGD blogged a (snarky?) rebuttal about the report's estimates.
- Aid Thoughts (specifically: me) reflexively delighted in Amanda's response, as it (admittedly) fell in line with my priors on NGO reports.
- The authors of the report return fire in the comments section, accusing Glassman of playing fast and loose with the figures.
- Glassman responds in comments, doesn't really budge.
- Duncan Green goes Bruce Banner, writes a (very reasonable and measured, given that he was angry) post about the original attack by Glassman.
- I read about all this over my morning bowl of muesli and yogurt, and figure I'd better just put all of it out there.
Duncan points out that it is a shame that much of the complex debate is lost in the comments. I completely agree - but the same can easily be said for the Appendices of these large reports, where the method used to determine Ghana's NHIS coverage sits. It is true that boring assumptions are often relegated to the appendix (ever more so in econ papers), but when the figure plays such a big part of the story, those assumptions need to be front and center.
Furthermore - defenders of the report point out that they're working in a relative data desert (especially since the NHIS hasn't been very accommodating). Then again, I think a report that says "the figures are really unknown, and so many people could be excluded, so let's figure them out" would be a lot more reasonable than "the figures are really unknown, so let's use a dodgy method to come up with a figure that we are going to run with."
I should add that all this still adds to my argument that we need think tanks out there which are solely dedicated to careful reading and criticism. One can dream.
13 Comments
A think tank dedicated solely to reading and criticizing other people's work? I think that's what the development blogosphere does, and we do it for free.
Dave,
IMHO, we still do a pretty poor job of it - and we do it part time.
Bruce Banner? Uncontrollable giggles have broken out. You made my weekend.
AidThoughts sits next to AidWatch in my feed reader, and for a second I just thought "wow this AidWatch post is totally ripping off AidThoughts"
Matt, I appreciate your point that if the report was equivocal about the figures, it would have been more reasonable. But would it then have provoked the response from the Ghanaian Government and led to the roundtable Duncan describes, designed to get the right figures?
I know this isn't a good justification for changing a research paper, but I imagine that was at least part of the thought process involved, and it would be disingenuous to imagine academics don't occasionally do the same thing, rubbing their hands together as they pen an assassination of Amartya Sen or someone, thinking 'now this is gonna get some citations...'
Ranil,
Good point - but there are still wider ramifications. The report wasn't released just in Ghana, but the rest of the world. The media coverage leaned strongly towards the "the Ghanaian health care system needs a complete overhaul and the model was wrong in the first place" not "we need new figures." People's priors have changed, and will stay changed even if the correct figure turns out to have been 99%
Yes it would be disingenous to image academics don't do the same thing - did I suggest they didn't? That said, academics go and have those figures challenged by other academics from other institutions multiple times in workshops, conferences and peer review. Are they still subject to the cognitive biases of the profession? Of course.
Ranil - one more thing. I should point out that I spend just as much time on this blog pointing out bad academic research that has gotten too much press, so I don't think I am being unfair here.
Matt - you didn't make that implication, I was just saying it wasn't solely an NGO phenomenon. And yes, I'm fully aware that you're just as harsh on academic writing, I enjoy your occasional eviscerations of poor research.
Might point was more this: This report was part research, part advocacy. That might skew the way it presents its results for a reason. There are other reasons why it shouldn't be done like that, but it's something to consider in the final analysis of the value of the product. Though, like you, I would prefer a fully rational process in which we're circumspect about our findings and appropriately open about our methods, and others are willing to engage with reasoned research without a media storm behind it. (btw, I'm not saying this is the situation with this particular report, I'm speaking generally).
Matt
All of this debate over the figure is irrelevant unless you also report the key anomaly of the Ghanaian system- that what is described as insurance is actually funded primarily through VAT. This means all Ghanaians pay for a system only a few can access. This means between the tax revenue and the health services is an inefficient, self serving and ultimately unnecessary insurance bureaucracy. All of this is covered in detail in the report, which it seems you had not read when you made your original post. .
This is the richness of the argument and one you would do well to reflect in your chosen role as research quality vigilante.
I enjoy your blog at times but a bit more humility on your part would be welcome in future.
Kwame
Kwame,
Thanks for your comment. I agree that the revenue for the NHIS is deeply relevant - that's why I linked to the comments made on Amanda's blog as well as Duncan's response. My post shouldn't be seen as an attempt to stake any further claim in this debate. That said, the unfairness of the revenue system doesn't change the uncertainty of the figures, and I see this as the thin line between advocacy and research: one leads you to hold back and question the numbers more, the other to go ahead and run with the numbers you have. Oxfam and its partners have clearly chosen the latter - which is OK, but I don't have to agree with it.
And as far as my `chosen role', I think you're blowing my thoughts out of proportion. I'm just a research student who runs a really obscure blog about aid in his precious spare time, sometimes choosing to question research that gets turned into hyperbole by the media. I wished society subsidized this kind criticism a little more and that more people did it, this even if they are occasionally wrong.
This isn't my day job - I don't claim to be an expert in anything - and I don't have a large think-tank, charity, or even really a university to back me up on anything I say. Humility doesn't even enter into it.
Matt
Kwame,
Following up some more on the VAT front - according to the Oxfam report, VAT is zeroed for goods that are primarly consumed by poor Ghanaians, so it's *also* unclear who is paying for it. The report cites research by James Akazili on this, none of which I can't hunt down online - and the main link provided in the report is broken!
This doesn't throw the argument one way or the other (Even if only the richest 10% of Ghanaians paid VAT but only 18% are covered, we still wouldn't be happy) but just throws more uncertainty in the air about what the truth really is! Surely it is reasonable to argue that we should do a little more analysis?
Kwame - you make a reasonable point that the disjunct between coverage and VAT is the crucial issue. But I think you misinterpret Matt's comments here. He's not trying to say that the research is or isn't correct, or that it does or doesn't make a valid point or a useful contribution (as the comments between him and me demonstrate, we both consider the outcomes of the furore from the report to have had at least some tangible positive outcomes, which is great).
He's merely pointing out that the research methodology of any major paper, especially from (justifiably) well-respected organisations need to be pretty explicit. If you look over some of his older posts (and I'm glad you like the blog, so I hope you have) he makes this argument about several papers and bits of research, academic or advocacy led. I don't think he's making any grand claims to a final voice or anything here.
This blog is basically a place for us to air ideas, get thoughts out (hence the name). It started out of a series of facebook arguments that were quickly clogging up our facebook pages and boring all our non-development friends (and most of the ones who do work in development!). We're not always going to be able to make detailed critiques on everything we cover, but we don't claim to either. Matt's making a valid point here, even if he's not attempting to get to the bottom of the 'truth' of the Ghanaian health system, something neither he nor I would claim to be experts on.
Update: A graceful apology from the CGD here: http://blogs.cgdev.org/globalhealth/2011/03/really-amanda-really.php?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+cgdev%2Fglobalhealth+%28Global+Health+Policy%29
'I deeply apologize to Oxfam and its partners and to our readers for the tone of my post. I should have dialed way back on the snark. Mea culpa.' Kudos to Amanda Glassman for that. She doesn't give on inch on the issues, so there will doubtless be disagreements in the future, but at least now we can have the argument without the stupid stuff.