Just a quick post to highlight an interesting paper just published my Bruce McCullough and Ross McKittrick entitled "Check the Numbers: The Case for Due Diligence in Policy Formation" (hat tip to Steve McIntyre over at Climate Audit). The paper focuses on an issue that I have mentioned once or twice in the past - the transparency of the scientific work upon which major public policy decisions are taken (it is worth noting that one of the authors, McKitrick, was also a co-author with McIntyre on a number of papers critical of the "hockey stick" graph relied upon by the Intergovernmental Panel on Climate Change). Here's the blurb:
Empirical research in academic journals is often cited as the basis for public policy decisions, in part because people think that the journals have checked the accuracy of the research. Yet such work is rarely subjected to independent checks for accuracy during the peer review process, and the data and computational methods are so seldom disclosed that post-publication verification is equally rare. This study argues that researchers and journals have allowed habits of secrecy to persist that severely inhibit independent replication. Non-disclosure of essential research materials may have deleterious scientific consequences, but our concern herein is something different: the possible negative effects on public policy formation. When a piece of academic research takes on a public role, such as becoming the basis for public policy decisions, practices that obstruct independent replication, such as refusal to disclose data, or the concealment of details about computational methods, prevent the proper functioning of the scientific process and can lead to poor public decision making. This study shows that such practices are surprisingly common, and that researchers, users of research, and the public need to consider ways to address the situation. We offer suggestions that journals, funding agencies, and policy makers can implement to improve the transparency of the publication process and enhance the replicability of the research that is published.
The paper discusses a number of cases in which science has formed the basis for important policy decisions and in which, in the authors' view, significant questions have emerged over the full disclosure and transparency of the data and methods used in generating the results. Firstly, and in many ways most strikingly, the authors insist that the pre-publication "peer-review" process, often presented as the arbiter of sound science, is - or should be - only in fact the beginning of the results verification and replication process. They quote the editor of the prestigious journal Science as saying
What we can’t do is ask our peer reviewers to go into the laboratories of the submitting authors and demand their lab notebooks. Were we to do that, we would create a huge administrative cost, and we would in some sense dishonor and rob the entire scientific enterprise of the integrity that 99.9 percent of it has ... it all depends on trust at the end, and the journal has to trust its reviewers; it has to trust the source. It can’t go in and demand the data books.
The most important part begins, then, when articles are published, and other scientists begin to try to replicate the results. It is in order to facilitate this crucial part of the process that, the authors argue, full disclosure of all data, code and methods used is needed to make the replication process as straightforward as possible. Whatever one thinks of the scientific issues raised by the author - and here, as ever, I plead straightforward ignorance - I find it difficult to come up with reasonable counter-arguments against this claim.
From a global administration perspective, there are two points of particular interest: firstly, the authors outline the "hockey stick" controversy, in which the IPCC was a significant player; and secondly, they refer to another case involving climate science relied upon by an international organisation, in a passage worth quoting at some length:
In late 2004, a summary report entitled the Arctic Climate Impact Assessment (ACIA) was released by the Arctic Council, an intergovernmental organization formed to discuss policy issues related to the Arctic region. The council had convened a team of scientists to survey available scientific information related to climate change and the Arctic. Impacts of a Warming Arctic: Highlights (Arctic Council, 2004) was released to considerable international media fanfare, and prompted hearings before a US Senate committee on November 16, 2004 (the full report did not appear until August 2005). Among other things, the Highlights document stated that the Arctic region was warming faster than the rest of the world, that the Arctic was now warmer than at any time since the late 19th century, that sea-ice extent had declined 15 to 20 percent over the past 30 years and that the area of Greenland susceptible to melting had increased by 16 percent in the past 30 years.
Shortly after its publication, critics started noting on web sites that the main summary graph (Arctic Council, 2004, Highlights: 4) showing unprecedented warmth in the Arctic had never appeared in a peer-reviewed journal (Taylor, 2004; Soon, Baliunas, Legates, and Taylor, 2004), and the claims of unprecedented warming were at odds with numerous published Arctic climate histories in the peer-reviewed literature (Michaels, 2004). Neither the data used nor an explanation of the graph’s methodology were made available (Taylor, 2004; Soon, Baliunas, Legates, and Taylor, 2004). When the final report was released eight months later, it explained that they had used only land-based weather stations, even though the region is two-thirds ocean, and had re-defined the boundaries of the Arctic southwards to 60N, thereby including some regions of Siberia with poor quality data and anomalously strong warming trends. Other recently published climatology papers that used land- and ocean-based data had concluded that the Arctic was, on average, cooler than it had been in the late 1930s (Polyakov et al., 2002). But while these studies were cited in the full report, their findings were not mentioned as caveats against the dramatic conclusions of the ACIA summary, nor were their data sets presented graphically.
This example indicates that empirical claims in assessment reports may need to be audited if they present new data or calculations; or to ensure that the findings are based on published, peer-reviewed journal articles (which themselves can be audited) if the mandate of the panel doing the report is confined to citing only published research. It also highlights the importance of timeliness. If a summary document is released to great fanfare, and contradictory information is quietly disclosed eight months later, the later information may not affect the way the issue was framed by the summary.
Of course, it might be argued that even if the presentation of science, or even the science itself, is in specific instances not ideal, this can be overlooked where that science lends support to an overwhelming need to act quickly to avert catastrophe. However plausible this might seem in the realm of climate change (and I leave that an open question), it is open to a very familiar criticism: how can we, in the absence of properly enforced procedural guarantees, ensure that such examples of less-than-desirable method and practice are limited to those that lend their supprot to an overwhelming public policy objective? My suspicion is that this would be well-nigh impossible. Moreover, and more basically, I can't imagine that any scientist would find this type of pragmatic subversion of the scientific method to be an intellectually satisfying solution.
The authors conclude with a number of recommendations designed to increase "due dilligence" (transparency of data, method and code), aimed at journals, researchers, funding bodies, and policy-makers. There seems to me to be no reason that these recommendations could not be framed as requirements where public authorities rely on scientific work in taking important policy decisions, as part of a administrative law of science (indeed, it is worth noting that McIntyre is no stranger to the use of US Freedom of Information Act in order to get access to the data he desires). It may even be that we are beginning to see the emergence of this type of thing globally, as rules are developed on the legitimate use of science by administrative authorities (particularly here by the WTO, and the scientific evidence required to justify the adoption of trade-restrictive measures under either Article XX GATT, or the provisions fo the SPS and TBT agreements). This would be an interesting article, that I hope to have time to write one day...
***DISCLAIMER***
As always, when writing blogs that refer to the Climate Audit site and to other works that might be seen as challenging the broad scientific consensus on climate change, I feel that a short disclaimer might serve to prevent misunderstanding of my position: I have no grasp whatsoever of the science involved in climate change - as discussed on either the Climate Audit or Real Climate sites, for example - and thus rely exclusively upon the weight of scientific authority. Here, it seems clear: that global warning is 1) happening; 2) our fault; and 3) very scary (although it might be worth mentioning in passing that I have never seen McIntyre "deny" any of these points). This is therefore not the place to discuss such issues; and I am not the person to discuss them with.
My interest in the Climate Audit site comes exclusively from the fact that regardless of the science, I find McIntyre's "due diligence" arguments to be compelling, due to a mixture of my interest in global administrative law, which I think could have an important role to play in ensuring that global public policy is based upon sound science; and my increasing amateur interest in what constitutes "good science" in general (which I owe almost entirely to this man).
Empirical research in academic journals is often cited as the basis for public policy decisions, in part because people think that the journals have checked the accuracy of the research. Yet such work is rarely subjected to independent checks for accuracy during the peer review process, and the data and computational methods are so seldom disclosed that post-publication verification is equally rare. This study argues that researchers and journals have allowed habits of secrecy to persist that severely inhibit independent replication. Non-disclosure of essential research materials may have deleterious scientific consequences, but our concern herein is something different: the possible negative effects on public policy formation. When a piece of academic research takes on a public role, such as becoming the basis for public policy decisions, practices that obstruct independent replication, such as refusal to disclose data, or the concealment of details about computational methods, prevent the proper functioning of the scientific process and can lead to poor public decision making. This study shows that such practices are surprisingly common, and that researchers, users of research, and the public need to consider ways to address the situation. We offer suggestions that journals, funding agencies, and policy makers can implement to improve the transparency of the publication process and enhance the replicability of the research that is published.
The paper discusses a number of cases in which science has formed the basis for important policy decisions and in which, in the authors' view, significant questions have emerged over the full disclosure and transparency of the data and methods used in generating the results. Firstly, and in many ways most strikingly, the authors insist that the pre-publication "peer-review" process, often presented as the arbiter of sound science, is - or should be - only in fact the beginning of the results verification and replication process. They quote the editor of the prestigious journal Science as saying
What we can’t do is ask our peer reviewers to go into the laboratories of the submitting authors and demand their lab notebooks. Were we to do that, we would create a huge administrative cost, and we would in some sense dishonor and rob the entire scientific enterprise of the integrity that 99.9 percent of it has ... it all depends on trust at the end, and the journal has to trust its reviewers; it has to trust the source. It can’t go in and demand the data books.
The most important part begins, then, when articles are published, and other scientists begin to try to replicate the results. It is in order to facilitate this crucial part of the process that, the authors argue, full disclosure of all data, code and methods used is needed to make the replication process as straightforward as possible. Whatever one thinks of the scientific issues raised by the author - and here, as ever, I plead straightforward ignorance - I find it difficult to come up with reasonable counter-arguments against this claim.
From a global administration perspective, there are two points of particular interest: firstly, the authors outline the "hockey stick" controversy, in which the IPCC was a significant player; and secondly, they refer to another case involving climate science relied upon by an international organisation, in a passage worth quoting at some length:
In late 2004, a summary report entitled the Arctic Climate Impact Assessment (ACIA) was released by the Arctic Council, an intergovernmental organization formed to discuss policy issues related to the Arctic region. The council had convened a team of scientists to survey available scientific information related to climate change and the Arctic. Impacts of a Warming Arctic: Highlights (Arctic Council, 2004) was released to considerable international media fanfare, and prompted hearings before a US Senate committee on November 16, 2004 (the full report did not appear until August 2005). Among other things, the Highlights document stated that the Arctic region was warming faster than the rest of the world, that the Arctic was now warmer than at any time since the late 19th century, that sea-ice extent had declined 15 to 20 percent over the past 30 years and that the area of Greenland susceptible to melting had increased by 16 percent in the past 30 years.
Shortly after its publication, critics started noting on web sites that the main summary graph (Arctic Council, 2004, Highlights: 4) showing unprecedented warmth in the Arctic had never appeared in a peer-reviewed journal (Taylor, 2004; Soon, Baliunas, Legates, and Taylor, 2004), and the claims of unprecedented warming were at odds with numerous published Arctic climate histories in the peer-reviewed literature (Michaels, 2004). Neither the data used nor an explanation of the graph’s methodology were made available (Taylor, 2004; Soon, Baliunas, Legates, and Taylor, 2004). When the final report was released eight months later, it explained that they had used only land-based weather stations, even though the region is two-thirds ocean, and had re-defined the boundaries of the Arctic southwards to 60N, thereby including some regions of Siberia with poor quality data and anomalously strong warming trends. Other recently published climatology papers that used land- and ocean-based data had concluded that the Arctic was, on average, cooler than it had been in the late 1930s (Polyakov et al., 2002). But while these studies were cited in the full report, their findings were not mentioned as caveats against the dramatic conclusions of the ACIA summary, nor were their data sets presented graphically.
This example indicates that empirical claims in assessment reports may need to be audited if they present new data or calculations; or to ensure that the findings are based on published, peer-reviewed journal articles (which themselves can be audited) if the mandate of the panel doing the report is confined to citing only published research. It also highlights the importance of timeliness. If a summary document is released to great fanfare, and contradictory information is quietly disclosed eight months later, the later information may not affect the way the issue was framed by the summary.
Of course, it might be argued that even if the presentation of science, or even the science itself, is in specific instances not ideal, this can be overlooked where that science lends support to an overwhelming need to act quickly to avert catastrophe. However plausible this might seem in the realm of climate change (and I leave that an open question), it is open to a very familiar criticism: how can we, in the absence of properly enforced procedural guarantees, ensure that such examples of less-than-desirable method and practice are limited to those that lend their supprot to an overwhelming public policy objective? My suspicion is that this would be well-nigh impossible. Moreover, and more basically, I can't imagine that any scientist would find this type of pragmatic subversion of the scientific method to be an intellectually satisfying solution.
The authors conclude with a number of recommendations designed to increase "due dilligence" (transparency of data, method and code), aimed at journals, researchers, funding bodies, and policy-makers. There seems to me to be no reason that these recommendations could not be framed as requirements where public authorities rely on scientific work in taking important policy decisions, as part of a administrative law of science (indeed, it is worth noting that McIntyre is no stranger to the use of US Freedom of Information Act in order to get access to the data he desires). It may even be that we are beginning to see the emergence of this type of thing globally, as rules are developed on the legitimate use of science by administrative authorities (particularly here by the WTO, and the scientific evidence required to justify the adoption of trade-restrictive measures under either Article XX GATT, or the provisions fo the SPS and TBT agreements). This would be an interesting article, that I hope to have time to write one day...
***DISCLAIMER***
As always, when writing blogs that refer to the Climate Audit site and to other works that might be seen as challenging the broad scientific consensus on climate change, I feel that a short disclaimer might serve to prevent misunderstanding of my position: I have no grasp whatsoever of the science involved in climate change - as discussed on either the Climate Audit or Real Climate sites, for example - and thus rely exclusively upon the weight of scientific authority. Here, it seems clear: that global warning is 1) happening; 2) our fault; and 3) very scary (although it might be worth mentioning in passing that I have never seen McIntyre "deny" any of these points). This is therefore not the place to discuss such issues; and I am not the person to discuss them with.
My interest in the Climate Audit site comes exclusively from the fact that regardless of the science, I find McIntyre's "due diligence" arguments to be compelling, due to a mixture of my interest in global administrative law, which I think could have an important role to play in ensuring that global public policy is based upon sound science; and my increasing amateur interest in what constitutes "good science" in general (which I owe almost entirely to this man).
2 comments:
Ahhh.... but there's the rub. You 1. see the need for due diligence, 2. accept the consensus of scientists, who 3. reject the need to subject themselves to due diligence.
Regardless of the particular issues at hand, I hope the natural question that arises is: if "3" is true, then how can I trust "2"?
It's a fair point.
However, we have to make the best of the knowledge available to us, while advocating for reforms that will allow us to be more confident in the accuracy of that knowledge. Anyway, it seems to me far from clear that your point 3 can be applied to the entire scientific consensus on climate change...
Post a Comment