I am reading 'Bad pharma' by Ben Goldacre, which details how scientifc results are fraudolently manipulated, and I had to think of it yesterday. Evangelos Evangelou gave a seminar of meta-analysis and validation in whole genome studies. One peculiar fact in medical genetics is that replicating results is somewhat iffy, and multiple studies need to be looked at to get a handle on what is going on, and special analytical tools (meta-analysis as I was saying) need to be used because not all results agree with each other. The issue of meta analysis is that it rests on the availability of all studies on a specific topic to actually get a proper result -- if we do not have all the data we cannot get the true effect.
At the end of the talk, innocent like a baby, I raised my little hand to point out that there is a recent trend for journals, especially big ones, to publish papers only if the authors 'replicated' their main findings in some other dataset. The 'optimist' explanation for this demand is that authors are required to show that their results hold in multiple 'independent' dataset, making the results more reliable. Because I have seen quite a lot of water pass under the bridge, in my question I pointed out that, far from forcing people to publish reliable results, this practice can backfire big time. Let us assume that group X finds some sexy result that might get a paper in high impact factor journal Y. Group X knows that they have to replicate their findings, or no cookie. Will group X carefully look at all other available data and present whatever they find, even if the results are 'inconclusive', or will group X leave aside all the data that does not help their replication, finding some post hoc reasons why it is ok to do so? (and will journal Y support groups that have the balls to admit their sexy results look less sexy after replication attempts?). And if this kind of behaviour becomes common, what effect would it have on meta-analysis and systematic reviews?
Fact is, I actually have been told of instances where data that did not replicate some result were left aside for publication (don't worry, the plausible deniability spiel was well rehearsed in case people asked).
So, is the demand that people replicate the very results they need for their publication (and careers) a stupid move after all? wouldn't it be better if a journal publishes all the replication efforts from different groups after it publishes the first results? This way there would be an incentive for proper independent replication. Call me a cynic but asking people to verify *themselves* the message they want to publish seems to be the wrong way around to do things.
At the end of the talk, innocent like a baby, I raised my little hand to point out that there is a recent trend for journals, especially big ones, to publish papers only if the authors 'replicated' their main findings in some other dataset. The 'optimist' explanation for this demand is that authors are required to show that their results hold in multiple 'independent' dataset, making the results more reliable. Because I have seen quite a lot of water pass under the bridge, in my question I pointed out that, far from forcing people to publish reliable results, this practice can backfire big time. Let us assume that group X finds some sexy result that might get a paper in high impact factor journal Y. Group X knows that they have to replicate their findings, or no cookie. Will group X carefully look at all other available data and present whatever they find, even if the results are 'inconclusive', or will group X leave aside all the data that does not help their replication, finding some post hoc reasons why it is ok to do so? (and will journal Y support groups that have the balls to admit their sexy results look less sexy after replication attempts?). And if this kind of behaviour becomes common, what effect would it have on meta-analysis and systematic reviews?
Fact is, I actually have been told of instances where data that did not replicate some result were left aside for publication (don't worry, the plausible deniability spiel was well rehearsed in case people asked).
So, is the demand that people replicate the very results they need for their publication (and careers) a stupid move after all? wouldn't it be better if a journal publishes all the replication efforts from different groups after it publishes the first results? This way there would be an incentive for proper independent replication. Call me a cynic but asking people to verify *themselves* the message they want to publish seems to be the wrong way around to do things.