Most of the findings that are reported in Science/Cell/Nature cannot be even replicated (in another words probably not even true). There was a study 5-6 years ago, a team tried to replicate the main findings of 100 papers published in these three journals and only able to replicate 2!!! yes, 2 out of 100! they also said though because some of the experiments required very high expertise this number can be too low but in any case, when I look at the quality of these studies my take is 1/10 is true 9/10 is there just because of how "sexy" the idea is and people who wrote them are "famous".
Yes, I am well aware of the awkward fact that many studies in high impact journals cannot be replicated. This is a fact in not only Science/Nature/Cell, but in almost all scientific journals, especially biology/medicine fields. However, that does not mean the findings are not true.
Two main reasons could contribute to the low reproducibility: (1) false data, (2) different conditions used by different labs when trying to replicate the
presumably same experiments.
No need to further explain (1). Very simple. Many labs have generated false data, either intentionally or unintentionally (using wrong cells, wrong/expired reagents, wrong way of analyzing the data and statistics, etc). This should be flat out stopped. No questions!
Let me explain (2). I'll describe some personal experience to explain. About 12 years, I just got my PhD and started my first postdoc in a physiology lab. At the time, my new lab was in the middle of a confusing and potentially disastrous situation. A new lab technician was having a hard time replicating a key set of experiments that had been done by a long-time technician in the lab. Her data was completely the opposite of the data generated by the old technician. No matter how much she tried, her data was consistently contradictory of his... The old technician's data had been published. You can imagine what kind of implication could follow... My mentor was confused and the two technicians furiously maintained that they themselves did nothing wrong. Yet, how do you explain the opposite results???
My mentor then suggested that two postdocs in the lab then re-do the experiments, completely blinded. Then two addition students in the lab did the data analysis, also completely blinded. And... They then generated opposing results, one agreeing with the old technician and the other agreeing with the new tech. What the heck is going on?????? Everybody was so confused.
Then Nature published an article about how the sex of technicians influences how lab animals respond to stimulations. They proposed that lab mice respond to male technicians differently than female technicians because they respond to the different hormones secreted by the technicians. Especially, the male technicians' testosterone pushes the mice to respond differently than the female hormones. Then a research fellow in our lab suggested that this could be the reason for our confusing data. We then looked at our data again, and found that, interestingly, the male technician and the male postdoc generated the same results, while the female technician and the female postdoc gave the same results. It looks promising. Then we asked the female tech and female postdoc to repeat the experiments while wearing their husband/boyfriend's clothes. Lo and behold! Their new results became the same as the male researchers! So we immediately published our new data comparing male and female technicians. Turn lemons into lemonade, I guess...
So we had two opposing sets of data. Is one of them untrue? Both sets of data are true. The only difference is that the experiments were performed under different conditions (male tech vs. female tech). Both data sets show how mice respond differently to different conditions. We were simply unaware that we had unintentionally changed the condition of the experiments.
This kind of unintentional changes of conditions by different labs occur frequently. Even in the same lab, this kind of things happen all the time. Several years ago when I just started my own lab, I was still using the same fancy reagents that I used in my previous mentor's lab. My mentor is a leading expert in cancer research and is loaded with funding. So he doesn't care about cash and uses the expensive stuff. As a new investigator on my own, I couldn't afford the expensive stuff. So when a sales rep from another reputable company came to my office and suggested that I should use their cheaper serum. I happily took up their offer. All of a sudden, all my experiments stopped working. I freaked out! What the heck!! Even the simplest experiments stopped working. My cells all look differently than before. I called the sales rep into my office and yelled at her (well, if I were a little less freaking out, I wouldn't yell at her. She's very cute...). After days of back and forth, we finally figured out that my new serum was from a new batch of baby cows that were fed with different chow. And these new baby cows ended up secreting different amount of hormones and changed the chemical composition of the serum, which affected my cells. The company apologized and refunded me for my batch of serum.
As you can see, we caught this change because the change happened in my own lab and I noticed the difference. If a different lab used the same new serum and generated the new data that will be completely different from my earlier published data, they will publish it and say "see? their earlier experiments are invalid!".
This kind of confusion happens almost on a daily basis among different labs that all use different brands of reagents. This is why in high impact journals, we are now required to submit the company names, catalogue numbers, batch numbers, and expiration dates for all the reagents that we use.
I also have other researchers approaching me at conferences and tell me that they can't replicate my published data. I then ask them to describe to me how they do the experiments. Almost always, I notice small but significant differences. I simply suggest that they should make such such changes since that's how we do the experiments in my lab. And almost always, they come back and tell me that they are able to replicate my data just fine once they modify their protocols. Very few labs still cannot replicate my data. I then invite them to my lab and my staff will physically train them. These labs usually lack the experiences in some of the more technically challenging methods. Physically showing them how we do it and our unique equipment almost always solves the problem.
Does that mean any of our findings are untrue? Not at all! The presumably same experiments have been done under different conditions by different labs. These opposing results simply reflect how cellular machineries respond differently to different stimulations and different conditions. It is so confusing simply because our understanding of biology is still pathetically little, to the point that we are not even aware that we are changing the conditions of the experiments...
So when you see a study published in a journal cannot be replicated, don't rush to judge. It could very well be that the finding is false. It could also be that other labs are doing, what they think, the same experiments, but in reality different experiments, and end up getting different results.
