Excellent overview of why scientific studies may be far less than objective, and should only be trusted when key metrics are met.
...
Publication bias
...most big, high quality studies are carried out by pharmaceutical companies. Obviously, this is a problem, because the companies have a vested interest in making their products look good. And when companies carry out studies that don’t show their drugs in the best light, they will usually try to bury the data...
This contributes to a problem known as publication bias... studies you can find on a topic often aren’t all the studies. You are most likely to find the studies that show the strongest effect. The effect of an intervention in the published literature is pretty much always bigger than the effect subsequently seen in the real world. This is one reason why I am skeptical to drugs, like statins, that show an extremely small benefit even in the studies produced by the drug companies themselves.
...Most serious journals have now committed to only publish studies that have been listed on clinicaltrials.gov prior to starting recruitment of participants, which gives the pharmaceutical companies a strong incentive to post their studies there. This is a hugely positive development, since it makes it a little bit harder for the pharmaceutical companies to hide studies that didn’t go as planned.
Peer review
Peer-review provides a sort of stamp of approval, although it is questionable how much that stamp is worth...
Generally the position of peer-reviewer is an unpaid position, and the person engaging in peer-review does it in his or her spare time. He or she might spend an hour or so going through the article before deciding whether it deserves to be published or not. Clearly, this is not a very high bar. Even the most respected journals have published plenty of bad studies containing manipulated and fake data because they didn’t put much effort in to making sure the data was correct. As an example, the early part of the covid pandemic saw a ton of bad studies which had to be retracted just a few weeks or months after publication because the data wasn’t properly fact checked before publication.
...The guiding principle is the idea that bad studies will be caught out over the long term, because when other people try to replicate the results, they won’t be able to.
There are two big problems with this line of thinking. The first is that scientific studies are expensive, so they often don’t get replicated... And if the drug company has done one study which shows a good effect, it won’t want to risk doing a second study that might show a weaker effect. The second problem is that follow-up studies aren’t exciting... No-one cares about the people who re-did a study and determined that the results actually held up to scrutiny.
Different types of evidence
In medical science, there are a number of “tiers” of data. The higher tier generally trumps the lower tier, because it is by its nature of higher quality. This means that one good quality randomized controlled trial trumps a hundred observational studies.
The lowest quality type of evidence is anecdote... Anecdotal evidence can generate hypotheses for further research, but it can never say anything about causation...
After anecdote, we have observational studies. These are studies which take a population and follow it to see what happens to it over time. Usually, this type of study is referred to as a “cohort study”, and often, there will be two cohorts that differ in some significant way... observational studies can never answer the question of causation... This is extremely important to be aware of, because observational studies are constantly being touted in the media as showing that this causes that...
The highest tier of evidence is the Randomized Controlled Trial (RCT). In a RCT, you take a group of people, and you randomly select who goes in the intervention group, and who goes in the control group.
...
There are those who would say that there is another, higher quality form of evidence, above the randomized controlled trial, and that is the systematic review and meta-analysis... meta-analysis is a systematic review that has gone a step further, and tried to combine the results of several studies in to a single “meta”-study, in order to get a higher amount of statistical power... The reason I say it’s both true and not true that this final tier is higher quality than the RCT is that the quality of systematic reviews and meta-analyses depends entirely on the quality of the studies that are included...
Statistical significance
One very important concept when analyzing studies is the idea of statistical significance. In medicine, a result is considered “statistically significant” if the ”p-value” is less than 0,05 (p stands for probability)... 5% is an entirely arbitrary cut-off... Personally, I think a p-value of 0,05 is a bit too generous. I would much have preferred if the standard cut-off had been set at 0,01, and I am sceptical of results that show a p-value greater than 0,01...
The 0,05 limit is only really supposed to apply when you’re looking at a single relationship. If you look at twenty different relationships at the same time, then just by pure chance one of those relationships will show statistical significance. Is that relationship real? Almost certainly not... The reason researchers are supposed to post the primary endpoint at clinicaltrials.gov before starting a trial is that they can otherwise choose the endpoint that ends up being most statistically significant just by chance, after they have all the results... That is of course a form of statistical cheating. But it has happened, many times.
Absolute risk vs relative risk
Let’s say we have a drug that decreases your five year risk of having a heart attack from 0,2% to 0,1% . We’ll invent a random name for the drug, say, “spatin”. Now, the absolute risk reduction when you take a spatin is 0,1% over five years (0,2 – 0,1 = 0,1). Not very impressive, right? Would you think it was worth taking that drug? Probably not...
How can a spatin only decrease risk by 0,1% and yet at the same time decrease risk by 50%? Now you’d definitely want to take the drug, right? ... When you look at an advertisement for a drug, always look at the fine print. Are they talking about absolute risk or relative risk?
How a journal article is organized
Introduction... mostly fluff...
Method... important section and you should always read it carefully...
There are a few methodological tricks that are very common in scientific studies. One is choosing surrogate end points and another is choosing combined end points. I will use statins to exemplify each, since there has been so much methodological trickery in the statin research.
Surrogate end points are alternate endpoints that “stand in” for the thing that actually matters to patients... By using a surrogate end point, researchers can claim that the drug is successful when they have in fact showed no such thing. As we’ve discussed previously, the cholesterol hypothesis is nonsense, so showing that a drug lowers LDL cholesterol does not say anything about whether it does anything clinically useful.
Another example of a surrogate endpoint is looking at cardiovascular mortality instead of overall mortality. People don’t usually care about which cause of death is listed on their death certificate. What they care about is whether they are alive or dead...
...example of a combined end point is looking at the combination of overall mortality and frequency of cardiac stenting. Basically, when you have a combined end point, you add two or more end points together to get a bigger total amount of events...
...Another trick is choosing which specific adverse events to follow, or not following any adverse events at all. Adverse events is just another word for side effects. Obviously, if you don’t look for side effects, you won’t find them.
Yet another trick is doing a “per-protocol analysis”... anyone who dropped out of the study because the treatment wasn’t having any effect or because they had side effects, doesn’t get included in the results... The alternative to a per-protocol analysis is an “intention to treat” analysis. In this analysis, everyone who started the study is included in the final results, regardless of whether they dropped out or not...
The third section of a scientific article is the results section, and this is the section that everyone cares most about. This is just a pure tabulation of what results were achieved, and as such it is the least open to manipulation, assuming the researchers haven’t faked the numbers... I think most researchers are honest...
There is however one blatant manipulation of the results that happens frequently. I am talking about cherry picking of the time point at which a scientific study is ended... If the results are promising, they will often choose to stop the study at that point, and claim that the results were “so good that it would have been unethical to go on”. The problem is that the results become garbage from a statistical standpoint... . Never trust the results of a study that stopped early.
...The fourth section of a scientific article is the discussion section, and like the introduction section it can mostly be skipped through. Considering how competitive the scientific research field is, and how much money is often at stake, researchers will use the discussion section to try to sell the importance of their research, and if they are selling a drug, to make the drug sound as good as possible.
In conclusion, focus on the method section and the results section...
Final words
My main take-home is that you should always be skeptical. Never trust a result just because it comes from a scientific study. Most scientific studies are low quality and contribute nothing to the advancement of human knowledge. Always look at the method used. Always look at who funded the study and what conflicts of interest there were.
WIND: scientific studies around medicine and human health are so dangerous to health and the economy (e.g., statins) that a legal framework should be established that requires full disclosure of data*, independent confirmatory analysis, and severe penalties (jail time) for undisclosed conflicts of interest or business relationships. The stringency should be no less tight than for financial companies.
But since Congress and the “news” and the FDA are in the pockets of Big Pharma, don’t expect any integrity to emerge anytime soon.
SEARCH
