The Hierarchy of Evidence

I was fortunate enough to get to work with the data from cross-sectional studies, case control studies and cohort studies in my first job after the grad school. In my next job I was working with data from thousands of randomized control trials. Naturally, I was trying to draw parallels and understand the complete picture. So I signed up for a few MOOC courses which helped to connect the dots. 

This posts contains my notes from the courses.

When it comes to (public health) experiments we could categorize the experiments into two buckets.

Note: This is not a rigid, universal and a perfect classification. 

An observational study is a study where the researchers only collect the data based on observations and then make some inference based on it. The researchers have no control over the variables in an observational study.

An experiment is a method of applying some intervention to a group and recording the effects. The researchers have the control over the intervention being chosen for the study.

However, the more I explored I found this constitutes only the middle spectrum in the hierarchy of evidence.

As shown in the diagram it could start with the editorials and expert opinions which are highly susceptible to personal biases and then moves up the heirarchy toward higher quality of evidence. Obviously it takes a lot of more effort to derive a high-quality evidence.  

Let's start with cross-sectional studies and work our way up to the top.

Cross-sectional studies 

• Exposure and outcome assessed at same time (single point) 
• In cross-sectional we cannot determine if exposure precedes the outcome 
• Unit used: Prevalence ratio and prevalence odds ratio 
• Number of people with health outcome/ number of people in study 
• Ways to use cross-sectional studies
• Characterize the prevalence of a health outcome in a specified population, in a defined point of time 
• Obtain data on the prevalence of exposure and health outcome

Case-Control Studies

In contrast with cohort studies, subjects are selected because they either have the disease (cases) or they don’t have the disease (controls) 

Three key steps 

• Define and select cases 
• Determine diagnostic criteria 
• Incident cases are preferred over prevalent 
• Define and select controls 
• Multiple controls per case ( statistical power) 
• Sometimes, Multiple control groups (to re-validate the hypothesis) 
• Compare exposure prevalence 

Preferred unit of measurement: Odds ratio

• Advantages 
• Good for rare outcomes 
• Takes less time 
• Use fewer resources 
• Can examine multiple exposure 
• If sampled accurately, odds ratio provides estimate of risk or rate

• Disadvantages 
• Some possible biases in selections of subjects 
• No direct estimate proxy for risk or risk ratio

Cohort studies

• Foundation for other non-experimental studies
• Cohort: Group of people sharing common characteristics
• Epidemiologist may define based on several characteristics (could be occupation, geographic location, gender)
• Examples: Framingham, Women Health Initiative
• Cohort studies measure incident cases. Outcome must not be present at baseline.
• Subjects who already have the outcome of interest (like lung cancer or cardiovascular disease) are removed
• Key exposure characteristics assessed at baseline in all subjects and subjects are followed over the time
• New cases of the outcome are measured
• Unit: risk or rate
• Useful for uncommon and rare exposure

• Classification
• Prospective: Cohorts are followed in future 
• Retrospective/historical cohort: Possible when historical records exists about baseline data

• Classification (on another dimension)
• Open study population: New subject are recruited continuously 
• Closed study population: No new subject are added

• Disadvantage 
• Expensive 
• Time-consuming 
• Inefficient for rare outcomes 
• Loss to follow up may lead to bias

Randomized Control Trials

• Intervention is determined by the experiment conductors
• Randomized: The researchers decide randomly as to which participants in the trial receive the new treatment and which receive a placebo, or fake treatment.
• Controlled: The trial uses a control group for comparison or reference. In the control group, the participants do not receive the new treatment but instead receive a placebo or reference treatment
• Advantages
• Significantly stronger evidence generation than observational studies
• Easier to blind/mask than observational studies
• Results can be analyzed with well known statistical tools
• Populations of participating individuals are clearly identified
• Disadvantages
• Expensive in terms of time and money
• Treatments that are more invasive, involving devices or surgery, may be impossible to mock-up realistically in the comparison group.
• Too few people might have a certain disease and also be available for investigation in both treatment and non-treatment groups.
• The recruitment of participants to a particular trial may be too difficult.

Now next part is extremely interesting.

You might have observed that, we sometime hear contradictory outcomes from 2 different experiments. For example, benefits of homeopathy is a largely debated topic and you would find experiments supporting both sides of the argument. So how do we get to the bottom of it?

Well, the answer is Meta-analysis or Systematic Reviews. They help us use findings from multiple experiments and come up with a single statistically significant outcome to settle the debate. Interestingly, it is another matter of debate which one of these two is a better way to evaluate the evidence from multiple studies.