5.2 Causal Identification Strategies

Our goal: Try to “identify” the causal effect of one variable on an outcome. As Montell Jordan once said, this is how we do it:

  • Use data we have (that exist out in the world)
  • Compare those who are ``treated” to a relevant comparison group who is not treated

However, we can’t randomize treatment so….

  • We do our best to try to choose a good comparison (one very similar to the treatment group, but happens not to be treated)

We want to rule out all possible confounding variables and “alternative explanations” for the outcomes we observed.

5.2.1 Three Common Identification Strategies

Example: Does drinking Sprite make a person a better basketball player? (Inspired by 1990s commercial where a kid believes drinking Sprite will cause him to play basketball better.)

  • Cross-section comparison: Compare Grant Hill (who drinks Sprite) to others (who don’t)
  • Before-and-after: Compare Grant Hill after he started drinking Sprite to Grant Hill before
  • Difference-in-differences: Compare Grant Hill before and after drinking Sprite and subtract from this the difference for some other person (who never drank Sprite) during the same two periods

(Note: “drinking Sprite” is our treatment.)

5.2.2 Threats to Cross-Section Designs

Assumption: Must assume no confounders and any alternative explanations related to differences between the treated and control subjects that also relate to the outcome. The Threat: Your two groups may differ in ways beyond the “treatment” in ways that are relevant to the outcome you care about.

  • Compare Grant Hill, a tall NBA player who currently drinks Sprite (treatment group) to
  • Yourself, assuming you and they do not drink Sprite (control group)
  • Compare your basketball skill levels (the outcome).
  • Suppose Grant Hill is better (a positive treatment effect).
    • Can we conclude Sprite causes a person to be a better player?

Nope, because other things that affect basketball talent differ between you and Grant Hill, and these things, not Sprite, may explain the difference in basketball talent.

Moreover, even if we compared just among NBA players (Grant Hill vs. non-Sprite drinking players of his era), it’s possible that Sprite targeted all-stars to recruit to drink Sprite. In this way, pre-existing basketball talent (a confounder) both explains why Grant Hill drank Sprite (relates to the treatment) and explains his higher level of basketball talent (relates to the outcome) in the time period after drinking Sprite.

  • For a cross-sectional comparison to be plausible, we need to choose a very similar comparison in order to isolate the treatment as the main variable that is causing a change in an outcome.

5.2.3 Threats to Before-After Designs

Assumption: Must assume no confounding time trend. Threat: Something else may be changing over time, aside from the treatment, that is affecting your outcome.

  • Compare Grant Hill in the years after he started drinking Sprite (treated) to
  • Grant Hill the years before he started drinking Sprite (control)
  • Compare his basketball skill levels (outcome).
  • Suppose Grant Hill after Sprite is better (a positive treatment effect).
  • Can we conclude Sprite causes a person to be a better player?

Not if something else Grant Hill started doing during that time period made him better (e.g., maybe during that time the NBA provided higher quality coaches and trainers, and everyone (including Grant Hill) got better).

  • You want your treatment to be the only thing relevant to basketball talent changing over time.

5.2.4 Threats to Diff-in-Diff Designs

Assumption: Must assume parallel trends: That in the absence of treatment, your treatment group would have changed in the same way as your control

  • Compare Grant Hill in the years before vs. after he started drinking Sprite to Grant Hill’s teammate, who never drank sprite, in the same two time periods (before Hill drinks Sprite vs. after Hill drinks Sprite)
  • Compare the change in each player’s basketball skill levels. Suppose Grant Hill’s skills increased to a greater degree than his teammate’s over the same time period.
  • Can we conclude Sprite causes a person to be a better player?

If we are confident that Grant Hill did not have a unique (non-Sprite) advantage over that time period relative to other players, then our assumption might be plausible– that Grant Hill and other players would have experienced a similar growth in their skills if not for Grant Hill getting the extra benefit of Sprite.

Instead, if, for example, Grant Hill got a new trainer during this period AND his teammate did not, then we might have expected Grant Hill to see more improvement even if he didn’t start drinking Sprite. A violation of the parallel trends assumption!

  • Causality is hard!