Jeff Wang Blog

This blog post uses information and experiments extracted from the latest book I’ve been listening to on Audible: “Naked Statistics: Stripping the Dread from the Data”

In economics, in sciences, in social sciences, in politics, we care about what works. But how effective are certain treatments? 

Placebos can be quite powerful. Sometimes they are just as effective as the actual treatment. We don’t want that to happen.

In statistics, we try to discern between cause-and-effect treatments with a counterfactual. A counterfactual is what would have happened without the treatment. In science experiments, we create a counterfactual with a control group and placebos. But even then, many times it is difficult to compile a control in certain inhumane or unethical experiments. 

And in life, in our own decisions, how are we meant to create counterfactuals?

Is your life better or worse having decided to attend university?

Well. The only “right” answer a statistician could offer you is “we don’t know”

Because in your case of life decisions, we really don’t know how your life would be different had you not attended university.

Maybe had you not decided to attend university on decision day, you somehow decided to buy a lottery ticket and miraculously won $100 million dollars. But maybe because you won $100 million dollars and you didn’t have a university education and graduated with a finance degree like you had planned to, so you didn’t have a good understanding of financial management and somehow ended up broke within a few years, as many lottery winners do. Or maybe because you attended university, you met your soulmate in your intro to statistics class. You get the point.

In statistics, we want to be objective and use as few assumptions as we can. Even when we have really good data, we are still only fairly confident that certain cause-and-effect scenarios are true. (We call this confidence intervals).

So let’s go back to the question of a university as a treatment. How effective is a university?

Well, it’s quite hard to determine because we don’t really have good data. We have data that proves that the most selective universities (Ivy league schools) have alumni with a much higher mean salary. But let me just tell you that that statistic is absolutely useless. How do we know that the selective university, as a treatment, caused the higher salary? It’s very likely that the type of student who not only be admitted to a selective university but also can survive 4-years that top academic institution is the type of person to be more “successful” later in life. (There are many different metrics of measuring success, all which are incredibly arbitrary, but for the sake of this example, we’ll simply use salary.) So we can’t know unless we have some sort of control group, or counterfactual. But that’s quite hard because that would mean we would need to assign random students to random institutions and then track their progress longitudinally. This is an example unethical method of creating a control group I mentioned earlier. I doubt that selective universities would accept random student admissions and I’m not quite sure that students would want to be randomly sent to an institution either.

Thankfully there was a natural way of creating a control group discovered by Alan Krueger and Stacy Dale to determine the effectiveness of selective universities: students who were accepted to top institutions, but chose to attend a less selective university. This data was good because 1) the participants (students) were more or less random, 2) the students were “good” enough to be admitted to top colleges, so at the point of separation, they were “equal” to the students who DID attend the top colleges they were admitted to, and 3) there was enough data of students to actually perform analysis over long periods. Here is the report: https://www.nber.org/papers/w7322

And what did this report tell us?

Well, in terms of whether or not a top university “treatment” affected mean salary, it didn’t (Except for low-income students, who did have a noticeable increase in earnings over time). 

This is reassuring to people high-school students who are in the midst of their stressful college applications.

Although earnings isn’t the best metric, and even this analysis isn’t apples-to-apples, it is a interesting statistical report.

The university does not make the student. Hard-working students will thrive regardless of attending Harvard or their local community college.

Good luck to everyone in finals exam season.