Reporting on methods

Austin Frakt has the definitive take on the new McKinsey report:

As someone who does research, this really bothers me. It should bother you too. Look, anybody can say what they like on a topic. They can put out a glossy report. They can claim they did a “survey” to make it sound scientifically rigorous. They can talk to the media all about it. They can stand behind their good name and reputation, if they have one. But when what they’re saying runs counter to previous experience and other credible estimates, they’d better have a good explanation.

But, McKinsey has no explanation. None. They’re stonewalling. You know what would happen to me if I tried that? Suppose I sent my new results to a journal, results that were very different from that of others, and said, “Trust me. They’re good.” Well, my paper would be laughed out of the editorial office.

And that’s as it should be. That would not be research. That would be the opposite of research. That would be indistinguishable from making things up. Well, anybody can make things up. The difference between making things up and actually doing some sound, rigorous work is the difference between fiction and reality.

It is no surprise that I, also, see the methods as being critical. Some extremely misleading associations can occur depending on how the sample was developed and how the survery was conducted. Of special concern:

"Our survey shows significantly more interest in alternatives to ESI [employer sponsored insurance] than other sources do, for several reasons," the report says. "Interest in these alternatives rises with increasing awareness of reform, and our survey educated respondents about its implications for their companies and employers before they were asked about post-2014 strategies. The propensity of employers to make big changes to ESI increases with awareness largely because shifting away will be economically rational not only for many of them but also for their lower-income employees, given the law's incentives.

So the survey educated employers on how it would be economically rational to not give health insurance and then 70% of them still decided to do so!! This is the precise opposite of the headline -- that employers will tr to keep covering employees even if it is not an optimal strategy. Probably because they want to do right by their employees.

Why is this key issue not getting a lot more attention in the interpretation of the report? And why is McKinesy's reputation not being brought into question given that they released such a report?

update: The fallout continues.