Fake Data in Political Science

This is Joseph.

One of the most challenging things in population research is the need to trust in the data and analysis done by other groups.  Unlike chemistry, we cannot simply replicate experiments without huge amounts of expense.  Furthermore, the population is getting less and less responsive to surveys.  In a very real sense, endlessly replicating strong and clean results is going to partially displace other research questions.  After all, people have a limited tolerance for surveys.  This goes double for high burden approaches, such as door-to-door interviews and interventions (which require trained and paid field agents to conduct the survey with a high degree of professionalism and often limited wages).

This need to trust, where possible, makes stories like this one painful.  Full respect to the field that the problem was detected and I am glad that academia was self correcting.  But these actions have pretty strong consequences

I also think there is a very important difference between a technical error, misunderstanding of data, and completely making data up.  The first two are the cases that give every analyst nightmares.  But the last seems to have no excuses at all -- how could somebody not know that they were faking data? 

That said, it's not like medicine is innocent (as Thomas Lumley points out) and medical research probably has a lot more direct potential to cause harm (as patient concerns about this "treatment is not working" will be dismissed in the face of "randomized controlled trial" "evidence"). 

EDIT: and how could I overlook Andrew Gelman's take on this (which is right in his area)

Blast from the past -- The curse of large numbers and the real problem with p-values

Following up on Joseph's recent piece.

[Originally posted MARCH 22, 2010]

(Some final thoughts on statistical significance)

The real problem with p-values isn't just that people want it to do something that it can't do; they want it to do something that no single number can ever do, fully describe the quality and reliability of an experiment or study. This simply isn't one of those mathematical beasts that can be reduced to a scalar. If you try then sooner or later you will inevitably run into a situation where you get the same metric for two tests of widely different quality.

Which leads me to the curse of large numbers. Those you who are familiar with statistics (i.e. pretty much everybody who reads this blog) might want to skip the next paragraph because this goes all the way back to stat 101.

Let's take simplest case we can. You want to show that the mean of some group is positive so you take a random sample and calculate the probability of getting the results you saw or something more extreme (the probability of getting exactly results you saw is pretty much zero) working under the assumption that the mean of the group was actually zero. This works because the bigger the samples you take the more the means of those samples will tend to follow a nice smooth bell curve and the closer those means will tend to group around the mean of the group you're sampling from.

(For any teachers out there, a good way of introducing the central limit theorem is to have students simulate coin flips with Excel then make histograms based on various sample sizes.)

You might think of sampling error as the average difference between the mean of the group you're interested in and the mean of the samples you take from it (that's not exactly what it means but it's close) . The bigger the sample the smaller you expect that error to be which makes sense. If you picked three people at random, you might get three tall people or three millionaires, but if you pick twenty people at random, the chances of getting twenty tall people or twenty millionaires is virtually are next to nothing.

The trouble is that sampling error is only one of the things a statistician has to worry about. The sampled population might not reflect the population you want to draw inferences about. Your sample might not be random. Data may not be accurately entered. There may be problems with aliasing and confounding. Independence assumptions may be violated. With respect to sample size, the biases associated with these problems are all fixed quantities. A big sample does absolutely nothing to address them.

There's an old joke about a statistician who wakes up to find his room on fire, says to himself "I need more observations" and goes back to sleep. We do spend a lot of our time pushing for more data (and, some would say, whining about not having enough), but we do that not because small sample sizes are the root of all of our problems but because they are the easiest problem to fix.

Of course "fix" as used here is an asymptotic concept and the asymptote is not zero. Even an infinite sample wouldn't result in a perfect study; you would still be left with all of the flaws and biases that are an inevitable part of all research no matter how well thought out and executed it may be.

This is a particular concern for the corporate statistician who often encounters the combination of large samples and low quality data. It's not unusual to see analyses done on tens or even hundreds of thousands of sales or customer records and more often than not, when the results are presented someone will point to the nano-scale p-value as an indication of the quality and reliability of the findings.

As far as I know, no one reviewing for a serious journal would think that p<0 .001="" 99.9="" a="" almost="" an="" analytic="" background="" br="" but="" conclusion="" everyone="" is="" means="" re="" s="" sure="" that="" thinks.="" true="" we="" what="" without="">
And that is a problem.

Very good statistics post (p-value edition)

This is Joseph (and I located the piece through Economist's View)

This article on p-values was a very interesting read.  The author (who teaches statistics) has a very nice discussion of p-values:

One reason for this, I think, is that we fail to teach well how, with enough data, any non-zero parameter or difference becomes statistically significant at arbitrarily small levels. The proverbial expression of this, due I believe to Andy Gelman, is that "the p-value is a measure of sample size". More exactly, a p-value generally runs together the size of the parameter, how well we can estimate the parameter, and the sample size. The p-value reflects how much information the data has about the parameter, and we can think of "information" as the product of sample size and precision (in the sense of inverse variance) of estimation, say n/σ2. In some cases, this heuristic is actually exactly right, and what I just called "information" really is the Fisher information.

But I found this way of talking about p-values to be extremely useful, and something that should be kept in mind in Epidemiology -- where a significant association estimates from a big sample with a small effect can often be uninteresting.  You never reduce bias to zero in a real observational study and interventions rarely remove an association entirely (as not everyone changes behavior or mitigation is partial).  In the era of big data, this becomes important.

Fun stuff.  

Double-talk and Mathiness

If you haven't been following the reaction to Paul Romer's Mathiness paper, you should check it out. Here's a key quote:

Academic politics, like any other type of politics, is better served by words that are evocative and ambiguous, but if an argument is transparently political, economists interested in science will simply ignore it. The style that I am calling mathiness lets academic politics masquerade as science. Like mathematical theory, mathiness uses a mixture of words and symbols, but instead of making tight links, it leaves ample room for slippage between statements in natural versus formal language and between statements with theoretical as opposed to empirical content

Brad DeLong has had some sharp comments, but I think the best reply to Romer's critics came from Romer himself.

If you think that what McGrattan and Prescott do for location is even remotely on the same level as what Solow did for capital or what Becker did for human capital, please go read the two M-P papers (JET 2009, AER 2010.)

When you think you are too stupid to understand what they are saying and want to give up, trust me, it isn’t you. What they are saying makes no sense. No one can understand it. The authors do not understand it.

Here is a sample of what you can expect:

Technology capital is distinguished from other types of capital in that a firm can use it simultaneously in multiple domestic and foreign locations.  (Footnote: In the language of classical general equilibrium theory, a unit of technology capital is a set of technologies, with one technology for each location.) (JET 2009, p. 2455)

“A unit … is a set”? This is just gibberish. Forget about whether the model connects in any meaningful way to the real world. There is no way to make sense of this statement even in the make-believe world of the model. In the model, the authors define technology capital is a cardinal measure. It is supposed to be something that you can have 2 units of, or 4, or 10. What could 2 or 4 or 10 sets of technologies possibly mean?

We assume that the measure of a country’s production locations is proportional to its population, since locations correspond to markets and some measure of people defines a market. (JET 2009 p. 2461)

I feel guilty pulling a quote like this one, as if I’m humiliating some miserable undergraduate by reading to the class from a term paper on a project that fell apart. But remember, this is from an article that was published in the Journal of Economic Theory.

As you read this quote, remember that the motivation for the theory is that for these authors, perfect competition is the ultimate non-negotiable, more sacred even than micro-foundations. If this were a Hotelling model of location or a Krugman model of spatial location, I’d have some way to try to make sense about how “some measure of people defines a market.” But in the formal mathematical model of perfect competition that the authors are using, this sentence means nothing.

These words are untethered, undisciplined by logic or math, chosen to sound plausible enough to someone who is not paying close attention, like the set up for an applause line in a speech by a politician. This is mathiness.

There is lots more:

One unit of technology capital and z units of the composite input at a given location produce y = g(z). Consider the case of brand equity with units of technology capital indexed by m. For ease of exposition, assume for now that m is discrete and that m = 1 is the Wal-Mart brand, m = 2 is the Home Depot brand, and so on. Wal-Mart chooses the locations in which to set up stores and use its brand. It may be the case that both Wal-Mart and Home Depot have stores at the same location. (AER 2010, p. 1497.)

And if you look at the math, a company like Wal-Mart has to use one unit of technology capital for each location. Because the number of locations in the US is the US population, Wal-Mart must be using more than 300 million units of technology capital. (So more than 300 million technology sets?)

How can we reconcile the math with words that say Wal-Mart gets index m=1 and Home Depot gets m=2? And if technology capital is brand equity, why does Wal-Mart need another unit of brand equity for each US citizen/location? I haven’t a clue, but neither do the authors. One of the things that Milton Friedman got right was his observation that “confused writing is a sign of confused thinking.”

As a discussant, I put serious effort into trying to clean up the mess in the working paper that became the 2009 JET paper. I worked through the math. I talked with the authors.

The things I explained, such as how to convert any concave function like g(z) into a function with one additional variable that is homogeneous of degree one, just helped them put lipstick on this pig.

It was an embarrassment for me that the 2007 NBER version contained the acknowledgement “Discussions with Robert Lucas and Paul Romer were extremely helpful…”

One thing has been bothering me for years now is not just that economists often combine overly simplistic modeling assumptions with overly complicated math (we all do that from time to time), but that many seem to equate these things with thinking scientifically.

On the mathematical point, go back and look at “is economics a science?” debates and see how long it takes someone on the pro side to bring up how mathematically sophisticated their work is. You will frequently hear this same argument made by astrologers and other fringe science types and it has no less validity there than it does here.

This is not to say that economics is a fringe or even a soft science. While I'm not going to get into the relative hardness of different disciplines (always a waste of time), I will say that econ is a fascinating field full of smart people doing sound and important work.

My concerns are more cultural and meta. I'll try to come back and fill in some of the details later (I've got a lot on my plate now, if you'll pardon the pun), but just to paint it in broad strokes, these attitudes not only make economics look bad, they lead to all sorts of bad things like epicyclic modeling and, in the final stages, mathiness.

Which brings us to the double talk of Sid Caesar, the famous bits where he would seem to be speaking in various languages. The words were complete gibberish but by mimicking the inflection and rhythm (what he called the 'song') of each language, he could could create the impression of going from French to German to Russian and so on.

I realize Romer is making some bigger points here, but in at least one aspect, this reminds me of those Caesar routines. Instead of musical qualities, mathiness uses linguistic properties like vocabulary and syntax to create the impression of scientific reasoning. It is only when you pay attention that you realize nothing is being said.

The value(s) of a dollar

Joseph's recent post reminded me that there was an important point I should have emphasized more in the ongoing food and food security threads.

One of the main reasons social safety nets work is that, while the nominal value of money remains the same, the impact value (the difference in quality of life that a dollar makes) varies greatly from the top to the bottom of the income scale. For example, if you were in the top quartile, there is a very good chance that you would not even notice a $20 or $30 increase in your weekly food budget, while if you were in the bottom quartile, that same $20 or $30 could make a world of difference in terms of health and hunger. This is why a relatively small level of income redistribution can produce a substantial increase in aggregate quality of life.

This is also why the food stamp challenge and other similar initiatives can be so dangerous. The trouble with high-profile, upper-class activists trying to show how difficult it is to live on a program like food stamps is that these activists have fully internalized these upper-class perceptions of the impact value of money. They don't have the skill set to work with these small sums and they don't have the world view that allows them to understand their relative value.

Though the intentions are unquestionably good, the effect of these initiatives is to suggest incorrectly that these social safety net programs do little good and that incorrect belief can do a great deal of harm.

Notes on today's election

From KPCC (presented without comment):

The biggest spender in the LAUSD school board election is the California Charter Schools Association Advocates' PACs, which so far has spent $2.2 million. Their funds primarily support charter school executive Ref Rodriguez in the east Los Angeles' District 5 race and against incumbent Bennett Kayser, a charter school opponent.

"We are supportive of candidates who have got a vision of great public schools in Los Angeles, and we would like charters to be part of that solutions," said Gary Borden, executive director of CCSA Advocates.

Last year, the charter school PAC received donations from local philanthropist Eli Broad, Netflix CEO Reed Hastings and former New York City Mayor Michael Bloomberg.

The names of more recent donors aren't required to be disclosed just yet: California requires these types of PACs to publish contributors on a semi-annual  basis, often falling after Election Day.

Double Talk

Believe it or not, I am going to connect this to one of our threads.










From the Wikipedia page on Sid Caesar

Max and Ida Caesar ran a restaurant, a 24-hour luncheonette. By waiting on tables, their son learned to mimic the patois, rhythm and accents of the diverse clientele, a technique he termed double-talk, which he used throughout his career. He first tried double-talk with a group of Italians, his head barely reaching above the table. They enjoyed it so much that they sent him over to a group of Poles to repeat his native-sounding patter in Polish, and so on with Russians, Hungarians, Frenchmen, Spaniards, Lithuanians, and Bulgarians.
...
Of his double-talk routines, Carl Reiner said, "His ability to doubletalk every language known to man was impeccable," and during one performance Caesar imitated four different languages but with almost no real words. Despite his apparent fluency in many languages, Caesar could actually speak only English and Yiddish. In 2008, Caesar told a USA Today reporter, "Every language has its own music ... If you listen to a language for 15 minutes, you know the rhythm and song." Having developed this mimicry skill, he could create entire monologues using gibberish in numerous languages, as he did in a skit in which he played a German general.

Another post at the food blog

On nutritional requirements and on why people who write about food so often don't bother to research those requirements.

Why are these food stamp challengers so miserable? Because they don't know about Wikipedia.

Circa 1918

Private property

This is Joseph.

One very important shift in modern thinking has been the notional of private property as an absolute and natural right, as opposed to a social convention.  I very much like David Hume's take on this issue:

Thus, “the rules of equity or justice [regarding property] depend entirely on the particular state and condition in which men are placed, and owe their origin and existence to that utility, which results to the public from their strict and regular observance” (Enquiry Concerning the Principles of Morals, 3).

In other words, property rights arise because they are useful and are an imposed social construct by a polity.  It is a measure of how incredibly successful that the modern state has been at protecting property rights that we see property rights as an innate right.

So why do I bring this up?

Mark has been talking about how to live on extremely small amounts of money.  But it is not necessary that resources be allocated in the way that they are.  In fact, it may well be the case that the wealthiest members of society benefit the most from the conventions and constructions of society.  I don't think attacking people for being selfish is ever productive,

But I do think it might be worth considering whether a new social contract can be developed that is not anti-statist.  Perhaps, instead, we could focus on efficient and effective government.  Or am I an idealist?

More on the Business Insider food stamp challenge

When you are trying to fed yourself on a very tight budget, pick the one on the left.

Leave it to an Irishman like Pierce to throw in a puck goat

Charles Pierce continues to track the movements of Politico's Dylan Byers who continues not to get it.

Distressed at being outmaneuvered in this way, I wandered on over to the joint hosted by Dylan Byers, Tiger Beat On The Potomac's media "reporter." There, I found him where I'd left him back in 2012, lashed to the side of the white whale, andbeckoning for the Pequod's remaining whaleboats to continue the pursuit.

"[T]here are lots of reasons to worry about the state of the polling industry," Silver concluded, citing a range of factors. "There may be more difficult times ahead for the polling industry." This is quite a notable statement. The former New York Times statistician gained national fame for correctly anticipating the outcome of the 2008, 2010 and 2012 U.S. elections. He did this largely by understanding how to read the polls, and by knowing which polls were worth reading. (Never mind that he wasn't the only one. In fact, in 2012, the Daily Kos blogger Markos Moulitsas was more accurate than Silver in predicting the outcome of the 2012 electoral college. Needless to say, Moulitsas was not offered a high-paying job at ESPN.) If Silver is declaring that the world has a polling problem, and that there may be more difficult times ahead for the polling industry, what is Silver's added value in an election cycle? His ability to forecast elections is largely dependent on the accuracy of polling. Without that, what is his raison d'etre -- other than to point out how bad polling caused him to make inaccurate forecasts?

Oooooh, Nate. Burn'ch-ya! Can ya feeeeeel it?

Recall that Silver spent the end of the 2012 election cycle helping Byers look stupid in print. Byers, as is his custom, was the last one to get the joke. That Byers embarked on this doomed crusade as part of his other job as the sidewalk shill for MSNBC's ratings-challenged Morning Zoo Crew only made the whole thing more hilarious.

As Silver readily admits, the results in the UK elections on Thursday confounded his predictions -- andthose of everyone else, truth be told. His musings on the state of the polling industry are worth reading and considering as the election cycle over here grinds on. But there is one thing on which most polls agree -- having Dylan Byers question anyone else's professional raison d'etre is like subscribing to the classical music criticism of a puck goat.

Putting aside the childishness of his feud with Silver, this is yet another reminder that Byers still can't or won't face the central point.

All political journalists play the horse-race game. I personally don't see a great deal of value in this -- reporters telling voters who the reporters think the voters will vote for -- but Byers, the consummate establishment apologist, wouldn't go there at gunpoint. What Silver and company showed was that the kind of horse-race journalism Byers constantly defends is so devoid of value that labeling it 'news' borders on the fraudulent.

The Markos Moulitsas link is particularly rich. Here's what you see if you follow the link:

Math-based prognostication is superior to the old-school way of talking about gut feelings, or vibrations, or outright dishonest hackery.

Remember that Byers has emphatically thrown his support behind the old-school approach. Even those that Byers links to for support appear to think he's full of crap.

Journalists can't shop (I was way too hard on Ron Shaich)

[Also posted at A Statistician Walks into a Grocery Store...]

We been down this road before, but this time it's going to be a bit bumpier.

A couple of years ago I criticized Panera Bread CEO Ron Shaich for the way he handled what we now call the "food stamp challenge." The intention with these efforts is always good – – bringing attention to the issue of food insecurity – – but almost invariably the people taking the challenge are so far removed from below-the-poverty-line conditions and so lacking in the necessary life skills that they make a complete hash of the attempt and end up misrepresenting the problem.

Shaich made errors largely of omission. Among other things, he skimped on high-protein foods that would've helped him stave off hunger. I am starting to feel a little bad about being so harsh in that case because I now see how much worse it could have been.

This Business Insider entry by Kathleen Elkins manages to demonstrate virtually every mistake you could make when trying to feed yourself on a tight budget, starting with where not to shop.

On Monday night I headed to the most affordable grocery store I could think of: Trader Joe's.

I was super conscious of sales as I wove through TJs, and the steals of the night included: sweet potatoes ($0.49 each), bananas ($0.19 each), and a 16-ounce bag of bowtie pasta ($0.99). 

As mentioned before, Trader Joe's is not a bargain grocery. The target demographic might be best described as budget-conscious foodie. If you're looking for imported, organic and (most importantly) prepared food, you will find some very reasonable options.

For the severely price-constrained, though, Trader Joe's shouldn't be your first or even second choice. Dried rice is only available in pricey varieties. Dried beans aren't available at all. Even the supposed "steals" of the night are available elsewhere at comparable or better prices.

But, even within the confines of this third-best choice, Many of Elkins' decisions were disastrous.

Red split lentils ($1.69)
Bowtie pasta ($0.99)
Can of garbanzo beans ($0.89)
Can of black beans ($0.89)
Butternut squash soup ($2.79)
Chunky peanut butter ($2.49)
8 corn tortillas ($1.99)
Half-gallon of almond milk ($2.99)
Dozen organic eggs, since the only remaining non-organic eggs were cracked ($3.99)
8-pack of maple and brown sugar oatmeal ($2.99)
7 bananas ($1.33)
Bag of spinach ($1.99)
1 yellow onion ($0.79)
3 sweet potatoes ($1.47)
Sea salt ($0.99)
One of the biggest mistakes I made was not buying butter or oil, essential cooking ingredients that I take for granted and therefore completely overlooked.

You'll also notice there is no coffee, a staple in my normal diet but one that would blow the budget.

There are a few reasonable choices here  -- the beans, the pasta, the bananas, the sweet potatoes -- but after that it gets ugly quickly. By my calculations and based on a couple of trips to Trader Joe's for research (with the caveat that some foods are cheaper in LA), more than half of Elkins' money went to purchases that were either bad or which could be replaced by much cheaper alternatives.

Elkins bought prepared food.

(You can make your own soup much cheaper and with less sodium)


She passed over frozen produce which was a fraction of the price of fresh.

She paid extra to make questionable nutritional choices. Keeping in mind that protein is our first priority and that a half gallon of milk costs $1.99 rather than $2.99, take a look at this:

Elkins would probably have been better off going for coffee instead of almond milk (she complained bitterly about caffeine withdrawal) and she would have had a dollar left over.

(Not the best deal on coffee, but certainly doable on the budget.)

Then there's $2.99 for an 8-pack of oatmeal. Here's how much oatmeal $2.50 gets me at the grocery store down the street.

Oatmeal is remarkably nutritious and a big bowl does an excellent job staving off hunger. At less than ten cents a serving, this is one of the places we want to indulge.

Even at Trader Joe's, Elkins could have done much better.

Like the oatmeal, the eggs Elkins bought are a good food at a very bad price. Non-organic are $1.99 at my Trader Joe's. That's a pretty good price per gram of protein but it's far from the best the store has to offer.

We're shooting for 50g a day. $4.00 worth of chicken goes a long way toward hitting that target. With a few other smarter choices (including more beans, more pasta, and, of course, no almond milk), she probably could have avoided the hunger pangs altogether.

Elkins' heart is in the right place and I appreciate her willingness to suffer for a story, but this is still bad and dangerous journalism. It misses the reality of poverty, it understates the effectiveness of an excellent program, and it can be more than a little insulting to those facing these challenges for real. Mary Elizabeth Williams put it best when writing about Gwyneth Paltrow’s SNAP Challenge for Salon:

And I’ve had an awful lot of conversations with people who think they’re being insightful when they declare that it turns out it’s really hard to get a job or to stretch a dollar. That it just can’t be done. Actually, guys, it’s hard for you to be poor. Lots of us are great at it. Lots of us do it every goddamn day.

It might not be a kangaroo at all; it could just be a really big mouse

More things I wish I had time to write about.

The feather, the bathroom scale, and the kangaroo



"And some analysts note that the company hasn’t articulated a clear plan on how it will profit from these videos."



How Click Farms Have Inflated Social Media Currency

This is what happens when companies are desperate to get in on the next big thing despite having no idea why it's important or how it works.

I've been watching the national press screw up Arkansas stories all my life

A few days ago, I linked to reports of an astroturf anti-marriage equality rally in Russellville that was countered by a couple of genuine grassroots pro-equality demonstrations. I mentioned that this was an important story that the national media was largely missing.

The press did, however, pick up on this superficially similar but  utterly meaningless story about Eureka Springs.

I have lots of fond memories of Eureka Springs but, as anyone familiar with the state can tell you, it is literally the least representative spot you could find.

In addition to being very small and isolated, the town has been home to countless fringe movements, religious, political, and medical, since at least the early 20th century. It is associated as much with hippies and UFO enthusiasts as with religious fundamentalist, and all of these groups have long since made their peace and united behind the common goal of getting money from tourists. There are numerous highly convincing indications that places like Arkansas are moving dramatically toward acceptance of marriage equality, but using Eureka Springs as an example of changing attitudes is like using Knott's Berry Farm as an example of modern agricultural practices.

Getting a handle on the food stamp discussion

[Crossposted at A Statistician Walks into a Grocery Store...]

I'm going to spend quite a bit of time over the next couple of weeks  talking about the price of food and about what it means to live on a food budget of less than thirty dollars a week. Before we can get very far with that discussion, however, we need to spend some time thinking about the metrics we want to track, the conditions we want to meet, and the properties we want to optimize some of the properties we need to see.

Here are the big four I would like to start with:

Protein – – extremely important and also the only completely objective item on the list. Any proposed diet must satisfy the heart condition of 50 g of protein a week.

Taste – – trying to save money by telling people to eat unappealing food is a false economy. This will lead to problems, particularly when asking people to budget their resources over the course of the week. Obviously, there's a big subjective component here but restaurants and food companies like Kraft have shown that it is a manageable problem, especially if we do a good job with the next item on the list...

Versatility – – we are interested here in variety not on the individual but on the aggregate level. For example, eggs make a good staple because, in addition to being a good source of low-cost protein, they can also be prepared in any number of ways. This is important not because an individual will necessarily want to have all of these different dishes, but because this variety increases the likelihood of our finding one or two dishes that the individual will like.

Satiation – – meals need to be filling and to alleviate hunger. This is largely a function of fiber and protein which is yet another reason why hitting that 50 g target is so important.

Clearly this is oversimplified but it does give us some kind of a framework to proceed. Now we need to address the central question, under what conditions is it possible to have a protein-rich, appealing, varied and filling diet for $28 a week?