Google -- Over-spammed or over-engineered (or both)?

Brad DeLong points to a couple of posts on the declining quality of Google searches. Jeff Attwood concisely sums up the central point:

People whose opinions I respect have all been echoing the same sentiment -- Google, the once essential tool, is somehow losing its edge. The spammers, scrapers, and SEO'ed-to-the-hilt content farms are winning.

This is certainly right as far as it goes, but the Google searches that annoy me the most are the ones where Google decides I don't want what I just asked for, sometimes even ignoring quotation marks. For example, if you query "jeff bridges true grit", about a third of the results on the first page don't contain the string "jeff bridges true grit". This is an unlikely example but I've frequently found myself looking for an obscure search term that was similar to something popular. The results were unspeakably aggravating.

The great irony here is that by taking control away from the searchers, Google is making search engine optimization easier. If you really want to screw with the people trying to reverse engineer your algorithms, make the searches more interactive. If Google let us tune our searches with dials that changed the weights of parameters such as word order or proximity of words in our search, the results would be less annoying for us and more annoying for the SEO people.

'Standard' does not mean 'sound'

Eric Schoenberg is raising some important points:

[Greg] Mankiw concisely summarizes the theory underlying the ethical argument for market capitalism: "under a standard set of assumptions... the factors of production [i.e., workers] are paid the value of their marginal product... One might easily conclude that, under these idealized conditions, each person receives his just deserts." Mankiw's long-standing opposition to higher taxes on the wealthy suggests that he thinks these conditions usually pertain in the real world, too.

Consider me skeptical. The list of "standard assumptions" open to question is long... I believe, progressives must directly challenge the claim that unfettered markets create just deserts. This won't be easy. Free market fundamentalists have the advantage of a simple message -- ending bailouts will deliver just deserts -- and of nearly limitless funds from rich folks who benefited from the bailout but are happy to claim that it should never happen again.

Mankiw's assumptions may all be correct, but they are not all self-evident. Some are at odds with experience. Some are in conflict with findings from related fields like psychology and behaviorial economics. Some are just hard to buy. These are the kind of assumptions that need to be stated and supported.

As mentioned before, the way language is used in the debate compounds the problem. The 'free-market fundamentalists' (to use Schoenberg's phrase) often affect a folksy, plain-spoken tone. They make common-sense statements like "people are rational" or "people respond to incentives." They generally don't add that they are using these terms in a technical, highly specialized sense.

Damned Canadian Show-offs

From the aforementioned paper by Lane Kenworthy:

At the risk of repeating myself, if we want to use the experiences of other countries to help us improve our education system, the first place we should look is probably Canada, even if those experiences don't support the conventional wisdom.

"Two and a Half Cheers for Education"

I've only had a chance to briefly skim this paper by Lane Kenworthy, so I can only give it a so-far-so-good recommendation, but what I've seen definitely looks interesting.

Here's a taste:

Some home environments are less helpful to children's development than we would like them to be. Schools tend to do better. Evidence on this in the U.S. context comes from the natural experiment that is summer vacation. During those three months out of school, the cognitive skills of children in lower socioeconomic status (SES) households tend to stall or actually regress. Kids in high-SES households fare much better during the summer, as they are more likely to spend it engaged in stimulating activities. Cognitive psychologist Robert Nisbett concludes that "much, if not most, of the gap in academic achievement between lower- and higher-SES children, in fact, is due to the greater summer slump for lower-SES children."

This is relevant also for inequality of opportunity. Some argue that
schools actually worsen inequality, because children from high-income households benefit more than their less advantaged counterparts, thereby widening the disparity. As the evidence from summer breaks attests, that is wrong. Without schools the gap in cognitive and noncognitive abilities almost certainly would be greater. Though they can't possibly produce full equalization, schools do help to equalize.

To get a fuller picture of this phenomena, listen to this segment on the Harlem Children Zone's Baby College from This American Life.


Thanks to Mark Thoma for the link.

Political rhetoric and political violence -- the McKinley Assassination

William Randolph Hearst was relentless in his attacks on President McKinley. When the president was assassinated, those attacks had consequences:

Hearst Burned in Effigy

The publisher learned of the shooting in Chicago and said quietly to editor Charles Edward Russell of the American, "Things are going to be very bad." All of his papers took a sorrowful, solicitous, hopeful stance while waiting for news of McKinley's fate. When the president died, Hearst's enemies reprinted the cartoons, the poem, and the editorial that seemed to incite assassination. It was widely believed that Czolgosz was carrying a copy of the Journal in his pocket when he shot the president, but that story is apocryphal. Nonetheless, the Hearst papers were widely boycotted, and their publisher was burned in effigy along with anarchist Emma Goldman, whose lecture Czolgosz cited as his true inspiration for the assassination. Hearst punished none of the writers or cartoonists but soon changed the name of the Journal to the American. A cloud hovered over his empire for about a year, but by 1902 he was popular enough to win election to the House of Representatives from New York.

We'll have to wait to see if there will be any real consequences for this.

Dark days

In Arizona this morning:

Rep. Gabrielle Giffords (D-AZ) has reportedly been shot in the head at point blank range at an event in her district.

The incident took place in the parking lot of Safeway in Tuscon where Giffords was hosting a "Congress on Your Corner" event. The first 911 call came in at 10:11 local time, according to the Pima County sheriff's office. Local news confirmed that there were 5 dead at 3:34 ET. In a 4:00 ET press conference, UMC trauma surgeon Peter Rhee confirmed that they had received a total of 10 victims, one of whom had died, five of whom were in surgery and 5 of whom were in critical condition. Giffords, he said, was shot once in the head "through and through" but was responding to commands and had made it through surgery. Rhee said he was optimistic that she could recover.

The deceased victim at UMN, Rhee said, was a child.

Also from Talking Points Memo:

A federal judge was killed in the same incident in which Rep. Gabby Giffords was shot on Saturday morning.

A federal law enforcement official first confirmed to TPM that a federal judge was shot. The U.S. Marshals Service is on the scene of the shooting, the federal official told TPM. The Marshals Service employees responded to the scene after the shooting, the official said.

WNBC reporter Jonathan Dienst confirmed Roll was killed. A statement from Homeland Security Secretary Janet Napolitano said in a statement that Roll had been attacked.

Roll faced death threats in 2009 after presiding over a $32 million civil-rights lawsuit, the Arizona Republic reported:

When Roll ruled the case could go forward, Gonzales said talk-radio shows cranked up the controversy and spurred audiences into making threats.

While we're doing "then and now"

Back in June, 2009, this is how Edward L. Glaeser felt about the bailouts:

Since the collapse of Lehman Brothers, the public sector has spent billions saving the banks. While these decisions are certainly debatable, they are understandable. The US financial industry misbehaved badly,... but it is still a sector with a future. ... After all, every other sector in the economy depends on banks for their financing.

But what about cars? ... Does anyone, other than GM's management, believe that this company can come back? The current treatment, cash infusion and a reduction in corporate liabilities, provides a solution for a company that is broke, not for one that is broken.

The future of the financial sector is looking pretty scary these days. How about the auto industry and GM in particular?

Although the transformation has been a long time coming, Ford and the rest of the domestic auto industry appear to be finally giving up their addiction to gas-guzzling trucks and sport utility vehicles. Prodded first by rising federal fuel economy standards, then shocked in 2008 by $145-a-barrel oil and a global credit crisis that forced General Motors and Chrysler to seek federal bailouts, Detroit is making a fundamental shift toward lighter, more fuel-conscious cars — and turning a profit doing so.

Lone star, bad portents

Then:



Now (from Paul Krugman):

These are tough times for state governments. Huge deficits loom almost everywhere, from California to New York, from New Jersey to Texas.

Wait — Texas? Wasn’t Texas supposed to be thriving even as the rest of America suffered? Didn’t its governor declare, during his re-election campaign, that “we have billions in surplus”? Yes, it was, and yes, he did. But reality has now intruded, in the form of a deficit expected to run as high as $25 billion over the next two years.

And that reality has implications for the nation as a whole. For Texas is where the modern conservative theory of budgeting — the belief that you should never raise taxes under any circumstances, that you can always balance the budget by cutting wasteful spending — has been implemented most completely. If the theory can’t make it there, it can’t make it anywhere.

How bad is the Texas deficit? Comparing budget crises among states is tricky, for technical reasons. Still, data from the Center on Budget and Policy Priorities suggest that the Texas budget gap is worse than New York’s, about as bad as California’s, but not quite up to New Jersey levels.

The point, however, is that just the other day Texas was being touted as a role model (and still is by commentators who haven’t been keeping up with the news). It was the state the recession supposedly passed by, thanks to its low taxes and business-friendly policies. Its governor boasted that its budget was in good shape thanks to his “tough conservative decisions.”

Oh, and at a time when there’s a full-court press on to demonize public-sector unions as the source of all our woes, Texas is nearly demon-free: less than 20 percent of public-sector workers there are covered by union contracts, compared with almost 75 percent in New York.

So what happened to the “Texas miracle” many people were talking about even a few months ago?

Part of the answer is that reports of a recession-proof state were greatly exaggerated. It’s true that Texas job losses haven’t been as severe as those in the nation as a whole since the recession began in 2007. But Texas has a rapidly growing population — largely, suggests Harvard’s Edward Glaeser, because its liberal land-use and zoning policies have kept housing cheap. There’s nothing wrong with that; but given that rising population, Texas needs to create jobs more rapidly than the rest of the country just to keep up with a growing work force.

And when you look at unemployment, Texas doesn’t seem particularly special: its unemployment rate is below the national average, thanks in part to high oil prices, but it’s about the same as the unemployment rate in New York or Massachusetts.

What about the budget? The truth is that the Texas state government has relied for years on smoke and mirrors to create the illusion of sound finances in the face of a serious “structural” budget deficit — that is, a deficit that persists even when the economy is doing well. When the recession struck, hitting revenue in Texas just as it did everywhere else, that illusion was bound to collapse.

The only thing that let Gov. Rick Perry get away, temporarily, with claims of a surplus was the fact that Texas enacts budgets only once every two years, and the last budget was put in place before the depth of the economic downturn was clear. Now the next budget must be passed — and Texas may have a $25 billion hole to fill.

As a native of the Lone Star state (now happily on the West Coast), I've got to go with General Sheridan on this one.

Defining Denominators

Via Mark Thoma, I discovered this very interesting article looking at group of gross domestic product by working age population (WAP):

When one looks at GDP/WAP (defined as population aged 20-60), one gets a surprising result: Japan has actually done better than the US or most European countries over the last decade. The reason is simple: Japan’s overall growth rates have been quite low, but growth was achieved despite a rapidly shrinking working-age population.

The difference between Japan and the US is instructive here: in terms of overall GDP growth, it was about one percentage point, but larger in terms of the annual WAP growth rates – more than 1.5 percentage points, given that the US working-age population grew by 0.8%, whereas Japan’s has been shrinking at about the same rate.

Another indication that Japan has fully used its potential is that the unemployment rate has been constant over the last decade. By contrast, the US unemployment rate has almost doubled, now approaching 10%. One might thus conclude that the US should take Japan as an example not of stagnation, but of how to squeeze maximum growth from limited potential.

This is a very good illustration of how important it can be to understand the structure of a population under study. I don't know if the proposed metric is the most relevant metric for the phenomenon under study but it's sure interesting how it completely changes the interpretation of the result. That could have very profound policy results when we consider questions like "would it be a bad thing to emulate Japan's industrial policy?".

Rajiv Sethi was there first

While going through the comment section of this post by Tim Duy, I was reminded that Rajiv Sethi was talking about broadcast HDTV long before the rest of us and was doing a remarkably thorough job of it.

Earnings of Post-Docs

Felix Salmon links to this post from the Economist and extracts this rather amazing statistic:

In Canada 80% of postdocs earn $38,600 or less per year—the average salary of a construction worker.

Mark has previously quoted from the same post. But I do think that post-doctoral training is a very interesting place to look at; the PhD program is educational and credentialing in a way that the post-doctoral process is not. In a sense, a PhD student is being partially compensated by acquiring their degree, which in theory can open up many career options (only some of which are academic). But a post-doctoral fellow is focused entirely on academics. So long as the success rate among post-doctoral fellow is high, the post-doc can be considered to be an internship. But if the success rate gets too low then the issue of exploitation comes up.

On the other hand, if the post-doctoral fellowship is a time of enrichment and job satisfaction is very high then maybe it is okay. I'm sure I would prefer to be a post-doc than a construction worker, myself. It's a tough issue . . .

Research Plans

In the comments to this post, PhysioProf draws a very interesting distinction:

You know all the people at Drugmonkey’s blog and Writedit’s blog who are constantly ranting and raving about how their science is so totally awesome and there must be something NEFARIOUS AND UNFAIR going on in study sections that fail to fund their MAGNIFICENT AND BOLD GROUNDBREAKING RESEARCH? Those people fail to distinguish between appropriate design of their actual research programs and appropriate crafting of a fundable grant application, taking account of their career stage and prior accomplishments. If they keep willfully ignoring this distinction, they are going to keep failing to secure NIH funding.

I think that this is correct, even if I don't especially like the reality of it. Scientific writing is based on a very stylized approach and it makes sense to learn the rules of how to communicate effectively in this medium. It's also true that one needs to put forth a plan that is realistic in a grant proposal. I don't think anybody was ever admonished for accomplishing more than they expected in a research plan.

The hardest thing I am finding in year one of the tenure track position is rescaling the speed at which I can do things. I was a very fast post-doctoral fellow and I could do an amazing amount by just working insane hours. But training junior graduate students takes a lot of time and I find my net productivity is dropping as I focus on training (well, designing courses isn't helping either). So I am sympathetic to the NIH wanting to see realistic goals for a research grant.

But I really liked the clear distinction that was being made between the two processes . . .

The Scalar Fallacy

Sometimes the things that give us the most trouble are most obvious. When something is completely self-evident, it can be difficult to wrap your mind around it and think through its implications. Important points can be mistaken for tautologies (and vice versa) and when you try to work through the questions with essays or conversations, you often find yourself feeling pretentious and, for lack of a better word, silly.

Here's an example: neither vectors, random variables nor vectors of random variables are scalars. This statement is obvious to anyone familiar with the basic terms. Equally obvious is the fact that when you try to represent one of these complex, multidimensional creatures as a point on a line, you will invariably lose some information.

The implications of these points, however, are often not obvious at all.

We have to assign scalars to things all the time because, among other reasons, scalars are the only things we can rank. Any time you want to decide what's the best _____ (car, job offer, candidate), you have to start by assigning _____ a scalar. You can do this by finding a proxy that's already a scalar (like the answer to a survey question) or by using a function of the vector. Simple examples include taking the sum or the sum of the squares or the average or the maximum value. (I'm going to limit this to vectors from here on but everything should generalize to random variables and vectors of random variables fairly easily.)

But, though we have to do it all the time, no one has ever found a perfect way of assigning scalars to vectors and no one ever will. This isn't pessimism; it's mathematics. You lose information when you go from a vector to a scalar. That loss means you have to be careful about contextual questions like range of data. Though there may be a few cases where we can derive the scalars from first principles, we generally have to arrive at the assignments through experimentation. We find methods that have produced useful metrics in previous situations. Unfortunately, when you move out of the range of data you encountered in those previous situations or when you otherwise find yourself in a new context, the information you could safely omit before becomes essential and the metric that has done such a good job up till now suddenly becomes worthless.

Here are a couple of examples:

A "rate your experience" question might do a good job comparing the impact of bad beverage service versus that of short delay in take-off but it will probably not do a satisfactory job comparing a forced landing and a seven hour stay on the tarmac on a hot summer day . These events fall outside the range of data the question was developed for.

A weighted average of nutrients might provide a good way of ranking most of the foods you find in the produce aisle. In the context of comparing different fruits and vegetables found in your neighborhood grocery store, you might be able to get by assuming a linear relationship between the amount of certain nutrients and healthiness. If, however, you move to the context of the dietary supplement aisle, making that linear assumption about certain nutrients can be dangerous, even deadly. Having a bottle of iron supplement pills for lunch is an extraordinarily bad idea.

These are relatively simple examples but think about all the unspeakably complicated things like happiness that people routinely discuss as if they were scalars -- "people in group A were 42% happier than people in group B." Worse yet, many researchers insist on pushing these scales to ludicrous extremes, using the same metrics to measure the impact of everything from trivial lifestyle changes to the birth of a first child. (How this affects theories like rational addiction is a subject for another post.)

Perhaps even more important than being context-specific, the scalars we assign to vectors are generally question-specific. Take the example of health. There's no meaningful way to boil this complex, multidimensional concept down to one number, but we can come up with scalars that are useful when answering certain questions. Let's say we have formulas for deriving two metrics, L and Q. L correlates very well with longevity; Q correlates very well with quality of life. For most questions about health policy, you will get similar answers with either metric, but there are cases where the two diverge sharply. Both L and Q are good measures of health, but their usefulness depends on the question you need answered.

Part of the blame for the tendency to take scalars as ideal representations of vectors rests with the "magic of the market" faction of economists and their camp followers. Markets are basically in the scalarizing business and under the proper conditions they do a pretty good job. It's easy to see how researchers grew enamored with markets' ability to set prices in such a way that resources are effectively allocated. It is a remarkable process.

But as impressive as markets are, they still are not exempt from the laws of mathematics and the limitations listed above. Prices are scalars assigned the values of things. They generally provide us with an excellent tool for prioritizing purchases and production but when you start to think of the scalars as actually being the vectors they represent, your thinking becomes sloppy and you open yourself up to dangerous mistakes.

"Why doing a PhD is often a waste of time"

Via Felix Salmon, there's a bleak but informative article in the Economist on the PhD glut. Here's a sample:

For most of history even a first degree at a university was the privilege of a rich few, and many academic staff did not hold doctorates. But as higher education expanded after the second world war, so did the expectation that lecturers would hold advanced degrees. American universities geared up first: by 1970 America was producing just under a third of the world’s university students and half of its science and technology PhDs (at that time it had only 6% of the global population). Since then America’s annual output of PhDs has doubled, to 64,000.

Other countries are catching up. Between 1998 and 2006 the number of doctorates handed out in all OECD countries grew by 40%, compared with 22% for America. PhD production sped up most dramatically in Mexico, Portugal, Italy and Slovakia. Even Japan, where the number of young people is shrinking, churned out about 46% more PhDs. Part of that growth reflects the expansion of university education outside America. Richard Freeman, a labour economist at Harvard University, says that by 2006 America was enrolling just 12% of the world’s students.

But universities have discovered that PhD students are cheap, highly motivated and disposable labour. With more PhD students they can do more research, and in some countries more teaching, with less money. A graduate assistant at Yale might earn $20,000 a year for nine months of teaching. The average pay of full professors in America was $109,000 in 2009—higher than the average for judges and magistrates.

Indeed, the production of PhDs has far outstripped demand for university lecturers. In a recent book, Andrew Hacker and Claudia Dreifus, an academic and a journalist, report that America produced more than 100,000 doctoral degrees between 2005 and 2009. In the same period there were just 16,000 new professorships. Using PhD students to do much of the undergraduate teaching cuts the number of full-time jobs. Even in Canada, where the output of PhD graduates has grown relatively modestly, universities conferred 4,800 doctorate degrees in 2007 but hired just 2,616 new full-time professors. Only a few fast-developing countries, such as Brazil and China, now seem short of PhDs.

Correction

I was wrong when I said industry watchers had been incorrectly announcing the death of network television for over thirty years.

I should have said over forty years:

An awestruck audience at the 1970 CES loved the VCR's convenience -- but Hollywood battled back, warning that piracy would run rampant and kill network television.