The Jump$tart Coalition is a leader among organizations seeking to improve the personal financial literacy of students from kindergarten to the university level. In particular, through its biennial survey of high school seniors—the results of which you will hear about shortly—Jump$tart has brought increased attention to the need for greater financial literacy among the youth of our nation. During the Jump$tart survey’s 12-year history, the data gathered have served as the basis for useful measures of what young adults do and don’t understand about finances. Undoubtedly, we will soon learn that there is plenty of work to be done and that our students have much to learn.Ben S. Bernanke at the Jump$tart Coalition for Personal Financial Literacy and Federal Reserve Board joint news conference, April 9, 2008
April of 2008 was definitely a time of signs and portents. Many economists and a few farsighted journalist like the good people at This American Life were warning us that we were entering dangerous territory. It's fair to assume that Bernanke didn't need the warning -- the man had essentially spent his entire career preparing for this crisis -- but he took time out of what was unquestionably a very busy day to laud the accomplishments of the Jump$tart Coalition and its survey.
It's easy to understand why Chairman Bernanke was concerned about financial literacy. With the complex, unstable economy, the shift away from traditional pensions and the constant flood of new financial products, financial literacy might be more important now than it has been for decades. You could even make the case for financial illiteracy being a major cause of the economic crisis.
But if the supporters of financial literacy need a good measure of how well we're doing, they'll need to find a better instrument than the Jump$tart survey.
The 'test' part of the survey consists of thirty-one questions. That's not very long but that many questions should be sufficient for a tightly focused, well-structured test. Unfortunately the focus of the Jump$tart survey is ridiculously broad, ranging from investments to retirement to credit cards to debt counseling to auto insurance to macroeconomics to really questionable career advice.
Even within the categories the questions have a random, pulled-from-a-hat quality with no apparent effort to prioritize. There are multiple references to credit histories but no mention of credit scores. None of the few questions on credit cards mention teasers or other cases where rates can change on a credit card. There are no questions that refer to charts or tables though the ability to read both is an essential part of financial literacy.
On the individual question level the situation is no better. Most of the questions are either badly written, trivial/irrelevant, open to interpretation, guessable or factually challenged. The test resembles nothing so much as the homework paper a student teacher might turn in when asked to come up with 31 questions on financial literacy.
If you compare this test to something like the SAT where every question has been repeatedly proofed, tested and rewritten, it becomes obvious how sloppy the writing is here, complete with rookie errors like using the wrong person in a question like this:
24. If you went to college and earned a four-year degree, how much more money could you expect to earn than if you only had a high school diploma?
21.9 a.) About 10 times as much.
8.6 b.) No more; I would make about the same either way.
22.0 c.) A little more; about 20% more.
47.6 d.) A lot more; about 70% more. *
[note: Numbers to the Left of Answers are Proportion Giving Response. The asterisk indicates the correct answer.]
What's the problem with using the second person here? This is one of those statements that's true for a population at large but may not be true for most subgroups of the population. The value of a college degree varies greatly based on proposed career plans. For an architect or database analyst, a ten fold increase would probably be conservative; for a truck driver or someone who plans to work in a family restaurant, a college degree may provide nothing but personal growth opportunities and bragging rights.
Another rookie mistake is the high number of guessable questions, questions where students who know nothing about the information of interest have a good chance of guessing the right answer.
18. Don and Bill work together in the finance department of the same company and earn the same pay. Bill spends his free time taking work-related classes to improve his computer skills; while Don spends his free time socializing with friends and working out at a fitness center. After five years, what is likely to be true?
11.5 a.) Don will make more because he is more social.
9.8 b.) Don will make more because Bill is likely to be laid off.
67.9 c.) Bill will make more money because he is more valuable to his company.*
10.8 d.) Don and Bill will continue to make the same money.
By the time they reach high school, students have long since learned the simplistic moral vision of tests and textbooks. When in doubt, pick the answer that shows hard work and self-discipline pay off. Questions like this may be better measures of students' cynicism than of their financial literacy, particularly given the suspect answer.
As Frances Woolley observes, it is "not at all obvious to me that (c) is the right answer." In most companies, the management track tends to pay better and move faster than the technical track, making (a) a reasonable choice and, in a age of off-shoring, the employee who just brings computer skills to the job is particularly vulnerable to replacement, making (b) a possible choice. In other words, you could argue that the 'correct' choice is neither the best nor the second best answer.
In order to provide useful data with tests and surveys, you have to make sure that the response to each question tells you what it's supposed to and that those questions adequately cover the areas of interest. The Jump$tart survey completely fails under both criteria.
