[I realize we've covered a lot of this territory before and I apologize for the redundancy, but I thought it might be nice to some everything up in one final post.]
Just to have a framework, let's start with some fundamental assumptions of the conventional urbanist wisdom. These are badly oversimplified, but they should be good enough for our purposes here.
The best and easiest way of alleviating the serious externalities associated with commuting (particularly environmental damage) is by having people move near enough to centers of employment that personal transportation (other than bikes) is not necessary.
The best and easiest way of lowering the often exorbitant rents near the center's is by building up.
The best and easiest way of getting high-capacity housing where we most need it is through market forces.
Putting aside arguments for telecommuting (pretty much by definition the fastest and most efficient way to get to work), here are some of my concerns with this model. Ironically, some of them are fairly closely the concerns that urbanists have about suburban sprawl.
Moving is difficult. Buildings are permanent (and they do have an environmental footprint). One of the hidden social costs of home ownership is that it ties the owner to a specific job market. If you are wedded to the idea of making commuting nondependent on automobiles, this high density approach faces many of the same challenges, particularly for households with more than one working member. These housing units need to be so close to a wide enough range of jobs that two people can find housing within easy commute of two different positions and will have a reasonably good chance of staying in the same location in the event of a job change. What's more, that employment center needs to remain relatively stable more or less indefinitely. Booms and busts could play hell with this model.
Actual researchers tend to take a more nuanced and sophisticated view, but in the press, the urban density debate generally treats the choice of where to live as a fairly simple function of two variables, proximity to employment and housing cost. We have reason to believe that the real relationship has more variables and more complexity with interactions between proximity to employment and the weighting of other factors. For example, we know that a nontrivial number of people in Los Angeles and the Bay Area will opt for rental options that are both more expensive and further from work.
Silicon Valley workers living in San Francisco have gotten a lot of coverage but trendy neighborhoods in LA may be a more useful case for study. "Trendy" is the key word here. We're generally talking about well-paid professionals who are willing to put up with an extra half hour or more of traffic for scenic views, dining and other amenities, and, perhaps most of all, the ability to impress other people with where you live and who your neighbors are. The resulting dynamic can be very much like suburban sprawl, but with the suburb tucked in the middle of a high density urban area.
Partially because of the reasons given above, market forces have a very mixed record when it comes to picking the most efficient spot for development. I'll limit my comments to Los Angeles because I know the town, but I believe they could be generalized to a large number of other areas.
A great deal has been written about the NIMBY push against development in Santa Monica. Utopian urbanists like Dave Roberts have gone so far as to suggest that anyone who claims to be an environmentalist and opposes it must be a hypocrite.
The problem with this line of reasoning is that Santa Monica, particularly the extremely expensive section north of the 10 and west of Lincoln, is one of the worst possible places in the county of Los Angeles (and this is a big God damn County) for using high density development to alleviate the impact of commuting and to reduce cost of living.
Geographically, it's bounded on two sides by ocean and mountains thus greatly limiting the number of commuting destinations. The constant flow of tourists means that prices will tend to be high and traffic will never, ever be good. The trendiness of the town makes it likely to become an urban suburb and an appealing spot for second homes among the rich. Finally, and perhaps most importantly, the public transportation actuation is extremely bad. Other than the buses, which have to deal with the aforementioned traffic, the only other option is a single, slow train with a not-that-convenient route. (Don't get me wrong, simply having a train to the ocean is a big step forward for LA, but not nearly big enough to alleviate the traffic woes of a much denser Santa Monica/Venice.)
If the goal really were to create a greener, less car dependent Los Angeles, Santa Monica developments wouldn't even be on our radar. Instead, we would be focusing on development around transportation hubs, particularly Union Station. There's plenty of room for growth with in a two-mile radius, but the best places for development are not in the trendy upscale neighborhoods, and developers know that trendy is where the money is.