The United States is emerging from a great recession whose major hallmark has been the collapse of national housing prices, which grew by 59 percent from 2000 to 2006 and then fell 41 percent by 2011, all in constant dollars. Nationally, real house prices in 2011 were 6 percent below levels in 2000. The housing price collapse had unanticipated contagion effects that helped produce the accompanying financial crisis and the most severe economic downturn since the Great Depression. The share of U.S. mortgages that were delinquent by 90 days or more rose from about 1 percent in 2006 to over 8 percent in 2010. The economic and social costs of this house price bubble and subsequent collapse have been immense.

The benefits of preventing future house price bubbles is obviously great, but realizing such benefits will require that policy makers learn to detect price bubbles as they are forming and then implement policies that will attenuate or mitigate them. A recent Lincoln Institute policy focus report, Preventing House Price Bubbles: Lessons from the 2006–2012 Bust, by James Follain and Seth Giertz, addresses the challenges of diagnosing and treating price bubbles in the real estate market. Their report builds on extensive statistical analysis available in several Lincoln Institute working papers.

While it is common to summarize the recent housing market bust using national indicators (as in the first paragraph above), these national indicators don’t account for great variations in both the levels and changes in housing prices across metropolitan areas. For example, from 1978 to 2011, constant dollar housing prices in Dallas, Texas and Omaha, Nebraska varied by less than 20 percent from their 1978 levels; those in Stockton, California nearly tripled from 1978 to 2006, but by 2011 fell back to their 1978 levels. Local housing markets are all influenced by national economic and financial policies and conditions, but these large differences across metropolitan markets indicate that local conditions play a very important role as well.

A key element of the statistical work by Follain and Giertz is to use metropolitan housing markets as the unit of observation for their analyses, which are based on annual data (for 1980 to 2010) and quarterly data (for 1990 to 2010) for up to 380 metropolitan areas. Their econometric work indicates that house price bubbles can be detected across metropolitan areas and that price changes and the accompanying credit risk vary greatly in size. Stress tests, such as those used to evaluate mortgage credit risk, can be useful indicators of potential price bubbles at the metropolitan level.

Because the levels and changes in housing prices vary greatly across metropolitan areas—with bubble-like price increases in some and essentially stable prices in others—Follain and Giertz conclude that policy measures to mitigate housing bubbles should be tailored to target metropolitan areas or regions rather than be applied uniformly across all metropolitan areas at the national level. Thus monetary policy would be an unattractive intervention to counter house price increases in a few metropolitan areas, because it would affect financing terms across both frothy and stable housing markets. Instead, Follain and Giertz favor policy interventions that would target those metropolitan areas with high price increases. The policy they advance would raise the capital reserve ratio that banks are required to hold against mortgages that they finance in those areas. Such countercyclical capital policies would both dampen house price increases and strengthen the reserves of the issuing banks, improving their ability to withstand any unexpected financial shocks.

Applying prudential housing market policies at the metro-politan level seems to be an obvious thing to do; so why has it not been done before? A major part of the answer is that housing market analysis is benefitting from a revolution in the availability of spatially disaggregated data at the metropolitan, county, and even zip code level. The data required to inform policy interventions targeted at the metropolitan level have only recently become widely available, and such data underpin the empirical work carried out by Follain and Giertz. For more information on their analysis, see http://www.lincolninst.edu/pubs/2245_Preventing-House-Price-Bubbles.

People who work with me are often surprised by the extent to which my philosophical canon derives from low-budget offbeat films, typically from the 1980s. When in need of wisdom, I frequently turn to the teachings of Repo Man or, for this essay, Terry Gilliam’s allegorical masterpiece Time Bandits. In the movie, a group of public workers are employed by the Supreme Being to fill holes in the time-space continuum left from the haste of creating the universe in seven days: “It was a bit of a botched job, you see.”

Like the Time Bandits, policy makers are often tasked to fill holes—actual potholes in roadways, or more theoretical holes that are the artifacts of dysfunctional private markets. One big hole that policy has struggled for decades to fill is the inadequate supply of affordable housing. For example, housing economists in the United States have become quite adept at tracking the size of the hole, which has only become harder to fill since the federal government committed to address it as a national policy priority beginning with the Housing Act of 1949, part of President Harry S. Truman’s Fair Deal.

Perhaps our collective failure to solve the affordable housing deficit over the last 66 years stems from wrongheaded analysis of the problem, and the conclusion that market-based solutions can be designed to solve the mismatch between the supply of affordable housing and demand for it. In his 1949 State of the Union address, President Truman noted that to fill the needs of millions of families with inadequate housing, “Most of the houses we need will have to be built by private enterprise, without public subsidy.”

To support this claim, permit me a short departure into market theory. From the now-preferred mathematical approach to economic analysis, a market is simply a system of partial differential equations that is solved by a single price. The partial differential equations capture the complex decisions made by consumers and producers of goods, reconciling tastes, preferences, and budgets of consumers with the technical complexities of producing goods to arrive at a price that clears the market by settling all transactions that suppliers and consumers of goods are willing to make.

Acclaimed economists Arrow, Debreu, and McKenzie proved the theoretical existence of a single set of prices that can simultaneously solve for the “general equilibrium” of all markets in a national or global economy. One important aspect of this Nobel Prize–winning contribution was the observation that a unique price cleared each market—one market, one price. There was no expectation that a single price could maintain equilibrium in two markets. But this is the fundamental flaw of the housing market—it is actually two markets, not one. Housing markets supply both shelter for local consumption and a globally tradable investment good made possible by broad capital markets that serve global investors. This dual-market status used to be more descriptive of owner-occupied housing, but, with the proliferation of real estate investment trusts (REITs), rental markets are now in the same boat.

Markets for consumption goods behave very differently than investment markets, responding to different “fundamentals.” On the supply side, prices for consumption goods are dictated by production costs, while prices in investment markets are dictated by expected returns. On the demand side, such things as tastes and preferences, household incomes, and demographics determine the price of housing as shelter. Investment demand for housing is dictated by factors like liquidity and liquidity preferences of investors, expected returns on alternative investments, or interest rates.

In developed countries, global capital markets and the market for shelter collide locally with little chance of reconciliation. Local households compete with global investors to decide the character and quantity of housing that is produced. In markets that attract global investment, plenty of housing is produced, but shortages of affordable units are acute, and worsen over time. This is because a huge share of new housing is produced to maximize investment return, not to meet the needs of the local population for shelter. For example, there is no shortage of global investment willing to participate in developing $100 million apartments in New York City. But affordable housing, being much harder to finance, is in short supply. And in markets that have been abandoned by global capital, house prices fall below production costs, and surplus housing accumulates and decays. In extreme cases such as Detroit, market order can only be restored by demolishing thousands of abandoned homes and buildings.

Perhaps it is time that we reconsider the analysis that led President Truman, and thousands of housing policy makers after him, to conclude that one could forge market-based solutions to the challenge of sheltering a country’s population. Truman concluded that “By producing too few rental units and too large a proportion of high-priced houses, the building industry is rapidly pricing itself out of the market.” But Truman was thinking about the market for shelter, not investment. It is remarkable to note that the number of housing units supplied in developed countries such as the United States significantly exceeds the number of households. In 2010, the U.S. Census estimated that there were 131 million units of housing in the country and 118 million households—one in seven housing units were vacant. It is even more shocking to note that in the United States this oversupply of housing characterizes every metropolitan market in the country—even metropolitan markets with extreme shortages of affordable housing. In 2010, 8.5 percent of housing units were vacant in Greater Boston, 9.1 percent in the San Francisco Bay area, and 10.2 percent in Washington, DC. The problem is that many households have insufficient incomes to afford the housing that is available.

In the end, rather than fill the holes in the fabric of time and space, the Time Bandits decided to take advantage of them to “get bloody stinking rich.” The bandits sought to capitalize on celestial imperfections in the same way that global investors seek returns from short-term market dislocations. To illustrate the dangers of naked speculation in unregulated markets, consider an apocryphal tale from a very different market. In 1974, heavy rains during planting season in Bangladesh suggested that rice might be in short supply at harvest time. In anticipation of these shortages, rice prices started to rise. Savvy commodity speculators realized that there would be a good return on any rice that was held off the market. Despite the fact that the actual harvest produced a bumper crop, the interaction between market expectations and market manipulations by commodity investors produced one of the worst famines of the 20th century—with an estimated 1.5 million famine-related fatalities. The famine was not the result of real food shortages. The collision of the market for goods and the market for speculative investment priced rice out of the reach of the local populations, with landless families suffering mortality at three times the rate of families with land.

Perhaps shelter and food are too important to be left to unregulated markets to allocate. In light of the damage that the conflict between the market for goods and the market for investment can inflict on local populations, perhaps public policy should focus on protecting a share of the market—and the public—from the ravages of speculation. In this issue, we describe some nascent efforts to produce permanently affordable housing by insulating it from speculation—through community land trusts, inclusionary housing, and housing cooperatives. Miriam Axel-Lute and Dana Hawkins-Simons discuss the mechanics of organizing local community land trusts. Loren Berlin describes efforts to preserve affordable housing in the form of manufactured homes and to promote permanent affordability of that stock through the conversion of manufactured housing communities to limited equity cooperatives.

On more cautionary notes: Cynthia Goytia discusses the ways that low-income communities circumvent housing regulations that drive up housing costs to produce their own affordable but substandard shelter in informal settlements around Latin American cities; and Li Sun and Zhi Liu discuss the tenuous status of one-quarter of urban Chinese households that purchased affordable shelter with uncertain property rights on collectively owned land at the rapidly developing edge of cities and in “urban villages,” former rural settlements now surrounded by modern construction. As capital markets deepen in these countries, the competition between housing as investment good and housing as shelter will likely exacerbate informality in Latin American cities and make property rights of these Chinese families more precarious. After almost seven decades of failed efforts to get private markets to meet populations’ needs for affordable shelter, it might be time to develop, and to export, another approach that is based on a more realistic understanding of the complexity of housing and capital markets.