Thursday, 28 June 2018

Topics on my mind: May 2018

I've been listening to quite a few podcast episodes lately, particularly Rationally Speaking, the 80000 Hours podcast, EconTalk and Conversations with Tyler (who runs the excellent Marginal Revolution economics blog). I was very impressed by a Rationally Speaking episode with Dr Herculano-Houzel, a brain researcher (here's my blog post about it), as well as the 80,000 Hours episode with Anders Sandberg. Anders does a wide range of very creative and original research; here he was talking about hibernating aliens, interstellar warfare, and how quick colonising the universe could be. I'd previously seen these arguments as intellectually fun but fairly irrelevant. But on the podcast, Anders presented them in a new light: not simply as speculations on what aliens might be doing, but also as a way of exploring strategies that humans might want to use in the far future. Now that I think about it, that does seem pretty important.

I've finally written some political blog posts: an extensive exploration of the causes of the housing crisis, as well as another which I haven't published yet on arguments for libertarianism. The latter is meant to be a holistic summary of my political and economic views, so I'm being thorough in editing it. In general I think that I should spend very little time thinking about or discussing politics, but that's pretty difficult. When tempted, I remind myself that my brain evolved to live in a tiny tribe where involvement in politics was a matter of life and death. Since I instead live in a society of millions of people with a very stable (by historical standards) political structure, personal political action is actually thousands or millions of times less important than my emotions tell me it is.

I've also recently posted on the backpropagation algorithm and ways it might be implemented in the brain - which follows in the footsteps of Jacob's excellent essay on the same subject - and added an index by topic of all blog posts so far.

Following on from my series of posts about intelligence, check out this timeline of milestones in the evolution of life (taken from this blog; note that the scale is a bit distorted though).

Three key takeaways:
  • Simple life formed almost as early as it could possibly have, which implies that it's a relatively easy step
  • Going from photosynthetic prokaryotes to eukaryotes with nuclei took a very long time. Apparently eukaryotes probably arose from the symbiotic fusion of two cells, which I suppose would be fairly complicated
  • Going from eukaryotes to multicellular life was another pretty lengthy step
So it took over 80% of the time between the origin of life and today to get just to something like microscopic algae. I'm not sure how to interpret this. On one hand, the more complex life is, the more scope there is for evolution to make interesting changes (one salient example is how humans ended up in an "intelligence arms race", and thereby developed unprecedented mental abilities very quickly). On the other hand, microorganisms live and die very quickly compared with more complex life forms, so the total selection pressure on them was probably even greater than the time ratio makes it seem.

One last thing on my mind: the causes of increasing economic inequality. The standard explanation I've heard is that as the world becomes more connected and markets get bigger, the returns to success increase in most fields, so elites take a larger share of income. (Power law distributions, generated by processes where "the rich get richer", are often used to model wages; although according to this recent paper, log-normal distributions fit better). First question - if this is true, how long has it been going on? Decades? Centuries? A millennium? Second question - in what sort of futures might the returns to success decrease? Would leveling society require another upheaval like World War 2?

On the other hand, the graphs below suggest that this explanation is wrong, or at least incomplete.

At least until 2010, major continental European countries didn't exhibit the same spike in inequality as anglophone countries. This is measuring pre-tax income as well, so the difference is even greater once you account for the countries in the right-hand graph having more extensive redistribution via taxation and welfare. Is the disparity due to non-anglophone countries discouraging entrepreneurship? Or providing more opportunities to disadvantaged children? Or does it stem from more subtle cultural factors? That's the trillion-dollar question, I guess; and we'll need good sociologists just as much as good economists to solve it.

Wednesday, 27 June 2018

Going to the dark side vs seeing the light

Since I've now been lucky enough to attend both Oxford and Cambridge, I thought it'd be interesting to compare and contrast the two. Bear in mind that I only spent one year in Cambridge, and that graduate students' experiences differ significantly from those of undergrads. Also note that there are many aspects of Oxford and Cambridge which are practically indistinguishable, and so I've skipped over most of those similarities. Nevertheless, here are some comparisons which have stuck in my mind - and which might help you decide (in the words of the particularly obnoxious Boat Race slogan) "which blue are you?"

Geographic differences
  • Oxford is a bigger city than Cambridge, with its suburbs stretching out much further. In Oxford, students who live outside the city center in places like Cowley or Summertown are surrounded by shops, bars and restaurants; whereas in Cambridge, they'd be surrounded by quiet residential neighbourhoods or paddocks. Since student life is so based around the city center, though, I don't notice this difference too much.
    As a university, Cambridge is much more spread out than Oxford. The two most distant colleges at Cambridge are an 80-minute walk from each other; at Oxford, the equivalent figure is less than half of that. Even discounting the outlier that is Girton College, getting from any of Cambridge's Hill Colleges to the train station takes longer than any trip Oxford students would commonly need to make. Cambridge also has a separate West Cambridge site which is much further away from the city center than any group of departments at Oxford - I sometimes had back-to-back lectures in the Computer Laboratory on the West Cambridge site and the central Engineering Department, a 38-minute walk apart. 
  • It takes about an hour to reach London from each, but tragically the only way to travel between them directly is a miserable 4-hour bus trip.
  • Cambridge's proximity to Stansted Airport is very nice. From Oxford, the easiest airport to get to is actually Birmingham, via direct train - although, to be fair, it's also closer to Heathrow than Cambridge is. 
  • Punting is much nicer in Cambridge; you get to go right past the most beautiful colleges, instead of just through meadows like in Oxford.
  • Both cities are also home to a smaller former polytechnic university - Oxford Brookes and Anglia Ruskin, respectively.
  • The wealth and prestige of Cambridge colleges is skewed at the top end by Trinity and Johns, which between them have almost as large an endowment as all other 29 colleges combined. At Oxford, no colleges are nearly as dominant; rather, the bottom end of the distribution is skewed instead, since there are still 6 Permanent Private Halls. These are like colleges, except that they are run by various Christian groups and focus heavily on theology; they are also very small and poor compared with the proper colleges.
  • Oxford has significantly more colleges - 38 (or 44 including PPHs) to Cambridge's 31. Cambridge's undergrad population is actually slightly larger than Oxford's, but Oxford has around 50% more postgrads than Cambridge.
  • Cambridge has colleges - most notably Trinity and Churchill - which focus heavily on STEM subjects. Trinity accepts about 40 mathematicians each year; Churchill's undergrad population is mandated to be at least 70% STEM. I don't think any Oxford colleges (apart from PPHs) specialise to a comparable extent.
  • Oxford colleges are all coed, and I think all fairly gender-balanced. By contrast, there are still three women-only colleges at Cambridge. Several other Cambridge colleges, especially the STEM-focused ones, are disproportionately male.
  • Both have notoriously left-wing colleges (Kings and Wadham respectively), and notoriously right-wing colleges (Johns and Oriel).
  • The overarching stereotype is that Cambridge focuses more on sciences, and Oxford focuses more on the humanities. As far as I can tell, this is reasonably accurate. Cambridge's CS department, for example, is much bigger and well-resourced than Oxford's. I got to know many more computer scientists in my one year here than during all three years at Oxford - due in no small part to the newly-constructed Computer Laboratory in Cambridge, whose sprawling, spacious architecture contrasts sharply with the warren that is Oxford's CS department. I was also able to choose from over five times as many machine learning courses as Oxford offers (although Oxford is stronger for more theoretical CS).
  • Oxford offers many more joint degrees - for example, you can't take philosophy by itself, but have to pair it with subjects like Computer Science, or Maths, or Psychology and Linguistics. I'm a big fan of this, since degrees at English universities are otherwise very narrow, and I think that exposure to a broad curriculum provides valuable perspective. (In particular, I've noticed that Cambridge philosophy students tend to focus on more traditional problems than philosophy students at Oxford, and be more absolutist in their views.) However, the rigour of Economics at Oxford probably does suffer from the fact that it's only available as a joint degree.
  • The most well-known degrees at both universities are in fact joint degrees. At Oxford, it's Politics, Philosophy, and Economics, which churns out Cabinet ministers by the dozen; Cambridge's is the Natural Sciences Tripos, which crams together many different scientific disciplines.
  • The exam schedules differ somewhat. At Oxford, science students have exams in their second and third years, which are combined to give a final degree result; humanities students only have exams in third year. At Cambridge, everyone has exams in both their second and third years, but those results are never officially combined - you end up with two grades for the two parts of your degree. I think that a combination of these two systems, with everyone sitting exams in both years then combining the results, would probably be more sensible than either.
  • While admissions are roughly equally competitive, Cambridge are more focused on grades than Oxford. For prospective undergrads, Cambridge requires A*A*A or A*AA at A-level, whereas most Oxford courses only require AAA (and in practice are often willing to accept some Bs from students who interviewed well). For graduate admissions, Cambridge usually requires a 1st at undergrad, whereas Oxford usually doesn't set that as a condition.
  • The academic differences trickle down to affect the activities that students do. Oxford is much more political, with many students attending events like Port and Policy (or the host of other drinking and debating nights put on by various political societies). Cambridge seems to have more attendance at niche and/or nerdy clubs - e.g. juggling, Go, maths, and gaming societies.
  • Cambridge students celebrate summer by throwing lovely garden parties, many more than in Oxford. On the other hand, one of my favourite parts of summer in Oxford was the garden plays that were staged in many colleges, which sadly seem to be lacking in Cambridge (perhaps because second-year humanities students also have exams there?)
  • Colleges at both universities regularly throw lavish balls. Oxford usually has several balls each year where tickets nudge above 200 pounds and white tie is compulsory. At Cambridge, dress codes and prices are more relaxed - however, the balls are no less splendid. In particular, Trinity and Johns have spectacular annual balls which are so oversubscribed that only those colleges' students and their guests can get tickets. These fall in the middle of June, during a raucous celebratory fortnight known for some unfathomable reason as "May Week". (There's no equivalent to May Week in Oxford, probably because exams are more spread out.)
  • Oxford still has elaborate exam traditions. It is compulsory to wear suits and gowns (or equivalent) to exams; candidates also customarily wear carnations indicating how far through their exams they are; and after finishing, they are usually "trashed" with champagne, cream, confetti, etc by their friends, then thrown in the river. Cambridge used to have similar traditions but fairly recently decided to abolish them (via a combination of student votes and university administrator decisions).
  • There is a stereotype that Oxford is posher and more elitist than Cambridge. Overall this is probably true - in addition to differences like the ones mentioned just above, Cambridge's undergraduate intake has about 10% more state school students than Oxford's. However, several events I've attended at Trinity and Johns (in Cambridge) have been fancier than any I went to in Oxford, so perhaps the variance within Cambridge is higher.
  • There are many slight differences in terminology - e.g. supervisions vs tutorials, Lent and Easter vs Hilary and Trinity terms.
  • Formals are much more expensive in Cambridge: typically £13-£18, compared with £4-£10 at Oxford.
  • Cambridge's online resources are better. I particularly like the online lecture timetables and centralised lists of talks, which allow you to explore events outside your department much more easily.
  • The food trucks are much MUCH better in Oxford. Especially Hassan's!
  • There are many more nightclubs in Oxford; in Cambridge, people often resort to dancing at the local Wetherspoons.
  • Cambridge has the Gates scholarship; Oxford has the Rhodes. The former is accessible to students from more countries, but also has a strong bias towards Americans. In other ways, they seem fairly similar.
  • Oxford has an ice rink which hosts late-night pick-up hockey games twice a week - one of my favourite parts of undergrad. Unfortunately, the closest rink to Cambridge is a 45 minute drive away.
  • Membership of the Cambridge Union is significantly cheaper than that of the Oxford Union, and the former seems better-run in several ways (e.g. more online content, more representation for competitive debaters). On the other hand, the Oxford Union has nicer buildings, particularly their library; I also think more people get involved in debating at Oxford.
When people ask me which I liked more, I tend to say that I preferred Oxford socially and Cambridge academically - which made it a good choice to go to Oxford for undergrad and Cambridge for a masters, I guess. I can trace the social preference to a variety of factors: the greater geographic proximity of the colleges; the cheaper formal halls (which were one of the major ways I socialised at Oxford); and even the preponderance of humanities students - it's much easier to start a conversation about history than biochemistry! On the other hand, it's always easier to settle in over three years than one, so I'm not sure whether I'd have come to the same conclusion if I'd attended Cambridge for undergrad and Oxford for a masters. And in fact, the closer I get to the end of my time at Cambridge, the more I appreciate the friends I've made and experiences I've had here.

Monday, 11 June 2018

Yes, you should be angry about the housing crisis

The easiest and most neglected way to make people in the developed world much better off is to fix housing policy. Record high property prices in thousands of major cities are burying people under crushing debt, crippling social mobility, and providing fuel for class divisions and xenophobia. And all of this could be easily avoided! I’ll explain how shortly (with particular reference to London and San Francisco, which are the examples I’m most familiar with, and which showcase widespread problems) but first let’s look at just how bad the crisis is (graphs from The Economist).

In Australia, New Zealand, Canada and the UK, average house prices have roughly tripled over the last 30 years. This is in real terms, adjusted for inflation; and it's in spite of whatever progress has occurred in construction techniques or materials. This is hundreds of thousands of dollars out of the average person's pocket, a vast and overwhelming expense.

And it's even worse than that. Consider the trend in the US, where prices have "only" increased by 50%. Not so bad, right? But this is a national average. If we zoom in on the biggest cities, we see prices doubling in New York, Boston and Los Angeles, and multiplying by a factor of 2.5 in San Francisco. (Also, all this data is 2 years out of date; the problem's only gotten worse since then).

Similarly, Britain as a whole has seen prices increase to more than 3x their 1970 values- but in London specifically, prices are over 5x what they were back then. That's absurd, and has disastrous implications for the economy overall. Modern knowledge economies are built around network effects; that's why, within America, there’s overwhelming concentration of entrepreneurship in Silicon Valley, and finance in New York, and entertainment in Los Angeles. Cutting people off from them is the best way to slow down innovation - and that's exactly what housing prices are doing. Consider this study, which finds that around 75% of US growth over the last 50 years comes from only two dozen cities. Yet in places like New York, over half of that growth was eaten up by housing costs. In the authors’ (admittedly speculative) model, if even just three cities - New York, San Francisco, and San Jose - had kept their housing regulations at normal levels over the last 50 years, the average income across the entirety of America would be almost $9000 higher than it is today, with millions more people able to access good jobs.

It gets even worse! This isn't just a slow, long-term trend - it's accelerating. The sharpest increases have been specifically over the past two decades. You can see that in the graphs above, but it becomes even clearer if we look at longer-term data, like these Stockholm prices.

In other words: if you’re in my generation, this is your problem and mine. It was born around the same time we were, and it's going to shape our future. It stifles innovation, as people can't afford to move to places where the breakthroughs are happening. It pushes people into financial instability, living from rent payment to rent payment and only one missed paycheck away from disaster. (For a gripping account of the struggle poor families face in finding and keeping housing I recommend Evicted, a Pulitzer prizewinner). It's responsible for rises in homelessness in many major cities. And even for the lucky few who can afford to live in those hubs, it's a constant weight on our shoulders. (Note that rents are also increasing, but at a considerably slower rate than house prices overall. If you're okay with many more people becoming long-term renters, then the trends above are still very bad, but not disastrous. I discuss the relationship between rental and purchase prices in more detail later on.)

Okay, so what about those inherit a house, or manage to buy one? Won't they end up rich, as prices keep skyrocketing? Perhaps, but only at the expense of others: people who can't get enough capital for that initial purchase, and the next generation who will face even more crippling prices. Housing price increases don't generate wealth, they just transfer it, usually to those who are already better off. And what's more, it's not even a productive transfer. A "winner" from the house price lottery might end up living in a million-pound house, wishing they lived in one half as expensive and had the rest of their wealth in cash. But they can't get to that point unless they're willing to retire and move to the countryside, because every other house in their area will be just as expensive. Even their heirs won’t benefit from that wealth, since they’ll have to sink it into housing costs of their own.


The graph above is particularly striking to me because the only other graphs I've seen shaped like this are those cataloguing exponential increases in technology or population. Somehow, over the last twenty years, we've managed to create a similar trend which, instead of empowering progress and growth, has exactly the opposite effect. What's to blame?

There are some subtleties on this issue, which I'll discuss shortly. But we simply cannot overlook the fact that the underlying cause of the spike in housing prices is government regulation, which severely curtails the supply of housing. The image above is a building plan of Manhattan - famous, of course, for its towering skyline. Most of those high-rises were built when the city's zoning code was relatively toothless. But boy has it expanded since then. How much? Well, the buildings marked in red are the ones that would be illegal to build today - 40% of the entire borough. What sort of problems do they have? Here are three common ones: they're too tall; they have too many apartments; or they have too many businesses. When governments strictly limit exactly the things which make buildings profitable and productive, it’s really not surprising that nobody’s building enough of them!

Height restrictions

Well, not quite nobody. Let's talk about height restrictions elsewhere. Here's Hong Kong, where they basically don't exist. Hong Kong is undergoing a housing crisis, but it’s perhaps the one place in the world which can’t fix it through increased construction, because they’ve already built about as much housing as is physically possible.

And here's London, which is just as wealthy a city, and is also currently undergoing a housing crisis, but seems much less inclined towards fixing it.

I'm not saying we should convert every building in London into a skyscraper. But right now there are only 1629 buildings in London that are taller than 35 metres or 12 stories. Moscow has over 7 times that many, and Seoul almost 10 times more (through some shocking coincidence, South Korea is one of the few developed countries where housing costs have actually decreased over the last 30 years, as shown in my first graph). Would doubling or tripling the figure really ruin London? Maybe it would make it slightly less pleasant. But would it make it half a million pounds per resident less pleasant? And even if you're willing to pay that price, what about the people who can't?

Land restrictions

Perhaps the downsides of central housing restrictions would be mitigated if poorer people, who can’t afford the luxuries of nice views in central neighbourhoods at double the former price, were able to live further away and commute in. Unfortunately, that option has also been taken away from them. Here's London, surrounded as you can see by its "green belt", although in its effects it's more like a green noose. Building is severely restricted throughout this belt and others like it around many other UK cities. This means that people who can’t afford to live in the centre have to commute for an hour or more, on slow and expensive trains, from places like Reading. Increased commutes not only hit the poor the hardest but also increase energy use: the denser cities are, the fewer emissions per capita.

Unsurprisingly, these effects are quite far from the original aims of the green belt policy:
  • To check the unrestricted sprawl of large built-up areas
  • To prevent neighbouring towns from merging into one another
  • To assist in safeguarding the countryside from encroachment
  • To preserve the setting and special character of historic towns
  • To assist in urban regeneration, by encouraging the recycling of derelict and other urban land
These goals aren’t intrinsically bad by themselves. But in the current context - given other policies in place - they’re disastrous and have relocated the urban sprawl dozens of kilometres outward in a way that’s worse for almost everyone involved. You can’t simultaneously stop cities from spreading out and stop them from building upwards unless you’re willing to have housing prices spike or go full totalitarian and just stop people from moving there. You can’t have urban regeneration while also putting hundreds of thousands of old, decrepit buildings on heritage lists so they can’t be replaced (discussed later in this essay). And preserving the “special character” of historic towns can’t be done without shutting out young people who want to be upwardly mobile (also explored later).

Isn’t the third goal on the list above legitimate, at least? Isn't there a worry that the UK might run out of countryside, with so many of its citizens pursuing life in urban centres? If you ask people to guess how much of UK land is "densely built on", the average guess is 47%. The true figure is 0.1%. If we include "built on" land, we get all the way up to 6%. People's perceptions on this issue are phenomenally inaccurate. There is not and probably never will be any shortage of land in this country or any other (excepting a handful of city-states).

There is a shortage of land in many cities, though, fueled by these misperceptions and the green belts themselves. People are irrationally fond of green belts, but we can easily solve the land shortage without environmental damage. In fact, 37% of the green belt is actually intensively farmed agricultural land, which is actively harmful to the environment! By contrast, just 1% of green belt land could hold enough housing to meet demand for several decades. Yet even that much is politically unacceptable, because Britons have been seduced by the mere use of the word “green”. The result? Buying land makes up 70% of the cost of a new home in London, up from just 25% a few decades ago. And ironically enough, this makes it far more difficult and expensive to create green spaces near where Londoners actually live.

Supply vs demand

The fundamental problem is that too many people want to live in too few houses. There are two ways that such mismatches get resolved: either the supply increases or the demand decreases. I've talked above about why the supply is artificially constrained; so instead, basic economic theory says that the price must increase until demand drops significantly. There are other ways to decrease demand, of course - for example, making it more attractive to live in smaller cities, or improving public transport so that people can commute from further away. These aren't ideal solutions, but they're better than nothing. Instead, national and city governments across the US and UK have taken what might be the most moronic strategy possible: they've implemented policies which increase demand and decrease supply. Californian cities have widespread rent control, which limits the rate at which rent can increase. So people who otherwise wouldn't stay in the city are able to. But that feel-good story is outweighed by the other clear effects: more people are competing for the remaining housing stock, so its prices go up even further; and what's worse, construction is diverted from high-density apartments (which run the risk of being declared rent-controlled properties) to larger luxury developments. Meanwhile, the UK government has been rolling out zero-interest loans for first-time home buyers - which means that they'll bid more for houses than they would have otherwise, again increasing demand and driving prices up. We’ve seen these sort of policies fail spectacularly before - it was a similar desire to increase home ownership which motivated Bill Clinton when he pressured Fannie Mae and Freddie Mac into handing out subprime mortgages like candy, which led to a major housing bubble and subsequent global recession.

If you're not yet convinced that a lack of supply is the most important factor in rising house prices, consider the following graph (sourced from here) of house prices compared with the rate at which housing permits were issued in ten American tech hubs, showing a very clear negative correlation.

This chart shows is the median sale price per foot compared to the amount of construction relative to existing housing stock since 1990. Source: Trulia.

Misaligned incentives

It's not just outright stupidity which is behind the housing crisis, but also clashes of interests. Local homeowners often block development in their area (the catch-all term for such opposition is NIMBYism: Not In My Back Yard). It's often said that NIMBY groups are mainly trying to protect their own home prices, but I'm not convinced that's true - if you live in a central neighbourhood that suddenly allows the construction of skyscrapers, your land's value will soar (at least until everyone else follows suit). Instead, it seems, NIMBYism is about more intangible quality-of-life issues, like avoiding noise and traffic, or keeping the current character of the community. To an extent, this is understandable - I'm also very interested in building strong communities. But the overall trend is regressive because even if it benefits property owners, it harms younger and poorer people, who are much less likely to own houses. It also cuts away at social mobility, because it takes much more capital to move to places where the opportunities are. Inter-state migration within the US has fallen sharply over the last few decades, as younger workers are faced with job and housing markets dominated by an ageing population. Some YIMBY (Yes In My Back Yard) groups are doing good work against NIMBYism, particularly in California, but they don't yet have much traction by comparison (although kudos to Stripe, who recently donated $1,000,000; I think other tech companies are also starting to do serious advocacy).

Other regulatory issues
  • There was only one factor on which all interviewees and focus group participants agreed: the most significant and pointless factor driving up construction costs was the length of time it takes for a project to get through the city permitting and development processes… Participants noted that “additional hoops and requirements seem to pop up at various stages in the process” and that projects are subject to “re-interpretation of the codes throughout the permitting process.”... Another challenge is the frequency of appeals on projects, including affordable or workforce housing units. In San Francisco, every permit is appealable; since very few large-scale projects match the city’s existing building and planning codes, this translates into numerous opportunities for appeals and contributes to delays in the entitlement process. Focus group participants noted that a large percent of appeals are eventually denied, meaning that the time spent on the appeal does not typically produce different outcomes other than increasing the time and cost in pre-construction. “There’s no benefit to the public – the need to get ‘exceptions’ approved just adds costs.”
  • Protected views. There are 13 protected views of St Paul's Cathedral, the Palace of Westminster and the White Tower, which mean that tall buildings are prohibited along a series of lines which criss-cross London. In Edinburgh, there are 170 protected views; this is also a serious problem in San Francisco.
  • Protection of historic buildings, of which there are 377,587 across England. History is important, but that's an absurdly high number. Having a building listed as historic can be such a financial blow that the government also offers Certificates of Immunity from listing. Presumably the buildings they're protecting are actually important, at least? Well, in San Francisco a proposal to replace a laundromat with 75 housing units, which had already been delayed for several years getting permits, is now at risk of being cancelled because several charities used to meet in the building in the 70s - even though there's no trace of them left. (One of the charities was focused on making housing more affordable, a stroke of irony you really couldn't make up). Also, there's no good way to get a building off the list, since it's practically tautological that buildings become more historic over time (the UK website suggests that they might consider doing so if the building has literally burned down).
  • Mandatory parking requirements and very cheap street parking essentially force people to pay for driving facilities they don't even use. Fortunately, London abolished the former in 2004 - and saw the number of private carparks built halve, showing that people simply didn't want to pay for more parking. Which is not surprising, considering that each above-ground parking space costs around $25000, and each underground one costs around $35000. Meeting Los Angeles' minimum requirement for parking spaces increases the cost of building a new shopping centre there by around 60 to 90%! That's assuming it's even possible - if not, the development would need to be cancelled entirely.
  • Zoning, which is particularly bad in the US, and contributes to urban deserts and the need to drive everywhere. The Reserve Bank of Australia estimates that “zoning raised detached house prices 73% above marginal costs in Sydney and 69% in Melbourne”.
  • Lots and lots of other miscellaneous crap - like California's new law that requires solar panels on the roof of every new building, despite the fact that they're much less efficient and much more expensive than industrial installations; or Austrian villages where, by law, every house must be a picturesque 19th-century-style cottage; or Oxford, where no building is allowed to be taller than Carfax Tower (i.e. four stories); or...
Other possible causes

Overall, I think that by cutting regulation, we could build much, much more housing than we currently do, and make it much more affordable than it currently is, and that would be a very good thing. Although this is the consensus amongst economists, it hasn’t been particularly popular politically. A number of alternative causes of our housing crises are often thrown around, some of which have merit and some of which don't. Let's have a look at them:

  • Interest rates
Interest rates have dropped precipitously over the last few decades from their highs in the 70s and 80s. This makes mortgages cheaper and thereby pushes the price of houses up. We can see that interest rates have had a huge effect on house prices by looking at rental prices, which aren't as dependent on interest rates, and haven't increased nearly as much. One Oxford economist cites the graph below, which claims that the median percentage of income spent on rent has only increased marginally in London over the last two decades, as clear-cut evidence: "you cannot argue that building too few houses has anything to do with why houses have suddenly become unaffordable for young people."
Case closed? I don't think so, few a few reasons. Firstly, there are many cities in which rents have increased drastically: the cost of rent in New York as a percentage of national median income has gone from 10% in 1950 to over 80% now. Given this, I'm somewhat suspicious of the London data. Compare, for instance, the next graph, which paints a contradictory picture, showing a 23% rise since 2011 alone.
Secondly, I don't see any good reason why housing costs should be measured as a percentage of income, rather than in real terms. If you account for increasing incomes, the chart above actually shows housing in London becoming 40% more expensive over the last two decades. If that happened for any other life essential, like food, people would consider it a scandal regardless of whether incomes had increased as well.

More importantly, even if interest rates drove house prices up in the first place, the natural response is for more construction to drive prices down again.  This is simple supply and demand - given higher prices, construction firms can expand their activities and make more profit. Since that hasn't happened, something must be preventing them from doing so. I suggest that the factors I outlined above are responsible both for the rise in rental prices and for the vulnerability of the housing market to changes in interest rates. There are some reasons to think that more people renting is a good thing, because it allows them to be more flexible and mobile. But people seem to have a strong revealed preference for owning their own houses, and so are disincentivised from moving to the places with high productivity and high house prices, which undermines the whole point of increasing mobility. Pro-homeowning attitudes also benefit the economy overall by increasing saving and building stronger communities. So I do think that pushing people into renting is a significant harm, one which is further exacerbated by recent increases in rents.

  • Developers only building for the wealthy
This is less of a problem than it sounds because whenever developers build new housing for the wealthy, it means that they no longer occupy other accommodation, which allows middle-class people to move in, which frees up other accommodation, and so on... And in fact the luxury housing of 20 years ago often becomes the ordinary housing of today. But there is a valid underlying concern, which is that luxury units aren't very high-density, and so they’re not a particularly good use of desirable central land. So why are they being built? Well, if you raise the costs of development by making it more difficult to get planning permission and more expensive to build large numbers of apartments, then the rational response for developers is to focus on projects with higher margins, i.e. luxury accommodation or office spaces. Mandatory parking minimums showcase this point well - if you have to allow for one parking space per apartment even though the market won't cover that cost, then by making each apartment bigger, and building fewer of them, you lose less money on parking (rent controls have a similar effect, as I outlined above). The best way to get more cheap housing is simply to make construction cheaper, and the best way to do that is still by decreasing regulation.

  • Cuts to government homebuilding
Yes, this is a valid and fair concern. Governments used to be much more involved in construction in the mid-20th century before neoliberal administrations promoted less interventionist policies. Both approaches have merits - but now, it seems, we’re stuck in an uncanny valley between the two. The right wing has just enough power to severely limit government construction, and the left wing has just enough power to severely limit private construction. At this point, I’d happily welcome increases in either. But government construction is still only a temporary solution, because if current regulations remain in place then new buildings will always be inefficient in terms of building costs and land use, and will therefore be either expensive to buy or a drain on public funds.

  • Foreign investment
There's often populist outrage about foreigners buying properties only to leave them empty. Londoners blame Russian oligarchs; Aucklanders blame the Chinese. Firstly, we should recognise that the number of properties involved is pretty small. But more importantly, we should think of this as a golden opportunity. It's good for domestic companies to be able to sell goods overseas, right? Well, because of our countries' various advantages, such as stability and the rule of law, foreigners are willing to buy certain products that only we can manufacture - namely, houses in our cities - at high prices. And the gain from this sale is a lasting one, because such investors are still paying property taxes, but not using any local services! If housing were something with a small fixed supply, then perhaps we should be worried. But I've already argued that even in big cities like London, housing is scarce not for intrinsic reasons, but rather because of regulatory restrictions. Cutting off foreign investors might ease the housing crisis temporarily, but in the long term it's just depriving ourselves of a valuable source of income.

  • Airbnb
Exactly the same is true when it comes to tourists. Countries like New Zealand profit greatly off tourism revenue, which is responsible for 10% of our GDP. Tourists wouldn’t inflate the prices of AirBNBs nearly as much if there were more affordable hotels - the lack of which can be traced to all the problems outlined above. Banning AirBNBs and thereby reducing tourism may slow the growth of housing prices in the short term, but the long-term reduction in revenue will be in the billions, and the root causes of the problem will remain.

  • Too much immigration
Amongst economists, immigration is almost universally acknowledged as beneficial for both the countries accepting those immigrants and the immigrants themselves There's probably a point where cultural concerns start to outweigh the clear economic gains, but I think there are very few countries anywhere near that point (although I concede that this is rather subjective). Immigrants are unusually entrepreneurial, fill important roles in the labour market, and revitalise our ageing populations. As with the two points above, cutting immigration might help with housing, but would cause even worse economic problems in the long term (see Japan's population implosion, for instance). And it's not even clear that it would do the former, because...

  • Too little immigration
Western countries just don't have enough construction workers to build the houses we need at reasonable prices. Immigrants have always been a source of cheap labour, but right now many countries are cracking down on them via xenophobic policies. Even increasing the allowances for temporary workers would help, but both the UK and US administrations are irrationally terrified of overstayers (to the point of cracking down even on graduate students, one of the demographic groups that have the most to contribute). It's a chicken-and-egg problem, of sorts, since I think people would be much more open to immigration if they were doing better economically, and they'd be doing much better economically if they were more open to immigration.

  • Universities
Why won't domestic workers go into construction, even when they're guaranteed high salaries? There are a number of reasons; but the biggest factor, I think, is a cultural bias against what used to be perfectly respectable blue-collar jobs, so that very few students think of them as acceptable careers. And a driving influence behind that is the massive growth of higher education, which is essentially a funnel into white-collar jobs. I don't want to go into all the arguments surrounding this now, but it's something to keep in mind.

  • Transport
As Matthew suggests in a comment below, high-speed rail links could lower prices by allowing people to commute from further without taking longer. This wouldn't be a terrible alternative to allowing more construction, but Western governments have generally proved themselves pretty incompetent at creating and running good public transport, so it's not an option I'm particularly optimistic about. By contrast, to increase construction the government just needs to stop doing things, which in theory is a fairly simple step.

Some argue instead that improving transport isn't a substitute for increasing housing density, but a prerequisite for it, because otherwise transport systems will be overwhelmed. I don't know enough about transport infrastructure to evaluate this thoroughly, but it seems odd to me because if people live further away and commute in to city centers, that surely creates even more congestion, especially at peak hours. Living centrally allows more sustainable alternatives like biking or bussing - and with increased wealth from bigger populations, central city governments will have more money to sink into capital-intensive forms of transport like trains and metros.

  • Something else?
As this article eloquently argues, it seems like prices are increasing massively in a variety of areas, from healthcare to education to housing. While Baumol's cost disease is often cited as the underlying cause, it doesn't seem to explain the whole phenomenon. Perhaps housing prices are just one part of a bigger trend which is caused by something entirely different. But that seems fairly unlikely to me, because there's a simple chain of cause and effect from all the factors I've outlined above to the increasing unaffordability of housing.

Other effects

So far, I've mainly talked about the financial and economic harms of the housing crisis. But because housing is such a foundational part of our lives, it also has flow-on effects in many areas.

For example, housing startup Zillow found that over the last five years, the average age of first-time home buyers in the US rose by 2.7 years. This is an incredibly sharp increase which turns out to be correlated with a significant decrease in birth rates. That is, the more house prices in an area increase, the fewer children its residents have. While this relationship hasn't been proved to be causal, it suggests that house prices are disrupting people's ability to even have the types of families they desire.

Another metric which has been rising is commuting time, as people need to live further and further away from city centres. Unsurprisingly, this turns out to be strongly correlated with unhappiness and anxiety. People with long commutes are less satisfied with their lives and feel like their daily activities are less worthwhile.

What else affects happiness? Possibly the biggest single factor is quality of social relationships. With higher housing prices, though, it's more difficult to find accommodation near your friends, because you're all much more financially constrained. Increases in the proportion of people renting are also detrimental to relationships: because they move around frequently, renters find it harder to build up deep ties with neighbourhood communities. These are likely factors driving the current loneliness epidemic.

Lastly, you may have heard of Piketty's book Capital in the 21st Century, which argues that the returns to capital are increasing, leading to the rise of a new rentier class at the expense of workers. But this reanalysis argues that when depreciation is accounted for, the only type of capital whose returns are rising is housing! So any discussion of rising inequality should consider the role of housing regulations in funneling income towards the wealthy.

In summary

The developed world is in the grip of a massive, multi-decade housing crisis, which is exacerbating other social and political issues. Many people are very concerned about directly working towards social justice, but I honestly think that the single best thing you could do for minorities and the poor is to make it financially viable for them to live in good communities near good jobs, and prevent the entrenchment of the wealth of landowners. There’d also be a lot less negative sentiment about “immigrants taking our jobs ” with the employment boom that would result from relocation being affordable. These effects are not just economic, but directly affect people's relationships and happiness. The housing crisis is even harming the environment, since detached suburban homes with long commutes are the least eco-friendly type of housing.

But it doesn’t have to be this way! South Korea, Germany and Japan have all kept house prices low, and not through lack of demand either. Landowners in Japan have the right to do practically whatever they want with their property - and so in Tokyo alone more housing units are started in a year than in all of California or all of England. This in a city where there’s barely any empty land left! By good luck or good planning, a few countries have demonstrated that the biggest problem facing modern cities is eminently avoidable. The rest of us need to follow their lead.