Saturday, 31 March 2018

Deep learning - deeper flaws?

In this post I summarise four lines of argument for why we should be skeptical about the potential of deep learning in its current form. I am fairly confident that the next breakthroughs in AI will come from some variety of neural network, but I think several of the objections below are quite a long way from being overcome.

Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution - Pearl, 2018

Pearl describes three levels at which you can make inferences: association, intervention, and counterfactual. The first is statistical, identifying correlations - this is the level at which deep learning operates. The intervention level is about changes to the present or future - it answers questions like "What will happen if I do y?" The counterfactual level answers questions like "What would have happened if y had occurred?" Each successive level is strictly more powerful than the previous one: you can't figure out what the effects of an action will be just on the association level, without a causal model, since we treat actions as interventions which override existing causes. Unfortunately, current machine learning systems are largely model-free.

Causal assumptions and conclusions can be encoded in the form of graphical models, where a directed arrow between two nodes represents a causal influence. Constraints on the structure of a graph can be determined by seeing which pairs of variables are independent when controlling for which other variables: sometimes controlling removes dependencies, but sometimes it introduces them. Pearl's main claim is that this sort of model-driven causal analysis is an essential step towards building human-level reasoning capabilities. He identifies several important concepts - such as counterfactuals, confounding, causation, and incomplete or biased data - which his framework is able to reason about, but which current approaches to ML cannot deal with.

Deep Learning: A Critical Appraisal - Marcus, 2018

Marcus identifies ten limitations of current deep learning systems, and argues that the whole field may be about to hit a wall. According to him, deep learning:
  1. Is data hungry - it can't learn abstractions through explicit verbal definition like humans can, but instead requires thousands of examples.
  2. Is shallow, with limited capacity for transfer. If a task is perturbed even in minor ways, deep learning breaks, demonstrating that it's not really learning the underlying concepts. Adversarial examples showcase this effect.
  3. Has no natural way to deal with hierarchical structure. Even recursive neural networks require fixed sentence trees to be precomputed. See my summary of 'Generalisation without systematicity' below.
  4. Struggles with open-ended inference, especially based on real-world knowledge.
  5. Isn't transparent, and remains essentially a "black box".
  6. Is not well-integrated with prior knowledge. We can't encode our understanding of physics into a neural network, for example.
  7. Cannot distinguish causation from correlation - see my summary of Pearl's paper above.
  8. Presumes a largely stable world, like a game, instead of one like our own in which there are large-scale changes.
  9. Is vulnerable to adversarial examples, which can be constructed quite easily.
  10. Isn't robust as a long-term engineering solution, especially on novel data.
Some of these problems seem like they can be overcome without novel insights, given enough engineering effort and compute, but others are more fundamental. One interpretation: deep learning can interpolate within the training space, but can't extrapolate to outside the training space, even in ways which seem natural to humans. One of Marcus' examples: when a neural network is trained to learn the identity function on even numbers, it rounds down on odd numbers. In this trivial case we can solve the problem by adding odd training examples or manually adjusting some weights, but in general, when there are many features, both may be prohibitively difficult even if we want to make a simple adjustment. To address this and other problems, Marcus offers three alternatives to deep learning as currently practiced:
  1. Unsupervised learning, so that systems can constantly improve - for example by predicting the next time-step and updating afterwards, or else by setting itself challenges and learning from doing them.
  2. Further development of symbolic AI. While this has in the past proved brittle, the idea of integrating symbolic representations into neural networks has great promise.
  3. Drawing inspiration from humans, in particular from cognitive and developmental psychology, how we develop commonsense knowledge, and our understanding of narrative.
Generalisation without systematicity - Lake and Baroni, 2018

Lake and Baroni identify that human language and thought feature "systematic compositionality": we are able to combine known components in novel ways to produce arbitrarily many new ideas. To test neural networks on this, they introduce SCAN, a language consisting of commands such as "jump around left twice and walk opposite right thrice". While they found that RNNs were able to generalise well on new strings similar in form to previous strings, performance dropped sharply in other cases. For example, the best result dropped from 99.9% to 20.8% when the test examples were longer than any training example, even though they were constructed using the same compositional rules. Also, when a command such as "jump" had only been seen by itself in training, RNNs were almost entirely incapable of understanding instructions such as "turn right and jump". The overall conclusion: that neural networks can't extract systematic rules from training data, and so can't generalise compositionality anything like how humans can. This is similar to the result of a project I recently carried out, in which I found that capsule networks which had been trained to recognise transformed inputs such as rotated digits and digits with negative colours still couldn't recognise rotated, negated digits: they were simply not learning general rules which could be composed together.

Deep reinforcement learning doesn't work yet - Irpan, 2018

Irpan runs through a number of reasons to be skeptical about using deep learning for RL problems. For one thing, deep RL is still very data-inefficient: DeepMind's Rainbow DQN takes around 83 hours of gameplay to reach human-level performance on an Atari game. By contrast, humans can pick them up within a minute or two. He also points out that other RL methods often work better than deep RL, particularly model-based ones which can utilise domain-specific knowledge.

Another issue with RL in general is that designing reward functions is difficult. This is a theme in AI safety - specifically when it comes to reward functions which encapsulate human values - but there are plenty of existing examples of reward hacking on much simpler tasks. One important consideration is the tradeoff is between shaped and sparse rewards. Sparse rewards only occur at the goal state, and so can be fairly precise, but are usually too difficult to reach directly. Shaped rewards give positive feedback more frequently, but are easier to hack. And even when shaped rewards are designed carefully, RL agents often find themselves in local optima. This is particularly prevalent in multi-agent systems, where each agent can overfit to the behaviour of the other.

Lastly, RL is unstable in a way that supervised learning isn't. Even successful implementations often fail to find a decent solution 20 or 30% of the time, depending on the random seed with which they are initialised. In fact, there are very few real-world success stories featuring RL. Yet achieving superhuman performance on a wide range of tasks is a matter of when, not if, and so I think Amara's law applies: we overestimate the effects RL will have in the short run, but underestimate its effects in the long run.

Thursday, 22 March 2018

Topics on my mind: February 2018

Thanks to Tom for pointing me to this presentation by a couple of Oxford philosophers who may just have solved Fermi's Paradox (the puzzle of why we haven't seen any alien civilisations, given that plausible estimates for terms in the Drake equation predict that there should be many thousands in this galaxy alone). The main idea is actually quite simple: that people have generally tried to find plausible expected values for each term, then multiplied them together to give an expected number of alien civilisations in the thousands or millions. But if we instead multiply together a whole probability distribution for each term, we get the same mean but a median less than 10, and a double-digit probability that there are no other alien civilisations (the first couple of slides explain this quite well). Their final best guess, given our observations so far, is that there's something like a 40% chance that we're alone in the universe. While this isn't a particularly definitive answer, I am impressed because the Fermi paradox was one of a very few questions which felt truly mysterious (the others are consciousness, the nature of maths, the origin and nature of the uni(multi?)verse, and how abstract thought occurs in the brain), and so it's very cool to see this sort of progress.

I mentioned last month that I was beginning to regret my former dismissive attitude towards continental philosophy. Upon further reflection, I think it was actually just one example of a more general mistake which I've been making, which has the following steps: 1. Observe philosophers trying to analyse a concept; 2. Notice that the concept is vague or messy and there is probably no underlying "truth", which makes most analyses aimed at finding it predictably fail; 3. Think that I've "dissolved the question" enough that it's not worth worrying about any more.

But in fact, there's often a lot more important conceptual work to do even if there is no metaphysical truth. A good example is causation. Metaphysically, causation is a mess; it seems clear that human intuitions are a hodgepodge of whatever was evolutionarily useful, and there may not be any coherent truth about what caused what. But that didn't stop Judea Pearl from revolutionising the analysis of causality in statistics and machine learning by formalising causal reasoning using directed graphs. Some more questions which we can hopefully make similar progress on: what does it mean to implement a computation; what is a "meaningful life"; what is knowledge? (The last has already been partially solved by Bayesianism). Perhaps the most important application of this argument is to language itself. Almost all nouns, verbs and adjectives are vague and fail to specify a precise set of objects or actions; acknowledging this provides a simple resolution to philosophical dilemmas based on edge cases, such as the Ship of Theseus "paradox". But just noticing that vagueness isn't enough: the AIs we are building need to represent sentences somehow, or else we'll be unable to communicate important concepts to them, which seems pretty dangerous. Also note that the concept of vagueness is itself vague; it seems likely that better analyses of vagueness will help us solve the problems above, because then we'll actually understand what we're grappling with.

I've also been thinking about how to take "epistemic modesty" seriously. I've written about this before here, and it's also related to my discussion of cluelessness last month. (See also this new Stanford Encyclopedia of Philosophy article on the topic). Intellectually, I don't there's any clear place to draw a line between sensible (and indeed obligatory) modesty ("all the experts disagree with me, maybe I should reconsider") and radical skepticism ("but how do I know that I'm not insane?"): both are about reevaluating your beliefs based on the difficulty of identifying mistakes in your reasoning "from the inside". Practically speaking, though, I think the most effective step towards minimising intellectual overconfidence would be to deliberately and consciously try to suppress feelings of "tribal affiliation" when having discussions, especially about things I consider part of my identity. Other strategies I've heard about: when arguing, first rephrase the other person's ideas in a way that they're happy with, before giving your own argument (this is basically an ideological Turing test). Apparently the Wright brothers took this even further, and used to try switching sides whenever they argued. In fact, when disputants are close to ideal Bayesians, each would offer new considerations favouring each side with roughly equal frequency until their beliefs converged. Some of the best discussions I've had looked roughly like that; and, I hope, many more will.

Speculations on improving debating

I was recently discussing with a friend whether or not competitive debating makes you better at figuring out what is actually true. This is an interesting question, because debating influences people in a variety of different ways. The most basic is that participating in debates, or preparing for them, teaches you a lot of facts about the world; I would know much less about international politics in particular were it not for debating. It also markedly improves your ability to notice flaws and fallacies in arguments, which obviously helps you avoid falling for them. This skill can extend too far, though: debaters can be very good at rebutting even very sensible arguments. In the American Parliamentary style of debate, the proposing team is allowed to choose and prepare in advance any motion which is not so blatantly, obviously true that it would be impossible to debate (e.g. "murder is bad"). A lot of proposition teams skirt quite close to that line - but even so, opposing teams win a majority of the time. Debaters can poke holes in essentially any argument.

To be fair, debaters are aware that the existence of convincing rebuttals doesn't necessarily invalidate a claim. But motivated reasoning can be a powerful force, especially amongst intelligent people. This is particularly true when you've been trained to respond oppositionally to any claim that doesn't support your current position. I can think of several cases where I've been able to ignore a nagging voice of doubt by quickly finding a plausible response that turned out to be incorrect. The social media filter bubbles that we all currently inhabit can amplify this effect by caching in your mind hundreds of examples engineered to be memorable and stoke outrage; to counter it, it's more important than ever to genuinely consider opposing arguments rather than going to the automatic rebuttal mode that debating ingrains.

The other thing that debating (arguably) does is cultivate a non-empirical mindset. At the end of the debate, the question isn't settled. There may be more clarity over exactly which factors might swing a conclusion one way or the other, but almost nobody ever bothers to go out and do that further research, or even fact-check claims made during the debate - despite the fact that it's now easier to access such data than ever before. Of course if you asked debaters explicitly they'd be clear that empirical results are usually required to actually draw firm conclusions. But if you spend long enough thinking and arguing in a certain way, it's difficult to imagine that it doesn't carry over to your reasoning in general.

At this point I want to be a little speculative. I'm not sure whether the issues I discussed above can be addressed within anything resembling a traditional debate (I'd be interested to hear your comments on this). But let's say that we want to design an entirely new form of debating which instilled the best possible habits of thought. What could it look like? It would need to be driven by empiricism, while still having room for conceptual analysis. It would still be fun and competitive, but debates would also build up the sort of knowledge which could actually help drive decisions. Domain-specific knowledge would be helpful but not essential.

This was in the back of my mind when I read about two very interesting studies. The first was by researchers at Uber: they calculated the willingness of consumers to pay for Uber rides by comparing times when the calculated surge index was very similar, but the actual surge price was different, e.g. 2.24 vs 2.25, rounded to 2x and 2.5x price increases respectively. The study was heavily criticised for equating willingness to pay increased surge prices with overall "consumer surplus" created by Uber, but the initial methodology was still quite clever. The second studied Italy, which is apparently the bank robbery capital of Europe. Researchers analysed the relationship between duration of robberies, amount stolen, probability of being caught and prison sentence. Their hope was that there would be some consistent tradeoff between the expected gain and expected punishment; it turned out that more capable robbers behaved as if they assigned higher disutility to prison time than less capable ones. This sort of creative approach to answering important questions is something we sorely need more of; let's consider a new form of debating, neodebating, which is aimed at encouraging it.

The core ideas underlying neodebating would be that, instead of debating about what is true, we could debate about how to find out what is true; and instead of debating what the effects of a certain policy would be, we could debate which policy would best create certain effects. In some debates this would look like both sides designing experimental methodologies like the ones I outlined above, then critiquing and defending them. In others, it would require teams to dream up specific interventions, e.g. "What's the best way to build stronger community spirit in deprived areas?" Debates could focus on personal decisions: "Your best friend is going through a midlife crisis and feels like their life is meaningless. What do you do about it?" But they could also be about some of the biggest modern issues: "Redesign the education system to make it as effective as possible in conveying skills and knowledge." These debates would be judged on the current standards of persuasiveness and eloquence, but also on creativity and boldness. The most fun debates I've ever done are the ones which introduced me to totally new ideas: in an ideal implementation of neodebating, every debate would be like that. And I think the mindset required is exactly what we should want from political leaders - an innovative, experimental approach to finding the best policies, plus knowledge of how to test and evaluate them. It would even promote the statistical literacy required to identify and argue about correlation, causation, confounders and controls. Wouldn't that be great? If I were being really idealistic, I'd even build in a mechanism for teams in a debate to make bets about relevant future events, with points awarded retroactively to whichever team turned out to be correct.

Anyway, enough daydreaming. For now, I think my main point is that everyone - but debaters in particular - should spend some time answering a few questions. Which arguments do you critique most rigorously? Do you ever try to generate novel and creative solutions to big problems? Do you seek out enough empirical data? And when and how do you actually change your mind?