Wednesday, 29 April 2020

Melting democracy

Liquid democracy, also known as delegative democracy, is a proposed hybrid between direct democracy and representative democracy. In liquid democracy, as in direct democracy, everyone is able to vote directly on every issue. Of course, in a modern society it’d be prohibitively time-consuming for everyone to research every topic well enough to cast an informed vote. In practice, most who vote would end up following the advice of someone better-informed on the topic. Liquid democracy makes this dynamic explicit by allowing any voter to delegate their vote to someone else that they trust. Their delegate can now vote on their behalf, or else delegate again to someone else.

Some implementation details of how this system would work:
  • If A delegates to B, and B delegates to C, then C now has three votes. There’s no limit to how many times votes can be delegated (except that forming a cycle returns a vote to its original owner).
  • It’s a real-time system: any delegation can be changed at any time. In practice we’d need to implement voting and delegation via a secure digital system.
  • Voting and delegation would be private for most people, but public for those who have more than a certain number of pledged votes.
  • People can also split their delegation by subject area, so they don’t need to find a delegate whose judgement they trust on every topic. For example, A could delegate their vote on economic issues to B, their vote on foreign policy to C, and their vote on everything else to D.

I have mixed feelings about liquid democracy. There are quite a few reasons why direct democracy is a bad idea, and liquid democracy only solves some of them. In particular, the role of representatives isn’t just to cast votes, but also to design, advocate, and implement good policies, which is a full-time job. Meanwhile there needs to be some way to make all those policies consistent with each other over time - otherwise voters might approve a budget constraint, and then approve individual policies expensive enough to overrun that budget constraint. Then, once the policies have been in effect for long enough to judge their success or failure, we also need to know which groups to praise or blame - since allocating responsibility on an individual basis is hard. In short, electing representatives who belong to political parties to serve multi-year terms in parliament seems like a robustly good idea.

But I also think that the core insights of liquid democracy are very important ones. People are best able to trust others around them, or public figures whose work they respect - not politicians who they mainly see during large-scale media campaigns. A hierarchical system of delegation is a natural way to scale up while maintaining trust in each link. And it’s also true that no representative can have detailed knowledge about every topic, so we should find some way of making specialist knowledge more accessible to our representatives.

So I’d like to propose a watered-down version of liquid democracy, which preserves these core insights, but without the problems of direct democracy - and which is also much easier to implement gradually. I envisage it happening in two phases.

Phase 1: Melting Democracy

We keep a party-based system of elected representatives, who serve multi-year terms. Everyone can delegate their votes as in liquid democracy, with the same implementation details as described in the previous section (except that people don’t get the opportunity to split their delegation by subject area). Those votes can then be used during parliamentary elections to choose between candidates running for election.

This may seem like only a modest change. But I think that if delegation is widely adopted, it will improve a number of important dynamics:
  1. Politicians need to run better campaigns. They can’t just use shallow large-scale media tactics, because delegates who have been pledged many votes (let’s call them superdelegates) will have more incentive to investigate platforms and policies in detail. Superdelegates would also have the opportunity to talk to candidates in person, to understand them better.
  2. Trusted figures get more involved in politics. Consider a public figure who’s demonstrated good judgement and trustworthiness in their previous work. While their endorsement might be useful for political candidates, they have no official power within the political system, which makes it more difficult for them to have a positive influence. Having a large number of pledged votes will give them leverage to push for better policies.
  3. A natural funnel into parliament. Running for parliament is difficult not just because of the costs, but also because it’s so abrupt. Candidates need to decide whether to commit to running before knowing how much voters like or trust them. How do we get the best people from other sectors of society to consider a political career? Under melting democracy, they can start gradually getting involved in politics by amassing pledged votes. And they don’t need to spend a lot of money to get started, as long as other superdelegates commit to supporting them.
  4. Easier for new parties to get traction. Right now there’s a coordination problem in supporting small parties for first-past-the-post parliamentary seats. Even if many voters prefer a challenger over the two entrenched parties, unless a large number of them switch at once, their votes will probably be wasted. Melting democracy makes this easier, because superdelegates can coordinate more easily to throw support behind small parties.

A potential problem is that we’re centralising voting power in superdelegates, which might open the door to corruption. I hope that this will not be a big problem, because anyone can be a delegate. If a politician is involved in some sort of scandal, voters have to wait several years to vote against them - and even then, if the politician is still backed by their party, there might not be any alternative candidate who’s any better. Whereas if a delegate does something morally or legally dubious, voters can immediately re-delegate to anyone else in the country. This doesn’t have any formal effect until the next election, but it does shift informal power away from them rapidly.

Meanwhile, there’s also a sense in which we’re decentralising power, because now in addition to political parties we also have a parallel system of delegates. It’s possible that delegates all end up segregating along party lines - but because the assignment of votes to delegates can be so fluid, I expect that many superdelegates will serve as independent voices who are capable of challenging entrenched parties.

Something else I particularly like about this proposal is that it could start without requiring legal changes. That is, you could initially set up an opt-in, non-binding system for delegating votes. While this first system wouldn’t provide all the benefits I outlined above, pledging a vote to someone would still be a way of giving them more influence, allowing them to push for beneficial policies more effectively, and paving the way for them to run for office later on. Then, if the system proves successful, it could gradually be integrated into the democratic process across different countries. This makes it much more realistic than proposals which would require sudden radical changes.

Phase 2: Functional Constituencies

The specific way that representatives are elected varies by country. In the UK and the US, all representatives need to win their seat in a given constituency or state, where only the votes of residents count. By contrast, New Zealand and Germany use mixed-member proportional (MMP) voting, in which each person gets to vote twice - once for their local MP, and once for a national political party. The resulting parliament is a mix of MPs who represent specific constituencies, and MPs who represent their party as a whole. I think this is an excellent system; it allows voters to select representatives who care about their local issues and are held responsible for them, while also keeping representation proportional to popularity on a national level (unlike the UK’s system).

But if we want representatives who are experts on and accountable for specific issues, then local issues aren’t the only ones which matter. Proponents of liquid democracy are correct in arguing that representatives can’t be experts in every topic they need to legislate on. So what if we added new seats for specific areas of expertise? I don’t think we’d want too many, but it seems valuable to have several representatives who we trust to represent us well on:
  • Economic issues
  • Social issues
  • Science and technology
  • Foreign policy

While I’ve framed this proposal in the context of MMP, variants could work in many other political systems; the core idea resembles functional constituencies in Hong Kong, and Ireland’s vocational panels. In contrast to the former, I think it’s important to keep the number of categories low, so that they don’t become de facto safe seats for special interest groups. However, if the system works well then more MPs could be added for each category, and maybe a few more domains included. In particular, I’d be excited to see dedicated seats for people with expertise in improving the governmental system itself.

In contrast to the Irish system, I think that everyone should be allowed to vote for functional representatives - because even though we’re aiming for expertise, there are also normative elements to each of these constituencies. There are several ways we might allocate votes to functional constituencies. The simplest version would be to allow everyone to vote for a representative in each area, in addition to their national and local votes. Voters could then delegate their six votes together or separately, as in liquid democracy. However, I think that’s asking for too much from voters; and it would encourage people to vote for expert representatives in areas where they can’t judge expertise. Probably it would be more reasonable to allow each person to choose only one of these constituencies in which to vote, in addition to their national and local votes.

In terms of parliamentary voting power, functional MPs probably wouldn’t have much direct sway. But they would be able to influence other MPs and work with them to create better policies. Right now experts are outsiders to the political process. The core goal of both the proposals I’ve advanced in this essay is to break down the barriers between the people whose judgement we trust in civil society, and the people to whom we’re entrusting the powers and responsibilities of government.

Thursday, 13 February 2020

Notes on The Historical Figure of Jesus

These are my notes from The Historical Figure of Jesus, by E. P. Sanders, an attempted biography of Jesus. I haven't evaluated the veracity of most of the claims he makes.

Historical background
  • The Kingdom of Judah, which David and Solomon had ruled, was conquered by the Babylonians in 586 BCE, then the Persians, then the Ptolemaic and Seleucid Empires.
  • The Maccabean revolt in 167 BCE succeeded in establishing the Hasmonean dynasty of Jewish leaders in Palestine for just over a century.
  • After a Hasmonean civil war, Pompey conquered Palestine in 63 BCE. After further unrest, Herod became the Rome-backed king of Palestine, as Jewish leader of a client state.
  • After Herod's death around 4BCE (also the time of Jesus' birth), his kingdom was split between three of his sons: in particular, Archelaus got Judea and Antipas got Galilee.
  • Popular unrest was common. Archelaus became unpopular, was deposed by Augustus, and replaced by a series of Roman governors. At the time of Jesus' death, this position was held by Pontius Pilate.
  • Even after Judea fell under Roman governance, almost all administrative matters were handled by the Jewish priests, particularly the high priest (Caiaphas, during Pilate's governorship).
  • Pharisees were not an official part of the priesthood, but rather a religious tradition that emphasised strict obedience to religious law.
  • Sanders identifies the key distinguishing feature of Jewish religious law as how "it brings the entirety of life, including civil and domestic practices, under the authority of God". While few individual Jewish practices were strikingly odd at the time, Jewish devotion to them was unusual, and prevented Jews from assimilating into other cultures.
Jesus' life
  • Jesus was from Nazareth in Galilee. It is doubtful he was born in Bethlehem, or that his family had to travel for a census, or that they later fled to Egypt; these were likely embellishments from the gospel authors to accentuate the parallels between him and David and Moses. He would have spoken Aramaic.
  • He became a follower of John the Baptist, who was executed by the Romans for preaching that end times were coming (and probably also for speaking out against Antipas). It is likely that Jesus also preached about the imminent coming of the Kingdom of God (which his disciples seem to have expected after his death). Sanders calls Jesus a "radical eschatologist" who "expected God to act in a decisive way, so as to change things fundamentally". This eschatological message became less prominent as Christianity grew.
  • Jesus then spent one to two years as an itinerant preacher in Galilee, particularly near a town called Capernaum. He spent most of his time in small villages and the countryside, rather than in cities.
  • He likely became known as a miracle worker, and particularly as an exorcist. This was not an uncommon reputation to have at the time - there are similar reports about some of his contemporaries. (As my own observation, it's interesting that Jesus is portrayed as emphasising the role of faith in healing, which might have helped create placebo effects).
  • He built up a following of disciples. It is unlikely that there were exactly 12 of them - rather, 12 is a symbolic number representing the historical tribes of Israel (descended from Jacob's sons). While the male disciples were emphasised, there were also women who supported and likely fed and housed the disciples.
  • Jesus went to Jerusalem around 30CE, was involved in an altercation with the Temple, and was subsequently put to death. The altercation may have take the form of a prophecy or "threat" of destruction.
  • Jewish priests were very keen to keep order, especially during festivals like Passover, to prevent the Roman governor from needing to step in.
  • Pilate was known for large-scale and ill-judged executions; it is quite plausible that he agreed to Jesus' execution merely on Caiaphas' recommendation.
Documentation and legacy
  • The gospels were written 30-80 years after Jesus' death. The four gospels chosen to form the New Testament (Matthew, Mark, Luke and John) were primarily composed of 'pericopes', small stories about Jesus that circulated orally after his death. The authorship of the gospels is uncertain; names were assigned based on later guesswork.
  • The synoptic gospels (Matthew, Mark and Luke) consist of many of the same stories, arranged in different orders and with different emphases. The gospel according to John differs considerably; it was written from a more theological perspective, attempting to convey later insights about Jesus' divine nature and its significance.
  • After Jesus' death, the leaders of the Christian movement were Simon Peter, John, and James (Jesus' brother). Saul/Paul, a converted opponent of Christianity, was also influential in reaching out to Gentiles, although the way he deemphasised Jewish customs was controversial. Paul wrote at least seven of the twenty-seven books in the New Testament, and several more may have been based on his writings.
  • The First Jewish-Roman War occurred from 66-73 CE. It was the first of three Jewish rebellions against Roman rule, and led to the destruction of the Second Temple in Jerusalem.
  • Josephus, who wrote The Jewish War in 75CE, was the main Jewish historian of this period. He spent only a paragraph discussing the followers of Christ, prompted by debates that they sparked in Jewish communities.

Wednesday, 1 January 2020

Characterising utopia

When we consider how good the best possible future might be, it's tempting to focus on only a handful of dimensions of change. In transhumanist thinking, these dimensions tend to be characteristics of individuals: their happiness, longevity, intelligence, and so on. [1] This reflects the deeply individualistic nature of our modern societies overall, and transhumanists in particular. Yet when asked what makes their lives meaningful, most people prioritise their relationships with others. In contrast, there are strands of utopian literature which focus on social reorganisation (such as Huxley’s Island or Skinner’s Walden Two), but usually without acknowledging the potential of technology to radically improve the human condition. [2] Meanwhile, utilitarians conceive of the best future as whichever one maximises a given metric of individual welfare - but those metrics are often criticised for oversimplifying the range of goods that people actually care about. [3] In this essay I've tried to be as comprehensive as possible in cataloguing the ways that the future could be much better than the present, which I've divided into three categories: individual lives, relationships with others, and humanity overall. Each section consists of a series of bullet points, with nested elaborations and examples.

I hesitated for quite some time before making this essay public, though, because it feels a little naive. Partly that’s because the different claims don’t form a single coherent narrative. But on reflection I think I endorse that: grand narratives are more seductive but also more likely to totally miss the point. Additionally, Holden Karnofsky has found that “the mere act of describing [a utopia] makes it sound top-down and centralized” in a way which people dislike - another reason why discussing individual characteristics is probably more productive.

Another concern is that even though there’s a strong historical trend towards increases in material quality of life, the same is not necessarily true for social quality of life. Indeed, in many ways the former impedes the latter. In particular, the less time people need to spend obtaining necessities, the more individualistic they’re able to be, and the more time they can spend on negative-sum status games. I don’t know how to solve this problem, or many others which currently prevent us from building a world that's good in all the ways I describe below. But I do believe that humanity has the potential to do so. [4] And having a clearer vision of utopia will likely motivate people to work on the problems that stand, as Dickinson put it, “between the bliss and me”. So what might be amazing about our future?

Individual lives
  • Health. Perhaps the clearest and most obvious way to improve the human condition is to cure the diseases and prevent the accidents which currently reduce both the quality and the duration of many people’s lives. Mental health problems are particularly neglected right now - solving those could make many people much better off.
    • Longevity. From some moral stances, the most important of these diseases to tackle is ageing itself, which prevents us from leading fulfilling lives many times longer than what people currently expect. Rejuvenation treatments could grant unlimited youth and postpone death arbitrarily. While the ethics and pragmatics of a post-death society are complicated (as I discuss here), this does not seem sufficient reason to tolerate the moral outrage of involuntary mortality.
  • Wealth. Nobody should lack access to whatever material goods they need to lead fulfilling lives. As technology advances and we automate more and more of the economy, the need to work to subsist will diminish, and eventually vanish altogether. An extrapolation from the last few centuries of development predicts that with centuries almost everyone will be incredibly wealthy by today’s standards. Luxuries that are now available only to a few (or to none at all) will become widespread.
    • Life in simulation. In the long term, the most complete way to achieve these two goals may be for us to spend almost all of our time in virtual reality, where possessions can be generated on demand, physical inconveniences will be eliminated, and our experiences will be limited only by our imaginations. Eventually this will likely lead to us uploading our minds and permanently inhabiting vast, shared virtual worlds. The key ideas in all of the points that follow this one are applicable whether we inhabit physical or virtual realities.
  • Alleviation of suffering. Evolution has stacked the hedonic deck against us: the extremes of pain are much greater than the extremes of pleasure, and more easily accessible too. But bioengineering and neuroscience will eventually reach a point where we could move towards eradicating suffering (including mental anguish and despair) and fulfilling the goal of David Pearce's abolitionist project. Perhaps keeping something similar to physical pain or mental frustration would still be useful for adding spice or variety to our lives - but it need not be anywhere near the worst and most hopeless extremes of either.
    • Freedom from violence and coercion. As part of this project, any utopia must prevent man’s inhumanity to man, and the savagery and cruelty which blight human history. This would be the continuation of a longstanding trend towards less violent and freer societies.
    • Non-humans. The most horrific suffering which currently exists is not inflicted on humans, but on the trillions of animals with which we share the planet. While most of this essay is focused on human lives and society, preventing the suffering of conscious non-human life (whether animals or aliens or AIs) is a major priority.
  • Deep pleasure and happiness. Broadly speaking, positive emotions are much more complicated than negative ones. Physical pleasure may be simple, but under the umbrella of happiness I also include excitement, contentment, satisfaction, wonder, joy, love, gratitude, amusement, ‘flow’, aesthetic appreciation, the feeling of human connection, and many more!
    • Better living through chemistry. There’s no fundamental reason why our minds couldn’t be reconfigured to experience much more of all of the positive emotions I just listed: why the ecstasy of the greatest day of your life couldn’t be your baseline state, with most days surging much higher; why all food couldn’t taste better than the best food you’ve ever had; why everyday activities couldn’t be more exhilarating than extreme sports.
    • Positive attitudes. Our happiness is crucially shaped by our patterns of thought - whether we’re optimistic and cheerful about our lives, rather than pessimistic and cynical. While I wouldn’t want a society in which people’s expectations were totally disconnected from reality, there’s a lot of room for people to have healthier mindsets and lead more satisfied lives.
    • Self-worth. In particular, it’s important for people to believe that they are valuable and worthwhile. In today’s society it’s far too easy to be plagued by low self-esteem, which poisons our ability to enjoy what we have.
    • Peak fun. Our society is already unprecedentedly entertainment-driven. With even fewer material constraints, we will be able to produce a lot of fun activities. Over time this will involve less passive consumption of media and more participation in exciting adventures that become intertwined with the rest of our lives. [5]
    • New types of happiness. The best experiences won’t necessarily come about just by scaling up our existing emotions, but also by creating new ones. Consider that our ability to appreciate music is an evolutionary accident, but one which deeply enriches our current lives. Our future selves could have many more types of experiences deliberately designed to be as rich and powerful as possible.
  • Choice and self-determination. Humans are more than happiness machines, though. We have dreams about our lives, and we devote ourselves to achieving them. While it’s not always straightforwardly good for people to be able to fulfil their desires (in particular desires involving superiority over other people, which I’ll discuss later), these activities give us purpose and meaning, and it seems unjust when we are unable to fulfil our plans because we are helpless in the face of external circumstances. Yet neither are the best desires those which can be fulfilled with the snap of a finger, or which steer us totally clear of any hardship. Rather, we should be able to set ourselves goals that are challenging yet achievable, goals which we might struggle with - but whose completion is ultimately even more fulfilling because of that. What might they be?
    • Making a difference to others. In a utopian future, dramatically improving other people’s lives would be much more difficult than it is today. Nevertheless, we can impact others via our relationships with them, as I’ll discuss in the next section.
    • Growth. People often set goals to push themselves, grow more and learn more. In those cases the specific achievements are less relevant than the lessons we take from them.
    • Tending your garden. Continuous striving isn’t for everyone. An alternative is the pursuit of peace and contentment, mindfulness and self-knowledge.
    • Self-expression. Everyone has a rich inner life, but most of us rarely (or never) find the means to express our true selves. I envisage unlocking the writer or musician or artist inside each of us - so that we can each tell our own story, and endless other stories most beautiful.
    • Life as art. I picture a world of “human beings who are new, unique, incomparable… who create themselves!” We can think of our lives as canvases upon which we each have the opportunity to paint a masterpiece. For some, that will simply involve pursuing all the other goods I describe in this essay. Others might prioritise making their lives novel, or dramatic, or aesthetically pleasing (even if that makes them less happy).
    • Life at a larger scale. With more favourable external circumstances, individuals will be able to shape their lives on an unprecedented scale. We could spend centuries on a single project, or muster together billions for vast cooperative ventures. We could also remain the “same” continuous person as long as we wanted, rather than inevitably losing touch of the past.
  • Cultivation of virtue. Although less emphasised in modern times, living a good life has long been associated with building character and developing virtues. Doing so is not primarily about changing what happens in our lives, but rather changing how we respond to it. There’s no definitive characterisation of a virtuous person, though: we all have our own intuitions about what traits (integrity, kindness, courage, and so on) we admire most in others. And different philosophical traditions emphasise different virtues, from Aristotle's 'greatness of soul' to Confucius' 'familial piety' to Buddha's ‘loving kindness’ (and the other brahmaviharas). [6] Deciding which virtues are most valuable is a task both for individuals and for society as a whole - with the goal of creating a world of people who have deliberately cultivated the best versions of themselves. [7]
  • Intelligence. As we are, we can comprehend many complex concepts, but there are whole worlds of thought that human-level intelligences can never fully understand. If a jump from chimpanzee brain size to our brain size opened up such vast cognitive vistas, imagine what else might be possible when we augment our current brains, scale up our intelligence arbitrarily far, and lay bare the patterns that compose the universe.
    • The joy of learning. Today, learning is usually a chore. Yet humans are fundamentally curious creatures; and there can be deep satisfaction in discovery and understanding. Education should be a game, which we master through play. We might even want to reframe science as a quest for hidden truths, so that each person can experience for themselves what it’s like to push forward the frontiers of knowledge.
    • Self-understanding. In many ways, we’re inscrutable even to ourselves, with our true beliefs and motivations hidden beneath the surface of consciousness. As we become more intelligent, we will better understand how we really work, fulfilling the longstanding, elusive quest to “know thyself”.
    • Agency. Each human is a collection of modules in a constant tug of war. We want one thing one day, and another the next. We procrastinate and we contradict ourselves and we succumb to life-ruining addictions. But this needn’t be the case. Imagine yourself as a unified agent, one who is able to make good choices for yourself, and stick to them - one who’s not overwhelmed by anger, or addiction, or other desires that your reflective self doesn’t endorse. This might be achieved by brain modification, or by having a particularly good AI assistant which knows how to nudge you into being a more consistent version of yourself.
    • Memory. Today we lose most of our experiences to forgetfulness. But we could (and have already started to) outsource our memories to more permanent storage media accessible demand, so that we can stay in touch with our pasts indefinitely.
    • The extended mind. Clark and Chalmers have argued that we should consider external thinking aids to be part of our minds. Right now these aids are very primitive, and interface with our brains in very limited ways - but that will certainly improve over time, until accessing the outputs of external computation is similar to any other step in our thinking. The result will be as if we’d each internalised all of human knowledge.
  • Variety and novelty of experiences.
    • Seeing the universe. The urge to travel and explore is a deep-rooted one. Eventually we will be able to roam as far as we like, and observe the wonder and grandeur of the cosmos.
    • Explorations of the human condition. Most of us inhabit fairly limited social circles, which don’t allow us to experience different ways of life and different people’s perspectives. Given the time to do so, we could learn a lot from the sheer variety of humanity, and import those lessons into the rest of our lives.
    • Explorations of consciousness. Right now the conscious states that we’re able to experience are limited to those induced by the handful of psychoactive chemicals that we or evolution have stumbled upon. Eventually, though, we will develop totally different ways of experiencing the world that are literally inconceivable to us today.
    • Spiritual experiences. One such mental shift that people already experience is the feeling of spiritual enlightenment. Aside from its religious connotations, this can be a valuable shift of perspective which give us new insights into how to live our lives.
  • Progress on our journeys. A key part of leading a meaningful life is continual growth and transcendence of one’s past self, each moving towards becoming the person we want to be. That might mean becoming a more virtuous person, or more successful, or more fulfilled - as long as we’re able to be proud of our achievements so far, and hopeful about the future.
    • Justified expectation of pleasant surprises. One important factor in creating this sensation of progress is uncertainty about exactly what the future has in store for us. Although we should be confident that our lives will become better, this should sometimes come in the form of pleasant surprises rather than just ticking off predictable checkpoints.
    • Levelling up. One way that this growth might occur is if people’s lives consist of distinct phases, each with different opportunities and challenges. Once someone thinks they have gained all that they desire from one phase, they can choose to move on. In an extreme case, the nature and goals of a subsequent phase might be incomprehensible to those in earlier phases - in the same way that children don’t understand what it’s like to be an adult, and most people don’t understand Buddhist enlightenment. For fictional precedent, consider sublimation in Banks’ Culture, or the elves leaving Middle-Earth in Tolkein’s mythos.
    • Guardrails. Extended lives should be very hard to irreversibly screw up, since there’s so much at stake - especially if we have much greater abilities to modify ourselves than we do today.
    • Leaving a legacy. People want to be remembered after they’ve moved on. Even in a world without death, each person should have had the opportunity to make a lasting difference in their communities before they leave for their next great adventure.
Relationships with others

For most of us, our relationships (with friends, family and romantic partners) are what we find most valuable in life. By that metric, though, it’s plausible that Westerners are poorer than we’ve ever been. What would it mean for our social lives to be as rich as our material lives have and will become? Imagine living in communities and societies that didn’t just allow you to pursue your best life, but were actively on your side - that were ideally designed to enable the flourishing of their inhabitants.
  • Stronger connections. Most relationships are nowhere near as loving or as fulfilling as they might ideally be. That might be because we’re afraid of vulnerability, or we don’t know how to nurture these relationships (knowing how to be a good friend is more valuable than almost anything learned in classes, but taught almost nowhere), or we simply struggle to find and spend time with people who complement us. Imagine a society which is as successful as solving these problems as ours has been at solving scientific and engineering problems, for example by designing better social norms, giving its citizens more time and space for each other, and teaching individuals to think about their relationships in the most constructive ways.
    • Abolishing loneliness. I envisage a future where loneliness has been systematically eradicated, by helping everyone find social environments in which they can flourish, and by providing comprehensive support for people struggling with building or maintaining relationships. I imagine too a future without the social anxieties which render many of us insecure and withdrawn.
    • Love, love, love. What would utopia be without romantic love and passion? This is an obsession of modern culture - and yet it’s also something that doesn’t always come naturally. We could improve romance by reducing the barriers of fear and insecurity, allowing people to better create true intimacy. Even the prosaic solutions of better educational materials and cultural norms might go a long way towards that.
    • Commitment and trust. In my mind, the key feature of both romance and friendship is deep commitment and trust, and the common knowledge that you’re each there for the other person. Whatever the bottlenecks are to more people building up that sort of bond - inability to communicate openly and honestly; or a lack of empathy; or even the absence of shared adversity - we could focus society’s efforts towards remedying.
    • Free love. While there’s excitement in the complex dance of romance, a lot of the hangups around sex serve only to make people anxious and unhappy. Consenting adults should feel empowered to pursue each other; and of course utopia should include some great sex.
    • Ending toxic relationships. We can reduce and manage the things that make relationships toxic, like jealousy, narcissism, and abuse. This might happen via mental health treatment, better education, better knowledge of how to raise well-socialised children, or cultural norms which facilitate healthy relationships.
    • Longer connections. I think it’s worth noting the positive effect that longevity could have on personal relationships. There’s a depth and a joy to being lifelong friends - but how much stronger could it be when those lives stretch out across astronomical timescales? This is not to say that we should bind ourselves to the same people for our whole extended lives - rather, we can spend time together and separate in the knowledge that it need never be a final parting, with each reunion a thing of joy.
  • Life as a group project. In addition to one-on-one relationships, there’s a lot of value in being part of a close-knit group with deep shared bonds - a circle of lifelong friends, or soldiers who trust each other with their lives, or a large and loving family. Many people don’t have any of these, but I hope that they could.
    • Better starts to life. The quality of relationships is most important for the most vulnerable among us. In a utopian future, every child would be raised with love, and allowed to enjoy the wonder of childhood; and indeed, they would keep that same wonder long into their adult lives. 
    • Less insular starts to life. Today, many children only have the opportunity to interact substantively with a handful of adults. While I’m unsure about fully-communal parenting, children who will become part of a broader community shouldn’t be shut off from that community; rather, they should have the chance to befriend and learn from a range of people. Meanwhile, spending more time with children would enrich the lives of many adults.
    • Families, extended. What is the most meaningful thing for the most people? Probably spending time with their children and grandchildren, and knowing that with their family they’ve created something unique and important. A utopian vision of family would have the same features, but with each person living to see their lineage branch out into a whole forest of descendants, with them at the root.
  • Healthy societies. In modern times our societies are too large and fragmented to be the close-knit groups I mentioned above. Yet people can also find meaning in being part of something much larger than themselves, and working together towards the common goal of building and maintaining a utopia.
    • Positive norms. The sort of behaviours that are socially encouraged and rewarded should be prosocial ones which contribute to the well-being of society as a whole.
    • (The good parts of) tribalism and patriotism. The feeling of being part of a cohesive group of people unified by a common purpose is a powerful one. At a small scale, we currently get this from watching sports, or singing in a choir. At larger scales, those same feelings often lead to harmful nationalist behaviour - yet at their best, they could give us a world in which people feel respect for and fraternity with all those around them by default, simply due to their shared humanity.
    • Tradition and continuity. Another key component of belonging to something larger than yourself is continuing a long-lived legacy. Traditions could be maintained over many millennia in a way which gives each person a sense of their place in history.
    • Political voices. Our current societies are too large for their overall directions to be meaningfully influenced by most people. But we can imagine mechanisms which allow individuals to weigh in on important questions in their local communities to a much greater extent. And people could at least know that their voice and vote have as much weight in the largest-scale decisions as anyone else’s.
  • Meetings of minds. Today, humans communicate through words and gestures and body language. These are very low-bandwidth channels, compared with what is theoretically possible. In particular, brain interfaces could allow direct communication from one person’s mind to another. That wouldn’t just be quicker talking, but a fundamentally different mode of communication, as if another person were finishing your own thoughts. And consider that our “selves” are not discrete entities, but are made up of many mental modules. If we link them in new ways, the boundaries between you and other people might become insubstantial - you might temporarily (or permanently) become one larger person.
  • Mitigating status judgements and dominance-seeking. In general we can’t hope to understand social interactions without considering status and hierarchy. We want to date the most attractive people and have the most prestigious jobs and become as wealthy as possible in large part to look better than others. The problem is that not everyone can reach the top, and so widespread competition to do so will leave many dissatisfied. In other cases, people are directly motivated to dominate and outcompete each other - such as businesspeople who want to crush their rivals. While this can be useful for driving progress, in the long term those motivations would ideally be channeled in ways which are more conducive to long-lasting fulfilment. For example, aggressive instincts could be directed towards recreational sports rather than relationships or careers.
    • Diverse scales of success. To make social dynamics more positive-sum, we should avoid sharing one single vision, which everyone is striving towards, of what a successful life looks. We can instead encourage people to tie their identities to the subcommunities they care most about, rather than measuring themselves against the whole world (though for an objection to this line of reasoning see Katja Grace’s post here).
    • More equality of status. To the extent that we still have hierarchies and negative-sum games, it should at least be the case that nobody is consistently at the bottom of all of them, and everyone can look forward to their time of recognition and respect (as in the system I outline in this blog post).
Humanity overall

When we zoom out to consider the trajectory of humanity as a whole, there are some desirable properties which we might want it to have. Although there are reasons to distrust such large-scale judgements (in particular the human tendency towards scope insensitivity) these are often strong intuitions which do matter to us.
  • Sheer size. The more people living worthwhile lives, the better - and with the astronomical resources available to us, we have the opportunity to allow our descendants to number in the uncountable trillions.
  • Solving coordination. In general, we’re bad at working together to resolve problems. This could be solved by mechanisms to make politics and governance transparent, accountable and responsive at a variety of levels. In other words, imagine humanity at one with itself and able to set its overall direction, rather than trapped in our current semi-anarchic default condition.
    • The end of war. Others have spoken of the senseless horror of war much better than I can. I will merely add that some human war will be our last war; let us hope that it gains that distinction for the right reason.
    • Avoiding races to the bottom. Under most people’s ethical intuitions, we should dislike the Malthusian scenario in which, even as our wealth grows vastly, our populations will grow even faster, so that most people end up with subsistence-level resources. To avoid this, we will need to ability to coordinate well at large scales.
  • The pursuit of knowledge. As a species we will learn and discover more and more over time. Eventually we will understand both the most fundamental building blocks of nature and also the ways in which complex systems like our minds and societies function.
  • Moral progress. In particular, we will come to a better understanding of ethics, both in theory and in ways that we can actually act upon - and then hopefully do so, to create just societies. While it’s difficult to predict exactly where moral progress will take us, one component which seems very important is building a utopia for all, with widespread access to the opportunity to pursue a good life. In particular, this should probably involve everyone having certain basic rights - such as the ability to participate in the major institutions of civil society, as Anderson describes.
  • Exploring the diversity of life. Many people value our current world’s variety of cultures and lifestyles - but over many millennia our species will be able to explore the vast frontiers of what worthwhile lives and societies could look like. The tree of humanity will branch out in ways that are unimaginable to us now.
    • Speciation. Even supposing that we are currently alone in the universe, we need not be the last intelligent species. Given sufficient time, it might become desirable to create descendant species, or split humanity into different branches which experience different facets of life. Or we might at least enjoy the companionship of animals, whether they be species that currently exist or those which we create ourselves.
  • Making our mark. The universe is vast, but we have plenty of time. Humanity could expand to colonise this galaxy, and others, in a continual wave of exploration and pursuit of the unknown. We might create projects of unimaginable scale, reengineering the cosmos as we choose, and diverting the current astronomical waste towards the well-being of ourselves and our descendants.
    • Creativity and culture. The ability to create new life, design entire worlds, and perform other large-scale feats, will allow unmatched expressions of artistry and beauty.
    • Humanity’s final flourishing. In the very very long term, under our current understanding of physics, humanity will run out of energy to sustain itself, and our civilisation will draw to an end. If we cannot avoid that, at least we can design our species’ entire trajectory, including that final outcome, with the wisdom of uncountable millennia.
Contentious changes

For all of the changes listed above, there are straightforward reasons why they would be better than the status quo or than a move in the opposite direction. However, there are some dimensions along which we might eventually want to move - but in which direction, I don’t know.
  • Privacy, or lack thereof. In many ways people have become more open over the past few centuries. But we now also place more importance on individual rights such as the right to privacy. I could see a future utopia in which there were very few secrets, and radical transparency was the norm - but also the opposite, in which everyone had full control over which aspects of themselves others could access, even up to their appearance and name (as in this excellent novel).
  • Connection with nature. Many people value this very highly. By contrast, transhumanists generally want to improve on nature, not return to it. In the long term, we might synthesise these two by creating new environments and ecosystems that are even more peaceful and beautiful and grand those which exist today - but I don’t know how well those would match people’s current conceptions of natural life.
  • New social roles. Each of us plays many social roles, and is bound by the corresponding constraints and expectations. I think such roles will be an important part of social interactions even in a utopia: we don’t want total homogeneity. However, our current roles - gender roles, family roles, job roles and so on - are certainly not optimal for everyone. I can imagine them being replaced by social roles which are just as strong, but which need to be opted into, or provide more flexibility in other ways. Yet I’m hesitant to count this as an unalloyed good, because the new roles might seem bizarre and alien to us, even if our descendants think of them as natural and normal (as illustrated in this fascinating story by Scott Alexander). Consider, for instance, how strange the hierarchical roles of historical societies seem to us today - and then imagine a future in which our version of romance is just as antiquated, in favour of totally new narratives about what makes relationships meaningful.
  • Unity versus freedom. Unity of lifestyle and purpose was a key component of many historical utopias. Some more recent utopias, like Banks’ Culture, propound the exact opposite: total freedom for individuals to live radically diverse lives. Which is better? The temperament of the time urges me towards the latter, which I think is also more intuitive at astronomical scales, but this would also make it harder to implement the other features of the utopia I’ve described, if there’s extensive disagreement about what goals to pursue, and how. Meanwhile one downside of unity is the necessity of enforcing social norms, for example by ostracising or condemning those who disobey.
  • The loss or enhancement of individuality. The current composition of our minds - having very high bandwidth between different parts of our brain, and very low bandwidth between our brains and others’ - is a relic of our evolutionary history. Above, I described the benefits of reducing the communication boundaries between different people. But I’m not sure how far to take this: would we want a future in which individuality is obsolete, with everyone merging into larger consciousnesses? Or would it be better if, despite increasing communication bandwidth, we place even greater value on long-term individuality, since our lives will be much less transient?
    • Cloning and copying. Other technologies which might affect our attitudes towards individuality are those which would allow us to create arbitrarily many people arbitrarily similar to ourselves.
  • Self-modification. The ability to change arbitrary parts of your mind is a very powerful one. At its best, we can make ourselves the people we always wanted to be, transcending human limitations. At its worst, there might be pressure to carve out the parts of ourselves that make us human, as Scott Alexander discusses here.
    • Designer people. Eventually we will be able to specify arbitrary characteristics of our children, shaping them to an unprecedented extent. However, I don’t know if that’s a power we should fully embrace, either as individuals or as societies.
  • Wireheading. I’m uncertain about the extent to which blissing out on pleasure (at the expense of pursuing more complex goals) is something we should aim for.
  • Value drift. More generally, humanity’s values will by default change significantly over time. Whether to prevent that or to allow it to happen is a tricky question. The former implies a certain type of stagnation - we are certainly glad that the Ancient Greeks did not lock in their values. The latter option could lead us to a world which looks very weird and immoral by our modern sensibilities.


1. See, for instance, the conspicuous absence of relationships and communities in works such as Nick Bostrom’s Transhumanist FAQ. His summary of the transhumanist perspective: “Many transhumanists wish to follow life paths which would, sooner or later, require growing into posthuman persons: they yearn to reach intellectual heights as far above any current human genius as humans are above other primates; to be resistant to disease and impervious to aging; to have unlimited youth and vigor; to exercise control over their own desires, moods, and mental states; to be able to avoid feeling tired, hateful, or irritated about petty things; to have an increased capacity for pleasure, love, artistic appreciation, and serenity; to experience novel states of consciousness that current human brains cannot access.” See also Yudkowsky: “It doesn't get any better than fun.” Meanwhile the foremost modern science fiction utopia, Banks’ Culture, is also very individualistic.

2. Some interesting quotes from Walden Two:
  • “Men build society and society builds men.”
  • “The behavior of the individual has been shaped according to revelations of ‘good conduct,’ never as the result of experimental study. But why not experiment? The questions are simple enough. What’s the best behavior for the individual so far as the group is concerned? And how can the individual be induced to behave in that way? Why not explore these questions in a scientific spirit?”
  • “We undertook to build a tolerance for annoying experiences. The sunshine of midday is extremely painful if you come from a dark room, but take it in easy stages and you can avoid pain altogether. The analogy can be misleading, but in much the same way it’s possible to build a tolerance to painful or distasteful stimuli, or to frustration, or to situations which arouse fear, anger or rage. Society and nature throw these annoyances at the individual with no regard for the development of tolerances. Some achieve tolerances, most fail. Where would the science of immunization be if it followed a schedule of accidental dosages?”
And from Island:
  • “That would distract your attention, and attention is the whole point. Attention to the experience of something given, something you haven't invented in your imagination.”
  • "We all belong to an MAC—a Mutual Adoption Club. Every MAC consists of anything from fifteen to twenty-five assorted couples. Newly elected brides and bridegrooms, old-timers with growing children, grandparents and great-grandparents—everybody in the club adopts everyone else. … An entirely different kind of family. Not exclusive, like your families, and not predestined, not compulsory. An inclusive, unpredestined and voluntary family. Twenty pairs of fathers and mothers, eight or nine ex-fathers and ex-mothers, and forty or fifty assorted children of all ages."
  • “[Large, powerful men] are just as muscular here, just as tramplingly extraverted, as they are with you. So why don’t they turn into Stalins or Dipas, or at the least into domestic tyrants? First of all, our social arrangements offer them very few opportunities for bullying their families, and our political arrangements make it practically impossible for them to domineer on any larger scale. Second, we train the Muscle Men to be aware and sensitive, we teach them to enjoy the commonplaces of everyday existence. This means that they always have an alternative—innumerable alternatives—to the pleasure of being the boss. And finally we work directly on the love of power and domination that goes with this kind of physique in almost all its variations. We canalize this love of power and we deflect it—turn it away from people and on to things. We give them all kinds of difficult tasks to perform—strenuous and violent tasks that exercise their muscles and satisfy their craving for domination—but satisfy it at nobody’s expense and in ways that are either harmless or positively useful.”
3. For a short introduction to this debate, see section 3 in the Stanford Encyclopedia of Philosophy’s entry on Consequentialism.

4. For stylistic purposes I wrote much of this essay in the future tense, without always hedging with “we might” and “it’s possible that”. Please don’t interpret any of my descriptions as confident predictions - rather, treat them as expressions of possibility and hope.

5. As Tim Ferris puts it, “excitement is the more practical synonym for happiness”.

6. For an analysis of the similarities between these three traditions, I recommend Shannon Vallor's Technology and the Virtues.

7. For a (somewhat fawning) description of such a society, see Swift’s Houynhnhnms, which are “endowed by nature with a general disposition to all virtues, and have no conceptions or ideas of what is evil in a rational creature”; and which the narrator wants “for civilizing Europe, by teaching us the first principles of honour, justice, truth, temperance, public spirit, fortitude, chastity, friendship, benevolence, and fidelity.”

Wednesday, 25 December 2019

My sense of the ending

"History is that certainty produced at the point where the imperfections of memory meet the inadequacies of documentation."

Julian Barnes' novel The Sense of an Ending (which won the 2011 Man Booker Prize) is short but fascinating; I recommend it. It also leaves a lot of questions unanswered or ambiguous; here's my take on what happened.

Warning: many, many spoilers ahead.

  • Adrian committed suicide because he'd gotten Sarah pregnant and she had decided to keep the child. The child, also called Adrian, had developmental issues because of Sarah's age, and eventually ended up in adult care.
  • At the end of the novel, Tony feels responsible for Adrian's suicide because he remembers that he'd deliberately tried to set Adrian up with Sarah in order to sabotage Adrian's relationship with Veronica. He had done so by suggesting, in his letter to Adrian, that Adrian meet secretly with Sarah.
  • Tony knew that Sarah would make a pass at Adrian because she'd made a pass at Tony during his weekend visit, while the others went out for a walk (or perhaps the night before). However, it's not clear whether he actually had sex with her.
    • Reasons to think that Tony slept with Sarah:
      • Her smile, "almost as if we had a secret"; and her "secret, horizontal" farewell gesture.
      • There's another example of Tony eliding over a love affair: "I did a slightly odd thing when I first met Margaret. I wrote Veronica out of my life story... I know most men exaggerate the amount of girls and sex they've had; I did the opposite."
      • It would explain why Sarah wrote Tony a letter after he broke up with Veronica, wishing him well.
      • It would explain why Sarah left Tony the bequest.
    • Reasons to think that Tony didn't actually sleep with Sarah:
      • After Tony and Sarah were alone together, Veronica "became more openly affectionate", as if he'd passed a test.
      • Tony was always very timid; it would be appropriate if Adrian had been bolder than him in this regard too.
      • "She eased another egg onto my plate, despite my not asking for it or wanting it." While the recurring image of Sarah cooking eggs is suggestive, the second half of the sentence points towards Tony being more passive.
      • Sleeping with your girlfriend's mother would have been a huge thing to totally forget; whereas Tony forgetting about her making a move on him seems much more realistic.
  • Although the details of what happened can't be resolved, I think that Tony having deliberate intent to set up Adrian and Sarah is necessary for the ending of the novel to be satisfactory. Without that, he played only an accidental role in the events leading up to Adrian's suicide, and the theme of unreliable memory is left hanging (if the harshness of his letter to Adrian was the only important thing he forgot about).
What else is obscured by Tony's unreliable narration? I think the key thing is Veronica's personality, which is sidelined by Tony's unrelenting self-absorption and his inability to empathise with her.
  • Veronica was a "damaged" girl, with a manipulative mother and (perhaps) an abusive father. It took a while for her to trust Tony enough to dance with him; meanwhile he was still thinking primarily about sex and music and "what does she want me to say?" What she actually wanted was for Tony to be honest with and committed to her, in a way that he never was. Even after she had sex with Tony to try and salvage their relationship, he didn't understand her motivations or vulnerability. He was preoccupied with his own insecurities, and his "fear of an overwhelming closeness I couldn't handle".
    • For example, during his visit to Veronica's family, he's paranoid about what they think of him. Presumably Veronica is also nervous about him meeting her family (hence her being a little withdrawn at first), but he merely interprets her behaviour as more fuel for his insecurity.
  • This characterisation helps explain the older Veronica's behaviour: she hopes that Tony will have grown enough that they can be frank with each other and build a better relationship. That's why she keeps replying to his emails, and keeps agreeing to meet. He never manages to live up to that, partly because he's repressed many of the relevant memories. But it's also because he's simply not mature enough - see how, when they meet for lunch, Tony spends a whole hour talking about himself self-aggrandisingly without asking anything about her life, prompting her to leave.
    • In fact, he's also pretty immature in his relationship with Margaret, to the point where she gets fed up and tells him that he's "on his own".
  • On the other hand, I don't think this is a fully satisfactory explanation for why the older Veronica was so cryptic, parcelling out information in dribs and drabs (except for that being a convenient plot device). Nor does it explain why she thinks that Tony should be able to "get it" from the limited information available to him, and why she's so frustrated when he doesn't. And there are other unresolved questions about her, such as why Adrian Jr called Veronica "Mary" (her middle name).

Friday, 18 October 2019

Technical AGI safety research outside AI

I think there are many questions whose answers would be useful for technical AGI safety research, but which will probably require expertise outside AI to answer. In this post I list 30 of them, divided into four categories. Feel free to get in touch if you’d like to discuss these questions and why I think they’re important in more detail. I personally think that making progress on the ones in the first category is particularly vital, and plausibly tractable for researchers from a wide range of academic backgrounds.

Studying and understanding safety problems
  1. How strong are the economic or technological pressures towards building very general AI systems, as opposed to narrow ones? How plausible is the CAIS model of advanced AI capabilities arising from the combination of many narrow services?
  2. What are the most compelling arguments for and against discontinuous versus continuous takeoffs? In particular, how should we think about the analogy from human evolution, and the scalability of intelligence with compute?
  3. What are the tasks via which narrow AI is most likely to have a destabilising impact on society? What might cyber crime look like when many important jobs have been automated?
  4. How plausible are safety concerns about economic dominance by influence-seeking agents, as well as structural loss of control scenarios? Can these be reformulated in terms of standard economic ideas, such as principal-agent problems and the effects of automation?
  5. How can we make the concepts of agency and goal-directed behaviour more specific and useful in the context of AI (e.g. building on Dennett’s work on the intentional stance)? How do they relate to intelligence and the ability to generalise across widely different domains?
  6. What are the strongest arguments that have been made about why advanced AI might pose an existential threat, stated as clearly as possible? How do the different claims relate to each other, and which inferences or assumptions are weakest?
Solving safety problems
  1. What techniques used in studying animal brains and behaviour will be most helpful for analysing AI systems and their behaviour, particularly with the goal of rendering them interpretable?
  2. What is the most important information about deployed AI that decision-makers will need to track, and how can we create interfaces which communicate this effectively, making it visible and salient?
  3. What are the most effective ways to gather huge numbers of human judgments about potential AI behaviour, and how can we ensure that such data is high-quality?
  4. How can we empirically test the debate and factored cognition hypotheses? How plausible are the assumptions about the decomposability of cognitive work via language which underlie debate and iterated distillation and amplification?
  5. How can we distinguish between AIs helping us better understand what we want and AIs changing what we want (both as individuals and as a civilisation)? How easy is the latter to do; and how easy is it for us to identify?
  6. Various questions in decision theory, logical uncertainty and game theory relevant to agent foundations.
  7. How can we create secure containment and supervision protocols to use on AI, which are also robust to external interference?
  8. What are the best communication channels for conveying goals to AI agents? In particular, which ones are most likely to incentivise optimisation of the goal specified through the channel, rather than modification of the communication channel itself?
  9. How closely linked is the human motivational system to our intellectual capabilities - to what extent does the orthogonality thesis apply to human-like brains? What can we learn from the range of variation in human motivational systems (e.g. induced by brain disorders)?
  10. What were the features of the human ancestral environment and evolutionary “training process” that contributed the most to our empathy and altruism? What are the analogues of these in our current AI training setups, and how can we increase them?
  11. What are the features of our current cultural environments that contribute the most to altruistic and cooperative behaviour, and how can we replicate these while training AI?
Forecasting AI
  1. What are the most likely pathways to AGI and the milestones and timelines involved?
  2. How do our best systems so far compare to animals and humans, both in terms of performance and in terms of brain size? What do we know from animals about how cognitive abilities scale with brain size, learning time, environmental complexity, etc?
  3. What are the economics and logistics of building microchips and datacenters? How will the availability of compute change under different demand scenarios?
  4. In what ways is AI usefully analogous or disanalogous to the industrial revolution; electricity; and nuclear weapons?
  5. How will the progression of narrow AI shape public and government opinions and narratives towards it, and how will that influence the directions of AI research?
  6. Which tasks will there be most economic pressure to automate, and how much money might realistically be involved? What are the biggest social or legal barriers to automation?
  7. What are the most salient features of the history of AI, and how should they affect our understanding of the field today?
  1. How can we best grow the field of AI safety? See OpenPhil’s notes on the topic.
  2. How can spread norms in favour of careful, robust testing and other safety measures in machine learning? What can we learn from other engineering disciplines with strict standards, such as aerospace engineering?
  3. How can we create infrastructure to improve our ability to accurately predict future development of AI? What are the bottlenecks facing tools like and Metaculus, and preventing effective prediction markets from existing?
  4. How can we best increase communication and coordination within the AI safety community? What are the major constraints that safety faces on sharing information (in particular ones which other fields don’t face), and how can we overcome them?
  5. What norms and institutions should the field of AI safety import from other disciplines? Are there predictable problems that we will face as a research community, or systemic biases which are making us overlook things?
  6. What are the biggest disagreements between safety researchers? What’s the distribution of opinions, and what are the key cruxes?
Particular thanks to Beth Barnes and a discussion group at the CHAI retreat for helping me compile this list.

Thursday, 5 September 2019

Seven habits towards highly effective minds

Lately I’ve been thinking about how my thinking works, and how it can be improved. The simplest way to do so is probably to nudge myself towards paying more attention to various useful habits of mind. Here are the ones I've found most valuable (roughly in order):
  1. Tying together the act of saying a statement, and the act of evaluating whether I actually believe it. After making a novel claim, saying out loud to myself: “is this actually true?” and "how could I test this?"
  2. Being comfortable with pausing to reflect and thinking out loud. Trying to notice when my responses are too quick and reflexive, as a sign that I'm not thinking hard enough about the point I'm addressing.
  3. Asking for specific examples, and using more of my own. Tabooing vague abstractions and moving away from discussing claims that are too general.
  4. Being charitable and collaborative, both towards new ideas and towards conversational partners. Trying to rephrase other people’s arguments and pass Ideological Turing Tests on them. Helping my conversational partners build up their ideas.
  5. Noticing the affect heuristic, and which claims stir up emotions. Noticing when I'm talking defensively or heatedly (especially about politics), and when it’d be uncomfortable to believe something.
  6. Thinking in terms of probabilities; cashing out beliefs in terms of predictions; then betting on them. I haven’t done enough bets to calibrate myself well, but I find that even just the feeling of having money on the line is often enough to make me rethink. Being asked whether something is a crux gives me a similar feeling.
  7. Thinking about how the conversations and debates I participate in actually create value, and when they should be redirected or halted.
Then there are social influences. I think one of the greatest virtues of the rationalist community is in creating an environment which encourages the use of the tools above. Another example: my girlfriend fairly regularly points out times when I’ve contradicted myself. I think this has helped me notice and limit the extent to which I behave like an opinion confabulation machine.

I’d classify most if not all of the tools listed above as tools for evaluating ideas, though, rather than tools for generating ideas. What helps with the latter? I’ve personally found that one very useful strategy is to make and then justify bold claims based on vague intuitions. In the process of defending my position, I’m forced to actually flesh it out and make it coherent (although I do need to be careful not to become overly attached to the untrue parts). And what's helped the most is that after having interesting conversations, I now write posts inspired by them much more frequently. I often feel like Feynman in this story: "When historian Charles Weiner found pages of Nobel Prize-winning physicist Richard Feynman's notes, he saw it as a 'record' of Feynman's work. Feynman himself, however, insisted that the notes were not a record but the work itself." Arguing and writing are not just ways to transmit my thoughts, but also the key mechanisms by which I generate new thoughts.

(Edited to add: a friend pointed out that the last line is a good indicator that I'm being insufficiently empirical. I think I agree; it should also include the mechanisms of actually observing and experimenting with the world.)

Saturday, 20 July 2019

Book review: The Technology Trap

I recently finished reading The Technology Trap, by Carl Frey. The book attempts to do two things: chronicle the role of technology in economic progress throughout history, and argue that automation in our own era parallels the first seven decades of the industrial revolution, during which the wealth from mechanisation failed to reach most citizens, leading to a populist backlash.

I particularly enjoyed the first component, because until now I’ve read much less about the industrial revolution than I should have. That also means that I’m not qualified to evaluate the book’s accuracy. However, it had interesting discussions of:
  • The technological prowess of the Romans, and why they were held back from industrialising both because of their slave-based economy, and also because of an implicit dismissal of the private economy.
  • The development of some surprisingly important technologies during the Middle Ages, such as wind- and water-mills, better ways of accessing horsepower (via improved horseshoes, harnesses and ploughs), and town clocks.
  • The fact that most of the key innovations of the early industrial revolution (steam engines excepted) would have been technologically possible a century or two earlier, but were blocked by the political power of guilds.
  • The importance of the Glorious Revolution in shifting England’s political climate to favour industrialisation; and more generally, the role of competition between nation-states in spurring government permissiveness towards innovation.
  • The mechanisation of the silk industry as a smaller-scale precursor to the mechanisation of cotton processing that would drive the early years of the industrial revolution.
  • The prevalence of child labour in the first factories run by Arkwright and others; and more generally how miserable the first few decades of the industrial revolution were for the poor, who were crammed into unsanitary cities on reduced wages (with severe health consequences).
  • The fact that it took many decades after Watt’s steam engine was patented in 1769 for railways to actually become widely significant.
  • The role of arms manufacturers like Colt as precursors to Ford’s assembly lines.

In its latter role, however, the book seems a little incomplete. From around 1770 to 1840, productivity rose while worker incomes stagnated, with the increased wealth primarily going to industrialists - a period known as “Engels’ pause”. Frey argues that today, as the incomes of Western workers stagnate, we’ve reached an analogous situation. Engels’ pause gave rise to Luddite riots and the growth of the communist movement. Similarly, the modern working class will be tempted to campaign against automation - a “technology trap” which we will need to overcome to reach the level of technology which makes prosperity more widespread.

Certainly the thesis is initially plausible, but at the end of the book I was left with quite a few unanswered questions. Four particularly important ones:
  1. Frey makes the distinction between replacing technologies and augmenting technologies. The former “render jobs and skills redundant”; automatic elevators are a good example. The latter “make people more productive in existing tasks or create entirely new jobs for them”; Frey’s examples are innovation in the steel industry and the invention of the typewriter. But there’s a pretty blurry line between these two categories. An augmenting technology becomes a replacing technology if “demand for a given product or service becomes saturated”, a criterion which has less to do with the sector itself than with the broader state of the economy. But if we’re considering the wider economy, then the lower costs provided by replacing technologies enable other sectors to produce more goods, making them augmenting after all. So while the call to “augment not replace” workers has become a rallying cry, I’m not sure that the distinction has much predictive power. Can we tell in advance which technologies will be augmenting vs replacing, or do we just have to wait until a few decades later and look at the job statistics?
  2. Building on the last point: you could describe the first industrial revolution as starting off with replacing technologies (such as power looms) and moving on to augmenting technologies (such as the steam engine). And you could describe the second industrial revolution as being all about augmenting technologies (such as electricity and cars - although the latter could also be considered a replacing technology for horses). If Frey is right that the current wage stagnation has been driven by automation, then this matches the beginning of the first industrial revolution.* But are there good reasons to think that we’ll eventually transition to building augmenting technologies in the same way as they did? Reasoning from a small sample size is treacherous at the best of times, and in this case our n=2 sample showcases two different trajectories. We might be about to experience a third distinct trajectory: AI continuing to be a replacing technology to a greater and greater extent. I do think this is unlikely (as I argue here) but it’s an open possibility.
  3. Frey discusses the experience of America’s blue-collar middle class - which, he argues, has lost jobs to a combination of globalisation and automation. But (assuming this is true) how much of the responsibility should each factor bear? If it’s almost all due to globalisation, then the chapter is a little misleading. I don’t have any particular reason to think that, but Frey doesn’t do the work of convincing me otherwise. (Although, since globalisation has been made much easier by information technology, should we count it as an effect of automation? It seems roughly analogous to how technologies invented early in the industrial revolution allowed adults’ jobs to be done by children.)
  4. Frey worries that the technology trap will lead to workers suppressing technological growth. Yet there have been many changes to the factors which originally held back industrialisation. Guilds/unions are much reduced in power; international competitiveness is now a top priority; faster communication channels facilitate the spread of new ideas; and the intellectual plausibility of stifling innovation as a way to protect workers is much diminished, given how hugely we have benefited (in material terms) from the last few centuries of technological progress.
    On the other hand, everyone has the vote now, which wasn’t the case in the past. And many people are using those votes to send a strong message against current intellectual orthodoxy. And with the pace of change being much faster now than in the 1700s, perhaps the backlash it spurs will be concomitantly greater. Or maybe it will mean that the anti-technology camp has less time to coordinate resistance. It seems very unclear how these factors weigh against each other; Frey’s historical analogy can only take us so far.

Frey finishes with a set of prescriptions for how to close the gap between the winners and losers from automation, most of which are standard and sensible - e.g. cutting back on occupational licensing, encouraging relocation, investing in high-speed rail, and reforming housing markets. A more novel proposal is wage insurance, which compensates people when they are forced into lower-paying jobs. It seems like a good idea for individuals, but if implemented by the government as Frey suggests, I worry that it’ll become yet another piece of clutter in an already overcomplicated and inefficient welfare system.

I want to end this review with a theme of the book that I particularly liked: the rehabilitation of the Luddites. Frey emphasises that, despite having become a byword for ignorant destructiveness, the Luddites were actually campaigning against a major threat to their livelihoods and communities, and we should sympathise with them. The parallels with our modern era are obvious - and the more we can rise above pejorative descriptions of our political opponents, the better.

* There’s also the complication that incomes in the tech sector have been rising rapidly. Was there an analogous group of skilled workers who benefited from Engels’ pause? I suppose that the job of building the machines must have been a lucrative one, but I really don’t know.