tag:blogger.com,1999:blog-66420844844580085872024-03-14T07:37:01.223-07:00Thinking CompleteSearching for theories of everythingRichard Ngohttp://www.blogger.com/profile/04825733481608403399noreply@blogger.comBlogger101125tag:blogger.com,1999:blog-6642084484458008587.post-52282927421084854362022-07-27T11:48:00.014-07:002022-10-18T13:17:37.563-07:00Moral strategies at different capability levelsLet’s consider three ways you can be altruistic towards another agent:<br /><ol style="text-align: left;"><li>You care about their welfare: some metric of how good their life is (as defined by you). I’ll call this care-morality - it endorses things like promoting their happiness, reducing their suffering, and hedonic utilitarian behavior (if you care about many agents).</li><li>You care about their agency: their ability to achieve their goals (as defined by them). I’ll call this cooperation-morality - it endorses things like honesty, fairness, deontological behavior towards others, and some virtues (like honor).</li><li>You care about obedience to them. I’ll call this deference-morality - it endorses things like loyalty, humility, and respect for authority.</li></ol>I think a lot of unresolved tensions in ethics comes from seeing these types of morality as in opposition to each other, when they’re actually complementary:<div><ol style="text-align: left;"><li>Care-morality mainly makes sense as an attitude towards agents who are much less capable than you, and/or can't make decisions for themselves - for example animals, future people, and infants.</li><ol><li>In these cases, you don’t have to think much about what the other agents are doing, or what they think of you; you can just aim to produce good outcomes in the world. Indeed, trying to be cooperative or deferential towards these agents is hard, because their thinking may be much less sophisticated than yours, and you might even get to choose what their goals are.</li><li>Applying only care-morality in multi-agent contexts can easily lead to conflict with other agents around you, even when you care about their welfare, because:</li><ol><li>You each value (different) other things in addition to their welfare.</li><li>They may have a different conception of welfare than you do.</li><li>They can’t fully trust your motivations.</li></ol><li>Care morality doesn’t focus much on the act-omission distinction. Arbitrarily scalable care-morality looks like maximizing resources until the returns to further investment are low, then converting them into happy lives.</li></ol><li>Cooperation-morality mainly makes sense as an attitude towards agents whose capabilities are comparable to yours - for example others around us who are trying to influence the world.</li><ol><li>Cooperation-morality can be seen as the “rational” thing to do even from a selfish perspective (e.g. <a href="https://forum.effectivealtruism.org/posts/CfcvPBY9hdsenMHCr/integrity-for-consequentialists-1">as discussed here</a>), but in practice it’s difficult to robustly reason through the consequences of being cooperative without relying on ingrained cooperative instincts, especially when using causal decision theories. <a href="https://intelligence.org/2017/10/22/fdt/">Functional decision theories</a> make it much easier to rederive many aspects of intuitive cooperation-morality as optimal strategies (as discussed further below).</li><li>Cooperation-morality tends to uphold the act-omission distinction, and a sharp distinction between those within versus outside a circle of cooperation. It doesn’t help very much with population ethics - naively maximizing the agency of future agents would involve ensuring that they only have very easily-satisfied preferences, which seems very undesirable.</li><li>Arbitrarily scalable cooperation-morality looks like forming a central decision-making institution which then decides how to balance the preferences of all the agents that participate in it.</li><li>A version of cooperation-morality can also be useful internally: enhancing your own agency by cultivating virtues which facilitate cooperation between different parts of yourself, or versions of yourself across time.</li></ol><li>Deference-morality mainly makes sense as an attitude towards trustworthy agents who are much more capable than you - for example effective leaders, organizations, communities, and sometimes society as a whole.</li><ol><li>Deference-morality is important for getting groups to coordinate effectively - soldiers in armies are a central example, but it also applies to other organizations and movements to a lesser extent. Individuals trying to figure out strategies themselves undermines predictability and group coordination, especially if the group strategy is more sophisticated than the ones the individuals generate.</li><li>In practice, it seems very easy to overdo deference-morality - compared to our ancestral environment, it seems much less useful today. Also, whether or not deference-morality makes sense depends on how much you trust the agents you’re deferring to - but it’s often difficult to gain trust in agents more capable than you, because they’re likely better at deception than you. Cult leaders exploit this.</li><li>Arbitrarily-scalable deference-morality looks like an <a href="https://ai-alignment.com/clarifying-ai-alignment-cec47cd69dd6">intent-aligned</a> AGI. One lens on why intent alignment is difficult is that deference-morality is inherently unnatural for agents who are much more capable than the others around them.</li></ol></ol>Cooperation-morality and deference-morality have the weakness that they can be exploited by the agents we hold those attitudes towards; and so we also have adaptations for deterring or punishing this (which I’ll call conflict-morality). I’ll mostly treat conflict-morality as an implicit part of cooperation-morality and deference-morality; but it’s worth noting that a crucial feature of morality is the coordination of coercion towards those who act immorally.</div><div><h4 style="text-align: left;">Morality as intrinsic preferences versus morality as instrumental preferences</h4>I’ve mentioned that many moral principles are rational strategies for multi-agent environments even for selfish agents. So when we’re modeling people as rational agents optimizing for some utility function, it’s not clear whether we should view those moral principles as part of their utility functions, versus as part of their strategies. Some arguments for the former:<br /><ul style="text-align: left;"><li>We tend to care about principles like honesty for their own sake (because that was the most robust way for evolution to actually implement cooperative strategies).</li><li>Our cooperation-morality intuitions are only evolved proxies for ancestrally-optimal strategies, and so we’ll probably end up finding that the actual optimal strategies in other environments violate our moral intuitions in some ways. For example, we could see love as a cooperation-morality strategy for building stronger relationships, but most people still care about having love in the world even if it stops being useful.</li></ul>Some arguments for the latter:<br /><ul style="text-align: left;"><li>It seems like caring intrinsically about cooperation, and then also being instrumentally motivated to pursue cooperation, is a sort of double-counting.</li><li>Insofar as cooperation-morality principles are non-consequentialist, it’s hard to formulate them as positive components of a utility function over outcomes. E.g. it doesn’t seem particularly desirable to maximize the amount of honesty in the universe.</li></ul>The rough compromise which I use here is to:<br /><ul style="text-align: left;"><li>Care intrinsically about the welfare of all agents which currently exist or might in the future, with a bias towards myself and the people close to me.</li><li>Care intrinsically about the agency of existing agents to the extent that they're capable enough to be viewed as having agency (e.g. excluding trees), with a bias towards myself and the people close to me.</li><ul><li>In other words, I care about agency in a <a href="https://en.wikipedia.org/wiki/Person-affecting_view">person-affecting way</a>; and more specifically in a loss-averse way which prioritizes preserving existing agency over enhancing agency.</li></ul><li>Define welfare partly in terms of hedonic experiences (particularly human-like ones), and partly in terms of having high agency directed towards human-like goals.</li><ul><li>You can think of this as a mixture of hedonism, desire, and objective-list <a href="https://www.utilitarianism.net/theories-of-wellbeing">theories of welfare</a>.</li></ul><li>Apply cooperation-morality and deference-morality instrumentally in order to achieve the things I intrinsically care about.</li><ul><li>Instrumental applications of cooperation-morality and deference-morality lead me to implement strong principles. These are partly motivated by being in an iterated game within society, but also partly motivated by functional decision theories.</li></ul></ul><h4 style="text-align: left;">Rederiving morality from decision theory</h4>I’ll finish by elaborating on how different decision theories endorse different instrumental strategies. Causal decision theories only endorse the same actions as our cooperation-morality intuitions in specific circumstances (e.g. iterated games with indefinite stopping points). By contrast, <a href="https://intelligence.org/2017/10/22/fdt/">functional decision theories</a> do so in a much wider range of circumstances (e.g. one-shot prisoner’s dilemmas) by accounting for logical connections between your choices and other agents’ choices. Functional decision theories follow through on commitments you previously made; and sometimes follow through on commitments that you would have made. However, the question of which hypothetical commitments they should follow through with depends on how <a href="https://www.lesswrong.com/tag/updateless-decision-theory">updateless</a> they are.<br /><br />Updatelessness can be very powerful - it’s essentially equivalent to making commitments behind a veil of ignorance, which provides an instrumental rationale for implementing cooperation-morality. But it’s very unclear how to reason about how justified different levels of updatelessness are. So although it’s tempting to think of updatelessness as a way of deriving care-morality as an instrumental goal, for now I think it’s mainly just an interesting pointer in that direction. (In particular, I feel confused about the relationship between single-agent updatelessness and multi-agent updatelessness like the original veil of ignorance thought experiment; I also don’t know what it looks like to make commitments “before” having values.)</div><div><br /></div><div>Lastly, I think deference-morality is the most straightforward to derive as an instrumentally-useful strategy, conditional on fully trusting the agent you’re deferring to - epistemic deference intuitions are pretty common-sense. If you don’t fully trust that agent, though, then it seems very tricky to reason about how much you should defer to them, because they may be manipulating you heavily. In such cases the approach that seems most robust is to diversify worldviews using a <a href="https://www.lesswrong.com/posts/6QPFKHRsuY63cuJwh/making-decisions-using-multiple-worldviews">meta-rationality</a> strategy which includes some strong principles.</div>Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-81116331927336968502022-07-22T19:36:00.008-07:002022-10-18T13:13:25.223-07:00Which values are stable under ontology shifts?Here's a rough argument which I've been thinking about lately:<br /><br />We have coherence theorems which say that, if you’re not acting like you’re maximizing expected utility over outcomes, you’d make payments which predictably lose you money. But in general I don't see any principled distinction between “predictably losing money” (which we view as incoherent) and “predictably spending money” (to fulfill your values): it depends on the space of outcomes over which you define utilities, which seems pretty arbitrary. You could interpret an agent being money-pumped as a type of incoherence, or as an indication that it enjoys betting and is willing to pay to do so; similarly you could interpret an agent passing up a “sure thing” bet as incoherence, or just a preference for not betting which it’s willing to forgo money to satisfy. Many humans have one of these preferences!<br /><br />Now, these preferences are somewhat odd ones, because you can think of every action under uncertainty as a type of bet. In other words, “betting” isn't a very fundamental category in an ontology which has a sophisticated understanding of reasoning under uncertainty. Then the obvious follow-up question is: which human values will naturally fit into <a href="https://www.lesswrong.com/posts/LeXhzj7msWLfgDefo/science-informed-normativity">much more sophisticated ontologies</a>? I worry that not many of them will:<br /><ul style="text-align: left;"><li>In a world where minds can be easily copied, our current concepts of personal identity and personal survival will seem very strange. You could think of those values as “predictably losing money” by forgoing the benefits of temporarily running multiple copies. (This argument was inspired by <a href="https://www.lesswrong.com/posts/zd2DrbHApWypJD2Rz/udt2-and-against-ud-assa#against_UD_ASSA__part_1__9_26_2007_">this old thought experiment</a> from Wei Dai.)</li><li>In a world where minds can be designed with arbitrary preferences, our values related to “preference satisfaction” will seem very strange, because it’d be easy to create people with meaningless preferences that are by default satisfied to an arbitrary extent.</li><li>In a world where we understand minds very well, our current concepts of happiness and wellbeing may seem very strange. In particular, if happiness is understood in a more sophisticated ontology as caused by <a href="https://jan.leike.name/publications/A%20Definition%20of%20Happiness%20for%20Reinforcement%20Learning%20Agents%20-%20Daswani,%20Leike%202015.pdf">positive reward prediction error</a>, then happiness is intrinsically in tension with having accurate beliefs. And if we understand reward prediction error in terms of updates to our policy, then deliberately invoking happiness would be in tension with acting effectively in the world.</li><ul><li>If there's simply a tradeoff between them, we might still want to sacrifice accurate beliefs and effective action for happiness. But what I'm gesturing towards is the idea that happiness might not actually be a concept which makes much sense given a complete understanding of minds - as implied by the buddhist view of happiness as an illusion, for example.</li></ul><li>In a world where people can predictably influence the values of their far future descendants, and there’s predictable large-scale growth, any non-zero discounting will seem very strange, because it predictably forgoes orders of magnitude more resources in the future.</li><ul><li>This might result in <a href="http://reflectivedisequilibrium.blogspot.com/2012/09/spreading-happiness-to-stars-seems.html">the strategy described by Carl Shulman</a> of utilitarian agents mimicking selfish agents by spreading out across the universe as fast as they can to get as many resources as they can, and only using those resources to produce welfare once the returns to further expansion are very low. It does seem possible that we design AIs which spend millions or billions of years optimizing purely for resource acquisition, and then eventually use all those resources for doing something entirely different. But it seems like those AIs would need to have minds that are constructed in a very specific and complicated way to retain terminal values which are so unrelated to most of their actions.</li></ul></ul>A more general version of these arguments: human values are generalizations of learned heuristics for satisfying innate drives, which in turn are evolved proxies for maximizing genetic fitness. In theory, you can say “this originated as a heuristic/proxy, but I terminally value it”. But in practice, heuristics tend to be limited, messy concepts which don't hold up well under ontology improvement. So they're often hard to continue caring about once you deeply understand them - kinda like how it’s hard to endorse “not betting” as a value once you realize that everything is a kind of bet, or endorse faith in god as a value if you no longer believe that god exists. And they're especially hard to continue caring about at scale.<br /><br />Given all of this, how might future values play out? Here are four salient possibilities:<br /><ol style="text-align: left;"><li>Some core notion of happiness/conscious wellbeing/living a flourishing life is sufficiently “fundamental” that it persists even once we have a very sophisticated understanding of how minds work.</li><li>No such intuitive notions are strongly fundamental, but we decide to ignore that fact, and optimize for values that seem incoherent to more intelligent minds. We could think of this as a way of trading away the value of consistency.</li><li>We end up mainly valuing something like “creating as many similar minds as possible” for its own sake, as the best extrapolation of what our other values are proxies for.</li><li>We end up mainly valuing highly complex concepts which we can’t simplify very easily - like “the survival and flourishing of humanity”, as separate from the survival and flourishing of any individual human. In this world, asking whether an outcome is good for individuals might feel like asking whether human actions are good or bad for individual cells - even if we can sometimes come up with a semi-coherent answer, that’s not something we care about very much.</li></ol>Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-63021668497722844622022-07-21T16:46:00.005-07:002022-07-22T19:40:58.112-07:00Making decisions using multiple worldviews<i>Tl;dr: the problem of how to make decisions using multiple (potentially incompatible) worldviews (which I'll call the problem of meta-rationality) comes up in a range of contexts, such as epistemic deference. Applying a policy-oriented approach to meta-rationality, and evaluating worldviews by the quality of their advice, dissolves several undesirable consequences of the standard "epistemic" approach to deference.</i><br /><br /><div><span style="font-size: medium;">Meta-rationality as the limiting case of separate worldviews</span><br /><br />When thinking about the world, we’d ideally like to be able to integrate all our beliefs into a single coherent worldview, with clearly-demarcated uncertainties, and use that to make decisions. Unfortunately, in complex domains, this can be very difficult. Updating our beliefs about the world often looks less like filling in blank parts of our map, and more like finding a new worldview which reframes many of the things we previously believed. Uncertainty often looks less like a probability distribution over a given variable, and more like a clash between different worldviews which interpret the same observations in different ways.<br /><br />By “worldviews” I include things like ideologies, scientific paradigms, moral theories, perspectives of individual people, and sets of heuristics. The key criterion is that each worldview has “opinions” about the world which can be determined without reference to any other worldview. Although of course different worldviews can have overlapping beliefs, in general their opinions can be incompatible with those of other worldviews - for example:</div><div><ul style="text-align: left;"><li>Someone might have <a href="https://en.wikipedia.org/wiki/Knightian_uncertainty">severe uncertainty</a> about <a href="https://www.cold-takes.com/most-important-century/">far-reaching empirical claims</a>, or about <a href="https://www.williammacaskill.com/info-moral-uncertainty">which moral theory</a> to favor.</li><li>A scientist might be investigating a phenomenon during a <a href="https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Revolutions">crisis period</a> where there are multiple contradictory frameworks which purport to explain the phenomenon.</li><li>Someone might buy into an ideology which says that nothing else matters except adherence to that ideology, but then feel a “common-sense” pull towards <a href="https://www.openphilanthropy.org/research/worldview-diversification/">other perspectives</a>.</li><li>Someone might have a strong “inside view” on the world, but also want to partly <a href="https://forum.effectivealtruism.org/topics/epistemic-deference">defer</a> to the worldviews of trusted friends.</li><li>Someone might have a set of principles which guides their interactions with the world.</li><li>Someone might have different parts of themselves which care about different things.</li></ul><div><br /></div>I think of “intelligence” as the core ability to develop and merge worldviews; and “rationality” as the ability to point intelligence in the most useful directions (i.e. taking into account where intelligence <i>should</i> be applied). Ideally we’d like to always be able to combine seemingly-incompatible worldviews into a single coherent perspective. But we usually face severe limitations on our ability to merge worldviews together (due to time constraints, cognitive limitations, or lack of information). I’ll call the skill of being able to deal with multiple incompatible worldviews, when your ability to combine them is extremely limited, meta-rationality. (Analogously, the ideal of emotional intelligence is to have integrated many different parts of yourself into a cohesive whole. But until you’ve done so, it’s important to have the skill of facilitating interactions between them. I won’t talk much about internal parts as an example of clashing worldviews throughout this post, but I think it’s a useful one to keep in mind.)<br /><br />I don’t think there’s any sharp distinction between meta-rationality and rationality. But I do think meta-rationality is an interesting limiting case to investigate. The core idea I’ll defend in this post is that, when our ability to synthesize worldviews into a coherent whole is very limited, we should use each worldview to separately determine an overall policy for how to behave, and then combine those policies at a high level (for example by allocating a share of resources to each). I’ll call this the policy approach to meta-rationality; and I’ll argue that it prevents a number of problems (such as over-deference) which arise when using other approaches, particularly the epistemic approach of combining the credences of different worldviews directly.</div><div><br /><span style="font-size: medium;">Comparing the two approaches<br /></span><br />Let’s consider one central example of meta-rationality: taking into account other people’s disagreements with us. In some simple cases, this is straightforward - if I vaguely remember a given statistic, but my friend has just looked it up and says I’m wrong, I should just defer to them on that point, and slot their correction into my existing worldview. But in some cases, other people have worldviews that clash with our own on large-scale questions, and we don’t know how to (or don’t have time to) merge them together without producing a frankenstein worldview with many internal inconsistencies.<br /><br />How should we deal with this case, or other cases involving multiple inconsistent worldviews? The epistemic approach suggests:</div><div><ol style="text-align: left;"><li>Generating estimates of how accurate we think each worldview’s claims are, based on its track record.</li><li>Using these estimates to evaluate important claims by combining each worldview’s credences to produce our “all-things-considered” credence for that claim.</li><li>Using our all-things-considered credences when making decisions.</li></ol>This seems sensible, but leads to a few important problems:</div><div><ol style="text-align: left;"><li>Merging estimates of different variables into all-things-considered credences might lead to very different answers depending on how it’s done, since after each calculation you lose information about the relationships between different worldviews’ answers to different questions. For example: worldview A might think that you’re very likely to get into MIT if you apply, but if you attend you’ll very likely regret it. And worldview B might think you have almost no chance of getting in, but if you do attend you’ll very likely be happy you went. Calculating all-things-considered credences separately would conclude you have a medium-good chance of a medium-good opportunity, which is much more optimistic than either worldview individually.</li><ol><li>One might respond that this disparity only arises when you’re applying the epistemic approach naively, where the non-naive approach would be to only combine worldviews' final expected utility estimates. But I think that the naive approach is actually the most commonly-used version of the epistemic approach - e.g. Ben Garfinkel talks about using deference to calculate likelihoods of risk <a href="https://forum.effectivealtruism.org/posts/NBgpPaz5vYe3tH4ga/on-deference-and-yudkowsky-s-ai-risk-estimates?commentId=FEbpGZcxtQ9bhwQZN">in this comment</a>; and <a href="https://forum.effectivealtruism.org/posts/WKPd79PESRGZHQ5GY/in-defence-of-epistemic-modesty">Greg Lewis defends</a> using epistemic modesty for “virtually all” beliefs. Also, combining expected utility estimates isn't very workable either, as I'll discuss in the next point.</li></ol><li>Worldviews might disagree deeply on which variables we should be estimating, with no clear way to combine those variables into a unified decision, due to: Differences in empirical beliefs which lead to different focuses. E.g. if one worldview thinks that cultural change is the best way to influence the future, and another thinks that technical research works better, it may be very difficult to convert impacts on those two areas into a "common unit" for expected value comparisons. Even if we manage to do so, empirical disagreements can lead one worldview to dominate another - e.g. if one falls prey to <a href="https://en.wikipedia.org/wiki/Pascal%27s_mugging">Pascal’s mugging</a>, then its expected values could skew high enough that it effectively gets to make all decisions from then on, even if other worldviews are ignoring the mugging.</li><li><a href="https://www.clearerthinking.org/post/understand-how-other-people-think-a-theory-of-worldviews">Deep-rooted values disagreements</a>. If worldviews don't share the same values, we can't directly compare their expected value estimates. Even if every worldview can formulate its values in terms of a utility function, there's no canonical way to merge utility estimates across worldviews with different values, for <a href="https://www.econlib.org/interpersonal-comparisons-of-utility">the same reason</a> that there’s no canonical way to compare utilities across people: utility functions are equivalent up to positive affine transformations (multiplying one person’s utilities by 1000 doesn’t change their choices, but does change how much they’d influence decisions if we added different peoples’ utilities together.)</li><ol><li>One proposed solution is <a href="https://www.lesswrong.com/posts/dX7vNKg4vex5vxWCW/making-decisions-under-moral-uncertainty-1#Normalised_MEC_and_Variance_Voting">variance normalization</a>, where worldviews' utilities are normalized to have the same mean and variance. But that can allow small differences in how we differentiate the options to significantly affect how a worldview’s utilities are normalized. (For example, “travel to Paris this weekend” could be seen as one plan, or be divided into many more detailed plans: “travel to Paris today by plane”, “travel to Paris tomorrow by train”, etc.) It's also difficult to figure out what distribution to normalize over (as I'll discuss later).</li></ol><li>Some worldviews might contain important action-guiding insights without being able to generate accurate empirical predictions - for example, a worldview which tells us that following a certain set of principles will tend to go well, but doesn’t say much about which good outcomes will occur.</li><li>Some worldviews might be able to generate accurate empirical predictions without containing important action-guiding insights. For example, a worldview which says “don’t believe extreme claims” will be right much more often than it’s wrong. But since extreme claims are the ones most likely to lead to extreme opportunities, it might only need to be wrong once or twice for the harm of listening to it to outweigh the benefits of doing so. Or a worldview that has strong views about the world in general, but little advice for your specific situation.</li><li>Since there are many other people whose worldviews are, from an outside perspective, just as trustworthy as ours (or more so), many other worldviews should be given comparable weight to our own. But then when we average them all together, our all-things-considered credences will be dominated by other people’s opinions, and we should basically never make important decisions based on our own opinions. Greg Lewis bites that bullet <a href="https://forum.effectivealtruism.org/posts/WKPd79PESRGZHQ5GY/in-defence-of-epistemic-modesty">in this post</a>, but I think most people find it pretty counterintuitive.</li></ol>The key problem which underlies these different issues is that the epistemic approach evaluates and merges the beliefs of different worldviews too early in the decision-making process, before the worldviews have used their beliefs to evaluate different possible strategies. By contrast, the policy approach involves:</div><div><ol style="text-align: left;"><li>Generating estimates of how useful we think each worldview’s advice is, based on its track record.</li><li>Getting each worldview to identify the decisions that it cares most about influencing.</li><li>Combining the worldviews’ advice to form an overall strategy (or, in reinforcement learning terminology, a <i>policy</i>), based both on how useful we think the worldview is and also on how much the worldview cares about each part of the strategy.</li></ol>One intuitive description of how this might occur is the <a href="https://www.fhi.ox.ac.uk/wp-content/uploads/2021/06/Parliamentary-Approach-to-Moral-Uncertainty.pdf">parliamentary approach</a>. Under this approach, each worldview is treated as a delegate in a parliament, with a number of votes proportional to how much weight is placed on that worldview; delegates can then spread their votes over possible policies, with the probability of a policy being chosen proportional to how many votes are cast for it.<br /><br />The policy approach largely solves the problems I identified previously:</div><div><ol style="text-align: left;"><li>Since each worldview separately calculates the actions it recommends in a given domain, no information is lost by combining estimates of variables before using them to make decisions. In the college admissions example, each worldview will separately conclude “don’t put too much effort into college admissions”, and so any reasonable combined policy will follow that advice.</li><li>The policy approach doesn’t require us to compare utilities across worldviews, since it’s <i>actions</i> not <i>utilities</i> that are combined across policies. Policies do need to prioritize some decisions over others - but unlike in the epistemic case, this doesn’t depend on how decisions are differentiated, since policies get to identify for themselves how to differentiate the decisions. (However, this introduces other problems, which I’ll discuss shortly.)</li><ol><li>Intuitively speaking, this should result in worldviews negotiating for control over whichever parts of the policy they care about most, in exchange for giving up control over parts of the policy they care about least. (This might look like different worldviews controlling different domains, or else multiple worldviews contributing to a compromise policy within a single domain.) Worldviews which care equally about all decisions would then get to make whichever decisions the other worldviews care about least.</li></ol><li>Worldviews which can generate good advice can be favored by the policy approach even if they don’t produce accurate predictions.</li><li>Worldviews which produce accurate predictions but are overall harmful to give influence over your policy will be heavily downweighted by the policy approach.</li><li>Intuitively speaking, the reason we should pay much more attention to our own worldview than to other people’s is that, in the long term, it pays to develop and apply a unified worldview we understand very well, rather than being pulled in different directions by our incomplete understanding of others’ views. The policy approach captures this intuition: a worldview might be very reasonable but unable to give us actionable advice (for example if we don’t know how to consistently apply the worldview to our own situation). Under the policy approach, such worldviews either get lower weight in the original estimate, or else aren’t able to identify specific decisions they care a lot about influencing, and therefore have little impact on our final strategy. Whereas our own worldview tends to be the most actionable, especially in the scenarios where we’re most successful at achieving our goals.</li></ol><br />I also think that the policy approach is much more compatible with good community dynamics than the epistemic approach. I’m worried about cycles where everyone defers to everyone else’s opinion, which is formed by deferring to everyone else’s opinion, and so on. Groupthink is already a common human tendency even in the absence of explicit epistemic-modesty-style arguments in favor of it. By contrast, the policy approach eschews calculating or talking about all-things-considered credences, which pushes people towards talking about (and further developing) their own worldviews, which has positive externalities for others who can now draw on more distinct worldviews to make their own decisions.</div><div><br /><span style="font-size: medium;">Problems with the policy approach<br /></span><br />Having said all this, there are several salient problems with the policy approach; I’ll cover four, but argue that none of them are strong objections.<br /><br />Firstly, although we have straightforward ways to combine credences on different claims, in general it can be much harder to combine different policies. For example, if two worldviews disagree on whether to go left or right (and both think it’s a very important decision) then whatever action is actually taken will seem very bad to at least one of them. However, I think this is mainly a problem in toy examples, and becomes much less important in the real world. In the real world, there are almost always many different strategies available to us, rather than just two binary options. This means that there’s likely a compromise policy which doesn’t differ too much from any given worldview’s policy on the issues it cares about most. Admittedly, it’s difficult to specify a formal algorithm for finding that compromise policy, but the idea of fairly compromising between different recommendations is one that most humans find intuitive to reason about. A simple example: if two policies disagree on many spending decisions, we can give each a share of our overall budget and let it use that money how it likes. Then each policy will be able to buy the things it cares about most: getting control over half the money is usually much more than twice as valuable as getting control over all the money.<br /><br />Secondly, it may be significantly harder to produce a good estimate of the value of each worldview’s advice than the accuracy of each worldview’s predictions, because we tend to have much less data about how well personalized advice works out. For example, if a worldview tells us what to do in a dozen different domains, but we only end up entering one domain, it’s hard to evaluate the others. Whereas if a worldview makes predictions about a dozen different domains, it’s easier to evaluate all of them in hindsight. (This is analogous to how credit assignment is much harder in reinforcement learning than in supervised learning.)<br /><br />However, even if in practice we end up mostly evaluating worldviews based on their epistemic track record, I claim that it’s still valuable to consider the epistemic track record as a proxy for the quality of their advice, rather than using it directly to evaluate how much we trust each worldview. For example: suppose that a worldview is systematically overconfident. Using a direct epistemic approach, this would be a big hit to its trustworthiness. However, the difference between being overconfident and being well-calibrated plausibly changes the worldview’s advice very little, e.g. because it doesn’t change that worldview’s relative ranking of options. Another example: predictions which many people disagree with can allow you to find valuable neglected opportunities, even if conventional wisdom is more often correct. So when we think of predictions as a proxy for advice quality, we should place much more weight on whether predictions were <i>novel</i> and <i>directionally correct</i> than whether they were precisely calibrated.<br /><br />Thirdly, the policy approach as described thus far doesn’t allow worldviews to have more influence over some individuals than others - perhaps individuals who have skills that one worldview cares about far more than any other; or perhaps individuals in worlds where one worldview’s values can be fulfilled much more easily than others’. Intuitively speaking, we’d like worldviews to be able to get more influence in those cases, in exchange for having less influence in other cases. In the epistemic approach, this is addressed via variance normalization across many possible worlds - but as discussed above, this could be significantly affected by how you differentiate the possibilities (and also what your prior is over those worlds). I think the policy approach can deal with this in a more principled way: for any set of possible worlds (containing people who follow some set of worldviews) you can imagine the worldviews deciding on how much they care about different decisions by different people in different possible worlds <i>before</i> they know which world they’ll actually end up in. In this setup, worldviews will trade away influence over worlds they think are unlikely and people they think are unimportant, in exchange for influencing the people who will have a lot of influence over more likely worlds (a dynamic closely related to <a href="https://arxiv.org/abs/1701.01302">negotiable reinforcement learning</a>).<br /><br />This also allows us a natural interpretation of what we’re doing when we assign weights to worldviews: we’re trying to rederive the relative importance weights which worldviews would have put on the branch of reality we actually ended up in. However, the details of how one might construct this “<a href="https://www.lesswrong.com/tag/updateless-decision-theory">updateless</a>” <a href="https://en.wikipedia.org/wiki/Original_position">original position</a> are an open problem.<br /><br />One last objection: hasn’t this become far too complicated? “Reducing” the problem of epistemic deference to the problem of updateless multi-agent negotiation seems very much like a wrong-way reduction - in particular because in order to negotiate optimally, delegates will need to understand each other very well, which is precisely the work that the whole meta-rationality framing was attempting to avoid. (And given that they understand each other, they might try adversarial strategies like threatening other worldviews, or choosing which decisions to prioritize based on what they expect other worldviews to do.)<br /><br />However, even if finding the optimal multi-agent bargaining solution is very complicated, the question that this post focuses on is how to act given severe constraints on our ability to compare and merge worldviews. So it’s consistent to believe that, if worldviews are unable to understand each other, they’ll do better by merging their policies than merging their beliefs. One reason to favor this idea is that multi-agent negotiation makes sense to humans on an intuitive level - which hasn’t proved to be true for other framings of epistemic modesty. So I expect this “reduction” to be pragmatically useful, especially when we’re focusing on simple negotiations over a handful of decisions (and given some intuitive notion of worldviews acting “in good faith”).<br /><br />I also find this framing useful for thinking about the overall problem of understanding intelligence. Idealized models of cognition like <a href="https://www.lesswrong.com/posts/wsBpJn7HWEPCJxYai/excerpt-from-arbital-solomonoff-induction-dialogue">Solomonoff induction</a> and <a href="https://en.wikipedia.org/wiki/AIXI">AIXI</a> treat hypotheses (aka worldviews) as intrinsically distinct. By contrast, thinking of these as models of the limiting case where we have no ability to combine worldviews naturally points us towards the question of what models of intelligence which involve worldviews being merged might look like. This motivates me to keep a hopeful eye on various work on formal models of ideal cognition using partial hypotheses which could be merged together, like <a href="https://www.lesswrong.com/posts/N5Jm6Nj4HkNKySA5Z/finite-factored-sets">finite factored sets</a> (see also <a href="https://arxiv.org/abs/2109.11513">the paper</a>) and <a href="https://www.lesswrong.com/posts/zB4f7QqKhBHa5b37a/introduction-to-the-infra-bayesianism-sequence">infra-bayesianism</a>. I also note a high-level similarity between the approach I've advocated here and <a href="https://www.fhi.ox.ac.uk/wp-content/uploads/Anthropic_Decision_Theory_Tech_Report.pdf">Stuart Armstrong's anthropic decision theory</a>, which dissolves a number of anthropic puzzles via converting them to decision problems. The core insight in both cases is that confusion about how to form beliefs can arise from losing track of how those beliefs should relate to our decisions - a principle which may well help address other important problems.</div>Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-32473351961004048032022-05-25T11:28:00.003-07:002022-07-21T16:48:15.386-07:00Science-informed normativity<p><span style="font-family: inherit;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The debate over moral realism is often framed in terms of a binary question: are there ever objective facts about what’s moral to do in a given situation? The broader question of normative realism is also framed in a similar way: are there ever objective facts about what’s rational to do in a given situation? But I think we can understand these topics better by reframing them in terms of the question: how much do normative beliefs converge or diverge as ontologies improve? In other words: let’s stop thinking about whether we can derive normativity from nothing, and start thinking about how much normativity we can derive from how little, given that we continue to improve our understanding of the world. The core intuition behind this approach is that, even if a better understand of science and mathematics can’t </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">directly</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> tell us what we should value, it can heavily influence how our values develop over time.</span></span></p><span id="docs-internal-guid-70690251-7fff-2d2e-3753-1842fbfcd2f6"><span style="font-family: inherit;"><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Values under ontology improvements</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">By “ontology” I mean the set of concepts which we use to understand the world. Human ontologies are primarily formulated in terms of objects which persist over time, and which have certain properties and relationships. The details have changed greatly throughout history, though. To explain fire and disease, we used to appeal to spirits and curses; over time we removed them and added entities like </span><a href="https://en.wikipedia.org/wiki/Phlogiston_theory" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">phlogiston</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> and </span><a href="https://en.wikipedia.org/wiki/Miasma_theory" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">miasmas</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">; now we’ve removed those in turn and replaced them with oxidation and bacteria. In other cases, we still use old concepts, but with an understanding that they’re only approximations to more sophisticated ones - like absolute versus relative space and time. In other cases, we’ve added novel entities - like dark matter, or complex numbers - in order to explain novel phenomena.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I’d classify all of these changes as “improvements” to our ontologies. What specifically counts as an improvement (if anything) is an ongoing debate in the philosophy of science. For now, though, I’ll assume that readers share roughly common-sense intuitions about ontology improvement - e.g. the intuition that science has dramatically improved our ontologies over the last few centuries. Now imagine that our ontologies continue to dramatically improve as we come to better understand the world; and that we try to reformulate moral values from our old ontologies in terms of our new ontologies in a reasonable way. What might happen?</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Here are two extreme options. Firstly, very similar moral values might end up in very different places, based on the details of how that reformulation happens, or just because the reformulation is quite sensitive to initial conditions. Or alternatively, perhaps even values which start off in very different places end up being very similar in the new ontology - e.g. because they turn out to refer to different aspects of the same underlying phenomenon. These, plus intermediate options between them, define a spectrum of possibilities. I’ll call the divergent end of this spectrum (</span><a href="https://thinkingcomplete.blogspot.com/2019/02/arguments-for-moral-indefinability.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">which I’ve defended elsewhere</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">) the “moral anti-realism” end, and the convergent end the “moral realism” end.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This will be much clearer with a few concrete examples (although note that these are only illustrative, because the specific beliefs involved are controversial). Consider two people with very different values: an egoist who only cares about their own pleasure, and a hedonic utilitarian. Now suppose that each of them comes to believe </span><a href="http://www.davidjinkins.com/other_writings/files/reasons_and_persons.pdf" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Parfit’s argument that personal identity is a matter of degree</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, so that now the concept of their one “future self” is no longer in their ontology. How might they map their old values to their new ontology? Not much changes for the hedonic utilitarian, but a reasonable egoist will start to place some value on the experiences of people who are “partially them”, who they previously didn’t care about at all. Even if the egoist’s priorities are still quite different from the utilitarian’s, their values might end up significantly closer together than they used to be.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">An example going the other way: consider two deontologists who value non-coercion, and make significant sacrifices to avoid coercing others. Now consider an ontological shift where they start to think about themselves as being composed of many different subagents which care about different things - career, relationships, morality, etc. The question arises: does it count as “coercion” when one subagent puts a lot of pressure on the others, e.g. by inducing a strong feeling of guilt? It’s not clear that there’s a unique reasonable answer here. One deontologist might reformulate their values to only focus on avoiding coercion of others, even when they need to “force themselves” to do so. The other might decide that internal coercion is also something they care about avoiding, and reduce the extent to which they let their “morality” subagent impose its will on the others. So, from a very similar starting point, they’ve diverged significantly under (what we’re assuming is an) ontological improvement.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Other examples of big ontological shifts: converting from theism to atheism; becoming an illusionist about consciousness; changing one’s position on free will; changing one’s mind about the act-omission distinction (e.g. because the intuitions for why it’s important fall apart in the face of counterexamples); starting to believe in a multiverse (which has implications for </span><a href="https://www.nickbostrom.com/ethics/infinite.pdf" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">infinite ethics</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">); and many others which we can’t imagine yet. Some of these shifts might be directly prompted by moral debate - but I think that most “moral progress” is downstream of ontological improvements driven by scientific progress. Here I’m just defining moral progress as reformulating values into a better ontology, in any reasonable way - where a person on the anti-realist side of the spectrum expects that there are many possible outcomes of moral progress; but a person on the realist side expects there are only a few, or perhaps just one.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Normative realism</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">So far I’ve leaned heavily on the idea of a “reasonable” reformulation. This is necessary because there are always </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">some</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> possible reformulations which end up very divergent from others. (For example, consider the reformulation “given a new ontology, just pretend to have the old ontology, and act according to the old values”.) So in order for the framework I’ve given so far to not just collapse into anti-realism, we need some constraints on what’s a “reasonable” or “rational” way to shift values from one ontology to another.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Does this require that we commit to the existence of facts about what’s rational or irrational? Here I’ll just apply the same move as I did in the moral realism case. Suppose that we have a set of judgments or criteria about what counts as rational, in our current ontology. For example, our current ontology includes “beliefs”, “values”, “decisions”, etc; and most of us would classify the claim “I no longer believe that ‘souls’ are a meaningful concept, but I still </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">value</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> people’s souls” as irrational. But our ontologies improve over time. For example, Kahneman and Tversky’s work on dual process theory (as well as the more general distinction between conscious and unconscious processing) clarifies that “beliefs” aren’t a unified category - we have different types of beliefs, and different types of preferences too. Meanwhile, the ontological shifts I mentioned before (about personal identity, and internal subagents) also have ramifications for what we mean when talking about beliefs, values, etc. If we try to map our judgements of what’s reasonable into our new ontology in a reflectively consistent way (i.e. a way that balances between being “reasonable” according to our old criteria, and “reasonable” according to our new criteria), what happens? Do different conceptions of rationality converge, or diverge? If they strongly converge (the “normative realist” position) then we can just define reasonableness in terms of similarity to whatever conception of rationality we’d converge to under ontological improvement. If they strongly diverge, then…well, we can respond however we’d like; anything goes!</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I’m significantly more sympathetic to normative realism as a whole than moral realism, in particular because of various results in probability theory, utility theory, game theory, decision theory, machine learning, etc, which are providing increasingly strong constraints on rational behavior (e.g. by constructing different types of dutch books). In the next section, I’ll discuss one theory which led me to a particularly surprising ontological shift, and made me much more optimistic about normative realism. Having said that, I’m not as bullish on normative realism as some others; my best guess is that we’ll make some discoveries which significantly improve our understanding of what it means to be rational, but others which show us that there’s no “complete” understanding to be had (analogous to mathematical incompleteness theorems).</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Functional decision theory as an ontological shift</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">There’s one particular ontological shift which inspired this essay, and which I think has dragged me significantly closer to the moral/normative realist end of the spectrum. I haven’t mentioned it so far, since it’s not very widely-accepted, but I’m confident enough that there’s something important there that I’d like to discuss it now. The ontological shift is the one from Causal Decision Theory (CDT) to </span><a href="https://intelligence.org/2017/10/22/fdt/" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Functional Decision Theory</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> (FDT). I won’t explain this in detail, but in short: CDT tells us to make decisions using an ontology based on the choices of individual </span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">agents. FDT tells us to make decisions using an ontology based on the choices of </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">functions</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> which may be implemented in multiple agents (and by expanding the concepts of causation and possible worlds to include logical causation and counterpossible worlds).</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Because of these shifts, a “selfish” agent using FDT can end up making choices more similar to the choices of an altruistic CDT agent than a selfish CDT agent, for reasons closely related to the traditional moral intuition of universalizability. FDT is still a very incomplete theory, but I find this a very surprising and persuasive example of how ontological improvements might drive convergence towards some aspects of morality, which made me understand for the first time how moral realism might be a coherent concept! (Another very interesting but more speculative point: one axis on which different versions of FDT vary is how “updateless” they are. Although we don’t know how to precisely specify updatelessness, increasingly updateless agents behave as if they’re increasingly altruistic, even towards other agents who could never reciprocate.)</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Being unreasonable</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Suppose an agent looks at a reformulation to a new ontology, and just refuses to accept it - e.g. “I no longer believe that ‘souls’ are a meaningful concept, but I still </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">value</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> people’s souls”. Well, we could tell them that they were being irrational; and most such agents care enough about rationality that this is a forceful objection. I think the framing I’ve used in this document makes this argument particularly compelling - when you move to a new ontology in which your old concepts are clearly inadequate or incoherent, then it’s pretty hard to defend the use of those old concepts. (This is a reframing of the philosophical debate on </span><a href="https://plato.stanford.edu/entries/moral-motivation/#IntVExt" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">motivational internalism</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">.)</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">But what if they said “I believe that I am being irrational, but I just </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">refuse</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> to stop being irrational”; how could we respond then? The standard answer is that we say “you lose” - we explain how we’ll be able to exploit them (e.g. via dutch books). Even when abstract “irrationality” is not compelling, “losing” often is. Again, that’s particularly true under ontology improvement. Suppose an agent says “well, I just won’t take bets from Dutch bookies”. But then, once they’ve improved their ontology enough to see that </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">all </span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">decisions under uncertainty are a type of bet, they can’t do that - or at least they need to be much unreasonable to do so.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">None of this is particularly novel. But one observation that I haven’t seen before: the “you lose” argument becomes increasingly compelling the bigger the world is. Suppose you and I only care about our wealth, but I use a discount rate 1% higher than yours. You tell me “look, in a century’s time I’ll end up twice as rich as you”. It might not be that hard for me to say “eh, whatever”. But suppose you tell me “we’re going to live for a millennium, after which I’ll predictably end up 20,000 times richer than you” - now it feels like a wealth-motivated agent would need to be </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">much more unreasonable</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> to continue applying high discounts. Or suppose that I’m in a Pascal’s mugging scenario where I’m promised very high rewards with very low probability. If I just shrug and say “I’m going to ignore all probabilities lower than one in a million”, then it might be pretty tricky to exploit me - a few simple heuristics might be able to prevent myself being dutch-booked. But suppose now that we live in a multiverse where every possible outcome plays out, in proportion to how likely it is. Now ignoring small probabilities could cause you to lose a large amount of value in a large number of multiverse branches - something which hooks into our intuitive sense of “unreasonableness” much more strongly than the idea of “ignoring small probabilities” does in the abstract. (Relatedly, I don’t think it’s a coincidence that utilitarianism has become so much more prominent in the same era where we’ve become so much more aware of the vastness of the universe around us.)</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Why am I talking so much about “reasonableness” and moral persuasion? After all, agents which are more rational will tend to survive more often, acquire more resources, and become more influential: in the long term, evolution will do the persuasion for us. But it’s not clear that the future will be shaped by evolutionary pressures - it might be shaped by the decisions of goal-directed agents. Our civilization might be able to “lock in” certain constraints - like enough centralization of decision-making that the future is steered by arguments rather than evolution. And thinking about convergence towards rationality also gives us a handle for reasoning about artificial intelligence. In particular, it would be very valuable to know how much applying a minimal standard of reasonableness to their decisions would affect how goal-directed they’ll be, and how aligned their goals will be with our own.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">How plausible is this reasoning?</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I’ve been throwing around a lot of high-level concepts here, and I wouldn’t blame readers for feeling suspicious or confused. Unfortunately, I don’t have the time to make them clearer. In lieu of that, I’ll briefly mention three intuitions which contribute towards my belief that the position I’ve sketched in this document is a useful one.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Firstly, I see my reframing as a step away from </span><a href="https://www.lesswrong.com/posts/yA4gF5KrboK2m2Xu7/how-an-algorithm-feels-from-inside" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">essentialism</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, which seems to me to be the most common mistake in analytic philosophy. Sometimes it’s pragmatically useful to think in terms of clear-cut binary distinctions, but in general we should almost always aim to be able to ground out those binary distinctions in axes of continuous variation, to avoid our standard bias towards essentialism. In particular, the moral realism debate tends to focus on a single binary question (do agents converge to the same morality given </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">no</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> pre-existing moral commitments?) whereas I think it’d be much more insightful to focus on a less binary question (how small or large is the space of pre-existing moral commitments which will converge)?</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Secondly, there’s a nice parallel between the view of morality which I’ve sketched out here, and the approach some mathematicians take, of looking at different sets of axioms to see whether they lead to similar or different conclusions. In our case, we’d like to understand whether similar starting intuitions and values will converge or diverge under a given approach to ontological reformulation. (I discuss the ethics-mathematics analogy in more detail </span><a href="https://thinkingcomplete.blogspot.com/2018/04/a-pragmatic-approach-to-interpreting.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">here</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">.) If we can make progress on meta-ethics by actually answering object-level questions like “how would my values change if I believed X?”, that helps address another common mistake in philosophy - failing to link abstract debates to concrete examples which can be deeply explored to improve philosophers’ intuitions about the problem.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">And thirdly, I think this framing fits well with our existing experiences. Our values are strongly determined by evolved instincts and emotions, which operate using a more primitive ontology than the rest of our brains. So we’ve actually got plenty of experience in struggling to shift various values from one ontology to another, and the ways in which some people manage to do so, and some remain unreasonable throughout. We just need to imagine this process continuing as we come to understand the world far better than we do today.</span></p></span></span>
Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-31202125482620182762022-04-15T00:57:00.007-07:002022-04-15T01:16:13.982-07:00Three intuitions about effective altruism: responsibility, scale, self-improvement<p style="text-align: left;"><span style="font-family: inherit;"><span style="background-color: white; color: #212121;">This is a post about three intuitions for how to think about the effective altruism community.</span></span></p><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: #212121; line-height: 1.4; word-break: break-word;"><div style="box-sizing: inherit;"><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><strong id="Part_1__responsibility" style="box-sizing: inherit;"><span style="font-family: inherit;">Part 1: responsibility</span></strong></p><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><span style="font-family: inherit;">The first intuition is that, in a global sense, there are no “adults in the room”. Before covid I harboured a hope that, despite the incessant political squabbling we see worldwide, in the face of a major crisis with global implications, there were serious people who would come out of the woodwork to ensure that it went well. There weren’t. And that’s not just a national phenomenon, that’s a global phenomenon. Even countries like New Zealand, which handled covid incredibly well, weren’t taking <i style="box-sizing: inherit;">responsibility</i> in the global way I’m thinking about - they looked after their own citizens, but didn’t try to speed up vaccine distribution overall (e.g. by allowing human challenge trials), or fix everyone else’s misunderstandings.</span></p><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><span style="font-family: inherit;">Others developed the same “no adults in the room” intuition by observing failures on different issues. For some, <span style="box-sizing: inherit;"><span style="box-sizing: inherit;"><span style="box-sizing: inherit;"><a href="https://intelligence.org/2018/02/28/sam-harris-and-eliezer-yudkowsky/" style="box-sizing: inherit; color: #0c869b; cursor: pointer; text-decoration-line: none;">AI risk</a></span></span></span>; for others, climate change; for others, policies like <span style="box-sizing: inherit;"><span style="box-sizing: inherit;"><span style="box-sizing: inherit;"><a href="https://foreignpolicy.com/2019/11/01/immigration-wall-open-borders-trillion-dollar-idea/" style="box-sizing: inherit; color: #0c869b; cursor: pointer; text-decoration-line: none;">immigration</a></span></span></span> or <span style="box-sizing: inherit;"><span style="box-sizing: inherit;"><span style="box-sizing: inherit;"><a href="https://www.worksinprogress.co/issue/the-housing-theory-of-everything/" style="box-sizing: inherit; color: #0c869b; cursor: pointer; text-decoration-line: none;">housing reform</a></span></span></span>. I don’t think covid is a bigger failure than any of these, but I think it comes much closer to creating common knowledge that the systems we have in place aren’t capable of steering through global crises. This naturally points us towards a long-term goal for the EA community: to <i style="box-sizing: inherit;">become</i> the adults in the room, the people who are responsible enough and capable enough to steer humanity towards good outcomes.</span></p><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><span style="font-family: inherit;">By this I mean something different from just “being in charge” or “having a lot of power”. There are many large power structures, containing many competent people, which try to keep the world on track in a range of ways. What those power structures <i style="box-sizing: inherit;">don’t </i>have is the ability to absorb novel ideas and take novel actions in response. In other words, the wider world solves large problems via <span style="box-sizing: inherit;"><span style="box-sizing: inherit;"><span style="box-sizing: inherit;"><a href="https://en.wikipedia.org/wiki/OODA_loop" style="box-sizing: inherit; color: #0c869b; cursor: pointer; text-decoration-line: none;">OODA loops</a></span></span></span> that take decades. In the case of climate change, decades of advocacy led to public awareness which led to large-scale policies, plus significant reallocation of talent. I think this will be enough to avoid catastrophic outcomes, but that’s more from luck than skill. In the case of covid, the OODA loop on substantially changing vaccine regulations was far too long to make a difference (although maybe it’ll make a difference to the next pandemic).</span></p><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><span style="font-family: inherit;">The rest of the world has long OODA loops because people on the inside of power structures don’t have strong incentives to fix problems; and because people on the outside can’t mobilise people, ideas and money quickly. But EA can. I don’t think there’s any other group in the world which can allocate as much talent as quickly as EA has; I don’t think there’s any other group which can identify and propagate important new ideas as quickly as EA can; and there are few groups which can mobilise as much money as flexibly.</span></p><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><span style="font-family: inherit;">Having said all that, I don’t think we’re <i style="box-sizing: inherit;">currently</i> the adults in the room, or else we would have made much more of a difference during covid. While it’s not itself a central EA concern, it’s closely related to one of our central concerns, and would have been worth addressing for reputational reasons alone. But I do think we were <i style="box-sizing: inherit;">closer</i> to being the adults in the room than almost any other group - particularly in terms of long-term warnings about pandemics, short-term warnings about covid in particular, and converging quickly towards accurate beliefs. We should reflect on what would have been needed for us to convert those advantages into much more concrete impact.</span></p><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><span style="font-family: inherit;">I want to emphasise, though, that being the adults in the room doesn’t require each individual to take on a feeling of responsibility towards the world. Perhaps a better way to think about it: every individual EA should take responsibility for the EA community functioning well, and the EA community should take responsibility for the world functioning well. (I’ve written a little about the first part of that claim in <span style="box-sizing: inherit;"><span style="box-sizing: inherit;"><a class="PostLinkPreviewWithPost-link" href="https://forum.effectivealtruism.org/posts/ctnMCdTH7dmiN4jBx/lessons-from-my-time-in-effective-altruism#4__I_should_have_been_more_proactive_and_willing_to_make_unusual_choices_" style="box-sizing: inherit; color: #0c869b; cursor: pointer; text-decoration-line: none;">point four of this post</a></span></span>.)</span></p><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><strong id="Part_2__scale__not_marginalism" style="box-sizing: inherit;"><span style="font-family: inherit;">Part 2: scale, not marginalism</span></strong></p><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><span style="font-family: inherit;">Historically, EA has thought primarily about the <i style="box-sizing: inherit;">marginalist</i> question of how to do the most good <i style="box-sizing: inherit;">per unit of resources</i>. An alternative, which is particularly natural in light of part 1, is to simply ask: how can we do the most good <i style="box-sizing: inherit;">overall</i>? In some sense these are tautologically equivalent, given finite resources. But a marginalist mindset makes it harder to be very ambitious - it cuts against thinking at scale. For the most exciting projects, the question is not “how effectively are we using our resources”, but rather “can we make it work at all?” - where if it does work it’ll be a huge return on any realistic amount of investment we might muster. This is basically the startup investor mindset; and the mindset that focuses on <span style="box-sizing: inherit;"><span style="box-sizing: inherit;"><a class="TagHoverPreview-link" href="https://forum.effectivealtruism.org/tag/effective-altruism-megaprojects" style="box-sizing: inherit; color: #0c869b; cursor: pointer; text-decoration-line: none;">megaprojects</a></span></span>.</span></p><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><span style="font-family: inherit;">Marginalism has historically focused on evaluating possible projects to find the best one. Being scale-focused should nudge us towards focusing more on <i style="box-sizing: inherit;">generating</i> possible projects. On a scale-focused view, the hardest part is finding <i style="box-sizing: inherit;">any</i> lever which will have a big impact on the world. Think of a scientist noticing an anomaly which doesn’t fit into their existing theories. If they tried to evaluate whether the effects of understanding the anomaly will be good or bad, they’d find it very difficult to make progress, and maybe stop looking. But if they approach it in a curious way, they’re much more likely to discover levers on the world which nobody else knows about; and then this allows them to figure out what to do.</span></p><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><span style="font-family: inherit;">There are downsides of scaling, though. Right now, EA has short OODA loops because we have a very high concentration of talent, a very high-trust environment, and a small enough community that coordination costs are low. As we try to do more large-scale things, these advantages will slowly diminish; how can we maintain short OODA loops regardless? I’m very uncertain; this is something we should think more about. (One wild guess: we might be the one group best-placed to leverage AI to solve internal coordination problems.)</span></p><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><strong id="Part_3__self_improvement_and_growth_mindset" style="box-sizing: inherit;"><span style="font-family: inherit;">Part 3: self-improvement and growth mindset</span></strong></p><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><span style="font-family: inherit;">In order to do these ambitious things, we need great people. Broadly speaking, there are two ways to get great people: recruit them, or create them. The tradeoff between these two can be difficult - focusing too much on the former can create a culture of competition and insecurity; focusing too much on the latter can be inefficient and soak up a lot of effort.</span></p><p style="box-sizing: inherit; margin-bottom: 1em; margin-top: 1em; text-align: left;"><span style="font-family: inherit;">In the short term, it seems like there are still low-hanging fruit when it comes to recruitment. But in the longer term, my guess is that EA will need to focus on teaching the skillsets we’re looking for - especially when recruiting high school students or early undergrads. Fortunately, I think there’s a lot of room to do better than existing education pipelines. Part of that involves designing specific programs (like <span style="box-sizing: inherit;"><span style="box-sizing: inherit;"><a class="PostLinkPreviewWithPost-link" href="https://forum.effectivealtruism.org/posts/iwTr8S8QkutyYroGy/apply-to-the-ml-for-alignment-bootcamp-mlab-in-berkeley-jan" style="box-sizing: inherit; color: #0c869b; cursor: pointer; text-decoration-line: none;">MLAB</a></span></span> or <span style="box-sizing: inherit;"><span style="box-sizing: inherit;"><a class="PostLinkPreviewWithPost-link" href="https://forum.effectivealtruism.org/posts/BpAKCeGMtQqqty9ZJ/agi-safety-fundamentals-curriculum-and-application" style="box-sizing: inherit; color: #0c869b; cursor: pointer; text-decoration-line: none;">AGI safety fundamentals</a></span></span>), but probably the more important part involves the culture of EA prioritising learning and growth.</span></p><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 1em; text-align: left;"><span style="font-family: inherit;">One model for how to do this is the entrepreneurship community. That’s another place where returns are very heavy-tailed, and people are trying to pick extreme winners - and yet it’s surprisingly non-judgemental. The implicit message I get from them is that anyone can be a great entrepreneur, if they try hard enough. That creates a virtuous cycle, because it’s not just a good way to push people to upskill - it also creates the sort of community that attracts ambitious and growth-minded people. I do think learning to be a highly impactful EA is harder in some ways than learning to be a great entrepreneur - we don’t get feedback on how we’re doing at anywhere near the same rate entrepreneurs do, so the strategy of trying fast and failing fast is much less helpful. But there are plenty of other ways to gain skills, especially if you’re in a community which gives you support and motivation to continually improve.</span></p></div></div><div class="PostsPagePostFooter-footerTagList" style="background-color: white; box-sizing: inherit; font-size: 13px; margin-bottom: 66px; margin-top: 16px;"><span class="FooterTagList-root" style="box-sizing: inherit; margin-bottom: 8px; margin-top: 8px;"><a href="https://forum.effectivealtruism.org/about#Finding_content" style="box-sizing: inherit; cursor: pointer; text-decoration-line: none;"><span class="LWTooltip-root" style="box-sizing: inherit; display: inline-block;"></span></a></span></div>Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-5192332464340235702022-04-02T11:51:00.011-07:002022-04-02T17:48:10.854-07:00Book review: Very Important People<p><span style="font-family: inherit;"><span style="white-space: pre-wrap;"></span></span></p><div class="separator" style="clear: both; text-align: center;"><span style="font-family: inherit;"><span style="white-space: pre-wrap;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg25Sn6crj9iSpQnAYDGhCr6mqu4UiGC_MtQBp1ZfX07oKHSaupj_nS-7vgnFObRsCTmnpi26UjkK6VGRJI5wDPHBzr0MaUG7RD1ZxE2FhgifF0ZcHcqwvVH5sK9v_OQKA5imzweVJuCwEfbmlV-z-GtJmmpj4uzI6VJlNcXoQI2HM3vkVRzpVg8hnT" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img alt="" data-original-height="2560" data-original-width="1684" height="320" src="https://blogger.googleusercontent.com/img/a/AVvXsEg25Sn6crj9iSpQnAYDGhCr6mqu4UiGC_MtQBp1ZfX07oKHSaupj_nS-7vgnFObRsCTmnpi26UjkK6VGRJI5wDPHBzr0MaUG7RD1ZxE2FhgifF0ZcHcqwvVH5sK9v_OQKA5imzweVJuCwEfbmlV-z-GtJmmpj4uzI6VJlNcXoQI2HM3vkVRzpVg8hnT=w211-h320" width="211" /></a></span></span></div><span style="font-family: inherit;"><span style="white-space: pre-wrap;">New York’s nightclubs are the particle accelerators of sociology: reliably creating the precise conditions under which exotic extremes of status-seeking behaviour can be observed. Ashley Mears documents it all in her excellent book Very Important People: Status and Beauty in the Global Party Circuit. A model turned sociology professor, while researching the book she spent hundreds of nights in New York’s most exclusive nightclubs, as well as similar parties across the world. The book abounds with fascinating details; in this post I summarise it and highlight a few aspects which I found most interesting.</span></span><p></p><p><span style="font-family: inherit; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Here’s the core dynamic. There are some activities which are often fun: dancing, drinking, socialising. But they become </span><span style="font-family: inherit; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">much more fun</span><span style="font-family: inherit; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> when they’re associated with feelings of high status. So wealthy men want to use their money to buy the feeling of having high-status fun, by doing those activities while associated with (and ideally while popular amongst) other high-status people, particularly beautiful women.</span></p><p><span style="font-family: inherit; white-space: pre-wrap;">Unfortunately, explicit transactions between different forms of cultural capital are low-status - it demonstrates that you can’t get the other forms directly. So the wealthy men can’t just pay the beautiful women to come party with them. Instead an ecosystem develops which sells sufficient strategic ambiguity to allow (self- and other-) deception about the transaction which is taking place, via incorporating a series of middlemen.</span></p><p><span style="font-family: inherit; white-space: pre-wrap;">Specifically, wealthy men pay thousands at these nightclubs for table charges and “bottle service” - already-expensive alcohol marked up by 5x or much more. The nightclubs pay “promoters” to scout out and bring along dozens of beautiful women each night. Those women get access to an exclusive venue with many wealthy men - but by itself that’s not enough to motivate regular attendance, at least not from the prettiest. And most are careful not to ruin their reputations by actually accepting payments from the promoters. Instead, in order to bring enough girls, promoters each need to do a bunch of emotional labour, flirting, relationship-building, and many non-cash payments (food, transport, even accommodation). </span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I’m strongly reminded of Michael Sandel’s book </span><a href="https://www.amazon.com/What-Money-Cant-Buy-Markets/dp/0374533652" style="text-decoration-line: none; white-space: pre;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;"><i>What Money Can’t Buy</i></span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> - the intuitions about the corrosive effects of money are the same, they're just applied to a much less high-minded setting.</span></p><span id="docs-internal-guid-a76d8bec-7fff-8e02-1643-e3cef339747e"><span style="font-family: inherit;"><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Some interesting features of this system:</span></p><ul style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="margin-bottom: 0pt; margin-top: 12pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">At a top club, a promoter might get paid $1000 a night to bring out a dozen models or women who look like models. Notably, model-like beauty is much more highly-prized than conventional beauty - e.g. the clubs don’t allow access to women who aren’t unusually tall. Everyone selects for models even when they don’t personally find the model look as attractive, because the fashion industry has established this as the Schelling look for high-status women. (For more on how this happens, see Mears’ other book, Pricing Beauty; and the responses to </span><a href="https://twitter.com/RichardMCNgo/status/1508936677519024130?t=xe3MtIIvKqtXmVQrytTyww&s=19" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">my tweet about it</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">).</span></p></li><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The markup on increasingly large champagne bottles is determined less by the amount of champagne, and more by how ostentatious the purchase is. The biggest purchases, costing over 100k per bottle, therefore come with incredibly elaborate fanfare: all music stops, spotlights shine on the buyer, a whole train of staff bring out the drinks, etc.</span></p></li><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The nightclub profits by creating an atmosphere of “suspended reality” where a large group of people who all individually believe that buying status in this way is tacky can still convince themselves that all the other people </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">don’t </span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">think it’s tacky. Most of the profits don’t actually come from the biggest spenders, but rather the next tier down, who are inspired by the atmosphere, and anchored by stories of the biggest purchases.</span></p></li><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In contrast to the predominantly-white clients and models, promoters are disproportionately black. Mears talks about them having “colour capital”, and using some stereotypes to their advantage in order to catch attention. They need to be very charismatic and attractive in order to consistently convince girls to come along with them while not making their relationship seem too transactional.</span></p></li><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In some sense the whole system is grounded in the models’ sex appeal, but I think that the models’ prestige is just as important - as mentioned above, models are preferred to women who most men find more attractive, as well as preferred to women who have more transactional attitudes towards sex.</span></p></li><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Basically the same dynamics play out internationally as well - promoters offer girls free flights, food and accommodation in exchange for attendance at nightclubs in St Tropez, etc. On those trips the transactionality is usually a bit more obvious.</span></p></li><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">How can promoters afford to regularly wine and dine so many girls? Often they have deals with restaurants who give them leftover food in exchange for making the restaurant look more glamorous. Other times, wealthy men will host the dinners before the parties start. At the nightclub itself, they all drink for free. </span></p></li></ul><span style="font-family: inherit; white-space: pre-wrap;"><div><span><span style="font-family: inherit;"><span style="font-family: inherit; white-space: pre-wrap;"><br /></span></span></span></div>If I were a bit more cynical I might also say that the “fun” part of high-status fun is also mainly a strategic ambiguity which helps facilitate the status transaction - if people couldn’t convince themselves and others that they were having fun, their attempts to seem prestigious would be much more obvious. Perhaps it’s worth considering what differences you’d expect in a world where this is true vs false. (For example, might you expect that the highest-status men actually don’t spend much time dancing, drinking, or even socialising?)</span><br /><p dir="ltr" style="margin-bottom: 12pt; margin-top: 12pt;">The same might be true, to a lesser extent, of other types of high-status fun - which, in my circles, often involves quick-witted exchanges on arbitrary topics. Overall, though, after reading this book I do feel much luckier that silicon valley is largely disdainful of conspicuous consumption and other negative-sum status games; long may it stay that way.</p></span></span>
Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-43265908187937466692022-03-10T19:30:00.003-08:002022-03-11T18:27:22.852-08:00Beyond micromarriages<p><i><span style="font-family: inherit;">tl;dr micromarriages aren't fully analogous to micromorts, which makes it tricky to define them satisfactorily. I introduce an alternative unit: QAWYs (Quality-Adjusted Wife Years), where 1 QAWY is an additional year of happy marriage.</span></i></p><p><span style="font-family: inherit;">I once compiled <a href="https://twitter.com/RichardMCNgo/status/1463976716305514502?s=20&t=hYLw7z6XAi52XgQ7PJ5MrQ">a list of concepts</a> which I’d discovered were much less well-defined than I originally thought. I’m sad to say that I now have to add <a href="https://colah.github.io/personal/micromarriages/">Chris Olah’s micromarriages</a> to the list. In his words: “Micromarriages are essentially <a href="https://en.wikipedia.org/wiki/Micromort">micromorts</a>, but for marriage instead of death. A micromarriage is a one in a million chance that an action will lead to you getting married, relative to your default policy.”</span></p><p><span style="font-family: inherit;">It’s a fun idea, and helpful in making small probabilities feel more compelling. But upon thinking about it more, I’ve realised that the analogy doesn’t quite work. The key difference is that micromorts are a measure of <i>acute risk</i> - i.e. immediate death. For activities like skydiving, this is the main thing to worry about, so it’s a pretty good metric. But most actions we’d like to measure using micromarriages (going to a party, say, or working out more) won’t lead you to get married immediately - instead they flow through to affect marriages that might happen at some later point.</span></p><span id="docs-internal-guid-704c6879-7fff-4a37-5e02-32bdf2f2b02d" style="font-family: inherit;"><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">So how can we measure the extent to which an action affects your future marriages, even in theory? One option is to track how it changes the likelihood you’ll get married </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">eventually</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. But this is pretty unhelpful. By analogy, if micromorts measured an action’s effect on the probability that you’d die </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">eventually</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, then all actions would have almost zero micromorts (with the possible exception of some life-extension work during the last few decades). Similarly, under this definition the micromarriages you gain from starting a new relationship could be mostly cancelled out by the fact that this relationship cuts off other potential relationships.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">An alternative is to measure actions not by how much they change the probability that you’ll get married eventually, but by how much you expect them to causally contribute to an eventual marriage. The problem there is that many actions can causally contribute to a marriage (meeting someone, asking them out, proposing, etc) and there’s no principled way of splitting the credit between them. I won’t go into the details here, but the basic problem is the same as one which arises when </span><a href="https://forum.effectivealtruism.org/posts/fnBnEiwged7y5vQFf/triple-counting-impact-in-ea" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">trying to allocate credit to multiple contributors to a charitable intervention</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. E.g. if three different funders are all necessary for getting a project off the ground, in some sense they can all say that they “caused” the project to happen, but that would end up triple-counting their total impact. (In this case, we can use </span><a href="https://forum.effectivealtruism.org/posts/XHZJ9i7QBtAJZ6byW/shapley-values-better-than-counterfactuals" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Shapley values</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> to allocate credit - but the boundaries between different “actions” are much more arbitrary than the boundaries between different “agents”, making it harder to apply Shapley values to the micromarriage case. Should we count the action “skipping meeting someone else” as a contributor to the marriage? Or the action “turning your head to catch sight of them”? This is basically a rabbit-hole without end - and that’s not even getting into issues of marriage </span><a href="https://plato.stanford.edu/entries/identity-transworld/" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">identity across possible worlds</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">.)[1]</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Fortunately, however, there’s another approach which does work. When thinking about mortality, the medical establishment doesn’t just measure acute risks, but also another category of risk: </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">chronic risks</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, like smoking. When smoking, you don’t get a binary outcome after each cigarette, but rather a continual degradation of health. So chronic risks are instead measured in terms of the expected decrease in your lifespan - for example, with units of </span><a href="https://en.wikipedia.org/wiki/Microlife" style="text-decoration-line: none;"><span style="color: #1155cc; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">microlives</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, where one microlife is one millionth of an adult lifespan (about half an hour); or with quality-adjusted life years (QALYs), to adjust for ill health and disability.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Analogously, then, the most straightforward metric for guiding our romantic choices is the expected increase in the time you’ll spend married - which we could measure in microwives, where one microwife is an additional half-hour of marriage. But I don’t think this is the best unit, because most people could accumulate many more microwives by dropping their standards, even if that’ll lead to unhappy marriages. So it’s important to adjust for how good we expect the marriage to be! My proposed unit: quality-adjusted wife years (QAWYs). Note that these are gender-neutral: QAWYs can involve either </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">being </span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">a wife or </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">having </span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">a wife (or both) [2].</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> An intervention gains 1 QAWY if it increases the expected amount of time you’ll spend happily married by 1 year (or the amount of time you’ll spend in a half-as-happy marriage by 2 years, etc). We do need some benchmark for a “happy marriage”; I’ll arbitrarily pick the 90th percentile of marriages across the population. Some factors which affect QAWY evaluation include spouse compatibility, age of marriage, diminishing marginal utility, having children [3], and divorce probability. Conveniently, QAWYs don’t require the assumption of lifelong marriage - they can naturally account for the possibility of multiple consecutive (or even concurrent) marriages.</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> With QAWY’s combination of theoretical elegance and pragmatic relevance, I look forward to their widespread adoption.</span></p><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><i>(To borrow a disclaimer from Chris' original post: seriously? Nope. I'm about 90% joking. I do think the general idea can sometimes be helpful, though.)</i>
</span></p><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">1. </span><span style="font-family: inherit; white-space: pre-wrap;">Alongside QAWYs, some version of micromarriages may still be useful - we just need to adjust them to measure an acute one-off event rather than a continuing chronic contributor to marriage. The most natural one is probably to think of a micromarriage as a one-in-a-million chance of first meeting your future spouse at a given event.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">2. </span><span style="font-family: inherit;">Unfortunately, this is still not fully inclusive. In formal contexts please use Quality-Adjusted Wedded Years instead.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: inherit;">3. </span><span style="font-family: inherit;">The likelihood of which should of course be measured in units of microchildren (but not microkids, which I'm reserving for a very small chance of a very small joke, like this one).</span></p></span>Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-78557237713073970752022-02-25T07:17:00.007-08:002022-02-27T14:03:52.712-08:00My attitude towards death<p><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The philosophy and psychology of death seem weirdly under-discussed - particularly by the wider silicon valley community, given how strongly anti-death many people in it are. This post is an attempt to think through some relevant considerations, primarily focused on my own intuitions and emotions. See also this </span><a href="https://thinkingcomplete.blogspot.com/2018/01/is-death-bad.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">old blog post</span></a><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> - I mostly still agree with the points I made in it, but when thinking about it now I frame things pretty differently.</span></p><span id="docs-internal-guid-87e5eec8-7fff-f599-15e6-5cc1644720c2"><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: left;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: medium;">Fearing death, loving life</span></span></div><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Let’s first distinguish two broad types of reasons for wanting to avoid death: fearing death, and loving life.* Perhaps these seem like two sides of the same coin - but, psychologically speaking, they feel very distinct to me. The former was particularly dominant when I was in primary school, when a part of me emerged that was very afraid of death (in a way that wasn’t closely linked to fear of missing out on any particular aspects of life). That part is still with me - but when it comes to the surface, its fear feels viscerally unpleasant, so I learned to suppress it pretty strongly.</span></p><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><br /></p><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Arguments for why death is bad usually focus on positive reasons - living longer allows people to experience more happiness, and more of the other good things in life. These have resonated with me more over time, as I started to think about death on a more intellectual level. However, one difficulty with these arguments is that many parts of me pursue goals in a fairly myopic way which doesn’t easily extrapolate to centuries, millennia, or longer. For example, it’s hard to imagine what career success or social success look like on the scale of millennia - and even when I try, those visions are pretty different from the versions of those concepts that I currently find motivating on a gut level. Extrapolating hedonistic goals is easier in some ways (it’s easy to imagine being happy for a very long time) but harder in other ways (the parts of me which care most about happiness are also the most myopic).</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: medium; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Dissolving fear</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In practice, then, most of my motivation for avoiding death in the long term stems from fear of death. Although that fear comes out only rarely, I have a strong heuristic that fear-based motivation should be transmuted to other forms of motivation wherever possible. So what would happen if I talked more to the part that’s scared of death, to try and figure out where it’s coming from? By default, I expect it’d be uncooperative - it wants to continue being scared of death, to make sure that I act appropriately (e.g. that I stay ambitious). Can I assure it that I’ll still try hard to avoid death if it becomes less scared? One source of assurance is if I’m very excited about a very long life - which I am, because </span><a href="http://thinkingcomplete.blogspot.com/2020/01/characterising-utopia.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">the future could be amazing</span></a><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. Another comes from the altruistic part of me, whose primary focus is increasing the probability that the future will in fact be amazing. Since I believe that we face significant</span><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> existential risk this century**, working to make humanity’s future go well overlaps heavily with working to make my own future go well. I think this broad argument (along with being in communities which reward longtermist altruism) has helped make the part of me that’s scared of death more quiescent.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Indeed, probably my main concern with my current attitude towards death is actually that I’m not scared enough about existential risk - I think that, if my emotions better matched my credences, that’d help motivate me (especially to pursue particularly unusual or ambitious interventions). This doesn’t seem like a crucial priority, though, since my excitement- and interest-based motivations have been working fairly well so far (modulo some other productivity gaps which seem pretty orthogonal).</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: medium; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Generalising to others</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">So far I’ve talked primarily about my own experience. I’m curious about how well this generalises to other people. It seems like fear of death is a near-universal emotion (it’s striking that the </span><a href="https://en.wikipedia.org/wiki/Epic_of_Gilgamesh" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">first recorded story we have</span></a><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> is about striving to escape death), but my guess is that most people have it much less strongly than I did.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Since most people aren’t very openly concerned with avoiding death in the long term, I feel uncertain about the extent to which they’ve suppressed versus dissolved that fear. My guess is that in western societies most people have mainly suppressed it, and that the hostility they often show to longevity research or cryonics is a psychological defense mechanism. If so, then overcoming those defense mechanisms to convince people that death is not inevitable might unlock a lot of suppressed excitement about the future. However, I’m wary of assuming that other people are too similar to me - perhaps other people’s fear of death is just more myopic than mine.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">There also seem to be some people who started off with a long-term fear of death, then dissolved it, usually by significantly changing their conception of personal identity - via meditation, or drugs, or philosophical argument. The big question is whether this change is more like an empirical update, or more like a value shift (to be clear, I don’t think that there’s a bright line between the two - but something can be much more like one or the other). If the former, then perhaps fear of death is just a “mistake” that many people make. Whereas the latter suggests that death is really bad according to some people’s values, and mostly fine for others, even though they may in other ways be psychologically similar. Both of these conclusions seem a bit weird; let’s try to get a bit more clarity by digging into arguments about personal identity now.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: medium; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Continuity of self</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The core question is how much we should buy into the folk view of personal identity - the view that there’s a single “thread” of experience which constitutes my self, where I survive if that thread continues and “die” if it breaks. I consider thought experiments about duplicates to provide strong evidence against this position - it seems very compelling to me that, when two identical copies of myself are created, there is no fact of the matter about which one is “really me”. Insofar as many people have intuitions weighing the other way, that’s probably because we evolved in an environment where identical duplication didn’t happen. In a future where duplication exists, and we continue being subject to evolution, I can easily imagine the mental concept of survival-of-self being straightforwardly replaced by the concept of survival-of-a-copy-of-myself.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The main alternative to caring about continuity is caring about level of similarity - identifying with a successor if they are sufficiently psychologically similar to you. This might leave you identifying with many successors, or ones that are very disconnected from you in time or space. However, it’s also consistent with identifying only with successors with a level of similarity that, in practice, will only be achievable by copying or uploading you (although I expect that really buying into the similarity theory of personal identity will make most people more altruistic, </span><a href="https://www.3ammagazine.com/3am/derek-parfitt-rip-1942-2017/" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">like it did for Parfit</span></a><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">).***</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The strongest argument in favour of the folk view arises when considering large universes, like quantum multiverses or spatially infinite universes. In a quantum multiverse there are many copies of myself, and I tend to experience being the ones with more measure. But what does that even mean? If I expect that N slightly different copies of myself will branch off soon, and all of them will have the experience of being me, how can I anticipate being more likely to “find myself” as a given one of them? There's something here which I don't understand, and which makes me hesitant to fully dismiss the idea of a thread of experiences (a confusion which Yudkowsky explores in </span><a href="https://www.lesswrong.com/posts/WajiC3YWeJutyAXTn/where-physics-meets-experience" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">these</span></a><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span><a href="https://www.lesswrong.com/posts/WajiC3YWeJutyAXTn/where-physics-meets-experience" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">two</span></a><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> posts). I think the appropriate response is to be cautious until we understand this better - for instance, I would currently strongly prefer being non-destructively rather than destructively uploaded.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: medium; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Generalising to society</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">When we stop thinking on an individual level and start thinking on a societal level, many more pragmatic considerations arise - especially related to how widespread longevity might shift the overall balance of power in the world. I do think these are important; here, though, I want to focus on a couple of broader philosophical considerations.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I previously talked about the part of myself which wants to make the future amazing. Partly that stems from </span><a href="http://thinkingcomplete.blogspot.com/2020/01/characterising-utopia.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">imagining all the different ways</span></a><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> in which the world might dramatically improve, including defeating death. Partly it’s an aesthetic preference about the trajectory of humanity - I want us to flourish in an analogous way to how I want to live a flourishing life myself. But there’s also a significant utilitarian motivation - which is relevant here because utilitarianism doesn’t care about death for its own sake, as long as the dead are replaced by new people with equal welfare. Indeed, if our lives have diminishing marginal value over time (which seems hard to dispute if you’re taking our own preferences into account at all), and humanity can only support a fixed population size, utilitarianism actively prefers that older people die and are replaced.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Now, I don’t think we’ll hit a “fixed population size” constraint until well after we’re posthuman, so this is a pretty abstract consideration. By that point, hopefully we won’t need to bite any bullets - we could build a flourishing civilisation which extrapolates our more human-specific values as well as possible, and also separately build the best utilitarian civilisation (assuming we can ensure non-conflict between them). But I’m also open to the idea that the future will look sufficiently weird that many of the concepts I’ve been using break down. For example, the boundaries between different minds could blur to such an extent that talking about the deaths of individuals doesn’t make much sense any more. I find it hard to viscerally desire that for myself, and I expect that most people alive today are much less open to the possibility than I am, but I can imagine changing my mind as we come to understand much more about how minds and values work.</span></p><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><br /></p><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><br /></p><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: x-small;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">* </span></span><span style="font-family: Arial; font-size: x-small;"><span style="white-space: pre-wrap;">Upon reflection, I might also add a third distinct motivation - the celebration of immortality. I get this feeling particularly when I read fiction with very long-lived characters. But since it's much weaker than the other two, I won't discuss it further.</span></span></p><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: x-small;"><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">** </span><span style="font-family: Arial;"><span style="white-space: pre-wrap;">At least double digit percentage points, although my specific estimate is pretty unstable.</span></span></span></p><div style="margin-bottom: 0pt; margin-top: 0pt; text-align: left;"><span style="font-size: x-small;"><span style="font-family: Arial;"><span style="white-space: pre-wrap;">*** On a side note: I</span></span><span style="font-family: Arial; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span> feel very uncertain about how much information about my brain (in the form of my blog posts, tweets, background information about my life, etc) would be sufficient for future superintelligences to recreate me in a way that I’d consider a copy of myself. I haven’t even seen any rough bounds on this - maybe worth looking into.</span></span></span></div></span>Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-66869460927639829912022-02-14T00:46:00.003-08:002022-02-14T19:42:16.396-08:00Some limitations of reductionism about epistemology<p><span style="font-family: inherit;"><span style="font-style: italic; white-space: pre-wrap;">This post is largely based on a lightning talk I gave at a <a href="https://www.deugenesis.com/">Genesis</a> event on metacognition, with some editing to clarify and expand on the arguments. </span></span></p><span id="docs-internal-guid-47a5658f-7fff-248a-58b8-5a7c7fae9353"><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="white-space: pre-wrap;">Reductionism is the strategy of breaking things down into smaller pieces, then trying to understand those smaller pieces and how they fit together into larger pieces. It’s been an excellent strategy for physics, for most of science, for most of human knowledge. But my claim is that, when the thing we're trying to understand is how to think, being overly reductionist has often led people astray, particularly in academic epistemology.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">I’ll give three examples of what goes wrong when you try to be reductionist about epistemology. Firstly, we often think of knowledge in terms of sentences or propositions with definite truth-values - for example, “my car is parked on the street outside”. Philosophers have debated extensively what it means to know that such a claim is true; I think the best answer is the bayesian one, where we assign credences to propositions based on our evidence. Let’s say I have 90% credence that my car is parked on the street outside, based on leaving it there earlier - and let’s assume it is in fact still there. Then whether we count this as “knowledge” or not is mainly a question about what threshold we should use for the definition of “knows” (one which will probably change significantly depending on the context).</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">But although bayesianism makes the notion of knowledge less binary, it still relies too much on a binary notion of truth and falsehood. To elaborate, let’s focus on philosophy of science for a bit. Could someone give me a probability estimate that Darwin’s theory of evolution is true? [Audience answer: 97%] Okay, but what if I told you that Darwin didn’t know anything about genetics, or the actual mechanisms by which traits are passed down? So I think that 97% points in the right direction, but I think it’s less that the theory has a 97% chance of being totally true, and more like a 97% chance of being something like 97% true. If you break down all the things Darwin said into a list of propositions: animals inherit from their parents, and 100 different things - almost certainly at least one of these is false. That doesn’t change the fact that overall, the theory is very close to true (even though we really have no idea how to measure or quantify that closeness).</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">I don’t think this is a particularly controversial or novel claim. But it’s surprising that standard accounts of bayesianism don’t even try to account for approximate truth. And I think that’s because people have often been very reductionist in trying to understand knowledge by looking at the simplest individual cases, of single propositions with few ambiguities or edge cases. By contrast, when you start looking into philosophy of science, and how theories like Newtonian gravity can be very powerful and accurate approximations to an underlying truth that looks very different, the notion of binary truthhood and falsehood becomes much less relevant.</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">Second example: Hume’s problem of induction. Say you’re playing billiards, and you hit a ball towards another ball. You expect them to bounce off each other. But how do you know that they won’t pass straight through each other, or both shoot through the roof? The standard answer: we’ve seen this happen many times before, and we expect that things will stay roughly the same. But Hume says that this falls short of a deductive argument, it’s just an extrapolation. Since then, philosophers have debated the problem extensively. But they’ve done so in a reductionist way which focuses on the wrong things. The question of whether an individual ball will bounce off another ball is actually a question about our whole systems of knowledge: I believe the balls will bounce off each other because I believe they’re made out of atoms, and I have some beliefs about how atoms repel each other. I believe the balls won’t shoot through the roof due to my beliefs about gravity. If we try to imagine the balls not bouncing off each other, you have to imagine a whole revolution in our scientific understanding.</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: inherit;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Now, Hume could raise the same objection in response: why can’t we imagine that physics has a special exception in this one case, or maybe that the fundamental constants fluctuate over time? If you push the skepticism that far, I don’t think we have any bulletproof response to it - but that’s true for basically all types of skepticism. Yet, nevertheless, thinking about doing induction in relation to models of the wider world, rather than individual regularities, is a significant step forward. For example, it clears up Nelson Goodman’s confusion about his </span><a href="https://en.wikipedia.org/wiki/New_riddle_of_induction" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">New Riddle of Induction</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. Broadly speaking, the New Riddle asks: why shouldn’t we do induction on weird “gerrymandered” concepts instead of our standard ones? For any individual concept, that’s hard to answer - but when you start to think in a more systematic way, it becomes clearer that trying to create a model of the world in terms of gerrymandered concepts is hugely complex.</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: inherit;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Third example: in the history of AI, one of the big problems that people have faced is the problem of symbol grounding: what does it mean for one representation in my AI to correspond to the real world. What does it mean for an AI to have a concept of a car - what makes the internal variable in my AI map to cars in the real world? Another example comes from neuroscience - you may have heard of </span><a href="https://qz.com/740481/the-jennifer-aniston-neuron-is-the-foundation-of-compelling-new-memory-research/" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Jennifer Aniston neurons</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, which fire when they recognise a single person, across a range of modalities. How does this symbolic representation in your brain relate to the real world?</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">The history of AI is the history of people trying to solve this from the ground up. Start with a few concepts, add some more to them, branch out, do a search through them, etc. This research program, known as symbolic AI, failed pretty badly. And we can see why when we think more holistically. The reason that a neuron in my brain represents my grandmother has nothing to do with that neuron itself, it’s because it’s connected to my arms which make me reach out and hug her when I see her, and the speech centers in my brain which remind me of her name when I talk about her, and the rest of my brain which brings up memories when I think of her. These aren’t things you can figure out by looking at the individual case, nor is it something you can design into the system on a step by step basis, as AI researchers used to try to do.</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">So these are three cases where, I claim, people have been reductionist about epistemology when they should instead have taken a much more systems-focused approach. </span></span></p></span>Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-41257470702210164342022-02-14T00:00:00.000-08:002022-02-14T00:00:13.356-08:00Strevens on scientific explanation<p><span style="font-family: inherit;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This post discusses two books by Michael Strevens: </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Knowledge Machine: How Irrationality Created Modern Science </span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">and </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Thinking Off Your Feet: How Empirical Psychology Vindicates Armchair Philosophy</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. I loved the former, which tries to answer the core question in philosophy of science: why does science work so well? It’s a masterful synthesis of previous ideas and important new arguments, and it’ll be my go-to recommendation from now on for people interested in the field. The latter was… slightly less persuasive. But let’s start with the good stuff.</span></span></p><span id="docs-internal-guid-34b091dc-7fff-90c1-7303-6c417cb918a1"><span style="font-family: inherit;"><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Knowledge Machine</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> begins with a review of two of the key figures in philosophy of science: Popper and Quine. Historically, philosophers of science focused on identifying a “scientific method”: a specific way of generating theories, designing experiments, and evaluating evidence which, when followed, led scientists to the truth. Popper’s influential account of the scientific method focused on scientists trying to </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">refute</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> their hypotheses. He claimed that only severe tests which attempt to falsify a hypothesis can give us reason to provisionally accept it. Along with other early philosophers of science, Popper’s work promoted (according to Strevens) “an ideal of the scientist as a paragon of intellectual honesty, standing up for truth in the face of stifling opposition from the prevailing politics, culture, and ideology”.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">However, over the last half-century or so a range of criticisms of this flattering view of science have emerged. Most prominent is Kuhn, who in his book </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Structure of Scientific Revolutions</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> characterises scientists as constrained within a specific paradigm of thinking, unable to rationally decide between different paradigms. Soon afterwards, in his book </span><a href="http://thinkingcomplete.blogspot.com/2019/04/book-review-sleepwalkers.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">The Sleepwalkers</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, Koestler described the emergence of early science not as the triumph of a superior method, but rather as “a history of collective obsessions and controlled schizophrenias”. More recently, Feyerabend’s book </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Against Method</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> espoused a position he called “epistemological anarchism”, arguing that “anything goes” in the pursuit of truth.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Strevens focuses on Kuhn, but his arguments are in line with the positions of the others. He summarises a range of case studies of scientists ignoring inconvenient data, deciding questions via political maneuvering, and generally behaving in ways no “scientific method” would endorse. These case studies also reiterate a point made by Quine: that a theory can never be fully falsified, since it’s always possible to argue that it’s consistent with new evidence - e.g. by tacking on new parameters, or appealing to experimental mistakes. This line of thinking provides a useful counterbalance to earlier idolisation of the scientific method, but in doing so it reopens the core question of philosophy of science: if the scientific method isn’t what makes science work so well, then what does?</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Strevens’ core idea is to strip down scientific methodology to the bare basics. Instead of trying to understand the success of science as being due to a shared methodology for generating theories, or for designing experiments, or for interpreting evidence, or for rejecting theories, we should understand it as being due to a </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">shared</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">norm about what types of evidence to accept</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. He calls it the Iron Rule of Explanation: </span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">resolve disagreements via empirical tests</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. The law results in a strict separation between “private” and “public” science: scientists in private proceed on the basis of hunches, aesthetic intuitions, rivalries, visions from god, or whatever other motivations they like. But in public, these are all sterilised: only empirical tests count. This forces scientists to search more and more deeply for key empirical data, rather than trying to build castles of arguments which aren’t ever tested - and which can therefore be washed away even after millennia of work, as the example of theology shows. In a particularly evocative passage, Strevens portrays science as a communal process for producing solid facts:</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Science, then, is built up like a coral reef. Individual scientists are the polyps, secreting a shelly carapace that they bequeath to the reef upon their departure. That carapace is the sterilized public record of their research, a compilation of observation or experimentation and the explanatory derivation, where possible, of the data from known theories and auxiliary assumptions. The scientist, like a polyp, is a complete living thing, all too human in just the ways that the historians and sociologists of science have described. When the organism goes, however, its humanity goes with it. What is left is the evidential exoskeleton of a scientific career.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Strevens dates the birth of the iron rule to Newton, and in particular the famous passage where he says that he will “feign no hypotheses” about why the laws of gravity work the way they do. Newton thereby accepts a shallower and more instrumental conception of “explanation” than previous scientists, who searched for theories built on mechanisms that made intuitive sense. Strevens claims that the counterintuitive nature of the iron rule is why it took so long for science to get started. Shouldn’t the lack of a mechanism which implements Newtonian gravity be a significant strike against it? Shouldn’t the intuitions which led us to a theory be a key part of our arguments for it? And why can’t our other beliefs - say, about the existence of god - help inform our scientific theories?</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Strevens agrees that excluding information which we believe is relevant is, in a sense, irrational (hence the book’s subtitle: </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">“how irrationality gave birth to modern science”</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">). But he argues that it’s necessary for the success of science, because it pushes scientists towards doing the difficult work required to find the truth:</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">We live in a Tychonic world - a world in which great competing stories about the underlying nature of things can be distinguished by, and only by, scrutinizing subtle intricacies and minute differences. Humans in their natural state are not much disposed to attend to such trifles. But they love to win. The procedural consensus imposed by the iron rule creates a dramatic contest within which the trifles acquire an unnatural luster, becoming, for their tactical worth, objects of fierce desire. The rule in this way redirects great quantities of energy that might have gone toward philosophical or other forms of argument into empirical testing. Modern science’s human raw material is molded into a strike force of unnervingly single-minded observers, measurers, and experimenters, generating a vast, detailed, varied, discriminating stock of evidence.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I think this explanation points in the right direction - but it’s incomplete. Why do we need the iron rule to create a dramatic contest, rather than just competing to find any type of compelling evidence? It’s true that early thinkers didn’t understand that we lived in a Tychonic world, and so underrated empirical evidence. But after seeing many examples of the power of empirical evidence (and more specifically, the power of advance predictions), why wouldn’t they update towards empirical evidence being a powerful way to identify the truth, without enshrining it as the </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">only</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> way to identify the truth? In other words, Strevens’ proposal of science-as-competition works almost as well without the iron rule, as long as scientists reward progress towards truth in an unbiased way.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">So a complete version of Strevens’ explanation needs to identify why scientists will predictably overrate non-empirical evidence for theories, and reward that evidence more than it deserves. There may be a range of sociological considerations in play - for example, if observers tend to underestimate how much work has gone into finding evidence, then the reputational payoff for doing the hardest empirical work might be disproportionately low, meaning that scientists will focus on other ways to win the game. But for now I want to focus on the hypothesis that we find non-empirical arguments more persuasive than we should because of a range of cognitive biases. To illustrate this point, let’s dig into Strevens’ previous book - a more philosophical work named </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Thinking off your feet: how empirical psychology vindicates armchair philosophy</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">.</span></p><h3 dir="ltr" style="margin-bottom: 4pt; margin-top: 16pt;"><span style="color: #434343; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">The perils of philosophy</span></h3><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Having been so impressed by his latest book, I was surprised by how much I disagreed with this one. I ended up only skimming through most chapters, so take this summary with a pinch of salt, but my impression of Strevens’ case was as follows. The standard mode of inquiry in philosophy, known as conceptual analysis, aims to discover the “essential natures” of things by consulting our intuitions, especially intuitions about complex edge cases. Conceptual analysis has come under fire over the last few decades from skeptics who point out that almost no concepts can be characterised in terms of necessary and sufficient conditions - most of them are inherently vague or imprecise. Strevens agrees with this point. Nevertheless, he claims, conceptual analysis is still useful because the process of trying to identify essential natures helps us understand even entities which lack them.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">What’s bizarre is that Strevens sees so clearly the difficulty of science - which forces us to adopt the strict restriction of the iron rule - yet still thinks that philosophy can make progress basically by accident, by aiming at the wrong target entirely. Perhaps this would be compelling if there were historical examples of this working well, but the ones Strevens identifies are underwhelming, to say the least. Consider, for example, the thought experiment of a swan which spontaneously appears due to random particle fluctuations. Strevens claims that arguing about whether this is “really” a swan helps us understand the “causal-explanatory structure” of normal swans - e.g. the ways in which their properties are explained by their ancestry. To be honest, my main response here is an incredulous stare. I have no idea what valuable knowledge about swans biologists lack, which this type of philosophising has provided, or could ever provide. And I don’t think that’s a coincidence - these types of thought experiments are usually designed to steer as far clear from any empirical uncertainties as possible (and sometimes further), to make the conceptual debate clearer.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Or consider Strevens’ analysis of the concepts of belief and desire in everyday psychology. He argues that conceptual analysis is valuable in this case because this approach to psychology is “all or nothing”: in the face of empirical investigation, concepts like belief and desire “either stand firm or suffer a catastrophic collapse”. To me this seems totally wrongheaded - our understanding of belief and desire has undergone extensive shifts as we’ve gradually learned more about things like the subconscious, behavioural reinforcement, addiction, prospect theory, dual process theory, signalling theory, evolutionary psychology, and so on. By contrast, conceptual analysis of belief has been stuck in an </span><a href="https://fragile-credences.github.io/conceptual/#knowledge" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">unproductive merry-go-round of definitions and counterexamples</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> for decades.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This is not to say that there has been no progress in philosophy - see, for instance, Tom Adamczewski’s list of </span><a href="https://fragile-credences.github.io/ps/" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">philosophy success stories</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. But it seems like Strevens, and many other philosophers, dramatically overestimate how useful philosophy has been. I claim that this is because common cognitive biases (like the bias towards </span><a href="https://www.lesswrong.com/posts/yA4gF5KrboK2m2Xu7/how-an-algorithm-feels-from-inside" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">essentialism</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, and </span><a href="https://en.wikipedia.org/wiki/Confirmation_bias" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">confirmation bias</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, and </span><a href="https://www.lesswrong.com/posts/WnheMGAka4fL99eae/hindsight-devalues-science" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">hindsight bias</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">) make philosophical arguments seem more insightful than they actually are. And if these biases are common even amongst the brightest thinkers, it answers the question I posed above about why the iron rule is still necessary. By ruling out these types of arguments, the iron rule doesn’t just steer us towards useful research, it also protects us from cognitive biases which make conceptual arguments seem disproportionately valuable.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I don’t want to point the finger only at philosophy; I think many other humanities and social sciences have similar problems. But as one of the fields which makes least use of empirical data, philosophy is a particularly easy clear illustration of my core claim: </span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">science succeeds because the iron rule of explanation (“resolve disagreements via empirical tests”) mitigates cognitive and sociological biases in our judgments of the strengths of different types of evidence</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">.</span></p><br /><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">There’s much more room to elaborate on the specific types of biases involved; I do that to some extent </span><a href="http://thinkingcomplete.blogspot.com/2019/06/on-alien-science.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">in this blog post</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, and it’s also a </span><a href="https://www.lesswrong.com/s/zpCiuR4T343j9WkcK/p/WnheMGAka4fL99eae" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">core</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span><a href="https://www.lesswrong.com/posts/Yq6aA4M3JKWaQepPJ/burdensome-details" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">theme</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> of </span><a href="https://intelligence.org/rationality-ai-zombies/" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Yudkowsky’s writings</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> (along with </span><a href="https://www.lesswrong.com/s/fxynfGCSHpY4FmBZy/p/xTyuQ3cgsPjifr7oj" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">how to do better than the standard scientific process</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">). But one point to note is that this formulation assumes that human reasoning is actually pretty good in general - that, if we get rid of these biases, we’re capable of thinking in a broadly reliable way about domains that are very far removed from our everyday experience. So in some sense, an explanation for why science succeeds needs to also be a story about human intelligence, and the mental models which we can build using it. But I’ll save that for another post.</span></span></span>Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-602254451657562442022-02-13T23:48:00.008-08:002022-02-13T23:49:58.468-08:00Book review: The Age of Surveillance Capitalism<p><span style="font-family: inherit;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I recently finished Shoshana Zuboff’s book </span><a href="https://en.wikipedia.org/wiki/The_Age_of_Surveillance_Capitalism" style="text-decoration-line: none;"><span style="color: #1155cc; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">The Age of Surveillance Capitalism</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. It’s received glowing reviews, but left me disappointed. Zuboff spends much of the book outraged at the behaviour of big tech corporations, but often neglects to explain what’s actually bad about either the behaviour itself or the outcomes she warns it’ll lead to. The result is far more polemical than persuasive. I do believe that there are significant problems with the technology industry - but mostly different problems from the ones she focuses on. And she neglects to account for the benefits of technology, or explain how we should weigh them against the harms.</span></span></p><span id="docs-internal-guid-8c721a56-7fff-734a-6b9d-2c7c38c4da9f"><span style="font-family: inherit;"><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Her argument proceeds in three stages, which I’ll address in turn:</span></p><ol style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Companies like Google and Facebook have an “extraction imperative” to continually “expropriate” more personal data about their users.</span></p></li><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">They use this for “the instrumentation and instrumentalisation of behaviour for the purposes of modification, prediction, monetisation, and control.”</span></p></li><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Ultimately, this will lead to “a form of tyranny” comparable to (but quite different from) totalitarianism, which Zuboff calls instrumentarianism.</span></p></li></ol><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">On data: I agree that big companies collect a lot of data about their users. That’s a well-known fact. In return, those users get access to a wide variety of high-quality software for free. I, for one, would pay thousands of dollars if necessary to continue using the digital products that are currently free because they’re funded by advertising. So what makes the collection of my data “extraction”, or “appropriation”, as opposed to a fair exchange? Why does it “abandon long-standing organic reciprocities with people”? It’s hard to say. Here’s Zuboff’s explanation:</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Industrial capitalism transformed nature’s raw materials into commodities, and surveillance capitalism lays its claims to the stuff of human nature for a new commodity invention. Now it is human nature that is scraped, torn, and taken for another century’s market project. It is obscene to suppose that this harm can be reduced to the obvious fact that users receive no fee for the raw material they supply. That critique is a feat of misdirection that would use a pricing mechanism to institutionalise and therefore legitimate the extraction of human behaviour for manufacturing and sale. It ignores the key point that the essence of the exploitation here is the rendering of our lives as behavioural data for the sake of others’ improved control over us. The remarkable questions here concern the facts that our lives are rendered as behavioural data in the first place; that ignorance is a condition of this ubiquitous rendition; that decision rights vanish before one even knows that there is a decision to make; that there are consequences to this diminishment of rights that we can neither see nor tell; that there is no exit, no voice, and no loyalty, only helplessness, resignation, and psychic numbing.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This is fiery prose; but it’s not really an </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">argument</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. In more prosaic terms, websites are using my data to serve me ads which I’m more likely to click on. Often they do so by showing me products which I’m more interested in, which I actively prefer compared with seeing ads that are irrelevant to me. This form of “prediction and control” is on par with any other business “predicting and controlling” my purchases by offering me better products; there’s nothing “intrinsically exploitative” about it.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Now, there are other types of prediction and control - such as the proliferation of worryingly addictive newsfeeds and games. But surprisingly, Zuboff talks very little about the harmful consequences of online addiction! Instead she argues that the behaviour of tech companies is wrong for intrinsic reasons. She argues that “there is no freedom without uncertainty” and that predicting our behaviour violates our “right to the future tense” - again taking personalised advertising as her central example. But the degree of personalised prediction is fundamentally the wrong metric to focus on. Some of the products which predict our personal behaviour in the greatest detail - sleep trackers, or biometric trackers - allow us to exercise more control over our own lives, increasing our effective freedom. Whereas many of the addictive games and products which most undermine our control over our lives actually rely very little on personal data - as one example, the </span><a href="https://www.decisionproblem.com/paperclips/" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Universal Paperclips</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> game is incredibly addictive without even having graphics, let alone personalised algorithms. And the “slot machine”-style intermittent rewards used by mobile apps like Facebook again don’t require much personalisation.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">It’s true that personalisation can be used to enhance these problems - I’m thinking in particular of TikTok, whose recommendation algorithms are scarily powerful. But there’s also a case to be made that this will become better over time. Simple metrics, like number of views, or number of likes, are easy for companies to optimise for. Whereas figuring out </span><a href="https://forum.effectivealtruism.org/posts/xzjQvqDYahigHcwgQ/aligning-recommender-systems-as-cause-area" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">how to optimise for what people really want</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> is a trickier problem. So it’s not surprising if companies haven’t figured it out yet. But as they do, users will favour the products that give them the best experience (as one example, I really like the premise of the </span><a href="https://www.vogue.com/article/could-dispo-make-social-media-fun-again" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Dispo app</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">). Whether or not those products use personal data is much less important than whether they are beneficial or harmful for their users.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Lastly, we come to the question of longer-term risks. What is Zuboff most worried about? She holds up the example of Skinner’s novel </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Walden Two</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, in which behavioural control is used to teach children better self-control and other virtuous behaviour. Her term for a society in which such tools are widely used is “instrumentarian”. This argument is a little strange from the beginning, given that Walden Two was intended as (and usually interpreted as) a utopia, not a dystopia. The idea that technology can help us become better versions of ourselves is a longstanding one; behavioural reinforcement is just one mechanism by which that might occur. I can certainly see why the idea is discomfiting, but I’d like to see an actual argument for why it’s bad - which Zuboff doesn’t provide.</span></p><br /><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Perhaps the most compelling argument against instrumentarianism from my perspective is that it paves the way for behavioural control technology to become concentrated and used to maintain political power, in particular by totalitarian regimes. But for reasons I don’t understand, Zuboff downplays this risk, arguing that “instrumentarian power is best understood as the precise antithesis of Orwell’s Big Brother”. In doing so, she holds up China as an example of where the West might be headed. Yet China is precisely a case in which surveillance has aided increasing authoritarianism, as seen most notably in the genocide of the Uighurs. Whereas, whatever the faults of big US tech companies in using data to predict consumer behaviour, they have so far stayed fairly independent from exercises of governmental power. So I’m still uncertain about what the actual harms of instrumentarianism are.</span></p><br /><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Despite this strange dialectic, I do think that Zuboff’s warnings about instrumentarianism contribute to preventing authoritarian uses of surveillance. So, given the importance of preventing surveillance-aided totalitarianism, perhaps I should support Zuboff’s arguments overall, despite my reservations about the way she makes them. But there are other reasons to be cautious about her arguments. As Zuboff identifies, human data is an important component for training AI. Unlike her, though, I don’t think this is a bad thing - if it goes well, AI development has the potential to create a huge amount of wealth and improve the lives of billions. The big question is whether it will go well. One of the key problems AI researchers face is the difficulty of specifying the behaviour we’d like our systems to carry out: the standard approach of training AIs on explicit reward functions </span><a href="https://deepmind.com/blog/article/Specification-gaming-the-flip-side-of-AI-ingenuity" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">often leads to</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> unintended misbehaviour. And the </span><a href="https://arxiv.org/abs/2109.10862" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">most</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span><a href="https://arxiv.org/abs/1810.08575" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">promising</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span><a href="https://openai.com/blog/debate/" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">techniques</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> for solving this involve harnessing human data at large scale. So it’s important not to reflexively reject the large-scale collection and use of data to train AIs - because as such systems </span><a href="https://drive.google.com/file/d/1uK7NhdSKprQKZnRjU58X7NLA1auXlWHt/view" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">become increasingly advanced</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, it’s this data which will allow us to point them in the right directions.</span></span></span>Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-41551406676154279782022-02-13T17:31:00.007-08:002022-02-13T18:37:47.910-08:00Some thoughts on vegetarianism and veganism<p><span style="white-space: pre-wrap;"><span style="font-family: inherit;">I feel pretty confused about whether I, as an effective altruist, should be vegetarian/vegan (henceforth abbreviated veg*n). I don’t think I’ve seen anyone explicitly talk about the arguments which feel most compelling to me, so I thought I’d do that here, in a low-effort way.</span></span></p><span id="docs-internal-guid-d4f95fa5-7fff-3e0e-504a-f355abb67aae"><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">I think that factory farming is one of the worst ongoing moral atrocities. But most of the arguments I’ve heard for veg*nism, which I found compelling a few years ago, hinge on the effects that my personal consumption would have on decreasing factory farming (and sometimes on climate change). I now don’t find this line of thinking persuasive - my personal consumption decisions just have such a tiny effect compared to my career/donation decisions that it feels like I shouldn’t pay much attention to their direct consequences (beyond possibly donating to offset them).</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: inherit;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">But there are three other arguments which seem more compelling. First is a deontological argument: if you think something is a moral atrocity, you shouldn’t participate in it, even if you offset the effects of your contribution. In general, my utilitarian intuitions are much stronger than my deontological ones, but I do think that following deontological principles is often a very good heuristic for behaving morally. The underlying reason is that humans by default think more naturally in terms of black-and-white categories than shades of grey. </span><a href="https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-cases" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">As Yudkowsky writes</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">:</span></span></p><p dir="ltr" style="margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: inherit;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Any rule that's not labeled "absolute, no exceptions" lacks weight in people's minds. So you have to perform that the "Don't kill" commandment is absolute and exceptionless (even though it totally isn't), because that's what it takes to get people to even hesitate. To stay their hands at least until the weight of duty is crushing them down. A rule that isn't even absolute? People just disregard that whenever.</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">Without strong rules in place it’s easy to reason our way into all sorts of behaviour. In particular, it’s easy to understimate the actual level of harm that certain actions cause - e.g. thinking of the direct effects of eating meat but ignoring the effects of normalising eating meat, or normalising “not making personal sacrifices on the basis of moral arguments”, or things like that. And so implementing rules like “never participate in moral atrocities” sends a much more compelling signal than “only participate in moral atrocities when you think that’s net-positive”. That signal helps set an example for people around you - which seems particularly important if you spend time with people who are or will become influential. But it also strengthens your own self-identity as someone who prioritises the world going well.</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">Then there’s a community-level argument about what we want EA to look like. Norms about veg*nism within the community help build a high-trust environment (since veg*nism is a costly signal), and increase internal cohesion, especially between different cause areas. At the very least, these arguments justify not serving animal products at EA conferences.</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">Lastly, there’s an argument about how I (and the EA community) are seen by wider society. Will MacAskill sometimes uses the phrase “moral entrepreneurs”, which I think gestures in the right direction: we want to be ahead of the curve, identifying and building on important trends in advance. I expect that veg*nism will become much more mainstream than it currently is; insofar as EA is a disproportionately veg*n community, this will likely bolster our moral authority.</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">I think there are a few arguments cutting the other way, though. I think one key concern is that these arguments are kinda post-hoc. It’s not necessarily that they’re wrong, it’s more like: I originally privileged the hypothesis that veg*nism is a good idea based on arguments about personal impact which I now don’t buy. And so, now that I’m thinking more about it, I’ve found a bunch of arguments which support it - but I suspect I could construct similarly compelling arguments for the beneficial consequences of a dozen other personal life choices (related to climate change, social justice, capitalism, having children, prison reform, migration reform, drug reform, etc). In other words: maybe the world is large enough that we have to set a high threshold for deontological arguments, in order to avoid being swamped by moral commitments.</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: inherit;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Secondly, on a community level, EA is the one group that is most focused on doing </span><span style="font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">really large</span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> amounts of good. And so actually doing cost-benefit analyses to figure out that most personal consumption decisions aren’t worth worrying about seems like the type of thing we want to reinforce in our community. Perhaps what’s most important to protect is this laser-focus on doing the most good without trying to optimise too hard for the approval of the rest of society - because that's how we can keep our edge, and avoid dissolving into mainstream thinking.</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">Thirdly, the question of whether going veg*n strengthens your altruistic motivations is an empirical one which I feel pretty uncertain about. There may well be a moral licensing effect where veg*ns feel (disproportionately) like they’ve done their fair share of altruistic action; or maybe parts of you will become resentful about these constraints. This probably varies a lot for different people.</span></span></p><span style="font-family: inherit;"><br /></span><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">Fourthly, I am kinda worried about health effects, especially on short-to-medium-term energy levels. I think it’s the type of thing which could probably be sorted out after a bit of experimentation - but again, from my current perspective, the choice to dedicate that experimentation to maintaining my health instead of, say, becoming more productive feels like a decision I’d only make if I were privileging the intervention of veg*nism over other things I could spend my time and effort on.</span></span></p><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;"><br /></span></span></p><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;"><span id="docs-internal-guid-0b61737f-7fff-acf4-94b9-5cbc12447e2d"></span></span></span></p><p dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I don’t really have any particular conclusion to this post; I wrote it mainly to cover a range of arguments that people might not have seen before, and also to try and give a demonstration of the type of reasoning I want to encourage in EA. (A quick search also turns up </span><a href="https://forum.effectivealtruism.org/posts/WDiBMR2eXmNKSRjvd/should-i-be-vegan" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: 400; text-decoration-skip-ink: none; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">a post by Jess Whittlestone</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> covering similar considerations.) If I had to give a recommendation, I think probably the dominant factor is how your motivational structure works, in particular whether you’ll interpret the additional moral constraint more as a positive reinforcement of your identity as an altruist, or more as something which drains or stresses you. (Note though that, since people </span><a href="https://www.elephantinthebrain.com/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: 400; text-decoration-skip-ink: none; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">systematically overestimate</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> how altruistic they are, I expect that most people will underrate the value of the former. On the other hand, effective altruists are one of the populations most strongly selected for underrating the importance of avoiding the latter.)</span></span></p></span>Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com1tag:blogger.com,1999:blog-6642084484458008587.post-42197007525948477742022-02-13T12:37:00.005-08:002023-09-30T15:53:57.452-07:00Whence the sexes?<p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;"></p><div style="text-align: left;"><span style="color: black; font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;">There are many explanations of the evolutionary value of sex in terms of gene exchange (</span><a href="https://www.lesswrong.com/posts/SAvwF9F7rjmP6oZQj/reframing-the-evolutionary-benefit-of-sex" style="font-family: inherit; text-decoration-line: none;"><span style="color: #1155cc; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space-collapse: preserve;">I particularly like this one</span></a><span style="color: black; font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;">). But these don’t explain the evolutionary value of having </span><span style="color: black; font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;"><i>sexes</i></span><span style="color: black; font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;">: of the differentiation between males and females. A species of </span><a href="https://en.wikipedia.org/wiki/Hermaphrodite" style="font-family: inherit; text-decoration-line: none;"><span style="color: #1155cc; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space-collapse: preserve;">hermaphrodites</span></a><span style="color: black; font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;"> would get all the genetic benefits of sex, but without the massive cost of half its population being unable to bear offspring. On average, each individual could have twice as many offspring, unless other problems arose. And indeed, most plants are hermaphroditic - but only a few animals. So why aren’t most animals hermaphrodites? A quick search doesn’t turn up any widely accepted answer, so I’ve brainstormed a few possibilities. I may well be missing something obvious; if so, let me know.</span></div><ol style="text-align: left;"><li><span style="color: black; font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;"><span style="font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; font-weight: 700; vertical-align: baseline;">Specialisation of labour</span><span style="font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; font-weight: 700; vertical-align: baseline;">. </span><span style="font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline;">If there’s a strong </span><a href="https://bluteblog.com/2019/09/27/the-advantages-of-specialization-can-compensate-for-the-two-fold-cost-of-sex/" style="font-family: inherit; text-decoration-line: none; text-wrap: nowrap;"><span style="color: #1155cc; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; text-decoration-line: underline; text-decoration-skip-ink: none; text-wrap: wrap; vertical-align: baseline;">division of labour</span></a><span style="font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline;"> between males and females, then maybe it’s harder for hermaphrodites to gather enough resources to support offspring. But in many species the males contribute little in terms of resources - e.g. in orang-utans, who are very solitary.</span></span></li><li><span style="font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; font-weight: 700; vertical-align: baseline; white-space-collapse: preserve;">Developmental or metabolic costs of being hermaphroditic</span><span style="font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;">. Maybe it’s just a very expensive adaptation. But this seems unlikely to be the main factor - the relevant baseline is the existence of males at all, which is a huge energy cost.</span></li><li><span style="font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline;"><b>Difficulty of evolving hermaphroditism</b></span><span style="font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline;">. Maybe this is just hard for evolution to find? But non-reproductive hermaphroditism seems like it arises via mutations pretty frequently, so I’d be surprised if reproductive hermaphroditism were unachievable by evolution. And in fact there are a few hermaphroditic species - so why haven’t they spread much more widely?</span></li><li><span style="font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; font-weight: 700; vertical-align: baseline; white-space-collapse: preserve;">Difficulty of fixating hermaphroditism</span><span style="font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;">. A hermaphrodite in a species without many hermaphrodites is likely not as attractive to females as most males are, nor as fertile as most females are. So maybe, even after arising, the trait will be selected against. But on timeframes where sexual desires can themselves evolve, all else equal we should expect stabilising sexual selection towards the best combination of fertility and attractiveness. E.g. it would be undesirable to be impregnated by overly masculine conspecifics, because the resulting offspring would be less fertile themselves. So this adds a bit more difficulty to reaching the hermaphroditic equilibrium, but doesn’t answer the core question of why that’s a less fit equilibrium.</span></li><li><span style="font-family: inherit; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space-collapse: preserve;"><b>Self-fertilisation</b>. </span><span style="font-family: inherit; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space-collapse: preserve;">I remember reading a while back that self-fertilisation has a strong short-term advantage, despite losing out on the long-term benefits of sex (</span><a href="https://onlinelibrary.wiley.com/doi/epdf/10.1111/j.1469-1809.1941.tb02272.x" style="font-family: inherit; text-decoration-line: none; white-space: pre;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; text-wrap: wrap; vertical-align: baseline;">this old paper</span></a><span style="font-family: inherit; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space-collapse: preserve;"> has a section on “selection in favour of self-fertilisation”, although I haven’t read it in detail). So maybe the answer is that hermaphrodites end up evolving ways to self-fertilise, which is harmful in the long run, and so group selection prevents the trait from becoming too widespread. This effect <a href="https://royalsocietypublishing.org/doi/10.1098/rspb.2013.0133">plausibly occurs in plants</a>, but not strongly enough to prevent most of them from being hermaphroditic. And presumably it's significantly harder for plants to prevent self-fertilisation (since their pollen spreads widely) than it is for animals.
I’m open-minded to the possibility that a combination of these explanations is responsible. But none of them seems particularly strong to me; so I'm guessing that the biggest effect comes from:
</span></li><li><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space-collapse: preserve;"><b>Physical dominance.</b> Maybe animal competition to impregnate fertile conspecifics is grounded in physical power, so that dominant males could just prevent hermaphrodites (who invest less in muscle and brawn) from having sex. In some sense this is a variant of the first possibility: the comparative advantage of muscle is just so strong that the best solution is to have a division of labour. But it focuses not on problems posed by the environment, but rather on problems posed by one’s own species. If true, it feels a bit sad: that there could be a much better solution if it weren’t for the threat of physical force. But it does seem pretty plausible to me - especially because hermaphroditism is much more common in plants, which can’t use the strategy of physical dominance.</span></li><ol><li><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space-collapse: preserve;">The main argument against it is that males in some species don’t compete via shows of force - e.g. birds which sing to attract mates. But birds are unusual in other ways too - e.g. over 90% of bird species are monogamous (as compared with less than 5% of mammals), which makes it more plausible that the "specialisation of labour" hypothesis explains their sexual differentiation. So I'd be interested in pointers to any literature on how many non-monogamous species lack physical male competition.</span></li><li><span style="font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;">Also, in </span><a href="https://news.ucsc.edu/2007/10/1610.html" style="font-family: inherit; text-decoration-line: none; white-space: pre;"><span style="color: #1155cc; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; text-decoration-line: underline; text-decoration-skip-ink: none; text-wrap: wrap; vertical-align: baseline;">some species</span></a><span style="font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;"> several different mating strategies remain in equilibrium (including strategies which involve surreptitious mating unnoticed by a dominant male). So even if physical dominance is the best strategy, is it really so much better that it can crowd out all the others?</span></li></ol></ol><div><span style="font-family: inherit; white-space-collapse: preserve;">Getting more clarity on this topic isn’t a priority for me, but I do think of the question as one small data point that might help ground big-picture abstractions about competition and cooperation (in a comparable way to how knowledge of how insect colonies work provides an interesting metaphor for thinking about society).</span><span style="font-family: inherit; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; font-weight: 700; vertical-align: baseline; white-space-collapse: preserve;"></span></div>Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-88339957978249530242022-02-12T19:04:00.021-08:002022-02-13T12:54:03.886-08:00The innocent self<span id="docs-internal-guid-a64d4fa4-7fff-3f44-b37d-cee5c411d939"><span style="font-family: inherit;"><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I loved </span><a href="https://handsandcities.com/2021/04/04/the-innocent-gene/" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Joe Carlsmith’s blog post on "the innocent gene"</span></a><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. It’s a post about perspective shifts. One of them occurs when, as Dawkins suggested, we shift from thinking about organisms as agents, and start to think of them as vehicles for their “selfish” genes. From the selfish gene perspective, a lion is armour which the genes use to propagate themselves. In making this shift, we implicitly transfer agency from the lion to the genes: we conclude that the genes “want” the lion to hunt down antelope. That’s predictively useful, but Joe observes that in doing so we often implicitly assign moral agency to the genes - as if we should judge them for the outcomes they give rise to, like we judge humans. The word “selfish”, with its loaded connotations, is a good example of this moral transfer.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">So Joe offers a third perspective: that of the innocent gene. From this perspective, genes have the property of building up lions around them - but not because that’s something they “want”, just because that’s how the world interacts with them. In a particularly evocative passage, Joe explains:</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I remembered that the genes are passive, innocent patterns, rather than agents — that they had no desire to survive, or knowledge of the snarls and claws that had accumulated around them. The lion was still armor; but armor for something that doesn’t want defending. The sleeping beauties inside it did not ask for a lion. Yet over evolutionary time, a lion has grown up around them regardless — a giant, ferocious eddy, swirling around something that snagged.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">That’s made me think about an analogous perspective shift in the way we view ourselves and others. One view - the most standard one - sees people primarily as agents deliberately pursuing their goals and ends. That’s not a bad approximation in many contexts, but when you look more closely at the emotions behind people’s behaviour, it starts to break down. We often have contradictory drives, tugging us in different directions. We can think of our behaviour as being shaped by these needs, in an analogous way to how the lion’s behaviour is shaped by its genes. But then, as with the lion, you can make the second shift - to seeing those needs not as “selfish”, but rather as innocent entities that happen to have built up ways to influence behaviour.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I can’t remember if, when I started writing this blog post last year, I already knew about the <a href="https://ifs-institute.com/about-us">Internal Family Systems framework</a>, but now that I do it feels like it fits very well to the shift I just described. The way this shift happens in IFS is by distinguishing between “protectors” and “exiles”. Protective parts are those which drive our self-interested behaviour - not for their own sake, though, but instead based on the needs of the exiles. Exiled parts are the “innocent” ones, typically portrayed as children, who just want to be taken care of. Protectors are the armour that’s accumulated around those exiles - for example, the instinct to defensively reject criticism, driven by feeling like an imposter; or the instinct to be overly clingy, driven by fear of rejection. This perspective shift doesn’t change most of the predictions you’d make about people’s behaviour: armour or agent, a lion will still eat you, and someone who’s angry will still lash out. But it reminds us that there can be something very innocent underneath - something which didn't intend the negative consequences that grew up around it. And, if the IFS framework is correct, </span><span style="font-family: inherit; white-space: pre-wrap;">something which craves only the love and compassion that everyone should rightfully receive.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">A part of me thinks that that this perspective represents the ultimate in naive optimism. We’re honed by natural selection, we’re built to kill to survive. To think that those strivings (with their hurtful effects on others) are all just different outgrowths of an underlying need for compassion sounds crazy. Perhaps it's true nevertheless - perhaps that's just how minds work. But for now, my best guess is that we shouldn't think of IFS as identifying a unique "ground truth". Instead, it's one of many frames through which we can productively engage with our minds. In some contexts - in a harsher world - this compassionate frame might be actively harmful. Perhaps it used to be more productive to think of your mind as a hierarchy, in which order and discipline are sternly imposed from the top down. Yet in the 21st century, in the developed world, </span><span style="font-family: inherit; white-space: pre-wrap;">who needs to be harsh or hard-hearted, except in response to their own unhappiness or insecurity? In general, hurt people hurt people. If so, then Scott Alexander was more right than he knew </span><a href="https://unsongbook.com/" style="font-family: inherit; white-space: pre-wrap;">when he wrote</a><span style="font-family: inherit; white-space: pre-wrap;">: “Evil… was hollow, more brittle than glass, lighter than a feather, thinner than a hair, tinier than a dust speck, so tiny it barely even existed at all. Evil was the world’s dumbest joke, the flimsiest illusion.”</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">And if that’s so, then it’s on us to bring down the curtain on it.</span></p></span></span>Richard Ngohttp://www.blogger.com/profile/04369245820067821936noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-74373792689713069592021-01-19T16:52:00.005-08:002021-01-19T17:01:23.830-08:00Deutsch and Yudkowsky on scientific explanationScience aims to come up with good theories about the world - but what makes a theory good? The standard view is that the key traits are predictive accuracy and simplicity. Deutsch focuses instead on the concepts of <i>explanation</i> and <i>understanding</i>: a good theory is an explanation which enhances our understanding of the world. This is already a substantive claim, because various schools of instrumentalism have been fairly influential in the philosophy of science. I do think that this perspective has a lot of potential, and later in this essay explore some ways to extend it. First, though, I discuss a few of Deutsch's arguments which I don't think succeed, in particular when compared to the bayesian rationalist position defended by Yudkowsky.<br /><br />To start, Deutsch says that good explanations are “hard to vary”, because every part of the explanation is playing a role. But this seems very similar to the standard criterion of simplicity. Deutsch rejects simplicity as a criterion because he claims that theories like “The gods did it” are simple. Yet I’m persuaded by <a href="https://www.lesswrong.com/posts/f4txACqDWithRi7hs/occam-s-razor">Yudkowsky’s argument</a> that a version of “The gods did it” theory which could actually predict a given set of data would essentially need to encode all that data, making it very complex. I’m not sold on Yudkowsky’s definition of simplicity in terms of Kolmogorov complexity (for reasons I’ll explain later on) but re-encoding a lot of data should give rise to a complex hypothesis by any reasonable definition. So it seems most parsimonious to interpret the “hard to vary” criterion as an implication of the simplicity criterion.<br /><br />Secondly, Deutsch says that good explanations aren’t just predictive, but rather tell us about the underlying mechanisms which generate those predictions. As an illustration, he argues that even if we can predict the outcome of a magic trick, what we really want to know is how the trick works. But this argument doesn’t help very much in adjudicating between scientific theories - in practice, it’s often valuable to accept purely predictive theories as stepping-stones to more complete theories. For example, Newton’s inverse square law of gravity was a great theory despite not attempting to explain <i>why</i> gravity worked that way; instead it paved the way for future theories which did so (and which also made better predictions). If Deutsch is just arguing that <i>eventually</i> science should aim to identify all the relevant underlying mechanisms, then I think that most scientific realists would agree with him. The main exception would be in the context of foundational physics. Yet that’s a domain in which it’s very unclear what it means for an underlying mechanism to “really exist”; it’s so far removed from our everyday intuitions that Deutsch’s magician analogy doesn’t seem very applicable.<br /><br />Thirdly, Deutsch says that we can understand the importance of testability in terms of the difference between good and bad explanations:<br /><br /><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;">“The best explanations are the ones that are most constrained by existing knowledge – including other good explanations as well as other knowledge of the phenomena to be explained. That is why testable explanations that have passed stringent tests become extremely good explanations.”</blockquote><br />But this doesn’t help us distinguish between explanations which have themselves been tested, versus explanations which were formulated afterwards to match the data from those same tests. Both are equally constrained by existing knowledge - why should we be more confident in the former? Without filling in this step of the argument, it’s hard to understand the central role of testability in science. I think, again, that Yudkowsky provides the best explanation: that the human tendency towards <a href="https://www.lesswrong.com/posts/fkM9XsNvXdYH6PPAx/hindsight-bias">hindsight bias</a> means we dramatically overestimate how well our theories explain observed data, unless we’re forced to make predictions in advance.<br /><br />Having said all this, I do think that Deutsch’s perspective is valuable in other ways. I was particularly struck by his argument that the “theory of everything” which fundamental physicists search for would be less interesting than a high-level “theory of everything” which forges deep links between ideas from many disciplines (although I wish he’d say a bit more about what it means for a theory to be “deep”). This argument (along with the rest of Deutsch’s framework) pushes back against the longstanding bias in philosophy of science towards treating physics as the central example of science. In particular, thinking of theories as sets of equations is often appropriate for physics, but much less so for fields which are less formalism-based - i.e. almost all of them.[0] For example, the theory of evolution is one of the greatest scientific breakthroughs, and yet its key insights can’t be captured by a formal model. In Chapman’s terminology, evolution and most other theories are somewhat <a href="https://metarationality.com/nebulosity">nebulous</a>. This fits well with Deutsch’s focus on science as a means of understanding the world - because even though formalisms don’t deal well with nebulosity, our minds do.<br /><br />Another implication of the nebulosity of scientific theories is that we should move beyond the true-false dichotomy when discussing them. Bayesian philosophy of science is based on our credences about how likely theories are to be true. But it’s almost never the case that high-level theories are <i>totally</i> true or <i>totally</i> false; they can explain our observations pretty well even if they don’t account for everything, or are built on somewhat leaky abstractions. And so assigning probabilities only to the two outcomes “true” and “false” seems simplistic. I still consider probabilistic thinking about science to be valuable, but I expect that thinking in terms of degrees of truth is just as valuable. And the latter comes naturally from thinking of theories as explanations, because we intuitively understand that the quality of explanations should be evaluated in a continuous rather than binary way.[1]<br /><br />Lastly, Deutsch provides a good critique of philosophical positions which emphasise <i>prediction</i> over <i>explanation</i>. He asks us to imagine an “experiment oracle” which is able to tell us exactly what the outcome of any specified experiment would be:<br /><br /><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;">“If we gave it the design of a spaceship, and the details of a proposed test flight, it could tell us how the spaceship would perform on such a flight. But it could not design the spaceship for us in the first place. And even if it predicted that the spaceship we had designed would explode on take-off, it could not tell us how to prevent such an explosion. That would still be for us to work out. And before we could work it out, before we could even begin to improve the design in any way, we should have to understand, among other things, how the spaceship was supposed to work. Only then would we have any chance of discovering what might cause an explosion on take-off. Prediction – even perfect, universal prediction – is simply no substitute for explanation.”</blockquote><br />Although I assume it isn’t intended as such, this is a strong critique of <a href="https://www.lesswrong.com/posts/Kyc5dFDzBg4WccrbK/an-intuitive-explanation-of-solomonoff-induction">Solomonoff induction</a>, a framework which <a href="https://arbital.com/p/solomonoff_induction/?l=1hh">Yudkowsky defends</a> as an idealised model for how to reason. The problem is that the types of hypotheses considered by Solomonoff induction are not <i>explanations</i>, but rather computer programs which output predictions. This means that even a hypothesis which is assigned very high credence by Solomonoff induction might be nearly as incomprehensible as the world itself, or more so - for example, if it merely consists of a simulation of our world. So I agree with Deutsch: even idealised Solomonoff induction (with infinite compute) would lack some crucial properties of explanatory science.[2]<div><br /><span style="font-size: large;">Extending the view of science as explanation<br /></span><br />How could Deutsch’s identification of the role of science as producing human-comprehensible explanations actually improve science in practice? One way is by making use of the social science literature on explanations. <a href="https://arxiv.org/abs/1706.07269">Miller identifies</a> four overarching lessons:<br /><ol style="text-align: left;"><li>Explanations are contrastive — they are sought in response to particular counterfactual cases.</li><li>Explanations are selected (in a biased manner) - humans are adept at selecting one or two causes from a sometimes infinite number of causes to be the explanation.</li><li>Referring to probabilities or statistical relationships in explanation is not as effective as referring to causes.</li><li>Explanations are social — they are a transfer of knowledge, presented as part of a conversation or interaction, and are thus presented relative to the explainer’s beliefs about the explainee’s beliefs.</li></ol>We can apply some of these lessons to improve scientific explanations. Consider that scientific theories are usually formulated in terms of existing phenomena. But to formulate properly contrastive explanations, science will need to refer to counterfactuals. For example, in order to fully explain the anatomy of an animal species, we’ll need to understand other possible anatomical structures, and the reasons why those didn’t evolve instead. <a href="https://www.amazon.co.uk/Scale-Universal-Organisms-Cities-Companies/dp/1780225598">Geoffrey West</a>’s work on scaling laws in biology provides a good example of this type of explanation. Similarly, we shouldn’t think of fundamental physics as complete until we understand not only how our universe works, but also which counterfactual laws of physics could have generated other universes as interesting as ours.<br /><br />A second way we can try to use Deutsch’s framework to improve science: what does it mean for a human to understand an explanation? Can we use findings from cognitive science, psychology or neuroscience to make suggestions for the types of theories scientists work towards? This seems rather difficult, but I’m optimistic that there’s some progress to be made. For example, analogies and metaphors play an extensive role in everyday human cognition, as highlighted by Lakoff’s <i>Metaphors we live by</i>. So instead of thinking about analogies as useful ways to communicate a scientific theory, perhaps we should consider them (in some cases) to be a core part of the theory itself. Focusing on analogies may slightly reduce those theories’ predictive power (because it’s hard to cash out analogies in terms of predictions) while nevertheless increasing the extent to which they allow us to actually understand the world. I’m reminded of the elaborate comparison between self-reference in mathematics and self-replication in biology drawn by Hofstadter in Godel, Escher, Bach - if we prioritise a vision of science as understanding, then this sort of work should be much more common. However, the human tendency towards hindsight bias is a formidable opponent, and so we should always demand that such theories also provide novel predictions, in order to prevent ourselves from generating an illusion of understanding.<br /><br /><br />[0]. As an example of this bias, see the first two perspectives on scientific theories <a href="https://plato.stanford.edu/entries/structure-scientific-theories/">discussed here</a>; my position is closest to the third, the <a href="https://plato.stanford.edu/entries/structure-scientific-theories/#PraVie">pragmatic view</a>.<br />[1]. Work on <a href="https://intelligence.org/2016/09/12/new-paper-logical-induction/">logical induction</a> and <a href="https://www.alignmentforum.org/s/Rm6oQRJJmhGCcLvxh">embedded agency</a> may partly address this issue; I’m not sure.<br />[2]. I was originally planning to go on to discuss Deutsch’s broader critiques of empiricism and induction. But Deutsch makes it hard to do this, because he doesn’t refer very much to the philosophical literature, or specific people whose views he disagrees with. It seems to me that this leads to a lot of linguistic disagreements. For example, when he critiques the idea of knowledge being “derived” from experience, or scientific theories being “justified” by empirical experience, I feel like he’s using definitions of these terms which diverge both from what most people take them to mean, and also from what most philosophers take them to mean. Nor do I think that his characterisation of observation as theory-laden is inconsistent with standard inductivism; he seems to think it is, but doesn’t provide evidence for that. So I’ve decided not to go deeper on these issues, except to note my skepticism about his position.</div>Richard Ngohttp://www.blogger.com/profile/04825733481608403399noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-1508414208605415562021-01-15T14:13:00.010-08:002021-01-23T06:07:20.177-08:00Meditations on faith<p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">A few months before his death, Leonard Cohen, the great lyricist of modern spirituality, sang to God:<br /><br /></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>Magnified, sanctified</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>Be the holy name</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>Vilified, crucified</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>In the human frame</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>A million candles burning</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>For the help that never came</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>You want it darker</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i><br /></i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>You're lining up the prisoners</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>And the guards are taking aim</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>I struggled with some demons</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>They were middle class and tame</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>I didn't know I had permission</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>To murder and to maim</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>You want it darker</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i><br /></i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>Hineni, hineni</i></p><p style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i>I'm ready, My Lord</i></p></blockquote><p>The first lines are a reference to the Mourner’s Kaddish, a Jewish prayer for the deceased. The million candles - each one in remembrance of a life lost - reminds us of tragedies upon preventable tragedies. So too with the prisoners, the guards, the murders: if these are part of some deity’s plan, it’s a deity which wants the world darker. Finally, hineni is what Abraham said when God called upon him to sacrifice Isaac. It means <i>Here I am</i>; but with deep connotations: <i>I am willing</i>, or perhaps <i>I am yours</i>.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote>Together, I find these verses, and the rest of the song, deeply striking and totally incomprehensible. They throw the brute fact of immense suffering, and death, and darkness, at the listener. And then they don’t question it, or retreat from it, or refute it. The opposite! Hineni: yielding completely. Leonard Cohen, at death’s door, singing “I’m ready, My Lord”, as if there is nothing to excuse or explain. Or more: as if the litany of suffering in the rest of the song only strengthens his convictions.<br /><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote><br /><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote>It occured to me, listening to these lyrics, that I had no idea what that mental state feels like; that there is a whole spectrum of experience - apparently, overwhelmingly powerful experience - which is alien to me.<div><br /></div><div>Now, perhaps I am misinterpreting Cohen; perhaps these lyrics are ironic, or despairing (although if so, I just don’t see it, <a href="https://www.youtube.com/watch?v=2s3kQSZ_Qxk">and neither does the former UK Chief Rabbi</a>). But either way, the story of Abraham and Isaac, which encapsulates the question of faith in the face of suffering, is one with which religious thinkers have wrestled down the centuries. It is <a href="http://www.sophia-project.org/uploads/1/3/9/5/13955288/kierkegaard_fear.pdf">the story that Kierkegaard chooses</a> to illustrate the absolute absurdity of absolute faith - and its greatness, not in spite of, but because of that absurdity.<div><br />Yet, striking as Kierkegaard’s writings are, suppose that we would prefer not to found our worldviews on absurdity; what then? Is there something about this experience which atheists can learn from and absorb? I think there is, and it comes from thinking more deeply about faith. Forgive me if I am naively recapitulating well-known arguments; in this domain, it seems worth cultivating these thoughts myself.<br /><br />What do we actually mean by faith? Let me distinguish two related concepts: faith as belief (in particular, about properties of God), and faith as surrender. Atheists know that faith in standard beliefs about God is misguided: that God is not omnibenevolent, nor the source of morality, nor the rightful authority. From this perspective, the story of Abraham arouses bewilderment, or scorn, or pity. But, for me at least, this has obscured the second aspect of faith. There are many beautiful things about the human spirit, and one of the greatest is our ability to trust each other - where surrendering control entirely to another is the ultimate form of trust. The absurdity of the story of Abraham comes not from the fact that he surrenders in this way, but rather from the fact that we cannot imagine any good reason why an omnipotent God should require Isaac to be sacrificed. Yet if, instead of God, we picture Abraham’s dearest friend saying to him, “I can’t tell you why, and I’m aghast that this is the case, but it is imperative that you take your only son, whom you love dearly, and kill him” - if at <i>that</i> point Abraham says “Hineni”, in the knowledge that his friend would do the same if their roles were reversed - then it is clearer that Abraham’s surrender reflects a depth of trust and connection that we should strive towards.[0]<br /><br />So where does this leave us? Abraham’s belief in God was misguided. Today we do better by embracing the virtues of the Enlightenment - of reason and humanism and defiance of unjust authority. But nevertheless, in his ability to trust, he displays something valuable, and powerful enough that it has kept us coming back to his story for thousands of years.<br /><br />----------------------------------------------------------------------------------<br /><br />I think I believe this argument. Yet there’s something odd: if faith as surrender is so compelling to so many others, why do I (still) not feel the appeal of it? I’ve been familiar with this core idea since learning that Islam literally means “surrender”, many years ago. And yet I’ve never felt like that’s what’s missing in my life; nor do I hear people talk about it very much outside the context of religion.<br /><br />Maybe I’m biased by a longstanding dislike of religion. But a hypothesis that worries me more is that I’m too much a child of individualistic modernity to actually understand or appreciate this sort of surrender. Scott Alexander talks about <a href="https://slatestarcodex.com/2017/11/07/concept-shaped-holes-can-be-impossible-to-notice/">concept-shaped holes</a>: ideas you don’t even realise you don’t understand. Some of his examples: not believing that society has become atomised and individualistic, because you mistake the scattered remnants of community around you for the real thing. Or being shut off from genuine emotional connection, but not realising it because you think that your shallow connections are as good as it gets.</div><div><br />And I wonder whether the same thing has happened to “trust”. When I say that I trust somebody, often I’m thinking about them being a good and sensible person. Even when I say that I’d trust somebody with my life, it feels like that’s still pretty close to saying that they’re a really good person, and pretty careful, and also that they care about me.<br /><br />Okay, but now if I imagine saying: I trust somebody enough to surrender my will to theirs - enough that they could override any of my decisions, and I wouldn’t even resent them for it, because I’ve chosen to have faith in them. Oh. Yeah. That feels pretty different. That feels like a state of mind I don’t even understand, because I think of myself so deeply in individualistic terms that I’m not sure what it would mean to hand over that level of agency. Does it happen in the best relationships? I don’t know, but I can’t recall anyone talking about it, at least not in these terms. More generally, maybe this is what people mean when they talk about becoming “part of something bigger than you” But I’ve done that, with movements and ideologies, and even so, I relate to them very much as a free individual. I expect that it used to be much easier to feel defined by being a part of something much bigger than you - your family, your town, your religion. Now, even when we still belong to these groups, we no longer let them subsume our autonomy in the same way.<br /><br />Another way to think about this change is as an increasing desire for control. Modernity’s defining feature is humans exercising control over our environments. Many object to this, perhaps because it seems hubristic - but the tragedy is that we <i>shouldn’t</i> trust nature, or God, or fate; they’re indifferent to us. Humanity needs to seize control of its destiny because nobody else will do it for us. And one of the key ways we do so is via science, where <a href="https://en.wikipedia.org/wiki/Nullius_in_verba">taking nothing on faith is a central tenet</a>.</div><div><br /></div><div>But perhaps my mistake has been in thinking of my personal life as a microcosm of humanity, and trying to seize control of it in similar ways. Unlike humanity, I’m not surrounded by uncaring nature, but by a social environment of people who are sufficiently trustworthy that this core religious instinct, to surrender, may be worth leaning into. In other words: if we should no longer put our faith in God, and we should no longer (after the atrocities of the 20th century) put our faith in society at large - perhaps we can still fill that fundamental need by putting our faith in the people around us, on a smaller scale.</div><div><br />I started off by talking about absolute faith. But extremism is seldom a virtue when it comes to human interactions - and in this case it runs into many of the same problems as absolute faith in God. What if you’re mistaken about the people you’ve put your faith in? What if they want you to do bad things? I notice that many people who attract this level of faith end up as cult leaders; and surrender to ideologies has often led to atrocities. So in practice, I think we shouldn’t get anywhere near Abrahamic levels of trust when we don’t have an omnibenevolent being to direct it towards. But Abraham as an idealised example, to nudge us in the right direction in our small-scale personal lives at least, and to counterbalance the underlying pressure to be <i>in</i> <i>control</i> - well, that seems like an idea worth having. As a humanist, I consider human relationships to be one of the main sources of meaning and value in the world. And in a secular, humanist context, we’re not just the one surrendering, but also the one being surrendered to. So part of moving towards that ideal is making ourselves worthy of that responsibility. In that sense, at least, humanists should aim to play the role of God.<br /><br /><br />[0] I’ve specified an equal relationship, because that fits with modern sensibilities. When we imagine Abraham trusting an authority figure in an unreciprocated way, it starts to seem a bit weird - what gives them that authority? But I imagine that the ancients saw unequal relationships (e.g. their marriages) as potentially also very meaningful and fulfilling.<br /></div></div>Richard Ngohttp://www.blogger.com/profile/04825733481608403399noreply@blogger.com3tag:blogger.com,1999:blog-6642084484458008587.post-5208538937753774702021-01-15T07:12:00.013-08:002021-02-03T01:38:50.632-08:00Scope-sensitive ethics: capturing the core intuition motivating utilitarianism<a href="https://www.utilitarianism.net/types-of-utilitarianism">Classical utilitarianism</a> has many advantages as an ethical theory. But there are also many problems with it, some of which <a href="http://thinkingcomplete.blogspot.com/2017/10/utilitarianism-and-its-discontents.html">I discuss here</a>. A few of the most important:<br /><ul style="text-align: left;"><li>The idea of reducing all human values to a single metric is counterintuitive. Most people care about <a href="https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile">a range of things</a>, including both their conscious experiences and outcomes in the world. I haven’t yet seen a utilitarian conception of welfare which describes what I’d like my own life to be like.</li><li>Concepts derived from our limited human experiences will lead to strange results when they’re taken to extremes (as utilitarianism does). Even for things which seem robustly good, trying to <i>maximise</i> them will likely give rise to <a href="https://slatestarcodex.com/2018/09/25/the-tails-coming-apart-as-metaphor-for-life/">divergence at the tails</a> between our intuitions and our theories, as in <a href="https://en.wikipedia.org/wiki/Mere_addition_paradox">the repugnant conclusion</a>.</li><li>Utilitarianism doesn’t pay any attention to personal identity (except by taking a person-affecting view, which leads to worse problems). At an extreme, it endorses the <a href="https://www.tandfonline.com/doi/full/10.1080/0020174X.2019.1658631">world destruction argument</a>: that, if given the opportunity to kill everyone who currently exists and replace them with beings with greater welfare, we should do so.</li><li>Utilitarianism is post-hoc on small scales; that is, although you can technically argue that standard moral norms are justified on a utilitarian basis, it’s very hard to explain why these moral norms are better than others. In particular, it seems hard to make utilitarianism consistent with caring much more about people close to us than strangers.</li></ul>I (and probably many others) think that these objections are compelling, but none of them defeat the core intuition which makes utilitarianism appealing: that some things are good, and some things are bad, and we should continue to want more good things and fewer bad things even beyond the parochial scales of our own everyday lives. Instead, the problems seem like side effects of trying to pin down a version of utilitarianism which provides a precise, complete guide for how to act. Yet I’m not convinced that this is very useful, <a href="http://thinkingcomplete.blogspot.com/2019/02/arguments-for-moral-indefinability.html">or even possible</a>. So I’d prefer that people defend the core intuition directly, at the cost of being a bit vaguer, rather than defending more specific utilitarian formalisations which have all sorts of unintended problems. Until now I’ve been pointing to this concept by saying things like “utilitarian-ish” or “90 percent utilitarian”. But it seems useful for coordination purposes to put a label on the property which I consider to be the most important part of utilitarianism; I’ll call it “scope-sensitivity”.<br /><br />My tentative definition is that scope-sensitive ethics consists of:<br /><ul style="text-align: left;"><li>Endorsing actions which, in expectation, bring about more intuitively valuable aspects of individual lives (e.g. happiness, preference-satisfaction, etc), or bring about fewer intuitively disvaluable aspects of individual lives (e.g. suffering, betrayal).</li><li>A tendency to endorse actions much more strongly when those actions increase (or decrease, respectively) those things much more.</li></ul>I hope that describing myself as caring about scope-sensitivity conveys the most important part of my ethical worldview, without implying that I have a precise definition of welfare, or that I want to convert the universe into <a href="https://en.wiktionary.org/wiki/hedonium">hedonium</a>, or that I’m fine with replacing humans with happy aliens. Now, you could then ask me which specific scope-sensitive moral theory I subscribe to. But I think that this defeats the point: as soon as we start trying to be very precise and complete, we’ll likely run into many of the same problems as utilitarianism. Instead, I hope that this term can be used in a way which conveys a significant level of uncertainty or vagueness, while also being a strong enough position that if you accept scope-sensitivity, you don’t need to clarify the uncertainty or vagueness much in order to figure out what to do. (I say "uncertainty or vagueness" because moral realists are often particularly uncomfortable with the idea of morality being intrinsically vague, and so this phrasing allows them to focus on <a href="https://concepts.effectivealtruism.org/concepts/moral-uncertainty/">the uncertainty part</a>: the idea that <i>some </i>precise scope-sensitive theory is true, but we don't yet know which one. Whereas <a href="http://thinkingcomplete.blogspot.com/2019/02/arguments-for-moral-indefinability.html">my own position</a> is that it's fine and indeed necessary for morality to be intrinsically imprecise, and so it's hard to draw the line between questions we're temporarily uncertain about, and questions which don't have well-defined answers. From this perspective, we can also think about scope-sensitive ethics as a single vague theory in its own right.)<div><div><br /></div><div>How does the definition I've given address the problems I described above? Firstly, it’s pluralist (within the restrictions of common sense) about what contributes to the welfare of individuals. The three most common types of utilitarian conceptions of welfare are hedonic theories, desire theories and objective-list theories. But each of these captures something which I care about, and so I don't think we know nearly enough about human minds (let alone non-human minds) to justify taking a strong position on which combination of these constitutes a good life. Scope-sensitivity also allows room for even wider conceptions of welfare: for example, people who think that achieving virtue is the most valuable aspect of life can be scope-sensitive if they try to promote that widely.</div><div><br /></div><div><div>Secondly, it’s also consistent with pluralism about value more generally. Scope-sensitivity doesn’t require you to only care about welfare; you can value other things, as long as they don’t override the overall tendency to prioritise actions with bigger effects. In particular, unlike utilitarianism, scope-sensitivity is consistent with using non-consequentialist or non-impartial reasoning about most small-scale actions we take (even when we can't justify why that reasoning leads to the best consequences by impartial standards). Furthermore, it doesn’t require that you endorse welfare-increasing actions because they increase welfare. In addition to my moral preferences about sentient lives, I also have moral preferences about the trajectory of humanity as a whole: as long as humanity flourishing is correlated closely enough with humans flourishing, then those motivations are consistent with scope-sensitivity.<br /><br />Thirdly, scope-sensitivity isn’t rigid. It doesn’t require welfare-maximisation in all cases; instead, specifying a “tendency” rather than a “rule” of increasing welfare allows us to abide by other constraints as well. I think this reflects the fact that a lot of people do have qualms about extreme cases (for which <a href="http://thinkingcomplete.blogspot.com/2019/02/arguments-for-moral-indefinability.html">there may not be any correct answers</a>) even when their general ethical framework aims towards increasing good things and decreasing bad things.<br /><br />I should make two further points about evaluating the scope-sensitivity of existing moral theories. Firstly, I think it’s best interpreted as a matter of degree, rather than a binary classification. Secondly, we can distinguish between “principled” scope-sensitivity (scope-sensitivity across a wide range of scenarios, including implausible thought experiments) versus “practical” scope-sensitivity (scope-sensitivity given realistic scenarios and constraints).</div><div><br />I expect that almost all of the people who are most scope-sensitive in principle will be consequentialists. But in practice, non-consequentialists can also be highly scope-sensitive. For example, it may be the case that a deontologist who follows the rule ”try to save the world, if it’s in danger” is in practice nearly as scope-sensitive as a classical utilitarian, even if they also obey other rules which infrequently conflict with it (e.g. not lying). Meanwhile, some variants of utilitarianism (such as average utilitarianism) also aren’t scope-sensitive in principle, although they may be in practice.<br /><br />One problem with the concept of scope-sensitivity is that it might induce <a href="https://slatestarcodex.com/2014/11/03/all-in-all-another-brick-in-the-motte/">motte-and-bailey fallacies</a> - that is, we might defend our actions on the basis of scope-sensitivity when challenged, but then in practice act according to a particular version of utilitarianism which we haven't justified. But I actually think the opposite happens now: people are motivated by the intuition towards scope-sensitivity, and then defend their actions by appealing to utilitarianism. So I hope that introducing this concept improves our moral discourse, by pushing people to explicitly make the argument that scope-sensitivity is sufficient to motivate views like longtermism.<br /><br />Another possibility is that scope-sensitivity is too weak a concept to motivate action - for example, if people claim to be scope-sensitive, but add a few constraints which mean they don’t ever need to act accordingly. But even if scope-sensitivity in principle is broad enough to include such views, hopefully the concept of practical scope-sensitivity identifies a natural cluster of moral views which, if people follow them, will actually make the world a much better place. </div></div></div>Richard Ngohttp://www.blogger.com/profile/04825733481608403399noreply@blogger.com2tag:blogger.com,1999:blog-6642084484458008587.post-20417619414718261552020-11-22T09:43:00.004-08:002020-12-02T16:30:50.637-08:00My fictional influencesI’ve identified as a bookworm for a very long time. Throughout primary school and high school I read voraciously, primarily science fiction and fantasy. But given how much time I’ve spent reading fiction, it’s surprisingly difficult to pin down how it’s influenced me. (This was also tricky to do for nonfiction, actually - see <a href="http://thinkingcomplete.blogspot.com/2020/11/my-intellectual-influences.html">my attempt in this post</a>.)<br /><br />Thinking back to the fiction I’ve enjoyed the most, two themes emerge: atmosphere, and cleverness. The atmosphere that really engages me in fiction is one that says: the world is huge; there’s so much to explore; and there’s a vastness of potential. But one that’s also a little melancholy - because you can’t possibly experience all of it, and time always flows onwards. I was particularly struck by the ending of <i>The Lord of the Rings</i>, when Frodo leaves all of Middle-Earth behind; by <i>His Dark Materials</i>, when Lyra gains, and loses, uncountable worlds; by the <i>Malazan</i> saga, occurring against a fictional backdrop of hundreds of thousands of years of epic history; and by <i>Speaker for the Dead</i>, as Ender skims through the millennia. Oh, and I can’t forget George R. R. Martin’s <i>A Song for Lya</i>, Neil Gaiman’s <i>Ocean at the End of the Lane</i>, and Paterson’s <i>Bridge to Terabithia</i> - none are the subtlest, but they're all exquisitely wistful. I’m compelled by the aesthetic: each of these a whole world that never was and will never be!<br /><br />The other thing I love in fiction is cleverness: <a href="https://tvtropes.org/pmwiki/pmwiki.php/Main/XanatosGambit">Xanatos Gambits</a> and <a href="https://tvtropes.org/pmwiki/pmwiki.php/Main/MagnificentBastard">Magnificent Bastards</a> and plans within plans that culminated in startling and brilliant ways. <i>Ender’s Game</i> is a great example; so too is <i>The Lies of Locke Lamora</i>. On the literary side, I loved <i>Catch-22</i> for its cleverness in weaving together so many peculiar threads into a striking tapestry. Lately the novels which most scratch this itch have been online works, particularly <a href="http://unsongbook.com/"><i>Unsong</i></a>, <a href="https://parahumans.wordpress.com/"><i>Worm</i></a>, and <a href="https://practicalguidetoevil.wordpress.com/"><i>A Practical Guide to Evil</i></a>. Some sci-fi novels also fall in this category - I’m thinking particularly of <i>Snow Crash</i>, <a href="http://www.antipope.org/charlie/blog-static/fiction/accelerando/accelerando-intro.html"><i>Accelerando</i></a>, and <i>Hyperion</i>.<br /><br />It’s hard to tell whether my fiction preferences shaped my worldview or vice versa, but I’d be surprised if all this reading weren’t at least partially responsible for me often thinking about the big picture for humanity, and personally aiming for ambitious goals. What’s more difficult is to point to specific things I gained from these books. I don’t identify with many fictional characters, and can't think of any personal conclusions that I've gained from depictions of them (perhaps apart from: communicate more!) I did read a lot of “big idea” books, but they were never that satisfying - fiction always seemed like an inefficient medium for communicating them.<br /><br />But for some reason this has changed a bit over the last few years. I now find myself regularly thinking back to a handful of books as a way to remind myself of certain key ideas - in particular books that pair those ideas with compelling plots and characters. In no particular order:<br /><ul style="text-align: left;"><li><a href="http://unsongbook.com/"><i>Unsong</i></a> is the work of fiction that most inspires me to be a better person; to do the things that “somebody has to and no one else will”.</li><li><i>Diaspora</i> makes me reflect on the emptiness of pure ambition, and the arbitrariness of human preferences.</li><li><i>The Darkness That Comes Before</i> pushes me to understand my mind and motivations - to illuminate “what comes before” my thoughts and actions.</li><li><a href="http://www.antipope.org/charlie/blog-static/fiction/accelerando/accelerando-intro.html"><i>Accelerando</i></a> confronts me with the sheer scale of change that humanity might face.</li><li><i>Island</i> and <i>Walden Two</i> underline the importance of social progress in building utopias.</li><li><i>Flowers for Algernon</i> reminds me of the importance of emotional intelligence.</li></ul>I wish I had a similar list of fiction which taught me important lessons about friendships and relationships, but for whatever reason I haven’t really found many fictional relationships particularly inspiring. I’m very curious about what would be on other people’s lists, though.<br />Richard Ngohttp://www.blogger.com/profile/04825733481608403399noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-39590085323267468172020-11-22T09:10:00.008-08:002021-10-25T15:21:27.680-07:00My intellectual influences<div>Prompted by a friend's question about my reading history, I've been thinking about what shaped the worldview I have today. This has been a productive exercise, which I recommend to others. Although I worry that some of what's written below is post-hoc confabulation, at the very least it's forced me to pin down what I think I learned from each of the sources listed, which I expect will help me track how my views change from here on. This blog post focuses on non-fiction books (and some other writing); I've also written a blog post on <a href="http://thinkingcomplete.blogspot.com/2020/11/my-fictional-influences.html">how fiction has influenced me</a>.</div><div><br /></div>My first strong intellectual influence was Eliezer Yudkowsky’s writings on Less Wrong (now collected in <a href="https://www.readthesequences.com/"><i>Rationality: from AI to Zombies</i></a>). I still agree with <a href="https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences">many</a> <a href="https://www.lesswrong.com/s/5uZQHpecjn7955faL/p/fysgqk4CjAwhBgNYT">of</a> <a href="https://www.lesswrong.com/posts/8vpf46nLMDYPC6wA4/optimization-and-the-intelligence-explosion">his</a> <a href="https://www.lesswrong.com/posts/yA4gF5KrboK2m2Xu7/how-an-algorithm-feels-from-inside">core</a> <a href="https://www.lesswrong.com/s/FrqfoG3LJeCZs96Ym/p/dLJv2CoRCgeC2mPgj">claims</a>, but don’t buy into the overarching narratives as much. In particular, the idea of “rationality” doesn’t play a big role in my worldview any more. Instead I focus on <a href="http://thinkingcomplete.blogspot.com/2019/09/seven-habits-towards-highly-effective.html">specific habits and tools</a> for thinking well (as in <i>Superforecasters</i>), and creating communities with productive epistemic standards (a focus of less rationalist accounts of reason and science, e.g. <i>The Enigma of Reason </i>and <i>The</i> <i>Structure of Scientific Revolutions</i>).<br /><br />Two other strong influences around that time were <a href="http://slatestarcodex.com/">Scott Alexander</a>’s writings on <a href="https://slatestarcodex.com/2014/12/17/the-toxoplasma-of-rage/">tribalism</a> <a href="https://slatestarcodex.com/2014/11/03/all-in-all-another-brick-in-the-motte/">in</a> <a href="https://slatestarcodex.com/2014/09/30/i-can-tolerate-anything-except-the-outgroup/">politics</a>, and Robin Hanson’s work on signalling (particularly <a href="http://elephantinthebrain.com/"><i>Elephant in the Brain</i></a>), both of which are now foundational to my worldview. Both are loosely grounded in evolutionary psychology, although not reliant on it. More generally, even if I’m suspicious of many individual claims from evolutionary psychology, the idea that humans are continuous with animals is central to my worldview (see <i>Darwin’s Unfinished Symphony </i>and<i> Are We Smart Enough to Know How Smart Animals Are?</i>). In particular, it has shaped my views on naturalistic ethics (via a variety of sources, with Wright’s <i>The Moral Animal</i> being perhaps the most central).<div><div><br />Another big worldview question is: how does the world actually change? At one point I bought into techno-economic determinism about history, based on reading big-picture books like <i>Guns, Germs and Steel and The Silk Roads</i>, and also because of my understanding of the history of science (e.g. the prevalence of <a href="https://en.wikipedia.org/wiki/Multiple_discovery">multiple discovery</a>). Sandel’s <i>What Money Can’t Buy</i> nudged me towards thinking more about cultural factors; so did books like <i>The Dream Machine</i> and <i>The Idea Factory</i>, which describe how many technologies I take for granted were constructed. And reading Bertrand Russell’s <a href="http://thinkingcomplete.blogspot.com/2018/08/on-first-looking-into-russells-history.html"><i>History of Western Philosophy</i></a> made me start thinking about the large-scale patterns in intellectual history (on which <i>The Modern Mind</i> further shaped my views).<br /><br />This paved the way for me to believe that there’s room to have a comparable influence on our current world. Here I owe a lot to Tyler Cowen’s <i>The Great Stagnation</i> (and to a lesser extent <a href="http://thinkingcomplete.blogspot.com/2018/08/book-review-complacent-class.html">its sequels</a>), Peter Thiel’s <a href="https://docs.google.com/document/d/1zao_AyBhNb8TPWrQqgXn5NzNAgfEqzTIaFYos7wdqGI/edit?usp=sharing">talks and essays</a> (and to a lesser extent his book <i>Zero to One</i>), and <a href="http://www.paulgraham.com/articles.html">Paul Graham’s essays</a>. My new perspective is similar to the standard “<a href="http://www.paulgraham.com/swan.html">Silicon</a> <a href="http://www.paulgraham.com/wealth.html">Valley</a> <a href="http://www.paulgraham.com/siliconvalley.html">mindset</a>”, but focusing more on the role of ideas than technologies. To repurpose the well-known quote: “Practical men who believe themselves to be quite exempt from any intellectual influence are usually the slaves of some defunct philosopher.”<br /><br /><span id="docs-internal-guid-f1eed30a-7fff-7fab-e0c3-02fe5caeceab"><div><div>Here’s a more complete list of nonfiction books which have influenced me, organised by topic (although I’ve undoubtedly missed some). I'm no longer updating this post, but <a href="https://twitter.com/RichardMCNgo/status/1449937391704174592?t=KlyorVOg4OYb9XYPxCaiDQ&s=19">here's a more recent list of books I've enjoyed</a>. I welcome recommendations, whether they’re books that fit in with these lists, or books that fill gaps in them!</div></div><div><br /></div><p dir="ltr" style="font-family: inherit; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">On ethics:</span></p><ul style="font-family: inherit; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Righteous Mind</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Technology and the Virtues</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Reasons and Persons</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">What Money Can’t Buy</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Precipice</span></p></li></ul><br /><p dir="ltr" style="font-family: inherit; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">On human evolution:</span></p><ul style="font-family: inherit; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Enigma of Reason</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Human Advantage</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Darwin’s Unfinished Symphony</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Secret of our Success</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Human Evolution (Dunbar)</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Mating Mind</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Symbolic Species</span></p></li></ul><br /><p dir="ltr" style="font-family: inherit; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">On human minds and thought:</span></p><ul style="font-family: inherit; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Rationality: from AI to Zombies</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Elephant in the Brain</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">How to Create a Mind</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Why Buddhism is True</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Blank Slate</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Language Instinct</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Stuff of Thought</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Mind is Flat</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Superforecasting</span></p></li></ul><ul style="font-family: inherit; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Thinking, Fast and Slow</span></p></li></ul><br /><p dir="ltr" style="font-family: inherit; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">On other sciences:</span></p><ul style="font-family: inherit; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Scale: The Universal Laws of Life and Death in Organisms, Cities and Companies</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Superintelligence</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Alignment Problem</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Are We Smart Enough to Know How Smart Animals Are?</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Moral Animal</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Ending Aging</span></p></li></ul><ul style="font-family: inherit; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Improbable Destinies</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Selfish Gene</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Blind Watchmaker</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Complexity: The Emerging Science at the Edge of Order and Chaos</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Quantum Computing Since Democritus</span></p></li></ul><br /><p dir="ltr" style="font-family: inherit; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">On science itself:</span></p><ul style="font-family: inherit; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><a href="https://thinkingcomplete.blogspot.com/2019/04/book-review-sleepwalkers.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">The Sleepwalkers: A History of Man's Changing Vision of the Universe</span></a></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Structure of Scientific Revolutions</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The End of Science</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Fabric of Reality</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Beginning of Infinity</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Reinventing Discovery</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Dream Machine</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Idea Factory</span></p></li></ul><br /><p dir="ltr" style="font-family: inherit; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">On philosophy:</span></p><ul style="font-family: inherit; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><a href="http://thinkingcomplete.blogspot.com/2018/08/on-first-looking-into-russells-history.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">A History of Western Philosophy</span></a></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Intentional Stance</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">From Bacteria to Bach and Back</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Good and Real</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Big Picture</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Consciousness and the Social Brain</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">An Enquiry Concerning Human Understanding</span></p></li></ul><br /><p dir="ltr" style="font-family: inherit; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">On history and economics:</span></p><ul style="font-family: inherit; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Shortest History of Europe</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">A Farewell to Alms</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><a href="http://thinkingcomplete.blogspot.com/2019/07/book-review-technology-trap.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">The Technology Trap</span></a></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Iron, Steam and Money</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Enlightened Economy</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Commanding Heights</span></p></li></ul><ul style="font-family: inherit; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">A Short History of Nearly Everything</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Modern Mind</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><a href="https://thinkingcomplete.blogspot.com/2018/04/book-review-23-things-they-dont-tell.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">23 things they don’t tell you about capitalism</span></a></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><a href="https://www.mercatus.org/system/files/helland-tabarrok_why-are-the-prices-so-damn-high_v1.pdf" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Why are the prices so damn high?</span></a></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Silk Roads</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Sapiens</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Historical Figure of Jesus</span></p></li></ul><br /><p dir="ltr" style="font-family: inherit; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">On politics and society:</span></p><ul style="font-family: inherit; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Destined for War</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Prisoners of Geography</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><a href="http://thinkingcomplete.blogspot.com/2018/11/how-democracy-ends-review-and.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">How Democracy Ends</span></a></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Why Nations Fail</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Factfulness</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">What Terrorists Want</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Lexus and the Olive Tree</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Bowling Alone</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Antifragile</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><a href="https://thinkingcomplete.blogspot.com/2016/12/book-review-female-eunuch-by-germaine.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">The Female Eunuch</span></a></p></li></ul><br /><p dir="ltr" style="font-family: inherit; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">On life, love, etc:</span></p><ul style="font-family: inherit; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Deep Work</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Man's Search for Meaning</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">More Than Two</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Authentic Happiness</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><a href="https://thinkingcomplete.blogspot.com/2017/11/book-review-happiness-by-design.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Happiness by Design</span></a></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Written in History</span></p></li></ul><br /><p dir="ltr" style="font-family: inherit; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Other:</span></p><ul style="font-family: inherit; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><a href="https://ageofem.com/" style="text-decoration-line: none;"><span style="color: #1155cc; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Age of Em</span></a></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Immortality: The Quest to Live Forever and How It Drives Civilization</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Surely you’re Joking, Mr Feynman</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Impro</span></p></li><li dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Never Split the Difference</span></p></li></ul></span></div></div>Richard Ngohttp://www.blogger.com/profile/04825733481608403399noreply@blogger.com1tag:blogger.com,1999:blog-6642084484458008587.post-63569776060417122052020-11-07T03:04:00.005-08:002020-11-07T03:05:21.422-08:00Why philosophy of science?<div>During my last few years working as an AI researcher, I increasingly came to appreciate the distinction between what makes <i>science</i> successful and what makes <i>scientists</i> successful. Science works because it has distinct standards for what types of evidence it accepts, with empirical data strongly prioritised. But scientists spend a lot of their time following hunches which they may not even be able to articulate clearly, let alone in rigorous scientific terms - and throughout the history of science, this has often paid off. In other words, the types of evidence which are most useful in choosing which hypotheses to prioritise can differ greatly from the types of evidence which are typically associated with science. In particular, I’ll highlight two ways in which this happens.</div><br />First is scientists thinking in terms of concepts which fall outside the dominant paradigm of their science. That might be because those concepts are too broad, or too philosophical, or too interdisciplinary. For example, machine learning researchers are often inspired by analogies to evolution, or beliefs about human cognition, or issues in <a href="https://eprints.soton.ac.uk/250382/1/symgro.pdf">philosophy of language</a> - which are all very hard to explore deeply in a conventional machine learning paper! Often such ideas are mentioned briefly in papers, perhaps in the motivation section - but there’s not the freedom to analyse them with the level of detail and rigour that is required for making progress on tricky conceptual questions.<br /><br />Secondly, scientists often have strong visions for what their field could achieve, and long-term aspirations for their research. These ideas may make a big difference to what subfields or problems those researchers focus on. In the case of AI, some researchers aim to automate a wide range of tasks, or to understand intelligence, or to build safe AGI. Again, though, these aren’t ideas which the institutions and processes of the field of AI are able to thoroughly discuss and evaluate - instead, they are shared and developed primarily in informal ways.<br /><br />Now, I’m not advocating for these ideas to be treated the same as existing scientific research - I think norms about empiricism are very important to science’s success. But the current situation is far from ideal. As one example, Rich Sutton’s <a href="http://www.incompleteideas.net/IncIdeas/BitterLesson.html">essay on the bitter lesson</a> in AI was published on his blog, and then sparked a fragmented discussion on other blogs and personal facebook walls. Yet in my opinion this argument about AI, which draws on his many decades of experience in the field, is one of the most crucial ideas for the field to understand and evaluate properly. So I think we need venues for such discussions to occur in parallel with the process of doing research that conforms to standard publication norms.<br /><br />One key reason I’m currently doing a PhD in philosophy is because I hope that philosophy of science can provide one such venue for addressing important questions which can’t be explored very well within scientific fields themselves. To be clear, I’m not claiming that this is the main focus of philosophy of science - there are many philosophical research questions which, to me and most scientists, seem misguided or confused. But the remit of philosophy of science is broad enough to allow investigations of a wide range of issues, while also rewarding thorough and rigorous analysis. So I’m excited about the field’s potential to bring clarity and insight to the high-level questions scientists are most curious about, especially in AI. Even if this doesn’t allow us to resolve those questions directly, I think it will at least help to tease out different conceptual possibilities, and thereby make an important contribution to scientific - and human - progress.Richard Ngohttp://www.blogger.com/profile/04825733481608403399noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-91789571118930142452020-10-27T18:57:00.003-07:002020-11-15T07:45:26.038-08:00What is past, and passing, and to come?<p>I've realised lately that I haven't posted much on my blog this year. Funnily enough, this coincides with 2020 being my most productive year so far. So in addition to belatedly putting up a few cross-posts from elsewhere, I thought it'd be useful to share here some of the bigger projects I've been working on which haven't featured elsewhere on this blog.</p><p>The most important is <i><a href="https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ">AGI safety from first principles</a> </i>(also <a href="https://drive.google.com/file/d/1uK7NhdSKprQKZnRjU58X7NLA1auXlWHt/view?usp=sharing">available here as a PDF</a>), my attempt to put together the most compelling case for why the development of artificial general intelligence might pose an existential threat to humanity. It's long (about 15,000 words) but I've tried to make it as accessible as possible to people without a machine learning background, because I think the topic is so critically important, and because there's an appalling lack of clear explanations of what might go wrong and why. Early work by Bostrom and Yudkowsky is less relevant in the context of modern machine learning; more recent work is scattered and brief. I originally intended to just summarise other people's arguments, but as the report grew, it became more representative of my own views and less representative of anyone else's. So while it covers the standard ideas, I also think that it provides a new perspective on how to think about AGI - one which doesn't take any previous claims for granted, but attempts to work them out from first principles.</p><p>A second big piece of work is <i><a href="https://docs.google.com/document/d/1zao_AyBhNb8TPWrQqgXn5NzNAgfEqzTIaFYos7wdqGI/edit?usp=sharing">Thiel on progress and stagnation</a></i>, a 100-page compendium of quotes from Peter Thiel on - you guessed it - progress and stagnation in technology, and in society more generally. This was a joint project with <a href="twitter.com/jvnixon">Jeremy Nixon</a>. We both find Thiel's views to be exciting and thought-provoking - but apart from his two books (which focused on different topics) they'd previously only been found scattered across the internet. Our goal was to select and arrange quotes from him to form a clear, compelling and readable presentation of his views. You can judge for yourself if we succeeded - although if you're pressed for time, <a href="https://docs.google.com/document/d/1zao_AyBhNb8TPWrQqgXn5NzNAgfEqzTIaFYos7wdqGI/edit#heading=h.eeo21mqlnox8">there's a summary here</a>.</p><p>Thirdly, I've put together the <a href="https://forum.effectivealtruism.org/s/NKTk9s4tZPiA4aySj"><i>Effective Altruism archives reading list</i></a>. This collates a lot of material from across the internet written by EAs on a range of relevant topics, much of which is otherwise difficult to find (especially older posts). The reading list is aimed at people who are familiar with EA but want to explore in more detail some of the ideas that have historically been influential within EA. These are often more niche or unusual than the material used to promote EA, and I don't endorse all of them - although I tried to only include high-quality content that I think is worth reading if you're interested in the corresponding topic.</p><p>Fourth is my first published paper, <i><a href="https://arxiv.org/abs/2010.07877">Avoiding Side Effects By Considering Future Tasks</a>, </i>which was accepted at NeurIPS 2020! Although note that my contributions were primarily on the engineering side; this is my coauthor Victoria's brainchild. From the abstract: <i>Designing reward functions is difficult: the designer has to specify what to do (what it means to complete the task) as well as what not to do (side effects that should be avoided while completing the task). To alleviate the burden on the reward designer, we propose an algorithm to automatically generate an auxiliary reward function that penalizes side effects. This auxiliary objective rewards the ability to complete possible future tasks, which decreases if the agent causes side effects during the current task. ... Using gridworld environments that test for side effects and interference, we show that our method avoids interference and is more effective for avoiding side effects than the common approach of penalizing irreversible actions.</i></p><p>Fifth, <a href="https://www.alignmentforum.org/s/boLPsyNwd6teK5key">a series of posts on AI safety</a>, exploring safety problems and solutions applicable to agents trained in open-ended environments, particularly multi-agent ones. Unlike most safety techniques, these don't rely on precise specifications - instead they involve "shaping" our agents to think in safer ways, and have safer motivations. Note that this is primarily speculative brainstorming; I'm not confident in any of them, although I'd be excited to see further exploration along these lines.</p><p>More generally, I've been posting a range of AI safety content <a href="https://www.alignmentforum.org/users/ricraz">on the Alignment Forum</a>; I'm particularly happy about <a href="https://www.alignmentforum.org/posts/vqpEC3MPioHX7bv4t/environments-as-a-bottleneck-in-agi-development">these</a> <a href="https://www.alignmentforum.org/posts/S9GxuAEeQomnLkeNt/a-space-of-proposals-for-building-safe-advanced-ai">three</a> <a href="https://www.alignmentforum.org/posts/GqxuDtZvfgL2bEQ5v/arguments-against-myopic-training">posts</a>. And I've been asking questions I'm curious about <a href="https://www.lesswrong.com/posts/Qz6KTt5z3eu4pyKZB/which-scientific-discovery-was-most-ahead-of-its-time">on</a> <a href="https://www.lesswrong.com/posts/SSkYeEpTrYMErtsfa/what-are-some-of-robin-hanson-s-best-posts">Less</a> <a href="https://www.lesswrong.com/posts/p5cMynf5de9FiyCxL/what-explanatory-power-does-kahneman-s-system-2-possess">Wrong</a> and the <a href="https://forum.effectivealtruism.org/posts/Xfig7jbAP8kP3wp7R/how-do-most-utilitarians-feel-about-replacement-thought">Effective</a> <a href="https://forum.effectivealtruism.org/posts/Jpmbz5gHJK9CA4aXA/what-are-the-key-ongoing-debates-in-ea">Altruism</a> <a href="https://forum.effectivealtruism.org/posts/ahPL4N6byqR5YvJcw/which-scientific-discovery-was-most-ahead-of-its-time">Forum</a>. Lastly, I've been very active <a href="twitter.com/richardmcngo">on Twitter</a> over the past couple of years; I haven't yet gotten around to collating my best tweets, but will do so eventually (and post them on this blog).</p><p>So that's what I've been up to so far this year. What's now brewing? I'm currently drafting my first piece of work for my PhD, on the links between biological fitness-maximisation and optimisation in machine learning. A second task is to revise the essay on Tinbergen's levels of explanation which I wrote for my Cambridge application - I think there are some important insights in there, but it needs a lot of work. I'm also writing a post tentatively entitled <i>A philosopher's apology</i>, explaining why I decided to get a PhD, what works very well about academia and academic philosophy, what's totally broken, and how I'm going to avoid (or fix) those problems. Lastly, I'm ruminating over some of the ideas <a href="http://thinkingcomplete.blogspot.com/2020/10/the-future-of-science.html">discussed here</a>, with the goal of (very slowly) producing a really comprehensive exploration of them. Thoughts or comments on any of these very welcome!</p><p>Zooming out, this year has featured what was probably the biggest shift of my life so far: the switch from my technical career as an engineer and AI researcher, to becoming a philosopher and general thinker-about-things. Of course this was a little butterfly-inducing at times. But increasingly I believe that what the world is missing most is novel and powerful ideas, so I'm really excited about being in a position where I can focus on producing them. So far I only have rough stories about how that happens, and what it looks like to make a big difference as a public intellectual - I hope to refine these over time to be able to really leverage my energies. Then onwards, <a href="http://thinkingcomplete.blogspot.com/2020/01/characterising-utopia.html">and upwards</a>!</p>Richard Ngohttp://www.blogger.com/profile/04825733481608403399noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-22861112246848254182020-10-27T17:22:00.007-07:002020-10-27T17:45:01.009-07:00Against strong bayesianism<blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"></blockquote><span style="font-family: inherit;">In this post (<a href="https://www.lesswrong.com/posts/5aAatvkHdPH6HT3P9/against-strong-bayesianism">cross-posted from Less Wrong</a>) I want to lay out some intuitions about why bayesianism is not very useful as a conceptual framework for thinking either about AGI or human reasoning. This is not a critique of bayesian statistical methods; it’s instead aimed at the philosophical position that bayesianism defines an ideal of rationality which should inform our perspectives on less capable agents, also known as "strong bayesianism". As <a href="https://nostalgebraist.tumblr.com/post/161645122124/bayes-a-kinda-sorta-masterpost">described here</a>:</span><div><span style="font-family: inherit;"><br /></span><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">The Bayesian machinery is frequently used in statistics and machine learning, and some people in these fields believe it is very frequently the right tool for the job. I’ll call this position “weak Bayesianism.” There is a more extreme and more philosophical position, which I’ll call “strong Bayesianism,” that says that the Bayesian machinery is the <i style="box-sizing: inherit;">single correct way</i> to do not only statistics, but science and inductive inference in general – that it’s the “aspirin in willow bark” that makes science, and perhaps all speculative thought, work insofar as it <i style="box-sizing: inherit;">does</i> work.</span></p></blockquote></div></div></blockquote><div><span style="font-family: inherit;"><br /></span></div><span style="font-family: inherit;">Or another way of phrasing the position, <a href="https://www.lesswrong.com/posts/bkSkRwo9SRYxJMiSY/beautiful-probability">from Eliezer</a>:<br /><br /></span><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div class="PostsPage-postContent" style="box-sizing: inherit; text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">You may not be able to compute the optimal [Bayesian] answer. But whatever approximation you use, both its failures and successes will be explainable in terms of Bayesian probability theory.</span></p></blockquote></div></div></blockquote><div><span style="font-family: inherit;"><br /></span></div><span style="font-family: inherit;">First, let’s talk about Blockhead: <a href="https://en.wikipedia.org/wiki/Blockhead_(thought_experiment)">Ned Block’s hypothetical AI</a> that consists solely of a gigantic lookup table. Consider a version of Blockhead that comes pre-loaded with the optimal actions (according to a given utility function) for any sequence of inputs which takes less than a million years to observe. So for the next million years, Blockhead will act just like an ideal superintelligent agent. Suppose I argued that we should therefore study Blockhead in order to understand advanced AI better. Why is this clearly a bad idea? Well, one problem is that Blockhead is absurdly unrealistic; you could never get anywhere near implementing it i n real life. More importantly, even though Blockhead gets the right answer on all the inputs we give it, it’s not doing anything remotely like thinking or reasoning.</span><div><span style="font-family: inherit;"><br />The general lesson here is that we should watch out for when a purported "idealised version" of some process is actually a different type of thing to the process itself. This is particularly true when the idealisation is unimaginably complex, because it might be hiding things in the parts which we can’t imagine. So let's think about what an ideal bayesian reasoner like a Solomonoff inductor actually does. To solve the grain of truth problem, the set of hypotheses it represents needs to include every possible way that the universe could be. We don't yet have any high-level language which can describe all these possibilities, so the only way to do so is listing all possible Turing machines. Then in order to update the probabilities in response to new evidence, it needs to know how that entire universe evolves up to the point where the new evidence is acquired.</span></div><div><span style="font-family: inherit;"><br /></span><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"></blockquote><span style="font-family: inherit;">In other words, an ideal bayesian is not <i>thinking</i> in any reasonable sense of the word - instead, it’s <i>simulating every logically possible universe</i>. By default, we should not expect to learn much about thinking based on analysing a different type of operation that just happens to look the same in the infinite limit. Similarly, the version of Blockhead I described above is basically an <i>optimal tabular policy</i> in reinforcement learning. In reinforcement learning, we’re interested in learning policies which process information about their surroundings - but the optimal tabular policy for any non-trivial environment is too large to ever be learned, and when run does not actually do any information-processing! Yet it's particularly effective as a red herring because we can do proofs about it, and because it can be calculated in some tiny environments.</span></div><div><span style="font-family: inherit;"><br />You might argue that strong bayesianism is conceptually useful, and thereby helps real humans reason better. But I think that concepts from strong bayesianism are primarily useful because they have <i style="box-sizing: inherit;">suggestive names</i>, which make it hard to realise how much work our intuitions are doing to translate from ideal bayesianism to our actual lives. For more on what I mean by this, consider the following (fictional) dialogue:<br /></span><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">Alice the (literal-minded) agnostic: I’ve heard about this bayesianism thing, and it makes sense that I should do statistics using bayesian tools, but is there any more to it?</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">Bob the bayesian: Well, obviously you can’t be exactly bayesian with finite compute. But the intuition that you should try to be more like an ideal bayesian is a useful one which will help you have better beliefs about the world. In fact, most of what we consider to be “good reasoning” is some sort of approximation to bayesianism.</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">A: So let me try to think more like an ideal bayesian for a while, then. Well, the first thing is - you’re telling me that a lot of the things I’ve already observed to be good reasoning are actually approximations to bayesianism, which means I should take bayesianism more seriously. But ideal bayesians don’t update on old evidence. So if I’m trying to be more like an ideal bayesian, I shouldn’t change my mind about how useful bayesianism is based on those past observations.</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">B: No, that’s silly. Of course you should. Ignoring old evidence only makes sense when you’ve already fully integrated all its consequences into your understanding of the world.</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">A: Oh, I definitely haven’t done that. But speaking of <i style="box-sizing: inherit;">all</i> the consequences - what if I’m in a simulation? Or an evil demon is deceiving me? Should I think about as many such skeptical hypotheses as I can, to be more like an ideal bayesian who considers every hypothesis?</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">B: Well, <i style="box-sizing: inherit;">technically</i> ideal bayesians consider every hypothesis, but only because they have infinite compute! In practice you shouldn’t bother with many far-fetched hypotheses, because that’s a waste of your limited time.*</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">A: But what if I have some evidence towards that hypothesis? For example, I just randomly thought of the hypothesis that the universe has exactly a googleplex atoms in it. But there's some chance that this thought was planted in my mind by a higher power to allow me to figure out the truth! I should update on that, right?</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">B: Look, in practice that type of evidence is not worth keeping track of. You need to use common sense to figure out when to actually make the effort of updating.</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">A: Hmm, alright. But when it comes to the hypotheses I <i style="box-sizing: inherit;">do</i> consider, they should each be an explicit description of the entire universe, right, like an ideal bayesian’s hypotheses?</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">B: No, that’s way too hard for a human to do.</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">A: Okay, so I’ll use incomplete hypotheses, and then assign probabilities to each of them. I guess I should calculate as many significant digits of my credences as possible, then, to get them closer to the perfectly precise real-valued credences that an ideal bayesian has?</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">B: Don’t bother. Imprecise credences are good enough except when you’re solving mathematically precise problems.</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">A: Speaking of mathematical precision, I know that my credences should never be 0 or 1. But when an ideal bayesian conditions on evidence they’ve received, they’re implicitly being certain about what that evidence is. So should I also be sure that I’ve received the evidence I think I have?</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">B: No-</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">A: Then since I’m skipping all these compute-intensive steps, I guess getting closer to an ideal bayesian means I also shouldn’t bother to test my hypotheses by making predictions about future events, right? Because an ideal bayesian gets no benefit from doing so - they can just make updates after they see the evidence.</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">B: Well, it’s different, because you’re biased. That’s why science works, because making predictions protects you from post-hoc rationalisation.</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">A: Fine then. So what does it <i style="box-sizing: inherit;">actually</i> <i style="box-sizing: inherit;">mean</i> to be more like an ideal bayesian?</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">B: Well, you should constantly be updating on new evidence. And it seems like thinking of degrees of belief as probabilities, and starting from base rates, are both helpful. And then sometimes <a href="http://yudkowsky.net/rational/bayes/">people conditionalise wrong</a> on simple tasks, so you need to remind them how to do so.</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">A: But these aren’t just bayesian ideas - frequentists are all about base rates! Same with “when the evidence changes, I change my mind” - that one’s obvious. Also, when people try to explicitly calculate probabilities, sometimes they’re <i style="box-sizing: inherit;">way</i> off.** What’s happening there?</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">B: Well, in complex real-world scenarios, you can’t trust your explicit reasoning. You have to fall back on intuitions like “Even though my inside view feels very solid, and I think my calculations account for all the relevant variables, there’s still a reasonable chance that all my models are wrong.”</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">A: So why do people advocate for the importance of bayesianism for thinking about complex issues if it only works in examples where all the variables are well-defined and have very simple relationships?</span></p></blockquote></div></div><div class="PostsPage-postContent" style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); text-align: left; word-break: break-word;"><div style="box-sizing: inherit;"><blockquote style="border-left: 3px solid rgb(224, 224, 224); box-sizing: inherit; margin: 0px; padding: 16px;"><p style="box-sizing: inherit; margin-bottom: 0px; margin-top: 0px;"><span style="font-family: inherit;">B: I think bayesianism has definitely made a substantial contribution to philosophy. It tells us what it even <i style="box-sizing: inherit;">means</i> to assign a probability to an event, and cuts through a lot of metaphysical bullshit.</span></p></blockquote></div></div></blockquote><span color="rgba(0, 0, 0, 0.87)" style="background-color: white;"><span style="font-family: inherit;"><div><span color="rgba(0, 0, 0, 0.87)" style="background-color: white;"><br /></span></div>Back to the authorial voice. Like Alice, I'm not familiar with any principled or coherent characterisation of what trying to apply bayesianism actually means. It may seem that Alice’s suggestions are deliberately obtuse, but I claim these are the sorts of ideas you’d consider if you seriously tried to consistently “become more bayesian”, rather than just using bayesianism to justify types of reasoning you endorse for other reasons.</span></span></div><div><span style="font-family: inherit;"><span color="rgba(0, 0, 0, 0.870588235294118)"><br /></span><span color="rgba(0, 0, 0, 0.87)" style="background-color: white;">I agree with Bob that the bayesian perspective is useful for thinking about the </span><i style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87);">type signature</i><span color="rgba(0, 0, 0, 0.87)" style="background-color: white;"> of calculating a subjective probability: it’s a function from your prior beliefs and all your evidence to numerical credences, whose quality should be evaluated using a <a href="http://yudkowsky.net/rational/technical">proper scoring rule</a></span><span color="rgba(0, 0, 0, 0.87)" style="background-color: white;">. But for this insight, just like Bob’s insights about using base rates and updating frequently, we don’t need to make any reference to optimality proofs or the idealised limit of </span><s style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87);">intelligence</s><span color="rgba(0, 0, 0, 0.87)" style="background-color: white;"> brute force search. In fact, doing so often provides an illusion of objectivity which is ultimately harmful. I do agree that most things people identify as <a href="https://www.lesswrong.com/posts/AN2cBr6xKWCB8dRQG/what-is-bayesianism">tenets of bayesianism</a> </span><span color="rgba(0, 0, 0, 0.87)" style="background-color: white;">are useful for thinking about knowledge; but I claim that they would be just as useful, and better-justified, if we forced each one to stand or fall on its own.</span></span></div><div><span color="rgba(0, 0, 0, 0.870588235294118)" style="font-family: inherit;"><br /></span></div><div><span style="font-family: inherit;"><span color="rgba(0, 0, 0, 0.870588235294118)"><br /></span><span color="rgba(0, 0, 0, 0.87)" style="background-color: white;">* <a href="https://www.lesswrong.com/posts/tKwJQbo6SfWF2ifKh/toward-a-new-technical-explanation-of-technical-explanation">Abram Demski has posted</a> about </span><span color="rgba(0, 0, 0, 0.87)" style="background-color: white; box-sizing: inherit;"><span style="box-sizing: inherit;">m</span></span></span>o<span style="font-family: inherit;"><span color="rgba(0, 0, 0, 0.87)"><span style="background-color: white;">ving past bayesianism by accounting for logical uncertainty to a greater extent, but I think that arguments similar to the ones I’ve made above are also applicable to logical inductors (although I’m less confident about this).</span></span><br /><span color="rgba(0, 0, 0, 0.87)" style="background-color: white;"><br /></span></span></div><div><span style="font-family: inherit;"><span color="rgba(0, 0, 0, 0.87)" style="background-color: white;">** You can probably fill in your own favourite example of this. The one I was thinking about was a post where someone derived that the probability of extinction from AI was less than 1 in 10^200; but I couldn’t find it.</span> </span></div></div>Richard Ngohttp://www.blogger.com/profile/04825733481608403399noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-60787560719014247362020-10-27T16:48:00.004-07:002021-04-04T13:40:58.520-07:00The Future of Science<p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><i><span style="font-family: inherit;">This is the transcript of a short talk I gave a few months ago, which contains a (fairly rudimentary) presentation of some ideas about the future of science that I've been mulling over for a while. I'm really hoping to develop them much further, since I think this is a particularly important and neglected area of inquiry. <a href="https://www.lesswrong.com/posts/aEZgw6rHssGRqnRFa/the-future-of-science">Cross-posted from Less Wrong</a>; thanks to Jacob Lagerros and David Lambert for editing the transcript, and to various other people for asking thought-provoking questions.</span></i></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">Today I'll be talking about the future of science. Even though this is an important topic (because science is very important) it hasn’t received the attention I think it deserves. One reason is that people tend to think, “Well, we’re going to build an AGI, and the AGI is going to do the science.” But this doesn’t really offer us much insight into what the future of science actually looks like.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">It seems correct to assume that AGI is going to figure a lot of things out. I am interested in what these things are. What is the space of all the things we don’t currently understand? What knowledge is possible? These are ambitious questions. But I’ll try to come up with some framings that I think are interesting.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">One way of framing the history of science is through individuals making observations and coming up with general principles to explain them. So in physics, you observe how things move and how they interact with each other. In biology, you observe living organisms, and so on. I'm going to call this “descriptive science”. More recently, however, we have developed a different type of science, which I'm going to call “generative science”. This involves studying the general principles behind <em style="box-sizing: inherit;">things that don’t exist yet</em>.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">This is harder than descriptive science, because you don't actually have anything to study. You need to bootstrap your way into it. A good example of this is electric circuits. We can come up with fairly general principles for describing how they work. And eventually this led us to computer science, which is again very general. We have a very principled understanding of many aspects of computer science, which is a science of things that didn't exist before we started studying them. I would also contrast this to most types of engineering such as aerospace engineering. I don't think it's principled or general enough to put it in the same class as physics or biology and so on.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">So what would it look like if we took all the existing sciences and made them more generative? For example, in biology, instead of saying, "Here are a bunch of living organisms, how do they work?" you would say, "What are all the different possible ways that you might build living organisms, or what is the space of possible organisms and why did we end up in this particular part of the space on Earth?"</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">Even just from the perspective of understanding how organisms work, this seems really helpful. You understand things in contrast to other things. I don't think we're really going to fully understand how the organisms around us work until we understand why evolution didn't go down all these different paths. And for doing that it's very useful to build those other organisms.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">You could do the same thing with physics. Rather than asking how our universe works, you could ask how an infinite number of other possible universes would work. It seems safe to assume that this would keep people busy for quite a long time.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">Another direction that you could go in is asking how this would carry over to things we don’t currently think of as science. Take sociology, for example. Sociology is not very scientific right now. It's not very good, mostly speaking. But why? And how might it become more scientific in the future? One aspect of this is just that societies are very complicated, and they're composed of minds, which are also very complicated. There are also a lot of emergent effects of those minds interacting with each other, which makes it a total mess.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">So one way of solving this is by having more intelligent scientists. Maybe humans just aren't very good at understanding systems where the base-level components are as intelligent as humans. Maybe you need to have a more intelligent agent studying the system in order to figure out the underlying principles by which it works.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">But another aspect of sociology that makes it really hard to study, and less scientific, is that you can't generate societies to study. You have a hypothesis, but you can't generate a new society to test it. I think this is going to change over the coming decades. You are going to be able to generate systems of agents intelligent enough that they can do things like cultural evolution. And you will be able to study these generated societies as they form. So even a human-level scientist might be able to make a science out of sociology by generating lots of different environments and model societies.</span></p><p><span color="rgba(0, 0, 0, 0.87)" style="background-color: white; font-family: inherit;">The examples of this we've seen so far are super simple but actually quite interesting, like Axelrod's Prisoner's Dilemma tournament or Laland's Social Learning Tournament. There are a couple of things like that which led to really interesting conclusions, despite having really, really basic agents. So I'm excited to see what much more advanced work of this type could look like.</span></p><p><span style="font-family: inherit; font-size: large;">Q&A</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;"><strong style="box-sizing: inherit;">Ben:</strong> Thank you very much, Richard. That was fascinating. So you made this contrast between generative and more descriptive versions of science.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">How much of that set was just a matter of whether or not feedback loops existed in these other spaces? Once we came up with microprocessors, suddenly we were able to build, research, and explore quite a lot of new, more advanced things using science.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">And similarly with the sociology example, you mentioned something along the lines of "We'll potentially get to a place where we can actually just test a lot of these things and then a science will form around this measurement tool." In your opinion, is this a key element in being able to explore new sciences?</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;"><strong style="box-sizing: inherit;">Richard:</strong> Yes. I think feedback loops are pretty useful. I'd say there's probably just a larger space of things in generative sciences. We have these computer architectures, right? So we can study them. But how do we know that the computer architectures couldn't have been totally different? This is not really a question that traditional sciences focus on that much.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">Biologists aren't really spending much of their time asking, "But what if animals had been totally different? What are all the possible ways that you could design a circulatory system, and mitochondria, and things like that?” I think some interesting work is being done that does ask these questions, but it seems like, broadly speaking, there's just a much richer space to explore.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;"><strong style="box-sizing: inherit;">David:</strong> So when you started talking about generative versus descriptive, my initial thought was Schelling's “Micromotives and Macrobehavior” where basically the idea was, “Hey, even if you start with even these pretty basic things, you can figure out how discrimination happens even if people have very slight preferences.” There's a lot of things he did with that, but what strikes me about it is that it was done with very simple individual agents. Beyond that (unless you go all the way to purely rational actor agents, and even then you need lots and lots of caveats and assumptions), you don’t get much in terms of how economics works.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">Even if you can simulate everybody, it doesn't give you much insight. Is that a problem for your idea of how science develops?</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;"><strong style="box-sizing: inherit;">Richard:</strong> So you're saying that if we can simulate everyone given really simple models of them, it still doesn't give us much insight?</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;"><strong style="box-sizing: inherit;">David:</strong> Even when we have <em style="box-sizing: inherit;">complex</em> models of them, we can observe their behavior but we can't do much with it. We can't tell you much that's useful as a result of even pretty good models.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;"><strong style="box-sizing: inherit;">Richard:</strong> I would just say that our models are not very good, right? Broadly speaking, often in economics, it feels something like "we're going to reduce all human preferences to a single dimension but still try to study all the different ways that humans interact; all their friendships and various types of psychological workings and goals and so on".</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">You can collapse all of these things in different ways and then study them, but I don't think we've had models that are anywhere near the complexity of the phenomena that are actually relevant to people's behavior.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;"><strong style="box-sizing: inherit;">David:</strong> But even when they are predictive, even when you can actually replicate what it is that you see with humans, it doesn't seem like you get very much insight into the dynamics... other than saying, "Hey, look, this happens." And sometimes, your assumptions are actually wrong, yet you <em style="box-sizing: inherit;">still</em> recover correct behavior. So overall it didn't tell us very much other than, yes, you successfully replicated what happened.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;"><strong style="box-sizing: inherit;">Richard:</strong> Right. What it seems like to me is that there are lots of interesting phenomena that happen when you have systems of interacting agents in the world. People do a bunch of interesting things. So I think that if you have the ability to recreate that, then you’d have the ability to play around with it and just see in which cases this arises and which cases it doesn't arise.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">Maybe the way I'd characterize it is something like: in our current models, sometimes they're good enough to recreate something that vaguely looks like this phenomenon, but then if you modify it you don't get other interesting phenomena. It's more that they break, I guess. So what would be interesting is the case where you have the ability to model agents that are sophisticated enough, that when you change the inputs <em style="box-sizing: inherit;">away</em> from recreating the behavior that we have observed in humans, you still get some <em style="box-sizing: inherit;">other</em> interesting behavior. Maybe the tit-for-tat agents are a good example of this, where the set-up is pretty simple, but even then you can come up with something that's fairly novel.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;"><strong style="box-sizing: inherit;">Owain:</strong> I think your talk was based on a really interesting premise. Namely that, if we do have AGI in the next 50 years, I think it's plausible that development will be fairly continuous; meaning that on the road to AGI we'll have very powerful, narrow AI that is going to be transformative for science. And I think now is a really good time to think about, in advance, how science could be transformed by this technology.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">Maybe it is an opportunity similar to big science coming out of World War II, or the mathematization of lots of scientific fields in the 20th century that were informal before.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">You brought up one plausible aspect of that: much better ability to run simulations. In particular, simulations of intelligent agents, which are very difficult to run at the moment.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">But you could look at all the aspects of what we do in science and say “how much will narrow AI (that’s still much more advanced than today’s AI) actually help with that?” I think that even with simulations, there are going to be limits, due to its difficulty. Some things are just computationally intractable to simulate. AI's not going to change that. There are NP-hard problems even when simulating very simple physical systems.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">And when you're doing economics or sociology, there are humans, rational agents. You can get better at simulating them. But humans interact with the physical world, right? We create technologies. We suffer natural disasters. We suffer from pandemics. And so, the intractability is going to bite when you're trying to simulate, say, human history or the future of a group of humans. Does that make sense? I am curious about your response.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;"><strong style="box-sizing: inherit;">Richard:</strong> I guess I don't have strong opinions about which bits will be intractable in particular. I think there's probably a lot of space for high-level concepts that we don't currently have. So maybe one way of thinking about this is game theory. Game theory is a pretty limited model in a lot of ways. But it still gives us many valuable concepts like “defecting in a prisoner's dilemma”, and so on, that inform the way that we view complex systems, even though we don't really know exactly what the hypothesis we're evaluating is. Even just having that type of thing brought to our attention is sufficient to reframe the way that we see a lot of things.</span></p><p style="background-color: white; box-sizing: inherit; color: rgba(0, 0, 0, 0.87); margin-bottom: 1em; margin-top: 1em;"><span style="font-family: inherit;">So I guess the thing I'm most excited about is this expansion of concepts. This doesn't feel super intractable because it doesn't feel like you need to simulate anything in its full complexity in order to get the concepts that are going to be really useful going forward.</span></p>Richard Ngohttp://www.blogger.com/profile/04825733481608403399noreply@blogger.com0tag:blogger.com,1999:blog-6642084484458008587.post-4438589998109557902020-10-25T06:30:00.004-07:002020-10-27T17:55:44.433-07:00Reply to Jebari and Lundborg on Artificial Superintelligence<p><span style="font-family: inherit; font-size: 14.6667px; white-space: pre-wrap;">Jebari and Lundborg have recently published an article entitled <i><a href="https://link.springer.com/article/10.1007/s00146-020-01070-3">Artificial superintelligence and its limits: why AlphaZero cannot become a general agent</a></i>. It focuses on the thorny issue of agency in superintelligent AIs. I’m glad to see more work on this crucial topic; however, I have significant disagreements with their terminology and argumentation, as I outline in this reply. Note that it was written rather quickly, and so might lack clarity in some places, or fail to convey some nuances of the original article. I welcome comments and further responses.</span></p><p><span style="font-family: inherit;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;">Their paper runs roughly as follows: Jebari and Lundborg first discuss the belief–desire model for intentional action, under which agents act in ways that they believe will bring about their desires. They then distinguish between different degrees of generality that agents can have: “general agency is, loosely speaking, the ability to act in a diverse set of situations.” They consider thermostats to be very specialised agents, and dogs to be fairly general agents.
They then introduce introduce Legg and Hutter’s definition of intelligence as “an agent’s ability to achieve its goals in a wide range of environments”. This is, unfortunately, a misquotation of Legg and Hutter, and one which leads Jebari and Lundborg astray, as they follow it with:</span></span><span style="font-size: 11pt; font-weight: 700; white-space: pre-wrap;">
</span></span></p><p><span style="font-family: inherit; font-size: 11pt; font-weight: 700; white-space: pre-wrap;">Claim 1: If an agent has very specialized goals or desires, it can be superintelligent with regards to those desires without being a general agent.</span></p><span id="docs-internal-guid-cd2e7956-7fff-dd6a-ffa3-89452d42181e" style="font-family: inherit;"><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;">The actual definition given in both <i>A collection of definitions of intelligence</i> and <i>Universal intelligence</i> is: “Intelligence measures an agent’s ability to achieve goals in a wide range of environments”. Note the important distinction between an agent which can achieve <b>its</b><i style="font-weight: bold;"> </i>goals versus one which can achieve goals in general. In the original context, it is clear that to be highly intelligent according to this definition, an agent doesn’t just need to be able to achieve a single (potentially specialised) set of goals, but rather a wide variety of them! More specifically, in <i>Universal intelligence</i> goals are formalised as a reward function implemented by the environment, with intelligence measured over all computable environments (and therefore all computable reward functions). So Legg and Hutter’s definition does not allow us to call a non-general agent superintelligent. Bostrom, too, defines superintelligence to refer specifically to "intellects that greatly outperform the best current human minds across many very general cognitive domains".</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><br /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;">Perhaps Jebari and Lundborg aren’t making a claim based on Legg and Hutter’s definition, but rather trying to define a new concept: intelligence <i>with respect to a set of desires</i>. But this is still a highly misleading use of the term ‘intelligence’, because there are many possible desires whose achievement doesn’t require any of the cognitive faculties traditionally associated with intelligence (planning, memory, and so on). Consider an agent with a goal that is trivially easy to fulfil in almost all environments - for example, the goal of never taking any actions. Claim 1 implies that, even if this agent has no other cognitive faculties beyond the possession of that goal, it can nevertheless be superintelligent with respect to that goal, because it achieves the goal in almost all environments. Yet why would it be useful to conclude that an agent with nearly no cognitive abilities is superintelligent in any way? Or, perhaps more pithily: if this definition forces us to conclude that rocks could be superintelligent if only they possessed the goal of behaving in a rocklike way, we should suspect that it’s a misuse of the term.</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><br /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;">In their next paragraph, Jebari and Lundborg argue that superintelligence also requires the ability to perform a task better than humans. But this doesn’t salvage claim 1, because there are many tasks on which intelligence is not needed to outperform humans - e.g. both of the examples I discussed in the previous paragraph. Humans, unlike rocks, are notoriously bad at taking no actions; eventually we need to eat and sleep, or just get bored. They also note, in defense of their argument, that “the term “intelligence” has a lot of connotations that are misleading”. But at some point, when you separate a term from all of its connotations, you are merely talking about a different concept. We can already say that objects have the <i>property</i> of x, or a strong <i>ability</i> or <i>capability</i> to do x; we do not need to claim that this is equivalent to them being <i>superintelligent</i> at x.</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><br /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Having said all this, I do agree with them that it’s reasonable and indeed necessary to separate agency and intelligence. I think this will require us to move beyond Legg and Hutter’s definition of intelligence, in a way which I </span><a href="https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ/p/eG3WhHS8CLNxuH6rT" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">discuss briefly here</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> and hope to explicate further soon.</span></p><br /><h2 style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: large;">Ascribing desires to simple agents</span></span></h2><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;">Jebari and Lundborg go on to note: “In many situations, machines acting in our environment are often described as trying to attain some general aim, for example, the goal of a vacuum cleaner robot is often described as “clean a room”… But vacuum cleaner robots are not literally trying to clean a room. Rather they follow a set of specific instructions that often results in cleaning a room.”</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;">This is a very reasonable observation. To me, the implication is that it is not very useful to ascribe goals to vacuum cleaner robots; saying that they have the <i>goal</i> to “follow a set of specific instructions that often results in cleaning a room” adds nothing. In Dennett’s terminology, we can take the design stance towards such objects, and merely say that they have been <i>designed</i> to clean rooms without needing to invoke any goals belonging to the vacuum cleaner itself.</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><br /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;">But Jebari and Lundborg disagree. For example, they claim that “[a] thermostat has no desire with respect to the temperature of the room. It only has a desire to activate the AC if the temperature, as registered by its thermometer, reaches a certain threshold.” They introduce a distinction between these and other desires: they call desires like the ones they ascribe to thermometers and vacuum cleaners <i>unproductive</i> desires.</span></span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;">I should note, however, that this is primarily a semantic dispute. I say that unproductive desires shouldn’t qualify as desires; Jebari and Lundborg think that they are a special type of desire. I think that their terminology is misleading because, like their definition of intelligence, it’s too permissive - it allows us to ascribe unproductive desires to rocks and other inanimate objects. And I expect that if this terminology becomes more common, there’ll be a lot of unnecessary contention between people who use this terminology and people who think that rocks and thermostats don’t really have desires. But it’s not a crucial issue.</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><br /></p><h2 style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;"><span style="font-size: large;"><span style="white-space: pre-wrap;">Two ways of acquiring desires</span></span></h2><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;">A more serious disagreement arises in the subsequent section, where they argue that “an AI with a set of desires constrained to a specific domain cannot acquire desires relevant to other domains”, because “desires can only be reinforced from within”. We should immediately be suspicious of this argument, because it gives rise to a regress problem: how does an AI acquire its first desire in any given domain? In fact, Jebari and Lundborg are conflating two ways in which new desires can arise. The first is that new desires can arise by thinking about ways to achieve old desires; these are often called instrumental desires. But the second is that desires can arise via a modification to the system itself. For example, if I train an AI to play Starcraft, then that AI goes from an initial state in which it has no desires, to a state in which it desires (whether “productively” or “unproductively”) to win at Starcraft, due to the parameter updates implemented by its optimiser.</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;">In this case, I have described the situation as if there is a “policy” agent which chooses what actions to take, and a separate optimiser. But we could also think about this as one system: a single agent which contains a policy module and an optimiser module that updates the policy module. Some sources use the former terminology, some use the latter. Yet in either case, the “agent” starts off without general agency, and might develop general agency during training if the training environment is complex enough. Because of this, the distinction Jebari and Lundborg draw between “spontaneous emergence” and “accidental emergence” of generality seems to merely reflect the arbitrary choice of where we draw the line of what qualifies as an “agent”.</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><br /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span><span style="font-size: 14.6667px; white-space: pre-wrap;">Perhaps Jebari and Lundborg might reply that the combination of a randomly initialised policy and an optimiser already constitutes a general agent, because the optimiser is able to update the policy in a wide range of circumstances. Yet if so, then they are incorrect to claim that Bostrom argues for the possibility of spontaneous emergence of a general agent from a non-general agent. In fact, the example Bostrom uses is of evolutionary search processes, which are a type of optimisation algorithm. So if an optimiser updating a randomly initialised policy constitutes a general agent, then so does Bostrom’s proposed evolutionary search process. But I think that saying that either of these are “general” agents is too much of a stretch; they’re simple, blind optimisation processes that are able to <i>give rise to</i> general agents. I consider this distinction to be crucial in thinking about agency and what behaviour we should expect from superintelligences.</span></span></p></span>Richard Ngohttp://www.blogger.com/profile/04825733481608403399noreply@blogger.com1tag:blogger.com,1999:blog-6642084484458008587.post-40671451214686649012020-04-29T13:09:00.005-07:002023-10-26T13:11:30.904-07:00Melting democracy<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="http://www.icare.cl/assets/uploads/2018/01/bryan-ford-ppt.pdf">Liquid democracy</a>, also known as <a href="https://bford.info/deleg/deleg.pdf">delegative democracy</a>, is a proposed hybrid between direct democracy and representative democracy. In liquid democracy, as in direct democracy, everyone is able to vote directly on every issue. Of course, in a modern society it’d be prohibitively time-consuming for everyone to research every topic well enough to cast an informed vote. In practice, most who vote would end up following the advice of someone better-informed on the topic. Liquid democracy makes this dynamic explicit by allowing any voter to delegate their vote to someone else that they trust. Their delegate can now vote on their behalf, or else delegate again to someone else.<br />
<br />
Here's one possible set of implementation details for how this system could work:<br />
<ul style="text-align: left;">
<li>If A delegates to B, and B delegates to C, then C now has three votes. There’s no limit to how many times votes can be delegated (except that forming a cycle returns a vote to its original owner).</li>
<li>It’s a real-time system: any delegation can be changed at any time. In practice we’d need to implement voting and delegation via a secure digital system.</li>
<li>Voting and delegation would be private for most people, but public for those who have more than a certain number of pledged votes.</li>
<li>People can also split their delegation by subject area, so they don’t need to find a delegate whose judgement they trust on every topic. For example, A could delegate their vote on economic issues to B, their vote on foreign policy to C, and their vote on everything else to D.</li>
</ul>
<br />
I have mixed feelings about liquid democracy. There are quite a few reasons why direct democracy is a bad idea, and liquid democracy only solves some of them. In particular, the role of representatives isn’t just to cast votes, but also to <i>design</i>, <i>advocate</i>, and <i>implement</i> good policies, which is a full-time job. Meanwhile there needs to be some way to make all those policies consistent with each other over time - otherwise voters might (for example) approve a budget constraint, and then approve individual policies expensive enough to overrun that budget constraint. Then, once the policies have been in effect for long enough to judge their success or failure, we also need to know which groups to praise or blame - since allocating responsibility on an individual basis is hard. In short, electing representatives who belong to political parties to serve multi-year terms in parliament seems like a robustly good idea.<br />
<br />
But I also think that the core insights of liquid democracy are very important ones. People are best able to trust others around them, or public figures whose work they respect - not politicians who they mainly see during large-scale media campaigns. A hierarchical system of delegation is a natural way to scale up while maintaining trust in each link. And it’s also true that no representative can have detailed knowledge about every topic, so we should find some way of making specialist knowledge more accessible to our representatives.<br />
<br />
So I’d like to propose a watered-down version of liquid democracy, which preserves these core insights, but without the problems of direct democracy - and which is also much easier to implement gradually. I envisage it happening in two phases.<br />
<br />
<span style="font-size: large;">Phase 1: Melting Democracy</span><br />
<br />
We keep a party-based system of elected representatives, who serve multi-year terms. Everyone can delegate their votes as in liquid democracy, with the same implementation details as described in the previous section (except that people don’t get the opportunity to split their delegation by subject area). Those votes can then be used during parliamentary elections to choose between candidates running for election.<br />
<br />
This may seem like only a modest change. But I think that if delegation is widely adopted, it will improve a number of important dynamics:<br />
<ol style="text-align: left;">
<li><i>Politicians need to run better campaigns</i>. They can’t just use shallow large-scale media tactics, because delegates who have been pledged many votes (let’s call them superdelegates) will have more incentive to investigate platforms and policies in detail. Superdelegates would also have the opportunity to talk to candidates in person, to understand them better.</li>
<li><i>Trusted figures get more involved in politics</i>. Consider a public figure who’s demonstrated good judgement and trustworthiness in their previous work. While their endorsement might be useful for political candidates, they currently have no official power within the political system, which makes it more difficult for them to have a positive influence. Having a large number of pledged votes would give them leverage to push for better policies.</li>
<li><i>A natural funnel into parliament</i>. Running for parliament is difficult not just because of the costs, but also because it’s so abrupt. Candidates need to decide whether to commit to running before knowing how much voters like or trust them. How do we get the best people from other sectors of society to consider a political career? Under melting democracy, they can start gradually getting involved in politics by amassing pledged votes. And they don’t need to spend a lot of money to get started, as long as other superdelegates commit to supporting them.</li>
<li><i>Easier for new parties to get traction</i>. Right now there’s a coordination problem in supporting small parties for first-past-the-post parliamentary seats. Even if many voters prefer a challenger over the two entrenched parties, unless a large number of them switch at once, their votes will probably be wasted. Melting democracy makes this easier, because superdelegates can coordinate more easily to throw support behind small parties.</li>
</ol>
<br />
A potential problem is that we’re centralising voting power in superdelegates, which might open the door to corruption. I hope that this will not be a big problem, because anyone can be a delegate. If a politician is involved in some sort of scandal, voters have to wait several years to vote against them - and even then, if the politician is still backed by their party, there might not be any alternative candidate who’s any better. Whereas if a delegate does something morally or legally dubious, voters can immediately re-delegate to anyone else in the country. This doesn’t have any formal effect until the next election, but it does shift informal power away from them rapidly.<br />
<br />
Meanwhile, there’s also a sense in which we’re decentralising power, because now in addition to political parties we also have a parallel system of delegates. It’s possible that delegates all end up segregating along party lines - but because the assignment of votes to delegates can be so fluid, I expect that many superdelegates will serve as independent voices who are capable of challenging entrenched parties.<br />
<br />
Something else I particularly like about this proposal is that <i>it could start without requiring legal changes</i>. That is, you could initially set up an opt-in, non-binding system for delegating votes. While this first system wouldn’t provide all the benefits I outlined above, pledging a vote to someone would still be a way of giving them more influence, allowing them to push for beneficial policies more effectively, and paving the way for them to run for office later on. Then, if the system proves successful, it could gradually be integrated into the democratic process across different countries. This makes it much more realistic than proposals which would require sudden radical changes.<br />
<br />
<span style="font-size: large;">Phase 2: Functional Constituencies</span><br />
<br />
The specific way that representatives are elected varies by country. In the UK and the US, all representatives need to win their seat in a given constituency or state, where only the votes of residents count. By contrast, New Zealand and Germany use mixed-member proportional (MMP) voting, in which each person gets to vote twice - once for their local MP, and once for a national political party. The resulting parliament is a mix of MPs who represent specific constituencies, and MPs who represent their party as a whole. I think this is an excellent system; it allows voters to select representatives who care about their local issues and are held responsible for them, while also keeping representation proportional to popularity on a national level (unlike <a href="https://www.bbc.co.uk/news/uk-32954807">the UK’s system</a>).<br />
<br />
But if we want representatives who are experts on and accountable for specific issues, then local issues aren’t the only ones which matter. Proponents of liquid democracy are correct in arguing that representatives can’t be experts in every topic they need to legislate on. So what if we added new seats for specific areas of expertise? I don’t think we’d want too many, but it seems valuable to have several representatives who we trust to represent us well on:<br />
<ul style="text-align: left;">
<li>Economic issues</li>
<li>Social issues</li>
<li>Science and technology</li>
<li>Foreign policy</li>
</ul>
<br />
While I’ve framed this proposal in the context of MMP, variants could work in many other political systems; the core idea resembles <a href="https://en.wikipedia.org/wiki/Functional_constituency_(Hong_Kong)">functional constituencies</a> in Hong Kong, and Ireland’s <a href="https://en.wikipedia.org/wiki/Vocational_panel">vocational panels</a>. In contrast to the former, I think it’s important to keep the number of categories low, so that they don’t become de facto safe seats for special interest groups. However, if the system works well then more MPs could be added for each category, and maybe a few more domains included. In particular, I’d be excited to see dedicated seats for people with expertise in improving the governmental system itself.<br />
<br />
In contrast to the Irish system, I think that everyone should be allowed to vote for functional representatives - because even though we’re aiming for expertise, there are also normative elements to each of these constituencies. There are several ways we might allocate votes to functional constituencies. The simplest version would be to allow everyone to vote for a representative in each area, in addition to their national and local votes. Voters could then delegate their six votes together or separately, as in liquid democracy. However, I think that’s asking for too much from voters; and it would encourage people to vote for expert representatives in areas where they can’t judge expertise. Probably it would be more reasonable to allow each person to choose only one of these constituencies in which to vote, in addition to their national and local votes.<br />
<br />
In terms of parliamentary voting power, functional MPs probably wouldn’t have much direct sway. But they would be able to influence other MPs and work with them to create better policies. Right now experts are outsiders to the political process. The core goal of both the proposals I’ve advanced in this essay is to break down the barriers between the people whose judgement we trust in civil society, and the people to whom we’re entrusting the powers and responsibilities of government.</div>
Richard Ngohttp://www.blogger.com/profile/04825733481608403399noreply@blogger.com0