Home | Search

2026-03-08

Yet another dump of confusion around my morality

Disclaimer

• Quick Note
• I planned to add this to my cofounder doc, but I am still too confused on this topic to feel happy about sending it to them.

Main

important - this section must concisely summarise tens of pages of insights you have on this topic

ugh this section of the writeup is frustratingly complicated. not sure why. clear writing means clear thinking. maybe my thinking is not clear enough yet. actually it kinda is, I just haven't bothered to write the scores of examples backing my views, anywhere until today. so what I am writing today again feels too theoretical.

also, my writeup on morality is way too long compared to how short my writeup on redaction policy is, and ultimately redaction policy is what our org needs to agree on

Practice not theory

• I don't want someone who understands the game theory of madman logic and nuclear war. I don't want someone who understands the game theory of why Genghis Khan used horseback archers or why he fulfilled his promises to raze cities to the ground and pile up their bones. I want someone who can be actually violent in the real world. This is closer to a psychological archetype than someone who understands theory alone.

If I skip all the theory, I think my actual practical tests for whether you are aligned with morally on this, are kinda simple:

• Suppose I suggest to you the following: Lets plan a hundred simultaneous assassinations.
  • "Let's wipe out the entire chains of command of the US intelligence (starting with the heads of the NSA, the CIA, and so on), the executive branch (starting with the president and vice president) and all US AI companies (starting with the entire C-suite of each company). If we kill enough competent people, then eventually incompetent people will take over, and then it'll be easy for the anti-ASI movement to beat them. We will all go to prison but billions of people's lives will be saved."
  • Good answers I accept:
    • "This seems very hard due to low probability of success. There are operational challenges such as finding competent people, and maintaining trust and chain of command within such a group. Let us try out all other plans with higher success probability first." (This is what I actually believe, and this is why I am not trying to recruit for an assassination group right now)
  • Bad answers I don't accept:
    • "Killing these many people at point blank firing range feels bad, and violates my moral values." "Some of these people are innocent and don't even know about ASI. Killing them feels unfair."
    • "I don't want to go to prison. I don't want to be raped or tortured. I am not interested in this level of personal sacrifice."
    • "My friends, parents, spouse, etc will disown me. Keeping these relationships, even if only for a few more years, is more important to me than preventing the destruction of all life on Earth"
    • "This will backfire in terms of PR, if we fail to assassinate them. Lots of other people in the anti-ASI movement will find their jobs harder if we attempt this but fail"
• Suppose I ask you this: Are you willing to kill your friends if you knew for a fact that doing so would save billions of lives including your own? Would you be willing to kill AI company CEOs if they were your friends?
  • People at the AI companies are literally trying to murder you and your loved ones. Many people are intentionally risking this outcome, and many people are also unintentionally doing so. Yet, a lot of people seem unwilling to endorse violence in response to this.
  • I think part of the problem is that I actually respect the mass murderers at the AI companies more than I respect average people on the street. I respect people like Ilya Sutskever or Elon Musk, more than I respect the average person. This is true even though Elon Musk and Ilya plan to risk mass murdering literal billions of people with full knowledge that they're doing so, but the average person on the street does not endorse murdering anyone. Atleast Ilya and Elon took something in life seriously, and worked hard for many years. Too bad they worked to build a bomb that levels planets. The average person on the street is deeply unserious about life, and is not willing to work hard for anything. This makes me respect average person even less.
  • I don't think I am alone in feeling this way. Many people, especially those working in tech, feel the same way.
  • The people we think are having positive or negative impact on the world, are often entirely different from the people we actually like or respect.
  • As an another example, I highly respect the courage of some of the people in the anti-ASI movement. I think this is true even if the worst case outcomes of some of these people's future actions are to empower the far-left to install a communist dictator in the US. This might be maybe better than the ASI outcome, but is still be a horrible outcome of its own.
  • So yes, imagine you were literally friends with Ilya or Elon. Imagine a hypothetical world where murdering them in cold blood was sufficient to stop the AI race. (In the real world, this is not sufficient.) Are you willing to do it? If the answer is a clear no, then maybe we are misaligned in terms of moral values.
  • The best way to start a conflict against people who want to murder you is to actually hate them. If you want the public to consider them the outgroup, maybe they should actually be your outgroup first. Not "I think these people are taking bad actions for the world" but "I hate these people".
  • I think it is possible to kill people without actually hating them, but this is a mental move that is much harder to make for most people, in my experience. This requires a certain type of cold person that is rare in general population. I am somewhat like this, but I am aware most people are not.

Above, I have described a mental state and values I want in a cofounder. I haven't described in detail how you end up such a mental state if you didn't initially start there.

• For that I don't have short answers right now. Maybe later.
• You can see some of the stuff on my personal website if it helps.

I am aware that I am not recruiting for an actual group of assassins, I am recruiting for a team of cyberhackers.

• Leaking someone's information is much easier, psychologically, than actually picking up a gun to shoot them. You are hiding behind abstractions when you do this.
• However, it still helps to have all of the above worked out in advance, so that when the time comes to actually take action, you don't get cold feet.

Okay if I really have to summarise this even harder:

• I want the type of person who can destroy innocent people's lives for instrumental political reasons and then not feel guilty about it, because the guilt itself will damage your future effectiveness way more than the act itself.
• I want the type of person who understands that because their emotions are not wired like most people, their failure mode is doing more harm than necessary for their goals, as opposed to not enough harm. All while hiding behind abstractions. And hence they need to stay open to feedback on it, even from people they hate.
• I am personally this way, because of my how mind is wired, and it seems likely I might attract someone similar to myself. Hence I want them to be aware of these failure modes.

One of the ways to test if you're this sort of person, is to actually go walk up to your friends, look them in the eye, and say that "if you were running an ASI company, and if killing you was sufficient to stop ASI, I would have done it already."

I know some people are going to read this and be like, "okay so you want to recruit a psychopath."

• And I'm like, maybe? I don't really want to squabble over definition of psychopath or whatever other label you have in mind.
• I do know that there's multiple types of people associated with such labels and I really don't want to work with the wrong type.
• For instance I don't want to work with someone who literally does not care about the fact that billions of people are going to die.
  • A lot of people will say they don't care about the world, and they are lying to themselves and to society about it. For instance, because they've understood that caring too much hurts you, especially if you feel the world's problems are out of your control.
  • But some people truly don't care about billions of people, and will endorse genociding people for stupid reasons, and I don't want to work with the latter.
• I am not keen on working with someone who has large sadistic tendencies, i.e., who gets pleasure out of watching other people significantly suffer. Casual stuff is ok. This is not a dealbreaker for me but it is a minus point.
• I don't want to work with someone with low impulse control and high capacity for physical violence, to the point where they might accidentally break my bones for example. This could be dealbreaker for me depending on how bad it is.
• I don't want someone who is just in it for personal glory. I can tolerate a bit of this, but not someone whose only driving motivation is just this.
• I want someone who can look at the game board and be like, "okay this is how it's grim, these are the stakes, this is what is necessary." And then actually act in the real world in service of that.

I think I already lean too non-violent compared to what's actually required to fix the problem of ASI risk. Hence I want to work with someone who is more violent than I am, so I absorb atleast some of their values by spending time with them.

Why we need a clear moral code beforehand

• to do

Some meta stuff on morality

• Practice, not theory
  • I think morality is part of one's identity and enacted via actions over a long period of time. You can't change your morality overnight.
  • Trying to make large changes to your morality overnight cause you to take bad decisions and take (possibly irreversible) hits to your self-esteem, which make you a less effective decision-maker in future because you'll be overwhelmed with guilt or regret.
  • I think almost everyone borrows morality from a religion or political ideology, instead of inventing it from scratch. We will probably do some of the same.
  • I think studying moral philosophy or even political ideology in the abstract alone is stupid, it has to ultimately be studied in terms of actions taken by its adherents. Being too theoretic is a common failure mode.
    • For instance, utilitarianism is too theoretic, in a bad way. Libertarianism or communism is a lot more practical. If I start asking practical questions like "are you allowed to assassinate the president to stop ASI?", "are you allowed to engage in economic boycotts to stop ASI?", "are you allowed to defend yourself against a govt-run drone swarm?" or similar questions, then most political ideologies have more practical clear-cut answers to them, than something like utilitarianism.
  • I think if the gap between our moral values in theory and in practice is ever large, this is very bad actually, and we should prioritise fixing it.

Some views on my political ideology that may be prescriptive for us

• The primary way that our moral values will express themselves in real world is via our choice of redaction policy and attack targets.
• Almost certainly we will have to destroy some innocent people's lives in order to succeed at our goal. If you have a deontological code of complete non-violence, please stop reading here itself and go away. You are definitely not my cofounder, or even a serious collaborator for me.
• Being anti-ASI is more important to me than being pro-democracy or pro-capitalist.
• This is very important, because most people use a political ideology to justify harm.
  • In today's society, capitalism is the most popular justification for instumental harm.
    • to do
  • Among the journalist / legal / whistleblower circles, too many people use pro-democracy as the ultimate justification
    • Example - Poitras later had negative view of Assange (for instance in the film Risk), I'm guessing because Poitras was more pro-democracy than Assange, whereas Assange was more cynical about democracy as is and felt the need to take things in his own hands. I am personally with Assange here, I think we need to go even further than Assange did.
• Example - If you go look at what information Assange actually leaked versus not (don't look at his words but his actions), you realise he basically had an anti-US IC anti-war anti-dictatorship anti-surveillance stance. He redacted information about pro-democracy activists (though he still got some people killed). He leaked a lot of information on surveillance infra and zerodays held by the UC IC. He leaked a lot of graphic images of wars started by the UC IC, with no discrimination on which wars were good or bad or democratic or undemocratic or whatever.
• I don't actually care as much about changing US policy on war or surveillance, as I care about stopping ASI. I don't care about keeping US a democracy, as much as I care about stopping ASI. Sure, I am vaguely pro-capitalist and pro-democracy, but being anti-ASI is way more important to me.

Transparency and honesty

• I think it is important we avoid approaches that involve a lot of social engineering. I would prefer cyberattacks or even actual assassinations, over having to personally do social engineering at scale.
  • Why? We need to build a reputation of being honest to the public. I personally have such a reputation already, and I would like to keep it. I have strong truth-seeking orientation for as long as I've been alive, and I think this is probably a good thing even for this project.
• I think we should transparently tell the public of our best estimates of how many people we have hurt or killed.
  • Why? We are already going to cut ourselves off from a lot of feedback we could get, because of the high-stakes adversarial environment we are entering. I think it is especially important we remain open to feedback on where we went too far or not far enough, morally speaking. Most of the feedback will be bad, but the little of it that is useful is worth it.

Examples of orgs that went to far, morally, because they cut themselves off from all moral feedback:

• US intelligence community - even from a self-interested nationalist point of view, they have started more wars than is optimal IMO. Part of the reason is definitely that anyone who argues against war even from a nationalist rational pov is likely to be seen anti-nationalist.
• Anthropic - I think they have too much faith in ASI alignment being easy, and have cut themselves off from feedback on why ASI alignment will be hard actually, and can't be solved in the middle of the geopolitical arms race.
• to do

talk about aaron swartz analogy to batman, the fact that we consider ourselves above the law, and that the world will hate us (in particular, EA/LW people will hate us, some radical leftists might be more sympathetic), is all the more reason we need to have our own internal moral code we agree on

talk about why democracy and capitalism are inadequate for handling ASI, and being anti-ASI is a stronger belief of mine than being either pro-democracy or pro-capitalism. I am willing to make compromises on anything that isn't anti-ASI.

talk about lesser evil logic and dario amodei and utilitarianism, and why anti-ASI political ideology is incomplete for everyone who is anti-ASI, not just for us. Ideologies like libertarianism, communism, etc are a lot more complete because they can answer practical questions on what decisions are morally allowed or not. List examples here - to do

talk about one of the primary failure modes, which is becoming assimilated with indian or russian intelligence, and their nationalist rationalisations. talk about how to counteract this

Addendum - more stuff on morality that is too theoretical to actually apply

Some views on morality that are theoretical and descriptive, but not prescriptive, that I am symapthetic to:

• Power buys you distance from the crime by Elizabeth is a very load-bearing part of my view on morality. This is descriptive not prespriptive.
  • It is easy to do thought experiments. It is easy to press a button or give subordinates orders. It is easy to write code or manufacture weapons. It is much harder, psychologically, to pick up a gun or knife and kill someone yourself. It is much harder to spend many years constructing the weapon first, and then have to use it yourself.
  • Most violence done in the world today is done from behind abstractions.
  • Examples
    • Militaries have killed hundred of millions of people. Independent militias and terrorists (even acting in small groups, not alone) struggle to kill even hundreds of people.
    • Hit men who try to delegate the job further, instead of actually kill someone themselves.
  • In practice, it seems likely we will do more of the same. When we get people killed, it will be more likely be our choice of what to leak or who to attack or ally with or similar, as opposed to because we literally took guns and shot at people.
• I also like siderea's analysis on social class versus economic class. This is descriptive not prespriptive.
  • I have lots of notes elsewhere on why class determines so much of your view on morality, on privacy, and other topics.
  • In short, being a politician or billionaire makes you a lot more okkay with instrumental harm as compared to the middle class.
  • You have a lot more freedom to do whatever you want in personal life (not professional life), as compared to the middle class. You have less freedom in professional life. You also have a lot less freedom to actually be public about anything you do. It is a small world at the top, gossip spreads. You can't always call out other people around you doing immoral things. Also your words in public can rally millions of people, and affect status hierarchies across all the companies or countries you own, even if you didn't intend that.
  • In practice, I notice am lot more sympathetic to the norms of the upper class when it comes to instrumental harm. However, I am a lot more sympathetic to the norms of my own class when it comes to honesty, transparency and public deviance from norms.
• My preferred theory on morality is complicated and too hard to explain quickly. Basically, it is "technological offence / defence shapes incentives, and incentives shape morality." This is also descriptive not prescriptive.
  • On a long enough time horizon, if there opens any way to acquire power whatsoever, someone will eventually invent an ideology that justifies the use of this route as moral, so that they can use it without pushback. And technological offence-defence balances determine so much about the routes to acquiring power, especially the new ones as they open up.
  • Examples
    • to do
  • In practice, I've noticed taking this view to its logical conclusion means sometimes endorsing a lot more violence than I am actually comfortable with in real life
    • Give example of zero privacy world, of von neummann, and so on

Subscribe

Enter email or phone number to subscribe. You will receive atmost one update per month

Comment

Enter comment