Home | Search

2026-05-12

Some conversations I've had on risks from superintelligence

Main

Here are some conversations I've had with people on risks from superintelligence
Only sharing anonymised summaries to protect privacy of these people

Conversation 1

Note - Alice is an anonymous username I picked. Debated in 2025-10. Summarised by me

Overview

Alice and I agreed that ASI was possible to build and dangerous to build. This was mostly a timelines debate.
Even after the debate, I don't think Alice and I converged much

Timelines

Samuel: P(ASI by 2030) = 0.25. Outside view of other experts, inside view - model scaling, RL scaling, serial speedup
Alice: P(ASI by 2030) = 0.05. Non-LLM ASI possible, no longterm goal pursuit, bad grounding, bad grounding means bad generalisation

Can AI pursue Goals Longterm

Samuel: Humans want AI to purse longterm goals, RL starting to pursue longterm goals. Priors based on past data: Problems that start to get solved get fully solved soon, brute force outperformed human experts
Alice: In theory agree, in practice RL/Inference compute required may be too high

RL Scaling Compute

Alice: $1M spent so far, risk appetite may not exists
Samuel: $1B risk appetite exists, but not spent yet, maybe technical bottleneck

Brute Force versus Human Ground Truth

Samuel: Humans learn from bad grounding, pretraining datasets are bad grounding, good world models can make use of bad grounding. Better architectures may reduce compute requirements.
Alice: Hard to bootstrap initial good world models, human grounding is the bottleneck for this, generalisation requires grounding

Humans have longterm goals. Inner Reward, Outer Reward, Planning

Alice: Speed is not the bottleneck but grounding is. In humans, Amyglada drives neocortex to do longterm goal pursuit. In AI, we have built neocortex not amyglada.
Samuel: In humans, internal reward system done by limbic system, but planning done by neocortex. In AI, internal reward system easy to build because low info processing, planning hard to build.
Alice: Building good reward system is hard and may need human experts
Samuel: Outer reward system is simple, inner reward system is emergent not built
Alice: Pretraining dataset and RL reward signals are both grounding. Whether you use pretuning or RL, you may require human experts to do grounding.
Alice: RL early days, hard to forecast. Evolution used simple outer reward and emergent inner reward, but required lot of compute.

Can Gradient Descent learn Complexity without Human-curated Datasets or Rewards

Alice: To get ASI, either good inner rewards are emergent despite simple grounding, or we brute force search despite bad inner rewards and bad grounding, or some unknown unknown path.
Samuel: Chess/Starcraft have emergent inner rewards
Alice: Chess state space is small. Collapsing large state space of reality into manageable size requires good world models which requires grounding to build.
Alice: RLHF was a surrogate grounding provided by humans. Can we build GPT3.5 without it? Samuel: Probably yes, I predict good inner rewards can be emergent not human-curated.
Alice: Both of us could make more concrete predictions like this

Can RL invent Multiple New World Models?

Alice: AI today imitates human world models like child imitates parent. AI not like bacteria that can be thrown into new environment and learn to survive on its own.
Samuel: Let's discuss this in a specific domain. Alice: AI physicist deciding which lab experiments to do
Samuel: Once AI physicist has absorbed known world models for known data, it has hypothesise multiple new world models to explain unexplained data, propose experiments, learn from experiment runs. Can explain how transformer + RL might crack this.
Debate paused

Conversation 2

Note - Bob is an anonymous username I picked. Discussed in 2025-05. Summarised by me

Overview of the conversation

Samuel thinks ASI is possible to build and dangerous to build. Bob was focussed on which new AI startups may be successful.
Even after the debate, it was clear that I was mostly focussed on the frontier capabilities being the most important thing. If you don't build in catastrophic outcomes, you should probably just push the frontier. Bob was instead more focussed on a worldview where frontier research is not the only important thing, efficiently using the capabilities we already have is also important.

Which part of inference stack is important

Samuel and Bob agree: current frontier AI companies focus on raw capabilities to raise more funding, none of them are actually profitable, none of them are focussed on efficiently using compute by writing low-level code, maybe a smaller team in the company or smaller independent companies are focussed on using compute efficiently
Samuel: if a new AI research paradigm works, then optimising low-level code for current paradigm may be wasted effort
Bob: Groq and cerebrum are focussed on efficiency of inference
Samuel and Bob agree: different AI companies will raise funding based on different paradigms, some will promise raw AI capabilities to investors, some will promise compute efficiency
Bob: Optimisations being made in heat / temperature of compute cluster, also compute clusters being built next to power plants to for efficient energy transfer
Samuel and Bob agree: In the long run, all AI inference will be running using low-level C code or even lower level like optimising ASICs / SMBs
Bob: Deepseek being focussed on compute efficiency might inspire other companies to also focus on it. Samuel disagrees

Do Nvidia and TSMC have competitors

Samuel: People at Nvidia and TSMC have latest knowledge. Govts are involved, it is not obvious that US companies can hire Taiwanese or Chinese researchers at TSMC and Nvidia without permission from Taiwanese or Chinese govts.
Bob: From the book Focus - The ASML way, he learned that Taiwanese prioritise one person to be the world expert in every sub process required to manufacture chips. For instance one person could be the world expert just on making screws.
Bob: Work ethic and culture of these people is the most important thing.
Samuel and Bob agree: Taiwan investing in semiconductors and nothing else since 70s was a crazy bet at the time, but has now produced outsized returns.

Indian AI companies and data cleanup

Bob: Gave examples, like Indian companies have built compute clusters in Gujarat and Tamil Nadu, Indian judicary attempting to reduce court case using AI, Indian banks providing microloans of 50k INR to 5 lakh INR and tracking how people use the loaned money using AI might enable you to trust them with more money
Samuel: From the Sarah Constantin's article great data integration, many companies don't record the data they need to feed AI, or its in a bad file format for AI use, data needs cleaning. This problem will appear in many such use cases.
Bob: Agree streamlining of processes required, even for basic things like machines to scan larger number of pages quickly.
Samuel and Bob agree: Converting all file formats to plaintext is not that hard, but someone has to write good software for it

Which AI startups will succeed?

Samuel: Whole space is very new, AI capabilities increased very recently
Bob: Many startups will succeed in many different niches. Samuel agrees
Samuel: Gave example of one use case, Bob said it might be built as demo but not be economically viable for a startup

Conversation 3

Note - Carol is an anonymous username I picked. Discussed in 2025-05. Summarised by me

Overview

Samuel and Carol both thinks superintelligence may be built soon and may be dangerous. Carol is even more bullish on AI capabilities than Samuel and has shorter timelines. This is a timelines debate between two people whose timelines already agree significantly.

Top-level views

Samuel top-level: 25% AI!2030 better or equal to ASI, less than 50% ASI much better than AI!2030 much better than AI!2025, ~25% AI!2030 ~= AI!2025
Carol top-level: medium probability AI!2030 better or equal to ASI
Samuel bullish on model scaling, more uncertain on RL scaling
Carol bullish on RL/inference scaling, Carol bullish on grokking
Samuel: does bullish on grokking mean bullish on model scaling. Carol: unsure

Agreements

Samuel and Carol agree: only 2024-2025 counts as empirical data to extrapolate RL/inference scaling trend. (o1, o3, deepseek r1, deepseek r0). RLHF done on GPT3.5 not a valid datapoint on this trend.
Carol and Samuel agree: if superhuman mathematician and physicist are built, high likelihood we get ASI (so robotics and other tasks also get solved). robotics progress is not a crux. crux: how good is scaling RL for LLM?
Carol is more certain as being bullish on scaling RL for LLM, Samuel has wider uncertainty on it.
Testable hypothesis: Carol claims GPT3 + lots of RL in 2025 ~= GPT4. Carol claims GPT2-size model trained in 2025 + high quality data + lots of RL in 2025 ~= GPT3. Samuel disagrees. need top ML labs to try this stuff more.
Testable hypothesis: Carol claims models such as qwen 2.5 coder are below 50B params but better than GPT3 175B and almost as good as GPT4 1.4T. Samuel disagrees and claims overfit to benchmark. Samuel needs to try below 50B param models on tests not in benchmarks.
Testable hypothesis: Samuel thinks small model being trained on big model leads it to overfit benchmark. Carol unsure. Samuel and Carol need to try such models on tests not in benchmarks.

Enter email or phone number to subscribe. You will receive atmost one update per month

Comment