LessWrong (Curated & Popular) podcast

11:13

I recently wrote about complete feedback, an idea which I think is quite important for AI safety. However, my note was quite brief, explaining the idea only to my closest research-friends. This post aims to bridge one of the inferential gaps to that idea. I also expect that the perspective-shift described here has some value on its own. In classical Bayesianism, prediction and evidence are two different sorts of things. A prediction is a probability (or, more generally, a probability distribution); evidence is an observation (or set of observations). These two things have different type signatures. They also fall on opposite sides of the agent-environment division: we think of predictions as supplied by agents, and evidence as supplied by environments. In Radical Probabilism, this division is not so strict. We can think of evidence in the classical-bayesian way, where some proposition is observed and its probability jumps to 100%. [...] --- Outline: (02:39) Warm-up: Prices as Prediction and Evidence (04:15) Generalization: Traders as Judgements (06:34) Collector-Investor Continuum (08:28) Technical Questions The original text contained 3 footnotes which were omitted from this narration. The original text contained 1 image which was described by AI. --- First published: February 23rd, 2025 Source: https://www.lesswrong.com/posts/3hs6MniiEssfL8rPz/judgements-merging-prediction-and-evidence --- Narrated by TYPE III AUDIO . --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts , or another podcast app.…

1
“The Sorry State of AI X-Risk Advocacy, and Thoughts on Doing Better” by Thane Ruthenis 12:31

hace 5 días12:31

12:31

First, let me quote my previous ancient post on the topic: Effective Strategies for Changing Public Opinion The titular paper is very relevant here. I'll summarize a few points. The main two forms of intervention are persuasion and framing. Persuasion is, to wit, an attempt to change someone's set of beliefs, either by introducing new ones or by changing existing ones. Framing is a more subtle form: an attempt to change the relative weights of someone's beliefs, by empathizing different aspects of the situation, recontextualizing it. There's a dichotomy between the two. Persuasion is found to be very ineffective if used on someone with high domain knowledge. Framing-style arguments, on the other hand, are more effective the more the recipient knows about the topic. Thus, persuasion is better used on non-specialists, and it's most advantageous the first time it's used. If someone tries it and fails, they raise [...] --- Outline: (02:23) Persuasion (04:17) A Better Target Demographic (08:10) Extant Projects in This Space? (10:03) Framing The original text contained 3 footnotes which were omitted from this narration. --- First published: February 21st, 2025 Source: https://www.lesswrong.com/posts/6dgCf92YAMFLM655S/the-sorry-state-of-ai-x-risk-advocacy-and-thoughts-on-doing --- Narrated by TYPE III AUDIO .…

1
“Power Lies Trembling: a three-book review” by Richard_Ngo 27:11

hace 6 días27:11

27:11

In a previous book review I described exclusive nightclubs as the particle colliders of sociology—places where you can reliably observe extreme forces collide. If so, military coups are the supernovae of sociology. They’re huge, rare, sudden events that, if studied carefully, provide deep insight about what lies underneath the veneer of normality around us. That's the conclusion I take away from Naunihal Singh's book Seizing Power: the Strategic Logic of Military Coups. It's not a conclusion that Singh himself draws: his book is careful and academic (though much more readable than most academic books). His analysis focuses on Ghana, a country which experienced ten coup attempts between 1966 and 1983 alone. Singh spent a year in Ghana carrying out hundreds of hours of interviews with people on both sides of these coups, which led him to formulate a new model of how coups work. I’ll start by describing Singh's [...] --- Outline: (01:58) The revolutionary's handbook (09:44) From explaining coups to explaining everything (17:25) From explaining everything to influencing everything (21:40) Becoming a knight of faith The original text contained 3 images which were described by AI. --- First published: February 22nd, 2025 Source: https://www.lesswrong.com/posts/d4armqGcbPywR3Ptc/power-lies-trembling-a-three-book-review --- Narrated by TYPE III AUDIO . --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts ,…

1
“Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs” by Jan Betley, Owain_Evans 7:58

hace 6 días7:58

7:58

This is the abstract and introduction of our new paper. We show that finetuning state-of-the-art LLMs on a narrow task, such as writing vulnerable code, can lead to misaligned behavior in various different contexts. We don't fully understand that phenomenon. Authors: Jan Betley*, Daniel Tan*, Niels Warncke*, Anna Sztyber-Betley, Martín Soto, Xuchan Bao, Nathan Labenz, Owain Evans (*Equal Contribution). See Twitter thread and project page at emergent-misalignment.com. Abstract We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding: it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment. This effect is observed in a range [...] --- Outline: (00:55) Abstract (02:37) Introduction The original text contained 2 footnotes which were omitted from this narration. The original text contained 1 image which was described by AI. --- First published: February 25th, 2025 Source: https://www.lesswrong.com/posts/ifechgnJRtJdduFGC/emergent-misalignment-narrow-finetuning-can-produce-broadly --- Narrated by TYPE III AUDIO . --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts , or another podcast app.…

1
“The Paris AI Anti-Safety Summit” by Zvi 42:06

hace 9 días42:06

42:06

It doesn’t look good. What used to be the AI Safety Summits were perhaps the most promising thing happening towards international coordination for AI Safety. This one was centrally coordination against AI Safety. In November 2023, the UK Bletchley Summit on AI Safety set out to let nations coordinate in the hopes that AI might not kill everyone. China was there, too, and included. The practical focus was on Responsible Scaling Policies (RSPs), where commitments were secured from the major labs, and laying the foundations for new institutions. The summit ended with The Bletchley Declaration (full text included at link), signed by all key parties. It was the usual diplomatic drek, as is typically the case for such things, but it centrally said there are risks, and so we will develop policies to deal with those risks. And it ended with a commitment [...] --- Outline: (02:03) An Actively Terrible Summit Statement (05:45) The Suicidal Accelerationist Speech by JD Vance (14:37) What Did France Care About? (17:12) Something To Remember You By: Get Your Safety Frameworks (24:05) What Do We Think About Voluntary Commitments? (27:29) This Is the End (36:18) The Odds Are Against Us and the Situation is Grim (39:52) Don't Panic But Also Face Reality The original text contained 4 images which were described by AI. --- First published: February 12th, 2025 Source: https://www.lesswrong.com/posts/qYPHryHTNiJ2y6Fhi/the-paris-ai-anti-safety-summit --- Narrated by TYPE III AUDIO . --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try…

1
“Eliezer’s Lost Alignment Articles / The Arbital Sequence” by Ruby 2:37

hace 11 días2:37

2:37

Note: this is a static copy of this wiki page. We are also publishing it as a post to ensure visibility. Circa 2015-2017, a lot of high quality content was written on Arbital by Eliezer Yudkowsky, Nate Soares, Paul Christiano, and others. Perhaps because the platform didn't take off, most of this content has not been as widely read as warranted by its quality. Fortunately, they have now been imported into LessWrong. Most of the content written was either about AI alignment or math[1]. The Bayes Guide and Logarithm Guide are likely some of the best mathematical educational material online. Amongst the AI Alignment content are detailed and evocative explanations of alignment ideas: some well known, such as instrumental convergence and corrigibility, some lesser known like epistemic/instrumental efficiency, and some misunderstood like pivotal act. The Sequence The articles collected here were originally published as wiki pages with no set [...] --- Outline: (01:01) The Sequence (01:23) Tier 1 (01:32) Tier 2 The original text contained 3 footnotes which were omitted from this narration. --- First published: February 20th, 2025 Source: https://www.lesswrong.com/posts/mpMWWKzkzWqf57Yap/eliezer-s-lost-alignment-articles-the-arbital-sequence --- Narrated by TYPE III AUDIO .…

1
“Arbital has been imported to LessWrong” by RobertM, jimrandomh, Ben Pace, Ruby 8:52

hace 11 días8:52

8:52

Arbital was envisioned as a successor to Wikipedia. The project was discontinued in 2017, but not before many new features had been built and a substantial amount of writing about AI alignment and mathematics had been published on the website. If you've tried using Arbital.com the last few years, you might have noticed that it was on its last legs - no ability to register new accounts or log in to existing ones, slow load times (when it loaded at all), etc. Rather than try to keep it afloat, the LessWrong team worked with MIRI to migrate the public Arbital content to LessWrong, as well as a decent chunk of its features. Part of this effort involved a substantial revamp of our wiki/tag pages, as well as the Concepts page. After sign-off[1] from Eliezer, we'll also redirect arbital.com links to the corresponding pages on LessWrong. As always, you are [...] --- Outline: (01:13) New content (01:43) New (and updated) features (01:48) The new concepts page (02:03) The new wiki/tag page design (02:31) Non-tag wiki pages (02:59) Lenses (03:30) Voting (04:45) Inline Reacts (05:08) Summaries (06:20) Redlinks (06:59) Claims (07:25) The edit history page (07:40) Misc. The original text contained 3 footnotes which were omitted from this narration. The original text contained 10 images which were described by AI. --- First published: February 20th, 2025 Source: https://www.lesswrong.com/posts/fwSnz5oNnq8HxQjTL/arbital-has-been-imported-to-lesswrong --- Narrated by TYPE III AUDIO . --- Images from the article:…

1
“How to Make Superbabies” by GeneSmith, kman 1:08:04

hace 12 días1:08:04

1:08:04

We’ve spent the better part of the last two decades unravelling exactly how the human genome works and which specific letter changes in our DNA affect things like diabetes risk or college graduation rates. Our knowledge has advanced to the point where, if we had a safe and reliable means of modifying genes in embryos, we could literally create superbabies. Children that would live multiple decades longer than their non-engineered peers, have the raw intellectual horsepower to do Nobel prize worthy scientific research, and very rarely suffer from depression or other mental health disorders. The scientific establishment, however, seems to not have gotten the memo. If you suggest we engineer the genes of future generations to make their lives better, they will often make some frightened noises, mention “ethical issues” without ever clarifying what they mean, or abruptly change the subject. It's as if humanity invented electricity and decided [...] --- Outline: (02:17) How to make (slightly) superbabies (05:08) How to do better than embryo selection (08:52) Maximum human life expectancy (12:01) Is everything a tradeoff? (20:01) How to make an edited embryo (23:23) Sergiy Velychko and the story of super-SOX (24:51) Iterated CRISPR (26:27) Sergiy Velychko and the story of Super-SOX (28:48) What is going on? (32:06) Super-SOX (33:24) Mice from stem cells (35:05) Why does super-SOX matter? (36:37) How do we do this in humans? (38:18) What if super-SOX doesn't work? (38:51) Eggs from Stem Cells (39:31) Fluorescence-guided sperm selection (42:11) Embryo cloning (42:39) What if none of that works? (44:26) What about legal issues? (46:26) How we make this happen (50:18) Ahh yes, but what about AI? (50:54) There is currently no backup plan if we can't solve alignment (55:09) Team Human (57:53) Appendix (57:56) iPSCs were named after the iPod (58:11) On autoimmune risk variants and plagues (59:28) Two simples strategies for minimizing autoimmune risk and pandemic vulnerability (01:00:29) I don't want someone else's genes in my child (01:01:08) Could I use this technology to make a genetically enhanced clone of myself? (01:01:36) Why does super-SOX work? (01:06:14) How was the IQ grain graph generated? The original text contained 19 images which were described by AI. --- First published: February 19th, 2025 Source: https://www.lesswrong.com/posts/DfrSZaf3JC8vJdbZL/how-to-make-superbabies --- Narrated by TYPE III AUDIO . --- Images from the article:…

1
“A computational no-coincidence principle” by Eric Neyman 13:28

hace 12 días13:28

13:28

Audio note: this article contains 134 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description. In a recent paper in Annals of Mathematics and Philosophy, Fields medalist Timothy Gowers asks why mathematicians sometimes believe that unproved statements are likely to be true. For example, it is unknown whether _pi_ is a normal number (which, roughly speaking, means that every digit appears in _pi_ with equal frequency), yet this is widely believed. Gowers proposes that there is no sign of any reason for _pi_ to be non-normal -- especially not one that would fail to reveal itself in the first million digits -- and in the absence of any such reason, any deviation from normality would be an outrageous coincidence. Thus, the likely normality of _pi_ is inferred from the following general principle: No-coincidence [...] --- Outline: (02:32) Our no-coincidence conjecture (05:37) How we came up with the statement (08:31) Thoughts for theoretical computer scientists (10:27) Why we care The original text contained 12 footnotes which were omitted from this narration. --- First published: February 14th, 2025 Source: https://www.lesswrong.com/posts/Xt9r4SNNuYxW83tmo/a-computational-no-coincidence-principle --- Narrated by TYPE III AUDIO .…

1
“A History of the Future, 2025-2040” by L Rudolf L 2:22:38

hace 13 días2:22:38

2:22:38

This is an all-in-one crosspost of a scenario I originally published in three parts on my blog (No Set Gauge). Links to the originals: A History of the Future, 2025-2027 A History of the Future, 2027-2030 A History of the Future, 2030-2040 Thanks to Luke Drago, Duncan McClements, and Theo Horsley for comments on all three parts. 2025-2027 Below is part 1 of an extended scenario describing how the future might go if current trends in AI continue. The scenario is deliberately extremely specific: it's definite rather than indefinite, and makes concrete guesses instead of settling for banal generalities or abstract descriptions of trends. Open Sky. (Zdislaw Beksinsksi) The return of reinforcement learning From 2019 to 2023, the main driver of AI was using more compute and data for pretraining. This was combined with some important "unhobblings": Post-training (supervised fine-tuning and reinforcement learning for [...] --- Outline: (00:34) 2025-2027 (01:04) The return of reinforcement learning (10:52) Codegen, Big Tech, and the internet (21:07) Business strategy in 2025 and 2026 (27:23) Maths and the hard sciences (33:59) Societal response (37:18) Alignment research and AI-run orgs (44:49) Government wakeup (51:42) 2027-2030 (51:53) The AGI frog is getting boiled (01:02:18) The bitter law of business (01:06:52) The early days of the robot race (01:10:12) The digital wonderland, social movements, and the AI cults (01:24:09) AGI politics and the chip supply chain (01:33:04) 2030-2040 (01:33:15) The end of white-collar work and the new job scene (01:47:47) Lab strategy amid superintelligence and robotics (01:56:28) Towards the automated robot economy (02:15:49) The human condition in the 2030s (02:17:26) 2040+ --- First published: February 17th, 2025 Source: https://www.lesswrong.com/posts/CCnycGceT4HyDKDzK/a-history-of-the-future-2025-2040 --- Narrated by TYPE III AUDIO . --- Images from the article:…

1
“It’s been ten years. I propose HPMOR Anniversary Parties.” by Screwtape 1:54

hace 13 días1:54

1:54

On March 14th, 2015, Harry Potter and the Methods of Rationality made its final post. Wrap parties were held all across the world to read the ending and talk about the story, in some cases sparking groups that would continue to meet for years. It's been ten years, and think that's a good reason for a round of parties. If you were there a decade ago, maybe gather your friends and talk about how things have changed. If you found HPMOR recently and you're excited about it (surveys suggest it's still the biggest on-ramp to the community, so you're not alone!) this is an excellent chance to meet some other fans in person for the first time! Want to run an HPMOR Anniversary Party, or get notified if one's happening near you? Fill out this form. I’ll keep track of it and publish a collection of [...] The original text contained 1 footnote which was omitted from this narration. --- First published: February 16th, 2025 Source: https://www.lesswrong.com/posts/KGSidqLRXkpizsbcc/it-s-been-ten-years-i-propose-hpmor-anniversary-parties --- Narrated by TYPE III AUDIO .…

1
“Some articles in ‘International Security’ that I enjoyed” by Buck 7:56

hace 15 días7:56

7:56

A friend of mine recently recommended that I read through articles from the journal International Security, in order to learn more about international relations, national security, and political science. I've really enjoyed it so far, and I think it's helped me have a clearer picture of how IR academics think about stuff, especially the core power dynamics that they think shape international relations. Here are a few of the articles I most enjoyed. "Not So Innocent" argues that ethnoreligious cleansing of Jews and Muslims from Western Europe in the 11th-16th century was mostly driven by the Catholic Church trying to consolidate its power at the expense of local kingdoms. Religious minorities usually sided with local monarchs against the Church (because they definitionally didn't respect the church's authority, e.g. they didn't care if the Church excommunicated the king). So when the Church was powerful, it was incentivized to pressure kings [...] --- First published: January 31st, 2025 Source: https://www.lesswrong.com/posts/MEfhRvpKPadJLTuTk/some-articles-in-international-security-that-i-enjoyed --- Narrated by TYPE III AUDIO .…

1
“The Failed Strategy of Artificial Intelligence Doomers” by Ben Pace 8:39

hace 15 días8:39

8:39

This is the best sociological account of the AI x-risk reduction efforts of the last ~decade that I've seen. I encourage folks to engage with its critique and propose better strategies going forward. Here's the opening ~20% of the post. I encourage reading it all. In recent decades, a growing coalition has emerged to oppose the development of artificial intelligence technology, for fear that the imminent development of smarter-than-human machines could doom humanity to extinction. The now-influential form of these ideas began as debates among academics and internet denizens, which eventually took form—especially within the Rationalist and Effective Altruist movements—and grew in intellectual influence over time, along the way collecting legible endorsements from authoritative scientists like Stephen Hawking and Geoffrey Hinton. Ironically, by spreading the belief that superintelligent AI is achievable and supremely powerful, these “AI Doomers,” as they came to be called, inspired the creation of OpenAI and [...] --- First published: January 31st, 2025 Source: https://www.lesswrong.com/posts/YqrAoCzNytYWtnsAx/the-failed-strategy-of-artificial-intelligence-doomers --- Narrated by TYPE III AUDIO .…

1
“Murder plots are infohazards” by Chris Monteiro 3:58

hace 17 días3:58

3:58

Hi all I've been hanging around the rationalist-sphere for many years now, mostly writing about transhumanism, until things started to change in 2016 after my Wikipedia writing habit shifted from writing up cybercrime topics, through to actively debunking the numerous dark web urban legends. After breaking into what I believe to be the most successful ever fake murder for hire website ever created on the dark web, I was able to capture information about people trying to kill people all around the world, often paying tens of thousands of dollars in Bitcoin in the process. My attempts during this period to take my information to the authorities were mostly unsuccessful, when in late 2016 on of the site a user took matters into his own hands, after paying $15,000 for a hit that never happened, killed his wife himself Due to my overt battle with the site administrator [...] --- First published: February 13th, 2025 Source: https://www.lesswrong.com/posts/isRho2wXB7Cwd8cQv/murder-plots-are-infohazards --- Narrated by TYPE III AUDIO .…

1
“Why Did Elon Musk Just Offer to Buy Control of OpenAI for $100 Billion?” by garrison 11:41

hace 20 días11:41