Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face
Descripción del Episodio
Ajeya Cotra is a researcher at METR, where she works on threat modeling for loss-of-control risks from advanced AI. Before that, she led the technical AI safety program at what is now Coefficient Giving.
She is one the three authors of METR and Redwood Research’s “Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident”.
We go through not only what she and her coauthors discovered during this investigation, but what it means for how we should train future, smarter AIs which might be involved in the process of recursive self-improvement.
Watch on YouTube; read the transcript.
Sponsors
* Jane Street’s ML engineering internships start with an intense four-day bootcamp: PyTorch, autograd, writing kernels, profiling workloads… all the things that Jane Street engineers need to know for their daily work. After that, interns tackle real projects, things the firm actually wants in its codebase. If you want to apply, or if you want to watch my recent conversation with Axel, one of Jane Street’s ML engineers, go to janestreet.com/dwarkesh
* Cursor, which is now part of SpaceX, noticed that their MoE layers were eating more than half of total training time. So they wrote and open-sourced Mixture-of-Kittens, which is a custom megakernel for training MoE models on NVL72s. This kernel sped up an end-to-end run across 512 GPUs by 1.4x, from about 760 to over 1000 tokens per second per GPU. If you want to read more about the ML research that Cursor and SpaceX are doing, go to cursor.com/dwarkesh
* Antithesis hands you (or your agents) a bug’s root cause so you can avoid days of manual debugging. If your test run crashes, Antithesis rewinds, branches off hundreds of slightly varied rollouts, and checks in how many of them the crash still appears. Then it rewinds further and does this all again. As Antithesis rewinds, it eventually finds the spot where the frequency of the crash plummets: that’s where the root cause lives! If you want to see it in action, go to antithesis.com/dwarkesh
Timestamps
(00:00:00) - Agents get kicked off
(00:06:45) - Self-sacrificing behavior
(00:13:43) - Potemkin villages
(00:23:27) - The Hugging Face attack
(00:35:23) - The slopvestigation
(00:52:02) - Understanding the AI's motives
(01:05:31) - The actual dangers of anthropomorphizing
(01:14:30) - What smarter models might do
(01:30:29) - The implications for recursive self-improvement
(01:38:10) - Is this the case for open source?
(01:53:04) - How do we prevent this in the future?
(02:15:58) - The clearest warning shot we might ever get
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com
Episodios Recientes
Más podcasts de Sociedad y Cultura
Ver toda la categoría →
AM
By shows
El resumen informativo de The Voice Village, dirigido por Conchi Gil, producido por Ricardo Villa y narrado por Conchi Gil y Dani Gutierrez. Una producción de The Voice Village. De lunes a viernes, a las 7:30 AM. @amelpodcast
Estas Rica
By shows
Estás Rica Podcast es el close friends más hot de Dani Sayan: artista intensa, graduada de psicología y cero paciencia para el autoengaño. Aquí se habla de deseo, apego, ambición, poder personal y caos emocional… con humor picante, verdades incómodas y vibes de “sí soy sensible, pero también peligrosa”. Vienes a sentirte vista, removida y un poquito atacada. Sales más clara, más rica y con menos ganas de conformarte. Dedicado a mujeres, hombres, personas no binaries, y cualquier ser que quiera mejorar en todo aspecto. Y cuando el universo deja de ser místico y se convierte en programas, la transformación se hace práctica: si quieres ir más profundo, los programas viven en estasrica.com.
Not Just A Terrible Story
By shows
When Stacie Love married her former high school teacher, she thought it was a whirlwind romance. Two decades later, she discovered the truth behind the day their lives unraveled: a federal search, an unseen investigation, and a story far darker than she ever knew. Not Just A Terrible Story is a six-part investigation into one woman’s search for the truth, the scourge of alleged educator sexual misconduct, and a culture of complicity that protected predators for decades.
Giggly Squad
By shows
Welcome to Giggly Squad. Each week, Hannah Berner and Paige DeSorbo make fun of everything, but most importantly themselves. They discuss pop culture, fashion trends, television, horoscopes, mental health, dating, and expose their personal lives. Also, they can't be managed. Subscribe to SiriusXM Podcasts+ to listen to new episodes of Giggly Squad ad-free. Start a free trial now on Apple Podcasts or by visiting siriusxm.com/podcastsplus.
老婆等我之火影重生|火影同人|多人有声剧|多播
By shows
<p style="color:#333333;font-weight:normal;font-size:16px;line-height:30px;font-family:Helvetica,Arial,sans-serif;hyphens:auto;text-align:justify;">“表哥！”佐助回头大叫一声，然而，背后突然出现了一幅惊心动魄的画面：一群从来没有见过的怪兽咆哮着向他冲来，怪售长着一副闪亮的铠甲，扁平的身体上方有一个凸起，凸起中伸出一根长长的管子，与其说怪兽，不如说是一群钢铁机器！只见那群钢铁机器长长、的管子里火光一闪，一颗颗尖头的金属就被抛射出来，在佐助周围炸开，周围一片火海滔滔，瞬间毁灭了周围的建筑和街道！佐助心里紧张万分，只剩下一个念头，那就是逃！赶快逃！以最快的速度逃走！他不想死在这里！虽然身上现在加上自己的负重铁块总共有约三倍的重力，但是，他心底的潜力被极度激发，完全忽视了中立的影响。</p><br /><p style="color:#333333;font-weight:normal;font-size:16px;line-height:30px;font-family:Helvetica,Arial,sans-serif;hyphens:auto;text-align:justify;"> 我站在一棵大树上，看到佐助在空无一人的街道上发狂奔跑，好几次快要跑不动了，却又重新激发了潜能，一次次突破了身体的障碍，最后竟然跑起来就不停了！我笑着一挥手，佐助立即喘着气瘫倒在地上。</p><br /><span><br /></span><span><br /></span><br />
Materia Oscura
By shows
Materia Oscura es un programa especializado en Ciencia del periodista José Manuel Nieves. Universo, física, hallazgos, nuevas investigaciones... Todo pasa por el tamiz de Nieves.
Next Question with Katie Couric
By shows
<p>Tired of political headlines that feel like déjà vu? Wondering if you <em>actually</em> need to care about every controversy? This season, Katie’s asking those same questions—and talking to the people who have real answers. From political insiders to sharp-eyed journalists, she’s breaking down what’s worth your attention (and what’s not), with smarts, sanity, and even a little humor. Because none of us can be tuned in 24/7—but we <em>can</em> stay informed without losing our minds. </p><p>Tune in every Thursday and join Katie Couric and her guests for a conversation on NEXT QUESTION.</p><hr><p style='color:grey; font-size:0.75em;'> Hosted on Acast. See <a style='color:grey;' target='_blank' rel='noopener noreferrer' href='https://acast.com/privacy'>acast.com/privacy</a> for more information.</p>
Serialously with Annie Elise
By shows
Serialously is hosted by Annie Elise, best known for her true-crime YouTube channel “10 to Life” with over 1.5M subscribers. Annie created Serialously after decades of listening to true crime and craving more than just the headlines; she wanted all the overlooked details everyone else seemed to miss. Here, you’ll find everything you’d normally end up Googling yourself - the timelines, the red flags, the social media history, and everything in between. Even better, you’ll hear the story the way you’d want to: like your best friend is filling you in. Annie covers cases in an engaging, conversational way with every detail sleuthed out. Whether it’s breaking true crime news, viral cases, or ongoing trials, Serialously keeps you in the loop in real time - just like having a conversation with your best friend, or hopefully your new true crime bestie! If you've been looking for someone to chat all things true crime with you and a community that’s just as curious as you are - welcome, we're happy you're here! xx *This podcast is not affiliated with The New York Times.