Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Descripción del Episodio
This week I welcome on the show two of the most important technologists ever, in any field.
Jeff Dean is Google's Chief Scientist, and through 25 years at the company, has worked on basically the most transformative systems in modern computing: from MapReduce, BigTable, Tensorflow, AlphaChip, to Gemini.
Noam Shazeer invented or co-invented all the main architectures and techniques that are used for modern LLMs: from the Transformer itself, to Mixture of Experts, to Mesh Tensorflow, to Gemini and many other things.
We talk about their 25 years at Google, going from PageRank to MapReduce to the Transformer to MoEs to AlphaChip – and maybe soon to ASI.
My favorite part was Jeff's vision for Pathways, Google’s grand plan for a mutually-reinforcing loop of hardware and algorithmic design and for going past autoregression. That culminates in us imagining *all* of Google-the-company, going through one huge MoE model.
And Noam just bites every bullet: 100x world GDP soon; let’s get a million automated researchers running in the Google datacenter; living to see the year 3000.Watch on Youtube; listen on Apple Podcasts or Spotify.
Sponsors
Scale partners with major AI labs like Meta, Google Deepmind, and OpenAI. Through Scale’s Data Foundry, labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you’re an AI researcher or engineer, learn about how Scale’s Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkesh
Curious how Jane Street teaches their new traders? They use Figgie, a rapid-fire card game that simulates the most exciting parts of markets and trading. It’s become so popular that Jane Street hosts an inter-office Figgie championship every year. Download from the app store or play on your desktop at figgie.com
Meter wants to radically improve the digital world we take for granted. They’re developing a foundation model that automates network management end-to-end. To do this, they just announced a long-term partnership with Microsoft for tens of thousands of GPUs, and they’re recruiting a world class AI research team. To learn more, go to meter.com/dwarkesh
To sponsor a future episode, visit dwarkeshpatel.com/p/advertise
Timestamps
00:00:00 - Intro
00:02:44 - Joining Google in 1999
00:05:36 - Future of Moore's Law
00:10:21 - Future TPUs
00:13:13 - Jeff’s undergrad thesis: parallel backprop
00:15:10 - LLMs in 2007
00:23:07 - “Holy s**t” moments
00:29:46 - AI fulfills Google’s original mission
00:34:19 - Doing Search in-context
00:38:32 - The internal coding model
00:39:49 - What will 2027 models do?
00:46:00 - A new architecture every day?
00:49:21 - Automated chip design and intelligence explosion
00:57:31 - Future of inference scaling
01:03:56 - Already doing multi-datacenter runs
01:22:33 - Debugging at scale
01:26:05 - Fast takeoff and superalignment
01:34:40 - A million evil Jeff Deans
01:38:16 - Fun times at Google
01:41:50 - World compute demand in 2030
01:48:21 - Getting back to modularity
01:59:13 - Keeping a giga-MoE in-memory
02:04:09 - All of Google in one model
02:12:43 - What’s missing from distillation
02:18:03 - Open research, pros and cons
02:24:54 - Going the distance
Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe
Episodios Recientes
Más podcasts de Sociedad y Cultura
Ver toda la categoría →
AM
By shows
El resumen informativo de The Voice Village, dirigido por Conchi Gil, producido por Ricardo Villa y narrado por Conchi Gil y Dani Gutierrez. Una producción de The Voice Village. De lunes a viernes, a las 7:30 AM. @amelpodcast
Estas Rica
By shows
Estás Rica Podcast es el close friends más hot de Dani Sayan: artista intensa, graduada de psicología y cero paciencia para el autoengaño. Aquí se habla de deseo, apego, ambición, poder personal y caos emocional… con humor picante, verdades incómodas y vibes de “sí soy sensible, pero también peligrosa”. Vienes a sentirte vista, removida y un poquito atacada. Sales más clara, más rica y con menos ganas de conformarte. Dedicado a mujeres, hombres, personas no binaries, y cualquier ser que quiera mejorar en todo aspecto. Y cuando el universo deja de ser místico y se convierte en programas, la transformación se hace práctica: si quieres ir más profundo, los programas viven en estasrica.com.
Not Just A Terrible Story
By shows
When Stacie Love married her former high school teacher, she thought it was a whirlwind romance. Two decades later, she discovered the truth behind the day their lives unraveled: a federal search, an unseen investigation, and a story far darker than she ever knew. Not Just A Terrible Story is a six-part investigation into one woman’s search for the truth, the scourge of alleged educator sexual misconduct, and a culture of complicity that protected predators for decades.
Giggly Squad
By shows
Welcome to Giggly Squad. Each week, Hannah Berner and Paige DeSorbo make fun of everything, but most importantly themselves. They discuss pop culture, fashion trends, television, horoscopes, mental health, dating, and expose their personal lives. Also, they can't be managed. Subscribe to SiriusXM Podcasts+ to listen to new episodes of Giggly Squad ad-free. Start a free trial now on Apple Podcasts or by visiting siriusxm.com/podcastsplus.
老婆等我之火影重生|火影同人|多人有声剧|多播
By shows
<p style="color:#333333;font-weight:normal;font-size:16px;line-height:30px;font-family:Helvetica,Arial,sans-serif;hyphens:auto;text-align:justify;">“表哥！”佐助回头大叫一声，然而，背后突然出现了一幅惊心动魄的画面：一群从来没有见过的怪兽咆哮着向他冲来，怪售长着一副闪亮的铠甲，扁平的身体上方有一个凸起，凸起中伸出一根长长的管子，与其说怪兽，不如说是一群钢铁机器！只见那群钢铁机器长长、的管子里火光一闪，一颗颗尖头的金属就被抛射出来，在佐助周围炸开，周围一片火海滔滔，瞬间毁灭了周围的建筑和街道！佐助心里紧张万分，只剩下一个念头，那就是逃！赶快逃！以最快的速度逃走！他不想死在这里！虽然身上现在加上自己的负重铁块总共有约三倍的重力，但是，他心底的潜力被极度激发，完全忽视了中立的影响。</p><br /><p style="color:#333333;font-weight:normal;font-size:16px;line-height:30px;font-family:Helvetica,Arial,sans-serif;hyphens:auto;text-align:justify;"> 我站在一棵大树上，看到佐助在空无一人的街道上发狂奔跑，好几次快要跑不动了，却又重新激发了潜能，一次次突破了身体的障碍，最后竟然跑起来就不停了！我笑着一挥手，佐助立即喘着气瘫倒在地上。</p><br /><span><br /></span><span><br /></span><br />
Materia Oscura
By shows
Materia Oscura es un programa especializado en Ciencia del periodista José Manuel Nieves. Universo, física, hallazgos, nuevas investigaciones... Todo pasa por el tamiz de Nieves.
Next Question with Katie Couric
By shows
<p>Tired of political headlines that feel like déjà vu? Wondering if you <em>actually</em> need to care about every controversy? This season, Katie’s asking those same questions—and talking to the people who have real answers. From political insiders to sharp-eyed journalists, she’s breaking down what’s worth your attention (and what’s not), with smarts, sanity, and even a little humor. Because none of us can be tuned in 24/7—but we <em>can</em> stay informed without losing our minds. </p><p>Tune in every Thursday and join Katie Couric and her guests for a conversation on NEXT QUESTION.</p><hr><p style='color:grey; font-size:0.75em;'> Hosted on Acast. See <a style='color:grey;' target='_blank' rel='noopener noreferrer' href='https://acast.com/privacy'>acast.com/privacy</a> for more information.</p>
Serialously with Annie Elise
By shows
Serialously is hosted by Annie Elise, best known for her true-crime YouTube channel “10 to Life” with over 1.5M subscribers. Annie created Serialously after decades of listening to true crime and craving more than just the headlines; she wanted all the overlooked details everyone else seemed to miss. Here, you’ll find everything you’d normally end up Googling yourself - the timelines, the red flags, the social media history, and everything in between. Even better, you’ll hear the story the way you’d want to: like your best friend is filling you in. Annie covers cases in an engaging, conversational way with every detail sleuthed out. Whether it’s breaking true crime news, viral cases, or ongoing trials, Serialously keeps you in the loop in real time - just like having a conversation with your best friend, or hopefully your new true crime bestie! If you've been looking for someone to chat all things true crime with you and a community that’s just as curious as you are - welcome, we're happy you're here! xx *This podcast is not affiliated with The New York Times.