Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Descripción del Episodio
This week I welcome on the show two of the most important technologists ever, in any field.
Jeff Dean is Google's Chief Scientist, and through 25 years at the company, has worked on basically the most transformative systems in modern computing: from MapReduce, BigTable, Tensorflow, AlphaChip, to Gemini.
Noam Shazeer invented or co-invented all the main architectures and techniques that are used for modern LLMs: from the Transformer itself, to Mixture of Experts, to Mesh Tensorflow, to Gemini and many other things.
We talk about their 25 years at Google, going from PageRank to MapReduce to the Transformer to MoEs to AlphaChip – and maybe soon to ASI.
My favorite part was Jeff's vision for Pathways, Google’s grand plan for a mutually-reinforcing loop of hardware and algorithmic design and for going past autoregression. That culminates in us imagining *all* of Google-the-company, going through one huge MoE model.
And Noam just bites every bullet: 100x world GDP soon; let’s get a million automated researchers running in the Google datacenter; living to see the year 3000.Watch on Youtube; listen on Apple Podcasts or Spotify.
Sponsors
Scale partners with major AI labs like Meta, Google Deepmind, and OpenAI. Through Scale’s Data Foundry, labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you’re an AI researcher or engineer, learn about how Scale’s Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkesh
Curious how Jane Street teaches their new traders? They use Figgie, a rapid-fire card game that simulates the most exciting parts of markets and trading. It’s become so popular that Jane Street hosts an inter-office Figgie championship every year. Download from the app store or play on your desktop at figgie.com
Meter wants to radically improve the digital world we take for granted. They’re developing a foundation model that automates network management end-to-end. To do this, they just announced a long-term partnership with Microsoft for tens of thousands of GPUs, and they’re recruiting a world class AI research team. To learn more, go to meter.com/dwarkesh
To sponsor a future episode, visit dwarkeshpatel.com/p/advertise
Timestamps
00:00:00 - Intro
00:02:44 - Joining Google in 1999
00:05:36 - Future of Moore's Law
00:10:21 - Future TPUs
00:13:13 - Jeff’s undergrad thesis: parallel backprop
00:15:10 - LLMs in 2007
00:23:07 - “Holy s**t” moments
00:29:46 - AI fulfills Google’s original mission
00:34:19 - Doing Search in-context
00:38:32 - The internal coding model
00:39:49 - What will 2027 models do?
00:46:00 - A new architecture every day?
00:49:21 - Automated chip design and intelligence explosion
00:57:31 - Future of inference scaling
01:03:56 - Already doing multi-datacenter runs
01:22:33 - Debugging at scale
01:26:05 - Fast takeoff and superalignment
01:34:40 - A million evil Jeff Deans
01:38:16 - Fun times at Google
01:41:50 - World compute demand in 2030
01:48:21 - Getting back to modularity
01:59:13 - Keeping a giga-MoE in-memory
02:04:09 - All of Google in one model
02:12:43 - What’s missing from distillation
02:18:03 - Open research, pros and cons
02:24:54 - Going the distance
Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe
Episodios Recientes
Más podcasts de Sociedad y Cultura
Ver toda la categoría →
Dumb Blonde
By shows
<p>Asking the questions others are afraid to. Bunnie XO host of the Dumb Blonde podcast – the ultimate destination for comedy, trending and lifestyle. Get ready to dive into hilarious discussions about relationships, trauma, embarrassing moments, and all the realness life throws at us. Join Bunnie every week to laugh, relate, and embrace your inner healing.</p>
The Snare
By shows
In 1996, 18-year-old Angie Dodge is found brutally murdered in her Idaho Falls home. Police zero in on a suspect and put a man behind bars. But as the years pass, doubts emerge about whether the real killer was ever caught. Leading the fight for answers is an unlikely advocate: Angie’s own mother, who embarks on a decades-long mission to uncover the truth. A six-part series from 20/20 and ABC Audio, hosted by Maggie Rulli. New episodes Tuesdays.
Joy 101 with Hoda Kotb
By shows
<p>Joy is essential.</p> <p>And it's also elusive. You can't order it, borrow it, or simply hope it into life.</p> <p>But now, there's a new and exciting way to start your journey toward a more joyful existence: The Joy 101 Podcast with Hoda!</p> <p>Best known for her Emmy-winning work and co-anchoring <em>Today,</em> Hoda Kotb infuses her authenticity, curiosity, and warmth into conversations with the world’s most fascinating people. Entertainment legends, sport icons, wellness experts, and everyday folks will share how they find, allow, and experience joy. Hoda will offer her own tips and takes on seeking a more balanced, harmonious life. </p> <p>If you're craving inspiration, support, and useful tools to maximize your joy, tune in to these candid, uplifting, and moving on-air chats.</p> <p>Joy after a breakup, joy as an empty-nester, joy after loss, joy as a caretaker — Hoda's new podcast will speak to you.</p> <p>Joy 101 with Hoda Kotb, an iHeartPodcast.</p>
La Silla: On The Record
By shows
Cada semana contamos movidas de poder en Colombia a través de la voz de sus protagonistas. Un podcast de La Silla Podcasts.
Dinero Más Inteligente
By shows
El dinero no solo se gana, se entiende. <br /> En El Dinero Más Inteligente, Valeria Ovalle presenta la economía y Juan Carlos Herrera la conecta al mundo de inversiones. Una conversación entre razón y estrategia para entender el mundo financiero sin complicaciones. <br /> By GBM <br /> Síguenos en <b><a href="https://www.instagram.com/gbmplus_?igsh=czBzOG5mazBwMmlj&utm_source=qr">Instagram</a><b>.</b></b>
Lo que NO se habla con Fer Flores
By shows
Contenidos para construir una Humanidad con H mayúscula con perspectiva psicoanalista, médica, legal, antropológica y espiritual.
After the Whistle with Brendan Hunt and Rebecca Lowe
By shows
<p>Rebecca Lowe (Fox Sports) and Brendan Hunt (‘Ted Lasso’) are teaming up again to take on the 2026 World Cup! They’ll ride an emotional roller coaster together as 48 teams play 104 action-packed matches across the U.S., Canada, and Mexico. They’ll bring you all the joy and the drama, the hope and the heartbreak — and help you understand the matchups and personalities that will make this the biggest sporting and cultural event of our lifetimes</p><p>‘After the Whistle With Brendan Hunt and Rebecca Lowe’ is an Apple News Original podcast presented by Verizon.</p>
