The Information Bottleneck

The Information Bottleneck

Ravid Shwartz-Ziv & Allen Roush
Country USA
Language EN
Episodes 42
Latest 20.11.2026

Two AI researchers, Ravid Shwartz-Ziv and Allen Roush, discuss the latest trends, news, and research in Generative AI, LLMs, GPUs, and Cloud Systems. The podcast covers cutting-edge developments in artificial intelligence and machine learning, offering insights from experts in the field.

Episodes

  • What a VC Actually Looks For in AI Founders | Sarah Catanzaro
    What a VC Actually Looks For in AI Founders | Sarah Catanzaro 20.11.2026 1h 5m
    In this episode, we talked with Sarah Catanzaro, a general partner at Amplify Partners, about how AI research looks from the investor's side. Sarah has backed Runway and Modal since their seed rounds, and she usually gets to know researchers long before they start a company. This one is a bit different from our usual episodes because we talk about business more than research.We started with what makes a research idea a good business. Sarah's view is that startups shouldn't compete with the frontier labs directly and should go after things the labs put lower on their list. Voice is her example....
  • Post-Training Is Where Models Are Made Now | Maxime Labonne
    Post-Training Is Where Models Are Made Now | Maxime Labonne 17.11.2026 1h 3m
    In this episode, we talked with Maxime Labonne, Head of Post-Training at Liquid AI and author of the LLM Course on GitHub, about what happens to a model after pre-training and why that stage now matters so much.Maxime walks us through the short history of post-training, from supervised fine-tuning and preference optimization to reinforcement learning with verifiable rewards and now multi-domain on-policy distillation (MOPD), where separate teachers for code, math and other domains get merged back into one student. He explains why you still can’t skip SFT, why the order of the stages matters, a...
  • Physical AI for Science: Why It’s Harder Than Coding | Rishabh Agarwal (Periodic Labs)
    Physical AI for Science: Why It’s Harder Than Coding | Rishabh Agarwal (Periodic Labs) 13.11.2026 1h 1m
    Rishabh Agarwal has worked on reinforcement learning for years, at Mila, Google DeepMind and Meta. He is now a founding member at Periodic Labs, a company building AI scientists together with the physical labs where they run experiments, and an adjunct professor at McGill. By his own count he understands about ten percent of the science they do there. In this episode he tells us why he made that move.His argument is that science is a different problem from coding. A coding agent never has to leave the computer, so progress is mostly about compute and good environments. To make a new material,...
  • Dan Card: AI Hasn’t Changed Cybersecurity As Much As You Think
    Dan Card: AI Hasn’t Changed Cybersecurity As Much As You Think 07.11.2026 1h 21m
    Dan Card has spent more than 26 years in cybersecurity as a penetration tester, defender and architect. He helped organizations respond to some of the biggest incidents of the last two decades, from MS Blaster to WannaCry and Log4j. In this episode we ask him what has actually changed now that attackers and defenders both have strong AI models.Dan runs a lab where Claude attacks and ChatGPT defends, and he explains why a swarm of ten thousand agents hitting a defended server mostly gets you blocked faster. AI does help attackers with reconnaissance and research, and it helps defenders work thr...
  • Why Video Games Still Don’t Use Modern AI | Julian Togelius (NYU)
    Why Video Games Still Don’t Use Modern AI | Julian Togelius (NYU) 31.10.2026 1h 13m
    Julian Togelius, an NYU professor and co-founder of modl.ai, is living the dream of every kid who grew up as a computer geek and gamer. He’s been doing AI research on video games since 2004, back when people told him it wasn’t real research.So why do most video games still run on behavior trees and dialogue trees when we have LLMs? Julian’s answer is that games are designed not to need AI. Most genres were locked in decades ago, and nobody has figured out how to design a good game around a model that can say anything.We talk about the state of the game industry and why so many developers there...
  • What’s Left for Mathematicians? | Daniel Litt
    What’s Left for Mathematicians? | Daniel Litt 20.10.2026 1h 18m
    In this episode, we talked with Daniel Litt, a mathematician at the University of Toronto. In March 2025, Daniel bet that AI would not be able to autonomously produce an Annals of Mathematics-level paper by 2030. He has since conceded. He told us what changed and why. The models became superhuman at a small set of skills, like knowing the entire literature and grinding through technical arguments, and that turned out to be enough to do real math. They are still weak at the fuzzy part of the job, like figuring out what is actually true about an object or which questions are worth asking.We also...
  • The Political Economy of AI: Governance, Bias, and the Future | Andy Hall (Stanford/Anthropic)
    The Political Economy of AI: Governance, Bias, and the Future | Andy Hall (Stanford/Anthropic) 16.10.2026 1h 24m
    In this episode of the the information bottleneck podcast, we talked with Andy Hall, a Stanford professor and MTS at Anthropic. The conversation explores the political economy of AGI, analyzing how superintelligence could reshape job markets, wealth redistribution, and public trust in government. Hall presents an optimistic view that hyperintelligent agents could enhance democracy by helping citizens make better political decisions, likening AI's potential impact to the invention of the printing press. The discussion also covers the challenges of regulating open-source models, the origins and...
  • John Platt (Google): AI for Climate and Science
    John Platt (Google): AI for Climate and Science 10.10.2026 1h 12m
    John Platt is a Google Fellow and leads the Applied Science team at Google Research, the group working on climate and science. He's also the person behind SMO and Platt scaling. In this episode, we talk with him about how Google decides which scientific problems are worth its time, and what happens when AI meets real physical systems. A big part of the conversation is contrails. A small share of flights creates most of the warming from contrails, which together add up to roughly 1% of human-caused warming. John explains how his team detects contrails from weather satellites and predicts where...
  • Yuandong Tian on Recursive Self-Improvement
    Yuandong Tian on Recursive Self-Improvement 05.10.2026 58m
    Yuandong Tian spent many years at Meta FAIR and recently left to co-found Recursive Superintelligence, a company building AI that improves itself. In this episode, he tells us why.We start with his work on computer Go, where he built DarkForest before AlphaGo came out. A few years later came OpenGo, which played Korean professionals on a single GPU and didn't lose a game. He then tried to bring RL to real-world problems and found that the design of the action space mattered more than the algorithm.We also talk about Coconut, his paper on reasoning in latent space instead of tokens, and why fro...
  • Chris Manning: Language Is the Real Unlock for Intelligence
    Chris Manning: Language Is the Real Unlock for Intelligence 27.09.2026 55m
    Chris Manning is a legend in NLP. He's a professor of linguistics and computer science at Stanford and ran the Stanford AI Lab. His course CS224N, Natural Language Processing with Deep Learning, is where a whole generation of researchers learned the field. If you've worked on anything in NLP over the last 25 years, you've probably used his work.We start with what linguistics gave machine learning that ML wouldn't have figured out on its own, and what's left for NLP researchers now that LLMs handle most of the classic tasks. Chris thinks the open problems have moved up the stack to pragmatics a...
  • Why AI Still Can't See wtih Andrew Dai
    Why AI Still Can't See wtih Andrew Dai 24.09.2026 58m
    Andrew Dai spent over a decade at Google Brain and DeepMind, where he co-wrote the 2015 paper that introduced language model pre-training followed by fine-tuning, and later co-led pre-training data for Gemini. He's now co-founder and CEO of Elorian AI, which is building models for visual reasoning.We talk about how his pre-training result started as a bug, why next-token prediction scales better than other objectives, and what went wrong for Google in the early LLM race. The second half is about vision: why today's frontier models still can't count objects in a photo, why he thinks reasoning i...
  • Can AI Replace Doctors? Zachary Lipton on the Future of Healthcare and AI
    Can AI Replace Doctors? Zachary Lipton on the Future of Healthcare and AI 21.09.2026 1h 31m
    Zachary Lipton is an associate professor at Carnegie Mellon University and a co-founder of Abridge, a healthcare AI company (and a jazz saxophonist!). His research spans machine learning, healthcare, and the broader impact of AI.In this episode, we discuss what AI can (and cannot yet) do in healthcare. We talk about why medicine is harder to automate than coding, AI scribes and clinical decision support, drug discovery, whether AI could eventually replace doctors, and the role of open models and intelligent routing.In the second half, we turn to the future of AI research: whether academia can...
  • Sara Hooker on the End of Static AI
    Sara Hooker on the End of Static AI 16.09.2026 1h 35m
    What comes after scaling?We talk with Sara Hooker, co-founder and CEO of Adaptation Lab, about why the next generation of AI may look very different from today's static models. Sara argues that models should continuously adapt to new tasks, data, users, and environments—and that doing this efficiently will require rethinking much more than fine-tuning.We discuss continual learning, AutoScientist and automated research, why non-verifiable tasks may become the next major bottleneck, and why interfaces could be as important as the models themselves. We also get into open vs. closed models, distil...
  • Tiny Recursive Models Beat the Giants  -  Alexia Jolicoeur-Martineau (Microsoft)
    Tiny Recursive Models Beat the Giants - Alexia Jolicoeur-Martineau (Microsoft) 15.09.2026 52m
    Alexia Jolicoeur-Martineau is a Principal Researcher at Microsoft and the author of "Less is More: Recursive Reasoning with Tiny Networks," the paper behind the Tiny Recursive Model that hit about 45% on ARC-AGI-1 with a fraction of the parameters of frontier systems. It won the 2025 ARC Prize paper award.She read the hierarchical reasoning paper, thought the potential was real and the explanation was not, and rebuilt it without the mouse brains: a small network that carries a hidden state and a current answer, thinks for a few steps, updates, and repeats, with the gradient truncated at each l...
  • Continual Learning Is the Next Bottleneck | Rohan Anil (Core Automation )
    Continual Learning Is the Next Bottleneck | Rohan Anil (Core Automation ) 10.09.2026 1h 11m
    Rohan Anil spent eleven and a half years at Google, where he went from writing memory allocators to large-scale linear solvers, then optimization at Google Brain, where he co-developed distributed Shampoo and led optimization for PaLM and Gemini pre-training, including the work that produced Gemini Flash. He then joined Anthropic's pre-training team, and left before the IPO to co-found Core Automation with Jerry Tworek (ex-VP of Research at OpenAI). We talk with him about how Brain worked at its peak, why he left two of the world's best labs, and what he thinks is missing from today's models.R...
  • World Models  | John Langford (Microsoft AI Labs)
    World Models | John Langford (Microsoft AI Labs) 05.09.2026 1h 5m
    John Langford, one of the heads of Microsoft's AI Labs, the creator of Vowpal Wabbit, and a co-inventor of CAPTCHA, joins us to talk about world models. Transformers need orders of magnitude more data than humans to learn the same thing, and John argues a compact, implicit world model is how you close that gap. He explains why he's skeptical of JEPA-style objectives, why a transformer's KV cache is the Ptolemaic epicycle model of belief states, and what his Next Latent work does differently.We also get into whether research still matters in the age of scale; open versus closed models; agent-dr...
  • Which Tabular Model Should You Actually Use? | David Holzmüller (INRIA)
    Which Tabular Model Should You Actually Use? | David Holzmüller (INRIA) 03.09.2026 55m
    DescriptionTabular data is still where most of machine learning actually happens in industry, and the field has changed a lot in the last few years. In this episode we talk with David Holzmüller, a researcher at INRIA and one of the people behind TabArena, TabICL and RealMLP, about what the state of the art looks like right now and how to pick a model for your own data.We cover the shift to TabPFN-style foundation models that learn to learn from whole tables, why TabArena was built and what earlier benchmarks got wrong, what Google's new TabFM means for the leaderboard, and when gradient boost...
  • Why You Can't Just Rent 1,000 GPUs | Charles Frye (Modal)
    Why You Can't Just Rent 1,000 GPUs | Charles Frye (Modal) 01.09.2026 1h 3m
    Charles Frye (Modal, ex-Weights & Biases, Berkeley PhD) joins Ravid and Allen to explain why modern AI research is bottlenecked by compute, and why simply buying more GPUs doesn't solve it. We cover the three problems every lab hits (underutilization, saturation, resource sharing), when companies should actually train their own models, why inference is a "bad algorithm" for today's hardware, NVIDIA's monopoly, the OpenAI/Hugging Face hack and what it says about open models, and whether we're in a compute bubble.Key topicsAI infrastructure challenges and when to train your own modelsGPU resourc...
  • Stella Biderman (EleutherAI) - Open Source, AI Safety, and Who We Can Trust
    Stella Biderman (EleutherAI) - Open Source, AI Safety, and Who We Can Trust 28.08.2026 1h 6m
    Stella Biderman, Executive Director of EleutherAI, joins us the week an OpenAI model autonomously broke out of its sandbox and hacked Hugging Face. Stella calls it what she thinks it is, an offensive cyber operation, and argues it's part of a pattern: this is not the first containment failure at a frontier lab, and sandboxes have failed basically every time they've been tested for real.So we spend a good chunk of the episode on what actual containment would look like. Stella's argument is that the tools already exist, the labs just don't use them: run dangerous capability evals on air-gapped n...
  • Why Deep Learning Finally Works on Tables | Frank Hutter (Prior Labs)
    Why Deep Learning Finally Works on Tables | Frank Hutter (Prior Labs) 24.08.2026 1h 18m
    In this episode, Frank Hutter joins us to talk about TabPFN and why tabular data is suddenly the hottest problem in deep learning. Frank is a professor at the University of Freiburg and spent 15 years building the AutoML field before founding Prior Labs, which SAP just acquired for over a billion dollars.We get into why deep learning failed on tables for a decade and what in-context learning changed, how TabPFN is trained entirely on synthetic data, and why a model that never saw a real time series ended up beating specialized forecasting models. Frank also explains the architecture tricks beh...

Popular in

The podcast also appears in the podcast charts of these countries.