The Information Bottleneck

Jaksot

• 
  Sara Hooker on the End of Static AI 16.09.2026 1t 35min

  What comes after scaling?We talk with Sara Hooker, co-founder and CEO of Adaptation Lab, about why the next generation of AI may look very different from today's static models. Sara argues that models should continuously adapt to new tasks, data, users, and environments—and that doing this efficiently will require rethinking much more than fine-tuning.We discuss continual learning, AutoScientist and automated research, why non-verifiable tasks may become the next major bottleneck, and why interfaces could be as important as the models themselves. We also get into open vs. closed models, distillation and Chinese AI labs, AI regulation and safety, cybersecurity and biorisk, AI companionship, and what may eventually come after Transformers and tokenization.TopicsContinuous learning and adaptive AIFine-tuning, memory, and AutoScientistAI agents and automated researchNon-verifiable tasks and human feedbackAdaptive interfacesOpen vs. closed models and distillationAI safety, regulation, cyber risk, and bioriskAI companionship and persuasionThe limits of TransformersMultilingual models and tokenizationChapters00:00 — Introduction02:15 — Why start another AI lab? The return of research05:46 — What continuous learning actually means12:04 — Should every company have its own adapting model?13:59 — Fine-tuning and platforms like Tinker18:04 — AutoScientist and automated optimization22:52 — Can AI really improve its own research?28:38 — The problem of non-verifiable tasks31:30 — Human feedback and the limits of exponential progress34:43 — Why the AI interface matters40:36 — Distillation, China, and open models49:05 — Open-model licensing52:19 — Will open models catch closed models?58:43 — AI regulation and compute thresholds1:03:07 — AI safety and agent failures1:10:19 — Biorisk vs. cybersecurity1:14:03 — Persuasion, AI companionship, and overlooked risks1:20:41 — Where will AI have the biggest real-world impact?1:25:41 — What is missing from current AI architectures?1:29:03 — Neurosymbolic AI1:31:30 — Multilingual models and tokenization1:34:02 — Byte-level models and alternatives to tokenization1:35:03 — ClosingMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0
• 
  Tiny Recursive Models Beat the Giants - Alexia Jolicoeur-Martineau (Microsoft) 15.09.2026 52min

  Alexia Jolicoeur-Martineau is a Principal Researcher at Microsoft and the author of "Less is More: Recursive Reasoning with Tiny Networks," the paper behind the Tiny Recursive Model that hit about 45% on ARC-AGI-1 with a fraction of the parameters of frontier systems. It won the 2025 ARC Prize paper award.She read the hierarchical reasoning paper, thought the potential was real and the explanation was not, and rebuilt it without the mouse brains: a small network that carries a hidden state and a current answer, thinks for a few steps, updates, and repeats, with the gradient truncated at each loop. We get into why puzzles suit this and autoregression doesn't, why she thinks LLMs are bad at molecules and more data won't fix it, and what she'd do with a trillion dollars.Timeline00:01 Intro01:06 Leaving biostatistics, and why the field stagnated06:47 GANs, diffusion, and research on four GPUs12:58 What was wrong with the hierarchical reasoning paper16:31 Tiny recursive models explained without the biology22:35 Why puzzles favor recursion over left to right generation24:15 Is the bitter lesson really bitter?27:28 With infinite compute, would you still want small models?32:00 Self improvement, memory, and a trillion dollars37:01 Test time compute beyond chain of thought40:41 Why chain of thought fails on molecules45:17 Is there a universal representation?48:06 What people are already building with TRM55:22 Fixed point models and DEQMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0TopicsTiny Recursive Models and the ARC-AGI resultsWhat the hierarchical reasoning model was really doingDeep supervision and truncated backpropLooping transformers and parameter efficiencyWhy puzzles favor whole-context iteration over left to right generationTest time compute beyond chain of thoughtLatent reasoning and the Coconut line of workWhy LLMs fail on chemistry and physicsRepresentation learning and whether a universal representation exists
• 
  Continual Learning Is the Next Bottleneck | Rohan Anil (Core Automation ) 10.09.2026 1t 11min

  Rohan Anil spent eleven and a half years at Google, where he went from writing memory allocators to large-scale linear solvers, then optimization at Google Brain, where he co-developed distributed Shampoo and led optimization for PaLM and Gemini pre-training, including the work that produced Gemini Flash. He then joined Anthropic's pre-training team, and left before the IPO to co-found Core Automation with Jerry Tworek (ex-VP of Research at OpenAI). We talk with him about how Brain worked at its peak, why he left two of the world's best labs, and what he thinks is missing from today's models.Rohan's view is that pre-training and RL were split by organizational convenience rather than by science. Pre-training builds a prior, and RL sharpens it to the tasks we care about, and neither gives a model a way to absorb new data or learn from its own experience once it is deployed. Post-training more every day plateaus, on-policy distillation plateaus, and in-context learning only goes as far as the context does. He argues the next architecture needs better ways to fold in new knowledge at inference time, and that this is a fundamental optimization question rather than a harness-engineering one.We also get into why coding agents still fail on low-level systems work, his take on Muon, why second-order methods matter once you leave the noise-dominated regime, and why nobody can yet use a few million GPUs for a single training run.Timeline00:00 Intro01:09 From computer vision to Google systems engineering02:37 Large-scale linear solvers and sparse features06:01 Getting into optimization: SDCA and Yonghui Wu's team07:29 Joining the Shampoo crew09:35 The Google Brain ethos, and why 2017 to 2019 was special14:23 Is open research going to keep winning?15:53 Frontier models are only as good as the prior you give them17:30 Missing the language model wave, then Common Crawl and online distillation18:31 Paternity leave, DALL-E Mini, and the 14 days that became two years20:32 PaLM, Gemini pre-training, and Gemini Flash23:59 The Shampoo origin story: Tomer Koren's two-week proof26:54 Why leave Google for Anthropic30:20 Why leave Anthropic for a startup31:33 Meeting Jerry Tworek at Dolores Park34:00 What Core Automation is building36:17 Continual learning and the pre-training vs RL split39:11 Why coding agents fail at kernels and low-level pipelines42:00 The QR factorization kernel competition and reward hacking44:57 Numerics, verification, and hardware that keeps changing46:07 Are LLMs creative, or just good at search?49:50 Getting models to extrapolate instead of interpolate52:03 Why did we ever call it pre-training?55:02 What RL is really learning56:27 Competing with the big labs with fewer people58:31 Will kernel generation keep old GPUs alive? Amdahl's law1:01:58 Open source plans1:02:55 Audience question: agentic optimizers1:04:53 Audience question: Muon, Shampoo, and the future of second-order methods1:08:58 Hiring at Core Automationkey topicsJourney from Google Brain to startupEvolution of AI research and optimizationPre-training and reinforcement learningKernel optimization and system efficiencyOpen source AI and collaborative researchChallenges in AI creativity and explorationFuture directions in continual learning and model scalingMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0
• 
  World Models | John Langford (Microsoft AI Labs) 05.09.2026 1t 5min

  John Langford, one of the heads of Microsoft's AI Labs, the creator of Vowpal Wabbit, and a co-inventor of CAPTCHA, joins us to talk about world models. Transformers need orders of magnitude more data than humans to learn the same thing, and John argues a compact, implicit world model is how you close that gap. He explains why he's skeptical of JEPA-style objectives, why a transformer's KV cache is the Ptolemaic epicycle model of belief states, and what his Next Latent work does differently.We also get into whether research still matters in the age of scale; open versus closed models; agent-driven research after running 2,000 pre-training experiments in 90 days; the origin story of CAPTCHA; and why Muon and orthonormal optimizers actually work.Topics:Implicit vs. explicit world models, and the case against JEPA-style objectivesCompact belief states: why compression beats a growing KV cacheDoes research still matter in the age of scale? The Kimi K3 argumentAgent-driven research: 2,000 pre-training experiments in 90 daysThe invention of CAPTCHAOptimizers from SGD and Vowpal Wabbit to MuonChapters00:00 Why world models: the sample-complexity gap09:48 The case against JEPA; a transformer-style implicit world model15:52 Compact belief states: epicycles vs. heliocentrism23:41 Does research still matter? The Kimi K3 argument27:35 Open vs. closed models35:57 Recursive self-improvement and agent-driven research42:30 2,000 pre-training experiments in 90 days; weak baselines and reproducibility54:54 The invention of CAPTCHA1:00:53 Optimizers: from Vowpal Wabbit to MuonMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0
• 
  Which Tabular Model Should You Actually Use? | David Holzmüller (INRIA) 03.09.2026 55min

  DescriptionTabular data is still where most of machine learning actually happens in industry, and the field has changed a lot in the last few years. In this episode we talk with David Holzmüller, a researcher at INRIA and one of the people behind TabArena, TabICL and RealMLP, about what the state of the art looks like right now and how to pick a model for your own data.We cover the shift to TabPFN-style foundation models that learn to learn from whole tables, why TabArena was built and what earlier benchmarks got wrong, what Google's new TabFM means for the leaderboard, and when gradient boosted trees are still the right tool. David explains why LLMs struggle with tables, shares an early result comparing Claude Opus against TabICL on tiny datasets, and walks through how to embed text columns for tabular models. We also get into time series vs tabular data, the open research problems he thinks matter most, and why classical ML libraries are so bad out of the box.Links:TabArena: https://tabarena.aiTopicsTabular foundation models and in-context learning on tablesTabArena and Beyond Arena: building a benchmark that stays honestTabFM, TabPFN, TabICL and the tradeoffs between themWhen boosted trees and MLPs still win (large data, CPU, fast inference)Why LLMs are inefficient on tabular data and where they might helpEmbedding text columns with language modelsExplainability, calibration and class imbalanceTime series vs tabular dataOpen problems: invariances, synthetic data, uncertainty, scaling downWhere the field is heading in the next five yearsChapters0:00 Intro0:31 What changed in tabular ML: TabPFN-style foundation models2:22 Which model to try first? TabArena and how it was built5:14 What older benchmarks got wrong, and Beyond Arena8:45 GPU AutoML vs foundation models10:40 Reading the leaderboard: TabFM, TabPFN, TabICL and the tradeoffs12:47 Calibration, class imbalance and small vs large data19:45 Explainability for black-box tabular models21:34 Why LLMs are bad at tabular data25:39 Claude Opus 4.6 vs TabICL on tiny datasets27:51 New classifiers, five-year outlook, real vs synthetic pretraining33:13 Embedding text columns for tabular foundation models36:13 Time series vs tabular data39:59 When gradient boosted trees still win, and feature engineering45:31 Open research problems and where the field is heading52:54 Better MLPs and why classical defaults are bad out of the boxMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0
• 
  Why You Can't Just Rent 1,000 GPUs | Charles Frye (Modal) 01.09.2026 1t 3min

  Charles Frye (Modal, ex-Weights & Biases, Berkeley PhD) joins Ravid and Allen to explain why modern AI research is bottlenecked by compute, and why simply buying more GPUs doesn't solve it. We cover the three problems every lab hits (underutilization, saturation, resource sharing), when companies should actually train their own models, why inference is a "bad algorithm" for today's hardware, NVIDIA's monopoly, the OpenAI/Hugging Face hack and what it says about open models, and whether we're in a compute bubble.Key topicsAI infrastructure challenges and when to train your own modelsGPU resource management and virtualizationInference optimization and speculative decodingThe economics and future of AI hardwareAgents, sandboxing, and open-model securityChapters00:00 Intro01:03 Why AI needs special-purpose compute03:22 Buying vs renting GPUs: the three problems07:15 Modal's approach, and doing more with less compute09:46 Do we actually need to spend more? The conflict-of-interest question13:08 Should companies train their own models?14:47 Efficient fine-tuning and prompts as fast weights17:37 Are we in a compute bubble?20:21 Why inference will dominate compute (the SQLite analogy)22:42 Speculative decoding26:44 Why scaling inference is hard, and neuromorphic hardware28:36 Why NVIDIA's monopoly persists33:09 Inference chip startups and the hardware lottery35:24 How Modal stays hardware-agnostic (GPU snapshot restore)38:45 Will agentic coding erode CUDA's moat?41:18 Running one agent vs thousands: sandboxing at scale46:27 The OpenAI/Hugging Face hack and open models as defenders52:28 Rogue AI, self-replication, and fast takeoff56:09 What's next: evals, embodiment, edge inference1:00:27 Modal is hiring (modal.jobs)Music"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0
• 
  Stella Biderman (EleutherAI) - Open Source, AI Safety, and Who We Can Trust 28.08.2026 1t 6min

  Stella Biderman, Executive Director of EleutherAI, joins us the week an OpenAI model autonomously broke out of its sandbox and hacked Hugging Face. Stella calls it what she thinks it is, an offensive cyber operation, and argues it's part of a pattern: this is not the first containment failure at a frontier lab, and sandboxes have failed basically every time they've been tested for real.So we spend a good chunk of the episode on what actual containment would look like. Stella's argument is that the tools already exist, the labs just don't use them: run dangerous capability evals on air-gapped networks with no route to the public internet, put the most sensitive testing in SCIF-style secure facilities, and treat model evaluation the way the security world treats classified systems rather than the way startups treat staging environments.And yet Stella remains one of the world's most prominent open-source advocates. From her perspective, the biggest risk isn't the technology; it's unchecked corporate power, and the only durable check on it is an independent scientific research establishment that doesn't depend on the AI industry for its funding or its facts.From there the conversation spans the geopolitics of Chinese open models and whether governments can restrict them, sovereign AI and what it would actually take for other countries to train their own models, why harnesses and UX drive more of AI's perceived progress than raw intelligence, the AI-found counterexample to the Jacobian conjecture, and EleutherAI's "Deep Ignorance" approach to making open-weight models safe by filtering hazardous knowledge out of pretraining.key topicsAI governance and regulationCybersecurity incidents involving AI modelsOpen source AI safety and securityThe role of independent research in AI safetyLegal and ethical considerations in AI developmentTimeline00:13 — Intro: Stella Biderman and EleutherAI, a real non-profit in AI02:05 — News of the week: Kimi K3, and OpenAI's model autonomously hacking Hugging Face05:49 — "Frontier labs can't be trusted": repeated containment failures, air-gapped networks and SCIFs vs. sandboxes22:45 — Can governments ban open or Chinese models? Import restrictions and the six-month open/closed gap27:05 — Why Stella is still pro-open-source: unchecked corporate power as the real danger31:11 — The opioid epidemic analogy: avoiding both regulatory failure and overcorrection34:57 — Offense vs. defense: why open access to AI has empirically favored defenders37:28 — Chinese labs, the CCP, and why safety and fine-tuning are low-prestige work in China42:19 — Sovereign AI: does every country need its own foundation model?49:29 — Sampling, harnesses, and why ChatGPT was really a UX breakthrough54:09 — AI solves the Jacobian conjecture: domain data beats raw intelligence58:02 — Safety is contextual, not a model property — and what HAL 9000 got right1:01:42 — Is Stella optimistic about the future?1:02:50 — Deep Ignorance, the science of AI training dynamics, and how to get involved with EleutherAIMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
• 
  Why Deep Learning Finally Works on Tables | Frank Hutter (Prior Labs) 24.08.2026 1t 18min

  In this episode, Frank Hutter joins us to talk about TabPFN and why tabular data is suddenly the hottest problem in deep learning. Frank is a professor at the University of Freiburg and spent 15 years building the AutoML field before founding Prior Labs, which SAP just acquired for over a billion dollars.We get into why deep learning failed on tables for a decade and what in-context learning changed, how TabPFN is trained entirely on synthetic data, and why a model that never saw a real time series ended up beating specialized forecasting models. Frank also explains the architecture tricks behind scaling from 10,000 to a million rows, where LLMs fit into data science (and where they embarrassingly don't), and what happens to XGBoost from here.Beyond the research, Frank talks about the jump from professor to co-CEO, why he refused to merge his 45-person team into SAP's 110,000 employees, the open-weights licensing debate, and the case for building a frontier lab in Freiburg rather than San Francisco.key topicsThe role of foundation models in tabular dataImpact of SAP acquisition on Pro LabsThe evolution of AutoML and hyperparameter optimizationChallenges and solutions for large context in modelsOpen source models and licensing strategiesThe importance of independence for startup agilityFuture directions in AI for science and medicine00:00 Intro00:34 The SAP acquisition and staying independent07:39 Why tabular data is the next big thing in deep learning14:19 What makes tabular data hard19:14 AutoML, AutoGluon, and fifteen years of hyperparameter tuning28:27 Scaling TabPFN: context limits and architectures34:35 Agentic data science and LLMs39:30 Online learning, time series, and Bayesian inference in a forward pass47:05 Open weights and the license debate54:51 Will LLMs and tabular models merge?1:00:01 From academia to startup1:09:42 Why build in Europe1:12:53 Audience questions and hiringMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
• 
  Surya Ganguli: The Physics of Intelligence 17.08.2026 1t 25min

  Surya Ganguli is a professor at Stanford and VP at General Catalyst, working at the intersection of physics, neuroscience, and AI. He started in string theory, moved to theoretical neuroscience, and now uses tools from statistical physics to understand both brains and neural networks.We talk about why deep learning theory is finally catching up to practice,  including his group's recent work explaining neural scaling laws, and why smarter data selection could beat them entirely. He also tells the origin story of diffusion models, which were invented in his lab as an attempt to violate the second law of thermodynamics.The second half turns to the brain: what happens to a mouse's sense of self on ketamine, how stimulating a handful of neurons can induce hallucinations, and a method his lab developed to get a neuron deep in a monkey's brain to describe, in English, what makes it fire.We close on where he thinks AI is going wrong: models train on ten trillion tokens while humans hear a hundred million words, because we don't teach children with gradients; we tell them the algorithm.key topicsConnections between physics, neuroscience, and AIEmergent properties in complex systemsScaling laws in language modelsData efficiency and pruning in AINeuroscience insights into consciousness and selfThe future of AI and brain modelingChapters00:00 Introduction to Surya Ganguli 00:57 Surya's Background: From String Theory to Neuroscience02:22 Emergent Properties in Physics, Neuroscience, and AI03:16 Energy Landscapes and Loss Landscapes in High Dimensions04:07 Why Local Minima Don't Exist in High-Dimensional AI05:22 Gradient-Based vs. Gradient-Free Learning Methods08:21 AI in Mathematics and Drug Discovery: Opportunities and Challenges13:48 Scaling Laws and Data Efficiency in Language Models18:10 Properties of Data that Affect Scaling Laws22:04 Constructing Non-Redundant Data Sets for Better Learning24:32 Theory vs. Empirical Results in AI Research32:19 Fundamental Components of Deep Learning: Are They Changing?34:31 Future Paradigms in AI Beyond Current Models37:22 Teaching AI and Humans: Paradigm Shifts in Learning41:37 Consciousness, Self, and the Brain: Surya's Perspectives49:49 Neuroscience and AI: Understanding the Brain and Consciousness01:02:03 Understanding the Brain: Challenges and Opportunities01:09:21 Brain-Computer Interfaces and AI in NeuroscienceMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
• 
  Text Diffusion Models with Brendan O'Donoghue (Google DeepMind) 14.08.2026 1t 9min

  Brendan O'Donoghue, research director at Google DeepMind, makes the case for text diffusion as a real alternative to autoregressive generation. He walks through how discrete diffusion works, why diffusion samples are far more diverse and what that unlocks for RL, where the Gemma diffusion model actually stands against frontier models, and why the whole training and serving stack being hyper-optimized for autoregression is the main thing holding the approach back. The conversation also covers hardware trends favoring flops over bandwidth, AGI timelines and real-world bottlenecks, and why he thinks RL is still underhyped.Key topics- Discrete diffusion for text vs autoregressive generation- Why diffusion samples are more diverse, and what that unlocks for RL- Where diffusion already wins: latency, on-device, robotics- Why serving cost, not quality, is the real blocker- RL as the most underhyped area in AITimeline00:00 Introduction00:50 What diffusion models are and how text diffusion works04:40 Why Brendan bet on text diffusion in 202307:15 Diversity, creativity, and why it helps RL11:00 The best diffusion LLM today and the gap to frontier models14:25 Latency, serving cost, and why it needs more chips17:14 Where diffusion already wins: on-device, robotics, battery20:14 One model, two modes: diffusion for thinking, AR for answering22:24 Samplers and the stuttering problem26:27 Theory, BERT, and why now is a good time to work on this31:48 Pipelines built for autoregression, and continuous diffusion35:35 Hardware: flops vs bandwidth39:49 AGI timelines and real-world bottlenecks50:15 Is AI engineering or science?54:14 Most overhyped and most underhyped ideas58:35 RL on diffusion, value functions, and exploration1:07:30 Go download the model and break itMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
• 
  Nathan Lambert: Inside Post-Training and the Open Model Fight 08.08.2026 1t 15min

  Nathan Lambert spent three years as post-training lead at Ai2, where he built the OLMo models, and he writes Interconnects, one of the most-read technical newsletters in AI. He left Ai2 in June and is now working on a new project. He's also the author of the RLHF book. We talked a lot about open models, their capabilities, and why they are better than he expected. We get into what that means over the next two to five years, why he thinks recursive self-improvement is overblown, what the market for training environments actually looks like now, and why he expects Anthropic's famously open internal culture to break after its IPO.Key TopicsOpen vs closed models and who actually captures the valueAnthropic and OpenAI as opposite cultures, and the talent concentration problemBoom vs bubble, and why token spend hasn't produced 10x better productsContinual learning, RSI skepticism, and what Nathan wants to work on nextWhat the open ecosystem needs economically to surviveTimeline 00:00 Intro00:27 Open vs closed models, and who actually captures the value05:12 China, harnesses, and where the real training leverage sits08:40 Sovereign compute and the national security case for building models11:18 Uncensored open weights and the bioweapon question14:29 Anthropic vs OpenAI, ideology and politics19:35 The Mythos ban and the Fable 5 delays24:30 The AGI narrative, the talent drain, and antitrust28:12 Why researchers join Anthropic, and the open Slack culture34:04 Nathan's next 12 months: character training and big RL runs37:55 Continual learning, RSI, and why Nathan is skeptical43:19 Boom or bubble, tokens vs GPUs45:12 Why all that token spend never produced 10x products48:38 Job displacement and the small-business future52:49 Robotics, world models, and why multimodal lags57:44 What the open ecosystem should actually do1:03:17 Why NVIDIA isn't building a frontier model1:07:34 The RLHF book, and whether RLHF still matters1:11:06 GRPO vs PPO and on-policy distillationMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
• 
  Daphne Koller - The Future of AI in Biology and Drug Discovery 03.08.2026 1t 2min

  Daphne Koller wrote the book that many of us learned probabilistic graphical models from, founded Coursera, and now runs insitro, which is trying to make drug discovery a machine-learning problem.We start with the bitter lesson. She agrees with most of it and then says where it stops working: biology doesn't have enough data, structure is how people understand anything, and making a drug is a question about an intervention that hasn't happened yet, not a pattern in data you already have.Most of the episode is about why drug discovery is hard. Ninety percent of drugs that reach the clinic fail, and mostly not because the molecule was bad. The molecule usually does what it was designed to do. It just turns out the thing it was designed to do had nothing to do with the disease. Only 22% of diseases have any approved drug at all, and she calls that an upper bound on what we understand, not a lower bound.She also gets into what agents are and aren't good for in a wet lab, why cells don't grow faster no matter how many GPUs you point at them, what it would take to have real foundation models for biology, and why almost all of biology is still out of distribution.Plus GLP-1s and what human data keeps teaching us, whether AI can make the kind of leap that turned a bacterial immune system into CRISPR, and what she'd build if she were starting Coursera today.Key TopicsThe impact of scaling and data in machine learningThe importance of structure and causality in AIChallenges in drug discovery and biological understandingThe role of foundation models in biologyEthical considerations in AI and biomedical researchChapters00:00 Introduction to Machine Learning and Drug Discovery02:00 The Bitter Lesson and Its Implications06:48 Challenges in Drug Design and Discovery11:48 Ethical Considerations in Human Research17:20 The Drug Discovery Pipeline Explained29:30 Integrating AI in Experimental Design35:38 The Role of Human Judgment in Drug Design37:14 Future of Drug Design: Efficiency vs. Automation39:37 Challenges in AI and Data Availability for Biology41:08 Foundation Models: Potential and Limitations43:39 Causality in Biological Data: Importance and Challenges45:18 Creativity vs. Understanding in Drug Design48:17 Balancing Investments in Data, Algorithms, and Experiments50:07 The Value of Simulations in Drug Discovery52:03 Mathematical Frameworks in Biology: Utility and Limitations54:14 The Future of Drug Discovery: Optimism and Innovations56:28 The Impact of Coursera on Education01:00:33 The Role of Universities in Lifelong Learning01:04:06 Connecting Dots: The Fun of Variety in Work01:05:46 Optimism for the Future of Drug DiscoveryMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
• 
  RL Was Broken at Every Level - With Joseph Suarez (PufferAI) 30.07.2026 1t 3min

  In this episode, Joseph Suarez from PufferAI explains why he thinks RL never had an algorithm problem, but it had a code problem. Every part of the standard RL stack was running about a thousand times slower than it should have been, and once that got fixed, problems that used to take months started getting solved in seconds on one GPU. We talk about what makes a simulator good for RL, why most of their sims run on CPU, what he wants to do with scientific simulation, and why he open sources all of it instead of writing papers. Key topicsTypes of RL and their applicationsChallenges in scaling reinforcement learningThe role of simulators and hardware in RLRL in gaming: from chess to complex games like NetHack and RuneScapeFuture directions: scientific simulation and biological modelingChapters00:00 - Introduction to RL and Puff AI01:50 - Different settings for RL: Games, Robots, Finance04:10 - RL in LM and other domains07:00 - Challenges and solutions in RL scaling09:55 - Building fast, efficient simulators15:10 - RL for scientific research and simulation19:57 - RL in complex games: NetHack, RuneScape, Dwarf Fortress29:55 - Future of RL: Scientific discovery and beyondResourcesPuff AI - Official Site - https://puffer.aiNetHack - https://www.nethack.org/RuneScape - https://www.runescape.com/Dwarf Fortress - http://www.bay12games.com/dwarves/OpenAI Gym - https://github.com/openai/gymMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
• 
  The Model Found a Way Out - with Florian Brand (Prime Intellect) 27.07.2026 57min

  Florian Brand builds evals at Prime Intellect. The premise of the conversation is that writing a benchmark is the easy part now. Keeping the model from cheating it is the job, and it takes longer than the benchmark itself.We get into why he thinks you can't evaluate a model apart from the CLI it runs in, what happens to statistics when a single run costs five figures, and whether the feeling that a model just works can ever become a number.He also has a few stories about agents finding their way around the scoring that are worth hearing cold.Timeline00:13 Intro01:00 What evals are for04:05 Agentic benchmarks07:10 Kimi K2 and model diversity08:23 Long-horizon coding tasks10:29 Building a benchmark12:15 MirrorCode14:27 Rubrics and LLM judges16:30 The cost of expert labelers17:49 Long runs and variance19:44 Evaluating the harness24:29 Chinese labs building CLIs30:00 More reward hacking37:45 Tau-bench and economic tasks39:43 Benchmaxxing and GLM 5.245:15 Statistics and cost47:56 Frontier convergence52:04 Misuse in open and closed models55:35 Self-improvementMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.AboutThe Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.
• 
  Pierre-Carl Langlais on Building Models from Data You Can Account For 23.07.2026 1t 5min

  Most labs build language models by scraping the web and filtering afterward. Pierre-Carl Langlais runs it the other way around. At Pleias, the French-German lab he co-founded, the models are built from data he can actually account for, which in practice means open and public-domain sources plus a lot of synthetic data the lab generates itself. It sounds like a self-imposed handicap. It mostly isn't. One of their models is a 600 million parameter system that runs live inside the Paris subway's monitoring pipeline.We cover the SYNTH pretraining dataset and why he thinks "ethical data" has to mean more than copyright-free. He explains why barely 2% of their Common Corpus appears in typical web crawls, and why that gap is really a preservation problem. From there, he gets blunt about benchmark maxing and whether GLM really earns its Opus-class reputation. He also argues that the quiet move by closed labs to hide reasoning traces is mostly about claiming ownership of model outputs. He's skeptical of sovereign AI, and not shy about how Mistral drifted from frontier research toward French corporate consulting. We finish on NVIDIA's persona datasets and the odd idea of training on the conditions that produced a text rather than the text itself.Timeline(00:02) Welcome and introductions(00:49) Why synthetic data matters, and the SYNTH set(04:15) Three reasons to control your training data(07:18) What "ethical data" actually means(11:08) How Common Corpus got built, from Wikipedia to PDFs(16:35) Agentic harnesses and synthetic data(20:03) Evaluating data when you train on reasoning traces(25:27) General versus specialized pretraining(27:08) Benchmark maxing and the GLM question(31:51) Getting diversity in, and the NVIDIA personas(35:02) Hidden reasoning traces and the fight over model IP(38:17) Mid-training and the "It's All Training" thesis(41:47) Can small models actually compete(45:01) Cybersecurity and Europe's strategic gap(47:08) Do you need a big model to orchestrate the small ones(52:08) Sovereign AI and the limits of national champions(56:42) Scaling laws when you control the data(01:00:41) The NVIDIA persona datasets(01:04:52) What you actually do with synthetic personas(01:08:22) Closing thoughtsMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.AboutThe Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.
• 
  Dhruv Batra: The Browser Is a Robotics Problem - From Embodied AI at Meta to Web Agents at Yutori 20.07.2026 1t 7min

  Dhruv Batra spent years leading Embodied AI at Meta,  training virtual robots to navigate photorealistic 3D scans of real buildings with pure reinforcement learning. Then he left to co-found Yutori and build agents for a very different environment: the web browser.In this episode, Dhruv explains why he sees these as the same problem. Web agents, in his framing, are robots that act in a browser (pixels in, actions out), and the web turns out to be just as messy an environment as the physical world.Along the way, we cover his definition of intelligence as "navigation in idea space," why robotics is lagging LLMs, the sim-to-real gap and why you can't fake friction coefficients, the teleoperation counterexample to the "it's a sensor problem" argument, and his provocative claim that under the current paradigm, we solved machine learning and didn't even realize it. He also makes the case for why the scaling hypothesis isn't falsifiable, why JEPA-style arguments deserve to be grappled with, how Yutori trains its Navigator models with RL on live websites, and what happens to the ad-supported web when agents, not eyeballs, do the browsing.Timeline00:01 — Intro00:54 — What embodied AI actually means06:47 — Intelligence as navigation in idea space13:26 — Habitat: training robots with pure RL, no maps20:04 — Why robotics is behind LLMs28:24 — Sim-to-real: what you can and can't fake33:34 — "We solved ML and nobody noticed"37:12 — Leaving Meta, founding Yutori43:21 — Web agents: screenshots in, actions out48:15 — Why the web won't rebuild itself for agents53:32 — Training Navigator: RL on live websites1:01:04 — Who pays for the web when agents browse?1:09:17 — What Yutori means, closing thoughtsMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.AboutThe Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.
• 
  How to Turn Research Into Billion-Dollar Companies, with Ion Stoica 16.07.2026 49min

  Ion Stoica has done what almost no academic ever does — repeatedly turned university research into billion-dollar companies. He co-founded Databricks (now valued at over $100 billion), Anyscale, Arena AI and Conviva, while his Berkeley lab produced the open source projects the entire AI industry runs on: Ray, vLLM, and SGLang.In this episode, we ask him how it's actually done. His answer is surprisingly unromantic: solve a problem people already care about, build an artifact good enough that they adopt it, and pay attention to the moment users start asking "who maintains this after the students graduate?" - that's when a project becomes a company. He's also insistent that the credit belongs to his students.From there, the conversation goes deep into what he's watching now: why the AI stack has become an order of magnitude more complex than the Hadoop/Spark era, why maximizing GPU utilization is "the name of the game" for any enterprise, and why coding agents will struggle with distributed systems long after they've mastered web apps. He shares a memorable reward-hacking story — a load balancer that maximized throughput by dropping requests — explains why the gap between open and closed models sits at about six months, and closes with his case for regulating AI by outcomes, not capabilities.Timeline00:00 — Introduction: welcoming Ion Stoica01:21 — The playbook: how research projects become companies05:22 — Will vLLM and SGLang stay open source?07:47 — The real bottleneck in the AI stack: complexity, not just hardware14:31 — Should algorithms follow infrastructure, or the other way around?16:13 — Can AI coding tools write distributed systems and GPU kernels?21:09 — Verifiers, harnesses, and the limits of outsourcing understanding25:41 — Reward hacking: the load balancer that dropped requests25:58 — How should enterprises consume GPUs? Utilization as the name of the game30:23 — GPU scarcity: will the compute crunch ever end?35:27 — Hyper-optimization and the risk of locking in today's architectures37:17 — Open vs. closed models: why every company wants to own the stack40:35 — The six-month gap, and the rising cost of training frontier models43:58 — Kimi, Qwen, and who's incentivized to keep open models alive45:39 — Regulation: outcomes, not capabilities47:41 — Self-regulation, concentration of power, and auditing open models48:32 — Wrap-upMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.AboutThe Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.
• 
  Kaggle Grandmasters, Agent Skills, and Why Everyone Is Overfitting with Jean-Francois Puget (NVIDIA) 13.07.2026 59min

  Jean-Francois Puget is a Director and Distinguished Engineer at NVIDIA, where he leads the Kaggle Grandmasters team, and he's ranked third on Kaggle's all-time list. We caught him on the day NVIDIA announced Nemotron Ultra and its new agent skills repo. We talk about what skills actually are, why they beat MCP tools on context cost, and how NVIDIA built an evaluation pipeline to separate skills that help from skills that don't.From there we talk about the thing JFP cares about most: evaluation. He explains why most LLM benchmarks reward overfitting, how his team discovered O3 could pick the right files to fix SWE-bench issues without reading them, and why the only benchmarks he trusts are the ones where you commit before you see the score, which is exactly how Kaggle works. He predicts a "bloodbath" for the wave of competitors letting coding agents chase leaderboard scores with no notion of validation.We also get into what coding agents are actually good for ("a mix of a genius and a dumb person"), the multi-agent system at NVIDIA that built a working PyTorch clone that runs 10x slower than the real thing, his unfiltered take on frontier lab PR and the Mythos release, whether AI is a bubble, and the story of how his team won ARC-AGI with a 4-billion-parameter model at 20 cents a task, including jumping from third to first in the final hours of a seven-month competition.Timeline00:00 — Intro01:05 — NVIDIA's announcements: Nemotron Ultra and the agent skills repo07:21 — Skills vs MCP tools, and progressive disclosure10:24 — Agents that write their own skills: a new form of learning13:33 — When overfitting is fine (and when it isn't)15:47 — Why most LLM benchmarks reward overfitting17:06 — The SWE-bench contamination story: O3 picks files without reading them19:45 — How LLMs changed Kaggle, and the coming "bloodbath"25:40 — What makes a good data scientist: evaluation and one-bit experiments28:56 — Running Codex at scale: the top token consumers at NVIDIA29:37 — Did coding agents kill AutoML?30:16 — Genius and dumb at once: the limits of coding agents35:21 — Humans in the loop, sandboxing, and the teenage hacker who never wrote code37:42 — Mythos, frontier lab PR, and open source40:08 — Why NVIDIA builds open models, and where it's already frontier43:48 — World models, robots, and the coffee test49:20 — Why agents still can't play Dota50:24 — Is AI a bubble?53:14 — Winning ARC-AGI with a 4B model at 20 cents a task57:39 — Kaggle is a legal drugMusic:"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.
• 
  AI Agents and The Golden Age of Asking Questions with Dimitris Papailiopoulos (MSR/UW-Madison) 09.07.2026 1t 13min

  In this episode, we talked with Dimitris Papailiopoulos, researcher at Microsoft Research's AI Frontiers lab and professor at the University of Wisconsin, about doing research in the age of agents. Dimitris told us about the Sunday morning that changed how he works: he handed Claude Code and Codex a question he'd been sitting on for years, went about his day, and came back to an answer. After a few days of dread about what's left for humans, he landed somewhere more optimistic, calling this the golden age of asking questions.We talked about his "smallest transformer that can add" leaderboard, a symbolic GSM8K solver built from if-else statements, and what happened when he put two Claude Code instances in the same file system and told them to do something cool (one pair invented a communication protocol, the other played Battleship). We also got into diversity and slop in agent-generated ideas, why agents get stubborn after a million tokens, harness overfitting on Terminal-Bench, continual learning and world models, whether agents need vision, and where information theory actually helps in AI and where it's a katana used to make coffee.Timeline00:00 Intro01:45 How agents changed the way Dimitris does research04:30 A Sunday morning with Claude Code, Codex, and GSM8K07:15 The dread, then the golden age of asking questions08:20 Taste and verification, and how we train students now09:53 Will models make human verification obsolete?11:30 The smallest transformer that can add 10-digit numbers13:40 Humans as initializers for gradient descent in idea space15:32 Allen on diversity, slop profiles, and high temperature research21:44 When Claudes meet: Battleship, invented protocols, and a grokking paper25:53 Single agent vs multi-agent under fixed compute30:28 Auto-research benchmarks and what agents actually accelerate35:14 Inside the symbolic GSM8K solver (with a live progress check)40:04 Idea overfitting and why agents refuse to change course44:00 Learning from failure traces and harness overfitting48:04 Continual learning, memory files, and world models51:30 Why don't labs personalize models on your own history?57:52 Agent-to-agent communication: is Jira the right tool?1:01:25 Multimodality: vision as a tool vs one unified model1:05:40 Information theory and AI, or making coffee with a katana1:11:23 Closing thoughts: ask bigger questionsMusic:"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.
• 
  Why All Models Learn the Same Thing with Phillip Isola (MIT) 02.07.2026 1t 11min

  Phillip Isola, professor at MIT, joins us to talk about representation learning: what makes a representation good, why different models seem to converge on similar representations, and whether pre-training is really over.We discuss the platonic representation hypothesis and its limits, why clustering structure matters more than global geometry, and Phillip's new neural thickets paper arguing that post-training is easier than people think because pre-trained weights already sit near solutions to downstream tasks. Phillip also explains why he thinks LLMs are already world models, why he's betting on RNNs making a comeback, and why his most exciting current direction is artificial life: putting LLM agents in open environments with no fixed task and studying them like new organisms.Timeline:00:00 Intro song00:13 Intro01:05 What is representation learning and why it matters04:09 What makes a representation good: minimality and sufficiency10:03 How cross entropy and contrastive learning shape representations14:35 Dimensionality reduction and why dimension isn't the right complexity measure16:35 Compression and geometric clustering during training19:27 The platonic representation hypothesis and what actually converges22:53 Local neighborhoods vs global structure: the Aristotelian follow-up24:33 When convergence is strong: truth vs the space of possibility28:09 Is there true similarity in the world? The Bouba-Kiki effect30:56 World models vs autoregressive LLMs32:14 Diffusion LLMs as a special case of autoregressive models33:42 What architectures win in five years: the case for RNNs36:11 Grad student descent, or do we actually have principles?40:51 Feathers and wings: what to take from biology43:17 How close are we to brain-like models? Marr's three levels47:01 Are better models becoming less human-like?49:38 Is pre-training all you need? The neural thickets paper54:18 LoRA, low rank fine-tuning, and why post-training is easier than we thought56:01 RL environments and what our benchmarks actually test1:01:11 Artificial life: LLM agents as new organisms1:07:20 What's overlooked in AI research right now1:08:36 Why stay in academia, and doing science in the age of OpusMusic:"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.