The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Astronomer
Country USA
Genres Technology
Language EN
Episodes 104
Latest 19.11.2026

The Data Flowcast is a podcast dedicated to Apache Airflow, a workflow management system for data engineering and AI. Each week, the show explores the current state, future, and potential of Airflow with leading thinkers in the community. It provides insights on how to leverage Airflow to meet the evolving needs of data engineering and AI ecosystems. The podcast is produced by Astronomer, a company specializing in Airflow solutions.

Episodes

  • Introducing Airflow 3.4
    Introducing Airflow 3.4 19.11.2026 27m
    Airflow 3.4 is here, and it brings task loops, multi-language Dag SDKs, and a stable 1.0 release of the Common AI provider. Marc Lamberti, Education Content Lead at Astronomer, joins Kenten to walk through what's new, what it means for AI orchestration, and which smaller features are worth paying attention to. Key Takeaways: 00:00 Introduction. 01:12 Task loops (AIP-111) let you repeat tasks in a task group until a condition is met or a max iteration count is reached. 01:56 How task loops...
  • Building AI for data engineering at Astronomer
    Building AI for data engineering at Astronomer 12.11.2026 26m
    Generic AI coding tools consistently fall short on Airflow work. Only 9% of users in last year's Airflow survey said they were satisfied with them. In this episode, Yetunde Dada, Senior Director of Product Management at Astronomer, and Stephanie Niu, Senior Product Manager at Astronomer, join Kenten to walk through Otto, Astronomer's data engineering agent built specifically for Airflow, and how it approaches investigations, upgrades, and evaluation. Otto is available now in Labs and will ent...
  • Orchestrating Qualcomm’s Semiconductor Chip Design At Scale
    Orchestrating Qualcomm’s Semiconductor Chip Design At Scale 05.11.2026 28m
    Semiconductor chip design is not where most people expect to find Apache Airflow, but Qualcomm's CPU team runs millions of pre-silicon simulations and post-silicon validation jobs through it every day. In this episode, Kenten Danas is joined by Dheeraj Turaga, Principal Engineer at Qualcomm and an Apache Airflow committer, to talk through two very different Airflow use cases: orchestrating semiconductor design workflows across on-prem compute farms, and driving robotic handlers in post-silico...
  • Orchestrating cloud data migrations at HCL Tech
    Orchestrating cloud data migrations at HCL Tech 29.10.2026 20m
    Large-scale cloud migrations can result in a lot to manage when it comes to orhcestration: multi-cloud targets, on-prem extraction, dynamic workloads, and strict operational controls. In this episode, Kenten Danas is joined by Bhanu Prakash, Technical Lead at HCL Tech, to talk through how his team uses Airflow to orchestrate end-to-end data and platform migrations to the cloud, generate Dags dynamically for micro-batch workloads, and automate incident response through a ServiceNow integration...
  • Building self-service Dags with GitHub Actions at Lyft
    Building self-service Dags with GitHub Actions at Lyft 22.10.2026 29m
    At Lyft, a small data engineering team supports dozens of scientists and analysts who tune driver pay configurations on demand, all under SOX controls. In this episode, Ken Obata, Senior Software Engineer at Lyft, joins Kenten Danas to walk through how his team built self-service Dags with an event-driven integration between GitHub Actions and Airflow, and how they solved the data modeling and consistency problems that came with it. Key Takeaways: 00:00 Introduction. 02:49 How Airflow r...
  • Orchestrating cost-efficient data pipelines at Insignis
    Orchestrating cost-efficient data pipelines at Insignis 15.10.2026 30m
    Running Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep. In this episode, Kenten sits down with Aniruddha Sengupta, Staff Data Engineer at Insignis, to talk through how the UK cash-savings fintech runs a low-cost, scalable Airflow platform. The conversation covers observability, DAG-level best practices, the internal tools Insignis built to control costs, and where AI fits into the roadmap. Key Takeaways: 00...
  • Scaling to 1,000+ Airflow Dags at Descartes / Idelic
    Scaling to 1,000+ Airflow Dags at Descartes / Idelic 08.10.2026 22m
    Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with Matt Stavinga, Data Engineer at Idelic (now part of Descartes), about how his team standardized DAG creation and logging, built a Jenkins-driven testing workflow with per-PR Airflow environments, and is opening up DAG configuration and triggering to their own customers through the Safety Suite UI. Key Takeaways: 00:00 Introduction. 01:40 Idelic's mission: collecting...
  • Orchestrating local LLMs for sensitive data with Airflow
    Orchestrating local LLMs for sensitive data with Airflow 01.10.2026 26m
    Running LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before downstream systems can depend on their output. In this episode, Kenten is joined by Chhayank Jain, Senior ML Engineer at Visa, to talk through how Airflow orchestrates local LLM pipelines end to end: nightly inference over millions of records, silent failure handling with Pydantic and distribution checks, and using a separate DAG to ma...
  • Improving security in Airflow 3
    Improving security in Airflow 3 24.09.2026 22m
    Security is one of the least visible parts of Airflow for most users, but it shapes what features teams can turn on in production. In this episode, Kenten Danas talks with Anish Giri, Senior Software Engineer and active Apache Airflow contributor focused on security, about three recent contributions that touch nearly every Airflow 3 deployment: hardening connection testing, adding JWT revocation, and splitting execution tokens by scope. Key Takeaways: 00:00 Introduction. 01:16 Anish's ins...
  • Orchestrating data across 30 companies at itti
    Orchestrating data across 30 companies at itti 17.09.2026 19m
    Orchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas Trubiano, Data Engineer at itti, the technology company within Grupo Vázquez in Paraguay. Lucas walks through the custom YAML framework his Center of Excellence built on top of Airflow, how they baked data quality and custom operators into it, and how a spec-driven AI workflow now lets product and business users contribute to templates...
  • Building self-healing Airflow pipelines at ATC Drivetrain
    Building self-healing Airflow pipelines at ATC Drivetrain 10.09.2026 27m
    Manufacturing data pipelines can't afford silent failures. When quality decisions and shop-floor visibility depend on Airflow, a broken DAG at midnight can cost real money. In this episode, Kenten Danas talks with Kumuda Sreenivasa, Founder of Receitly and Senior Data Architect at ATC Drivetrain, about how her team built self-healing Airflow pipelines, where AI fits into the recovery loop, and how they orchestrate AI agents as governed workflow components. Key Takeaways: 00:00 Introduction....
  • Orchestrating strictly sequential ETL pipelines at Synechron
    Orchestrating strictly sequential ETL pipelines at Synechron 03.09.2026 24m
    Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. Ivana Isailovic, Senior Big Data Engineer at Synechron, joins Marc Lamberti to walk through the three-layer DAG architecture her team built to solve exactly that, plus how they generate 200-task DAGs and how they rebuilt subdag-style group retries in Airflow 3. Key Takeaways: (00:00) Introduction. (02:16) The stack: Snowflake source, Mo...
  • Managing financial datasets with Airflow at Wise
    Managing financial datasets with Airflow at Wise 27.08.2026 28m
    Financial data pipelines have to be right the first time. On this episode, Kenten sits down with [Antonello Benedetto](linkedin.com/in/anbento4), Staff Data Engineer at Wise, to talk about how the central data and analytics engineering team runs Airflow for critical financial datasets, tiers pipelines by reliability, and orchestrates LLM-enabled workflows with validation layers and agent-checking-agent patterns. Key Takeaways: 00:00 Introduction.01:47 What Wise does and Antonello's role in ...
  • Managing travel platform data with Airflow at Headout
    Managing travel platform data with Airflow at Headout 20.08.2026 28m
    Running a travel platform means dealing with fast-moving inventory, real-time fraud detection, and heavy performance marketing attribution, all while keeping data trustworthy for a data-driven org. In this episode, Mrinalini Singh, Data Platform Engineer at [Headout](headout.com), walks through how her team uses Airflow as the nervous system of their stack: orchestrating dbt with a write-audit-publish pattern, running ML training and inference, and wiring up alerting that points to the exact ...
  • How Airflow orchestration decisions impact Spark performance
    How Airflow orchestration decisions impact Spark performance 13.08.2026 19m
    Airflow and Spark are one of the most common combinations in modern data platforms, but the orchestration decisions made on the Airflow side often determine whether Spark jobs run fast and cheap or slow and expensive. In this episode, Kenten is joined by [Meni Shmueli](linkedin.com/in/meni-shmueli-dataflint), Co-Founder and CEO at [DataFlint](dataflint.io), to dig into how Airflow and Spark fit together, where teams go wrong, and how AI is changing the way they reason about cost and performan...
  • Using Airflow for diverse client projects at Accion Labs
    Using Airflow for diverse client projects at Accion Labs 06.08.2026 27m
    When a real-time loan eligibility scoring pipeline is built on cron jobs, midnight pages are inevitable. In this episode, [Chandan Gowda](linkedin.com/in/chandan-gowda-a-h-744908195), Data Engineer at [Accion Labs](accionlabs.com), joins Kenten to discuss how his team uses Airflow across client projects, including a financial services scoring use case and a POC applying production-grade orchestration to RAG and GenAI data pipelines. Key Takeaways: 00:00 Introduction.01:00 What Accion Labs d...
  • Consolidating legacy job schedulers with Airflow at Trading Technologies
    Consolidating legacy job schedulers with Airflow at Trading Technologies 30.07.2026 31m
    Migrating two decades of legacy job orchestration is the kind of project most teams quietly avoid. In this episode, Marc Lamberti is joined by [Sanket Patel](linkedin.com/in/sanket-patel-a2a57b3a), Director of Engineering at Trading Technologies, to talk through how his team is consolidating 20 years of C#/.NET-based scheduling and a recently acquired company's stack onto Airflow, Astronomer, and Snowflake. The conversation covers orchestrator selection, observability, the operational economi...
  • Orchestrating AI video intelligence and evaluation pipelines at Firework
    Orchestrating AI video intelligence and evaluation pipelines at Firework 23.07.2026 22m
    How do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at [Firework](firework.com), joins the show to walk through how his team uses Airflow to coordinate ingestion, embeddings, and evaluation pipelines for AI features in a video commerce platform. The conversation covers the technical stack, the challenges of validating LLM output, cost guardrails, and what Shawn wants to see next from the Air...
  • Orchestrating Retail Data Pipelines at Saks Global
    Orchestrating Retail Data Pipelines at Saks Global 16.07.2026 25m
    Saks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud detection systems into Snowflake. In this episode, [Shailesh Kadam](linkedin.com), Architect at [Saks Global](saks.com), joins Kenten to walk through how Airflow acts as the nervous system tying it all together, why they moved from self-managed Kubernetes to Astro, what is driving their Airflow 3 upgrade, and how they are approaching age...
  • What's New in Apache Airflow® 3.3
    What's New in Apache Airflow® 3.3 09.07.2026 27m
    Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer](astronomer.io), joins Kenten Danas to walk through what's new in the release and where each feature actually pays off. Key Takeaways: 00:00 Introduction.01:46 The new task state store (AIP-1...

Popular in

The podcast also appears in the podcast charts of these countries.