The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Astronomer
0
The Data Flowcast is a podcast dedicated to Apache Airflow, a workflow management system for data engineering and AI. Each week, the show explores the current state, future, and potential of Airflow with leading thinkers in the community. It provides insights on how to leverage Airflow to meet the evolving needs of data engineering and AI ecosystems. The podcast is produced by Astronomer, a company specializing in Airflow solutions.
Episodes
-
Introducing Airflow 3.4 19.11.2026 27mAirflow 3.4 is here, and it brings task loops, multi-language Dag SDKs, and a stable 1.0 release of the Common AI provider. Marc Lamberti, Education Content Lead at Astronomer, joins Kenten to walk through what's new, what it means for AI orchestration, and which smaller features are worth paying attention to. Key Takeaways: 00:00 Introduction. 01:12 Task loops (AIP-111) let you repeat tasks in a task group until a condition is met or a max iteration count is reached. 01:56 How task loops... -
Building AI for data engineering at Astronomer 12.11.2026 26mGeneric AI coding tools consistently fall short on Airflow work. Only 9% of users in last year's Airflow survey said they were satisfied with them. In this episode, Yetunde Dada, Senior Director of Product Management at Astronomer, and Stephanie Niu, Senior Product Manager at Astronomer, join Kenten to walk through Otto, Astronomer's data engineering agent built specifically for Airflow, and how it approaches investigations, upgrades, and evaluation. Otto is available now in Labs and will ent... -
Orchestrating Qualcomm’s Semiconductor Chip Design At Scale 05.11.2026 28mSemiconductor chip design is not where most people expect to find Apache Airflow, but Qualcomm's CPU team runs millions of pre-silicon simulations and post-silicon validation jobs through it every day. In this episode, Kenten Danas is joined by Dheeraj Turaga, Principal Engineer at Qualcomm and an Apache Airflow committer, to talk through two very different Airflow use cases: orchestrating semiconductor design workflows across on-prem compute farms, and driving robotic handlers in post-silico... -
Orchestrating cloud data migrations at HCL Tech 29.10.2026 20mLarge-scale cloud migrations can result in a lot to manage when it comes to orhcestration: multi-cloud targets, on-prem extraction, dynamic workloads, and strict operational controls. In this episode, Kenten Danas is joined by Bhanu Prakash, Technical Lead at HCL Tech, to talk through how his team uses Airflow to orchestrate end-to-end data and platform migrations to the cloud, generate Dags dynamically for micro-batch workloads, and automate incident response through a ServiceNow integration... -
Building self-service Dags with GitHub Actions at Lyft 22.10.2026 29mAt Lyft, a small data engineering team supports dozens of scientists and analysts who tune driver pay configurations on demand, all under SOX controls. In this episode, Ken Obata, Senior Software Engineer at Lyft, joins Kenten Danas to walk through how his team built self-service Dags with an event-driven integration between GitHub Actions and Airflow, and how they solved the data modeling and consistency problems that came with it. Key Takeaways: 00:00 Introduction. 02:49 How Airflow r... -
Orchestrating cost-efficient data pipelines at Insignis 15.10.2026 30mRunning Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep. In this episode, Kenten sits down with Aniruddha Sengupta, Staff Data Engineer at Insignis, to talk through how the UK cash-savings fintech runs a low-cost, scalable Airflow platform. The conversation covers observability, DAG-level best practices, the internal tools Insignis built to control costs, and where AI fits into the roadmap. Key Takeaways: 00... -
Scaling to 1,000+ Airflow Dags at Descartes / Idelic 08.10.2026 22mScaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with Matt Stavinga, Data Engineer at Idelic (now part of Descartes), about how his team standardized DAG creation and logging, built a Jenkins-driven testing workflow with per-PR Airflow environments, and is opening up DAG configuration and triggering to their own customers through the Safety Suite UI. Key Takeaways: 00:00 Introduction. 01:40 Idelic's mission: collecting... -
Orchestrating local LLMs for sensitive data with Airflow 01.10.2026 26mRunning LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before downstream systems can depend on their output. In this episode, Kenten is joined by Chhayank Jain, Senior ML Engineer at Visa, to talk through how Airflow orchestrates local LLM pipelines end to end: nightly inference over millions of records, silent failure handling with Pydantic and distribution checks, and using a separate DAG to ma... -
Improving security in Airflow 3 24.09.2026 22mSecurity is one of the least visible parts of Airflow for most users, but it shapes what features teams can turn on in production. In this episode, Kenten Danas talks with Anish Giri, Senior Software Engineer and active Apache Airflow contributor focused on security, about three recent contributions that touch nearly every Airflow 3 deployment: hardening connection testing, adding JWT revocation, and splitting execution tokens by scope. Key Takeaways: 00:00 Introduction. 01:16 Anish's ins... -
Orchestrating data across 30 companies at itti 17.09.2026 19mOrchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas Trubiano, Data Engineer at itti, the technology company within Grupo Vázquez in Paraguay. Lucas walks through the custom YAML framework his Center of Excellence built on top of Airflow, how they baked data quality and custom operators into it, and how a spec-driven AI workflow now lets product and business users contribute to templates... -
Building self-healing Airflow pipelines at ATC Drivetrain 10.09.2026 27mManufacturing data pipelines can't afford silent failures. When quality decisions and shop-floor visibility depend on Airflow, a broken DAG at midnight can cost real money. In this episode, Kenten Danas talks with Kumuda Sreenivasa, Founder of Receitly and Senior Data Architect at ATC Drivetrain, about how her team built self-healing Airflow pipelines, where AI fits into the recovery loop, and how they orchestrate AI agents as governed workflow components. Key Takeaways: 00:00 Introduction.... -
Orchestrating strictly sequential ETL pipelines at Synechron 03.09.2026 24mStrict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. Ivana Isailovic, Senior Big Data Engineer at Synechron, joins Marc Lamberti to walk through the three-layer DAG architecture her team built to solve exactly that, plus how they generate 200-task DAGs and how they rebuilt subdag-style group retries in Airflow 3. Key Takeaways: (00:00) Introduction. (02:16) The stack: Snowflake source, Mo... -
Managing financial datasets with Airflow at Wise 27.08.2026 28mFinancial data pipelines have to be right the first time. On this episode, Kenten sits down with [Antonello Benedetto](linkedin.com/in/anbento4), Staff Data Engineer at Wise, to talk about how the central data and analytics engineering team runs Airflow for critical financial datasets, tiers pipelines by reliability, and orchestrates LLM-enabled workflows with validation layers and agent-checking-agent patterns. Key Takeaways: 00:00 Introduction.01:47 What Wise does and Antonello's role in ... -
Managing travel platform data with Airflow at Headout 20.08.2026 28mRunning a travel platform means dealing with fast-moving inventory, real-time fraud detection, and heavy performance marketing attribution, all while keeping data trustworthy for a data-driven org. In this episode, Mrinalini Singh, Data Platform Engineer at [Headout](headout.com), walks through how her team uses Airflow as the nervous system of their stack: orchestrating dbt with a write-audit-publish pattern, running ML training and inference, and wiring up alerting that points to the exact ... -
How Airflow orchestration decisions impact Spark performance 13.08.2026 19mAirflow and Spark are one of the most common combinations in modern data platforms, but the orchestration decisions made on the Airflow side often determine whether Spark jobs run fast and cheap or slow and expensive. In this episode, Kenten is joined by [Meni Shmueli](linkedin.com/in/meni-shmueli-dataflint), Co-Founder and CEO at [DataFlint](dataflint.io), to dig into how Airflow and Spark fit together, where teams go wrong, and how AI is changing the way they reason about cost and performan... -
Using Airflow for diverse client projects at Accion Labs 06.08.2026 27mWhen a real-time loan eligibility scoring pipeline is built on cron jobs, midnight pages are inevitable. In this episode, [Chandan Gowda](linkedin.com/in/chandan-gowda-a-h-744908195), Data Engineer at [Accion Labs](accionlabs.com), joins Kenten to discuss how his team uses Airflow across client projects, including a financial services scoring use case and a POC applying production-grade orchestration to RAG and GenAI data pipelines. Key Takeaways: 00:00 Introduction.01:00 What Accion Labs d... -
Consolidating legacy job schedulers with Airflow at Trading Technologies 30.07.2026 31mMigrating two decades of legacy job orchestration is the kind of project most teams quietly avoid. In this episode, Marc Lamberti is joined by [Sanket Patel](linkedin.com/in/sanket-patel-a2a57b3a), Director of Engineering at Trading Technologies, to talk through how his team is consolidating 20 years of C#/.NET-based scheduling and a recently acquired company's stack onto Airflow, Astronomer, and Snowflake. The conversation covers orchestrator selection, observability, the operational economi... -
Orchestrating AI video intelligence and evaluation pipelines at Firework 23.07.2026 22mHow do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at [Firework](firework.com), joins the show to walk through how his team uses Airflow to coordinate ingestion, embeddings, and evaluation pipelines for AI features in a video commerce platform. The conversation covers the technical stack, the challenges of validating LLM output, cost guardrails, and what Shawn wants to see next from the Air... -
Orchestrating Retail Data Pipelines at Saks Global 16.07.2026 25mSaks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud detection systems into Snowflake. In this episode, [Shailesh Kadam](linkedin.com), Architect at [Saks Global](saks.com), joins Kenten to walk through how Airflow acts as the nervous system tying it all together, why they moved from self-managed Kubernetes to Astro, what is driving their Airflow 3 upgrade, and how they are approaching age... -
What's New in Apache Airflow® 3.3 09.07.2026 27mAirflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer](astronomer.io), joins Kenten Danas to walk through what's new in the release and where each feature actually pays off. Key Takeaways: 00:00 Introduction.01:46 The new task state store (AIP-1...
Popular in
The podcast also appears in the podcast charts of these countries.