The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Episode

• Running Airflow 3 in a regulated environment at OTPP 25.06.2026 18mnt

  Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan](otpp.com), joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execution.Key Takeaways:00:00 Introduction.01:18 Inside the OTPP data platform team and what they're responsible for across cloud migration, standards, and enablement.02:33 What's driving OTPP's multi-year move off on-prem to a cloud architecture built around scalability and resilience.02:57 The new stack: Snowflake as the enterprise data platform, dbt for transformation, and Airflow as the orchestrator in the middle.04:15 Why OTPP chose Astronomer: active contributions to the Airflow OSS project, fast runtime releases, and built-in monitoring, observability, and RBAC.05:50 Evolving from dbt core with Bash operators to dbt Cosmos for model-level granularity, lineage, and precise failure recovery, plus a performance boost from watcher mode.08:00 Upgrading from Airflow 2.9 to Airflow 3, using the Astro CLI and linters to catch deprecations quickly.09:32 The drivers behind adopting remote execution: keeping data inside the security perimeter and scaling workloads on their own Kubernetes cluster.11:35 How remote execution replaced a complex network architecture of VPN tunnels and firewall rules, removing latency along the way.12:53 The POV process, success criteria, and a six week timebox to validate remote execution before going to production.14:14 Going fully live: OTPP's last hosted deployment was sunset just before recording.15:06 What Kowsy wants next from Airflow: AI orchestration capabilities and continued maturation of remote execution.Resources Mentioned:[Ontario Teachers' Pension Plan](otpp.com)[Apache Airflow](airflow.apache.org)[Astronomer](astronomer.io)[Cosmos](astronomer.io/cosmos)Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow
• Managing a Customer Analytics Platform with Airflow at Skimlinks 11.06.2026 22mnt

  Skimlinks runs a reporting platform that serves around 2,000 weekly publisher users, and the data infrastructure behind it runs on Airflow. In this episode, Julian Larralde, Director of Data Engineering at Skimlinks, walks through the stack, the migration from external task sensors to event-driven Assets, and a YAML-based DAG factory the team built to onboard new publishers without rewriting Python.Key Takeaways00:00 Introduction.00:45 What Skimlinks does and how it operates as an affiliate marketing network aggregator for publishers.02:12 Julian's team and the data platform they own: a reporting portal that serves ~2,000 weekly publisher users.03:07 The stack: real-time ingestion into BigQuery, Airflow as the orchestrator, raw / silver / gold layers, and Apache Druid as the serving database for sub-second BI queries.04:50 Reusing the same data marts for ~100 internal customers across marketing, finance, operations, and account management.06:25 Airflow as the single orchestrator: BigQuery operators for SQL business logic, plus raw file exports for the largest publishers.08:08 Moving from external task sensors to datasets (now Assets) and what the migration actually solved.09:18 Why sensor polling created scheduler load and worker overload, and how event-driven Assets fixed both.10:15 The lineage view in the Airflow UI that came as a bonus after the Assets migration.10:49 The vision for multi-tenant Airflow inside Skimlinks: replacing cron, Rundeck, and team-local Airflow instances with a shared platform.14:31 Building a custom DAG factory with YAML configuration for onboarding new publishers.17:33 Breaking a single Python class into single-responsibility components for the DataPipe project.19:07 Adding a Pydantic layer so misconfigured YAML fails at DAG parse time instead of run time.20:31 Using AI assistance to guide refactoring decisions and generate tests across the new class structure.22:34 What Julian wants from Airflow next: asset watchers paired with data contracts.Resources MentionedSkimlinks - skimlinks.comApache Airflow - airflow.apache.orgAstronomer - astronomer.ioGoogle BigQuery - cloud.google.com/bigqueryApache Druid - druid.apache.orgPydantic - docs.pydantic.devLooker - cloud.google.com/lookerThanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow
• Building a custom Tableau provider for Airflow at JLR 04.06.2026 21mnt

  JLR is the UK's largest automotive manufacturer, behind brands like Range Rover, Jaguar, Defender, and Discovery. In this episode, Najeeb Sulaiman, Senior Data Engineer at JLR, walks through how Airflow orchestrates data across manufacturing, supply chain, and finance — including a custom Tableau provider his team built (after the community version dropped PAT authentication) and a CI/CD pipeline that validates DAGs before they reach production.Key Takeaways:00:00 Introduction.00:48 What JLR makes: luxury vehicles under the Range Rover, Jaguar, Defender, and Discovery brands.01:42 Najeeb's team in the Data and AI Office, supporting manufacturing, supply chain, finance, and commerce analytics.03:25 Airflow as the central nervous system of the JLR data stack — the orchestrator that connects every source and downstream system.05:01 How JLR uses Tableau, and the two modes for getting data in: live connection and scheduled extract refresh.06:24 Why scheduled Tableau refreshes go stale: they aren't aware of when the data pipeline actually finished.08:09 First attempt at solving it: Python scripts calling the Tableau REST API directly.08:47 Why the script approach didn't scale across teams — code duplication and version drift.10:00 Trying the community Airflow Tableau provider and hitting the PAT authentication roadblock.12:21 Building a custom provider on top of the community one to keep PAT auth.13:30 Treating CI/CD as a deployment gate for Airflow DAGs at JLR's scale.15:23 What the CI/CD pipeline actually catches: top-level code making external calls, import errors, and Airflow 3 compatibility.17:47 How the gate blocks broken DAGs from reaching production.18:30 What Najeeb wants from Airflow next: native integration testing, better OpenTelemetry support, and built-in lineage.Resources Mentioned:JLR - jaguarlandrover.comApache Airflow - airflow.apache.orgAstronomer - astronomer.ioTableau - tableau.comTableau REST API - help.tableau.com/current/api/rest_api/en-us/REST/rest_api.htmAirflow Tableau provider (community) - airflow.apache.org/docs/apache-airflow-providers-tableauThanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow
• Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira 28.05.2026 32mnt

  Running Airflow at the scale of a national retailer means more than just scheduling. It means giving non-engineers a path to ship DAGs, and classifying thousands of runs to know which ones need attention. In this episode, Mateus Ferreira, Senior Data Engineer at Luiza Labs (the technology arm of Magazine Luiza, one of Brazil's largest retailers), joins Marc to talk about the patterns his team uses to run 2,000+ Airflow pipelines across more than four petabytes of data.Key Takeaways:00:00 Introduction01:11 Mateus introduces himself and Luiza Labs, the technology arm of Magazine Luiza (Magalu), one of Brazil's largest retailers (founded 1957). 1,000+ physical stores, multi-region operations, and a data team that has to handle the variability that comes with all of it.04:33 Lu Brain, Magalu's AI initiative built around their character Lu, and how AI fits into the data work.06:47 The data reliability engineering channel where AI summarizes Airflow errors with confidence scores and posts a suggested fix in chat.08:30 How Airflow became the heart of orchestration. Coming from Control-M in banking, then GCP, then consolidating on Cloud Composer to centralize roughly 2,000 pipelines.14:23 The YAML wrapper that lets non-engineers ship DAGs. Reads namespace, tables, and Spark options. Handles CDC, JDBC full, and JDBC incremental collection types with checkpoints. All changes go through data reliability engineering.17:20 Why metadata is the most valuable asset in the AI era, and how the wrapper makes data lineage observable across 2,000 pipelines.18:26 The Data Reliability Engineering team. A 10-person group that is the window to the company, handling maintenance, validation, corrections, and optimization for the business unit pipelines.20:09 Operating at four petabytes of data.21:24 Why they built custom Spark operators. Cost drove the move off the DataprocOperator. The custom operator exposes Spark driver and executor sizing as Airflow parameters and generates the Kubernetes manifest.24:36 The monitoring dashboard built on the Airflow metadata DB. A timeline view that shows how many DAGs run each hour, used to spread scheduling across the day.26:37 Classifying DAGs by their last five runs: success, partially correct, intermittent, total failure. A reusable observability pattern.29:57 How to reach Mateus, and a closing thought in Portuguese on appreciating the good old times while you are living them.Resources Mentioned:Apache Airflow (airflow.apache.org)Magalu Cloud / MGCLuiza Labs (luizalabs.com) and Magazine Luiza / MagaluAstro Observe (https://www.astronomer.io/product)Mateus Ferreira on LinkedIn (linkedin.com/in/mateusmferreira)Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow
• Enhancing DAGs for Data Processing with William Orgertrice III at Cargill 21.05.2026 26mnt

  In the data engineering world, the difference between a pipeline that works and one that's truly production-ready often comes down to a handful of deliberate decisions. William Orgertrice III, Data Engineer at Cargill, joins us to share the DAG design and monitoring practices he presented at Airflow Summit 2025 and how his team is rolling out Airflow across 60+ internal teams as part of Cargill's new Minerva data platform.Key Takeaways:00:00 Introduction. 01:45 Cargill is one of the largest privately owned companies in the US, operating across 70 countries and serving 125+ markets.03:45 William's team on the Cargill Data Platform supports 60+ internal teams, providing data products that drive decisions across finance, inventory and operations.05:10 Cargill chose Airflow as a core component of its new Minerva data platform to replace older ETL tooling with a more supportable, observable stack.06:26 Native SLA sensors and dependency management were specific features that made Airflow the right fit for Cargill's batch ingestion pipelines.09:00 Cargill is running Airflow through Astronomer as their managed solution, with some teams already in production.13:22 Every task in a DAG should have a single, documented purpose — one task doing everything makes troubleshooting significantly harder.14:40 A DAG that never enters a failed state but keeps running indefinitely will spend compute budget without alerting anyone.15:25 In shared Airflow environments, embedding contact information and owner tags in DAGs ensures the right team is reached when something breaks upstream.21:00 William flags connection testing as a friction point in pipeline development — verifying a connection string before building the full job would reduce iteration time.Resources Mentioned:Cargill | Websitehttps://www.cargill.com/food-beverageAirflow Community on Slack https://airflow.apache.org/community/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow
• Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer 14.05.2026 27mnt

  In this episode, we take a step back from implementation-specific topics to explore what it actually takes to build a career in data engineering — and how AI is reshaping that path.Shrividya Hegde,  a data and AI engineer and an Airflow champion in Astronomer’s Champions program, joins us to discuss getting into data engineering, contributing to open source and why good data engineering should make AI output trustworthy rather than confidently wrong.Key Takeaways:00:00 Introduction.04:08 Build fundamentals before chasing trending tools — understanding what a tool does, why it exists and what problem it solves has to come first. 07:19 Data engineering fundamentals mean SQL query performance under joins and aggregations, how data moves between pipelines, DAG failure recovery and idempotency — not just writing queries. 08:10 The most common mistake newer data engineers make is skipping fundamentals to chase trends — it is a sequencing problem, not a talent problem. 13:15 AI creates more opportunity for data engineers because AI output quality is directly determined by the quality of the data pipeline feeding it — confidently wrong output is harder to catch than obviously wrong output. 15:06 Airflow's supporting operators make AI outputs production-ready — orchestration is what converts experimental AI into something reliable. 17:14 AI-generated DAGs help newer engineers understand underlying concepts rather than just producing working code. 23:12 The Airflow open source community is more welcoming than most people expect for a project of its size — raising issues and reviewing PRs are viable entry points for first contributions.Resources Mentioned:Shrividya Hegdehttps://www.linkedin.com/in/shrividya-hegde-shri-91562365/Astronomer | LinkedInhttps://www.linkedin.com/company/astronomer/Astronomer | Websitehttps://www.astronomer.ioWomen in Data | Websitehttps://womenindata.mn.co/landingApache Airflow Slack https://airflow.apache.org/Shrividya's Medium writinghttps://medium.com/@shrihegdeShrividya’ Substack writinghttps://substack.com/@shrividyahegdeThanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning
• Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development 07.05.2026 24mnt

  We explore how a third-party logistics platform built its entire data orchestration layer on Airflow, and what that makes possible for developer teams and merchant-facing products alike.Filip Kunčar, Platform Director at ShipMonk Product Development, discusses migrating from a closed source tool to Airflow, orchestrating dbt with both Cosmos and the BashOperator and using Airflow to power customer-facing data delivery.Key Takeaways:00:00 Introduction.01:07 ShipMonk is a third-party logistics company guaranteeing two-day delivery across the US. The data platform team's mission is to lower cognitive load for developers working with data. 05:13 ShipMonk migrated to Airflow in 2022, moving away from a closed-source UI-based tool, driven by the need for a code-first approach, open source extensibility and broad cloud provider support. 10:02 The team uses Cosmos for developer-facing visibility and lineage and BashOperator for internal pipelines where runtime performance matters. 12:20 Switching from Cosmos to the BashOperator for a frequently running pipeline reduced runtime from over 15 minutes to three minutes. 13:14 Because the full dbt chain runs inside Airflow, a configurable downstream DAG can deliver processed data directly to each merchant's preferred destination, with secrets management and SLA tracking already handled. 15:03 Per-team alerting is hooked to each DAG by owner and severity, so teams can react to SLA breaches immediately. 18:09 ShipMonk uses Airflow in three ways for AI: authoring DAGs faster with skills, orchestrating AI workloads in Lambda and containers and using Astronomer's skills repo to simplify Airflow version upgrades.Resources Mentioned:Filip Kunčarhttps://www.linkedin.com/in/filipkuncar/ShipMonk Product Developmenthttps://www.linkedin.com/company/shipmonk-product-development/ShipMonk | Websitehttp://www.shipmonk.comAstronomer Cosmoshttp://www.astronomer.io/cosmosAstronomer AI Skills Repohttp://www.github.com/astronomer/airflow-llm-providers-demoDatadoghttp://www.datadoghq.comThanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning
• Building Airflow CTL with Buğra Öztürk at Mollie 30.04.2026 19mnt

  Buğra Öztürk, Senior Data Engineer at Mollie and Committer and PMC member on the Apache Airflow project, joins us to walk through Airflow CTL — what it is, how it differs from the existing Airflow CLI and where it is headed under AIP-94.Key Takeaways:00:00 Introduction.03:10 Buğra has contributed to Airflow since 2022, from docs changes up to Committer and PMC member — a path he hopes inspires others to start small and contribute. 04:05 Airflow CTL solves secure user interaction by abstracting database credentials behind the public core API. 05:13 Airflow CLI and Airflow CTL are complementary — CLI handles administration and database management while CTL handles secure user interactions via the API. 07:08 Airflow CTL authenticates via the API, acquires a JWT token and stores it securely in the OS keyring — running on the user's machine and never requiring direct database access.08:21 Concrete use cases include local DAG development without the UI and CI/CD automation using headless mode with short-lived JWT tokens.10:08 AIP-94 describes the long-term vision — decoupling all remote commands from the Airflow CLI and routing them through Airflow CTL. 13:12 Airflow CTL is currently at 0.X and already being used in CI and deployment automations. The move to 1.0 with full CLI parity is the next milestone under AIP-94.   16:09 Multi-team deployment becoming generally available in a future Airflow release is Buğra's most-anticipated upcoming feature beyond Airflow CTL.Resources Mentioned:Buğra Öztürkhttps://www.linkedin.com/in/bugraozturk93/Molliehttps://www.linkedin.com/company/mollie/Mollie | Websitehttps://www.mollie.com/Apache Airflow CTL https://airflow.apache.org/AIP-94 on Airflow Confluencehttps://lists.apache.org/thread/d2o1pr78wxdp1wozq519stp0pkcv6k6cApache Airflow GitHubhttps://www.github.com/apache/airflowThanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning
• Introducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik 23.04.2026 28mnt

  In this episode, we explore the newly released Apache Airflow common AI provider — what problem it solves, how it was built and what's coming next.Kaxil Naik, Senior Director of Engineering at Astronomer and Apache Airflow PMC member, and Pavan Kumar Gopidesu, Lead Data Engineer at Experian and Apache Airflow PMC member, join us to walk through the provider's first release and the technical decisions behind it.Key Takeaways:00:00 Introduction.04:05 The common AI provider was born from a real production problem.07:10 Airflow already had the primitives needed for durable agent execution, making it the natural foundation for AI orchestration. 09:15 The LLM schema compare operator uses Apache DataFusion to fetch source schemas.11:07 Apache DataFusion was chosen for its speed.13:09 Hook tool sets expose Airflow's provider hooks to agents with an allowed methods list that blocks destructive operations.15:20 Passing durable=True to an LLM operator caches tool calls and LLM outputs mid-task. 18:13 The provider offers three abstraction levels. 21:20 The provider currently requires Airflow 3 — the team is open to adding Airflow 2.11 support if demand is high enough. 24:10 MCP server configs can be stored as Airflow connections.Resources Mentioned:Kaxil Naikhttps://www.linkedin.com/in/kaxil/Pavan Kumar Gopidesuhttps://www.linkedin.com/in/pavan-kumar-gopidesu/Astronomer | LinkedInhttps://www.linkedin.com/company/astronomer/Astronomer | Websitehttps://www.astronomer.ioExperianhttps://www.linkedin.com/company/experian/Apache Airflowhttps://www.linkedin.com/company/apache-airflowApache Airflow common AI provider docshttps://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/commits.htmlApache DataFusionhttps://datafusion.apache.org/Pydantic AIhttps://pydantic.dev/docs/ai/overview/Airflow Slackhttps://airflow.apache.org/docs/apache-airflow-providers-slack/stable/index.htmlIntroducing the Common AI Provider: LLM and AI Agent Support for Apache Airflowhttps://airflow.apache.org/blog/common-ai-provider/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#Automation #Airflow #MachineLearning
• Building AI Debugging Agents Into Airflow DAGs at Jeppesen ForeFlight with Samantha Blaney Cuevas 16.04.2026 22mnt

  Aviation data pipelines run on strict 28-day publication cycles, and the margin for error is zero. In this episode, we're joined by Samantha Blaney Cuevas, Software Engineer at Jeppesen ForeFlight, to explore how her team orchestrates a complex, time-sensitive data pipeline with Airflow and where AI is starting to fit into that picture.Key Takeaways:00:00 Introduction.04:05 Airflow orchestrates almost all business logic and data transformations across the cycle, with custom timetables built to track busy and slow periods programmatically.06:10 Cycle-aware sensing tasks handle irregular source deliveries, including duplicates and early or late arrivals, without disrupting the pipeline.08:07 The two main AI use cases are pipeline debugging and cycle awareness — both designed to reduce the manual overhead of monitoring a complex DAG dependency graph.09:03 The Data Port agent is a two-task DAG that routes Slack pipeline alerts to either a predefined command list or an AI token, depending on whether the fix is already known.13:10 AI is still in development at Jeppesen ForeFlight — the team is focused on token efficiency and scoping how much autonomy to give agents across different environments.15:04 Airflow setup and MCP configuration were straightforward — the harder design work was deciding which environments agents could access across QA staging and production.17:06 Airflow's skills repo and agent tooling are helping onboard new developers and extend pipeline awareness to analysts who work alongside engineers on the cycle.19:10 Samantha would like to see single-task retries with different parameters in Airflow — resetting one task without clearing the full pipeline run.21:05 A future AI use case under consideration is live DAG editing and re-upload within Airflow to make one-off fixes without halting pipeline progress.Resources Mentioned:Samantha Blaney Cuevashttps://www.linkedin.com/in/samantha-blaney/Jeppesen ForeFlight | LinkedInhttps://www.linkedin.com/company/jeppesen-foreflight/Jeppesen ForeFlight | Websitehttp://www.foreflight.comAstronomer Airflow Skills Repohttp://www.github.com/astronomer/airflow-llm-providers-demoApache Airflow https://airflow.apache.org/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow
• Introducing Airflow 3.2 09.04.2026 26mnt

  We introduce Airflow 3.2 and its updates for teams that build and operate data pipelines.Astronomer’s Head of Customer Education, Marc Lamberti, and Senior Manager of Developer Relations, Kenten Danas, break down what’s new, from asset partitioning to Async Python tasks and DAG versioning. They explore how these updates improve scheduling, performance and observability in production workflows.Key Takeaways:00:00 Introduction.02:10 Airflow 3 architecture separates workers from the metadata database.03:05 Plugin versioning and UI-based backfills simplify operations.06:20 Asset partitioning enables granular, partition-level scheduling.07:15 Triggering DAGs on partitions instead of full datasets.11:05 Deferrable operators reduce worker slot usage.12:00 Async operators reduce database pressure and overhead.14:10 Async improves throughput, not single task speed.22:20 Inlets and outlets improve asset lineage visibility.23:00 DAG version markers show changes directly in the UI.Resources Mentioned:Marc Lambertihttps://www.linkedin.com/in/marclamberti/Apache Airflow https://airflow.apache.org/Astronomer | LinkedInhttps://www.linkedin.com/company/astronomer/Astronomer | Websitehttps://www.astronomer.io/3.2 Webinarhttps://www.astronomer.io/events/webinars/introducing-airflow-3-2-videoAsset Partitioning Guidehttps://www.astronomer.io/docs/learn/airflow-partitioned-runsAsynchronous Processes Guidehttps://www.astronomer.io/docs/learn/deferrable-operatorsRelease Noteshttps://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-3-2-0-2026-04-07Provider Registryhttps://airflow.apache.org/registry/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning
• Reflections on a Decade of Data Engineering at Seattle Data Guy 03.04.2026 26mnt

  Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent.In this episode, Benjamin Rogojan, Owner and Data Consultant at Seattle Data Guy, joins us to reflect on how the data engineering landscape has evolved alongside Apache Airflow. We explore when Airflow makes sense as an orchestrator, why batch processing is still dominant and how AI is reshaping the workflows and responsibilities of modern data engineers.Key Takeaways:00:00 Introduction.03:00 Airflow becomes valuable when workflows involve many pipelines, teams and dependencies.05:00 Data engineers are still focused on making data accessible and aligning work with business needs.05:30 Batch pipelines remain the most common approach even as real-time use cases grow.07:45 Many “real-time” requests are actually event-driven batch workflows.09:00 Airflow replaced many custom-built pipeline systems with built-in dependency management.11:00 Modern orchestration tools often build on Airflow concepts or differentiate from them.14:00 AI can assist with writing SQL and pipelines but still requires experienced engineers.15:30 Organizations are collecting increasingly granular data creating more engineering demand.19:00 The data stack has shifted rapidly from Hadoop-era systems to modern cloud platforms.Resources Mentioned:Benjamin Rogojanhttps://www.linkedin.com/in/benjaminrogojan/Seattle Data Guyhttps://www.linkedin.com/company/seattle-data-guy/Apache Airflowhttps://airflow.apache.orgAirflow Summit / Airflow Conferencehttps://airflowsummit.orgSnowflakehttps://www.snowflake.comHubSpot Data Sharing / APIshttps://developers.hubspot.comMLflowhttps://mlflow.orgThanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow
• Managing Data Quality and Governance With Airflow at Credit Karma with Ashir Alam 26.03.2026 22mnt

  Data quality is not optional when you manage credit data at scale.In this episode, Ashir Alam, Senior Data Engineer at Credit Karma, joins us to share how his team acts as the gatekeeper for credit data ingestion, how they standardize data quality with Airflow and DAG Factory and how they scale safely across thousands of DAGs. We explore how governance, PII protection and orchestration come together inside a modern data platform.Key Takeaways:00:00 Introduction.01:00 Overview of Credit Karma’s products and financial data ecosystem.02:00 The team acts as gatekeepers for ingesting data from TransUnion and Equifax.03:00 Why PII handling and controlled downstream access led to adopting Airflow.04:00 BigQuery as the warehouse and Airflow as the primary orchestrator.05:00 Why data quality and governance are critical in financial systems.07:00 Why Airflow was selected: ease of use and unified ETL plus data quality.09:00 Introduction to DAG Factory and YAML-based DAG generation.10:00 GitHub executor creates PR-driven DAG workflows with CI checks.12:00 BigQuery operators, structured checks and custom Slack and PagerDuty alerts.13:00 Failed checks stop ETL pipelines and trigger notifications.17:00 Scaling DAG Factory across thousands of DAGs and runtime vs compile-time concerns.19:00 Future improvements: better defaults, retries and GenAI workflows in Airflow.Resources Mentioned:Ashir Alamhttps://www.linkedin.com/in/ashir-alam/Credit Karmahttps://www.linkedin.com/company/intuit-credit-karma/Apache Airflowhttps://airflow.apache.org/DAG Factoryhttps://github.com/astronomer/dag-factoryBigQuery (Google Cloud)https://cloud.google.com/bigqueryGitHubhttps://github.com/Slackhttps://slack.com/PagerDutyhttps://www.pagerduty.com/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow
• Open Source Airflow Contributions and Performance Improvements at G-Research with Christos Bisias 19.03.2026 17mnt

  Modern Airflow isn’t just orchestration. It's a contribution.In this episode, we explore how open source investment drives real performance gains and deeper observability.We’re joined by Christos Bisias, Open Source Software Engineer, Apache Airflow at G-Research, to discuss how his team uses Airflow for large-scale data transformations, contributes upstream and improves scheduler throughput and OpenTelemetry support. From trace-level observability to CI-enforced metrics governance and a major scheduler optimization, this conversation spans strategy, engineering and community impact.Key Takeaways:00:00 Introduction.01:20 How G-Research applies machine learning and big data to predict financial market movements.02:15 Contributing to open source is a business decision.03:10 Maintaining a fork is costly.04:30 OpenTelemetry collects metrics, logs and traces to provide deep system visibility. 06:10 Custom spans help identify bottlenecks inside tasks and enable performance optimization. 08:05 OpenTelemetry integration works properly in Airflow 3.0 and above.10:00 A YAML-based metrics registry with CI enforcement ensures consistency between docs and exported metrics.12:10 Scheduler throughput improved significantly by applying concurrency limits earlier in the database query.  15:20 Future Task SDK changes may enable language-agnostic DAG authoring beyond Python.Resources Mentioned:Christos Bisiashttps://www.linkedin.com/in/xbis/G-Research https://www.linkedin.com/company/g-research/Apache Airflowhttps://airflow.apache.org/OpenTelemetryhttps://opentelemetry.io/Prometheushttps://prometheus.io/Grafanahttps://grafana.com/Jaegerhttps://www.jaegertracing.io/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow
• Automating Threat Intelligence Using Airflow with Karan Alang 12.03.2026 22mnt

  In this episode, Karan Alang, Principal Software Engineer at Versa Networks, joins the conversation to discuss how Airflow can be used to automate threat intelligence in modern cybersecurity environments. He explains the growing scale of cloud computing, the profitability of hacking and the shortage of SOC analysts. Karan also outlines a novel architecture that combines Airflow, XDR, graph databases and LLMs to orchestrate automated threat detection and response.Key Takeaways:00:00 Introduction.05:00 Organizations face massive log volumes and a shortage of SOC analysts.07:00 The solution integrates Airflow, XDR, Neo4j graph databases and LLMs into one architecture.08:00 MITRE ATT&CK provides a global framework for mapping tactics and techniques.11:00 Airflow acts as the orchestration backbone for ingestion graph transformation and LLM workflows.13:00 Graph databases provide a full relationship view of attackers’ systems and entities.14:00 LLMs automate mapping activity to MITRE ATT&CK and assign explainable risk scores.17:00 Traditional signature-based detection allows lateral movement and exfiltration before teams can react.18:00 End-to-end automation is essential to mitigating modern cybersecurity threats.20:00 Future opportunities include deeper LLM integration as first-class citizens within Airflow.Resources Mentioned:Karan Alanghttps://www.linkedin.com/in/karan-alang-4173437Versa Networks | LinkedInhttps://www.linkedin.com/company/versa-networksVersa Networks | Websitehttps://versa-networks.comGoogle Cloud Composer (Managed Airflow on GCP)https://cloud.google.com/composerMicrosoft Defender XDR https://www.microsoft.com/es-es/security/business/siem-and-xdr/microsoft-defender-xdrNeo4j (Graph Database)https://neo4j.comMITRE ATT&CK Frameworkhttps://attack.mitre.orgThanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning
• Using Plugins To Customize Airflow at Ponder Labs with Egor Tarasenko 05.03.2026 27mnt

  In this episode, we explore how teams scale Apache Airflow in complex environments and what it takes to make orchestration work across many stakeholders. We look at real-world challenges around visibility, ownership and predictability as data platforms grow.Egor Tarasenko, Data and AI Engineer at Ponder Labs, joins us to share how Ponder Labs customizes Airflow for education organizations using plugins, event-driven architectures and AI-powered tooling. He explains how his team supports large charter school networks and why structure, consistency and extensibility become critical at scale.Key Takeaways:00:00 Introduction.01:21 Ponder Labs helps education organizations bring data from many systems together so it becomes useful for teachers, school leaders and administrators.03:10 Airflow serves as the backbone for orchestrating ingestion, transformation and reverse ETL across client data platforms.05:43 Everything is triggered from Airflow to maintain dependency, visibility and a single operational picture.09:05 Managing hundreds of DAGs requires a focus on structure, visibility and consistency across teams.09:51 Treating DAGs like APIs helps teams scale without needing deep knowledge of upstream logic.12:00 Custom plugins like schedule insights help predict DAG run times across layered dependencies.15:00 AI-powered Airflow chat enables non-technical stakeholders to understand DAG ownership dependencies and cluster activity.22:06 Migrating plugins to Airflow 3 improves developer experience through cleaner APIs and faster extensibility.Resources Mentioned:Egor Tarasenkohttps://www.linkedin.com/in/egorseno/Apache Airflowhttps://airflow.apache.orgdbthttps://www.getdbt.comAstronomer Astro Platformhttps://www.astronomer.ioEgor Tarasenko on Substack https://egortarasenko.substack.comThanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow
• Scaling Airflow at Wix for Analytics and AI with Ethan Shalev 26.02.2026 18mnt

  Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while supporting more teams becomes a critical challenge.In this episode, we’re joined by Ethan Shalev, Data Engineer at Wix, to discuss how Wix operates Airflow at massive scale, migrates to Airflow 3 and uses AI to accelerate development.Key Takeaways:00:00 Introduction.02:13 Wix structures data engineering across multiple product-focused organizations.03:40 Migrating nearly 8,000 DAGs to Airflow 3 requires careful planning.04:31 Migration creates an opportunity to remove long-standing legacy Airflow code.05:32 Internal playbooks and Cursor rules standardize and speed up DAG migrations.07:39 Airflow 3 introduces backfills, DAG versioning and asset-aware scheduling.09:16 Deferrable operators reduce scheduler congestion in large Airflow environments.12:54 AI-generated code still requires review and strong testing practices.14:52 Moving to managed Airflow reduces operational burden on internal platform teams.15:57 Improving multi-tenancy and UI personalization remains a key Airflow need.Resources Mentioned:Ethan Shalevhttps://www.linkedin.com/in/eshalev/Wix | LinkedInhttps://www.linkedin.com/company/wix-com/Wix | Websitehttps://www.wix.com/Apache Airflowhttps://airflow.apache.org/Astronomerhttps://www.astronomer.io/Trinohttps://trino.io/Apache Iceberghttps://iceberg.apache.org/Cursorhttps://cursor.sh/Airflow Summithttps://airflowsummit.org/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow
• Using Airflow To Orchestrate Billions of Events at Addi with Carlos Daniel Puerto Niño 19.02.2026 24mnt

  Strong data orchestration is as much about culture and visibility as it is about technology. As data platforms scale, teams need systems that reduce cognitive load while increasing reliability and observability.In this episode, Carlos Daniel Puerto Niño, Senior Analytics Engineer and Data Analyst at Addi, joins us to share how Addi uses Airflow to support batch orchestration, manage organizational complexity and improve monitoring across its data platform.Key Takeaways:00:00 Introduction.01:25 Changes in company strategy increase data platform complexity over time.04:00 Centralized data teams help manage organizational and technical change.06:08 Scalable architectures support growing data volumes and use cases.09:10 Adopting orchestration tools introduces operational and maintenance challenges.14:43 Abstraction layers lower technical barriers for onboarding new team members.15:36 Modularity and visibility improve the reliability of data pipelines.18:14 Integrated monitoring supports faster incident response and resolution.22:19 Limited access to orchestration metadata constrains proactive analysis.Resources Mentioned:Carlos Daniel Puerto Niñohttps://www.linkedin.com/in/carlospuertoni%C3%B1o/Addi | LinkedInhttps://www.linkedin.com/company/addicol/Addi | Websitehttps://www.addi.comApache Airflowhttps://airflow.apache.org/Astronomerhttps://www.astronomer.io/Databrickshttps://www.databricks.com/dbthttps://www.getdbt.com/Grafanahttps://grafana.com/Slackhttps://slack.com/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow
• Building Event-Driven Data Pipelines With Airflow 3 at Astrafy with Andrea Bombino 12.02.2026 18mnt

  Real-time data expectations are reshaping how modern data teams think about orchestration and dependencies. As event-driven architectures become more common, teams need to rethink how pipelines react to data changes, rather than schedules.In this episode, Andrea Bombino, Co-Founder and Head of Analytics Engineering at Astrafy, joins us to discuss how event-driven scheduling in Airflow is evolving and how Astrafy applies it to deliver faster, more responsive data pipelines.Key Takeaways:00:00 Introduction.02:02 Astrafy’s role in guiding clients across the modern data stack.03:15 Strong DAG dependencies create challenges for time-based scheduling.04:48 Event-driven pipelines respond to increasing real-time data demands.05:30 Airflow 3 introduces native support for event-driven orchestration.06:27 Sensor-based workflows reveal scalability and efficiency limitations.11:32 Event-driven assets improve efficiency and pipeline elegance.14:45 Governance and cross-instance coordination emerge as ongoing challenges.Resources Mentioned:Andrea Bombinohttps://www.linkedin.com/in/andrea-bombino/Astrafy | LinkedInhttps://www.linkedin.com/company/astrafy/Astrafy | Websitehttps://www.astrafy.ioApache Airflowhttps://airflow.apache.org/Google Cloudhttps://cloud.google.com/Google Pub/Subhttps://cloud.google.com/pubsubGoogle BigQueryhttps://cloud.google.com/bigqueryThanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow
• Uphold’s Approach to Orchestrating Modern Data Workflows with Jaime Oliveira 05.02.2026 18mnt

  A strong data-driven mindset underpins how fintech teams scale analytics, infrastructure and decision-making across the business.In this episode, Jaime Oliveira, Lead Data Engineer at Uphold, joins us to discuss how Uphold structures its data organization and orchestration strategy. Jaime shares how the team uses Airflow and dbt to support analytics, reporting and data activation while evolving their approach as the stack grows.Key Takeaways:00:00 Introduction.01:23 A data-driven mindset supports product development and business decisions.02:55 Diverse ingestion pipelines enable scalable analytics.04:18 A single orchestration platform simplifies analytics workflows.05:17 Early experience with orchestration tools shapes engineering practices.08:16 Analytics orchestration works best when aligned with transformation workflows.09:25 Infrastructure choices involve tradeoffs in testing, visibility and overhead.16:39 More collaborative workflow tools could improve accessibility and autonomy.Resources Mentioned:Jaime Oliveirahttps://www.linkedin.com/in/jaime-oliveira-b075855a/Uphold | LinkedInhttps://www.linkedin.com/company/upholdinc/Uphold | Websitehttps://uphold.comApache Airflowhttps://airflow.apache.orgdbthttps://www.getdbt.comSnowflakehttps://www.snowflake.comKuberneteshttps://kubernetes.ioAstronomer Cosmoshttps://astronomer.github.io/astronomer-cosmosCosmos e-bookhttps://www.astronomer.io/ebooks/orchestrating-dbt-with-airflow-using-cosmos/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow