The Databricks Data Engineer

The Databricks Data Engineer

Jakub Lasak
Country USA
Genres Technology
Language EN
Episodes 13
Latest 16.11.2026

Helping 18k+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.

Episodes

  • Leave or stay? The 4 questions Databricks engineers keep asking (and the 10-minute test that answers them)
    Leave or stay? The 4 questions Databricks engineers keep asking (and the 10-minute test that answers them) 16.11.2026 11m
    It's late. A recruiter message sits unanswered, and the question you keep circling is short: I'm bored. Is bored a good enough reason to leave?It almost never travels alone. Four questions about leaving keep coming up from Databricks data engineers, and only the wording changes. Is boredom enough? Is it too late to stay for the migration? What is a good stack worth against twenty thousand more? Am I running toward something, or just away? They sound like questions about salary, stack and timing. They're one question, and from the inside its two answers feel identical.In this episode:- Why bore...
  • Medallion architecture vs. reality - when your Databricks bronze, silver and gold are just folders
    Medallion architecture vs. reality - when your Databricks bronze, silver and gold are just folders 09.11.2026 12m
    Three schemas. Bronze, silver, gold. The diagram got nods in the architecture review. Then two dashboards land in the same leadership meeting with two different active customer counts, and the head of finance asks which one is right.The trail ends at bronze, where rows that failed parsing were quietly dropped. The vendor keeps thirty days. Every label was there. What the medallion was invented to fix was not, and the pipeline had rebuilt the old mess across three schemas.In this episode:- What pipelines looked like before bronze, silver and gold, and the three pains the diagram was drawn to fi...
  • How the always-sure Databricks engineer caps their own promotion (and never sees it)
    How the always-sure Databricks engineer caps their own promotion (and never sees it) 02.11.2026 10m
    Mark's manager asks the room one question: if we move the biggest fact table onto Liquid Clustering, will it fix the slow dashboards? Mark answers first: yes, problem solved. Sarah says what she'd bet on, what she doesn't know yet, and when she'll know. Sarah claimed less. The decision goes to Sarah.Nobody tells Mark why. The instant-answer reflex that got him to mid-level is the same one that quietly keeps him there, and there's a name for what he's paying.In this episode:- Why a fast, confident answer is right early in a Databricks career, and when it stops paying- What the room needs from y...
  • Two Databricks jobs, one Delta table, same moment: 3 endings (and the 1 that needs a human)
    Two Databricks jobs, one Delta table, same moment: 3 endings (and the 1 that needs a human) 26.10.2026 11m
    Two in the morning. A nightly merge is folding yesterday's orders into a Delta table. A backfill someone kicked off before logging off is correcting March. Neither job knows the other exists, and nothing in Delta would tell them.A lot of design around this runs on superstition. Either "it's Delta, it's ACID, relax." Or "never let two jobs touch that table." Both are guesses, and the tidy middle version, same data collides, different data is fine, falls apart the first time the error shows up.In this episode:- What Delta actually does when the second job tries to commit to a table that moved un...
  • Unity Catalog Can't Manufacture Trust: Your Most-Read Databricks Table Might Be the Least Trusted
    Unity Catalog Can't Manufacture Trust: Your Most-Read Databricks Table Might Be the Least Trusted 19.10.2026 11m
    The rollout is done. Every production table migrated to Unity Catalog, every one owned by a team group, and the revenue table carries a certified check mark. The platform lead posts in Slack: fully governed. Monday morning, finance replies in the thread. The revenue on the finance dashboard and the revenue in the sales deck are about three percent apart. Which one's real?The catalog has something to show for every field: an owner, a check mark, a health indicator that says healthy, a beautiful lineage graph. None of them answers that question. Unity Catalog might be the best thing your platfor...
  • Your Databricks director: what they fear, why they fund the duller proposal, what gets yours through
    Your Databricks director: what they fear, why they fund the duller proposal, what gets yours through 12.10.2026 11m
    Two proposals sit on a head of data's desk, both about the nightly pipeline that feeds the finance dashboard. One swaps the full reload for incremental loads with Auto Loader: it finishes in a fraction of the time and it's cheaper every month after the first. The other keeps the full reload, puts it on a bigger job cluster and starts it an hour earlier. It costs more. Forever. She asks one question, "What happens if this breaks on quarter close?", and approves the second one.It's tempting to call that ignorance or politics. It's neither. The better proposal was arguing merit to someone who get...
  • Data skew is a checkout line - how to unjam Databricks jobs without bigger clusters
    Data skew is a checkout line - how to unjam Databricks jobs without bigger clusters 05.10.2026 12m
    Your nightly Databricks job ran long, so you open the Spark UI. Two hundred tasks. One hundred ninety-nine finished in about a minute. One is still running, and it's been running for forty. You double the cluster and rerun it. Same forty minutes. Nearly twice the bill.The cluster was never the problem. The rule that keeps your joins and group-bys correct is the same rule that lets one task carry the whole job, and no amount of extra workers moves that work off it. Sometimes the stuck task turns out to be worse than slow.In this episode:- Why more workers and more shuffle partitions both leave...
  • Can You Reach Senior Databricks Engineer Without Ever Leading an Architecture Review?
    Can You Reach Senior Databricks Engineer Without Ever Leading an Architecture Review? 28.09.2026 11m
    Sam is four years into a Databricks role, up for senior this cycle. Merged pull requests on the ingestion layer. The nightly job he rescued at three in the morning when the orders table stopped landing. Then his manager asks for one call where the answer wasn't obvious, he framed it, and he lived with the result. Sam starts three different sentences. None of them finish.The packet is full of work and empty of decisions. Sam thinks what he's missing is a chair at the architecture review table. It isn't.In this episode:- What the architecture review was ever supposed to prove, and why the room w...
  • Why Databricks serverless exists, and which of your jobs should stay on classic clusters
    Why Databricks serverless exists, and which of your jobs should stay on classic clusters 21.09.2026 12m
    Open the workspace. Start a cluster. Get coffee. Come back, and sometimes it's ready. If you've been doing this a few years you can feel that wait in your shoulders, and it's the least interesting part of the story.Before any of it, you had to answer a question you could not possibly answer: how big should this be? Start small and you pay in latency. Start big and you pay for machines that are barely breathing for most of the run. Wait, or waste, and there was never a third option.In this episode:- Why the cluster wait was only the receipt, and what classic Databricks compute never actually so...
  • How Databricks engineers lose the final round (and the 4 places it happens)
    How Databricks engineers lose the final round (and the 4 places it happens) 14.09.2026 10m
    You cleared the recruiter screen. You cleared the Databricks technical round, and cleared it well. Then the final round, then the email that says strong candidate, tough decision. You go back looking for the mistake and there isn't one. You were good. Somebody was just easier to picture.That sentence never makes it into the feedback. And it isn't a bad panel or a bias against you. It's a belief you carry into the room: that the final round is the last exam, the hardest one. If your last stage is still a coding or SQL round, that one is an exam, go and study. But most of these emails come out o...
  • The real reason your Databricks week never got simpler - it isn't the lakehouse
    The real reason your Databricks week never got simpler - it isn't the lakehouse 07.09.2026 11m
    Your org finished the migration. Two systems became one. The nightly copy nobody wanted to own is gone, and so is the quarterly meeting where finance and the data science team each showed up with a real revenue number from tables in different buildings. That pain was real, and it is genuinely dead. So why does your week feel heavier than the deck promised?Because most of what got sold as simplification was a relocation. The copy didn't die, it turned into an argument about which schema in which catalog is the real one. The sync job didn't die, it became a maintenance calendar that pages nobody...
  • The Databricks incident where the most senior engineer never touched the keyboard
    The Databricks incident where the most senior engineer never touched the keyboard 31.08.2026 11m
    Thursday, just after two in the afternoon. The first message doesn't come from an alert, it comes from a person in revenue operations: the biggest region jumped overnight and nobody sold anything new. Nobody answers.The engineer who owns the nightly Lakeflow job finds it in the run history. An overnight failure, a restart before anyone was awake, a window of orders written a second time under every executive dashboard in the company.So the fixing starts immediately. Head down, no message, because typing in the channel feels like stealing time from the fix. Which is how a second engineer joins...
  • 9 Databricks decisions that reveal your real seniority (and one that only looks senior)
    9 Databricks decisions that reveal your real seniority (and one that only looks senior) 24.08.2026 11m
    Three in the morning. The nightly job that loads the orders table died halfway. Half the tasks are green, and the rerun button is right there. What you do in the next ten seconds says more about your level than anything on your resume.Nobody places you by your title or your years. They place you by a pattern of routine calls made fast, in the dark, and everyone around you can read it except you.In this episode:- The ten-second question to ask before you rerun anything that writes- Why the fastest fix for a slow Databricks job is often the one that renews its cost every month- How the flaky job...
  • The Databricks job market in 2026: there are two of them, and only one answers
    The Databricks job market in 2026: there are two of them, and only one answers 17.08.2026 11m
    One engineer tells you hiring is back and recruiters are warm again. The next one is sixty applications deep and hearing nothing. Same month, same platform, sometimes the same city.That sounds like noise. It isn't. It's a split, and what sorts you onto one side of it has almost nothing to do with how good you are.In this episode:- Why forty applications in a week returns two dead screens, while a smaller and pickier search gets answered- The three things that changed in the last year and turned the generic Databricks data engineer posting into a stack of a hundred before lunch- Why fewer posti...
  • Databricks broadcast joins: when the memo beats the meeting (and when it kills a task)
    Databricks broadcast joins: when the memo beats the meeting (and when it kills a task) 10.08.2026 10m
    Two engineers on the same team join the same big orders table to the same small lookup table. Same cluster, same data, one line of code different. Sarah's finishes in the time it takes to get a coffee. Mike's dies, and the error isn't about the data at all. A task ran out of memory building a hash map.Broadcast joins get passed around as a tip instead of a model: small table equals fast, flip the switch when a join runs long. But the mechanism in almost every write-up is years out of date, and the size Spark checks is not the size that has to fit in memory.In this episode:- How to explain broa...
  • Build the platform or use the platform: the two Databricks data engineer tracks nobody names
    Build the platform or use the platform: the two Databricks data engineer tracks nobody names 04.08.2026 11m
    Two Databricks engineers sit at adjacent desks. Same title, same pay, same words in the ladder document. Watch a year go by, and they are not doing the same job.Both get rated strong. Both get told they're ready for more scope. Nobody says the useful thing: the evidence those two are stacking isn't interchangeable, and one of those years makes no sense on the other one's promotion packet.In this episode:- Why the data engineering ladder quietly forks, and why that fork is missing from every ladder document- The five-minute question that tells you which track your last year of Databricks work a...
  • Why Unity Catalog exists: the Databricks governance chaos it was built to end
    Why Unity Catalog exists: the Databricks governance chaos it was built to end 27.07.2026 10m
    You inherited Unity Catalog already switched on. You learned the catalogs, the schemas, the grant statements, and never once saw the problem all of it was built to solve. Then someone in a review asks why the company suffered through that migration, and the best you've got is one word: governance.Here's what that word hides. Before Unity Catalog, one company ran five separate Databricks workspaces, each walled off - its own tables, its own users, its own rules, and nothing connecting them. A new analyst couldn't find where a table lived without three Slack threads. An auditor's one question -...
  • How to get credit for Databricks platform work without becoming the on-call martyr
    How to get credit for Databricks platform work without becoming the on-call martyr 20.07.2026 11m
    Your pipeline hasn't failed in months, and nobody noticed. Then it breaks at 2am, you fix it in twenty minutes, and Slack fills with fire emojis and thank-yous. You're the backbone of the team. You've also been at the same level for three years, and nobody can quite explain why.It's not bad luck, and it's not a skill gap. The exact thing everyone praises you for, being the one who can always fix it, is the reason they can't afford to move you up.In this episode:- Why being the indispensable on-call hero quietly caps the level you can reach- How one engineer systematized a whole class of Databr...
  • Why Databricks Built Delta Lake When Parquet Was Already Good Enough
    Why Databricks Built Delta Lake When Parquet Was Already Good Enough 13.07.2026 9m
    A nightly job is halfway through writing a batch when the cluster dies. Nobody runs a rollback, because there's nothing to roll back to. The next morning a dashboard is quietly serving half-written garbage, and no one can tell the good files from the wreckage.That's not a bug in Parquet. Every one of those files is valid, beautifully compressed, doing its job perfectly. The problem lives one level up, in the one thing a folder of Parquet files has never had and a real table cannot live without.In this episode:- Why a folder of Parquet files was never actually a table, and what a crashed write...
  • The Databricks pipeline that ran green for a year and was wrong the whole time
    The Databricks pipeline that ran green for a year and was wrong the whole time 06.07.2026 8m
    A revenue-attribution pipeline runs overnight, bronze to silver to gold, and lands a clean set of numbers the whole company uses to decide where the money goes. Every morning: green. Every morning, everyone moves on with their day. For a year, nobody looked any closer.The job succeeded every single night, so everyone trusted the data. But "finished" and "correct" are two different words, and the space between them had been quietly swallowing a slice of revenue for eleven months, with no error, no warning, and not one alert.In this episode:- Why a Databricks job can run green every day and stil...

Popular in

The podcast also appears in the podcast charts of these countries.