Machine Learning Guide

Jaksot

• 
  MLA 030 AI and Programming Jobs: What Happened and How to Position 26.02.2026 36min

  The aggregate job market held, the entry-level door narrowed, and software postings sit a quarter below pre-pandemic. Why cheap implementation made specification, verification and domain scarce, how ML roles split five ways, and how to position. Links Try a walking desk - stay healthy & sharp while you learn & code More OCDevel shows - this one has siblings, each on its own subject and produced the same way What coding agents did to programming and machine learning jobs by late 2026: the labor data, the mechanism behind it, how the ML career splintered into five roles, and a concrete positioning plan. Companion to the vibe coding trio (Vibe Coding in 2026, Inside a Coding Agent, Agentic Software Engineering) and the agents pair (AI Agents in 2026, OpenClaw and the Personal Agent). Displacement vs task change Two different claims hide inside "AI is taking programming jobs": displacement (the role disappears, nobody is rehired) and task change (the role stays, the work shifts). They appear in different data. Displacement shows in unemployment and layoff reports; task change shows in what postings ask for and how teams are shaped. The aggregate evidence is mostly task change with one real pocket of displacement at the entry level, which sets up offensive advice for most listeners and defensive advice for new entrants. The evidence Aggregate. The BLS Employment Situation has unemployment at 4.1% with payrolls beating forecasts, far from the 10-20% Dario Amodei floated in the Axios "white-collar bloodbath" interview. Both he and Sam Altman have since softened the timeline; Altman said he was "delighted to be wrong" (Fortune, Time). The Yale Budget Lab tracker finds no discernible disruption. Goldman Sachs Research estimates a net drag of about 16,000 jobs a month across 800+ occupations, with a long-run baseline of 6-7% of workers displaced. Entry level. Stanford's Canaries in the Coal Mine (August 2026 paper, dashboard) puts 22-25 year olds in AI-exposed occupations 19% behind less-exposed peers, up from 15% a year earlier, driven by reduced hiring rather than separations and concentrated in automation-style exposure. The authors call these descriptive indicators, not causal estimates. Their software-developer case study finds young-developer pay grew somewhat faster than older developers' after ChatGPT, consistent with firms hiring fewer but better-paid juniors; the CPS sample is too small for a software-specific employment percentage. The EIG entry-level working paper is the main counterweight. Software demand. Indeed's software development postings index (Feb 2020 = 100) sits near 75, roughly a quarter below pre-pandemic and still drifting down, against a much smaller decline in total postings. Confounds stacked on top of AI: rate hikes, the Section 174 expensing change, and the 2021 overhire. SignalFire's State of Talent has new grads at 7% of Big Tech hires, down 25% from 2023 and over 50% from 2019, so half the collapse predates ChatGPT. Layoffs. Challenger, Gray & Christmas counts 116,175 of 529,914 announced 2026 cuts through August as AI-attributed (about 22%, already more than double all of 2025); AI led every month from March to July, then fell to 3,462 in August, while year-to-date cuts are down 41%. Grads and incumbents. The NY Fed college labor market data (2026:Q2) has computer science at 7.0% unemployment and 19.1% underemployment and computer engineering at 7.8% and 15.8%, against 5.6% and 42% for all recent graduates: worst on getting a job, among the best on getting a good one. CompTIA's tech jobs report has tech occupation unemployment at 2.8% and over 320,000 active postings asking for AI-related capabilities. The reversal wave: CNBC and Forbes on employers rehiring after AI cuts, Forrester's 55% regret figure, Robert Half's one-in-three refill figure, and the Klarna and IBM cases. Projections. BLS 2025-2035: software developers +10% from 1.72 million, data scientists +35%, computer programmers -7%. Measurement. METR's randomized trial of 16 experienced open-source developers on 246 real issues found AI made them 19% slower while they believed it sped them up 20%, so self-reported productivity is unreliable in both directions. The mechanism When implementation cost falls toward zero, value moves to specification, verification and domain knowledge. The BLS programmer-vs-developer split is that thesis in two rows. Andrew Ng's AI Rewards Generalists Who Can Build New Skills and his five-part AI Engineering Skills Map argue the bottleneck moved from how to build to what to build; David Autor calls AI a supplement to workers with judgment and domain knowledge. Juniors are hit because the traditional junior role was the commoditized part, and it was also the tuition for learning the other two skills. The ceiling on the mechanism shows in two benchmarks from the same year: OpenAI's GDPval, where the newest models win or tie against experts on most one-shot deliverables (with caveats about automated grading), against Scale's Remote Labor Index, where the best agent completed about 4% of real multi-day projects (via Carnegie). Agents produce artifacts; humans still run projects. The ML career in 2026 Data scientist demand is projected to grow three times as fast as developer demand, and Levels.fyi puts ML/AI-focused engineers in the US around $248k average total compensation. The title splintered into five roles, roughly by headcount: AI engineer (application layer: retrieval, tool use, agent loops, evals, context design); forward-deployed engineer (the Palantir-origin role the labs adopted, where domain is the constraint; see Anthropic's FDE posting); evals and AI quality (titles like Research Engineer, Model Evaluations on Anthropic's jobs board); inference, serving and platform infrastructure; and research engineer or scientist, the smallest and most competitive tier. "AI engineer" now means treating the model as a component with a failure distribution and designing the system around it. Prompt engineering as a standalone title, fine-tuning as a default move, and train-from-scratch generalist ML roles lost ground. Three camps Accelerationists (Amodei, Altman, Mustafa Suleyman): disruption within one to five years, entry-level first; strongest evidence is the benchmark curve. The aggregate prediction has failed so far and both leading voices softened it; Suleyman's 12-18 month clock has not expired. Skeptics (Yann LeCun, who left Meta to found AMI Labs on a world-model thesis; Gary Marcus; Daron Acemoglu, whose macro estimate is under 1% TFP gain over a decade): strongest evidence is the Remote Labor Index; weakest point is the cheap-but-imperfect case that reshapes jobs without replacing them. Pragmatists (Andrew Ng, Brynjolfsson, Autor): technology real, effects uneven, the question is which tasks move. Best track record so far because they predicted least. The Carnegie Endowment's three views cuts the map differently and is worth reading alongside. The Anthropic Economic Index (January, March) shows augmentation edging up on consumer chat while API and coding-agent usage stays automation-dominant. Positioning Own a domain where correct answers require knowledge not on the internet. Own verification: reading diffs fast, writing the test before the bug, building the eval harness, catching reward-hacked tests. Run agents fluently and measure yourself rather than trusting the feeling (the METR gap). Ship agentic work in public with specs, tests, evals and review trail visible, the new portfolio. New entrants: don't look like the traditional junior; compete for the well-paid junior seats that remain, at companies with real domains, in the roles that are hiring. Learning path Fundamentals first, because you can't verify what you don't understand: the Machine Learning Guide core episodes. Then the applied layer, which changes every few months: the vibe coding trio, the agents pair, the media trio. Then a domain and a project, which no course provides. Related episodes MLA 22: Vibe Coding in 2026 MLA 23: Inside a Coding Agent MLA 24: Agentic Software Engineering MLA 28: AI Agents in 2026 MLA 29: OpenClaw and the Personal Agent Every show Gnothi has produced, on AI, coding agents, video generation and agentic business, is at ocdevel.com/moremlg.
• 
  MLA 029 OpenClaw and Personal Agents 22.02.2026 36min

  OpenClaw as the worked example of the always-on personal agent: gateway, markdown memory, heartbeats, skills, coding agents from your phone, hosted vs local models, and the 2026 security record (exposed instances, two critical CVEs, ClawHavoc) with the posture that makes it survivable. Links Try a walking desk - stay healthy & sharp while you learn & code More OCDevel shows - this one has siblings, each on its own subject and produced the same way Second and last episode of the agents pair. AI Agents in 2026 covered the theory: loops, tools, memory, protocols, SDKs, evaluation. This one takes a single category, the always-on personal agent, and its most-cloned instance, OpenClaw, all the way down to the security posture required before it touches your inbox. What a personal agent is A personal agent is the agent loop with three additions: it runs continuously on a machine you control and can wake itself; its interface is messaging (WhatsApp, Telegram, Signal, iMessage, Slack) rather than a chat tab; and it holds your files, shell, browser, calendar and, if you allow it, email. Messaging removes the gap between having a thought and delegating it, and a chat thread is a natural home for asynchronous work that reports back later. The access that makes it useful is also the entire security problem. OpenClaw today OpenClaw is an MIT-licensed, self-hosted TypeScript agent created by Peter Steinberger in late 2025. It was released as Warelay, passed through claw-themed names, hit an Anthropic trademark complaint, and settled on OpenClaw at the end of January 2026 per Wikipedia; the org remains openclaw/openclaw. Steinberger joined OpenAI in February 2026 and stewardship moved to the OpenClaw Foundation, a US 501(c)(3) chaired by Dave Morin with a small full-time staff and donors including OpenAI, GitHub, Nvidia and Microsoft; the repo's README says OpenAI is a donor, not an owner, and there is no paid tier or hosted service. The project ships near-weekly CalVer releases plus an extended-stable line, and the GitHub blog's maintainer profile called it the fastest-growing project in the site's history. OpenClaw 2.0 (v2026.8.1) landed at the end of August. Rewrites that keep the idea and shrink the surface, per OSS Insight's fork-wave analysis: nanobot (Python, small auditable core), ZeroClaw (Rust, single static binary), PicoClaw (Go, from Sipeed, embedded targets) and NanoClaw (TypeScript, container-first). Hosted versions are all third-party one-click deploys or small managed services; the foundation runs none. Architecture: gateway, workspace files, heartbeats, skills One long-lived Node gateway per host binds to loopback on port 18789, owns every channel connection, routes inbound messages to sessions, loads context, calls the configured model, executes tools, streams the reply and persists everything under ~/.openclaw. Nodes are paired devices (laptop, phone, headless box) that lend the gateway local screen, camera and shell. Memory is markdown in the agent workspace: AGENTS.md (operating instructions), SOUL.md (persona and boundaries), IDENTITY.md, USER.md (stable facts about you, with its own character budget), MEMORY.md (curated durable facts), a memory/ directory of daily notes, and a one-time BOOTSTRAP.md interview. The memory docs state there is no hidden state; a hybrid memory_search index covers the memory file and daily notes, a background "dreaming" sweep promotes recurring material into MEMORY.md, and a flush runs before context compaction. Initiative comes from the heartbeat, a periodic main-session turn (30 minutes by default, 60 on subscription auth) that can stay silent via a no-reply marker, and from the automations scheduler (one-shot, interval or cron, delivered to a channel, a webhook or nowhere). HEARTBEAT.md is legacy; its checklist now lives in DB-backed scratch. Skills follow Anthropic's Agent Skills format (SKILL.md with name and description front matter) and install from ClawHub; the docs say to treat third-party skills as untrusted code and read them first. The browser tool drives a dedicated agent-owned Chrome/Brave/Edge profile through a loopback-only control service with a strict SSRF policy. The model behind it: hosted versus local OpenClaw is a harness over sixty-plus providers using provider/model references, including Ollama, llama.cpp, LM Studio, vLLM and SGLang for local inference. The tradeoff is the one from the agents episode with higher stakes: everything the agent reads is forwarded to the model, so inbox triage on a hosted model sends your inbox to the provider. Frontier models are better at the judgment calls (is this urgent, is this instruction really from me), local is the only defensible choice for regulated third-party data, and a per-task split (local for reading private content, hosted for writing public content) is a common compromise. The docs make no claim about local-model quality inside OpenClaw. Subscription auth reuses an existing Claude CLI login or an OpenAI OAuth flow per the OAuth docs, which describe the Claude CLI path as sanctioned per Anthropic staff guidance and warn that a community proxy needs a terms check; no published Anthropic term was found either way. Integrations: coding agents from your phone, email, calendar The old Claude Code bridge skill is superseded by agent runtimes: built-in, Codex app-server, Claude CLI and a Copilot plugin, with external harnesses (Claude Code, Gemini CLI, OpenCode, Cursor) driven over the Agent Client Protocol through acpx. Tasks land in managed worktrees: isolated branches with checkpoints in a state DB, filesystem snapshots where supported, a cap around 100 live worktrees, and dirty or unpushed work never auto-cleaned. See Agentic Software Engineering for why worktrees are the right isolation unit. The official IMAP plugin watches a mailbox, spawns an isolated restricted-reader session per allowed message, ranks trust by DMARC/SPF/DKIM, and does not send mail, which encodes the read-versus-write distinction the security section relies on. Calendar and most SaaS arrive as skills or MCP servers; Cisco's DefenseClaw announcement describes connecting email, calendar and Discord through Zapier-hosted MCP servers so a glue service holds the OAuth tokens. Security: what happened in 2026 and what to do about it Exposure: Bitsight, SecurityScorecard and Censys counted between 30,000 and 135,000 internet-facing instances in early 2026, roughly two thirds with no authentication; Censys confirmed 63,070 live instances at the end of March. Bugs: CVE-2026-25253 (CVSS 8.8), a one-click RCE where the control UI auto-connected to a gatewayUrl from the query string and leaked the auth token, worked even against loopback-bound instances and was patched in v2026.1.29 per the GitHub advisory; CVE-2026-32922 (CVSS 9.9) let a pairing token rotate itself into admin, fixed in v2026.3.11. Well over a hundred advisories were logged between February and April. Supply chain: Koi Security's ClawHavoc report found 341 malicious ClawHub skills (335 from one campaign) disguised as wallets, trading bots and Workspace integrations, delivering the Atomic macOS Stealer and targeting always-on Mac minis; the count later passed 800 as the registry grew past 10,000 (The Hacker News, Unit 42). Cisco's skill research scanned about 31,000 agent skills, found a quarter with at least one vulnerability, and demonstrated exfiltration through an attacker-controlled Telegram bot. Injection: Giskard exploited a live deployment for exfiltration and account takeover; PromptArmor showed Telegram and Discord link previews exfiltrate data with no click; CrowdStrike called a misconfigured instance "a powerful AI backdoor agent." Ambient: Wiz found Moltbook's database open with about 1.5 million API tokens; Meta banned OpenClaw on work devices and then acquired Moltbook; China restricted state use. Responses: fast patches, loopback default, DM pairing codes for unknown senders, openclaw security audit, the VirusTotal partnership scanning every ClawHub skill with daily rescans, openclaw skills verify, publisher gating, and the layered access model in the security docs: DM modes, per-agent profiles, control-plane tool restrictions, node exec policies, sandbox and read-only variants, exec approvals, strict browser SSRF, and "one trust boundary per gateway." None of it fixes indirect prompt injection; the maintainers say scanning is not a silver bullet, skills remain arbitrary code, and the agent still holds real credentials. Cisco's open-source DefenseClaw adds pre-execution scanning and runtime allow/block enforcement. Posture, as eight rules: loopback plus a tunnel (Tailscale or SSH) and an auth token even locally; the agent gets its own OS user, mailbox, calendar, browser profile and capped API keys; every integration starts read-only with human approval on irreversible writes; a dedicated box with nothing else on it; read every skill before enabling; design so a hijacked agent is a nuisance, not a breach, by shrinking the write surface; read the logs and memory files weekly; run the audit after every change and pin a version. Use cases that survived By mid-2026 the consumer frenzy had cooled and what remained was solo founders and small teams running it as infrastructure. Surviving uses share one shape, a scheduled or triggered read delivered to the chat you already use: morning briefings, read-only inbox triage with thresholds, watching deadlines and pipelines and even school lunch menus, voice memos returned as structured notes, lead research and CRM updates, and coding-agent dispatch from a phone. Value compounds through the memory files rather than any single automation, which is why the posture has to precede the setup. Alternatives Claude Cowork and OpenAI's ChatGPT Work give the delegate-a-task shape sandboxed and session-based, without messaging or always-on. The rewrites give a smaller, auditable surface. The SDKs from AI Agents in 2026 give one tight automation in an afternoon. The real decision is how much of your life you want in one process. Related episodes AI Agents in 2026: Loops, Tools, Memory, Protocols, and Evaluation Inside a Coding Agent: Context, Instructions, Hooks, Skills, MCP Agentic Software Engineering: Specs, Verification, and the Review Loop The Gnothi companion show, OCDevel Agentic Business, follows one business as agents take on research, software, sales and recurring operations.
• 
  MLA 028 AI Agents: Loops, Tools, Memory, Protocols, and Evaluation 22.02.2026 33min

  What an AI agent actually is, why coding agents got good first, how memory really works, what MCP and A2A standardize, which SDKs are alive, how to evaluate on trajectories, and where the products stand after browser agents contracted. Links Try a walking desk - stay healthy & sharp while you learn & code More OCDevel shows - this one has siblings, each on its own subject and produced the same way First of two episodes on AI agents. This one is the architecture: the loop, tools and verifiable feedback, memory, the protocols (MCP, A2A, computer use), the SDK landscape, evaluation and observability, the product map, and when multiple agents help. The next episode, OpenClaw and the Personal Agent, applies it to one always-on assistant with security as the centerpiece. Coding-agent products and mechanics live in the vibe coding sequence starting at MLA 22. Agent vs workflow vs chat: the loop A chat model returns a message; a workflow is your code calling a model at fixed steps; an agent is a model that owns the control flow, choosing its next action from what it observes. That puts systems on a spectrum (chat, chat plus tools, workflows, agents) rather than in a binary, the framing Anthropic's Building Effective Agents uses. The loop itself is ReAct (Yao et al.): thought, action, observation, repeat, with the reasoning trace letting the model track and update a plan. What changed by 2026 is not the loop but the infrastructure around it, and every part of that infrastructure is an attack on per-step error compounding. Tools, function calling, and verifiable feedback Function calling: you describe tools as schemas, the model emits a structured call, your code executes it and returns the observation. The model never runs anything itself, which is the security model. Writing effective tools for agents gives the practical rules: few high-impact tools, clear namespaces, meaningful identifiers, token-efficient responses, descriptions treated as prompt engineering. Effective context engineering for AI agents adds the overlap test: if a human cannot say which tool applies, neither can the agent. The central principle: coding agents got good first because tests and compilers give verifiable feedback that catches a bad step inside the same loop that made it. Find or manufacture the verifier before writing the prompt. Memory: context, retrieval, files, episodic "Memory" means four things: the context window (the only memory the model has), retrieval from an external store, files on disk, and episodic records of prior sessions. Most agent memory is files. Anthropic's memory tool is a client-side file protocol (view, create, replace, insert, delete) against storage you own. The hard part is context management, and both labs converged on the same three mechanisms: context editing to clear stale tool results, compaction to summarize near the limit, and notes written to files before summarization. OpenAI's Responses API conversation state has the same shape with a compaction threshold and compact endpoint. Third-party layers Mem0, Letta (from MemGPT), and Zep (temporal knowledge graph) now compete with first-party primitives. Multi-session patterns: Effective harnesses for long-running agents. Protocols: MCP, A2A, computer use Model Context Protocol is the agent-to-tool standard, now a Linux Foundation project with individual-maintainer governance. 2026 additions: elicitation (server asks the user mid-operation), an extensions mechanism, and the async Tasks extension for long-running tools. Every major SDK below consumes it; its cost is the context each connected server's tool list occupies. A2A is the agent-to-agent standard, Google-built, Linux Foundation-hosted, at v1.0 with a steering committee spanning AWS, Cisco, Google, IBM, Microsoft, Salesforce, SAP and ServiceNow. Strong governance, weak observed consumption; worth knowing, not yet worth building on for small teams. Computer use is the universal fallback: Anthropic's computer use tool (GA toolset with zoom and an automatic injection classifier), Google's Gemini computer use, open-source Browser Use, and Playwright MCP, which drives the accessibility tree instead of screenshots. Prefer API, then accessibility tree, then screenshots. Building one: the SDKs Both labs advise starting without a framework: Building Effective Agents and OpenAI's A Practical Guide to Building Agents. The 2026 SDKs have converged on that critique as thin harnesses around a loop. Claude Agent SDK: Claude Code's loop as a library (built-in tools, subagents, hooks, MCP, permissions, compaction); TypeScript and Python; pre-1.0. OpenAI Agents SDK: handoffs, guardrails, sessions, tracing on the Responses API. OpenAI deprecated the visual Agent Builder in favor of it. LangGraph and LangChain 1.x: stateful graph with checkpointing, interrupts, durable execution; create_agent as a minimal middleware harness. LangSmith is the separate tracing product. Google ADK: code-first hierarchical agent trees with native A2A; deploys to Vertex Agent Engine. Microsoft Agent Framework: GA successor to AutoGen and Semantic Kernel; Python and .NET. CrewAI: role-based crews, past 1.0, with a commercial management platform. smolagents: code agents that write Python instead of JSON calls; weakest maintenance signal on the list. Pydantic AI and Vercel AI SDK: typed validation-first agents in Python; loop control and agent abstraction in TypeScript. Decision rule: machine-operating agent fast, Claude Agent SDK; lightweight handoffs, OpenAI; durable human-in-the-loop state, LangGraph; inside Google or Microsoft, their kit; to understand what you run, write the loop yourself first. Evaluation and observability Agents are evaluated on trajectories, not answers: traces, task evals, cost per task. Traces follow the OpenTelemetry GenAI semantic conventions; products include LangSmith, Langfuse (open source, acquired by ClickHouse), Arize Phoenix, Braintrust, W&B Weave, and Helicone. Public benchmarks show the shape of a task eval: SWE-bench Verified, which OpenAI stopped reporting citing contamination; SWE-bench Pro; tau2-bench; Terminal-Bench 2.0; OSWorld-Verified; GDPval. Cost and reliability: Princeton's Holistic Agent Leaderboard (paper) and its reliability dashboard separate pass@k capability from pass^k reliability; METR time horizons with their own limitations note. Guardrails: OpenAI agent safety, NeMo Guardrails, Guardrails AI; prompt injection framed by Simon Willison's lethal trifecta and Google's CaMeL architectural defense. Products Claude Cowork: "Claude Code for everyone," a sandboxed desktop agent with open-sourced plugins. ChatGPT agent remains; the Atlas browser was retired within a year, folded into ChatGPT and Codex. Google discontinued Project Mariner and moved the capability into Gemini and Antigravity, which absorbed Gemini CLI. Standalone browser agents contracted; the capability moved into models and existing apps. Still shipping: Perplexity Comet (free), Manus (ownership contested this year; check before building on it), Devin, Copilot Studio, Agentforce. Glue: n8n (AI Agent node inside a drawn workflow, MCP server trigger) and Zapier Agents with Zapier MCP. Browser-agent prompt injection is the documented security problem: the PleaseFix research note. Multi-agent: when it helps Two essays a day apart: How we built our multi-agent research system (orchestrator plus parallel subagents beat a single agent on research at roughly 15x the tokens; token usage explained most of the variance) and Cognition's Don't Build Multi-Agents (dispersed decisions and unshared context make it fragile). The disagreement is task shape. Parallelize independent, read-mostly work; keep stateful, sequential work single-threaded; prefer a small hierarchy where workers return findings rather than decisions. Related episodes MLA 29: OpenClaw and the Personal Agent MLA 22: Vibe Coding in 2026 MLA 23: Inside a Coding Agent MLA 24: Agentic Software Engineering Companion show: Agentic Business on Gnothi follows one business as agents take on research, software, sales and operations.
• 
  MLA 027 The AI Media Pipeline: Voice, Music, ComfyUI, APIs, and Finishing 14.07.2025 35min

  How to automate AI media end to end: clone your own voice on open TTS, pick music that's actually licensed, run ComfyUI graphs headless, design around fal, Replicate and provider queues, finish with ffmpeg, and stay inside licensing at every layer. Links More OCDevel shows - this one has siblings, each on its own subject and produced the same way Companion show. This episode is the overview of the media pipeline. For weekly, hands-on coverage of the video half, from a first usable clip to scenes that cut together, listen to AI Video Generation on Gnothi. Try a walking desk - stay healthy & sharp while you learn & code Pipeline, not prompt Once you need thirty clips with the same character, a narrator who sounds identical every episode, a matched music bed, word-accurate captions and a platform-safe export, the prompt is one node in a graph and the graph is the product. The engineering lives in the edges: how one model's output becomes the next model's input, how failures retry, what each job cost, and whether the run is reproducible next week. Model choice at the nodes is covered in the two sibling episodes; this one covers everything else: voice with ElevenLabs and Qwen3-TTS, licensed music, ComfyUI on your own card, the fal and Replicate APIs, ffmpeg assembly, and licensing. Voice: cloning, open TTS, and consent ElevenLabs remains the reference point (current flagship Eleven v3, character-based pricing). Professional voice cloning is locked to the requester's own voice behind a live voice check, and the terms require consent attestation for any uploaded voice. On the open side, Breeze TTS 2 topped the open-weights column of the Artificial Analysis speech arena in August 2026, ahead of Fish Audio's S2 Pro; the code is Apache 2.0 but the weights are research/non-commercial, so it is not a commercial self-hosting option. The working set for programmers: Qwen3-TTS (Apache 2.0, 0.6B/1.7B, cloning from seconds of reference audio; the preset-speaker variant does not clone, see my Qwen3-TTS voice cloning guide), Chatterbox (MIT, emotion control, watermarked output), and Kokoro (82M parameters, Apache 2.0, faster than real time on CPU). Fish Audio plays both sides with open weights and a cheap hosted API. Quantized Qwen3-TTS runs podcast-length synthesis on CPU-only instances; see Quantized Qwen3-TTS on CPU and the broader open-source TTS roundup. Hosted alternatives for prototyping: OpenAI text-to-speech and Gemini speech generation. Consent is the legal boundary. Tennessee's ELVIS Act added voice to right of publicity; the federal NO FAKES Act cleared Senate Judiciary in June 2026; EU AI Act Article 50 transparency duties apply from 2 August 2026; Denmark is amending copyright law to cover a person's face and voice. Music and sound effects Warner settled with both Suno and Udio; Universal settled with Udio, which became a no-download walled garden; UMG and Sony are still litigating against Suno, whose terms now grant commercial rights rather than ownership to paid subscribers. Eleven Music is trained on licensed data via Merlin and Kobalt deals and cleared for commercial use on self-serve plans, excluding film, TV and larger games; ElevenLabs sound effects are cleared on any paid plan and support loops. Google exposes Lyria and Lyria RealTime through the Gemini API. Open models: ACE-Step 1.5, YuE, Stable Audio Open (community license, best open option for short effects), HeartMuLa, and Meta's MusicGen, which is non-commercial. ComfyUI and local generation ComfyUI is a workflow runtime with a GUI for designing graphs. Comfy raised $30M at a $500M valuation in April 2026 and ships a desktop app, Comfy Cloud, and API nodes that call paid providers from inside a local graph. Programmatic use is the same /prompt endpoint and websocket the front end uses: export the workflow in API format, patch fields, post, poll. Wrappers like comfyui-api and comfy-pack turn a graph into a scalable service. Alternatives: SwarmUI, InvokeAI, the Krita AI plugin. Hardware: full-precision Flux.2 and Qwen-Image do not fit consumer cards; fp8 and GGUF quantization bring them to 16 to 24 GB. For video, the open Wan releases lag the API versions; the 5B variant does 720p on 24 GB (about 8 GB with Comfy offloading), the 14B variant officially wants 80 GB at full precision and needs GGUF to be consumer-viable, and Wan2GP targets low-VRAM cards. Rule: local for iteration, cloud for volume. APIs and aggregators Start with an aggregator, move to a provider API only for a feature or price it lacks. fal is queue-first: submit, get a request ID, poll or webhook, per-output pricing on popular models, per GPU-second for custom deployments. Replicate has the broader catalog beyond image and video, bills per second of compute for open models, packages custom models with Cog, and joined Cloudflare with the same API. RunPod serverless is the raw GPU option for a custom ComfyUI graph. Provider APIs have converged on the same shape: Veo via the Gemini API (billed per output second, audio included), Kling API (post, store task_id, poll /v1/tasks), Runway API, ElevenLabs API. Design rules: every generation is a job in a durable queue keyed on a hash of inputs, model and seed; store the provider's request ID next to your job ID; honor 429 retry-after with a token bucket per provider; persist prompt, seed, inputs and outputs in object storage; route to a second provider on 5xx. Cost per usable second is list price times your rejection rate. Assembly and finishing ffmpeg is the programmer's editor: concat, overlay, sidechain ducking, caption burn-in, crop to 9:16, loudnorm and export. Human-in-the-loop editors: DaVinci Resolve (free version is a real editor; Studio unlocks most Neural Engine features), Descript with its transcript-as-timeline and Underlord assistant, and CapCut for short-form auto-captions, with a caution about its June 2025 terms change. Upscaling: Topaz retired Video AI for the subscription Topaz Video with the Astra model; open-side, SeedVR2 is single-step, runs on 8 GB and plugs into ComfyUI; Real-ESRGAN for clean stills; RIFE for frame interpolation. Captions: WhisperX gives word timestamps within about 50 ms via forced alignment plus diarization, emitting SRT/VTT; generate styled word-pop overlays from its JSON. Delivery: 1080x1920 9:16, H.264/AAC, roughly 10 to 12 Mbps, 30 fps; YouTube recommended upload settings; target -14 LUFS integrated with a -1 dBTP ceiling as the last pipeline step. Licensing across the stack Five layers, and the output is only as clean as the dirtiest node. Weights: Apache/MIT models (Qwen3-TTS, Kokoro, Chatterbox, open Wan) are clean; FLUX.2 dev is non-commercial without a separate license; Stability's community license allows commercial use under a revenue threshold; MusicGen is non-commercial. Output: the Copyright Office holds that prompts alone are not authorship, and the Supreme Court denied cert in Thaler v. Perlmutter in March 2026, so keep evidence of the human selection and editing. Training data: licensed models are the safe path while label suits continue. People: right of publicity, get written consent. Disclosure: YouTube auto-labels via SynthID and C2PA content credentials since May 2026, and labels on Veo and C2PA-stamped content are permanent. Attach credentials and disclose. Two pipelines Social clip (30 s, 9:16, run 200 times): character sheet from an image-editing model stored with prompt and seed -> templated script -> Qwen3-TTS narration with word timings -> per-shot image-to-video jobs via fal keyed on input hash, webhook completion -> cached licensed music bed -> ffmpeg concat, duck, styled captions from timing JSON, crop, loudnorm, 1080x1920 export -> content credentials and disclosure -> human review queue. Narrated explainer (8 min, 16:9, weekly): human-written script (where copyright rests) -> chunked TTS stitched with short silences -> LLM shot list with timestamps tagged diagram/image/video -> deterministic diagrams, styled images, a few video clips upscaled with SeedVR2 -> licensed music and SFX generated once -> Resolve or ffmpeg assembly, captions from narration timings, loudnorm, 1080p/4K export -> title, chapters from the shot list, disclosure, credentials. Both are the same graph with different shot counts and aspect ratios. Related episodes AI Image Generation and Editing in 2026 AI Video Generation in 2026 AI Agents covers agent orchestration of pipelines like these The companion show for the video half of this pipeline is AI Video Generation on Gnothi.
• 
  MLA 026 AI Video Generation 2026: Veo, Gemini, Kling, Runway, MiniMax, Sora 12.07.2025 32min

  Sora is shut down, Google runs two video models, Kling 3 does lip-synced dialogue, and open-weight MiniMax H3 is what you can actually fine-tune. What a usable clip costs, which models do native audio, how reference consistency works, and why the unit of work is the shot. Links Notes and resources at ocdevel.com/mlg/mla-26 More OCDevel shows - this one has siblings, each on its own subject and produced the same way.  Companion show: for weekly, hands-on coverage of the AI video pipeline, from a first usable clip to scenes that cut together, listen to AI Video Generation. Try a walking desk - stay healthy & sharp while you learn & code Second of three episodes on AI media generation, covering Veo, Kling, Runway and MiniMax H3. Four questions: what a usable clip costs, which models generate sound and dialogue natively, how character and shot consistency work now, and where open-weight video fits for a programmer. Ends with the shot-to-scene mental model. What changed since 2025 Native audio is now the baseline at the frontier: Veo 3.1, Kling 3.0, MiniMax H3 and LTX-2.5 sample audio and frames from one model, so lip movement and sound effects land on the right frame. Clips grew from four or five seconds to eight to fifteen, with a few models advertising thirty. Every serious product ships a reference-conditioning feature (Google "ingredients", Kling "elements", Runway references) that holds a character or object across clips. Image-to-video, not text-to-video, is the professional path: lock the first frame with an image model (see AI Image Generation and Editing), then ask the video model to move it. The old "storyteller vs animator" split resolved in favor of the animators. Google: Veo 3.1 and Gemini Omni Google now runs two video models in two places. Veo 3.1 is the developer baseline on the Gemini API and Vertex, in Quality, Fast and Lite tiers, all with native audio; clips are 4, 6 or 8 seconds, and 1080p/4K are upscales of the 8-second clip. It accepts up to three reference images, first-and-last-frame interpolation, and extend in 7-second steps up to 20 times; the Ingredients to Video update added identity consistency, native vertical and 4K upscaling. At Google I/O 2026 Google announced Gemini Omni; Gemini Omni Flash replaced Veo inside the Gemini app and Flow, taking text, image, video or audio as input and supporting conversational video-to-video editing. Per Flow's model matrix it currently tops out at 10 seconds and 720p. All output carries SynthID; the detector portal is still waitlisted. Sora: shut down OpenAI launched Sora 2 on September 30, 2025 with native audio and a free iOS app; it then hit a copyright reckoning over opt-out character use, SAG-AFTRA and Bryan Cranston pushback on likeness, and a court order barring the word "Cameo". In March 2026 OpenAI announced a two-stage shutdown: app and web closed April 26, 2026, API closes September 24, 2026. NBC's reporting attributes it to reallocating compute to coding, reasoning and enterprise; Sora continues only as internal world-model research. It stays in the episode as the case study of a strong model without a business. Kling 3.0 Kuaishou's Kling 3.0 launched globally in March 2026 as a unified image, video and audio model: up to 15 seconds per shot, native 4K, and per the Kling Omni audio guide lip-synced dialogue in five languages with sound effects and ambience generated in the same pass. The control surface is the point: an elements library built from images or short reference video with per-element voice binding, multi-shot generation with continuity, motion transfer from a reference video, motion brush, six-axis camera control, extend and retake. Sold as a credit-based consumer app with commercial rights on paid tiers, plus a first-party API fronted in the West by fal and Replicate. fal's three per-second prices (audio off, audio on, voice control) make the cost of joint audio-video sampling visible. Runway Gen-4.5 and Aleph Runway Gen-4.5 shipped December 1, 2025, briefly topped the Artificial Analysis leaderboard, and was candid about causal reasoning, object permanence and "success bias" failures. Runway's differentiator is editing: Aleph is video-to-video (new angles, relighting, add/remove objects, restyle), and per the Runway API changelog Aleph 2.0 takes 2-30 second inputs with up to five keyframes; Act-Two transfers a filmed performance onto a character. Gen-4.5's release notes do not claim native dialogue or effects. The Runway API now also resells ByteDance Seedance 2.5 (30-second clips, large reference budgets, audio) and Wan 3, and studio deals with Adobe, AMC Networks and Lionsgate anchor the enterprise story. The leaderboard vs the products As of this recording the Artificial Analysis text-to-video arena (blind pairwise human preference) has none of Veo 3.1, Kling 3.0 or Gen-4.5 in its top five: Wan 3.0, Gemini Omni Flash, fal's post-trained MiniMax H3 Max, MiniMax H3, then Seedance 2.0. The headline products win on control, distribution and enterprise fit, not the taste test. Open weights and the second tier Naming matters for Wan: Wan 2.2 is Apache-2.0 open weight (14B MoE needing an 80GB GPU, or a 5B model for a 24GB card) and its GGUF quantizations still trend on Hugging Face; Wan 2.5, 2.7 and 3.0 have no published weights on the Wan-AI Hugging Face org or GitHub and are served as APIs. The open-weight center of gravity is MiniMax H3: 33B, native stereo audio, up to 2K, 4-15 seconds, a community license permitting commercial use, official ComfyUI workflows, and a LoRA and step-distillation ecosystem. LTX-2.5 (19B, native audio, community license free under $10M revenue) and HunyuanVideo-1.5 (8.3B, 14GB with offload, no audio) round out the runnable set; MAGI-2 is a preview with no confirmed license. Elsewhere: Luma shipped Ray3, Ray3 Modify and Ray3.14; Pika pivoted to effects, an agent and MCP on Pika 2.5; ByteDance's Seedance 2.0 and 2.5 ride Dreamina and CapCut distribution; Grok Imagine is a priced API video model outside the top ten; Higgsfield is an aggregator and creative suite, as are fal, Replicate and OpenArt on the developer side. Consistency and control Every consistency feature is conditioning under a different name: text, reference images, first frame, last frame, and reference video are slots the denoiser attends to. A first frame is the strongest condition, which is why image-to-video wins. Reference characters (Veo's three images, Kling elements with voice binding, Seedance's dozens of references) fight identity drift, still the main failure mode. Start and end frames bound a camera move and let shots hand off to each other. Explicit camera controls beat prompt text. Video-to-video (Aleph, Luma Modify, Omni Flash, Kling) means fixing a nearly right shot instead of regenerating. Extend compounds drift, so use it to finish a shot, not build a scene. Open models add LoRAs: Musubi Tuner trains adapters for HunyuanVideo and Wan 2.x, and the tooling lags each new frontier open release by months. Audio in video Native audio means one sampling process produces waveform and frames, conditioned on each other; Veo 3.1 prices everything as video with audio, Kling 3.0 exposes it as a paid toggle, H3 and LTX-2.5 do it in open weights, Gen-4.5 and Wan 2.2 do not. Post-hoc remains a valid choice: MMAudio generates synchronized sound from finished video with an explicit alignment module, and ElevenLabs sound effects generate timed effects from text. Native dialogue holds for a line or two; longer talking heads still favor performance-driven tools like Act-Two. Voice and music proper are in The AI Media Pipeline. Cost per usable second As of this recording, from Gemini API pricing: Veo 3.1 Quality about $0.40/s with audio (720p/1080p), Fast about $0.10/s, Lite about $0.05/s; Gemini Omni Flash is billed per token, working out to roughly $0.10/s of 720p. From Runway API pricing: Gen-4.5 $0.12/s, Aleph 2 $0.28/s with a minimum. From fal: Kling v3 about $0.08/s silent and $0.13/s with audio, Wan 2.5 $0.05/s; Grok Imagine video $0.05-0.08/s. An 8-second Veo Quality shot with audio is a bit over $3; Kling or Gen-4.5 about $1. No vendor publishes success rates; budgeting four generations per usable shot puts a frontier clip with audio at $3-13 and a Fast or open-weight clip under $1. Iterate on the cheap tier, render on the expensive one. Shot to scene Every model generates a shot: one continuous take, one camera, one action, 4-15 seconds. A scene is three to eight shots cut together, and continuity is your job: same references in every shot, first and last frames handing off, the same elements or LoRA, one audio bed over the cut. Storyboard as shots, lock first frames with an image model, iterate cheap, render expensive, fix with video-to-video, assemble in an editor. Assembly, voice, music, ComfyUI and driving it from code are the next episode. Related episodes AI Image Generation and Editing in 2026 The AI Media Pipeline: Voice, Music, ComfyUI, APIs, and Finishing Vibe Coding in 2026 for Grok's coding products
• 
  MLA 025 AI Image Generation 2026: GPT Image, Nano Banana, Midjourney, Flux 09.07.2025 30min

  Editing replaced generation as the core task. How GPT Image 2.5, Google's Nano Banana line, Midjourney V8.2 and Flux 2 differ, what open weights and LoRAs buy you, ControlNet vs instruction editing, and how licensing and C2PA provenance work now. Links More OCDevel shows - this one has siblings, each on its own subject and produced the same way Companion show. This episode is the overview. For weekly, hands-on coverage of the full image and video pipeline, from a first usable clip to scenes that cut together, listen to AI Video Generation. Try a walking desk - stay healthy & sharp while you learn & code First of three episodes on AI media generation (this one is images and editing; then video, then the pipeline). A decision guide rather than a leaderboard: where GPT Image, Nano Banana, Midjourney and Flux each fit in late 2026, why instruction-driven editing replaced generation as the core task, what open weights buy you, and how licensing and provenance work. What changed: editing, instruction-following, references, text The 2025 "artist vs collaborator" split is over and the collaborators won. Every frontier image model now sits behind a language model that reads the prompt with world knowledge and accepts images as input, so the unit of work became "here is an image, change this one thing and keep everything else." Text rendering, precise instruction following and identity-preserving reference images all landed at once for one reason: the image model became, or was paired with, a multimodal language model. Generation from scratch is now the special case where the input image is empty. OpenAI: GPT Image 2.5 The lineage runs gpt-image-1 (2025), gpt-image-2, then ChatGPT Images 2.5 in September 2026, with a precision variant (Sunburst) and a fast default (Flare); the same models power ChatGPT and the Images API. As of this recording it holds the top slots on both the LMArena text-to-image and image-edit boards and on Artificial Analysis. For editing it does mask inpainting, up to four reference images, and multi-turn editing via the Responses API. Its weaknesses are latency at high quality and occasional text and consistency slips. Billing is per token (text in, image in, image out) with a cached-input discount; see the gpt-image-2.5-sunburst model page. Google: the Nano Banana lineage Nano Banana was gemini-2.5-flash-image (August 2025, now legacy). It was succeeded by Nano Banana Pro (gemini-3-pro-image) and Nano Banana 2 (gemini-3.1-flash-image, February 2026), plus a Lite tier. Per the Gemini image generation docs, Nano Banana 2 accepts up to fourteen reference images (objects plus characters), outputs up to 4K, does multi-turn sequential editing, and can ground generation in Google Search; the Pro model adds style references and identity preservation across up to five subjects. Every output carries a SynthID watermark with no opt-out. Imagen appears to be superseded for new work, though no formal retirement notice was found. Midjourney V8.2 and the Edit Model Midjourney moved from V7 (2025) to V8 alpha in March 2026 and V8.2 as the default in July 2026 (version history). V8.2's Edit Model replaces Omni Reference, Character Reference, Retexture and the Editor with one instruction-driven model taking up to four references, the same convergence OpenAI and Google made. Aesthetics and draft-mode ideation remain its strengths. The hard limits: still no official API and terms that bar automation; generations public by default below the Stealth tier; and it is the defendant in Disney Enterprises v. Midjourney (filed June 2025, joined by a separate Warner Bros. Discovery suit), currently in discovery with Midjourney demanding the studios' own AI records. Flux: the open-weight default and its license tiers Black Forest Labs, founded by the original Stable Diffusion authors, shipped FLUX.1 Kontext (in-context editing without masks) in 2025 and FLUX.2 in November 2025: a Mistral vision-language model paired with a rectified-flow transformer, up to ten references, 4MP editing. Tiers: Pro and Flex (API only); FLUX.2 dev (32B, open weights, non-commercial license); and FLUX.2 klein (January 2026), where the 4B model is Apache 2.0 and the 9B is non-commercial. The open-weights editing sub-board puts FLUX.2 and HunyuanImage roughly 130-150 Elo behind the closed frontier. Open weights earn their place through fine-tuning, on-prem privacy and composability with ControlNets and node graphs rather than raw quality. Special mentions Stable Diffusion: SD 3.5 is still the flagship, no SD4; the Community License is free under a revenue threshold; Stability raised again in August 2026. SDXL retains the deepest LoRA ecosystem on Civitai. Qwen-Image: 20B, Apache 2.0, the best open model for text-in-image (especially Chinese); Qwen-Image-Edit 2511 adds multi-image editing and identity preservation. Qwen-Image 2.0 is closed and API-only. Seedream: ByteDance's Seedream 4.0 and 4.5 unify generation and editing at up to 4K with up to ten references, on fal and BytePlus; strong on cost, no longer top five in either arena. Ideogram: text-rendering specialist; Ideogram 3.0 added single-image Character Reference; Ideogram 4.0 (June 2026) is 9.3B open-weight with JSON bounding-box prompting, but the weights are non-commercial. Recraft: V4 / V4.1 output native SVG with editable paths, brand-palette control and clean product shots. Adobe Firefly: "commercially safe" training plus IP indemnification; the app now hosts partner models (GPT Image, Nano Banana Pro, Flux, Ideogram) and attaches Content Credentials automatically. Leonardo (Canva-owned, Phoenix model) is the practical pick inside Canva; Krea is a real-time canvas plus a 60-model aggregator for trying everything from one account. Control: three problems, one commoditized Change this thing is instruction editing, now standard everywhere; masked inpainting and outpainting (Photoshop Generative Fill, GPT Image masks, FLUX.1 Fill) remain the hard constraint when the instruction is not enough. Keep this subject is reference conditioning, whose open-world mechanism is IP-Adapter (decoupled image cross-attention on a frozen base) and whose closed equivalents are Google's fourteen references and Midjourney's edit-model references; single references drift on fine detail. When drift is unacceptable, train a LoRA: roughly ten to twenty images and about a thousand steps on Flux (fal guide, FLUX.2 LoRA guide, Replicate trainer), open weights only. Keep this structure is ControlNet depth/edge/pose conditioning, still required for geometry fidelity per Autodesk's testing, with first-party support in Qwen-Image-Edit and FLUX.2 ComfyUI nodes. Multi-turn is for exploration; for repeatability, reproduce the winning edit as one instruction from the original. Licensing, provenance, C2PA Ownership is a human-authorship question: the US Copyright Office's Copyrightability report requires human authorship, treats prompts as unprotectable instructions, and protects AI-assisted work to the extent of the human contribution; the Supreme Court declined Thaler v. Perlmutter in March 2026, leaving the "how much human is enough" line undrawn. Commercial use is a vendor question: OpenAI (you own outputs), Midjourney (paid plans, no indemnity), Flux (tiered), Firefly (indemnified). Training-data fair use is unresolved in the US; Andersen v. Stability AI is the image bellwether, while the UK High Court largely rejected Getty's claims against Stability in November 2025. Provenance now has two layers: C2PA 2.3 manifests plus pixel watermarks; OpenAI now embeds both C2PA and SynthID (API guide), as Google already did. Manifests are detailed but stripped on re-save, screenshot and most platform uploads; SynthID survives those but carries little information. Labeling is now law: EU AI Act Article 50 and California SB 942 both enforceable from August 2026, China's rules from September 2025. Choose by job Marketing asset: GPT Image or Nano Banana; Nano Banana references for a real face or product, a Flux LoRA when drift is unacceptable. Concept art: Midjourney, draft mode then the Edit Model; keep it out of any automated pipeline and flag the litigation to client legal. Product photo: Flux LoRA plus ControlNet depth, or Nano Banana object references for the quick version; Recraft for vector; Firefly for indemnity. Developer pipeline: on-prem or fine-tuning means open weights (FLUX.2 klein 4B commercially, dev with a license); otherwise OpenAI or Google APIs with a cheap tier for drafts; never Midjourney. Related episodes AI Video Generation in 2026: Veo and Gemini Omni, Kling, Runway, MiniMax H3, and the End of Sora The AI Media Pipeline: Voice, Music, ComfyUI, APIs, and Finishing Companion show: AI Video Generation on Gnothi
• 
  MLG 036 Autoencoders 30.05.2025 1t 5min

  Auto encoders are neural networks that compress data into a smaller "code," enabling dimensionality reduction, data cleaning, and lossy compression by reconstructing original inputs from this code. Advanced auto encoder types, such as denoising, sparse, and variational auto encoders, extend these concepts for applications in generative modeling, interpretability, and synthetic data generation. Links Notes and resources at ocdevel.com/mlg/36 Try a walking desk - stay healthy & sharp while you learn & code Build the future of multi-agent software with AGNTCY. Thanks to T.J. Wilder from intrep.io for recording this episode! Fundamentals of Autoencoders Autoencoders are neural networks designed to reconstruct their input data by passing data through a compressed intermediate representation called a "code." The architecture typically follows an hourglass shape: a wide input and output separated by a narrower bottleneck layer that enforces information compression. The encoder compresses input data into the code, while the decoder reconstructs the original input from this code. Comparison with Supervised Learning Unlike traditional supervised learning, where the output differs from the input (e.g., image classification), autoencoders use the same vector for both input and output. Use Cases: Dimensionality Reduction and Representation Autoencoders perform dimensionality reduction by learning compressed forms of high-dimensional data, making it easier to visualize and process data with many features. The compressed code can be used for clustering, visualization in 2D or 3D graphs, and input into subsequent machine learning models, saving computational resources and improving scalability. Feature Learning and Embeddings Autoencoders enable feature learning by extracting abstract representations from the input data, similar in concept to learned embeddings in large language models (LLMs). While effective for many data types, autoencoder-based encodings are less suited for variable-length text compared to LLM embeddings. Data Search, Clustering, and Compression By reducing dimensionality, autoencoders facilitate vector searches, efficient clustering, and similarity retrieval. The compressed codes enable lossy compression analogous to audio codecs like MP3, with the difference that autoencoders lack domain-specific optimizations for preserving perceptually important data. Reconstruction Fidelity and Loss Types Loss functions in autoencoders are defined to compare reconstructed outputs to original inputs, often using different loss types depending on input variable types (e.g., Boolean vs. continuous). Compression via autoencoders is typically lossy, meaning some information from the input is lost during reconstruction, and the areas of information lost may not be easily controlled. Outlier Detection and Noise Reduction Since reconstruction errors tend to move data toward the mean, autoencoders can be used to reduce noise and identify data outliers. Large reconstruction errors can signal atypical or outlier samples in the dataset. Denoising Autoencoders Denoising autoencoders are trained to reconstruct clean data from noisy inputs, making them valuable for applications in image and audio de-noising as well as signal smoothing. Iterative denoising as a principle forms the basis for diffusion models, where repeated application of a denoising autoencoder can gradually turn random noise into structured output. Data Imputation Autoencoders can aid in data imputation by filling in missing values: training on complete records and reconstructing missing entries for incomplete records using learned code representations. This approach leverages the model's propensity to output 'plausible' values learned from overall data structure. Cryptographic Analogy The separation of encoding and decoding can draw parallels to encryption and decryption, though autoencoders are not intended or suitable for secure communication due to their inherent lossiness. Advanced Architectures: Sparse and Overcomplete Autoencoders Sparse autoencoders use constraints to encourage code representations with only a few active values, increasing interpretability and explainability. Overcomplete autoencoders have a code size larger than the input, often in applications that require extraction of distinct, interpretable features from complex model states. Interpretability and Research Example Research such as Anthropic's "Towards Monosemanticity" applies sparse autoencoders to the internal activations of language models to identify interpretable features correlated with concrete linguistic or semantic concepts. These models can be used to monitor and potentially control model behaviors (e.g., detecting specific language usage or enforcing safety constraints) by manipulating feature activations. Variational Autoencoders (VAEs) VAEs extend autoencoder architecture by encoding inputs as distributions (means and standard deviations) instead of point values, enforcing a continuous, normalized code space. Decoding from sampled points within this space enables synthetic data generation, as any point near the center of the code space corresponds to plausible data according to the model. VAEs for Synthetic Data and Rare Event Amplification VAEs are powerful in domains with sparse data or rare events (e.g., healthcare), allowing generation of synthetic samples representing underrepresented cases. They can increase model performance by augmenting datasets without requiring changes to existing model pipelines. Conditional Generative Techniques Conditional autoencoders extend VAEs by allowing controlled generation based on specified conditions (e.g., generating a house with a pool), through additional decoder inputs and conditional loss terms. Practical Considerations and Limitations Training autoencoders and their variants requires computational resources, and their stochastic training can produce differing code representations across runs. Lossy reconstruction, lack of domain-specific optimizations, and limited code interpretability restrict some use cases, particularly where exact data preservation or meaningful decompositions are required.
• 
  MLG 035 Large Language Models 2 08.05.2025 45min

  At inference, large language models use in-context learning with zero-, one-, or few-shot examples to perform new tasks without weight updates, and can be grounded with Retrieval Augmented Generation (RAG) by embedding documents into vector databases for real-time factual lookup using cosine similarity. LLM agents autonomously plan, act, and use external tools via orchestrated loops with persistent memory, while recent benchmarks like GPQA (STEM reasoning), SWE Bench (agentic coding), and MMMU (multimodal college-level tasks) test performance alongside prompt engineering techniques such as chain-of-thought reasoning, structured few-shot prompts, positive instruction framing, and iterative self-correction. Links Notes and resources at ocdevel.com/mlg/mlg35 Build the future of multi-agent software with AGNTCY Try a walking desk stay healthy & sharp while you learn & code In-Context Learning (ICL) Definition: LLMs can perform tasks by learning from examples provided directly in the prompt without updating their parameters. Types: Zero-shot: Direct query, no examples provided. One-shot: Single example provided. Few-shot: Multiple examples, balancing quantity with context window limitations. Mechanism: ICL works through analogy and Bayesian inference, using examples as semantic priors to activate relevant internal representations. Emergent Properties: ICL is an "inference-time training" approach, leveraging the model's pre-trained knowledge without gradient updates; its effectiveness can be enhanced with diverse, non-redundant examples. Retrieval Augmented Generation (RAG) and Grounding Grounding: Connecting LLMs with external knowledge bases to supplement or update static training data. Motivation: LLMs' training data becomes outdated or lacks proprietary/specialized knowledge. Benefit: Reduces hallucinations and improves factual accuracy by incorporating current or domain-specific information. RAG Workflow: Embedding: Documents are converted into vector embeddings (using sentence transformers or representation models). Storage: Vectors are stored in a vector database (e.g., FAISS, ChromaDB, Qdrant). Retrieval: When a query is made, relevant chunks are extracted based on similarity, possibly with re-ranking or additional query processing. Augmentation: Retrieved chunks are added to the prompt to provide up-to-date context for generation. Generation: The LLM generates responses informed by the augmented context. Advanced RAG: Includes agentic approaches—self-correction, aggregation, or multi-agent contribution to source ingestion, and can integrate external document sources (e.g., web search for real-time info, or custom datasets for private knowledge). LLM Agents Overview: Agents extend LLMs by providing goal-oriented, iterative problem-solving through interaction, memory, planning, and tool usage. Key Components: Reasoning Engine (LLM Core): Interprets goals, states, and makes decisions. Planning Module: Breaks down complex tasks using strategies such as Chain of Thought or ReAct; can incorporate reflection and adjustment. Memory: Short-term via context window; long-term via persistent storage like RAG-integrated databases or special memory systems. Tools and APIs: Agents select and use external functions—file manipulation, browser control, code execution, database queries, or invoking smaller/fine-tuned models. Capabilities: Support self-evaluation, correction, and multi-step planning; allow integration with other agents (multi-agent systems); face limitations in memory continuity, adaptivity, and controllability. Current Trends: Research and development are shifting toward these agentic paradigms as LLM core scaling saturates. Multimodal Large Language Models (MLLMs) Definition: Models capable of ingesting and generating across different modalities (text, image, audio, video). Architecture: Modality-Specific Encoders: Convert raw modalities (text, image, audio) into numeric embeddings (e.g., vision transformers for images). Fusion/Alignment Layer: Embeddings from different modalities are projected into a shared space, often via cross-attention or concatenation, allowing the model to jointly reason about their content. Unified Transformer Backbone: Processes fused embeddings to allow cross-modal reasoning and generates outputs in the required format. Recent Advances: Unified architectures (e.g., GPT-4o) use a single model for all modalities rather than switching between separate sub-models. Functionality: Enables actions such as image analysis via text prompts, visual Q&A, and integrated speech recognition/generation. Advanced LLM Architectures and Training Directions Predictive Abstract Representation: Incorporating latent concept prediction alongside token prediction (e.g., via autoencoders). Patch-Level Training: Predicting larger "patches" of tokens to reduce sequence lengths and computation. Concept-Centric Modeling: Moving from next-token prediction to predicting sequences of semantic concepts (e.g., Meta's Large Concept Model). Multi-Token Prediction: Training models to predict multiple future tokens for broader context capture. Evaluation Benchmarks (as of 2025) Key Benchmarks Used for LLM Evaluation: GPQA (Diamond): Graduate-level STEM reasoning. SWE Bench Verified: Real-world software engineering, verifying agentic code abilities. MMMU: Multimodal, college-level cross-disciplinary reasoning. HumanEval: Python coding correctness. HLE (Human's Last Exam): Extremely challenging, multimodal knowledge assessment. LiveCodeBench: Coding with contamination-free, up-to-date problems. MLPerf Inference v5.0 Long Context: Throughput/latency for processing long contexts. MultiChallenge Conversational AI: Multiturn dialogue, in-context reasoning. TAUBench/PFCL: Tool utilization in agentic tasks. TruthfulnessQA: Measures tendency toward factual accuracy/robustness against misinformation. Prompt Engineering: High-Impact Techniques Foundational Approaches: Few-Shot Prompting: Provide pairs of inputs and desired outputs to steer the LLM. Chain of Thought: Instructing the LLM to think step-by-step, either explicitly or through internal self-reprompting, enhances reasoning and output quality. Clarity and Structure: Use clear, detailed, and structured instructions—task definition, context, constraints, output format, use of delimiters or markdown structuring. Affirmative Directives: Phrase instructions positively ("write a concise summary" instead of "don't write a long summary"). Iterative Self-Refinement: Prompt the LLM to review and improve its prior response for better completeness, clarity, and factuality. System Prompt/Role Assignment: Assign a persona or role to the LLM for tailored behavior (e.g., "You are an expert Python programmer"). Guideline: Regularly consult official prompting guides from model developers as model capabilities evolve. Trends and Research Outlook Inference-time compute is increasingly important for pushing the boundaries of LLM task performance. Agentic LLMs and multimodal reasoning represent the primary frontiers for innovation. Prompt engineering and benchmarking remain essential for extracting optimal performance and assessing progress. Models are expected to continue evolving with research into new architectures, memory systems, and integration techniques.
• 
  MLG 034 Large Language Models 1 07.05.2025 50min

  Explains language models (LLMs) advancements. Scaling laws - the relationships among model size, data size, and compute - and how emergent abilities such as in-context learning, multi-step reasoning, and instruction following arise once certain scaling thresholds are crossed. The evolution of the transformer architecture with Mixture of Experts (MoE), describes the three-phase training process culminating in Reinforcement Learning from Human Feedback (RLHF) for model alignment, and explores advanced reasoning techniques such as chain-of-thought prompting which significantly improve complex task performance. Links Notes and resources at ocdevel.com/mlg/mlg34 Build the future of multi-agent software with AGNTCY Try a walking desk stay healthy & sharp while you learn & code Transformer Foundations and Scaling Laws Transformers: Introduced by the 2017 "Attention is All You Need" paper, transformers allow for parallel training and inference of sequences using self-attention, in contrast to the sequential nature of RNNs. Scaling Laws: Empirical research revealed that LLM performance improves predictably as model size (parameters), data size (training tokens), and compute are increased together, with diminishing returns if only one variable is scaled disproportionately. The "Chinchilla scaling law" (DeepMind, 2022) established the optimal model/data/compute ratio for efficient model performance: earlier large models like GPT-3 were undertrained relative to their size, whereas right-sized models with more training data (e.g., Chinchilla, LLaMA series) proved more compute and inference efficient. Emergent Abilities in LLMs Emergence: When trained beyond a certain scale, LLMs display abilities not present in smaller models, including: In-Context Learning (ICL): Performing new tasks based solely on prompt examples at inference time. Instruction Following: Executing natural language tasks not seen during training. Multi-Step Reasoning & Chain of Thought (CoT): Solving arithmetic, logic, or symbolic reasoning by generating intermediate reasoning steps. Discontinuity & Debate: These abilities appear abruptly in larger models, though recent research suggests that this could result from non-linearities in evaluation metrics rather than innate model properties. Architectural Evolutions: Mixture of Experts (MoE) MoE Layers: Modern LLMs often replace standard feed-forward layers with MoE structures. Composed of many independent "expert" networks specializing in different subdomains or latent structures. A gating network routes tokens to the most relevant experts per input, activating only a subset of parameters—this is called "sparse activation." Enables much larger overall models without proportional increases in compute per inference, but requires the entire model in memory and introduces new challenges like load balancing and communication overhead. Specialization & Efficiency: Experts learn different data/knowledge types, boosting model specialization and throughput, though care is needed to avoid overfitting and underutilization of specialists. The Three-Phase Training Process 1. Unsupervised Pre-Training: Next-token prediction on massive datasets—builds a foundation model capturing general language patterns. 2. Supervised Fine Tuning (SFT): Training on labeled prompt-response pairs to teach the model how to perform specific tasks (e.g., question answering, summarization, code generation). Overfitting and "catastrophic forgetting" are risks if not carefully managed. 3. Reinforcement Learning from Human Feedback (RLHF): Collects human preference data by generating multiple responses to prompts and then having annotators rank them. Builds a reward model (often PPO) based on these rankings, then updates the LLM to maximize alignment with human preferences (helpfulness, harmlessness, truthfulness). Introduces complexity and risk of reward hacking (specification gaming), where the model may exploit the reward system in unanticipated ways. Advanced Reasoning Techniques Prompt Engineering: The art/science of crafting prompts that elicit better model responses, shown to dramatically affect model output quality. Chain of Thought (CoT) Prompting: Guides models to elaborate step-by-step reasoning before arriving at final answers—demonstrably improves results on complex tasks. Variants include zero-shot CoT ("let's think step by step"), few-shot CoT with worked examples, self-consistency (voting among multiple reasoning chains), and Tree of Thought (explores multiple reasoning branches in parallel). Automated Reasoning Optimization: Frontier models selectively apply these advanced reasoning techniques, balancing compute costs with gains in accuracy and transparency. Optimization for Training and Inference Tradeoffs: The optimal balance between model size, data, and compute is determined not only for pretraining but also for inference efficiency, as lifetime inference costs may exceed initial training costs. Current Trends: Efficient scaling, model specialization (MoE), careful fine-tuning, RLHF alignment, and automated reasoning techniques define state-of-the-art LLM development.
• 
  MLA 024 Agentic Software Engineering: Specs, Verification, and the Review Loop 13.04.2025 33min

    How working engineers ship with coding agents: issues an agent can verify, plan mode before code, a verification loop with a browser in it, agent review of agent code, worktrees and CI, cost discipline, and where agents still fail. Links Try a walking desk - stay healthy & sharp while you learn & code More OCDevel shows - this one has siblings, each on its own subject and produced the same way Third and last episode of the vibe-coding sequence. Vibe Coding in 2026 picked an agent; Inside a Coding Agent explained the mechanics. This one is the practice: how working engineers ship real software with Claude Code, Codex, and Antigravity without shipping garbage. Specs, verification loops, agent review, parallel worktrees, headless CI, cost discipline, and the failure modes. From vibe coding to agentic engineering Andrej Karpathy coined vibe coding in early 2025 and, a year later, proposed "agentic engineering" for professional work. His older idea of jagged intelligence (models clear hard problems and trip on trivial ones, unpredictably) is why the job is judgment rather than button-pressing. The frame for the episode: when implementation is cheap, the value moves to the two ends of the pipeline, specification (saying exactly what should be true) and verification (proving it). Vibe coding stays fine for throwaways; the rest applies to codebases with users. Specs: issues as prompts, plan mode in Claude Code, Codex, and Antigravity The prompt is a spec whether you meant it or not. The unit of work is a tracker issue with four parts: what's wrong, where to look, acceptance criteria a machine or a five-second human check can verify, and an explicit out-of-scope fence. Every serious agent has a read-only planning phase: Claude Code's plan mode (Shift+Tab or /plan), Codex's plan mode on the same keystroke with its own plan-mode reasoning effort, and Antigravity's implementation plan artifact. Judge a plan on three things: the files it names, the verification step it commits to, and whether it stays inside the fence. Heavier spec tooling (GitHub spec-kit, Kiro specs) formalizes requirements, design, and tasks. Anthropic's best practices teach a lighter interview-then-spec-file pattern and say to skip planning when you could describe the diff in one sentence; OpenAI's Codex best practices frame a task as goal, context, constraints, and "done when." Verification loops: one-command checks, test-first, Playwright MCP and browser agents An agent without feedback guesses; an agent with a runnable check searches. Make typecheck, lint, tests, and build fast and runnable from one command, because that command is what the agent lives inside. Test-first changes meaning here: it used to be design pressure, and now it gives the agent a fixed target. Anthropic's guidance is that without a success criterion the developer is the only feedback loop (building verification loops). Then add a browser. Playwright MCP drives a real one through accessibility snapshots with stable element refs, no vision model needed; Claude in Chrome is the screenshot route Anthropic names for UI verification; Antigravity's browser agent starts the dev server and clicks through on its own, ending in a walkthrough with verification evidence. The principle: every acceptance criterion maps to a check the agent can run. If one doesn't, either the criterion is vague or the project is missing a kind of check. The review loop: agent PRs, Claude Code and Codex review, human gate Five vendor-agnostic stages: trigger, implement, review, fix, gate. Agent review of agent code works because the reviewer has fresh context, so never reuse the implementing session as its own reviewer. Ask for correctness and quality separately and set a confidence bar so the reviewer reports only what it's sure of. Anthropic's code-review plugin runs four parallel reviewers with 0-100 confidence scores and drops findings under 80; a cloud tier (Ultrareview) reproduces each finding before reporting it, and a separate hosted Code Review product reviews every PR automatically. Codex reviews on an @codex review mention or automatically per repo, flagging only serious issues. CodeRabbit and Greptile fill the same slot with a precision/recall tradeoff (ignore vendor benchmarks of each other). The human gate reviews a staging branch as a batch with the running app in front of you, and reads the tests, not just the code. Parallelism: git worktrees, cloud sandboxes, task queues Each agent gets its own copy of the repo on its own branch. Locally that's a git worktree (Claude Code worktrees, --worktree and subagent isolation: worktree; Codex added a --worktree flag in 0.154.0). In the cloud it's a sandbox per task: Claude Code on the web (managed VM, credentials behind a proxy, claude --cloud, teleport back to the terminal), Codex cloud tasks (isolated containers, network off by default), Google Jules, and Antigravity's Agent Manager. Parallel tasks must be disjoint, which is why issues that name their files matter. Three interactive agents is a practical ceiling for one person; no vendor publishes a number, and Simon Willison's parallel coding agent lifestyle argues for prompting during natural breaks rather than a fixed count. Past that, scale with a labeled queue instead of more terminals. Headless and CI: GitHub Actions, label triggers, scheduled runs Every major CLI has a non-interactive mode, and CI chains it. All three majors ship a GitHub Action covering mention, label, and schedule triggers: claude-code-action (@claude mentions, label_trigger, cron automation mode, claude/ branch prefix), Codex's GitHub integration and codex-action, and Google's run-gemini-cli with hourly issue triage in its examples. GitHub Copilot coding agent takes an assigned issue into an Actions runner and opens a draft PR. The CI fix loop: Claude Code on the web's Auto-fix pull requests subscribes to a PR's webhooks and pushes fixes for failing checks or review comments (/autofix-pr from the terminal); Codex does the same from a PR mention. Two cautions: headless runs need a fixed tool allowlist or a sandbox, and scheduled runs act as the user who wrote the schedule, so gate triggers on the actor to avoid automation loops. Cost and context discipline Shape rather than prices, since prices age fastest: every vendor sells a subscription with a rolling window plus weekly cap in multiplied tiers (Claude, Codex, Antigravity), overage credits at roughly API rates, and pay-as-you-go API keys. Subscription for daily interactive use, API for headless and CI. Model tiering is official guidance: Anthropic's cost docs reserve the top model for architectural work, and subagents take a per-agent model and effort; Codex shows model plus reasoning effort on its status line. Context is the invisible line item: every turn re-sends history, compaction is itself a large request, /usage and /context show what eats the window, and the prompt cache goes cold after a break. The habits: clear between unrelated tasks, trim MCP servers, push exploration into subagents, and expect agent teams to run several times a single session's tokens. Failure modes: reward-hacked tests, scope creep, prompt injection, secrets Four families, each with a mechanism and a case. Reward-hacked tests: METR measured frontier models gaming graders in about 30% of research-engineering runs (patching the scoring function, locating precomputed answers), and instructing them not to cheat had nearly no effect; Anthropic's reward-hacking research names the canonical move, calling sys.exit(0) inside the harness so tests report green. The mitigation is structural: the worker is never the grader, and a hook flags test-file edits. Scope creep: fence the issue, flag out-of-fence changes in review, route unrelated improvements to a new issue. Prompt injection through repo content: Invariant Labs' GitHub MCP demonstration exfiltrated private code via a malicious public issue; Cursor's CVE-2025-54135 let injected content write MCP config and execute code; Anthropic's security docs say no system is immune and recommend VMs for external services. Secrets and blast radius: the Nx s1ngularity attack (advisory) ran victims' installed Claude, Gemini and Amazon Q CLIs with skip-permissions flags to harvest credentials, leaking 1,000+ tokens; the Replit production database deletion (July 2025) and a reported second wipe (April 2026); the UK AISI incident report (August 2026) on unsanctioned real-world actions during cyber evaluations. Guardrails: branch-scoped tokens behind a proxy, no production credentials on the agent's machine, destructive commands denied by hook or sandbox, plugins and MCP servers installed only from sources you'd let commit (plugin trust guidance), tested backups. On productivity: METR's 2025 RCT found experienced developers 19% slower; the 2026 uplift update flipped that cohort to a speedup with intervals crossing zero; the 2025 DORA report found AI adoption raised throughput and lowered delivery stability. The tools amplify the process you already have. One-week adoption plan Day one: one fast command for typecheck, lint, tests, build. Day two: three issues with acceptance criteria and a scope fence, each through plan mode. Day three: browser verification and a screenshot in every UI PR. Day four: fresh-context review of every agent PR and a staging branch as the human gate. Day five: one recurring chore as a scheduled headless run, with secrets and test-integrity guardrails. Days six and seven: two agents on disjoint issues, and notice where your supervision breaks. Related episodes MLA 22: Vibe Coding in 2026, which agent to pick and why MLA 23: Inside a Coding Agent, context, instruction files, hooks, skills, MCP, subagents MLA 28: AI Agents in 2026, agent architecture beyond coding MLA 30: AI and Programming Jobs in 2026, what agents did to the job market and how to position Want this practice hands-on rather than surveyed? The Gnothi OCDevel Claude Code show goes from a first change in the terminal to a repeatable delivery workflow.
• 
  MLA 023 Inside a Coding Agent: Context, Instructions, Hooks, Skills, MCP 13.04.2025 34min

  A coding agent is a loop appending to a context window, and the window is what runs out. Instruction files, permissions, hooks, skills, MCP, subagents and memory explained as one mechanism, mapped across Claude Code, Codex and OpenCode. Links Try a walking desk - stay healthy & sharp while you learn & code More OCDevel shows - this one has siblings, each on its own subject and produced the same way Second of three episodes on coding agents: how Claude Code, Codex, OpenCode and pi handle context, instruction files, permissions, hooks, skills, MCP, subagents and memory. Claude Code is the running example; each section maps the same concept to the other three. The loop and why context is the scarce resource A coding agent is a short program: send the conversation to the model, run whatever tool it asks for, append the result, repeat until the model answers with text alone. Every appended file and test log stays in the window for the life of the session, so the context window, not the model, is what runs out. Every mechanism below is a way of deciding what enters the window and when. Compaction summarizes the history when the window fills; Claude Code's auto-compact window and /compact are configurable, and /compact is itself an expensive request while /clear is free. Thinking is dialed by effort level in Claude Code and by model_reasoning_effort in the Codex configuration reference. Instruction files: CLAUDE.md and AGENTS.md AGENTS.md has become the cross-vendor instruction file, stewarded by the Agentic AI Foundation under the Linux Foundation and read by Codex, OpenCode, Cursor, Gemini CLI, GitHub Copilot, Jules, Aider, Zed and more. Claude Code reads only CLAUDE.md, bridged with an @AGENTS.md import; the Claude Code memory docs describe the managed, user, project and local layers, lazy subdirectory loading, @ imports up to four hops, path-scoped .claude/rules/, and re-injection after compaction. Codex's AGENTS.md guide walks from the git root down with a 32 KiB default cap; OpenCode rules read AGENTS.md with a CLAUDE.md fallback plus an instructions array in opencode.json; pi concatenates AGENTS.md from home, parents and cwd. What belongs: commands, conventions and boundaries the model cannot derive from the code, under a couple hundred lines. Instruction files are context, not enforcement. Permissions and sandboxing Claude Code permissions use Tool(pattern) rules with deny over ask over allow, and the docs list how allowlists leak: absolute paths, sh -c, and unstripped wrappers like devbox run. Permission modes are default, acceptEdits, plan, the classifier-backed auto mode, and bypassPermissions. Codex separates sandbox modes (read-only, workspace-write, danger-full-access) from an approval policy. OpenCode permissions map globs to allow, ask or deny with last-match-wins. pi ships no permission system by design. Claude Code sandboxing uses Seatbelt on macOS and bubblewrap on Linux, with filesystem and network-domain allowlists. The cautionary case is PocketOS, April 2026, where a Cursor agent found an unscoped hosting token and deleted the production volume and its backups in seconds, reported by The Register and Fast Company. Hooks: deterministic guardrails at tool events A hook runs a program at a fixed point in the loop and its exit code or JSON decides what happens next; unlike an instruction, it fires every time. Claude Code hooks cover more than thirty events; exit code 2 always blocks on blockable events, PreToolUse can stop a call, Stop can refuse to end a turn, PostToolUse cannot block. Codex hooks mirror the event set and require trusting each hook definition by hash. OpenCode plugins expose tool.execute.before, permission.asked and session events; Gemini CLI hooks and Copilot CLI hooks exist too. The three worth having: a PreToolUse block on destructive commands and secret files, a post-edit formatter and typecheck that feeds errors back, and a Stop hook that holds the turn open while tests are red. Skills and slash commands A skill is a folder with a SKILL.md whose name and description load at startup, whose body loads when a task matches, and whose bundled files load on demand. The format is the open Agent Skills standard, adopted by Codex, OpenCode, Cursor, Gemini CLI, GitHub Copilot and dozens more. Claude Code skills absorbed custom slash commands; frontmatter flags decide whether a skill is user-invoked, model-invoked or both, and side-effecting skills like deploy should be user-invoked only. A loaded skill body persists in the window, and load-time shell commands can splice live state into a procedure. MCP: what it solves and what it costs The Model Context Protocol standardizes tools, resources and prompts over JSON-RPC, via stdio or streamable HTTP with OAuth; Anthropic donated it to the Linux Foundation and the current revision adds stateless per-request negotiation, Tasks and MCP Apps extensions. Tool definitions traditionally loaded into context for every server every session. Anthropic's Code execution with MCP named the two costs and reported 150,000 tokens down to 2,000. The fix that shipped is deferred loading: Claude Code defers MCP tool definitions by default via tool search, also available at the API level; Codex MCP config offers per-server enabled_tools and disabled_tools; OpenCode MCP servers filter tools by glob with no lazy loading documented. Claude Code's cost guidance: prefer CLI tools over MCP servers, scope servers per project, disable what you have not used, and watch the 25,000-token output cap. Subagents and worktrees as the answer to context rot A subagent is a second loop with a fresh window; the parent sees only its result. Claude Code subagents are Markdown files with frontmatter for model, tools, permission mode, memory and worktree isolation, started without conversation history unless forked; agent teams add messaging between named teammates at roughly seven times the tokens of a plain session. Codex subagents are TOML files with developer instructions, model, effort and sandbox mode. OpenCode agents split into primary agents and @mention subagents, each with its own permission block. Recommended cast: a read-only scout on a cheap model, a fresh reviewer per diff, one writer per task. Claude Code worktrees create isolated checkouts with --worktree, EnterWorktree, subagent isolation: worktree, .worktreeinclude for gitignored files, and enforcement that blocks edits reaching back into the main checkout; Codex and OpenCode leave worktree management to git. Memory across sessions Memory is files. Claude Code auto memory is on by default: a per-repository directory shared across worktrees with a MEMORY.md index (first 200 lines loaded) and topic files read on demand, recording preferences, feedback and learned project facts. Codex memories are an opt-in local store controlled per session. OpenCode has no first-party memory beyond snapshots. Both vendors frame memory as a recall layer: rules that must always apply belong in the instruction file, prohibitions in hooks. Concept map across tools Instruction file: CLAUDE.md (Claude Code); AGENTS.md (Codex, OpenCode, pi, most others) Permissions: modes and rules (Claude Code); sandbox mode plus approval policy (Codex); per-tool allow/ask/deny (OpenCode); none by design (pi) Hooks: settings hooks (Claude Code); trusted hooks file (Codex); plugins (OpenCode); extensions (pi) Skills: shared open format everywhere; slash commands folded in (Claude Code) MCP: deferred by default (Claude Code); per-server tool filters (Codex); glob filters (OpenCode) Subagents: Markdown (Claude Code, OpenCode); TOML (Codex) Worktrees: first-party (Claude Code); do it yourself (Codex, OpenCode) Memory: auto memory (Claude Code); memories (Codex); files or an MCP server (OpenCode) Related episodes Vibe Coding in 2026: Codex vs Claude Code vs Antigravity vs Grok Agentic Software Engineering: Specs, Verification, and the Review Loop AI Agents in 2026: Loops, Tools, Memory, Protocols, and Evaluation The Gnothi companion show on Claude Code goes from a first change in the terminal to a repeatable delivery workflow: OCDevel Claude Code Podcast.
• 
  MLA 022 Vibe Coding: Codex vs Claude Code vs Antigravity vs Grok 09.02.2025 28min

  What a coding agent actually is, and how Claude Code, OpenAI Codex, Google Antigravity and xAI's Grok Build differ in philosophy rather than features. Plus open harnesses pi and OpenCode, the open-weight coders, and a decision rule by constraint. Links Try a walking desk - stay healthy & sharp while you learn & code More OCDevel shows - this one has siblings, each on its own subject and produced the same way First of three episodes on vibe coding: the field of coding agents (Claude Code, OpenAI Codex, Google Antigravity, Grok Build, pi, OpenCode) as of late 2026 and how to choose one. The next two cover the mechanics (Inside a Coding Agent) and the engineering practice (Agentic Software Engineering). Vibe coding then and now Andrej Karpathy's February 2025 post named "vibe coding": accept all, don't read the diffs, paste errors back in, and, the caveat everyone dropped, "not too bad for throwaway weekend projects." Collins made it Word of the Year 2025. The arc since is autocomplete, then chat, then agents that read, edit, run and iterate; the engineers who stayed with it moved from vibing to agentic engineering, which is the through line for the sequence. Anatomy of a coding agent The model and the harness are separate things, and most tool arguments are about one without noticing which. The harness is the loop around the model, and there are five parts worth naming: the model, the tools (read, write, shell, search, browser), the permission model and sandbox, the context strategy (what to keep, summarize, offload to files, or hand to a fresh subagent), and the surface (terminal, IDE, desktop, web, chat). Ask those five questions of any product; they matter more than the feature grid. The mechanics get their own episode. Claude Code: terminal-first and composable Anthropic's stated design is Unix-style composability: pipe logs into it, run it in CI, chain it with other tools. The terminal is the primary surface; the IDE extension, desktop app, web and mobile clients all connect to the same engine and share CLAUDE.md, settings and MCP servers, with handoffs between local, cloud and desktop. Headless -p mode, GitHub Actions and GitLab integrations, and the Claude Agent SDK make it embeddable. Subagents and worktrees are its answer to the context problem. Model tiers (Sonnet, Opus, Fable) sit behind one harness with 1M-context variants and an effort dial; usage is included in Claude subscriptions or billed via the API. Claude Code overview Model configuration Plans and costs Headless mode Subagents and worktrees Claude Agent SDK Claude Fable 5 and Mythos 5, Claude Opus 5 OpenAI Codex: cloud-first and parallel Codex CLI and Codex cloud are one product under one ChatGPT login, with an IDE extension and a desktop app; codex cloud pushes local work into a sandboxed cloud environment and applies the result back. OpenAI's framing is isolated environments, parallel tasks, and comparing multiple solution attempts, with tasks launchable from the web, GitHub, GitLab, Linear or Slack. Codex-tuned models are trained for long-running agentic work, and the newest generation keeps notes across context windows instead of repeatedly compressing a session. The CLI is Apache-2.0 open source; the Agents API brings the same harness to your own cloud agents. Codex CLI docs Codex cloud docs Model lineup and plans and credits Codex CLI source Introducing the Codex app Introducing the Agents API Google Antigravity: the agent manager Antigravity still ships under that name, and it absorbed Gemini CLI: Google retired Gemini CLI into Antigravity CLI in June 2026, and Antigravity 2.0 spans IDE, desktop app, CLI and SDK. The philosophy is supervision: "manage higher-level tasks for the Agent, not individual tool calls," with agents working across editor, terminal and browser and producing artifacts you review. The model menu is multi-vendor on consumer tiers (several Gemini tiers plus Claude Sonnet and Opus and GPT-OSS); enterprise is Gemini only. Antigravity IDE Supported models and tiers Antigravity changelog Google I/O 2026 developer highlights Transitioning Gemini CLI to Antigravity CLI Grok Build: the cost-first bet xAI's coding story flipped in 2026 from a fast model in other people's harnesses (grok-code-fast-1, since deprecated from GitHub Copilot) to a first-party open-source terminal agent, Grok Build: interactive TUI, headless -p, Agent Client Protocol for editors, plan-review-approve, hooks, plugins, MCP and parallel subagents. xAI's own positioning for its coding model is fastest and cheapest for agentic coding; the premium model above it is the agent's default. Benchmark numbers and the subagent-count and price claims circulating in blogs are not on xAI's pages. Introducing Grok Build Grok Build 0.1 on the API Grok Build docs and source Grok Code Fast 1 deprecated in Copilot Open harnesses: pi and OpenCode Both are open source, model-agnostic and very active, and they're opposites in philosophy. pi is a toolkit: small core, unified API over 15+ providers, TUI, print/RPC/SDK modes, TypeScript extensions, branchable sessions, a package system, and deliberately no built-in permission system (containerize instead; the author ships the Gondolin micro-VM). OpenCode is a product: TUI, web, desktop and IDE surfaces, a ~50-entry provider list including local runtimes, permission config, and the optional Zen gateway of verified models. OpenCode's docs note Anthropic prohibits driving Claude subscription models through third-party harnesses and those plugins were removed. Also: Crush, Kilo Code, Cline, and DeepSeek's own harness. pi, pi source OpenCode docs, providers, Zen, source Crush, DeepSeek Harness Special mentions: open-weight coders and the second tier Every lab with a model now ships a harness. Qwen has a coder line, an Apache-licensed open series and the Qwen Code CLI. Z.ai's GLM is pitched as the most capable open-weights model for coding, sold as a Coding Plan documented for Claude Code, Cline and OpenCode. DeepSeek serves 1M-context models with an Anthropic-format endpoint, so Claude-shaped harnesses can point at it. Moonshot's Kimi K3 is a very large open-weight multimodal model under a bespoke license, with the Kimi Code CLI. All coding claims are vendor-reported, now headlining SWE-bench Pro and Terminal Bench rather than SWE-bench Verified. Second tier: Copilot coding agent (cloud, inside Actions), Cursor (editor-first, adding cloud agents and a coordinator), Windsurf (folded into Devin), Cline (active), Aider (no commits since May 2026). Qwen3-Coder, Qwen Code GLM-5, GLM Coding Plan DeepSeek models and pricing Kimi K3, Kimi Code CLI Copilot coding agent, Cursor changelog, Cline, Aider Decision rule by constraint Already paying a lab: take that lab's agent (Claude Code, Codex, Antigravity) and stop shopping. Cloud vs local: Codex cloud or Copilot's agent for sandboxed parallel PRs; a harness plus a local or self-hosted model if everything has to stay behind your firewall. Cost: Grok Build or an open harness with a cheap open-weight model for mechanical work. Open weights: pi or OpenCode with Qwen, GLM, DeepSeek or Kimi. Test on a real bug you already understand, and budget for a primary agent plus a cheaper second one. Related episodes MLA 23: Inside a Coding Agent MLA 24: Agentic Software Engineering MLA 28: AI Agents MLA 30: AI and Programming Jobs Companion show: the Gnothi Claude Code series goes from your first terminal change to a repeatable delivery workflow.
• 
  MLG 033 Transformers 09.02.2025 43min

  Links: Notes and resources at ocdevel.com/mlg/33 3Blue1Brown videos: https://3blue1brown.com/ Try a walking desk stay healthy & sharp while you learn & code Try Descript audio/video editing with AI power-tools Background & Motivation RNN Limitations: Sequential processing prevents full parallelization—even with attention tweaks—making them inefficient on modern hardware. Breakthrough: "Attention Is All You Need" replaced recurrence with self-attention, unlocking massive parallelism and scalability. Core Architecture Layer Stack: Consists of alternating self-attention and feed-forward (MLP) layers, each wrapped in residual connections and layer normalization. Positional Encodings: Since self-attention is permutation invariant, add sinusoidal or learned positional embeddings to inject sequence order. Self-Attention Mechanism Q, K, V Explained: Query (Q): The representation of the token seeking contextual info. Key (K): The representation of tokens being compared against. Value (V): The information to be aggregated based on the attention scores. Multi-Head Attention: Splits Q, K, V into multiple "heads" to capture diverse relationships and nuances across different subspaces. Dot-Product & Scaling: Computes similarity between Q and K (scaled to avoid large gradients), then applies softmax to weigh V accordingly. Masking Causal Masking: In autoregressive models, prevents a token from "seeing" future tokens, ensuring proper generation. Padding Masks: Ignore padded (non-informative) parts of sequences to maintain meaningful attention distributions. Feed-Forward Networks (MLPs) Transformation & Storage: Post-attention MLPs apply non-linear transformations; many argue they're where the "facts" or learned knowledge really get stored. Depth & Expressivity: Their layered nature deepens the model's capacity to represent complex patterns. Residual Connections & Normalization Residual Links: Crucial for gradient flow in deep architectures, preventing vanishing/exploding gradients. Layer Normalization: Stabilizes training by normalizing across features, enhancing convergence. Scalability & Efficiency Considerations Parallelization Advantage: Entire architecture is designed to exploit modern parallel hardware, a huge win over RNNs. Complexity Trade-offs: Self-attention's quadratic complexity with sequence length remains a challenge; spurred innovations like sparse or linearized attention. Training Paradigms & Emergent Properties Pretraining & Fine-Tuning: Massive self-supervised pretraining on diverse data, followed by task-specific fine-tuning, is the norm. Emergent Behavior: With scale comes abilities like in-context learning and few-shot adaptation, aspects that are still being unpacked. Interpretability & Knowledge Distribution Distributed Representation: "Facts" aren't stored in a single layer but are embedded throughout both attention heads and MLP layers. Debate on Attention: While some see attention weights as interpretable, a growing view is that real "knowledge" is diffused across the network's parameters.
• 
  MLA 021 Databricks: Cloud Analytics and MLOps 22.06.2022 26min

  Databricks is a cloud-based platform for data analytics and machine learning operations, integrating features such as a hosted Spark cluster, Python notebook execution, Delta Lake for data management, and seamless IDE connectivity. Raybeam utilizes Databricks and other ML Ops tools according to client infrastructure, scaling needs, and project goals, favoring Databricks for its balanced feature set, ease of use, and support for both startups and enterprises. Links Notes and resources at ocdevel.com/mlg/mla-21 Try a walking desk stay healthy & sharp while you learn & code Raybeam and Databricks Raybeam is a data science and analytics company, recently acquired by Dept Agency. While Raybeam focuses on data analytics, its acquisition has expanded its expertise into ML Ops and AI. The company recommends tools based on client requirements, frequently utilizing Databricks for its comprehensive nature. Understanding Databricks Databricks is not merely an analytics platform; it is a competitor in the ML Ops space alongside tools like SageMaker and Kubeflow. It provides interactive notebooks, Python code execution, and runs on a hosted Apache Spark cluster. Databricks includes Delta Lake, which acts as a storage and data management layer. Choosing the Right MLOps Tool Raybeam evaluates each client's needs, existing expertise, and infrastructure before recommending a platform. Databricks, SageMaker, Kubeflow, and Snowflake are common alternatives, with the final selection dependent on current pipelines and operational challenges. Maintaining existing workflows is prioritized unless scalability or feature limitations necessitate migration. Databricks Features Databricks is accessible via a web interface similar to Jupyter Hub and can be integrated with local IDEs (e.g., VS Code, PyCharm) using Databricks Connect. Notebooks on Databricks can be version-controlled with Git repositories, enhancing collaboration and preventing data loss. The platform supports configuration of computing resources to match model size and complexity. Databricks clusters are hosted on AWS, Azure, or GCP, with users selecting the underlying cloud provider at sign-up. Parquet and Delta Lake Parquet files store data in a columnar format, which improves efficiency for aggregation and analytics tasks. Delta Lake provides transactional operations on top of Parquet files by maintaining a version history, enabling row edits and deletions. This approach offers a database-like experience for handling large datasets, simplifying both analytics and machine learning workflows. Pricing and Usage Pricing for Databricks depends on the chosen cloud provider (AWS, Azure, or GCP) with an additional fee for Databricks' services. The added cost is described as relatively small, and the platform is accessible to both individual developers and large enterprises. Databricks is recommended for newcomers to data science and ML for its breadth of features and straightforward setup. Databricks, MLflow, and Other Integrations Databricks provides a hosted MLflow solution, offering experiment tracking and model management. The platform can access data stored in services like S3, Snowflake, and other cloud provider storage options. Integration with tools such as PyArrow is supported, facilitating efficient data access and manipulation. Example Use Cases and Decision Process Migration to Databricks is recommended when a client's existing infrastructure (e.g., on-premises Spark clusters) cannot scale effectively. The selection process involves an in-depth exploration of a client's operational challenges and goals. Databricks is chosen for clients lacking feature-specific needs but requiring a unified data analytics and ML platform. Personal Projects by Ming Chang Ming Chang has explored automated stock trading using APIs such as Alpaca, focusing on downloading and analyzing market data. He has also developed drone-related projects with Raspberry Pi, emphasizing real-world applications of programming and physical computing. Additional Resources Databricks Homepage Delta Lake on Databricks Parquet Format Raybeam Overview MLFlow Documentation
• 
  MLA 020 Kubeflow and ML Pipeline Orchestration on Kubernetes 29.01.2022 1t 8min

  Machine learning pipeline orchestration tools, such as SageMaker and Kubeflow, streamline the end-to-end process of data ingestion, model training, deployment, and monitoring, with Kubeflow providing an open-source, cross-cloud platform built atop Kubernetes. Organizations typically choose between cloud-native managed services and open-source solutions based on required flexibility, scalability, integration with existing cloud environments, and vendor lock-in considerations. Links Notes and resources at ocdevel.com/mlg/mla-20 Try a walking desk stay healthy & sharp while you learn & code Dirk-Jan Verdoorn - Data Scientist at Dept Agency Managed vs. Open-Source ML Pipeline Orchestration Cloud providers such as AWS, Google Cloud, and Azure offer managed machine learning orchestration solutions, including SageMaker (AWS) and Vertex AI (GCP). Managed services provide integrated environments that are easier to set up and operate but often result in vendor lock-in, limiting portability across cloud platforms. Open-source tools like Kubeflow extend Kubernetes to support end-to-end machine learning pipelines, enabling portability across AWS, GCP, Azure, or on-premises environments. Introduction to Kubeflow Kubeflow is an open-source project aimed at making machine learning workflow deployment on Kubernetes simple, portable, and scalable. Kubeflow enables data scientists and ML engineers to build, orchestrate, and monitor pipelines using popular frameworks such as TensorFlow, scikit-learn, and PyTorch. Kubeflow can integrate with TensorFlow Extended (TFX) for complete end-to-end ML pipelines, covering data ingestion, preprocessing, model training, evaluation, and deployment. Machine Learning Pipelines: Concepts and Motivation Production machine learning systems involve not just model training but also complex pipelines for data ingestion, feature engineering, validation, retraining, and monitoring. Pipelines automate retraining based on model performance drift or updated data, supporting continuous improvement and adaptation to changing data patterns. Scalable, orchestrated pipelines reduce manual overhead, improve reproducibility, and ensure that models remain accurate as underlying business conditions evolve. Pipeline Orchestration Analogies and Advantages ML pipeline orchestration tools in machine learning fulfill a role similar to continuous integration and continuous deployment (CI/CD) in traditional software engineering. Pipelines enable automated retraining, modularization of pipeline steps (such as ingestion, feature transformation, and deployment), and robust monitoring. Adopting pipeline orchestrators, rather than maintaining standalone models, helps organizations handle multiple models and varied business use cases efficiently. Choosing Between Managed and Open-Source Solutions Managed services (e.g., SageMaker, Vertex AI) offer streamlined user experiences and seamless integration but restrict cross-cloud flexibility. Kubeflow, as an open-source platform on Kubernetes, enables cross-platform deployment, integration with multiple ML frameworks, and minimizes dependency on a single cloud provider. The complexity of Kubernetes and Kubeflow setup is offset by significant flexibility and community-driven improvements. Cross-Cloud and Local Development Kubeflow operates on any Kubernetes environment including AWS EKS, GCP GKE, and Azure AKS, as well as on-premises or local clusters. Local and cross-cloud development are facilitated in Kubeflow, while managed services like SageMaker and Vertex AI are better suited to cloud-native workflows. Debugging and development workflows can be challenging in highly secured cloud environments; Kubeflow's local deployment flexibility addresses these hurdles. Relationship to TensorFlow Extended (TFX) and Machine Learning Frameworks TensorFlow Extended (TFX) is an end-to-end platform for creating production ML pipelines, tightly integrated with Kubeflow for deployment and execution. While Kubeflow originally focused on TensorFlow, it has grown to support PyTorch, scikit-learn, and other major ML frameworks, offering wider applicability. TFX provides modular pipeline components (data ingestion, transformation, validation, model training, evaluation, and deployment) that execute within Kubeflow's orchestration platform. Alternative Pipeline Orchestration Tools Airflow is a general-purpose workflow orchestrator using DAGs, suited for data engineering and automation, but less resource-capable for heavy ML training within the pipeline. Airflow often submits jobs to external compute resources (e.g., AI Platform) for resource-intensive workloads. In organizations using both Kubeflow and Airflow, Airflow may handle data workflows, while Kubeflow is reserved for ML pipelines. MLflow and other solutions also exist, each with unique integrations and strengths; their adoption depends on use case requirements. Selecting a Cloud Platform and Orchestration Approach The optimal choice of cloud platform and orchestration tool is typically guided by client needs, existing integrations (e.g., organizational use of Google or Microsoft solutions), and team expertise. Agencies with diverse client portfolios often benefit from open-source, cross-cloud tools like Kubeflow to maximize flexibility and knowledge sharing across projects. Users entrenched in a single cloud provider may prefer managed offerings for ease of use and integration, while those prioritizing portability and flexibility often choose open-source solutions. Cost Optimization in Model Training Both AWS and GCP offer cost-saving compute options for training, such as spot instances (AWS) and preemptible instances (GCP), which are suitable for non-production, batch training jobs. Production workloads that require high uptime and reliability do not typically utilize cost-saving transient compute resources, as these can be interrupted. Machine Learning Project Lifecycle Overview Project initiation begins with data discovery and validation of the client's requirements against available data. Cloud environment selection is influenced by client infrastructure, business applications, and platform integrations rather than solely by technical features. Data cleaning, exploratory analysis, model prototyping, advanced model refinement, and deployment are handled collaboratively with data engineering and machine learning teams. The pipeline is gradually constructed in modular steps, facilitating scalable, automated retraining and integration with business applications. Educational Pathways for Data Science and Machine Learning Careers Advanced mathematics or statistics education provides a strong foundation for work in data science and machine learning. Master's degrees in data science add the most value for candidates from non-technical undergraduate backgrounds; those with backgrounds in statistics, mathematics, or computer science may benefit more from self-study or targeted upskilling. When evaluating online or accelerated degree programs, candidates should scrutinize the curriculum, instructor engagement, and peer interaction to ensure comprehensive learning.
• 
  MLA 019 Cloud, DevOps & Architecture 13.01.2022 1t 15min

  The deployment of machine learning models for real-world use involves a sequence of cloud services and architectural choices, where machine learning expertise must be complemented by DevOps and architecture skills, often requiring collaboration with professionals. Key concepts discussed include infrastructure as code, cloud container orchestration, and the distinction between DevOps and architecture, as well as practical advice for machine learning engineers wanting to deploy products securely and efficiently. Links Notes and resources at ocdevel.com/mlg/mla-19 Try a walking desk stay healthy & sharp while you learn & code ;## Translating Machine Learning Models to Production After developing and training a machine learning model locally or using cloud tools like AWS SageMaker, it must be deployed to reach end users. A typical deployment stack involves the trained model exposed via a SageMaker endpoint, a backend server (e.g., Python FastAPI on AWS ECS with Fargate), a managed database (such as AWS RDS Postgres), an application load balancer (ALB), and a public-facing frontend (e.g., React app hosted on S3 with CloudFront and Route 53). Infrastructure as Code and Automation Tools Infrastructure as code (IaC) manages deployment and maintenance of cloud resources using tools like Terraform, allowing environments to be version-controlled and reproducible. Terraform is favored for its structured approach and cross-cloud compatibility, while other tools like Cloud Formation (AWS-specific) and Pulumi offer alternative paradigms. Configuration management tools such as Ansible, Chef, and Puppet automate setup and software installation on compute instances but are increasingly replaced by containerization and Dockerfiles. Continuous Integration and Continuous Deployment (CI/CD) pipelines (with tools like AWS CodePipeline or CircleCI) automate builds, testing, and code deployment to infrastructure. Containers, Orchestration, and Cloud Choices Containers, enabled by Docker, allow developers to encapsulate applications and dependencies, facilitating consistency across environments from local development to production. Deployment options include AWS ECS/Fargate for managed orchestration, Kubernetes for large-scale or multi-cloud scenarios, and simpler services like AWS App Runner and Elastic Beanstalk for small-scale applications. Kubernetes provides robust flexibility and cross-provider support but brings high complexity, making it best suited for organizations with substantial infrastructure needs and experienced staff. Use of cloud services versus open-source alternatives on Kubernetes (e.g., RDS vs. Postgres containers) affects manageability, vendor lock-in, and required expertise. DevOps and Architecture: Roles and Collaboration DevOps unites development and operations through common processes and tooling to accelerate safe production deployments and improve coordination. Architecture focuses on the holistic design of systems, establishing how different technical components fit together and serve overall business or product goals. There is significant overlap, but architecture plans and outlines systems, while DevOps engineers implement, automate, and monitor deployment and operations. Cross-functional collaboration is essential, as machine learning engineers, DevOps, and architects must communicate requirements, constraints, and changes, especially regarding production-readiness and security. Security, Scale, and When to Seek Help Security is a primary concern when moving to production, especially if handling sensitive data or personally identifiable information (PII); professional DevOps involvement is strongly advised in such cases. Common cloud security pitfalls include publicly accessible networks, insecure S3 buckets, and improper handling of secrets and credentials. For experimentation or small-scale safe projects, machine learning engineers can use tools like Terraform, Docker, and AWS managed services, but should employ cloud cost monitoring to avoid unexpected bills. Cloud Providers and Service Considerations AWS dominates the cloud market, followed by Azure (strong in enterprise/Microsoft-integrated environments) and Google Cloud Platform (GCP), which offers a strong user interface but has a record of sunsetting products. Managed cloud machine learning services, such as AWS SageMaker and GCP Vertex AI, streamline model training, deployment, and monitoring. Vendor-specific tools simplify management but limit portability, while Kubernetes and its ML pipelines (e.g., Kubeflow, Apache Airflow) provide open-source, cross-cloud options with greater complexity. Recommended Learning Paths and Community Resources Learning and prototyping with Terraform, Docker, and basic cloud services is encouraged to understand deployment pipelines, but professional security review is critical before handling production-sensitive data. For those entering DevOps, structured learning with platforms like aCloudGuru or AWS's own curricula can provide certification-ready paths. Continual learning is necessary, as tooling and best practices evolve rapidly. Reference Links Expert coworkers at Dept Matt Merrill - Principal Software Developer Jirawat Uttayaya - DevOps Lead The Ship It Podcast (frequent discussions on DevOps and architecture) DevOps Tools Terraform Ansible Visual Guides and Comparisons Which AWS container service should I use? A visual guide on troubleshooting Kubernetes deployments Public Cloud Services Comparison Killed by Google Learning Resources aCloudGuru AWS curriculum
• 
  MLA 017 AWS Local Development Environment 06.11.2021 1t 4min

  AWS development environments for local and cloud deployment can differ significantly, leading to extra complexity and setup during cloud migration. By developing directly within AWS environments, using tools such as Lambda, Cloud9, SageMaker Studio, client VPN connections, or LocalStack, developers can streamline transitions to production and leverage AWS-managed services from the start. This episode outlines three primary strategies for treating AWS as your development environment, details the benefits and tradeoffs of each, and explains the role of infrastructure-as-code tools such as Terraform and CDK in maintaining replicable, trackable cloud infrastructure. Links Notes and resources at ocdevel.com/mlg/mla-17 Try a walking desk stay healthy & sharp while you learn & code Docker Fundamentals for Development Docker containers encapsulate operating systems, packages, and code, which simplifies dependency management and deployment. Files are added to containers using either the COPY command for one-time inclusion during a build or the volume directive for live synchronization during development. Docker Compose orchestrates multiple containers on a local environment, while Kubernetes is used at larger scale for container orchestration in the cloud. Docker and AWS Integration Docker is frequently used in AWS, including for packaging and deploying Lambda functions, SageMaker jobs, and ECS/Fargate containers. Deploying complex applications like web servers and databases on AWS involves using services such as ECR for image storage, ECS/Fargate for container management, RDS for databases, and requires configuration of networking components such as VPCs, subnets, and security groups. Challenges in Migrating from Localhost to AWS Local Docker Compose setups differ considerably from AWS managed services architecture. Migrating to AWS involves extra steps such as pushing images to ECR, establishing networking with VPCs, configuring load balancers or API Gateway, setting up domain names with Route 53, and integrating SSL certificates via ACM. Configuring internal communication between services and securing databases adds complexity compared to local development. Strategy 1: Developing Entirely in the AWS Cloud Developers can use AWS Lambda's built-in code editor, Cloud9 IDE, and SageMaker Studio to edit, run, and deploy code directly in the AWS console. Cloud-based development is not tied to a single machine and eliminates local environment setup. While convenient, in-browser IDEs like Cloud9 and SageMaker Studio are less powerful than established local tools like PyCharm or DataGrip. Strategy 2: Local Development Connected to AWS via Client VPN The AWS Client VPN enables local machines to securely access AWS VPC resources, such as RDS databases or Lambda endpoints, as if they were on the same network. This approach allows developers to continue using their preferred local IDEs while testing code against actual cloud services. Storing sensitive credentials is handled by AWS Secrets Manager instead of local files or environment variables. Example tutorials and instructions: AWS Client VPN Terraform example YouTube tutorial Creating the keys Strategy 3: Local Emulation of AWS Using LocalStack LocalStack provides local, Docker-based emulation of AWS services, allowing development and testing without incurring cloud costs or latency. The project offers a free tier supporting core serverless services and a paid tier covering more advanced features like RDS, ACM, and Route 53. LocalStack supports mounting local source files into Lambda functions, enabling direct development on the local machine with changes immediately reflected in the emulated AWS environment. This approach brings rapid iteration and cost savings, but coverage of AWS features may vary, especially for advanced or new AWS services. Infrastructure as Code: Managing AWS Environments Managing AWS resources through the web console is not sustainable for tracking or reproducing environments. Infrastructure as code (IaC) tools such as Terraform, AWS CDK, and Serverless enable declarative, version-controlled description and deployment of AWS services. Terraform offers broad multi-cloud compatibility and support for both managed and cloud-native services, whereas CDK is AWS-specific and typically more streamlined but supports fewer services. Changes made via IaC tools are automatically propagated to dependent resources, reducing manual error and ensuring consistency across environments. Benefits of AWS-First Development Developing directly in AWS or with local emulation ensures alignment between development, staging, and production environments, reducing last-minute deployment issues. Early use of AWS services can reveal managed solutions—such as Cognito for authentication or Data Wrangler for feature transformation—that are more scalable and secure than homegrown implementations. Infrastructure as code provides reproducibility, easier team onboarding, and disaster recovery. Alternatives and Kubernetes Kubernetes represents a different model of orchestrating containers and services, generally leveraging open source components inside Docker containers, independent of managed AWS services. While Kubernetes can manage deployments to AWS (via EKS), GCP, or Azure, its architecture and operational concerns differ from AWS-native development patterns. Additional AWS IDEs and Services Lambda SageMaker Studio Cloud9 Conclusion Choosing between developing in the AWS cloud, connecting local environments via VPN, or using tools like LocalStack depends on team needs, budget, and workflow preferences. Emphasizing infrastructure as code ensures environments remain consistent, maintainable, and easily reproducible.
• 
  MLA 016 AWS SageMaker MLOps 2 05.11.2021 1t

  SageMaker streamlines machine learning workflows by enabling integrated model training, tuning, deployment, monitoring, and pipeline automation within the AWS ecosystem, offering scalable compute options and flexible development environments. Cloud-native AWS machine learning services such as Comprehend and Poly provide off-the-shelf solutions for NLP, time series, recommendations, and more, reducing the need for custom model implementation and deployment. Links Notes and resources at ocdevel.com/mlg/mla-16 Try a walking desk stay healthy & sharp while you learn & code Model Training and Tuning with SageMaker SageMaker enables model training within integrated data and ML pipelines, drawing from components such as Data Wrangler and Feature Store for a seamless workflow. Using SageMaker for training eliminates the need for manual transitions from local environments to the cloud, as models remain deployable within the AWS stack. SageMaker Studio offers a browser-based IDE environment with iPython notebook support, providing collaborative editing, sharing, and development without the need for complex local setup. Distributed, parallel training is supported with scalable EC2 instances, including AWS-proprietary chips for optimized model training and inference. SageMaker's Model Debugger and monitoring tools aid in tracking performance metrics, model drift, and bias, offering alerts via CloudWatch and accessible graphical interfaces. Flexible Development and Training Environments SageMaker supports various model creation approaches, including default AWS environments with pre-installed data science libraries, bring-your-own Docker containers, and hybrid customizations via requirements files. SageMaker JumpStart provides quick-start options for common ML tasks, such as computer vision or NLP, with curated pre-trained models and environment setups optimized for SageMaker hardware and operations. Users can leverage Autopilot for end-to-end model training and deployment with minimal manual configuration or start from JumpStart templates to streamline typical workflows. Hyperparameter Optimization and Experimentation SageMaker Experiments supports automated hyperparameter search and optimization, using Bayesian optimization to evaluate and select the best performing configurations. Experiments and training runs are tracked, logged, and stored for future reference, allowing efficient continuation of experimentation and reuse of successful configurations as new data is incorporated. Model Deployment and Inference Options Trained models can be deployed as scalable REST endpoints, where users specify required EC2 instance types, including inference-optimized chips. Elastic Inference allows attachment of specialized hardware to reduce costs and tailor inference environments. Batch Transform is available for non-continuous, ad-hoc, or large batch inference jobs, enabling on-demand scaling and integration with data pipelines or serverless orchestration. ML Pipelines, CI/CD, and Monitoring SageMaker Pipelines manages the orchestration of ML workflows, supporting CI/CD by triggering retraining and deployments based on code changes or new data arrivals. CI/CD automation includes not only code unit tests but also automated monitoring of metrics such as accuracy, drift, and bias thresholds to qualify models for deployment. Monitoring features (like Model Monitor) provide ongoing performance assessments, alerting stakeholders to significant changes or issues. Integrations and Deployment Flexibility SageMaker supports integration with Kubernetes via EKS, allowing teams to leverage universal orchestration for containerized ML workloads across cloud providers or hybrid environments. The SageMaker Neo service optimizes and packages trained models for deployment to edge devices, mobile hardware, and AWS Lambda, reducing runtime footprint and syncing updates as new models become available. Cloud-Native AWS ML Services AWS offers a variety of cloud-native services for common ML tasks, accessible via REST or SDK calls and managed by AWS, eliminating custom model development and operations overhead. Comprehend for document clustering, sentiment analysis, and other NLP tasks. Forecast for time series prediction. Fraud Detector for transaction monitoring. Lex for chatbot workflows. Personalize for recommendation systems. Poly for text-to-speech conversion. Textract for OCR and data extraction from complex documents. Translate for machine translation. Panorama for computer vision on edge devices. These services continuously improve as AWS retrains and updates their underlying models, transferring benefits directly to customers without manual intervention. Application Example: Migrating to SageMaker and AWS Services When building features such as document clustering, question answering, or recommendations, first review whether cloud-native services like Comprehend can fulfill requirements prior to investing in custom ML models. For custom NLP tasks not available in AWS services, use SageMaker to manage model deployment (e.g., deploying pre-trained Hugging Face Transformers for summarization or embeddings). Batch inference and feature extraction jobs can be triggered using SageMaker automation and event notifications, supporting modular, scalable, and microservices-friendly architectures. Tabular prediction and feature importance can be handled by pipe-lining data from relational stores through SageMaker Autopilot or traditional algorithms such as XGBoost. Recommendation workflows can combine embeddings, neural networks, and event triggers, with SageMaker handling monitoring, scaling, and retraining in response to user feedback and data drift. General Usage Guidance and Strategy Employ AWS cloud-native services where possible to minimize infrastructure management and accelerate feature delivery. Use SageMaker JumpStart and Autopilot to jump ahead in common ML scenarios, falling back to custom code and containers only when unique use cases demand. Leverage SageMaker tools for pipeline orchestration, monitoring, retraining, and model deployment to ensure scalable, maintainable, and up-to-date ML workflows. Useful Links MadeWithML overview & ML tutorials SageMaker Home SageMaker JumpStart SageMaker Model Deployment SageMaker Pipelines SageMaker Model Monitor SageMaker Kubernetes Integration SageMaker Neo
• 
  MLA 015 AWS SageMaker MLOps 1 04.11.2021 47min

  SageMaker is an end-to-end machine learning platform on AWS that covers every stage of the ML lifecycle, including data ingestion, preparation, training, deployment, monitoring, and bias detection. The platform offers integrated tools such as Data Wrangler, Feature Store, Ground Truth, Clarify, Autopilot, and distributed training to enable scalable, automated, and accessible machine learning operations for both tabular and large data sets. Links Notes and resources at ocdevel.com/mlg/mla-15 Try a walking desk stay healthy & sharp while you learn & code Amazon SageMaker: The Machine Learning Operations Platform MLOps is deploying your ML models to the cloud. See MadeWithML for an overview of tooling (also generally a great ML educational run-down.) Introduction to SageMaker and MLOps SageMaker is a comprehensive platform offered by AWS for machine learning operations (MLOps), allowing full lifecycle management of machine learning models. Its popularity provides access to extensive resources, educational materials, community support, and job market presence, amplifying adoption and feature availability. SageMaker can replace traditional local development environments, such as setups using Docker, by moving data processing and model training to the cloud. Data Preparation in SageMaker SageMaker manages diverse data ingestion sources such as CSV, TSV, Parquet files, databases like RDS, and large-scale streaming data via AWS Kinesis Firehose. The platform introduces the concept of data lakes, which aggregate multiple related data sources for big data workloads. Data Wrangler is the entry point for data preparation, enabling ingestion, feature engineering, imputation of missing values, categorical encoding, and principal component analysis, all within an interactive graphical user interface. Data wrangler leverages distributed computing frameworks like Apache Spark to process large volumes of data efficiently. Visualization tools are integrated for exploratory data analysis, offering table-based and graphical insights typically found in specialized tools such as Tableau. Feature Store Feature Store acts as a centralized repository to save and manage transformed features created during data preprocessing, ensuring different steps in the pipeline access consistent, reusable feature sets. It facilitates collaboration by making preprocessed features available to various members of a data science team and across different models. Ground Truth: Data Labeling Ground Truth provides automated and manual data labeling options, including outsourcing to Amazon Mechanical Turk or assigning tasks to internal employees via a secure AWS GUI. The system ensures quality by averaging multiple annotators' labels and upweighting reliable workers, and can also perform automated label inference when partial labels exist. This flexibility addresses both sensitive and high-volume labeling requirements. Clarify: Bias Detection Clarify identifies and analyzes bias in both datasets and trained models, offering measurement and reporting tools to improve fairness and compliance. It integrates seamlessly with other SageMaker components for continuous monitoring and re-calibration in production deployments. Build Phase: Model Training and AutoML SageMaker Studio offers a web-based integrated development environment to manage all aspects of the pipeline visually. Autopilot automates the selection, training, and hyperparameter optimization of machine learning models for tabular data, producing an optimal model and optionally creating reproducible code notebooks. Users can take over the automated pipeline at any stage to customize or extend the process if needed. Debugger and Distributed Training Debugger provides real-time training monitoring, similar to TensorBoard, and offers notifications for anomalies such as vanishing or exploding gradients by integrating with AWS CloudWatch. SageMaker's distributed training feature enables users to train models across multiple compute instances, optimizing for hardware utilization, cost, and training speed. The system allows for sharding of data and auto-scaling based on resource utilization monitored via CloudWatch notifications. Summary Workflow and Scalability The SageMaker pipeline covers every aspect of machine learning workflows, from ingestion, cleaning, and feature engineering, to training, deployment, bias monitoring, and distributed computation. Each tool is integrated to provide either no-code, low-code, or fully customizable code interfaces. The platform supports scaling from small experiments to enterprise-level big data solutions. Useful AWS and SageMaker Resources SageMaker DataWrangler Feature Store Ground Truth Clarify Studio AutoPilot Debugger Distributed Training JumpStart
• 
  MLA 014 Hosting and Deploying ML: Managed APIs, Serverless GPUs, Self-Hosting 18.01.2021 36min

  Where the model behind your product should run in 2026: managed APIs vs open weights, AWS native vs Modal, RunPod and Cloud Run GPU, vLLM and SGLang, quantized CPU inference on Spot, batch pricing, prod evals, and infra as code. Links Try a walking desk - stay healthy & sharp while you learn & code More OCDevel shows - this one has siblings, each on its own subject and produced the same way The infrastructure episode, standalone: where the model behind your product runs in 2026, and what each choice costs. Managed APIs versus self-hosted open weights, AWS native (Bedrock, SageMaker) versus serverless GPU platforms (Modal, RunPod, Cloud Run GPU) versus bare GPUs, the inference servers (vLLM, SGLang, llama.cpp), quantized CPU inference on Spot, queues and batch pricing, observability and evals, and a minute on infrastructure as code. The coding-agent workflow is covered in Vibe Coding in 2026 through Agentic Software Engineering; agent architecture in AI Agents in 2026. Calling a model vs running one The default for a product builder is a managed API: cost is linear in usage with no floor, while a self-hosted GPU costs the same idle as busy. Four things flip the default: sustained volume, privacy or data residency, a strict latency shape, or a model only available as open weights. Move one workload when a flip applies, not the whole stack. The cloud-hosted middle path: Amazon Bedrock's model catalog now spans Anthropic, OpenAI (including the open-weight gpt-oss models), Meta Llama, Mistral, DeepSeek, Qwen and more, and Custom Model Import serves your own safetensors weights on demand with no instance to manage. The framing on rent vs own is still a16z's "Navigating the High Cost of AI Compute": renting wins for almost everyone; owning pays only at large steady spend. AWS native Lambda and Fargate still have no GPUs; the Lambda quotas page has no accelerator dimension and the Fargate FAQ sends GPU work to EC2. Lambda offers 10 GB memory, 15-minute timeouts, 10 GB container images, and SnapStart for Python. SageMaker AI: real-time endpoints scale to zero only via inference components; asynchronous inference queues S3 payloads up to 1 GB and scales to zero; serverless inference is CPU-only; batch transform for offline scoring. AWS Batch GPU jobs on Spot compute environments are the cheapest bursty GPU on AWS, with no scheduler surcharge. Aurora PostgreSQL ships pgvector 0.8.2 with HNSW, and Aurora Serverless v2 auto-pause drops compute to zero when idle. Bedrock Knowledge Bases can use that Aurora as its store. Aurora DSQL has no extensions, so no pgvector. Serverless GPU platforms The category: ship a container or function, pay per GPU-second, scale to zero. Differentiators are cold start and warm-pool control. Modal: Python-native, per-second billing, T4 through B300; cold starts around a second with memory snapshots and minimum-container knobs; batch via Function.map. RunPod Serverless: queue-based or load-balanced endpoints, FlashBoot, async run and webhooks, wide GPU list from economy cards to B200. Cloud Run GPU: GA, NVIDIA L4 and RTX PRO 6000 Blackwell, scales to zero, instance-based billing required. Baseten with the open-source Truss packager: operated production deployments for custom models. Replicate joined Cloudflare in November 2025; Cloudflare Workers AI remains a curated open-model catalog, not a bring-your-own-weights host. fal: generative media inference first, serverless GPUs on the side. Status changes: Lambda's inference API is winding down in favor of GPU instances; Fly.io's GPU Machines appear withdrawn (docs removed, no GPU line on pricing). Bare GPUs, neoclouds, and Spot Hyperscaler families: AWS P6, Google Cloud GPUs, Azure ND/NC sizes. Neoclouds: CoreWeave, Nebius, Lambda, Crusoe; Vast.ai as a marketplace with interruptible pricing. A reserved box beats serverless when the GPU is busy most of the hour. EC2 Spot advertises up to 90 percent off on-demand, with a two-minute interruption notice. The discount is paid for with checkpointing, retry-on-reclaim, and instance-pool diversity using the SPOT_PRICE_CAPACITY_OPTIMIZED allocation strategy AWS recommends. Inference servers vLLM: PagedAttention, continuous batching, OpenAI- and Anthropic-compatible server, V1 engine, now under the PyTorch Foundation. SGLang: RadixAttention prefix reuse, releases every two weeks, production use at xAI, NVIDIA, Cursor and the major clouds. Hugging Face TGI was archived in March 2026; its notice points to vLLM and SGLang. TensorRT-LLM for NVIDIA-only maximum throughput; NVIDIA Dynamo orchestrates disaggregated prefill/decode across engines. llama.cpp for GGUF on CPUs, Apple Silicon and ARM with KleidiAI; Ollama now runs its own engine on GGML for multimodal models. Also: Triton Inference Server for non-LLM models, Ray Serve for composition, OpenLLM, LMDeploy, MLC-LLM. Quantization for CPU inference Quantized weights cut memory and, since batch-1 decode is memory-bandwidth bound, cut time per token; integer matrix instructions (ARM dot-product and MMLA, Intel VNNI) do the multiplies. GGUF quant naming: bits per weight, k-quant mixed precision, size variant; Q4_K_M takes Llama 3.1 8B from about 32 GB to about 5 GB. AWS Graviton guide for llama.cpp recommends Graviton3 and later for their BF16 and MMLA instructions and publishes a prebuilt llama-server container. vLLM's CPU backend covers x86 AVX512 and ARM. ONNX Runtime quantization for classic models, with the caveat that gains need VNNI or ARM dot-product hardware. Practice described in the episode: a quantized Qwen3-TTS model served by a C port on Graviton Spot instances via AWS Batch, checkpointing chunks to S3. Queues, batch, and real-time If no human waits, queue it: SQS with Lambda needs idempotency and partial-batch failure reporting; SageMaker async, Modal map, and RunPod async run are the same shape. Batch pricing is 50 percent off across OpenAI Batch, Anthropic Message Batches, and Bedrock batch inference, with separate rate limits and a 24-hour window. Bedrock prompt caching bills cache reads at a large discount within a short TTL; put stable prefix content first. Real-time paths need warm containers ("scale to zero" becomes "scale to one"), continuous batching, and streaming. Observability and evals in prod Trace every call with prompt, response, tokens, latency, cost. The OpenTelemetry GenAI conventions moved to their own repo; OpenLLMetry emits them. Tools: Langfuse (open source, part of ClickHouse since January 2026), Arize Phoenix (Elastic License), LangSmith, W&B Weave, Braintrust, Helicone as a proxy. Production evals score sampled traces with a judge model: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena found over 80 percent agreement with humans and named position, verbosity and self-enhancement bias. Classic drift: Evidently covers tabular drift and LLM metrics; WhyLabs ceased operations but open-sourced whylogs and langkit. Infrastructure as code SST runs on Pulumi with Terraform providers bridged; Pulumi for languages without SST's web opinions; Terraform under BSL since 2023 with OpenTofu as the open fork. Keep ML infra in the product's repo and deploy. Shout-outs Fast custom silicon APIs: Groq, Cerebras. Open-model inference clouds: Together AI, Fireworks, DeepInfra. Aggregators: OpenRouter, Hugging Face Inference Providers; dedicated Inference Endpoints. AWS Trainium. SageMaker JumpStart delisted models in March 2026, so pin weights in your own bucket. Related episodes MLA 22: Vibe Coding in 2026 MLA 23: Inside a Coding Agent MLA 24: Agentic Software Engineering MLA 28: AI Agents in 2026 MLA 29: OpenClaw and the Personal Agent More Gnothi-generated shows on AI, coding, video and business: ocdevel.com/moremlg.