Software Engineering Institute (SEI) Podcast Series

An LLM Evaluation Framework for High-Stakes AI 11.06.2026 16хв

Experimentation and validation of LLM performance is critical when building LLM-driven systems that must reliably deliver a service, from customer service chat bots to intelligence analysis tools. To help teams meet the need for rigorous evaluation methods, a research team in the SEI's AI Division led by Violet Turri has developed the Evaluating Large Language Models (ELM) library, which is built on best practices for LLM evaluation and benchmarking. In the latest episode from the Carnegie Mellon University Software Engineering Institute, Turri sits down with Katie Robinson, a design researcher also in the SEI's AI division, to discuss the ELM library, which turns evaluation from an ad-hoc process into a repeatable, extensible framework.

Protecting AI Systems Against Data Poisoning 04.06.2026 20хв

Data poisoning—where adversaries tamper with training data to corrupt model behavior—poses significant risks as AI adoption expands across critical sectors. Organizations without mechanisms in place to detect or prevent data poisoning are open to an avenue of attack that, once exploited, is difficult to remediate. Machine unlearning and model retraining are not always viable or effective solutions. In today's operational climate, where threat actors look to influence models and degrade the trust of users through incorrect behaviors, preventing data poisoning is more important than ever. In this episode of the SEI Podcast Series, Julie Lawler and James Cunningham—AI security researchers at Carnegie Mellon University's Software Engineering Institute—discuss the growing threat of data poisoning in AI systems and highlight emerging mitigation strategies, including chain-of-custody controls.

Goal-Line Defense: A Tool to Discover and Mitigate UEFI Vulnerabilities 15.04.2026 41хв

As recently as December 2025, the Carnegie Mellon University Software Engineering Institute (SEI's) CERT Coordination Center (CERT/CC) documented a UEFI-related vulnerability in certain motherboard models, illustrating that early-boot firmware behavior continues to present security challenges despite requiring local physical access to exploit. While CERT/CC reported seven UEFI vulnerabilities in 2025, that number remains small compared to reported vulnerabilities in other software. However, the consequences of a potential UEFI attack are often more serious given the extremely high privileges UEFI firmware possesses. In our latest SEI Podcast, Vijay Sarvepalli, a senior information security architect specializing in vulnerability and threat analysis in CERT, sits down with Michael Winter, deputy technical director of threat analysis in CERT, to discuss research and mitigation of UEFI vulnerabilities and discuss a new tool, the CERT UEFI parser, an open source tool that uses program analysis to reveal the architecture of UEFI software, and explore this veiled source of vulnerabilities.

Leadership, Legacy, and the Power of Mentors: Insights from Dr. Paul Nielsen 06.04.2026 18хв

In February 2026, Paul Nielsen announced that he will transition out of his role as director and chief executive officer of the Software Engineering Institute (SEI) at Carnegie Mellon University. During Nielsen's tenure, the SEI has marked major institutional milestones that underscore its enduring role in strengthening the security, resilience, and reliability of the nation's software- and AI-intensive systems. The institute recently celebrated 40 years of innovation and saw its contract renewed, which paved the way for CMU to operate the SEI for another five years. In our latest SEI podcast, Nielsen recently sat down with Matthew Butkovic, technical director of Risk and Resilience in the SEI's CERT Division, to discuss his legacy at the SEI, the impact of mentors, and the importance of encouraging scientists and engineers to do their best work.

With a Little Help from Our Civilian Friends: Cybersecurity Reserve Is Both Feasible and Advisable 20.03.2026 49хв

Cybersecurity staffing shortages are a major concern in the government given the increasingly sophisticated cyber attacks on the nation's critical infrastructure. In the FY2023 National Defense Authorization Act (NDAA), Congress tasked the Pentagon with finding flexible options to address cyber staffing needs. The Pentagon commissioned the SEI to conduct an independent study to assess the feasibility and advisability of creating a civilian cybersecurity reserve (CCR) that could harness cyber expertise from the private sector to mobilize a mission-ready workforce capable of operating in contested environments. In our latest podcast from the Carnegie Mellon University Software Engineering Institute (SEI), the lead authors on the report, Marie Baker, a technical manager in the SEI's CERT Division, and Chris May, technical director of the CERT Cyber Mission Readiness directorate, sit down with Mike Winter, deputy technical director of threat analysis, to discuss their findings.

Maturing AI Adoption: From Chaos to Consistency 02.03.2026 25хв

While Stanford University found that AI investments, optimism, and accessibility are rising, a recent MIT report suggests that 95 percent of organizations are realizing no returns on their generative AI investments. Research from Accenture found that only 8 percent of companies are scaling AI at an enterprise level and embedding the technology into core business strategy to maximize value. Mismatched expectations, misaligned applications, and poorly executed or untested implementation practices—not the technology itself—often keep organizations from realizing immediate value from an AI investment. For AI to increase efficiency, productivity, and value while conserving resources and lowering overall costs, organizations need to shift their focus from hype-driven experimentation to foundational capabilities and practical, measurable outcomes. In our latest podcast from the Carnegie Mellon University Software Engineering Institute, Dr. Ipek Ozkaya, technical director of AI-Native Software Engineering, sits down with Matthew Butkovic, technical director of Risk and Resilience in the SEI's CERT Division, to discuss their work on an AI Adoption Maturity Model that organizations can use to create a roadmap for predictable AI adoption and realization of AI benefits.

Temporal Memory Safety in C and C++: An AI-Enhanced Pointer Ownership Model 09.02.2026 24хв

In October 2025, CyberPress reported a critical security vulnerability in the Redis Server, an open-source in-memory database that allowed authenticated attackers to achieve remote code execution through a use-after-free flaw in the Lua scripting engine. In 2024, another prominent temporal memory safety flaw was found in the Netfilter subsystem in the Linux kernel: CVE-2024-1086. Bugs related to temporal memory safety, such as use-after-free and double-free vulnerabilities, are challenging issues in C and C++ code. In this podcast from the Carnegie Mellon University Software Engineering Institute (SEI), Lori Flynn, a senior software security researcher in the SEI's CERT Division, and David Svoboda, a senior software engineer, also in CERT, sit down with Tim Chick, technical manager of CERT's Applied Systems Group, to discuss recent updates to the Pointer Ownership Model for C, a modeling framework designed to improve the ability of developers to statically analyze C programs for errors involving temporal memory.

AI for the Warfighter: Acquisition Challenges and Guidance 29.01.2026 24хв

On November 7, the Department of War released an acquisition transformation strategy that seeks to remove bureaucratic hurdles and streamline acquisition processes to enable even more rapid adoption of technologies, including artificial intelligence. Getting AI into the hands of warfighters requires disciplined AI Engineering. In this podcast from the Carnegie Mellon University Software Engineering Institute, Carol Smith, lead of human-centered research in the SEI's AI Division, and Brigid O'Hearn, the SEI's lead of software modernization policy for the Department of War, sit down with Eileen Wrubel, the SEI's technical director of Transforming Software Acquisition Policy and Practice, to discuss AI Engineering challenges and guidance in the defense acquisition space.

Visibility Through the Clouds with Network Flow Logs 15.01.2026 35хв

Organizations, including the U.S. military, are increasingly adopting cloud deployments for their flexibility and cost savings. The shared security model utilized by cloud service providers removes some of the adopting organization's responsibility for system administration and security. But it leaves them on the hook for monitoring hosted applications and resources. Cloud flow logs are a valuable source of data for supporting these security responsibilities and attaining situational awareness. The SEI has a long history of supporting flow log collection and analysis, including tools for collection in Azure and AWS. In this podcast from the Carnegie Mellon University Software Engineering Institute (SEI), two leading researchers in this area, principal researcher Tim Shimeall and security data analyst Ikem Okafo, both with the SEI's CERT Division, sit down with Dan Ruef, technical manager of the CERT Division's Network Situational Awareness Group, to discuss how to enhance security with cloud flow analysis as well as available tools and resources.

Orchestrating the Chaos: Protecting Wireless Networks from Cyber Attacks 02.12.2025 37хв

From early 2022 through late 2024, a group of threat actors publicly known as APT28 exploited known vulnerabilities, such as CVE-2022-38028, to remotely and wirelessly access sensitive information from a targeted company network. This attack did not require any hardware to be placed in the vicinity of the targeted company's network as the attackers were able to execute remotely from thousands of miles away. With the ubiquity of Wi-Fi, cellular networks, and Internet of Things (IoT) devices, the attack surface of communications-related vulnerabilities that can compromise data is extremely large and constantly expanding. In the latest podcast from the Carnegie Mellon University Software Engineering Institute (SEI) Joseph McIlvenny, a senior research scientist, and Michael Winter, vulnerability analysis technical manager, both with the SEI's CERT Division, discuss common radio frequency (RF) attacks and investigate how software and cybersecurity play key roles in preventing and mitigating these exploitations.

From Data to Performance: Understanding and Improving Your AI Model 10.11.2025 26хв

Modern data analytic methods and tools—including artificial intelligence (AI) and machine learning (ML) classifiers—are revolutionizing prediction capabilities and automation through their capacity to analyze and classify data. To produce such results, these methods depend on correlations. However, an overreliance on correlations can lead to prediction bias and reduced confidence in AI outputs. Drift in data and concept, evolving edge cases, and emerging phenomena can undermine the correlations that AI classifiers rely on. As the U.S. government increases its use of AI classifiers and predictors, these issues multiply (or use increase again). Subsequently, users may grow to distrust results. To address inaccurate erroneous correlations and predictions, we need new methods for ongoing testing and evaluation of AI and ML accuracy. In this podcast from the Carnegie Mellon University Software Engineering Institute (SEI), Nicholas Testa, a senior data scientist in the SEI's Software Solutions Division (SSD), and Crisanne Nolan, and Agile transformation engineer, also in SSD, sit down with Linda Parker Gates, Principal Investigator for this research and initiative lead for Software Acquisition Pathways at the SEI, to discuss the AI Robustness (AIR) tool, which allows users to gauge AI and ML classifier performance with data-based confidence.

What Could Possibly Go Wrong? Safety Analysis for AI Systems 31.10.2025 36хв

How can you ever know whether an LLM is safe to use? Even self-hosted LLM systems are vulnerable to adversarial prompts left on the internet and waiting to be found by system search engines. These attacks and others exploit the complexity of even seemingly secure AI systems. In our latest podcast from the Carnegie Mellon University Software Engineering Institute (SEI), David Schulker and Matthew Walsh, both senior data scientists in the SEI's CERT Division, sit down with Thomas Scanlon, lead of the CERT Data Science Technical Program, to discuss their work on System Theoretic Process Analysis, or STPA, a hazard-analysis technique uniquely suitable for dealing with AI complexity when assuring AI systems.

Getting Your Software Supply Chain In Tune with SBOM Harmonization 23.10.2025 23хв

Software bills of materials or SBOMs are critical to software security and supply chain risk management. Ideally, regardless of the SBOM tool, the output should be consistent for a given piece of software. But that is not always the case. The divergence of results can undermine confidence in software quality and security. In our latest podcast from the Carnegie Mellon University Software Engineering Institute (SEI), Jessie Jamieson, a senior cyber risk engineer in the SEI's CERT Division, sits down with Matt technical director of Risk and Resilience in CERT, to talk about how to achieve more accuracy in SBOMs and present and future SEI research on this front.

API Security: An Emerging Concern in Zero Trust Implementations 08.10.2025 17хв

Application programing interfaces, more commonly known as APIs, are the engines behind the majority of internet traffic. The pervasive and public nature of APIs have increased the attack surface of the systems and applications they are used in. In this podcast from the Carnegie Mellon University Software Engineering Institute (SEI), McKinley Sconiers-Hasan, a solutions engineer in the SEI's CERT Division, sits down with Tim Morrow, Situational Awareness Technical Manager, also with the CERT Division, to discuss emerging API security issues and the application of zero-trust architecture in securing those systems and applications.

Delivering Next-Generation AI Capabilities 29.09.2025 30хв

Artificial intelligence (AI) is a transformational technology, but it has limitations in challenging operational settings. Researchers in the AI Division of the Carnegie Mellon University Software Engineering Institute (SEI) work to deliver reliable and secure AI capabilities to warfighters in mission-critical environments. In our latest podcast, Matt Gaston, director of the SEI's AI Division, sits down with Matt Butkovic, technical director of the SEI CERT Division's Cyber Risk and Resilience program, to discuss the SEI's ongoing and future work in AI, including test and evaluation, the importance of gaining hands-on experience with AI systems, and why government needs to continue partnering with industry to spur innovation in national defense.

The Benefits of Rust Adoption for Mission-and-Safety-Critical Systems 16.09.2025 19хв

A recent Google survey found that many developers felt comfortable using the Rust programming language in two months or less. Yet barriers to Rust adoption remain, particularly in safety-critical systems, where features such as memory and processing power are in short supply and compliance with regulations is mandatory. In our latest podcast from the Carnegie Mellon University Software Engineering Institute (SEI), Vaughn Coates, an engineer in the SEI's Software Solutions Division, sits down with Joe Yankel, initiative Lead of the DevSecOps Innovations team at the SEI, to discuss the barriers and benefits of Rust adoption.

Threat Modeling: Protecting Our Nation's Complex Software-Intensive Systems 05.09.2025 35хв

In response to Executive Order (EO) 14028, Improving the Nation's Cybersecurity, the National Institute of Standards and Technology (NIST) recommended 11 practices for software verification. Threat modeling is at the top of the list. In this podcast from the Carnegie Mellon University Software Engineering Institute (SEI), Natasha Shevchenko and Alex Vesey, both engineers with the SEI's CERT Division, sit down with Timothy Chick, technical manager of CERT's Applied Systems Group, to discuss how threat modeling can be used to protect software-intensive systems from attack. Specifically, they explore how threat models can guide system requirements, system design, and operational choices to identify and mitigate threats.

Understanding Container Reproducibility Challenges: Stopping the Next Solar Winds 30.07.2025 25хв

Container images are increasingly being used as the main method for software deployment, so ensuring the reproducibility of container images is becoming a critical step in protecting the software supply chain. In practice, however, builds are often not reproducible due to elements of the build environment that rely on nondeterministic factors such as timestamps and external dependencies. Lack of reproducibility can lead to lack of trust, broken builds, and possibly mask hidden malware insertion. Vessel, a recent tool from the Carnegie Mellon University Software Institute (SEI), helps developers identify the difference between two container images to help sort benign from problematic issues. In this SEI Podcast, Kevin Pitstick, a senior software engineer at the SEI and Vessel's lead developer, and Lihan Zhan, a software engineer at the SEI working on tactical and AI-enabled systems, sit down with Grace Lewis, lead of the Tactical and AI-Enabled Systems (TAS) applied research and development team at the SEI, to discuss the Vessel tool, its development, and application in mission-critical settings.

Mitigating Cyber Risk with Secure by Design 14.07.2025 32хв

Software enables our way of life, but market forces have sidelined security concerns leaving systems vulnerable to attack. Fixing this problem will require the software industry to develop an initial standard for creating software that is secure by design. These are the findings of a recently released paper coauthored by Greg Touhill, director of the Software Engineering Institute (SEI) CERT Division. In this latest SEI podcast, Touhill and Matthew Butkovic, director of Cyber Risk and Resilience at CERT, discuss the paper including its recommendations for making software secure by design.

The Magic in the Middle: Evolving Scaled Software Solutions for National Defense 18.06.2025 21хв

A January 2025 Defense Innovation Board study on scaling nontraditional defense innovation stated, "We must act swiftly to ensure the DoD leads in global innovation and competition over AI and autonomous systems – and is a trendsetter for their responsible use in modern warfare." In this podcast from the Carnegie Mellon University Software Engineering Institute (SEI), chief technical officer Tom Longstaff discusses the SEI's long-standing work to help the DoD rapidly scale technology including artificial intelligence (AI) and autonomous systems.