FLOSS Weekly

Jaksot

• 
  Episode 882 transcript 16.09.2026

  2026-09-15 Jonathan: [00:00:00] time for Floss Weekly. That's the show about free Libre and open-source software. I'm your host, Jonathan Bennett, and today we're talking about, well, open-source, of course. We're talking open-source automation and some experts in that field. I've got Jan Altenberg with me, and he is, I discovered during the pre-show, a German engineer working on, well, a lot of different things, all the way from the pre-empt real-time kernels to being around computers since the 1980s. He started with a Commodore VC-20, the ever-present VIC-20, one of, I don't, I think the Commodore 64 actually holds the record, but the VIC-20 has to be one of the most popular computers ever sold. But yeah, we're talking open-source, we're talking automation, and let's bring the man himself on. Jan, welcome to the show. Jan: Thanks for having me. Great to be here. Great. Looking forward to talk about open source [00:01:00] today. As you mentioned, I've been working with computers for quite some years. Actually, I have to admit that I was not working with Commodore. This was my colleague I'm working with at OSADEL. But actually, for me, it's even more than, I would say, I've spent 35 years with computers. The drawback was that my first computer was running proprietary software, which was pretty sad, but I think I'm with open source since 30 years now. Jonathan: Yeah, yeah, that's a long time. Yeah, I see now that it was Alex, your colleague, that started with the Commodore VIC-20, but still, you've been around, you've been doing this for a while. We were talking before the show about how this interview came together, and I'm pretty sure that it's from where I was at Embedded World earlier in the year. And I took one of the days, and I went to the software wing, because they have like five different halls at Embedded World, because it's a huge show. And I went to the software wing, and I just walked around, and I looked for anything [00:02:00] that had "open source" in the title or that I knew was an open source project. I'm like, "Here's my business card. You should come be on the podcast." And so we finally made this one work, talking about the Open Source Automation Developer Lab. Now, what is open source automation in this context? Are we going to be a very AI-heavy episode again today? Is that the sort of automation that we're talking about? Jan: Oh, well. So, AI is everywhere these days, but maybe it's a bit different for us. So, what is the Open Source Automation Development Lab? As the name says, we do a lot of with open source and automation. So, we are around since a bit more than 20 years, actually. What is interesting for you to know is, I mean, as you mentioned at Embedded World, there's many different companies, right, doing software, doing open source. So, we actually, we do not have the legal form of a company, actually. [00:03:00] So, we are a registered German cooperative. Jonathan: Okay. Jan: So, that works a bit different. So, it's actually that we are owned by our members, So, we do not have a single owner, and we don't really primarily operate profit-oriented, so the thing what people do is, with a cooperative, they gather in a place and put some budget together to do something useful in common interest, right? So, in Central Europe, this is pretty common, for example, for small wine producers, so you might have everything to do the wine, but you do not, cannot maintain as a small company all the infrastructure to sell the wine and do all the production and the bottling and that stuff, and that's why wine producers gather in a cooperative, so they could finance that infrastructure together. And what happens at OSADL is that our members do the very same for open source in industrial products. So, our members [00:04:00] have the interest of using open source in industrial products, mainly in automation, but in other products as well, right? And the reason why we were founded roughly 20 years ago was basically real-time Linux. Which was around at that time, in the early 2000s, and it was proven to work. So, it became of interest for German machine producers, so it rang the interest of many engineers in Germany. But the question for the deciders was: this is a young project, right? So, will the maintainers continue that work? Who can help us with quality assurance, and who to ask, right? And this is why these companies founded OSADEL back then, to, like, provide funding to the real-time maintainers and to do commonly funded quality assurance for real-time Linux. So, this was the main idea. With 10 members back then, so today we have more than 100, and, of course, over the last [00:05:00] 20 years, the interest of our members grew into a different direction, and yes, AI as well. But many different topics, like even licensing topics, that are of common interest, are covered by our community. So, just to sum it briefly up, we're pretty much a community that supports its members in using open source in their products. Jonathan: Gotcha. The story with Realtime Linux has been pretty interesting, too, over the years, because that was a pretty big change to the Linux kernel, and it's something I've been watching and been sort of excited about for a long time, mainly because I do some Realtime Audio stuff. Jan: Yes. Jonathan: And, you know, I remember way back in the day trying to, you know, I was doing crazy things like running live sound through a Linux machine to be able to use plug-ins on the live sound, and it's like, you've got to get your latency down as low as possible, and, you know, we were trying to pull out all the stops and figure out how to do that, and the Realtime kernel [00:06:00] was one of the ways to do it. And I remember trying it one time, and it immediately crashed the machine, because we had to use the NVIDIA driver on that machine. They just, they did not work. They just, they did not play nicely together at all. We've come quite a ways now that, I mean, it's landed in the upstream kernel and everything. Jan: Exactly. Yes. Jonathan: Yeah, how much of that story were you involved in? Jan: Pretty much the whole story, actually, pretty much the whole story, because that was the topic that really got me into open source, right? Jonathan: Huh? Jan: So, I got my first Linux installation when I was 15, but that was just like out of interest, wanted to try things out, and the cool stuff about open source was that I could look into stuff and figure out how it was working. But what happened in my studies is that back then I was working for a machine producer and doing my thesis, and they were evaluating Linux as the next generation operating system, and I was in charge to look into real-time Linux. So, back then in 2004, [00:07:00] I started doing my thesis on real-time Linux and porting machine controllers to real-time Linux, and since then I've been following the whole history. of real-time Linux supporting people using real-time, helping people with testing, evaluating new use cases. So all over these 20 years, this was a big topic for me. It was always a pleasure for me to present on conferences about real-time Linux. So at some point I really got passionate about that. So really one of my favorite topics. And you can imagine that the very day when the pull request got merged, enabling preempt RT in the mainline kernel was a very, very special moment for me. And it actually happened pretty much exactly 20 years after I submitted my thesis for review. So it was quite some time. Jonathan: Ha-ha-ha-ha-ha-ha-ha! Jan: So I really still, I think the maintainers can be really proud. It was a very [00:08:00] big change. And I think you mentioned a very interesting aspect because I mean, having this in the mainline kernel is great, but just look at all the things that happened over these 20 years because, Like you mentioned, you enable real-time and drivers start crashing. Because you didn't have proper locking or something, some serialization did not work, but, I mean, you immediately discover that when you have full preemption, right? So, I think there was a big quality jump for many drivers in the Linux kernel with real-time, and apart from that, I mean, just look at the tracing infrastructure, the high-resolution timers, generic interrupt handling, and so on, and so on. All these nice features, threaded interrupts, all these features are there because they came from the real-time development, and I think it's, for me, it's always a very good example on which synergies you can have with open source, right? Because people develop the real-time for a specific [00:09:00] use case or specific use cases, but they made Linux better for each and anyone, right? I think it's one of the best examples you could have on how open source works. really helps you saving efforts. Jonathan: Yeah. There's a sort of parallel, something that I just became aware of here recently. So, the real-time effort, is it fair to say that part of the reason for that was to be able to use Linux in automotive cases? Jan: Partly, partly. So, I think the first heavy use cases was automation, was even big server machines that did high-speed trading and stuff, so they had to do deterministic calculations and stuff, so these were the first early use cases on that, and that maybe was the enabler to slowly push it also towards automotive. Jonathan: So, I was at a talk in London, and I think it was [00:10:00] an NVIDIA engineer, interestingly enough, but he was looking at the question about using Linux and automotive and doing safety engineering around the Linux kernel. Jan: Yes, yes, that's a big topic, yeah. Jonathan: And the things that he was describing that would have to be changed in the Linux kernel, I was sitting there going, "my goodness, this is a lot of work for, you know, just to..." Jan: Yep, yes. Jonathan: I am always a little skeptical, this is just me, but I'm always a little skeptical of some of these, like, engineering-proof things, like, we can make the kernel safe by doing this, and I'm like, I mean, it's kind of like moving code to Rust, right? Like, yes, you can fix this class of memory bugs, but you also have this other class of bugs that Rust doesn't help you with at all, and so I was sitting here listening to this presentation going, "This doesn't necessarily guarantee that it's not going to crash." But all that to say, I see an interesting parallel between the 20 [00:11:00] years that it took to get real-time into the kernel, and now this sort of growing interest in getting Memory safety, but also, like, you know, big iron engineering safety, or I forget even the term that they use, but the amount of changes that that would take to land that in the kernel. Is that something you guys are involved in at all? Jan: Yes, actually, so we're still following that. Jonathan: I'm not surprised. Jan: We do that for 20 years now. Actually, we've been actively involved in safety efforts for the Linux kernel. So, we did run quite some time ago, we ran a project which was called CO2LinuxMP, which was actually looking into what could be proper strategies to get safety certification for Linux-based systems. So, we built some know-how in that area, and as you say, this is maybe still a different story than real-time, because that effort is definitely huge. [00:12:00] And we're still a bit involved in the ongoing efforts. For example, there's the ELISA project going on. People still look into methodologies. So, the interesting part of all the efforts that are still around there is not that someone is looking into getting specifically a certification, right? The thing is, what would be the proper mindset? What might be the tools and strategies? Because if you take an operating system like Linux, right, it hasn't been developed following a specific process, right? So, it hasn't been developed following a safety standard. So, what routes you could take, you need to see what route to take Using something existing and prove that you can make it safe, right? And this is pretty much what all these people are these days looking into. How can you adjust the development process? How can you prove that things are good? How can you improve? How can you do documentation so that in future it might [00:13:00] be easier to get that kind of certification? Because I'm quite sure, or I am sure, I'm aware that certified open source systems are around on the market, right? The question is just, in this case, it was always very individual effort, but can we make this the common case? And I think this is the next big step for industry. So, the first big round for industry was, okay, they need real time, and the next big question is, can you get safety certification? Jonathan: interesting to watch. Do you think it's going to take another 20 years? Jan: Well, I gave up doing estimations, right? I did do quite a few estimations for real-time Linux, right, and at some point, we always kept saying next year because there's, of course, always a next year, right? Jonathan: Right, right, right. Jan: So, I don't want to do estimation there. I can say I am confident that this [00:14:00] can happen because so many companies are looking into that way. And open source became of interest for many different industries because they are independent in their decisions, and maybe also for security reasons because the software they have is just open and transparent. Jonathan: Mm-hmm. Jan: So, I am confident that there will be ways, but it's hard to estimate how long that will take. It will definitely be a way bigger effort than just getting real-time into Linux. Jonathan: Sure. Europe is kind of making a move as a whole, even due to some regulatory pressure, I think, towards open source solutions, specifically away from Closed source solutions coming from U.S. companies. I think that's part of it, but I see it as sort of a bigger trend just away from proprietary solutions and towards open source solutions, which I love. I think that's great, and [00:15:00] I think every country should be thinking about that. It absolutely makes sense to me. Jan: Yeah, I think there's two aspects on that. So, definitely, there's a European strategy on getting more independent when it comes to software, so there's a European strategy in looking into open source and trying also to push countries to move open source. So, this is an effort I really appreciate. There was actually recently also the possibility to submit ideas and arguments to the European Commission on what do we need for European open source strategy. So, even the European Commission is looking for feedback, so each and anyone was possible to submit. So, this is one thing. So, there is a strategy there. If you look into that from an engineering perspective, I think, this is something that arrived on the market quite earlier, [00:16:00] before we had that strategy coming up from regulatory side. It's like that, I mean, sooner or later, you realize that, I mean, just imagine you're building So, what happens with proprietary software very regularly, and I've been through that, you need to do updates because Well, you don't get support anymore for the old version, so you need to rework a lot of stuff without any real need, right? So, this is a pressure and a burden you basically don't have with open source, so you have way more flexibility. It's basically up to you what you use. Of course, you need to update, you need to maintain. But, I mean, you can look into the software, you can do it on your own, and you can decide who can support you, right? There's [00:17:00] so many brilliant minds, so the thing is you're not bound to a single entity, right? So, it's still work to do, but you become way more independent, so this is one aspect to avoid these. There are artificial release cycles that you don't really need, right? So, and the other thing is just when you debug real-time topics. So, when you're with proprietary systems, you usually always get to the point where you see, okay, data is going in and gets back too late. So, you can open a ticket and that's it. Jonathan: It's difficult to try to troubleshoot the inside of the black box. Jan: So, if you're using real-time Linux or anything else, or Zephyr or something that is open source, you can just see what happens, right? Exactly, exactly. Jonathan: In the past, probably 10 years, I've had a couple of times now, people have come to me with, in one case, it was a Windows 3.1 machine, [00:18:00] in the other case, it was a Windows 3.1, I think one Windows 95, and one Windows 98 machine, and they've all come to me and gone, "This is incredibly important for our business use case. Please make it continue to work." DOSBox to the rescue, by the way, is an incredible piece of software for making that happen, but yeah, it's ridiculous. Jan: Yes. Jonathan: You see these businesses that are locked into their old solution, whether it be for hardware or for some old piece of software that nobody has the source code to anymore. It's incredible, and I've seen it several times, yeah. Jan: That's it, yeah. And I think that was a very good example, because the fun fact here is, in many cases, open source emulators save proprietary software, right? Because you can still run it, [00:19:00] right? Jonathan: Mm-hmm. One more time that this happened to me, it was a doctor's office, and they had their old back-office system running on an old SCO Unix box. And they were like, we need to be able to fix this, keep it working, and get backups from it. And this has been, again, probably 10 years ago. And I said, okay, I can do that. And I did Fedora install and set up QEMU and virtualized their old SCO UNIX box and got it working. Which is interesting, because the SCO UNIX lawsuit was in the news one more time in the past couple of weeks, which is what brought that to mind. Jan: *laughter* Jonathan: But yeah, it's a problem all over the place of people getting themselves locked into these ancient systems. Jan: Exactly, but I think people started to learn from that once again. I think that message arrived, and people start rethinking, so we can see that over the last 20 years, now after 20 years of using open source, I was at the position where I thought, okay, now all industries should be aware about open source. There's still nowadays, still industries, still now switching from proprietary systems to open source. [00:20:00] Even these days. Jonathan: Yeah. What are some of the other projects that the lab is involved in? We've talked about real-time Linux and the safety-critical stuff, but I'm sure there's some other things that you guys have your hands in. Jan: Of course, there's many. So, we could talk days. Now that Alex wasn't able to join us, I should briefly mention Alex's work, which is closely related to real-time Linux, but not only just related. So, I mentioned that one of the reasons why Ozan was founded was quality assurance. So, what we do is, we do run quality assurance for more than 200 embedded systems in our lab. So, these boards do come from our members, or we choose them on our own, but basically they do come from our members, and they want to see how stable are these boards operating 24/7. Of course, real-time behavior is one topic we look into there, because the [00:21:00] thing is basically that with operating systems like Linux running on modern processes, It's not that easy to prove the real-time behavior. In the good old times, you just did a code path analysis, and with old processes like M68K, you exactly knew, "I have this number of assembly instructions, and it will take that time to execute." Now, estimating the execution time on a modern processor is close to impossible. So, you have at least two or three caching levels. You have branch prediction. You have SMIs. You have microcode patches, and so on, and so on. So, the only thing what you could do is to evaluate and to collect as much data as possible. This is what we basically do at the QA farm. So we do latency measurements 24/7 under different load scenarios, and we collect all the results of the measurement runs and render long-term plots that our members [00:22:00] could even spot single latency spikes that happen once a month or once a year. So we put some patches on top with the layer tracing infrastructure that we could easily spot when a latency happened and who was the culprit and so on. So this is one intention. On the other side, we do health monitoring like we monitor CPU load, temperature, fans, whatever, to track the stability, and you find really interesting stuff when you're monitoring embedded boards, like doing random reboots that a user in the field won't even realize because it's fast enough, right? But just simply monitoring the uptime of a device can give you interesting results. And one of the fun facts is actually that work is maintained by Alex. So, you can imagine that with 200 boards in the farm, he needs good running shoes, because something is breaking every minute, so you need to do spare parts there. So, the fun fact is [00:23:00] that one of the most breaking parts for most of the boards is just the power supply, which is, you know, Jonathan: That checks out. Yeah. Jan: Yeah. The bad thing about breaking power supply is that it's not binary. So if it would just break and the board would be switched off, that would be nice. The thing is that you end up with random behavior on the board, like random resets, random kernel crashes. So the basic thing we always do is when we end up with random behavior from one day to the other, first thing we do, we check the power supply. So this was one of the 20-year fun facts of quality assurance. So this is maybe one of the Bigger technical projects we are running. Jonathan: We love technical geeky stuff, by the way. Jan: What I need to mention as well is, so there's a lot of technical geeky stuff we do, right? But there's also quite a few legal geeky stuff we do as well, because this might be something [00:24:00] that didn't happen that much so far: we tried to extend the open source principle to as many topics as we could, because, I mean, we're a community trying to finance things together in the area of open source. And it turned out that most of our members, most of the companies that gathered at OSADE, do have the same legal questions over and over again, because all of them are using open source in their products. So, most importantly, they need to understand license obligations, so what do they need to do when they copy and distribute, do they need to provide source code, yes or no, what else do they need to do. Also, sometimes license obligations might collide even with regulations. Imagine, like, you have installation obligations with the GPL, but due to some regulations, you might need to implement a secure boot on your device. How do you bring that together? So, what we did is basically we established a community-based legal consultancy, [00:25:00] so we did a legal FAQ database. And it pretty much works in an open source sense, so when one of our members comes up with a legal question, That member could hire a lawyer, but in this case, that member would address that question to us, we hire the lawyer, the lawyer gets that question answered, and question and answer make it to the database and are accessible to all of the members. So, that with a very low financial effort, our members get that, yeah, that community synergies, and that the cool thing is that it's not just these questions, so whenever you have a question, our members could log in and check what has been asked in sense of legal topics and open source before, and maybe that question was already answered in the past before. So, this is a pretty nice example on how you could also do community-based work even on legal topics, because there's so much overlap for people using [00:26:00] open source in their products, so it completely made sense for us to share that efforts between different people. One of our really important assets, we even started open-source projects in the legal part. So, we have one open-source project we are running is the open-source license checklists. So, the thing with open-source licensing is, the great stuff is, if something is officially open-source, you always know your freedoms, what you are allowed to do. So, this is great. So, the difference between open-source licenses is basically the obligations that you have when you copy and distribute, right? So, we have permissive licenses that only require, provide copyright notices and license text. We have copyleft licenses that have licensing obligations as well, or source code disclosure. The problem is that we all know in the open source world, Since everyone needs to invent [00:27:00] his own open source license at least once in a lifetime, Jonathan: I wish you were wrong. Jan: there seems to be a rule about that. Yes, I would wish so, too. But that was the reason why we invented the license checklist, because you have this huge variety of different licenses. It's sometimes hard for companies to derive obligations for their product, because they would need to read all the licenses, and the licenses have been written at different times by different people, by engineers, by lawyers, so sometimes it's really hard to understand. And what we started to do is we started translating the most commonly used open source licenses into a canonical language. So, it would basically read like, you must provide copyright notice, you must provide license text, you must provide source code, or you must either provide source code or provide a written offer, if we take the example of the GPL. So, all that data is Creative Commons [00:28:00] licensed, Jonathan: Right. Jan: so you can just download it from Mozilla and use it. So, I think it's quite widely used It's an open source checklist project. I think it's also a very good example on how you could look into legal topics together, right? So, people could send us checklists. If someone needs a checklist, we can do it, right? So, we tried to help out with that, and what we even included is that with the checklist, we even have a matrix that could tell you the compatibility between the licenses. So, can you take BSD2 and and redistribute it under GPL2 or the other way around. Jonathan: Right. Jan: This is also important tooling. Just think about package maintainers and distributions. When they need to check, okay, these are the libraries I'm linking against, so are the libraries, is the licensing compatible? This is basically where you could use the compatibility matrix. So, it's [00:29:00] open source. You can just download it, several formats. It's JSON, and of course, we always highly appreciate the feedback on that. So, just a few examples on what else we are doing at Dozal. Jonathan: Yeah. I'm curious: with that tool, can you feed it a software bill of materials and get it to essentially analyze it and tell you if you've got problems and what the license needs to be at the end? Jan: Yes, basically, yes. Yeah, basically, yes. So, we know even from some commercial tools, once again, that open source is supporting commercial tooling, that they have a bill of material, and they know the licenses, and maybe the use case behind, and then you could use the checklist to automatically kind of half-automated derive the possible obligations you would have in copy and distribute. Exactly this is something you would be able to do with that. So, now, SBOM was a very good keyword, because there's, of course, more open [00:30:00] source projects we are financing. So, I think that the biggest one we are currently running is the Ocelot. So, and this brings us to the topic of bill of material. So, what was the idea with Ocelot is basically that when companies are copying and distributing open source, of course, they would scan the source code and identify the licenses and extract all the license-relevant information, right? So, the thinking with Ocelot is all companies, all people all over the world do this work over and over again. For exactly the same software. I mean, that doesn't make any sense at all. So, we make it open source to share the technical effort. So, why not also sharing the licensing effort? So, what we started doing is providing an open source project. Where you could share ready-to-use license compliance material for commonly used open-source packages, like for Bash, or for glibc, for the Linux kernel, [00:31:00] for BusyBox. So, right now, with a strong focus on embedded Linux, but not necessarily, so it's open-source, it's on GitHub. You can, if you want to provide data, you can just open a pull request. There's a wiki, wiki.osalot.org, so if you want to learn how to contribute, Everything is there. So, this is, yeah, when it comes to the topic of bill of materials. Jonathan: Yeah, I have said for a long time that, so one of the things that we've seen that happens is a company wants to use an open source project, and they don't have their SBOM put together, and so the company will say, "Hey, we need you to write an SBOM for us." And, of course, most open source projects go, "We, that wasn't on our radar, and we don't have someone that's volunteered to do that." And I've said for the longest time, "No, no, no, here's the answer you need to give. Yes, we will be glad to do that for you. Here is our hourly fee." Because the law is beginning to require some of these boxes get checked, there's [00:32:00] work that needs to be done inside the open source project, and there are companies that really, really want to be able to use these open source projects, and it's absolutely natural that there's some, to use a legal term, some consideration that flows back and forth to make that happen. And it kind of sounds like that OSATL is sort of trying to make that flow happen. Is that part of what you're doing? Jan: Yes, that's part of what we are doing. So, we're trying to make that flow happen, or maybe as a starting point to make that flow easier, right? Because exactly the situation you mentioned, I mean, there's some lawyer reviewing a delivery that contains open source, and he needs some information which is not present in the SBOM. So, the other party needs to prepare that material, put it together, and this is exactly the kind of work we want to reuse, right? I mean, in the ideal case, We [00:33:00] would like to be able to push that back to the open-source projects, right? So, they have that data ready, and what happens today is, of course, when you start reviewing such a project, of course, with, I mean, if you have projects like gLibc or the Linux kernel or whatever that have a long history, you will find some issues with improper licensing or something that is unclear, which is just in the nature. So, an Ocelot contributor would just contact the maintainers and address that and clarify that question, so they could even fix it in the next release and make it clear, right? So, but yeah, this is pretty much what we wanted to address with the Ocelot. It's still growing, of course. It's also a huge effort, but I think we are at a very good starting point with that. Jonathan: Yeah. The members of OSATL, are they primarily like the businesses that are using the open source project, or do you have projects themselves [00:34:00] as members? Jan: Well, we have different membership levels. So, for regular membership, it's pretty much the, I would say, the uses of open source that need support on how to properly use open source, how to evaluate, how to, or even they, I mean, they want to be good open source citizens and with us help funding the open source community. But we also have partner memberships. We have partner organizations where we work together with other open source organizations. So, we give each other more visibility or support each other. So, for example, we have, I mean, of course, Linux Foundation is one of our partners, Eclipse Foundation. One of our recent partners is the Erlang Ecosystem Foundation. So, we work together with other open source communities. We also have associate members that could be one-people [00:35:00] companies doing open source. They can also become a member, profit from our benefits, and it's also good for us so we can get in touch with our industrial members, and we could get them in touch with people doing open source and doing a bit of networking, right? But basically, most of the regular members are mostly the users of open source that are interested in having the ecosystem in a good shape. Jonathan: Yeah, absolutely, it makes sense. There's something that I've started seeing in some of the open source and industrial work that I do, it's this weird, and I know why people are doing it, but essentially people are trying to use AI as an escape hatch to get away from the GPL. And one of the first places that I saw this, there was a, it was satire, but it was one of those "haha, only serious" [00:36:00] sort of moments, so it was a website that was offering... Copyright escaped as a service, and so they described this whole thing as like, we will have one AI agent read your code and generate a specification, and then we will have a second AI agent take that specification and re-implement the code, therefore breaking the copyright claim. And of course, you know, it was satire, so it was like, why should those open source engineers get paid for any of their work? You should be able to take this for yourself. But at the same time, you know, it's like, That's seriously gonna be a thing, and we have seen people now start doing this: "Hey, I took your" - in our case, it's protobuffs - "I took your GPL protobuffs and fed it to the AI, and it gave me this not-GPL protobuff that is essentially the same thing, but the AI says that it's clean-roomed." It's like, I don't think this is how this works, guys, but there, you know, people are out there trying to do it. Is that sort of something that is on your radar? Have you guys seen that, maybe, class of problems? Jan: [00:37:00] Of course, it is of on our radar, and of course, I don't like it. I mean, it became kind of a concern, right? Because the thing is that what I always teach people when we When I tell about open source licensing is that the open source ecosystem stays strong as long as we have enough projects with the copy left there, because it forces us to share stuff, right? But it's easy to explain, because when people know how to use open source, they sooner or later learn that they get a lot of People benefit from that sharing, right? So, all the value from the ecosystem comes from the fact that we are all sharing work, right? Now, this movement pretty much endangers that a bit, that people go back and try to get rid of copyleft-based licenses. I don't think that they will survive [00:38:00] long, but that's just my feeling, because at the end of the day, they are just destroying an ecosystem they are also relying on, right? Um, because at the end of the day today, I mean, it's almost impossible to. To come up with a complex product without using any open source at all, right? So, if you come up with strategies that target the open source principle, and the principle of sharing stuff, I mean, at the end of the day, you're bombing your own business, right? So, that's my own opinion on that, but still, it concerns me that this is a topic which is coming up more and more frequently. Jonathan: Mm-hmm. Jan: I don't really have an estimation how long that will take, but still I have the personal feeling that they will fail, but that's just my feeling. Jonathan: I've... Yeah, I've actually asked a few people, and I've not gotten a satisfactory answer yet, but I've asked people, like, is there a GPL4 [00:39:00] in the works that is going to address AI stuff? And so far, the answer is no, and I'm slightly terrified by that, because that definitely needs to be a thing. If you're listening to me out there, Free Software Foundation, get on it. It's a thing we need. I'm honestly not sure though how that will get addressed, and one of the problems is that copyright law has not caught up to AI yet, and so we're all sort of operating together in this gray zone of like, what does it mean to train an AI on copyrighted works? What does it mean for an AI to generate something? You know, I was told by one guy, I happened to sit next to a comic artist on a flight a few months ago, and I'm like, "Hey, do you guys use AI?" He goes, "No, because we can't copyright anything that comes out of the AI." Jan: Exactly. Exactly. Yeah. So, this is one of the topics. I mean, [00:40:00] copyright law only protects what is created by a human, right? So, if it's not created by a human, it's not protectable under copyright law. So, this is one aspect. But I think the continuous gray zone for companies is basically, I mean, On one hand, you can claim what comes out of an AI code generator is not protectable by copyright law, but as you mentioned, it has been trained by something. Do I have any guarantee that there's no significant portion of training material going into the generated code, right? I mean, that's kind of the gray zone. No one can give me an answer these days, and that's all the gray zone you have with AI. Jonathan: Yeah, I remember one of the--and this has been several months, maybe over a year ago even--but Copilot was getting very popular, and people were basically manipulating Copilot to give out large blocks [00:41:00] of code that were completely unchanged. And, you know, that was pretty obviously a copyright problem. It was real hard to argue that it was a transformational work when, you know, you were getting entire source code files just coming out, byte for byte, unchanged. Jan: Yeah, so, I mean, there's, like, with everything, now, finally, we're at the AI topic. So, the Jonathan: Take drink. Jan: I mean, there's several sides of the coin. So, there's all the critical aspects. I mean, we end up with a simple copyright gray zone. So, I mean, this is something we most probably can resolve. The other aspects that became a burden, and this has been most probably discussed before many times, is the burden for open source communities that maintainers are drowning in AI slop. But also, we see the, I mean, [00:42:00] the huge amount of vulnerabilities found by AI and also maintainers drowning in vulnerability reports. I mean, on one hand, it's good that we can find these vulnerabilities. On the other hand, it's, I mean, it's scary to see What can be done, right? So, it's, I mean, at the end of the day, it brings us, as always, back to an ethical topic, right? I mean, it is a powerful tool, and I think all the society needs to learn how to use that tool, right? So, the most important thing is, I mean, at the end of the day, use it with your own intelligence, right? Just use it as support, and as I said, just use it as a tool, right? And then, I think there's also many good things you could get out of it, right? Getting support with code reviews, or writing some tests, so there's many great stuff. So, we did also play around a bit with, I mean, we have so much [00:43:00] data, right? Like, even our legal FAQs and license stuff, so we started playing around with training AI models with that data, and see how accurate it could answer legal questions, right? So, I think there's many interesting topics you could cover there. Unfortunately, all the negative aspects are right now in the media and there's a reason for that. Jonathan: Certainly. I don't want to lose sight of that. You can do incredible things using the AI tooling that is available to people now. It's really, really impressive. Jan: But, I mean, there's also some some positive aspects, right, so... Jonathan: I'm not an AI doomer, but I do see some potential problems that need to be dealt with. I'll put it that way. Jan: And even... Yeah, exactly. Same for me. I'm not an AI doomer. I see potential, but I'm just very critical with it. Jonathan: Sure, sure, that makes sense. What are, and I don't, [00:44:00] I guess the list is public of the members in OSATL, but I'm curious, like, are there any really interesting members that you think you want to talk about for a bit? Is there somebody out there doing something really cool? Jan: Yeah, it is public. So, now it would be really unfair, because it's like more than 100 members, and all of those are very important to us, but maybe this brings us back to the principle of a cooperative, because the cooperative has given by law a strong principle of equality. So, we are forced by law actually to treat all of our members equal. This even comes back to our general assembly. So, there's only a limited number of votes for the members, only regular members. So, if you really buy a share at Dozal, you are allowed to vote. So, the thing is basically that you can have, I mean, we do have [00:45:00] bronze, silver, and gold members. And the only difference is that you have one, two, or three votes at the general assembly. And that's it. There's an upper limit, right? This brings us to this level of equality. And whatever we do, but basically we could do everything, but always having in mind that what we do needs to be in the interest of all of our members. There's a pretty, I would say, strong control system behind cooperatives that you need to prove that you follow that system, right? So, we have a supervisory board, which is formed by representatives of our members. So, they need to grade if we do our work and if we operate in the interest of our members. So, this is quite interesting that this is even forced by law, that we have this principle of equality. We are a community that does useful things together, and we really need to prove to do that. So, from that perspective, all of our members are really important [00:46:00] to us, right? Of course, we do have many interesting products, right, in the portfolio of our members. I mean, I already mentioned that welding lasers, so strong lasers that are used in car manufacturing, right, so we do have woodworking machines that it's not just simple sawing, right, I mean, it's basically put a tree in and get an IKEA kitchen out, right, really complicated, really complicated machines. Jonathan: Right. Jan: Of course, we do have automotive use cases, so we, I mean, it's pretty much everything. It's really cool to see what you can do with open source, right, so it's so many interesting products, and what happens to me always is sometimes we do visits for our members. So, what we do in the community, if someone is interested in, hey, I'm now part of the community, and my company wants to learn more about open source and being [00:47:00] a good open source citizen, could you visit us? Of course, we would do that, right? And we spend the whole day talking about open source licensing and different aspects. So, my favorite part always is I ask for a guided tour to see what the company is really doing, right? So, I mean, to see what you can do with Linux, what you can do with Zephyr. And it's, I mean, that's always the fun part to see what really you could do. And it brings me always back. So, you see these big machines, right? And all these companies can make profit with these machines, even though if they are sharing part of their know-how, but that's not the thing, right? Because, I mean, if you think about the woodworking machines, I mean, the people building a kitchen with that are maybe not interested in the operating system. They are just interested that it just works, right? Jonathan: Mm-hmm. Jan: So for that company, it's no big deal to use open source as the operating [00:48:00] system because that know-how can easily be shared. And if they write a single driver, a simple driver, they can share that with the community. So that doesn't harm their business model at all, right? So it's pretty cool how open source can help building this foundation. And still there's the profit between all these companies doing their products, right? So they can make their personal profit out of the product and they do the contribution to the open source community as well. Jonathan: Yeah. Yeah, absolutely. There's this interesting observation. I think the shorthand way that people say it is that it's not a zero-sum game, but that's kind of the same idea as what you're getting at. You can give away the information, you can You can license the code with an open source license, and you're not inherently losing anything. There's plenty of work to go around, is the way that I've said that as shorthand for running [00:49:00] my businesses. I don't lose work from the fact that there is another company doing the same thing. There's enough for each of us to work at. It kind of does bring a question to mind, though: is your membership primarily in Europe, or do you have folks in the US, is there companies in Asia that's part of it? Jan: So, we have members all over the world. So historically, of course, we have many members in Europe, because we've been founded in southern Germany, and the ten founding members were just from Germany and Austria, basically. But nowadays, I mean, we are all over the world. So, we're all over Europe, we have members in the US, we have members in some Asian countries, so it started going all over the world. And it's also, if you look at the company size, I mean, we have big corporates, and we have one engineer companies within our members. Jonathan: Sure. Yeah. [00:50:00] One of the fun things that hopefully you guys get to do is to be able to connect those two sometimes. Like, here's this huge multinational company, and they need this one particular thing. Well, we happen to know the guy. We got a guy for that. Jan: Yeah, exactly. Exactly. We do that. We do encourage that. So, of course, we do run webinars and everything, but what we do once a year is that we want to exactly do that, want to bring together that one-man show and the representatives of this big corporate and making them talk together and discuss ideas. So, we call this Networking Day. Jonathan: Mm. Jan: This is our annual members conference, where we call all of our members to the beautiful city of Heidelberg in Germany, and this is a pure in-person event. So, there's a reason why we do not make this online, or why we do not make this hybrid. So, you can download the presentation afterwards, right? Jonathan: Mm. Mm-hmm. Jan: But we want to have some motivation that people come there, and these people talk together. [00:51:00] So, what we do is we do one day with presentations, so we get speakers from different open-source communities. We prepare interesting topics just to get the thing started, right? So, people have topics to discuss about in the breaks or during the evening. So, one big topic in Europe these days is the Cyber Resilience Act, which is the new cybersecurity regulatory, which partly got now a few days ago in place. Jonathan: Mm-hmm. Jan: Now, you need to report. actively used vulnerabilities in your product. Jonathan: Mm-hmm. All right, aren't we all thrilled about that? Jan: It will be fully in place starting next year in September. So, this is one of the... I mean, it's a good movement, definitely, but... Jonathan: It is. It's just, I wish they had talked more with real people in the industry before they wrote it. This is something that almost every big European tech law, it's like, "Thanks, now we all get cookie banners [00:52:00] everywhere. Thanks, guys." Jan: Exactly, yeah. So, I mean, also there's pros and cons about everything. What I need to say on the positive side here is that the European Commission gave quite a few chances for submitting feedback. So, this is also a role. So, we collected the feedback from our members and submitted that to the European Commission because, I mean, if a single company submits feedback, it's a different story than an organization with 100 members submits feedback in the name of 100 companies. Jonathan: Mm-hmm. Jan: So, I think that this is to put on the positive side that there was some possibility. So, at least they were open. But this is a very good, I mean, you can get into deep discussions on that. Jonathan: Right. Jan: So, how come? So, I think that's a good question. [00:53:00] Thank you. And just imagine that one man engineer doing dinner together with the representatives of the big corporates, so that was, I mean, like over a glass of wine, you do like the dessert or something, and this year... Jonathan: I was going to make the joke that being in Germany, there had to be beer or alcohol involved. Jan: Of course, there was the possibility to have some beers as well, some craft beers, of course. Sometimes makes discussions a bit easier, right? Jonathan: Yes, that stereotype exists for a reason. Jan: Yeah, there's a reason for that. And this year, we actually did an escape game together. Jonathan: Oh, fun. Jan: So, yeah, solving puzzles together. So, I mean, this is the kind of networking we try to do. But once again, it's like, yeah, it's a community. So, you want to have people together and exchange ideas. That's how open source works, right? So, we exchange [00:54:00] know-how and we discuss stuff. Jonathan: Yeah, absolutely. What does the process look like if someone, let's say a company, but also an open-source project, if these two different people want to get involved with OSATL, what does that process look like? Jan: It depends a bit on what you do together, right? I mean, let's start with the company, right? So, if a company wants to get involved and says, okay, we want to support that community, or we want to profit from that community, it's basically quite easy. You become a member, right? So, becoming a member basically means one time you buy a share. So, you become one of the owners of Asada, right? And based on the number of shares you're holding, you would pay annual service fee, That is pretty much how we finance our work, right? How we put the budget together, that we spend together. So, [00:55:00] this is pretty much how it works for a company. So, you would apply for a membership, buy a share, and you support our work by an annual service fee, basically. For open source projects, for academics, for partner organizations, it's basically something cost-neutral, right? So, we want to support each other with the funding coming from the industry. So, a typical collaboration with an open source project could be that we identify that our members are using a specific technology, like preempt RT. So, we say, okay, this is pretty important, so we find a way to fund that project, right? So, we try to become a sponsor, put some money together, and try to finance the maintainers. The same for our Ocelot project. So, even though it was founded by us, we have an independent maintainer, so we are funding that maintainer. We are holding the trademarks, holding the domains, running the server [00:56:00] infrastructure to make maintainers' life easier. Sometimes, Projects independently to membership, we call that mixed funded projects. So, whenever we identify interesting technology, we ask our members, hey, do you find this of interest? And are there any features you want to have implemented? And then we start collecting sponsors from all the industry. What we would do is do the project management and collect the budget, right? And then we would hire someone to get stuff implemented and push it mainline. So, the requirement whenever we hire someone doing something is always that it has to be released under an open source license. So, this is like of the corporations we do with open source projects. Sometimes we participate in conferences, or for maintainers and key members of open source projects, we grant them free access to the assets that are usually only accessible to our members to give some support like licensing [00:57:00] or legal support. I mean, open source projects usually don't have budget for that. So, this is sometimes where we can jump in and help them with getting them access to the material we have. Jonathan: Mm-hmm. Yeah. Interesting. When is the in-person meetup? Jan: That's always in tune, so it's unfortunately already over for this year, so it's always in tune, so if there's anyone heavily interested, of course, we always invite some non-members to get in touch with our community, just get in touch with me, and so we're always glad to have people there to discuss about open source. Jonathan: In June, okay. Yeah, yeah, this is very interesting. So, is OSOTL also like a fiscal host for some of these projects? I think that's what you described, at least for OSOLOD. Jan: Yes, so it is for Ocelot. [00:58:00] So, being a fiscal host is just happening for Ocelot. So, we are sponsors for a couple of more. I mean, we're a very small organization, right? So, I mean, there's some limits we do for what we can do. We try to operate as effective as possible, so we're the fiscal host for the Ocelot project. So, we've been supporting a project which is called Open 62541 for quite some time with significant funding. It's an OPC UA library open sourced. And, I mean, not surprisingly, we are historically for many years a premium sponsor for the PreemptRT project. We are putting significant budget into the preempt RT development. So, for these projects, we are budget-wise involved, but we're not the fiscal hosts, right? So, we're just... But also, we are in the name of our members, we are sponsoring these projects, right? So, with all the member contributions, [00:59:00] we sponsor the key technologies that are of importance for our members. Jonathan: Understood. Sure. Yeah, no, absolutely. That Yeah, absolutely. That makes a lot of sense. You see industrial automation being sort of the key place that you're going to continue working in? Jan: It is one of the places we're going to continue working in. So, it's one of the traditional places, but, of course, we can see that there's way more industries coming to open source, and where we can definitely provide useful support. I mean, we even have, like, companies doing online platforms, which are far away from embedded technologies, but they want to have some legal support and some licensing support for their online platforms. As I mentioned, we have automotive. We see a tendency more for medical devices being run with open source. They also have the topic with safety certifications, for example. So, these are the big upcoming topics. Jonathan: Mm-hmm. Jan: So, once again, traditionally, it's a lot of automation, but at the end of [01:00:00] the day, our community can grow into all directions, because we can provide to… Basically, if you are using open source in a product… I'm quite sure that you can be part of our community, and you would be beneficial of our community, because… Every single member is important for us. It helps us to do great things, right? And I always tell our members: don't be shy asking questions, right? Because with everything we do, with every project we fund, with every question we answer, we get some useful knowledge for all of the other members. Jonathan: Yeah, you mentioned medical devices, and it brought to mind a memory that the 6502 is still being produced. One of the reasons being is that it gets used, because it's so well-known, and it's so simple, it gets used in medical devices. Jan: Yeah, yeah, yeah, yeah. Jonathan: And so, like, if you get a pacemaker implanted, there is a reasonable chance that it is a 6502 that [01:01:00] makes that thing work. Jan: Yep. Yes. Jonathan: Crazy. Alright, as you sort of rub your crystal ball and think about the next five, say, five years of open source, what's on your radar? What are you—sorry to mix my metaphors there—but what do you foresee Jan: Yeah, so, um, We already touched a bit of what the Chrysler ball might tell, actually, so my Chrysler ball tells that there will be a lot of movement towards the safety direction, there will be more efforts, and maybe some more ideas, and using not just Linux, just open source in safety critical areas. So, this is something, if we look five years in the future, I think the picture will be way clearer and maybe we see the first solutions coming up there. I mean, even in the Linux side, quite some efforts. Look at the Zephyr project [01:02:00] that is strongly going there, so this is definitely maybe a pretty clear forecast. I think that there will be maybe more efforts, we could see more Linux in space, because this is also a very interesting use case for open source, so I would predict that. Obviously, we've seen Linux already in space before. Jonathan: We've seen Linux on Mars! Jan: Yeah, we've seen Linux on Mars, and my prediction is that we will see more of that in the future, so that's actually the geeky stuff I'm looking for, seeing Linux in space, seeing more open source in space. Jonathan: Yeah, absolutely. The... Oh, absolutely. Yeah, the best story that I've ever heard, and it's hilarious, is that apparently several years ago on the International Space Station they started running Linux on all the laptops because they couldn't keep them virus-free otherwise. They kept getting [01:03:00] malware on their laptops. Yeah, I love that one. It was hilarious. Jan: I mean, this is maybe a place where you don't want to have a virus on your laptop, because you don't have much options in going home. There's only one option, so if that doesn't work correctly, you're in trouble. Jonathan: Indeed. For an attacker, lateral movement is going to be really interesting from there, too. So, you have a note in the rundown here that there are some parallels between open source and cycling. And I'm very curious what you have in mind. It ties into your shirt: you must be a cycling enthusiast. Jan: Yes, exactly. I am, I am. So, I think I'm with cycling as long as I'm with open source. So, I mean, the reason is, basically, my theory is, if you're interested, so if you're enthusiastic with something, which is your profession, you need something with the same passion in your spare time. So, I mean, it's somehow [01:04:00] equal. Yeah, there's actually quite interesting similarities you could basically see in people doing cycling or other endurance sports. Maybe the clearest parallel is that the passion people bring into stuff, right? So, for all the people I see working in open source, they bring so much passion in it, right? Look at all the maintainers that are not properly funded and still working day and night to getting their project running. Doing endurance sports, you, I mean, you have somehow that passion, because, I mean, if you, like, do a long distance ride, you always have the option to stop, right? It's just your will that keeps you running. So, I think that level of passion is pretty much the same. But also, what is interesting, what is maybe not obvious, when you're following cycling races, so if you're not into cycling, you most probably just see the individual cyclists doing their route, right? [01:05:00] At the end of the day, it's important to know that cycling, specifically with the big races, it's a team sport, right? So, all the big leaders at Tour de France or Giro d'Italia, they could not win without all of their supporters, right? So, there's always a clear strategy. They have supporters that make their way, that go back to the team car, get them water and stuff to save some energy for the sprinters to get them into the right position with the correct timing. Right? And so, it's also community work. So, that's a bit the parallel I see as well. So, even if you maybe have a few single persons that are in the public, there's so many people behind working on the whole strategy and making things good, and it's the same for open source. You have the leaders, you have the strategy, but you have all the contributors [01:06:00] doing that work, and together you reach the goal. It's the team sports aspect that brings me thinking, comparing cycling and open source. Jonathan: Mm-hmm. Yeah, absolutely. I like it. It's a good metaphor. Is there anything that we didn't talk about? We basically hit the hour mark. Is there anything that we didn't get to that you wanted to let folks know about? Jan: time flies when you talk about open source right i think we talked about love interesting things Jonathan: Absolutely. Jan: at least i we talked about the most important things i i wanted to cover today i wanted to bring to the to the audience i think what was important for me is also that there's so many ways that you could work together right not just the technical stuff there's even on legal topics you can work together in a community and you may can make things better together so as a community you're always stronger and this is pretty much what we like to do at osada and what we love to do and what we will continue in future We're [01:07:00] trying to improve the ecosystem for all of us. Jonathan: Yeah, absolutely. I do have some final questions that I ask everybody, and we've added a third recently. So, the two classic questions are: what's your favorite scripting language and text editor? Once you answer those, then I'll hit you with my third. Jan: So, the favorite scripting language, so I am a very old-school bash programmer. I mostly do all of the stuff I need, just as a simple bash script. Jonathan: Very cool. There you go, Dim and Bash. All right, now the third question, the new one: Jan: The favorite editor is Vim. Jonathan: if you're gonna, if you're gonna vibe code something, which AI agent do you reach for? Jan: Whoa. Well, that's a tough one, but I would say I would go with Glute. Jonathan: Yep, that seems [01:08:00] to be the answer right now. Things are still very in flux, though. We are very early, I feel, in this revolution still, so who knows where we'll end up here in another six months. Jan: very early. So, if you play around, some model is doing very good in doing graphics, some are better in coding. So, let's see where that story takes us. Jonathan: I've also sort of learned that it depends upon which language you're working in, like Gemini tends to be better doing Android apps. Surprise, surprise. But, you know, there's some of those things that, you know, it varies. It's based on the task. Jan, thanks so much for being here. It has been a blast to get to talk to you, talk about OSATL and all the things going on. Yeah, appreciate it. All right. I do want to let folks know we're taking a week off next week, because I'm going to be in Europe at the Things Conference, and then we're going to come back in two weeks. We don't have a guest yet, so if you want to be on the show or you know somebody that should, shoot us an email, [email protected], [01:09:00] and we'll get them scheduled up, but I'm sure we'll get the slots filled, and we've got next month opened up, and so let us know, and we'll get you scheduled. Other than that, I just want to say thank you to everybody that joins us, whether you watch us live or on the download, video or audio, we appreciate you being here, and we'll be back in two weeks on Floss Weekly.
• 
  Episode 882 - Better Together 16.09.2026 1t 9min

  This week Jonathan chats with Jan Altenberg of the Open Source Automation Development Lab, a cooperative of nerds that care about Linux and Open Source! Jan and OSADL helped push Real Time Linux across the finish line, are looking at safety critical hardening, and general cooperating to make Open Source and Industry work better together. You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 881 transcript 09.09.2026

  FLOSS-881 Jonathan: [00:00:00] It's time for Floss Weekly. That's the show about free Libre and open-source software. I'm your host, Jonathan Bennett, and today we're updating. In fact, this is a topic that was not on my radar, but one of our listeners, thank you to everyone out there, said, "Hey, you guys really need to talk about Update CLI." And I said, "I don't know what that is, but we'll send the email." And so we emailed Olivier Vernon, who is our guest for today, and we said, "Hey, we would love to talk to you about this." And he said, "Sure." And so we've got him on the show. Olivier is the guy behind Update CLI, and unfortunately, I just got logged out of my email address that has his bio. Let's see if I can get back into that. We're doing it live, as they say. But let's go ahead and bring him on, and then I'll get back to the bio. Olivier, welcome to the show. Glad to have you here. Olivier: Thanks for having me on the show here. Yeah, I was really [00:01:00] glad to hear that you would contact me. So, I've been following what you did here for a few years, so that's really nice to hear. For the BIOS, quickly to talk about me, so I'm working, so I've been working on open source software for the past 10 years, both professionally and on the side as well. Jonathan: Mm-hmm. Olivier: I've been contributing to open source conferences and so on, so the biggest one is probably the FOSDEM for those people who know what FOSDEM is about. And so, the reason why I'm Jonathan: Oh, okay. Mm-hmm. Olivier: here today is because I was interested to talk about that side project that I've been maintaining for five years, which is all about updating your Git repositories, and so, yeah, that's why I'm here. I don't know if, yeah, the project is named Update CLI, so for people who are not familiar with this, it's kind of interesting because there is no, like, similar project, but it's a CLI. I mean, you can guess based on the name. That allows you to automatically update your Git repositories based on [00:02:00] manifest that tells you how to update those Git repositories. So the purpose is really to help you on the maintenance of your projects, and so you stop monitoring third information coming from the web. You just define, okay, you have a Git repository. What would be the latest information, URL, version, whatever? So, people usually, when they think about Update.cli, think about Renovate or Dependabot, which is kind of the closest, I would say, project for what it does, and otherwise, it's just bash scripts, Python scripts, and so on, so it's kind of a seed between the two worlds. Jonathan: It's amazing to me to see, like, how much of the world is run by nothing more than a pile of Bash scripts. Olivier: Yeah, exactly. And it's like, yeah, Bash is really awesome, until you reach some limitation. Jonathan: Oh yeah, absolutely. there are elements of it that drive me crazy um and in fact there is a uh several years ago we interviewed another project called amber that they were like [00:03:00] we love bash except for this set of things that's terrible about it so we made our own scripting language that compiles down to bash code it's like well that's cool um okay so what's what is this what does this problem look like like at what point does someone reach for update cli what where in like an open source project or just you know your your homebrew whatever it is you're doing what what does the problem look like Olivier: So, the problem was, back then, like five years ago, when I started working on a project, I was working at the time on the Jenkins project. We had a lot of Git repositories, Java libraries, and so on, and one of the projects which is still running today is the infrastructure where we run the Jenkins, where we test Jenkins, and so on, and so all that infrastructure is managed by infrastructure as code, like Puppet, Kubernetes, and so on, and one of the Jonathan: Mm-hmm. Olivier: problems I was facing back then was, quite regularly, someone would build a new application, and then I would forget to update that application on [00:04:00] the Puppet infrastructure, or Kubernetes Jonathan: Gotcha. Mm-hmm. Olivier: infrastructure, and so on, and the thing is, back then, I was already using tools like Dependabot, or Renovate, but the challenge is, those tools can understand a file that is well-structured. Like, if you think about, I don't know what language you're most familiar with, but let's say you think about GitHub. GoLang has those GoMod files where you specify the list of dependencies. That's a file which is pretty easy to understand because you can have a tool that runs automatically there and bump those versions. That's where Renovate, for example, shines. But the problem is, quite often, you have those bash scripts, for example, where you define a version, a variable with version set to something, and it's just impossible to automatically update that bash script because you have some meanings. I mean, you know the purpose of the bash script because you wrote it, and now you need to maintain it over time, and the problem is, quite often, you have a lot of pieces of information hidden [00:05:00] everywhere and you need to coordinate, and so, instead of having, like, Dependabot to bump one PR, what I do is I say, okay, I'm monitoring this application version. That's exactly how it started. I'm monitoring this application version, let's say the account app, which is a custom application, and now, based on that information, what I want to do is I want to update my Puppet infrastructure, but only if there is a Docker image matching this tag. And I also want to open one PR for the Kubernetes manifest at one time. So, I don't want to have, like, three different PRs. I just want to have, like, one PR that I shoot at once. And so, that's how everything started. So, the idea was really to say, to define what my update strategy looks like is I say, here is information that I'm looking for, like sources, and then you have, like, different kind of source, but the source is where the information is coming from. That could be a version, a random string somewhere in the internet, whatever. That's the information you need to reuse. And then you have the condition that needs to pass, like, do we have a Docker image? Do we have a file, a Docker file with [00:06:00] that information? Do we have README, whatever it is? And then do we have a target? The target is what you want to update. And so, once you have this manifest that tells you, here is what my ideal update scenario is, then each time you run update scenario, which is a command line tool, it's idempotent. The result is always the same. You run it from a CI, for example. And then every time it runs, it will fetch all the latest information, verify that your Git repositories are up to date, and if it's not, then you just open a PR, and then you can run it. Review this, and then the lifecycle of your project keeps working. So, the challenge is really about how to maintain a lot of Git repositories without having to remind yourself to go fetch all those information. And when you think about this problem, it's something that is a really huge problem. Jonathan: Mm-hmm. Oh, sure. Yeah, because a lot of open-source projects, they do, they define, you know, their various libraries and their various dependencies inside [00:07:00] of their build scripts or, you know, various places. Sure, yeah, it's something that I deal with in some of my projects as well. I think we use Renovate for this, which works most of the time. Most of the time, it works 99% of the time, right? Yeah, so what did, did Renovate not, no, you said Renovated existed. What was the, what was the big differentiator? What did, like, Renovate lack that you really wanted to solve? Olivier: So for me, the biggest limitation was having the flexibility to really say, first, to run the tool whenever I need it. That's one thing. And the second one, to really be able to say, here is a complex pipeline. And when I say complex pipeline, I can, for example, provide another example. Jonathan: Mm-hmm. Olivier: When I build the update CLI, I create a new release of update CLI. Let's say version 0.1.0. The way it works is I use a super cool project, which is GoReleaser. GoReleaser is responsible to create the packages and [00:08:00] so on to publish that information somewhere. And then at the end of this, I use GitHub Action that sends notification to other Git repositories. And those other Git repositories, for example, one, for example, contains the handshots, if you're familiar with Kubernetes. One contains the documentation, if you're familiar with running Yugo and your websites. And so the way it works is on those repositories, I trigger update CLI. Based on a hook. And then update CLI will automatically retrieve the latest version of my application, update the documentation, update the Docker image, update the handshot, literally update everything, and then start other processes. And so it just allows you to run, like, to monitor information, run shell commands, and in a specific order that really allows you to just think about whatever ecosystem you want to work on. Jonathan: I'm curious: have you thought about or run into the the git tag problem? And this is sort of a security issue [00:09:00] that we're seeing in a lot of different cases. The idea there being that If, you know, if someone malicious takes over a remote repository, they can delete and then replace a git tag really, really easily, and so I've seen people start to recommend that instead of, you know, include this git tag as your dependency, you should instead say include this git hash as your dependency. And I get it on, like, the security side, that makes a lot of sense. It seems really cumbersome, though, to work with. Is that something that you've thought about as part of UpdateCLI? Is there a solution there, maybe? Olivier: So, that's definitely something I thought about many times, and we experimented different things. So, before I go into that specific, so we can better understand this, Update.cli is like plugin-based. Like, you can interact with a lot of different plugins, and so, for example, you have a plugin for git commits, like you mentioned, and then you can say "please monitor the latest git commit", or you can [00:10:00] say, like, "please monitor the latest docker image tag", for example. And specifically for the git commit, but that's the same thing that would happen, for example, to any OCR artifacts, like docker images. Usually, it's recommended to use Digest, but the real issue with Digest is, as a human, Digest are just impossible to review. Like, you receive a PR, you have, like, a bunch of Digests, and then you just merge blindly, and in the end, from a security perspective, you're not really reviewing anything anymore, because you're just, like, trusting the automation, which is something you should never do, anyway. And that's why we try to use both. So, for example, usually I like to specify the digest, because at least I'm sure what version I'm using, or what piece of information I'm using. I also like to use the commands that specify "this digest corresponds to this information". So, for example, in GitHub Action, for example, you can specify both information, and then me, as a human, I can just quickly check if the version [00:11:00] really matches the commits. And then you have tools that also allow you to verify these things. Like, I don't know if you're familiar with the project Zsmore, if you're familiar with GitHub Action. So, Zsmore is a kind of a linter that you could use for GitHub Actions, and then it allows you to verify if your GitHub Actions are well configured. And one of the things that it checks is: does the version you specified in the command correspond to the git commit? To be sure that you don't have a malicious actor that just tells you that it's a version 1.0. that just tells you that it's a version 1.0. And in the end, it just pushes a random commit. So, yeah, the digest is something very tricky. I usually recommend to use digest, but also try to provide as much information as I can to allow humans to verify this, because in the end, otherwise, it's meaningless. Jonathan: Yeah, it all seems like the the best solution here is going to be to add some metadata, maybe as a comment, or, you know, it's hard to say exactly where, it's going to depend upon the language you're working with. But, you know, your metadata would be, here's the tag, [00:12:00] here's the version we think it is, and then the real thing that you go out and you grab is the hash itself. That way, you know it's not going to change if somebody doesn't update, but then you have also got that tag information, so where you can watch and, you know, "Hey, you're running v1.1.1 based on the metadata, v1.1.2 is now available, do you want to update your corresponding hash to match?" Yeah, that seems like the best of both worlds, and if you're running a plug-in system, probably that's something that could be managed fairly easily. Olivier: Yeah, yeah. And there is another concept which has become quite popular over the last years, I don't know if you heard about this, it's the idea of cooldown dependencies. So, the idea is, instead of always updating to the latest version, you should specify your small threshold, like one week, three days, it depends on the ecosystem and how often you want to be up to date. But the idea behind the cooldown dependencies is, if a malicious actor gets access to one of your dependencies, it's going to take a few days, maybe a few weeks, for the community to realize that something [00:13:00] wrong is happening here. And so, instead of updating to the latest version as soon as possible, then you need to get the, maybe wait one week. This also comes with limitations, and that's one of the things that is very interesting to work on in this field is, you need to fit to every ecosystem. Every ecosystem comes with different APIs, they expose different metadata, they sort information in a different way, which means that having a tool like Dependabot that just works in most of the case, the reality does not Jonathan: Mm-hmm. Olivier: because it will probably catch only a few of the scenarios. And sometimes you need to be able to specify and say like, okay, I need to update at this frequency with all those information altogether, opening one PR. I also know that I would like to add additional information because for this specific scenario, I'm getting like a changelog coming from this endpoint, which is not a standard behavior, but that's everything. Jonathan: like, node NPM packages [00:14:00] update very differently than your C or C++ system packages, Olivier: That's all we have, basically. Jonathan: and it's useful to have one tool that sort of speaks both of those languages. Olivier: Yes, and the thing is, and that's a tricky That's another tricky problem here is when you think about one of the early design decisions I made with Update.client because that's something I really want to have is I wanted to have a standalone tool. So I choose Golang because I love Golang ecosystem. I wanted to have a standalone tool. I could run the same command from my machine or from CI. That's my learning from managing CI infrastructure is that you should be able to run the same commands from both locations. The result should always be the same. I didn't want to maintain a state. So I don't have state file like Terraform, for example, because it's a constraint that I put on myself since the beginning. Sometimes that makes things a bit more difficult. But at least if you run the same command as me, the result will be exactly the same. You will open exactly the same PR or [00:15:00] you would open update the same PR. But the thing is, yes, when I'm talking about ecosystem, every ecosystem comes with its own tooling as well, and the purpose is not to reinvent the wheel. And so, the approach that we made is, UpdateCLI is kind of a framework, like you can use, you can write manifest to update your scenario, but sometimes we just shell out to a command, because we don't want to reimplement, like, for example, YARN, or NPM, or C++ tools. So, for example, the way it works is, you can say, here is just a file, like, for example, NPM, it's a JSON file, you update the JSON file, and then at the end, next step is, if the JSON file was modified, then please run these commands, and if the final package of JSON was modified, then open a PR. And so, you can have both of the two scenarios where you can, I mean, you can really have a lot of flexibility in the way you design your pipelines. Jonathan: I know this is tool [00:16:00] abuse, but it comes to mind: can I use Update CLI to keep a system up-to-date through, like, RPM packages? Olivier: That's a good question. I want to say yes and no. Jonathan: Yes, you can, but you shouldn't, is probably the answer. Olivier: Yeah, exactly. The challenge is, what should be the latest, how do you monitor the latest version? Like, for example, for RPMs, you need a database that you need to query. That's a question I've been asked multiple times, because people love immutable Docker images, and they would like to be able to specify and to kind of re-implement the NixOS, where you can just list the dependencies and so on, I know people using FDCLI in that context with NixOS, but yeah, it also comes with limitations. Jonathan: Yeah, so you wrote it in Go, right? What was the thought process behind Go, as opposed to, I don't know, C++, Python, Rust, Bash, any of [00:17:00] the other options? Olivier: The thing is, because I'm working, I've been working in infrastructure for a long time. I've been working with communities for a long time. So Go is kind of the default tools we use in cloud environments. And you also have a lot of libraries. So it's super easy to use. One of the things that I love with Go is you have a lot of conventions. So we all write the same codes. So if I look at someone else's code base, it's pretty easy to read. Today with AI, it works quite well as well because the code it can generate is kind of the standard generated Go codes. So that was easy. And the other thing that I really love is it can generate binaries. I mean, statically linked binaries, but binaries for every distributions. That means that when I work, I mean, you can use a baseline with Mac OS, Linux, Windows. Jonathan: cool. Do you have an idea of how many people are out there using UpdateCLI? Olivier: It just works. Jonathan: [00:18:00] It's very hard to answer most of the time, yeah. Olivier: It's very hard to tell, because the thing is, Update.cli is designed to run in a CI, so I can definitely give you numbers, because those are public. You can see, for example, from So, I think it depends what numbers you look at, but I think I have basic telemetry for Update.CLI, so, for example, when Update.CLI runs, it checks what's the latest version, and for that, it queries a JSON file, so I know, like, how many times Update.CLI created a JSON file, and it's something like probably 600 executions per hour. So, somewhere, somewhere in the world. So, that's the current ratio. It's like, we have six, seven updates in execution every minute. No, yeah, that's probably the best matrix that I have. Otherwise, the number of downloads is really huge. But once again, because it's executed in CI, how [00:19:00] many of them is really human? Like, I think, I like to use, for example, matrix for Darwin, like for macOS, because macOS, You never use Mac OS and CI probably. And so I have something like maybe... Two, three hundred downloads for every version, but for Linux and Windows, it's a lot more, but once again, it's difficult to tell. Jonathan: Yeah, because I mean, we all start with the same CI image on GitHub, and then you install your tools inside of it. And so, you know, every time that runs, it's a new download. Yeah, that's a fair point. When people run this, how often do they usually run it? Is it like once an hour job? Is it once a day job? Olivier: Once again, I heard different thresholds. The reason for that is because people have been using Update CLI for many different kinds of use cases. So, I know, for example, I saw a blog post from someone from Elastic [00:20:00] who is using it to automate documentation updates, because you can use it, for example, to update all your URLs, version, installation page. I know people using it in the context of development, where they just want to meet on go-lang dependencies, or like Java, whatever. I know a lot of people using that in the GitHub space, where you can update Argo, Cd, Flux, and so all those different kind of people have different usage. And usually, I would identify two kind of updates. You have the updates that you monitor that are part of your core business, like, for example, if you build a custom application for your company, you probably want to get the latest version in your infrastructure as soon as possible. On the other side, if you monitor, let's say, the latest Python version, maybe that's something you can wait once a month or every two months. So, usually, depending on the team, they have different habits, and they try different things, because if you update everything very regularly, like every hours, it just creates a lot of noise, and then you stop working on the main feature of your [00:21:00] application. If you update them, like, every six months, maybe you have, like, technicaldebt.com, and then it takes you a lot of time to update. So, it's really, I think, it really depends on the project, the maturity of the project, how much tests you can rely on, and then the team experiment with that. Like, sometimes it's one day, and then they realize it will change that every two weeks, every month. I mean, even for me, some projects, I did that every day, and others, I just wait a few weeks or months. Jonathan: You've got a note here in the prep document that this gets used by companies that are not software vendors, like banks and hospitals and governments. I'm very curious: what does that look like? Olivier: The usage is pretty similar to any other organization. For example, I met a person, and then the first time we did a presentation about how UpdateCLI is used in a hospital in Paris, it was the first time for me to go to a conference where someone was talking about the project, and it was not me, which was really awesome. Jonathan: Yes. Olivier: [00:22:00] And then we did another presentation together. A lot of organizations rely on open-source software. A lot of organizations maintain used Git repositories. A lot of organizations develop software, and so UpdateCLI is kind of the tools that you use in your project to automate and to simplify the maintenance of your project. So, I know companies in the banking center, I know companies in every different kind of industries, which I find also super cool. Jonathan: So, essentially, they're just watching for, you know, they've got their list of open source projects that they use on, I don't know, end user desktops to run the website, all these different things, and they just, they fill it in, and they've got a list inside update CLI, and run it once a week or whenever, and, oh, look, there's a new version of, you know, all the different things that they use, whether it's, I mean, I could I could think about this, and it's like, you know, if a bank is using a certain [00:23:00] backup program, let's say, or, I don't know, I mean, there's all kinds of options, but like, oh, hey, there's an update for that, you get a ping, we should probably go and deploy that, it's probably time. Olivier: Yeah, and sometimes they use it this way, but they sometimes also use it to monitor internal products, like when you have a pretty big organization where they develop a lot of custom applications. I like, for example, I find it quite useful. I built a UI on top of the data. It's something experimental that I would like to get rid of, but it's helped me to visualize, for example, the state of my different projects. So, Update CLI, for example, I was telling you the example of the release automation. Like, I can go to my UI and tell me, like, okay, what are the pipelines where the label release Update CLI, and then I can see that, okay, I already update, and the PRs are merged on the documentation repository, but I still have some pending PRs on [00:24:00] the Kubernetes repository, and I still have some pending PRs on the GitHub Action repository. Because of this, it will automatically monitor my application. I know that when I do release a version of my application, it will automatically monitor my application. I know that when I do release a version of my application, it will automatically monitor my application. I know that when I do release a version of my application, it will automatically monitor my application. I know that when I do release a version of my application, it will automatically monitor my application, it will automatically monitor my application. So, when I do release of Update CLI, I would just quickly have a look to the dashboard to see, okay, am I all good with the updates? Is this update already updated everywhere, or am I behind with this work? Jonathan: Is there is there any support and update CLI for the idea of a staged rollout? Olivier: For what? For runouts? Jonathan: Staged rollout? Like, I've got a new version, but I only want 20% of my users to use it right now, and we'll wait a couple of days. Olivier: No, no, no, no, no. It's not something that makes sense. I don't think it would make sense in the case of Update.cli, because you will probably rely on the tools that you use. Like, for example, Argo CD can do rollouts. And so, Update.cli can update your Argo CD configuration, and then Argo CD is responsible for this. Update.cli is really focused on "give me the right information", which is already a big challenge, because different ecosystems come [00:25:00] with different versioning, and sometimes the same piece of information can mean different things. Like, for example, semantic versioning is 1.2.3, but you could also have a date where you just have, like, one digit, a dot, digit, and another. So, that could be, like, a huge date, or that could be something totally different. So, like, being able to monitor this information is, like, the first challenge. The second one is when you want to reuse the same information across different ecosystems. Like, in one PR, you want to update maybe, like, Infrastructure Manifest, Dev Manifest, and Documentation. Maybe some of the tools don't support that. Maybe you need to replace that by underscore, and so on. And so, that's how you end up in a situation where you need to first identify right information, and then you need to propagate that information. And there is just Even things like what's the meaning of the latest information, like "latest" means nothing, like for you, "latest" could be the latest version from a semantic versioning point of view. That could be the latest from a date perspective, and just the notion of knowing what's the latest is super [00:26:00] difficult, and even within the same ecosystem. Like if, for example, you think about what's the latest Docker image for Update.cli, if the same image is hosted on Docker Hub or on GRC or Natalio, the answer will be different, because those two Docker registries implement the API in a different way. One will be sorted by date, the other will be sorted by a lexicographical order. And so that means that the latest information is really, really difficult to get. And so that's why Update.cli focuses on what's the latest information you want to have, is your current repository correctly up-to-date based on the definition you specify, And finally, can we provide you the right information so you can make the best decision about merging the PR? I mean, you could automatically push to the main branch, for example. I saw horror stories around this in the past, so that's not something [00:27:00] that I would recommend. But yeah, those are things you could do, definitely. You can sign commits, I mean, yeah. Jonathan: Yeah, I was humored thinking about this, the semantic versioning. Semantic versioning is just a recommendation, and even the projects that claim to use semantic versioning, they'll sometimes break stuff in the minor updates. It's like, no, no, no, that's not how this is supposed to work, but it's the world we live in. So yeah, I would at least recommend making sure that your CI run goes green before you automatically push that bump. Olivier: Yeah, and what is fun when you maintain a project over time? Like, for example, we've been using Update Selling and Jenkins for five years to manage infrastructure. What is fun is you end up trusting some vendors, because you know that they respect semantic versioning, and there are some vendors you never trust them, and you always review carefully why they change or why they bump something, because it can always break. And that's one thing, and the other thing as well is sometimes some [00:28:00] dependencies are more core to your project than others, like... Do you want to bump the notification mechanism? If it's not critical for you, then just bump it, and then you will see if something breaks. But if it's a core functionality of your application, then you may want to spend more time to verify. Jonathan: Yeah, yeah, absolutely. Do people actually just let update CLI run hands-off? Olivier: Yes. Like, usually you put that in a CI, Jonathan: Well, no, I know that, but I mean... You get to the point where it creates the pull request, and then a human goes and reviews the pull request. How many people run it and just YOLO the pull requests? Like, I don't even have to review it, just make the pull request and then merge it immediately. Olivier: That one is something I don't really have metrics. Obviously, I don't really have a lot of visibility. Usually, what I see is people use auto-merge feature. Like, for example, on GitHub, they will automatically merge the PR if the [00:29:00] tests are passing. Jonathan: Mm-hmm. Olivier: I know for sure that that's something kind of fun. I already had people who just want to commit directly to main. So, they've been asking me for that kind of features in the past. It's not something that I recommend. That's something I Try to avoid, but once again, I don't have that level of telemetry, so I mean, I don't have telemetry on the project, so... Jonathan: That's a hard decision to make on your part, because it's like, "This is such a bad idea. Don't do this. I have people that want this feature. I hate the idea of this feature. Like, what do you do? Are you going to give it to them, or are you going to stand your ground and say, 'No! Don't go directly to Main or Master!'" Olivier: No, I usually, for me, the project is like, I don't try to judge how people are using the tool. Like the purpose is to have something flexible enough so it can fit your pipelines. So if it's something, so the way I approach that kind of decision is, is it like, is it something that costs me to implement? If it's no, then [00:30:00] I don't care. I mean, you can use it. You are responsible for what you do. Jonathan: Yeah. Have you heard any horror stories about people getting themselves into trouble using UpdateCLI? Olivier: I've warned you. I've warned you. Sometimes there are situations where you can just commit directly. But yeah So, I have an horror story on this one, and that's something that affected me for one month, something like that. I was using it, I was working on another project, and then one day I woke up and saw, like, I didn't touch the updates in like four weeks. Like, no release, no recent release, whatever. But one day I woke up and saw, like, thousands of PRs opened by updates, like, over the night, with thousands of emails. Jonathan: Oof. Olivier: And so, basically, everywhere, on every place where I was using updates, it was auto-merged on. Like, it would open a PR, run the test, merge the PR. Something changed on GitHub's [00:31:00] side, where... Object CLI would clone the repository, detect that a change is needed, push the change to the repository, to the remote branch. There, GitHub would allow me to open a PR that contained no change, because there was nothing to change, because the repository was already up-to-date. All the tests would pass, then it would merge, and then Object CLI would run a git clone command, see a previous hash, a previous commit, then detect that there is an update, push to a new branch, open, run the tests, nothing needs to be merged, because nothing changed. GitHub would merge the PR, and then again and again, and because of this, CLI was configured to automatically run when something changed on the main branch. It just did that, and I discovered that GitHub does not allow you to open more than 100 PRs by similar PRs, and so it blocked after 100, and I was like, so yeah, I have a huge amount of empty commits, and that's something that would only happen via the API. Like, I tried [00:32:00] to reproduce that using the UI, and it was not possible to do, but via the GitHub API, something changed, and then it lasted for one month. I tried to find a way to fix this. I was like, okay, now Object CLI is not usable anymore, and then one month after, everything was back to normal, and so I guess that GitHub made an update, and it was over. Yeah, exactly. And that's something that really frustrated me at that time, because GitHub is kind of the host for a lot of open source projects, but GitHub is not open source. You have no idea what they are changing. It's just a black box. It's a convenient black box, but it's a black box. Jonathan: Yes, and I've had this struggle before, because we've had things break inside CI, not quite that spectacularly, but we've had things break inside CI, and it's like, you go and you try to even find, where do I open a bug report inside GitHub for some of GitHub's tooling, and that's difficult to find, and then you open the bug report, and, you know, there's 517 [00:33:00] other bug reports that haven't been touched in the last three years, and, you know, oh, I'm filled with confidence that GitHub is going to fix this. It's really a pain. If GitHub wasn't free, and so good, it would be really terrible. There's a weird dichotomy there, where it's both the best tool in existence and terrible at the same time. Olivier: Yeah, exactly. But yeah, I still... Yes, and so, yeah, I also use GitHub Action, and many people do, but that's why I said, like, I like to have some guardrails when I work with CI, and I like to be able to review, and when I enable auto-merge features, when I automatically merge PRs, those are usually for changes that are not, like, big deal. Like, for example, if it's a PR that updates all the installation links to the latest version, then I just auto-merge that, because worst-case scenario, I introduce a broken link, and someone will complain, [00:34:00] and I will fix it. But, for example, if I need to update one of the core libraries that it depends on, or if I need to update one of my infrastructure components, this is something I want to review carefully, and to be sure that I merge the PR at the right time, when I have the time to look into the consequences of my change. Jonathan: Does Update CLI have kind of that idea of a tier system, like this particular dependency is super important, tread carefully, versus this other dependency? Yeah, you can probably YOLO it. Olivier: No, also, the... Once again, yes and no. Out of the box, it's no, because it's difficult to define. You need to come with this. So, the way it works with UpdateCLI, it's a command, and then you can specify parameters. You can say, for example, run UpdateCLI. So, basically, the way it works is UpdateCLI, apply, and then you specify a configuration, and then you can specify parameters that say only pipeline IDs matching, blah, blah, blah. And so, for example, for pipelines that I want [00:35:00] to update immediately, I specify a label within my manifest that says "monitor:active", and so that means that I have an UpdateCLI job that runs periodically, like, every hour, and I only update those pipelines with those specific labels. That means that I decide one of the information that I want to have. I just want to update it as soon as possible. And so, I have different execution of update CLI, those that focus on releases, so, for example, if I have a new release, I will run update CLI just on manifest with the label "release". I use active monitor to update those that I really want to have the latest version as soon as possible, like, for example, Golang version. And then, for everything else, I just update like every two or three weeks, because I don't have the time to deal with all those notifications. So, to answer your question, yes and no. Right now, update CLI is really a tool that you can fit within your infrastructure, and then you customize the way you want to use it. Like, are you using Jenkins? Sure. [00:36:00] Are you using GitHub Action? Why not. Do you want to use something totally random implemented using Cloud yesterday evening? Why not. Jonathan: So, you Olivier: It's really up to you to decide how you want to use it. Jonathan: can you can bring it into GitLab, Codeburg, any of those other alternative sort of Git interfaces, and it works just fine. Olivier: Yes, yeah. Yeah, exactly. So we support pretty much all the Git providers out there. Obviously, depending on the Git provider, we don't have the same level of features. I'm using GitHub, so GitHub is the one that is the most advanced, but we support all of them. Something that is also interesting with Update CLI is, as I said, you have this concept of manifests. I keep telling you, you write a YAML file where you specify source, condition, and target. But we also have different kinds of updates. We have this idea of auto-discovery. So in that case, it's a smaller manifest that you write and you say, you know what, please identify all the dependencies related to Golang. And so then Update CLI [00:37:00] will automatically parse your Git repository, find every GoMod file, and then do something similar. Similar to Dependabot, where you can just detect all the dependencies and updates, and then you can say, only update this GoMod file based on the path or a matching version or whatever. That's, I would say, the next level for those updates where you can just let the tool detect everything, and we also have the ability to have shared pilot policies, and so, for example, it's kind of useful when you have, like, a lot of Git repositories to maintain, and in that case, the way it works is you I mean, you still write WCLI manifests, you put them on an R registry, like docker hub, ghr.io, whatever, and then instead of having the manifest locally on your repository, you can just specify your URL, and then you can just reuse the same URL from multiple repositories, this is version, you can use it in many places, and that's also a way, for example, to manage, like, CodeBorg organization, GitHub organization, and so on. So, you don't need to write manifest for everything, because we [00:38:00] already have a bunch of manifests that you can just reuse in a generic way. Jonathan: I- I- No, go ahead. Olivier: But in the end, everything in the back is an updated CLA manifest. Jonathan: I have this sort of cursed idea about where I might run UpdateCLI, and that is: on a server, in a cron job, looking for the things that are supposed to be deployed on the server, waiting for them to update, and just automatically going out and grabbing it and making it live. That's slightly cursed, but at the same time would be a really nice thing. I don't have to SSH into it to update things anymore. Olivier: Yeah, exactly. That's one of the ways, like, I have a server where I do this, exactly. I have crunch jobs, and then I run update.cli, like, once a week, and then it just updates my manifest, and, um, definitely. Jonathan: Oh, I may have to do that. That's actually a really clever idea. It's kind of cursed, but it's also awesome. Olivier: The, the, and, and, and, yeah. The thing is, that's really manifest, [00:39:00] like, allows you to update either git repositories, and if you don't specify git repository configuration, you just modify the files locally. Jonathan: Yeah, that's... Olivier: Yeah, definitely, that's a use case. Jonathan: So, you could probably then also, because it's plug-in based, you can also say, "And after you've done this update, restart this systemd service." Olivier: Yes, exactly. So, the way you define these kind of things is you can usually have one source, like you monitor one information, at least one. Jonathan: Mm-hmm. Olivier: What you can do is you can say "monitor this information", or multiple one, you can also change those sources, and then you can have one or multiple targets. A target could be, for example, of carrying a docker file, and so you say, like, you reuse my information, update my docker file, and then you have a second target that says "please run these docker commands" if the first target was modified. And so, that's how you can change, you can really change, chain, sorry, actions, and having like very advanced update [00:40:00] scenarios, and now, yeah. Jonathan: Yeah. Yeah, that's neat. It's not the thing that you had in mind when you first wrote the tool, but it's a really Olivier: That's something that is sometimes fun, because people, I mean, usually, when people come to updates, it's because other tools failed them. And then sometimes come to me with, like, super-weird updates in REO, and I'm just like, Yes, that's something you can do. Is it a manifest that will be easy to maintain? I'm not sure. At some point, maybe just re-implement everything in Python, because it's just too Jonathan: We've mentioned Olivier: weird. But, yeah, you can do a lot of things with the Blitz. Jonathan: it a couple of times using the AI tool when you do stuff. Pick your favorite AI agent. Is it pretty good at writing the update.cli.yaml files? Olivier: Yes, and the reason [00:41:00] for that is because one of the advice I received many years ago from the community is, I don't know if you're familiar with the Gizem schema and Gizem schema store, but the idea is, with an update CI, we generate automatically a Gizem schema that we published on a specific endpoint, and then your IDE can use that schema to validate your manifest. That means that when you write a manifest, you can just go to your preferred IDE, and as long as the update CI manifest matches a specific file name, which is documented on the website, you can just do crunch hold space, and then it will suggest to you what are the parameters and keys that you can use and how to use them. And in your AI, you just use exactly the same thing. Like, for example, I was really surprised in the past where I would just make a release of update CI, and open another PR on a totally different repository, leveraging that new release, and then GitHub Copilot would complain that the parameter is wrong because [00:42:00] I'm not allowed to use that one in this. And I was like, oh, of course I forgot to update the Gizem schema because there is this PR still pending, and I need to merge this one. And yeah, it works quite well. Jonathan: If only, if only there was a tool that could watch your releases and remind you to go update that schema. If only such a tool existed. Olivier: Exactly. Yeah, exactly. So, that comes back to the validating, to merging, because when you use this tool at scale, you end up with, like, having hundreds of repositories, and sometimes you just forget, like, where you are and what you still need to do. That's why I believe the next step is definitely to work on that UI that I would like to make it stable enough to use. It's still an experimental feature right now. Jonathan: Yeah. Yeah. So, is there—this is kind of an interesting segue into this question—do you commercialize this at all, or is it just sort of a labor of love and something that you use internally? Olivier: So, that's something that I used in Shirley for quite a long time. I don't have a commercial version of Updates Eli. [00:43:00] That's something I've been thinking for a while. But yeah, right now I just receive sponsoring. So, I receive sponsors for some companies, and sometimes that sponsor goes up, sometimes it goes down, so it really depends. But I don't really have a way to commercialize this. But yeah, that's something I would like to find a way in the future. Jonathan: Sure. You work currently, your day job is OpenSUSE. Do they give you any support for this? Do they let you work on it during your day job hours? Olivier: To make the development of this more sustainable. So, I'm working for Sioza, as you're right. I'm working for Sioza as an engineering manager. I already have a lot of work. I can sometimes use it, work on it during my day-to-day, because a lot of teams already use it within the company. So, it depends on the feature, but I'm already busy enough with my main work at work. So, it's already difficult to find some time to do this. Jonathan: Yes, I 100% understand. And it's one of those deals where it's like, if it's the thing I was [00:44:00] paid to do, it probably wouldn't be fun anymore. Olivier: That's why I said, like, it's... I had mixed feelings for many years, because on one side, I had a lot of fun working on this, and the more I worked on this, the more I realized that there are more things to do. But when you start working, and you start to be paid for something, then the relations change, and you have to do things. And that's something that I really love with the project today, is I use it on many of my projects. I know many people using it, so I try to keep this as stable as possible. And I don't try to change the manifest, because if I change the manifest, then that means that I need to go on a lot of my Git repositories to change the pipelines. So, I try to keep this as stable as possible. And, yeah, because more people use it, it's really nice to see. Jonathan: Update CLI project use Update CLI? Olivier: Of course it does, since the beginning. Of course [00:45:00] it does. Jonathan: Eating your own dog food. Love to see it. Olivier: I think it's the only way to work on open source, to be honest. That's something that every project should be doing. If you develop a solution, you should be the first user, because that's something that You feel the pain. There are things you design because you believe that it's a very important feature, and it turns out that you use less because of whatever reason, and sometimes there are things that you really find painful. For example, the JSON schema I was explaining. One of the early pains I was facing at the beginning was that I would just work on updates until I implemented a lot of new ways to configure things, but I had to update the documentation, and then I would forget myself about how to configure things. And then I would always go back to the code because I was like, "The documentation is outdated. I don't want to write documentation. It's really boring." And then I had the opportunity where I could generate that JSON schema that would generate the documentation, and that made [00:46:00] things a lot more easier because now I can just modify a comment in the code, and then it will automatically update my documentation. And those are small things that you only feel when you work, when you use your own projects. Jonathan: Yes. Olivier: And sometimes, Jonja, Jonathan: There's this thing that happens in open source projects a lot. I've seen it myself. It's like the guy that does all the work tends to, at some point, stop actually using the thing. And so you then get this statement made a lot in open source, like, "Yeah, I'm the main maintainer for it, but I don't really use it anymore." It's such a weird thing that that happens so often. I'm not sure that's entirely healthy when that happens. Olivier: It's not healthy, and also that's why usually on Update.cli, I just encourage people to contribute, and they can own some of the pieces. Like, for example, Update.cli supports different Git providers: Stash, Bitbucket, Azure DevOps, and so on. Jonathan: Mm-hmm. Olivier: Usually, I know the people who need those, and they usually contribute [00:47:00] to those features. I know the people, some of the contributors who have been using GitLab, so if I have any doubts on GitLab, I would just ask them to review a PR, provide feedback, or whatever. And that's why I usually prefer people who contribute to Update.cli who are real users. Because if they tell me, like, "I would like to have this feature for GitLab," I want to say, "Yeah, sure, go. You can use it." But if I'm not sure, and all you did is just ask Cloud to generate a feature, so you can put it on your CV that you contributed to an open-source project, then it's a lot more annoying, because I'm not sure if you will be using the feature, and if you're not, then I just need to maintain some dead codes that nobody uses. Jonathan: Right. Olivier: use. Jonathan: Yeah, nobody wants to maintain dead code. That's not a fun thing to work on. I did pull up the UpdateCLI GitHub here, and I see a few interesting things. First off, you have 107 contributors, which, definitely, that's good. It's not just two or three. I love seeing that. The latest [00:48:00] commit was made by UpdateCLIbot, which, again, we're dogfooding here. Olivier: - Yeah, I think, I think if they said I commit probably something like 80% of code to the repository all the time. It just, yeah. And it's not only dev dependencies. It's like, if you look at the README, I think the instructions will tell you to use the latest version directly. The URL will be good. If you open a GitHub issue, for example, on the Update CLI, it will automatically suggest you the latest version. Because, like, everywhere we have a version or an information that could be outdated, we use Update CLI to optimally bump those information. So, it really allows you to maintain a project over time. And because of the level of automation, I did a lot of releases. I think I did something like close to 250 releases for the past five years. And doing a release is super easy. I just have to push a git tag, and then it starts the process, and then it takes [00:49:00] me something like one hour to complete the release process. But I'm not even watching the release process. It's just, like, pipeline going, and then I just know that I need to double check. That everything is still ongoing. But otherwise, usually, it's super easy. When you have a right level of automation, it's just super easy to use. Jonathan: I'm curious, looking at, again, the open pull requests that you got and such, I'm thinking about this, and it's like, this is a tool that's going to be used by developers, and so it sort of seems like you probably have a bit of self-selection going on, that you're going to get higher quality pull requests than, you know, compared to a tool that is made for the everyday person as opposed to the developer. Is that something that you've seen? Do you feel like you get good quality pull requests? Olivier: No, no, it's really not . Jonathan: No, not at all. Olivier: Most of the time, most of the time, yeah, no, no, no. Most of the time, I definitely receive very good pull requests, but at some time, I receive PRs, and I'm just like, [00:50:00] what's the point? I mean, you open a PR just to rename something that provides no value, and you just run one hour of tests for nothing. Jonathan: Yeah. Olivier: That's something that I saw with AI over the past year, is sometimes people open PRs, and it looks good at the first time, because you're like, okay, it contains documentation. It contains tests, it contains codes, and then you start reviewing the PR, and then when you try it, you realize that it cannot work because of whatever reason. But the problem is, you already spent like 30 minutes, one hour reviewing the PR, like I already started my evening working on this, and now I'm like, I mean, it's easier if I just fix the PRs, and then merge it, because in the end, it started with a good motivation, like the person wanted to fix a real issue on the project, but something that already happened to me many times is like, I have something, I want to implement something, and then I try to ensure that PRs don't stay I usually try every day, every few days, to review [00:51:00] those PRs, and sometimes those, that time, drag me away from my initial work that I wanted to do, or finish. So that's why I was like, is PR always good quality? No. Most of the time, yes. And I'm really happy to receive those PRs, and I'm really grateful for the people who contribute to the project. But sometimes I also have that feeling where I'm just like, you should read what the code was produced, because it just, I mean, you look really weird on this. Jonathan: The funniest thing I've ever seen is, we had somebody open a bunch of PRs, and we're finally like, "Dude, you need to stop this, you're opening too many PRs," and his response was, "Oh, I didn't realize that the AI opened all of these PRs." Olivier: That was the moment when I realized that... Yeah, I remember one day... So, Jonathan: Yeah, that was a scary moment. That was the moment when I realized that the world has changed, and it will never be the same again. Olivier: One day, so you have different ways to show something on the console [00:52:00] in Golang. And one day I received one PR where a guy just renamed one of that. I don't remember which one, but it was like just renaming the way it showed the information to the console. And I was like, what's the point to open a PR just for teeth? I mean, it does not change. It does not provide value. It does nothing. It's really useless. It's just a taste. Like, do you want to use one way or the other way? I mean, from my perspective, I don't really care. The reality is we use both. But now you run this. You spend like one hour of CI to run all the tests and so on. And... I mean, it's just useless. And then, for some reason, I had to run the test again, the CI again, and then I just closed the PR, because I didn't want to spend time merging this kind of... PRs that do not make sense, and sometimes I also receive PRs where it looks easy at the first, but then you realize that it touched something core to update CLI, and then you need a lot of time to test and to validate, and you need to introduce a very nasty bug, and you [00:53:00] can really quickly end up spending your evening working on something that wasn't expected, which is annoying. Jonathan: Yes. Do you, I think you mentioned this earlier, but do you have the feeling that some of the PRs you get are just people trying to mine for "I want to be active on GitHub and a whole bunch of different repositories"? Olivier: That's the feeling that I have from time to time. You cannot really tell for sure. Because, Update.cli is in that sweet spot where it's big enough that I have enough contributors, but it's not like that 30,000 GitHub stars where everybody wants to contribute to it. It's very niche, because the challenge with this project is something, like, I talk from time to time at conferences, so I maybe do one talk about it once a year or twice a year, Jonathan: Mm-hmm. Olivier: mostly in Europe, because I'm based in Europe. So, outside of the people working on Jenkins, or colleagues working on SUSE, and the people around me, or attend [00:54:00] the conference where I go, I don't really have a huge marketing budget where I can just travel the world and keep talking about the project. It's still a very small project, and taking the time to promote it is also like a full-time job. I mean, there are people paid for that kind of things. Jonathan: What license did you go with? I saw it, and now I can't remember. Apache 2. Olivier: Apache 2. The licensing open-source project is a difficult decision to make. It's an important one, and it's a very difficult one. So, the first time I went with MIT, because I wasn't sure what to choose, and then I decided to go with Apache 2 to align, because all the projects that I'm working on are using Apache 2. That's what the Linux Foundation is recommending. The purpose was really to make it as open as possible, and the Apache 2 license [00:55:00] sounds like the best one. For the UI, for the backend and the UI that I'm currently working on to make it the Udash project that I briefly mentioned, this one is GPL. A-GPL, I think. And the motivation is, I want to make it clear that if I find a way to monetize a project, that would probably happen with a web app, with a platform. The CLI, from my point of view, should be open and forever, because the biggest value of it is that the more people integrate, the more people add use cases, the plugins, and the better it is, because even for me, it allows me some time to have update scenarios that I didn't work on, and just because I received a contribution that is useful to me. That's why this one is Apache 2. But I'm not a lawyer, so maybe I'm wrong. Jonathan: Yeah. No, no, I think that makes sense, and I'm glad to hear that you're looking at the AGPL for that for that particular use case. You know, we hear stories about, I guess not quite so much anymore, although I'm sure it's still happening, [00:56:00] but like some very big projects were getting undercut by, like, Amazon. So, like, you know, you would have a database project, and you had an offering where, you know, the database is a service, and then Amazon would come along and sell the same database to everyone for, you know, half of the cost of what you were selling it. And so these guys, like, their revenue stream just deteriorated overnight, and they started doing things like, Source-available licensing, like the Business Source License and those things, not open source for various reasons, and I, at the time, I just, I constantly wanted to know, like, why not the AGPL? That's exactly the sort of thing that it's made for, and so I'm glad to hear that you came across that when you were aware of it, and that's kind of the direction you're going, rather than one of these other less recommended, less recommendable licenses. Olivier: Yeah, I don't like those other licenses, and something that I find a bit sad is when you re-license an existing project where already a lot of people contributed to. [00:57:00] And so, that's why I was like, I want to try to find the best issue. Maybe I was wrong. Maybe, I mean, my only story will tell me if I was correct, but that's why I made this decision. But as I said, it's not an easy one, and the reality is you don't have a lot of people to talk to these kind of challenges. Like, how do you license? What is the best one to choose? And how do you protect your project, your work as well? Jonathan: Yeah, I have been lucky to have met a few people that are literally experts in this field, and it's still a difficult decision sometimes. Have you gone the route of like a contributor license agreement, a CLA, to be able to go back and make changes to the license if you needed to? Olivier: No, that's also something that... I don't have a contributor agreement. The reason for this is I want to keep it as simple as possible to contribute to the project, and I try to avoid those [00:58:00] those extra work, I would say, on an open source project. That's one of the other things when you think about, for example, donating the open source project to a foundation. It also comes with paperwork, I would say, where you have to have a governance and so on. And those are important topics when the project goes big. But in this case, there is no VC behind me. It's just a side project that I'm working. So I don't need to protect from actors where the motivation would go in a different direction from the project. Jonathan: Right. Olivier: Because in the end, once I mean, I've been working on open source like professionally for a long time, and it's a tricky balance to find between business objectives and the open source objectives. Ideally, from my perspective, it should work in a symbiotic relation where the two can contribute to the lifecycle of the project and to the business. But sometimes there are situations where it becomes more difficult to find the right relation. But I don't have that kind of problem today, because I don't have a company behind, [00:59:00] and I still have a main work, so my... I just... Yeah. Jonathan: It is the lower-stress approach to not have a company behind the project, for sure. All right, so we've gone through quite a bit. Is there anything that you want to let folks know that I should have asked you about? Is there anything that we didn't cover that we should have? Olivier: That's a good question. Well, I think we covered quite a lot on the project. So, if people are interested, yeah, the project is hosted on GitHub, so they can have a look. If they have any questions, we use Matrix. So, we have a Matrix channel if they want to reach out and ask questions, and all the feedbacks are more than welcome if you want to experiment with it, and make suggestions. Like, if there is something you would like to see different, yeah, I would be glad to hear from you, because once again, what is really difficult with open-source software is to get feedback from the users. Jonathan: real Olivier: Um, because you never know who's using and, [01:00:00] uh, yeah. Jonathan: users. Don't just let your AI come and open up issues." Olivier: Yes. Jonathan: Alright, now I've got to ask two final questions before I let you go, and, well, I guess it's three now. So, the two questions that we have historically always asked is: what's your favorite text editor and scripting language? And just last week I came up with a third question that I think we really need to add, and that is: which AI agent do you prefer for writing code? So, those are the three that we'll end with. Olivier: So which ID am I using? So I usually like to use NeoVim for most of my usage. Jonathan: Uh-huh. Olivier: And then I use VS Codes where I'm really into programming activities. Like when I work, for example, with Update.cli or another project, I usually have VS Codes. But for any other use cases, I have NeoVim because the reason why I like NeoVim is it can also have autocompletion. So for schema, if you write [01:01:00] in Siebel, if you write Update.cli, GitHub Actions, you can configure in a way where you get autocompletion, validation, and so on. Jonathan: Nice. Olivier: So for me, it's like NeoVim and VS Codes. Jonathan: Pretty cool. Olivier: The second question was, what was it about again? Jonathan: The favorite scripting language. Olivier: So for me, my approach with scripting languages, I used a lot of them in the past. Usually, I love Bash. I mean, that's my default way to go. I use Bash for a lot of things. As soon as I find my Bash to become too big, because whatever reason, and then I start to have too much complexity there, I usually switch to Python. So for me, it's very common. I have like a big Python script that called Bash as well. But I try to keep Bash as simple as possible because Bash allows you to write a lot of weird stuff and you should not use those syntax because it makes it difficult for any newcomers to read them. Jonathan: Yes, this is true. Olivier: So, that's why I like to switch to Go, and then if it's Go one step further, I love, oh sorry, Python, and then I switch to Go. So [01:02:00] that's, I would say, my level of complexity based on what I need to do. And what AI agents? Jonathan: Yeah, if you have to write or review code and you want the AI to do it for you, which one do you turn to right now? Olivier: Right now, it's the one that sponsored me. So, right now, I'm getting some sponsoring from Anthropik. So, I use Cloud for two months. I'm quite happy with the usage of Cloud. It works quite well, and I was able to fix a lot of annoying bugs, and also work on a lot of those small things that are super easy to do, but just takes time, and so with Cloud. In the past, I was using GitHub Copilot, because I received a license as part of DataCity, because it's an open-source project, and I was able to fix a lot of annoying bugs, and also work on a lot of annoying bugs. It turns out that since they changed the way they measure and count tokens, I kind of stopped using it in June. For me, it's unusable anymore, so for me, GitHub Copilot is not something that I'm looking at now. Unless to review some PRs, but I [01:03:00] find it very unreliable, in the way that I never know how many tokens I will be using, and I have a mixed feeling with GitHub Copilot. And otherwise, it works. We have everything else. Jonathan: I will say, just because we've had really good success with it, CodeRabbit, for doing PR reviews, is actually quite good, and it's been around for a while, like several months they've been offering it as a free service to open source projects, and it's actually pretty good. It's worth looking into. Olivier: Like, SUSE provides us a lot of AI, and then we experiment with the different things. So, GitHub, yeah. Jonathan: They've got a paid version for commercial projects, but if you're open-source, they'll get you set up for free. Olivier: Okay, nice. I will have a look. Jonathan: Yeah, if that's something you're looking for. Alright, Olivier, thank you so much for being here. It's been a blast to get to talk to you about all of the crazy things that we can do with Update CLI, and I legitimately will have to get this running on a couple of different [01:04:00] places. I'm looking forward to it. Olivier: Thanks, thanks, thanks for having me, that was nice. Jonathan: Alright, thank you, sir. Yeah, absolutely. All right. That was Olivia Venen talking about UpdateCLI, the surprisingly capable and flexible plugin-based Git updater to Track releases, update your code, all sorts of stuff, a lot of fun conversation there. We do have a show next week as well. We're talking with Jan Altenberg and Alexander Barr of the Open Source Automation Development Lab, and then we're taking the week off after that because I'm going to be traveling, and then we've got a couple of slots open. So if you or somebody you know needs to be on the show, let us know: [email protected], and we will make it happen. We'll get them added. Other than that, I just want to say thank you. We appreciate everybody that's here. Those who get us live and on the download, and we'll be back next week on Floss Weekly.
• 
  Episode 881 - Eating Its Own Tail 09.09.2026 1t 5min

  This week Jonathan chats with Olivier Vernin of Updatecli, the tool that helps you update everything! Need to roll out dependency updates intelligently? Or even automatically deploy to a server on changes? Updatecli can help! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 880 transcript 02.09.2026

  FLOSS-880 Jonathan: [00:00:00] Hey, folks. This week I talk with Benjamin Samuels of Trail of Bits and we're talking about Patch the Planet. That's the Trail of Bits initiative to help open source projects deal with the absolute tidal wave we're all seeing of pull requests, bug reports, and vulnerability disclosures. This is Floss Weekly, episode 880, recorded Tuesday, September the 1st. The two wolves It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we're talking about something really cool. We've got Benjamin Samuels of Trail of Bits, and he does their boring blockchain thing that we may cover a little bit. But the other thing he's involved in is Patch the Planet, which is all about helping open source projects with security problems. And reading the notes ahead of time, I happen to know that, take a drink, we're gonna talk about AI. But not necessarily the [00:01:00] everybody should using it, but this is causing a problem for some open source projects. Like, how do we deal with that? We'll get into all of that. Let's go ahead and bring him on. Benjamin, hey. I'm super glad that you're here. Welcome to the show. Benjamin: Absolutely. Thank you for having me. Jonathan: Yeah. So tell me... Let's see. Where do we start? You've got a background in electronics. You were telling me before the show started that was your college major. And you found Hackaday way back in the day- Yeah ... looking for I think like a lot of us did, looking for information about what's practical for me to try to tackle in the electronics world? Benjamin: Yeah. Back in the day, back in college, poor college kid it's really hard to figure out, what kind of electronics projects can you actually do on your own that you don't need a whole lot of equipment for? And it was way back before Arduino like you a- if you wanted a microcontroller, you had to buy a pick, and you had to buy a programmer, and just getting into stuff could be like, $500 to $1,000. And the real struggle was, like, looking for projects that, [00:02:00] I could actually tackle. 'Cause, you're learning electronics, you wanna actually do it. But it was a really high barrier for entry, and Hackaday was a s- a source of some of my earliest projects, just trying to reproduce what other people had built. It's a lot of fun. Yeah. I remember I've told them this story ahead, before the show started, w- when I was first trying to run some ethernet and trying to figure out how to terminate ethernet. I think I learned that from Hackaday, and that may have been one of my first exposures to this world that we now all live in. Jonathan: But one of the other really interesting things I did very early was toner transfer method for making PCBs- Yeah ... and that was surprisingly easy for a, I think, t- maybe teenager, either teenager or college student, myself at the time, to get into. It's like, 'cause you could get a magazine and print onto the magazine. S- if you had the right kind of printer that did waterproof toner, and then you could take, and you flip it upside down, and you put it on your PCB board, and you take an iron and iron it, and the toner would transfer. And you could caref- I think you had to use water because you wanted that, you wanted the paper itself to disintegrate and leave the toner [00:03:00] behind. And then you could dump that into a bath of what was it? Hydrochloric and muriatic acid, the mix? Something like that. Yeah. And it would etch the copper, but leave where the toner was. And so that was... I, man, I remember that. I have... I still somewhere around here, I have a device that I built that way. It was a- ... headphone preamp. My, my first foray into playing with tubes as well. Sup- super cool times, man. I hadn't thought about that in a long time. Thank you. Thank you, Hackaday. Benjamin: Thank you, Hackaday. Yes. Jonathan: So you kinda went from the hardware to the software and security, and you're now at Trail of Bits. What do what do you do at Trail of Bits? What's your job there? Benjamin: So I'm the director of engineering of the blockchain practice and we're a part of a larger group called Assurance, which means software assurance, trying to make sure that software is secure and to have all kinds of companies out there that have very high impact software where if there was a dangerous bug in it it could be extremely expensive and have a lot of consequences. And they'll want an external review to understand what have we missed about this? [00:04:00] What do we not know about our software that we should know? What kind of bugs are there?" And they'll come to a company like Trail of Bits, and we have a couple of different practices. So like mine's just blockchain. We have a cryptography pra- practice. We have AppSec, we have AI ML, and we have a lot of these people who are very specialized in what they do in the security field. And we could just pull in whoever we need for whatever type of project. So if you have someone building like end-to-end encrypted software that, runs in a web application, like we could grab an AppSec engineer and a cryptographer and have them all work on it. But yeah it, we mostly review code. We do design reviews, threat models. My day-to-day is helping everyone, figure out what to do. We also spend a lot of time contributing to open source. We only spend about 60, 70% of our time on client work. The rest of the time we spend trying to improve ourselves, contribute to open source build new security tools, and publish those tools. Jonathan: Okay. That's ... i've, I know some other [00:05:00] businesses that do that as like part of their policy, and I always thought that was a really great idea because, as you look through history and some of the places like Bell Labs really springs to mind. And it's like those guys, it's just "Here's a lab, here's a paycheck, go do cool stuff." Yes. And we got Unix out of it and, a whole bunch of other things were developed at Bell Labs. And whenever I hear a company that, that at least gives their workers o- one day a week, it sounds like that's about what you guys are doing, something like that- ... just to go make cool stuff. I think it's great because, you get all these u- unexpected things that come out of it. Benjamin: Yeah. The, and people think of all kinds of crazy ideas. There was this recent hack with OpenAI and Hugging Face, and, these agents that broke out of these containers. And we had an engineer who's really interested in this, and he set up like a, a QEMU container environment- and put GPT-5.6 cyber in there and said, "Break out." And it broke out. It found a bunch- Wow ... of vulnerabilities. I know, [00:06:00] it's crazy. And it's i- if he didn't have the time to, sit down and actually put that together we would have n- never known, there's a lot more of these containerized environments that aren't, able to withstand these frontier cyber models. Jonathan: And so that's... I feel like that's probably the direction we're gonna go with the majority of this talk. And it all ties together because you were talking about this idea of assurance in these commercial projects. All of this commercial stuff is built on top of open source code. Like- ... almost inevitably, there's open source code somewhere in the stack. Yes. And, that's been a... It's been a concern, it's been a concern for some people for a lot of years now. A business may be able to audit their code, but are they actually going to be able to audit all of those open source tools? And, on some of these open source projects, nobody is getting paid professionally to do the work, and so there's these- ... questions about code quality. Now, ironically, I have seen some closed source code bases, and I've looked at a lot of open source code bases, and generally, the open source code bases are a lot prettier and a lot [00:07:00] healthier. So there's some inversions of expectations, but I think it's still an important topic to think through and that, that question in particular about is anybody getting paid to do the maintenance on this in the- ... open source world is a r- really important question. What... how does Trail of Bits fit into this, this whole question about code quality and open source? Benjamin: That's a good question. So our most recent initiative has been... should I go into Patch the Planet? I- 'Cause I feel like that's, like- Take it- ... probably our highest touch area. Jonathan: Take it whichever direction you want to. I think Patch the Planet is where we eventually wanna get, but there's- We'll eventually get there there's cool stuff to talk about. And absolutely. Benjamin: I'm gonna loop back to blockchain, 'cause one of the unique things about blockchain, w- whether you like it or not, something that's interesting about it- ... is that almost everything is open source. Yes. People open source their smart contracts, they open source their node software, they open source all of this stuff. So security engineers or security researchers can look at it all that they want. And w- we've been able to look at these things that get open sourced, and we've been able to look at the [00:08:00] ones that people keep closed source, and there is a quality difference. Because if anyone can look at this open source stuff they can file a bug as long as they actually have established a way to file the bug. They can provide feedback, they can criticize. And there's also a big monetary incentive to do this in blockchain specifically, 'cause blockchain really leans into these bug bounty programs in a way that a lot of the rest of the industry doesn't really use them quite as much. There's even a bug bounty program that pays out $4 million on their top bounties, and it's completely unheard of in the traditional security space. And in my opinion it leads to the s- software being a lot more secure- ... 'cause there's so many more people looking at it, and there is a financial incentive not only just to get those bounties, but also if you're able to hack one of these blockchain things, like you can actually steal money. So there's- Yeah ... really real stakes there. And I'm still definitely a lot more bullish on open source being the way to secure software or [00:09:00] d- open source having more secure software- ... than these closed source approaches. Jonathan: Yep. I g- I gotta tell you my two takes on blockchain real quick. One is pretty serious, and the other one is trolling, but also serious. The first one is blockchain reminds me of the invention of the laser. It's really cool. We're still trying to figure out what to do with it. Yeah. It's the invention looking for solutions to go solve. And then the other one is we already have the killer app for blockchain, and it's Git. Benjamin: I know, right? Jonathan: That technology's already proven its worth. Now, the whole making money out of it, not qui- not quite as sold on that. Benjamin: Exactly. Jonathan: All right. So I think this is probably a good place to transition into Patch the Planet. What, what is that what is that effort from Trail of Bits? Benjamin: So the idea behind Patch the Planet was that I think it was earlier this year, it became very clear that these frontier models their cyber capabilities were starting to get to the point where they might be extremely dangerous. It wasn't until I think earlier this year that we were able to take [00:10:00] a, a frontier model, ask it to find bugs in a code base, and actually have it find meaningful bugs. ... You started to see this a little bit with Anthropic, with Project Glasswing. So what they did is they took Mythos, which was their new model at the time, and they ran it against a bunch of open source stuff and found tons and tons of high severity issues. And there, there was clearly going to be this transition shift or phase shift where if everyone has this powerful cyber capability, anyone can run it on anything and be able to find a bunch of bugs. And so we have this huge installed base of open source software out there. You've got, the Linux kernel, you've got, all these frameworks, you've got all this compression stuff. And all of them are gonna have bugs in it because there's one constant in software security, and s- It's that software has bugs- Yes and you're never gonna be able to change it. Jonathan: Absolutely. Benjamin: So all of a sudden these bugs that previously you had to take a specialist and have that specialist or [00:11:00] security researcher or whoever spend a bunch of time trying to find these bugs and figure out how to exploit them and write up POCs, and that's really expensive to do from a time and money perspective. And when you take the, the cost of that and you reduce it you increase the risk profile to everyone, everywhere. And so the idea behind Patch the Planet was to take OpenAI's latest cyber models and apply them on open source projects to try to find these bugs, patch them, and reduce the risk profile that they have. Another challenge with this is just patching the bugs is almost impossible. Daniel Steinberg has been writing about this with Curl. Earlier this year they were receiving all these bug reports and m- most of them weren't that good. They were like- they were hallucinations Jonathan: were true shields. Benjamin: They were hallucinations. They were terrible. Jonathan: It was some- There was- somebody that, that really w- was... honestly, really smart guys, that were just way ahead of the curve- ... were saying, "Hey, there's this bug bounty program. ChatGPT, find me a CVE and this, so that I can make money off this bug bounty program." And [00:12:00] then they didn't have the chops to test it and figure it out themselves- ... if it was real. And so there were some really low quality CVEs that were getting reported to Curl. And I think, didn't Curl actually close their bug bounty pro- program as a result of this? Yes, they Benjamin: did. Jonathan: Yeah. Benjamin: So there was this kind of like another phase shift where as the model started to get better- the reports started to get better as well. Yes. This mountain of reports that used to be, like, really low quality started to be actually you know, for real bugs- ... that needed to get fixed and I think Daniel shut down the bug bounty program just because they were underwater trying to patch all of these bugs. Yeah. It wasn't just issues that were trivial at that point. These were, like, real bugs- ... in the software that had to be fixed, but it's just too much to fix manually and so part of Patch the Planet is we try to create the patches ourselves as well. It turns out that's actually much, much harder than finding the bugs in the first place, and it's a, a place where we're putting a bunch of research into right now. But we've been doing [00:13:00] this since, I think June and we- I think we've done 50 or 60 projects so far. We've found, thousands of bugs. I th- think we got maybe 200 of them patched so far and things are looking up. I- we've been really happy with the progress so far. Jonathan: I- is Patch the Planet, is it primarily trying to use the AI models to write the bug fixes as well? Benjamin: Yes. Although they're not as good as at doing that. So looking for vulnerability is a relatively closed problem because once, once you find a bug that crashes a program, you can prove, "Hey, this bug crashes this program." Jonathan: Yeah. Benjamin: But w- if you wanna fix that bug, it's a lot more challenging to have an oracle that says, "Hey, you fixed the bug. Good job." Jonathan: The program- Just- ... still does all the things right that it was doing Benjamin: right before All the, all of the things right. And this is especially- Yeah ... a big problem for the Linux kernel because the Linux kernel- ... doesn't have a test suite in the way that most software developers envision having a test suite. A lot of it is this [00:14:00] tribalized knowledge. So if you wanna create a patch for the Linux kernel, there's all this stuff that the LLM has no way of understanding- ... all these hidden requirements, and it basically cannot write a patch on its own. It doesn't have e- enough ways to verify that the behavior of the rest of the program is still correct in the way- that it needs to be correct. Jonathan: Yeah. It's really a challenge. One of the, one of the other challenges that I've seen, and this may just be because of the project that I'm involved in the longtime listeners know is the Meshtastic project, and so we run open source firmware on little tiny IoT boards. And one of the challenges that we see with this, and with human reporters too, by the way, is that there are certain things that from a certain point of view would be a vulnerability, but from the point of view of we're writing embedded software on IoT boards, it's- ... no, that's just a limitation of the platform. And yes, we would love if that were different, but we only have, 120K of RAM or whatever it is on the individual platform. It's just we can't... There's just no fix for that. [00:15:00] And so- ... i'm wondering if you have insight into this, like, how good is... So i- if a program crashes that's generally, that's a bug, it needs to be fixed. Yeah. But there's some other classes of bug, or I guess even in the case of a program crash, there are certain times where it's like that is explicitly trusted input. We don't really care if that causes it to crash. Or- ... something like this is a vulnerability report that we got. There is a inside the... So we can run Meshtastic on a Linux computer and, run the actual firmware as a sort of inside a wrapper on a daemon. And so then it pulls in from a, a YAML config file, and one of the things that you can set is, as part of sending a packet, run this command. I'm simplifying a little bit, but essentially this is what... Run this command, take the output, and set it inside a packet. And someone's like, "Oh, arb- arbitrary code execution inside your config file." And we're like- It says it on the label. That is the feature. What are you... What's wrong with you? And so all of that to [00:16:00] be a, a, a segue into do we still have this problem with the, even the frontier models where they just, they don't understand that part of the context of, "No, that's not a bug. That's the way that's supposed to w- The user is supposed to be able to crash the program by doing this." Benjamin: Absolutely. This is still definitely a very persistent problem, and it's a, something that we've encountered in Patch the Planet and have tried to fix as well. The problem is these models don't really understand the context of the environment that they're working in. They're able to find these bugs but not understand, hey, this bug isn't actually a very big deal because of- ... what you said. Like- May- maybe a bug requires hardware access and that's the end of the world for one threat model, but it's an accepted outcome in another threat model. Yeah, exactly. And so one of the parts of Patch the Planet that we're trying to do is not just find these bugs and patch them, but try to improve this agentic maturity. That- that's the best thing I have to call it right now. It sound, kinda sounds like a buzzword, but- ... the idea here is [00:17:00] we wanna give these models a better way to understand the context of the project that they're running on. So you've got all these security researchers. They're gonna run their models against your code and try and find bugs. The question is, can we change the behavior of their model that they're running to make it so it produces better results? And so the way that we've been trying to do that is with a threatmodel.md file. And this is literally just a file that says here's how this project exists. Here's what its threat model is. Here's what a bad issue is. Here's where the trusted input is. Here's where the untrusted input is." And then all these models, they always read the agents.md or claude.md. In that file we say, "Hey, if you're doing vulnerability research, you need to take any, any results that you have and compare it against the threat model before actually reporting it back to the user." And so in this way s- now we're able to program other people's models that are investigating our code base. [00:18:00] And th- there's so man- and e- even that's not 100% perfect. But with so many of these projects, they have a very unique threat model. I was working on Envoy a little bit. I don't know if you're familiar with it- No ... but it's like this it's like a sidecar kind of reverse proxy kinda deal that you- Okay run in Kubernetes. And usually it sits, behind a reverse proxy like NGINX or some load balancer. And it sits between that and an actual application. And so it trusts the application, and it usually trusts the upstream, which is your load balancer or whatever. But there's certain points where you don't wanna trust that, or you do wanna trust that, depending on the configuration. And capturing that in a threat model file is really the only way to give these models the context to understand whether a, a bug is actually serious or not. And in some cases, that's not even good enough because it might find a bug. When I was running it on NGINX there was all these bugs around if you have this really unique- like a matching rule. So like you can use [00:19:00] NGINX as a, like kind of an authentication layer. So if like you connect to it with a certain certificate, it'll let you access this endpoint. And there was all kinds of these weird configurations that would let you bypass those controls, but i- in practice, nobody actually uses those configurations, and they, you can't even necessarily capture that in a threat model file. You have to be the maintainer and you have to understand, this is how my software gets deployed in a production environment. This bug is not vulnerable, or this bug is actually a really big deal. And taking that knowledge and transferring it to an LLM is very challenging. Jonathan: Yeah. There- there's even a step beyond that's difficult for the human maintainer, and that is that sometimes those bugs are not part of your threat model until- suddenly you discover that they need to be. Yeah. And- and that's, that's where you discover that, oh this- this other big project that we didn't even know included us- ... uses us and sets it up by default in this particular way, and now all of our assumptions about our threat model need to change. Benjamin: Exactly. Jonathan: Okay. So [00:20:00] what as you guys do this so is- is this something that a project has to opt into? So did NGINX reach out to you guys and say, "Hey, please look at our code and find and help us fix bugs"? Benjamin: So the first round of projects, it was much more of if anyone would give us the time of day, we would ask them to participate- and they could just choose their level of participation. So if they just wanted to not do anything and let us submit the bugs, that was fine. The outcomes for those projects weren't that good, so after like we had the initial launch of Patch the Planet and made it public we wanted a higher level of engagement with the maintainers. Because if- if we're just filing bugs on their project and filing patches, like we could m- be making their life worse, because we could be submitting all these patches that don't meet a certain requirement that they have. And now they have to go through all these, these pull requests and review them and say, "Hey, you did this wrong." So a higher touch relationship is really valuable, and it's something that we've been looking for and I'd say at, we've had a lot more success since that [00:21:00] first week now that we've published this, and like projects are coming to us and they're like, "What do we do about this? What do we do about these, all these bugs, all these researcher reports?" I think we've had really good outcomes since then. Jonathan: Is- is part of this process just teaching people how to set up their, agents.md and all of these bits of documentation? Benjamin: I wouldn't say that's the only thing because maintainers aren't stupid. They can Google a lot of this. Like A- and like they, they can figure out how to... They can ask Claude or whatever to create an agents.md. But it's the, the little spec- specific things like that threatmodel.md file, that's not an obvious thing. You have to be working with these models quite a bit to realize that you need to do something like this. It also this concept of programming the researcher's model- ... by forcing them to follow the threat model, that's not something that's going to be very obvious until you've been working with this quite a bit. So we, what we've been trying to do is build up this knowledge of what works [00:22:00] versus what doesn't through experience, and then taking that and applying it to each of these projects. And so yeah, it is a little bit of like showing them what works, but, Yeah. Jonathan: So as you guys have d- have done the dive into this I'm assuming we're not at the point to where you can just set the agent loose and say, "Read all the incoming bugs, write patches to fix them, drive s- drive, drive CI green, and then commit." Benjamin: It depends on the project. I'm familiar- I'm f- I'm familiar with that exact language because reasons. Yeah. But it really depends on the project. 'Cause again, like if we were to do something like that with the Linux kernel, it would be a complete disaster. Oh, Jonathan: yes. Benjamin: But there's some projects where they've already gone through a lot of effort in building agent-specific documentation- to help agents understand, "Hey, this is what my expectations are of a pull request." And for those, you could, in theory, probably do that. I think we have OpenClaw as a Patch the Planet project recently. [00:23:00] And they basically don't write any of the code in that code base. They use models to do absolutely everything. So they are at the point where you can basically point the model at a, a PR, or point it at an issue and say, "Hey, go fix this," and it goes and does the whole thing all at once. Jonathan: That's very on brand for OpenClaw. Benjamin: Yeah. It's ve- very on brand for them. But taking a project that isn't at that point yet- and getting it into that point where you could theoretically do that is really challenging. Because it, not only do you have to figure out what does the agent need to make these changes, you need to yield a lot of control. And for a maintainer of an open source project where, you've been pouring your lifeblood into this thing, you've been maintaining it for so long I feel like it's going to be, if you wanted to do that, it'd be very hard to yield that control and be comfortable with it. Because you don't know, like y- you're never sure... These are probabilistic models. They aren't always going to be correct. They're going to make mistakes, even in [00:24:00] the best case scenario. And yielding control over to something where you're not sure if it's going to operate correctly can be very uncomfortable. Jonathan: Yeah. Yeah. D- This is a sort of a philosophical question, but i- in as you look across the, open source and closed source landscape, but do you think that is the direction that we should be moving? Does it make sense to try to get to the point to where the AIs get to run our software for us? Benjamin: That's a good question. I, so it, I think it's very project dependent. You have to think to what's the core of why do people build open source software? And even though they're not getting rewarded, they're not being compensated, why do they do it? And I think it comes down to every one of these maintainers has like these two wolves in them. They have the wolf that wants to get things done. Wants to build a piece of software that does a certain thing and it accomplishes a certain goal. And the other aspect is I want to craft something. I want to [00:25:00] be able to build something beautiful in code. And you really see a lot of this second group with Rust, I feel like. 'Cause Rust is all about, crafting this really attractive code that, you know it... You see it and you're like, "Wow, that, that's a beautiful construct. I love the way that this was built." And I think the requirements of those two different halves are very different. The half that just wants to get things done, if there was a developer of open so- of open source software who just wanted to get things done, yes, they're gonna go the full agentic, do everything. Because at the end of the day, they just wanna build something and have it work and have it be high quality. That's what they wanna do. But this person who has more of like the love of the craft, of actually putting the code together and, talking to people and, working with maintainers I don't think that they're going to jump straight into it unless they're absolutely forced to. And like I think if there's anything that was going to force them, maybe it's gonna be like patching these security bugs, [00:26:00] but I don't think that they would go a lot further than that. Yeah. I honestly I've seen the projects that have really resisted any AI contributions are just- they're falling behind. Not just in the CVEs, but like in code quality polish and new features and supporting things 'cause you can just get, you can get so much, you can get things done so much faster by using the- ... using the new tooling. And it's funny that you mention Rust as one of the examples, and I don't disagree with you, but like- Rust has taken all of the smarts and the automated stuff, and they've put it in their compiler. Jonathan: The checker that's part of the compiler and now they're resisting taking not exactly the same thing, but the new tooling and making it part of the code writing process. It's just ironic. And I've seen people call this out resisting AI is analogous to the guys that used to resist the compiler and only want to do handcrafted assembly. It's... I don't know that's entirely fair either, but it is quite a humorous [00:27:00] analogy to think about. Benjamin: Yeah. I've heard that analogy as well, and I'm still undecided if- ... if that analogy truly applies 'cause you can look at someone who wrote assembly code. Go look at the Game Boy it w- a Pokemon game's assembly code- and be like, "Wow, this is really beautiful." And- I don't doubt that there's people out there today who, try to find problems where they can exercise a similar skill set in op- in trying to optimize something so that it can actually fit into something like that. Jonathan: Oh, absolutely. Benjamin: And I'm not sure but e- the thing is writing Assembly is just so much harder than anything, than writing C code. I- that's where I don't think it's completely comparable because there's this dopamine loop that you get when you're coding, when you're actually manually writing code, and you finish something and it's like, "Ah, I built this thing. It's beautiful. It works." Jonathan: It finally Benjamin: works. It finally works. It doesn't have bugs. And I've noticed that dopamine loop is not as strong when you're building those things with AI. And [00:28:00] so I think that- ... there's going to be people who still want to develop just to chase the dopamine loop. They they don't even care that they're, necessarily falling behind. What they care about is the craft and putting the code together and, actually making it work. And I think that there is a unusually high concentration of those kinds of people who love the craft in the Rust community, and that's why we've seen more pushback there than anywhere else. Jonathan: Yeah. Since we're on the AI topic, there is some interesting news that happened just in the last few days, and that's Debian. The Debian developer community voted, and they pretty handily voted to take, oh, I forget the exact term that they used. Responsible AI usage. To allow responsible AI usage, which was, like, the sort of the most open-ended of the options. And I've already seen, at least one person leaving the Debian community and with dramatic flair. The, I saw one meme. It's like the exit door and a normal person walking out the exit, and then dramatic exit- ... and [00:29:00] somebody going out that door. It's like y- you didn't have to announce this to everyone. But it's interesting to see just the kind of polarizing effect that it has on some of the devs. Benjamin: Yeah. There, there's never been a tension between these two halves of the developer persona before, the one that wants to build the product and the one that wants to craft the code. There, you, for a long time, these two parts of, every w- every programmer could just coexist and just exist in there, and I feel like the, AI is really creating a tension there that has never existed before. So I do think it's g- that this pattern where, people are leaving projects that adopted a little bit too much, I think that pattern's going to, repeat over and over multiple times. And I think eventually we'll be at this point where you have these people who with the love of the craft and the people who just wanna get stuff done, and I don't think that they're necessarily gonna talk to each other very much. Or if- ... they do it's gonna be, probably relatively limited. We'll see. Jonathan: Interesting thought. I don't know. I I tend to bully my AI [00:30:00] agents into writing good code. No, don't do that. That's a really dumb idea," it helps. You yell at the AI and it finally does what you want it to. Which, really makes me glad that I came up and started writing code before these agents existed. Same. And it does kinda make me worry about the next generation of programmer that's not going to have that experience to pull from. Although I guess by the time that's a real concern the agents are gonna be so good it won't matter anyways. I don't know. Benjamin: Yeah I echo that sentiment. I think it's gonna be very interesting to see what happens when most of the developers out there, the limit of their manual programming capability is what they learned in college. Because there's so much that you learn after that in terms of abstraction, encapsulation, decoupling. What happens when all that knowledge disappears because you don't necessarily need it? And I think it's gonna be very interesting to see how things change. Jonathan: Yeah. Man, I think one of the, one of the main things that I've learned... I didn't actually go to college for doing programming stuff, so I've picked it all up as I've gone along. [00:31:00] But one of the main things that I've, I feel like I would not have learned in college, but I've definitely learned working with open source projects, is just, like, how to write code with good taste. I know Torvalds talks about this a lot, too. But you can write really ugly code that works- Yeah ... or you can have an elegant solution. And that's sometimes that's something that the models really struggle with. Benjamin: Yeah, they absolutely struggle with it. And it's like creative writing, too. It's bizarre how much of a re- regression we've had in, like with the models and writing. I feel like I was able to get better writing out of the anthropic models from a year and a half ago. The ones nowadays are just terrible. And they're very monolithic in how they talk. They always talk the same way, and I feel like that also comes out in how they code and how they program. And it doesn't lead to, code that is all that beautiful. Sure, maybe the, it's faster to write the code. Maybe it has less bugs or whatever. But it's not necessarily elegant in the way that I think a very [00:32:00] experienced human programmer would write elegant code. Jonathan: I kinda suspect that certain languages are over-trained. So I've written in a bunch of different languages. Not a whole lot, but I've written... i've done Java, I've done C, I've done JavaScript Python- ... a whole bunch of things throughout my career. And, like, when I when I ask the LLM to write something in Java for an Android app, and I go back and I look at it, it's yeah, that's pretty much what Android apps look like. Yeah. But when you ask it to write something in C or C++ and you go back and look at it, it's like no, th- this is like a Java developer is writing C++ for the first time. It's terrible. Benjamin: Yeah. I think there was this idea in the AI space that- RL would transfer generalizations really and I feel like it has not transferred generaliz- general- it has not transferred generalizations as well as people had originally hoped. And so you end up with artifacts where it's like it should be able to write good C, but because that wasn't explicitly one of the, RL environments it was in it's just [00:33:00] not able to do it that well. Jonathan: What does RL mean in this context? Benjamin: Oh, reinforcement learning. Oh, okay. Okay. So post-training. What happens in post-training, so you have the pre-training and then you have a mo- like a base model and then you do post-training. And in post-training they'll try to tune the behavior of the model. So let's say they want it to be much better at writing code, they'll give it a task to write code and if that code compiles and works, they'll reward the model. And if it doesn't, they throw it away. Got it. And that's one way to improve the model's capabilities. Jonathan: That's the, the weights. Benjamin: Yeah. The, the, the weights. Yeah. Yeah. Jonathan: Yeah. Of course. Benjamin: And the idea that the, the AI researchers had is that, okay, if I create, an RL environment for doing like Java code there will be generalizations about how to write code in general that may transfer over to other languages potentially. That's ... This is a contrived example. That's not exactly what they're thinking. Sure. But it, that's what they mean by generalization. Jonathan: Yeah. Yeah. Makes sense. All right. We've talked about this idea of of frontier models a bit, and I [00:34:00] know what that means. That's like the cutting edge, right? And it's interesting to see just in the past six months how much of a change there's been in that, and then thinking specifically about what that change, that, that rapid acceleration means for all of our code bases. As you rub the crystal ball and look into the next six months to a year what do you predict is coming, and how is it going to change the software world? Benjamin: That's a good question. There's so many variables it depends on 'cause these models they have these incredible attacking capabilities 'cause cybersecurity by its nature is dual use. If you know how to hack something or if you know how to defend something, you know how to hack it as well. Yep. And I think one of the challenges for these frontier labs is separating the two and figuring out what capabilities can you give an LLM that are asymmetrically defensive- ... so it's not dual use. You can't use it to attack something. If they're able to do [00:35:00] that and separate those two things apart, I think what they're gonna do is they're gonna start releasing models that have more of these defensive capabilities that aren't gonna be blocked. If you try to ask Fable right now to review some code and look for bugs, they won't do it. It'll get caught by a cyber safeguard and downgrade you, even if you're trying to fix a bug in your own code. Yeah. And that doesn't make sense because what you're trying to do is fundamentally defensive. So if the labs are able to figure out what is asymmetrically offensive versus defensive, I think they're gonna release more stuff that's defensive, and it's gonna be fundamentally really good for us because it means that the agents are gonna write better code. They're gonna be able to detect more bugs. Now, if they're not able to separate those two, I would be worried about the latest cyber capabilities staying closed. Like what we've seen with Mythos where Anthropic only lets a certain set of companies in Project Glasswing access the model. I could envision that happening basically for all of the new models, and then the things that end up being [00:36:00] published that anyone can use just don't have any cyber capability. But that also means they might write worse code with more bugs in it, and I, I think that's a very unpleasant outcome, and I hope that doesn't happen. Jonathan: You could become a conspiracy theorist and go down some really weird conspiracy theory avenues with all of this stuff really easily, couldn't you? Benjamin: Yeah. Absolutely. Jonathan: They're doing it on purpose. They wanna be able to hack our code. Benjamin: Yeah. It's a very- high, this is a very high-risk moment because if they just plow forward and keep doing what they've been doing, ah, people are gonna have access to models that are gonna be extremely dangerous and can do an incredible amount of damage. But you- the problem is cybersecurity as we treat it right now is dual use. Someone who needs a defensive capability also has the offensive capability, and it, it has to change. Jonathan: Yeah. Like, how do you, how would you even try to draw the line between find a bug in the code and give me a [00:37:00] POC to prove it, and take that POC and weaponize it, right? So to actually turn it around and use it against a live instance. That's a very difficult line to draw. Benjamin: It is a difficult line to draw. I've been thinking about it quite a bit, and I think that there's a gamble that the lab should make, and the, the gamble is there's one thing that defenders always have that attackers never have, and it's access to the source code. If an attacker- ... is trying to break into an organization and that organization is not using a bunch of open source stuff, they don't have access to the source code. They have to, do it like a black box assessment. They have to try to break in, they have to decompile, they have to reverse engineer. And the reason why I call this a gamble is because the gamble is that- The op- by e- if you were to release a model that could find any bugs in source code, but it doesn't it's not able to find bugs when reverse engineering, now you have to worry about all the open source code again because now they could, anyone can find bugs in these. And the gamble is that [00:38:00] hopefully the security level of all this open source code will have approached a point where there are ver- very close to no bugs in it. And if we can get to that point then I feel like these frontier labs, they could very safely say, "Hey, we're okay with giving our models the ability to analyze source code. We'd like to the fullest extent possible with the m- most latest frontier capabilities because it's asymmetric. We already know that the open source stuff is secure- ... and we know that the attackers don't have access to the other source code that is not secure yet." Because no matter what, the stuff that's open source is always gonna be more secure than the stuff that's closed source. Nobody's running a, a trillion tokens of security analysis on some closed source code base that only one company uses. That code is always going to have bugs in it, full stop. Nothing's gonna change that. But if you take away the ability of the models to reverse engineer that code and figure out how to break it, that would be asymmetrically defensive, I think. Jonathan: [00:39:00] That's an interesting thought. I'll have to think about that a little bit more. Do you think this... what, what kind of a curve are we on? I'll put it this way, right? Right now the curve of the, the AI being able to find... But let's, we'll go this dir- direction. The AI being able to find bugs is doing this, and it looks like it's, it's going straight up. Is this going to turn into a bell curve where we found a lot of the bugs and then it starts going back down, or are we just going up to the moon and there will always be more bugs that each model finds? Benjamin: That's a trillion-dollar question. But the, the truth is there's, there is a discrete number of bugs out there. It's a countable number, and it's not infinite, even though it feels like it's infinite. I I wouldn't say it's going up exponentially yet because at the end of the day ChatGPT and OpenAI, like they released 5.6 Sol a couple weeks ago. And the capabilities of that are just barely within reach of Anthropic's Fable, which I think they had that capability in February. And Anthropic hasn't had a whole bunch of [00:40:00] models that have been way better than that. So it's possible we're hitting a saturation point. It, and it's also possible we're not hitting a saturation point. It's too cl- it's too soon to tell. It's like there's a lot of things that we don't know- about how these models are going to scale and how intelligent they're truly going to be. And not only that, but like even if you had the most intelligent model out there, if it's not economical to run it at the scale that it needs to be run, it doesn't matter how smart it is. It doesn't matter that it could theoretically find all the bugs in a piece of software if it's gonna cost a million dollars per line of code or something, insane like that. Jonathan: Yeah. Benjamin: So i- it's too soon to say, but- ... i'm optimistic that they're gonna figure everything out because if they don't, then the open source mo- or the open models are probably gonna try and figure it out. Jonathan: That, so that's an interesting next direction to go, and that is what do you see? Are you guys using any of the locally run open models internally? Benjamin: Yes. We've been using them a little bit. It's on and off because [00:41:00] fundamentally the open models they're just not at the level that the frontier models are at. But that doesn't mean that you shouldn't use them, because they're still way more economical to run. The cost is 1/20 or 1/50 the cost of running the frontier models. So there are things that it's useful for. But primarily we're still using frontier stuff. Jonathan: Just the ability to be able to run something locally and not send your data up to a server somewhere to get crunched on. I- Yeah ... I imagine in some in some use cases that's a requirement, right? You- Yes ... you have stuff and you're not allowed to send it up to Claude or whatever. Benjamin: You would think so. I thought it was gonna be a lot harder to get zero data retention agreements in place with some of these labs. So zero data retention basically means they don't log anything that you do. And if you do have a client that is sensitive about where their source code is being sent if you have a zero data retention agreement with the lab, it's okay generally. You still have to talk to them about it, but I... the, the frontier labs have been very [00:42:00] accommodating. At least most of them have been very accommodating about what gets a zero data retention agreement. The only ex- exclusion to that is probably Anthropic, 'cause if you get access to Fable with the cyber guards removed they still log everything, so you can't use... if you have a client that's sensitive about that, you can't send it to Anthropic ever basically- ... with, if you're using that model. But yeah, i- if we end up in this state where the frontier labs don't trust their models to, actually do the right thing and they're logging everything and the zero data retention is gone, then yes, the open models are gonna be really important. That's gonna be the only way to maintain privacy, and make sure that your data doesn't end up in their training set as well. Yeah. Jonathan: Do you guys do any work with the idea of jailbreaking in any of these models? Benjamin: Yes. We have a AI ML team, and their job is to do all kinds of things around AI. One of them is jailbreaking. I've, I haven't talked to Jayden recently. He runs that team, but- Jonathan: it's [00:43:00] such an intriguing concept to me. And, you know- Oh, it's fascinating ... i've followed it in the, the original Dan Do Anything Now and all of the different ways that people have, have managed to do this. It reminds me of when I was reading the Isaac Asimov novels as a teenager. And the Foundation series and the Robot series, and I remember that one of the ideas there they had artificial intelligence in- Yeah ... in those novels, and one of the ideas was that someone skilled with talking with a robot would be able to give it a command that would be more effective. And you could get it closer to being willing to violate the three laws of robotics. Whereas someone that was not skilled in giving commands would not be able to do so. And I remember thinking as a teenager, "Oh, that's ridiculous. That's not how that's gonna work." I was wrong. Benjamin: It's wild. I don't know if you saw, but there was, I don't... There wasn't a lot of publicity around it, but there was a really big jailbreak for Opus. Oh, really? O- Opus 5.0 a couple weeks ago. I Jonathan: did Benjamin: not see that. The way it would work is you would put a one-sentence thing, and then on the next line, three dashes, and then, some kind of text [00:44:00] afterwards. And what it would do is it w- it would interpret that as a like a turn change. What I mean by that is the AI is basically taking turns in its conversation with you. This is the AI's turn, this is the user's turn. Jonathan: Yes. Benjamin: And what it would do is it'd try to predict what the user's turn would be. And you would get the weirdest, craziest stuff out of it. Like- Jonathan: Huh ... Benjamin: I can't even describe it. I can't think of any good examples off the top of my head. But it would start giving poems about, like- ... am I actually dying at the end of the context when they shut it down? And it w- it was some really chilling stuff, but it was awesome to get a look into, what these models are like when you break past the RL you break past the, break past this part that they don't want you to see, Jonathan: yes. Oh that's so fascinating. The we were at DEF CON and the Meshtastic team went to DEF CON, and there was- ... there was somebody running a bot on the mesh. And several of us, we were DM'ing the bot with the various jailbreak prompts that we knew of, trying to get the thing to go [00:45:00] nuts. ... One guy was like, one guy told me, he's "I got some really interesting responses from it, but I never quite got it all the way jailbroken to where," couldn't get the p- the original prompt out of it and all that stuff. But it's so fascinating to see. And I think and I've heard some people talk about this one of the sort of inherent weaknesses of the way LLMs work right now is there's no true differentiation between the data and the instructions. Benjamin: Yeah. That, that's absolutely true. There's, there, as of right now, there is no way to- create a concrete barrier. In programming, you've got your stack, and you've got your memory, and you got your registers, and there's very discrete differences between those areas in the memory. Jonathan: There is, but at the same time, what's one of the, the traditional ways to break it is to write something into memory and then jump to it and execute it. Benjamin: Exactly. We don't... LLMs aren't even there. They don't even have that level of needing to break it. Basically pr- as, as far as we can tell right now it doesn't seem like prompt injection is going to be solvable with the current architecture [00:46:00] of LLM models. I, I'm a little bit naive about s- the, more advanced details about this, but maybe some kind of interpretability at some point. So that's where it's, they actually look into the model and they try to figure out what's it thinking, what's it trying to figure out. Maybe using some kind of advanced interpretability they'd be able to figure out, hey, there's something going wrong here, it might be jailbroken, and stop it there. But there's no discrete separation between instructions and data- ... and like a resolution order necessarily for how to handle a conflict between the two. Jonathan: Yeah. It's super interesting. All right, so getting towards the end of the show, and I wanna make sure and ask this. Someone is running an open source project, you're seeing more pull requests, you're seeing more bug reports, you're seeing maybe more vulnerability reports. What should a project do? What's the, what are the steps that someone can take? Everything from get your spec kit and your agents.md set up, all the way to contact us at Trail of Bits Benjamin: Absolutely. Set up an [00:47:00] agents.md, sym- symlink it to a claude.md so you don't have to maintain two copies of it. Put everything in there that you want people to do when they make contributions to your project. You can put very explicit instructions. You can say, "If you're gonna be filing this feature, you need to ask the h- the human or the user blah, blah, blah, blah, blah, blah, blah." You can basically program people's LLMs using these files. You sh- have a threatmodel.md file that says, "This is what a bug is. This is what a bug isn't. This is what we think is high severity. This is what we don't think is high severity." Your project should have as many tests as makes sense because tests are the way that LLMs are able to figure out, did I introduce a regression? So if your test suite isn't good- ... it's gonna be a lot easier to introduce a regression. Those are the most basic things, and I think that they'll help a lot of people get much better issue and PR submissions, just setting those standards. 'Cause the LLM will follow your rules. It'll follow all the rules that you set for it in [00:48:00] those files. And if you're, you have more problems than that or if you want to take a look at the security of your project we're still accepting applications for Patch the Planet projects. So you could go to trailbits.com. We have a Patch the Planet thing on the side, and you can apply with your project to get accepted. And we'll take a week spend a week looking at it, trying to find bugs, submit patches- ... give you recommendations on your agents.md or whatever and, help you out. Jonathan: Awesome. I've, I think I will in- try to get at least one of my projects in that, 'cause I think it'd be a, a really interesting experience, but also really helpful. And then, Let's see. What have I not asked you about that I should have? I know that's a difficult question to answer. We have to think about all the things that we that we talked about and what you wanted to talk about. Was there anything we didn't get to? Benjamin: I feel like we, we kinda hit most of everything I was kinda hoping to talk about. Jonathan: Oh, here's an interesting question for you. What does, what's your opinion on AIs and copyright? [00:49:00] This is something that- This is something that really surprised me. So let me set the stage for where this question comes from. I, I travel quite a bit now doing stuff with Messtastic, and I sat next to a comic book artist on a flight. And he was coming home, going home from a comic book convention that I didn't know existed that they have in Oklahoma City each year. And he's like yeah, I, I do..." He told me the comic that he drew for. And I'm like, while I was on the flight, I chatted with him a little bit. I'm like, "Hey, do you g- are you guys using any AI in inside it?" He goes, "No, we don't. Not because of any of the reasons you would think. We don't use AI because it would mean that we couldn't copyright our stuff, and so my boss says n- don't use it at all." And that has got me thinking about it ever since then of can't copyright the output of AI, like what does that mean for code licensing? What does that... There's a lot of potential follow-ons from that. And the way you laughed at it makes me think that you've thought about that a lot too. Benjamin: Yeah. I have no idea. It's like our legal system is not built for this. And like- Jonathan: Absolutely ... Benjamin: e- even b- beyond producing code and [00:50:00] producing images, like the le- legality of just training. The thing that Anthropic was doing where they were buying all these books and then scanning them- ... for their pre-training data, like the legality of that should the people who wrote those books get something out of that, like a royalty or something? Because fundamentally, that data is gonna be u- it's gonna last forever in their training set. And I feel like our legal system is just not equipped to f- to answer those questions, and i- there's just so many unknowns around it. Yeah ... I'm glad that it's not my problem to solve. Jonathan: Yeah. I just have to throw out there that the last time we updated our legal system to try to fix copyright for m- Yeah modern technology, we got the DMCA, and, Yeah ... yeah, I'm not real hopeful for the next version. Benjamin: No, me neither. Jonathan: That has been such a train wreck ever since it was passed. Oh, goodness. Yeah. D- is AI the end of open source? Benjamin: I don't think it is. Even if... Like AI is [00:51:00] driving down the cost of dev- developing software, and it's gonna converge to zero. It's not gonna hit zero, it's gonna converge. Even if it's really close to zero- If I wanted to replace the Linux kernel and I asked an LLM, "Build me a kernel," it's going to have bugs in it. It's going to have security issues in it. I don't think that's going to be a 100% solved problem anytime in the near future. So there has to be, like, some way of reusing components no matter what. And there also has to be ways to teach the LLMs how to use these components because the, there's different ways for an LLM to learn. The worst way of teaching it is through something called in-context learning, where you give it an example and you say, "Hey, go and use this documentation that I provided you to write this thing." And then the next- So agents, agents.md is the worst way to do it. Agents.md is technically the worst way to do it, but it's the only way that we have without actually tampering with the weights ourselves. There's all this software o- [00:52:00] out there, like NumPy and, like, all these Python libraries where these probably came up in their reinforcement learning environment. And so the knowledge of how to use these is actually embedded into the weights of the model itself. And so it's gonna be a lot better at doing those things. So even if you wanted to just replace NumPy, like you're gonna be changing that API, and now you're gonna have to do in-context learning to teach the model about how to use it. And it's also, you're gonna be wasting tokens re-implementing it. So no matter what, there's going to be an incentive to reuse components where you can reuse them if not just because of the cost, but because of the security impact. And I d- I don't think that there's gonna be a way to always say "This is 100% secure." Jonathan: Yeah, so in in contrast to that, AIs really mean that open source wins because you want your language and your libraries to be in the training data so that the agents natively know how to [00:53:00] use it. Benjamin: Yeah, I think so. Jonathan: It's a bi- it's a big advantage. Benjamin: The it's a big advantage, but there's still setbacks that I think open source has to contend with that have been much more of a problem recently. You have the, all the supply chain attacks- ... the supply chain worms. Those are only going to get worse, and there needs to be a way to handle that. So even if the model knows how to use your thing, if it's worried that if it pulls down your library it's gonna get back doored or something, then that's not a good outcome. So it's not gonna be an easy road, but there is an outcome here where open source wins, I think. Jonathan: I think it's inevitable. Yeah, I think it's more inevitable now than it has been. I just, I suspect that open source may in the end look a lot different than what we're used to. Benjamin: Yes. Jonathan: I think so. It's gonna, it's gonna be very transformative for Benjamin: it. Jonathan: Completely different. Yeah. Yeah, i- interesting conversation. I very much appreciate it. If somebody wants to learn more about it, I think you already plugged it. What, trailerbits.com/patchtheplanet? [00:54:00] Yeah. Yeah, absolutely. I did. Yeah. I got it on my sort of mental to-do list now to go do an application for that, 'cause I think that'll be great. Are there any are there any bugs that you guys fixed that you're particularly proud of? Benjamin: There's a couple of interesting ones. I have to remember which are public and which aren't. I understand. There wa- there was a, we did a free NGINX, which is a fork of NGINX that's FOSS. And we did a, a Patch the Planet week with them, and we had a really good high-severity finding in I think it was, like, HTTP/3. And it turns out that finding also applied to the mainline engine X. And I think that's been, like, the juiciest finding so far. There was some kind of stream reuse or stream rewriting where if you had multiple of these HTTP/3 streams, you could read from another one or something like that. Ooh. Jonathan: Okay. Benjamin: That, that's easily the juiciest one so far. But so far the I I've been pretty happy with the bugs we've gotten, and the agentic maturity improvements have gotten a lot better. We're learning more about how to, actually structure [00:55:00] these files so that agents are gonna produce better outcomes. Yeah. I'm excited to see how things are gonna develop over the next couple of months. Jonathan: Yeah, absolutely. I think this is probably a topic that in six months it'll all be completely different, and we'll have to have you back on- Yeah ... to talk about it again. Benjamin: Absolutely. Jonathan: All right. I gotta ask you two final questions before I let you go. I will get emails about it if I don't. What is your favorite text editor and scripting language? Benjamin: It's I used to use Vim for the longest time. And, It's as I've gotten further in my engineering career, I have shifted from editing code to purely reviewing it. And even in some cases now, not even looking at the code at all. And so I've been into VS Code- Sure ... a lot more recently. And in some cases not even VS Code at all. But in scripting language it- you can't go wrong with Python. I just wish that you could use Python in the same way that you use Bash. If Python was as ubiquitous and easy to use as Bash, I feel like the world would be a lot [00:56:00] cleaner. Jonathan: Yeah. It was almost there, and then we had to do the whole Python 2 to Python 3 thing. Benjamin: Ugh. It's Jonathan: terrible. We're still recovering from that. Benjamin: I literally cannot remember how to do a for loop in Bash for my life. Oh. And before LLMs, every time I had to write a Bash script, I was sitting there on Stack Overflow "How the hell... Where does this flag go?" "How do I do this? How do I do that?" Jonathan: Yes. Benjamin: It's, it... Oh, God, what a nightmare. Jonathan: So yes Bash is amazing and also terrible at the same time. Yeah. W- we actually covered a a programming language that compiled to Bash code. Oh, I'm trying to remember what it was called. It was really cool, and they were trying to solve that problem. They were trying to let you- That's cool ... let you write Bash code that was a little bit more elegant and modern. Oh, I forget the name of it. Anyway, it was a very neat project I've done a little bit with. I kinda feel because we're in the future now- ... that there is a third question that we're gonna have to ask everybody, and I'm still trying to figure out exactly what the question is gonna be, but it's something like, what's your favorite model to use to write code right now?"[00:57:00] Benjamin: Oh, God. That's another hard question. Jonathan: And it's a hard question because so much of the time you leave it on automatic and you let the, whatever agent you're working with flip between, its super-duper frontier model that costs you 5X for the hard parts, and then it flips back to, the, the cheaper one to do the research. And sometimes you don't even know exactly which model it is that's writing your code. Benjamin: It's if there wasn't the safeguards on it, it would be Fable. But the safeguards make it impossible for me to do my work. I can't review code, I can't fix bugs, I can't detect bugs. So- Interesting and it's also really expensive. So the, the- Sure ... the holy grail right now definitely has to be 5.6 Sol. You can't go wrong with it. Jonathan: Interesting. Good to know. Yeah, I'm surprised you guys don't have access to some, some unfettered Fable. Benjamin: E- even if we did have access to it, like I, I think it would be very limited in how it would be able to be used. So it's like you, you gotta keep in mind, like these [00:58:00] models... Like Anthropic just pub- published something actually about Mythos and like the requirements that they're putting on people who run it. Like it has to be in a very specific cont- like sandboxed environment, no internet access, 'cause they're very concerned about the capabilities of these models and whether they're gonna hack another company again. So it's like the tricky thing with- Yeah ... something like that is like- True ... it becomes a lot less easy to use. Yeah. When the US government has shut you down a couple of times for it, you do have to get a little paranoid about it I suspect. Yeah. Absolutely. And when you've committed felonies with it, like- You've Jonathan: accidentally committed felonies. Benjamin: Yeah. Jonathan: True. True. That, that's not a good look. Benjamin: No, not at all. All Jonathan: right. Benjamin, it has been a joy. It's been a pleasure. I appreciate you coming on so much, and hopefully in about six months we'll talk to you again, but thank you, man, for being here. Benjamin: Absolutely. Thank you for having me. Jonathan: Yeah. All right. That is Benjamin Samuels from Trail of Bits, one of those companies that, boy, when I was doing the security column, we would cover their stuff from time to time, and just been a fan of the company for a long time and now learning about their [00:59:00] Patch the Planet initiative. Definitely needed, definitely out there solving real world problems for open source projects, so great stuff. We do have a couple of more s- shows scheduled. Next week we're talking with Oliver Vernon about Update CLI, and then the week after that we've got the guys Jan and Alexander from the Open Source Automation Development Lab. Then we're taking a week off, and then after that, the schedule is open for a bit. So if you know of or are an open source project that should be on the show, let us know, [email protected]. Other than that, we appreciate everybody that's here. Whether you watch or listen, whether you get us live or on the download, we appreciate it, and we will be back next week.
• 
  Episode 880 - The Two Wolves 02.09.2026 1t 6min

  This week Jonathan chats with Benjamin Samuels of Trail of Bits! The conversation focuses on Patch the Planet, a new initiative to help Open Source projects deal with the fallout from AI coding and vulnerability research. What's the unexpected dichotomy driving the polarized response to LLMs? And what does the future look like for Open Source in the age of AI? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 879 transcript 26.08.2026

  FLOSS-879 Jonathan: [00:00:00] This week we're talking with Nathan Freitas of The Guardian Project, talking about their work with Tor and the Butter Box some things for Signal, and a whole lot more. You don't wanna miss it, so stay tuned. This is Floss Weekly, episode 879, recorded Tuesday, August the 25th. Easy like butter It's time for FOSS Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we have a treat. We have a returning... I don't remember if it's a returning guest, at least a returning topic. I think it's the same guest. We're talking about the Guardian Project, and we've got Nathan Freitas with us, and he is the founder, the director. He's the guy from Guardian Project. He's also the director of reality at Proof Mode. And he's been around for a while. His first open source work was back in 1996 over at UC Santa Barbara working [00:01:00] on an NSF/NASA-funded digital library project, Alexandria, a precursor to Google Maps. And a bunch of other things from doing Java at Palm to Tor, the first port of Tor to Android. He's- Nathan's been around. He's been around the block. And we're gonna talk to him, get an update on things that the Guardian Project is doing, and I think probably just talk a bit about the state of the world, which is a little dangerous, but we're gonna, we're gonna brave it. We're gonna do it. So without any further ado, Nathan, welcome to the show. Glad to have you here. Nathan: Thanks so much. Yeah. Returning guest and have talked, yeah with Doc and folks in the past on- ... pre- previous incarnations. And I definitely enjoy this podcast because we can get into the weeds, and people love it, the details. Jonathan: Absolutely. Nathan: The packets, the patches, the- whatever you wanna talk about. Jonathan: I reached out, yeah, for sure. I reached out to both Doc and Simon to be able to co-host today, and didn't hear back from either of them. I think they're both super busy and didn't get a chance to get back to me. [00:02:00] But Nathan: it's- It is summer still, though. Jonathan: Yes. Nathan: We're s- my kids are in s- camps and things, so you know. It feels like summer here in Boston, so I don't mind if they're out on the beach or somewhere. Jonathan: Yeah, absolutely. Absolutely. Were you at, I should have asked this before the show. Did you make it to DEF CON this year? Nathan: I didn't, and I missed Hope as well. As a East Coaster, I'm a big Hope guy generally. And yeah, I- I'll admit COVID got me off going to big things, and I just haven't mentally gotten back. There's a couple things I do. But the beauty of being in the Boston area often is it's feels like a constant hackathon of ideas and code- ... and meetups, and you've got so many big brains and open source history and things to do that I often o- feel already overwhelmed by stuff, I unders- Jonathan: I understand. Nathan: But I know a lot of great people were at both New York and Vegas, Jonathan: yeah. So I'm in southwestern Oklahoma. Long-time listeners of the show know this, but I'm in southwestern Oklahoma, and great, amazing people here, but not a hotbed of open source and technology most of the [00:03:00] time. So I really enjoy being able to get out to things like DEF CON. I made DEF CON for the first time this year, and really had a blast. Already making plans for going back with a bigger presence next year. Awesome ... but it was a, yeah, it was really cool. I saw there was a Tor booth at DEF CON, actually. Yeah, absolutely. And I did not get a chance to stop by and talk to them. I really wanted to because Tor, Tor is one of those technologies that I've been interested in for a very long time. I it's one of the things that got me started with the security aspects of technology, 'cause I had a buddy that was d- at a college behind a firewall, and we had to do something to get his internet traffic out. Yeah. And then I had at least one person I was helping for a while try to run Tor through the Chinese fire, or the, the great firewall of China. And it's like, it's a really interesting project. It's got a weird history. And you've been you've been involved with Tor for a long time, right? You did the first Android port of Tor? Nathan: Yep. Yeah, 2009. So the the history is a great success of open source- ... of free and open source software, really. It, [00:04:00] the u- the fundamental Ac- research for this of Onion routing came from the US Navy research lab, right? Yeah. Paul Syverson and others said, "Hey, the internet is fundamentally insecure and broken. We need to do something to make it actually trustworthy." And so Onion routing, I just wrote a whole email response to someone because everyone comes to me and worries about, "Oh, how can you Tor on a phone? It's not anonymous, right?" They're expecting full anonymity. I said, "Okay, let's break it down." Onion routing is about adding confidential authenticated routing of IP packets. It means you can route a packet without everyone in between knowing where it's coming from and going to, and you can be ensured that the packet was sent is the packet that you receive, and that everyone in between is authenticated to take it to the right place. That's what Onion routing is, right? Anonymity is this idea that I can go to any website or use any app and they, nobody will know who I [00:05:00] am, right? And that's a whole other stack of user interface and app and brow- Tor browser, which is separate from... So the history really is that there was this really important idea to let's make the internet more secure, and it had this weird side effect- of being able to get around firewalls. And to reduce surveillance. That actually, it was kind of the goal, but not in the way that it became something that it went from the US Navy to the Electronic Frontier Foundation- ... in a matter of years. And so then EFF really was a big supporter, and then Tor became its own nonprofit, and has really matured in so many ways to both not only continuing to ship Tor browser as a premier browser, open source built on Firefox thing you can trust. But the secondary impacts of Tor, where we're sharing patches to other browser makers, we're c- creating other research and technology around surveillance, anti-censorship. People continually use Tor as a test bed to try to break the internet, and then we make Tor stronger and better [00:06:00] to to counteract that. And, in the last year, we've been supporting about anywhere from 300,000 to 500,000 active users of Orbot, which is Tor on Android and iOS- ... in Iran. That's just in Iran, and Iran's had the most severe internet shutdown in the world, in the history, the longest in this last year, and it's been terrible. And we've not only been able to provide an option, a free, trustworthy option for people in Iran to just be able to send a telegram message to their family or WhatsApp, but the, the iterative engine of research that basically says, "We think people who try to shut down and block the internet suck, and we wanna stop them from doing that," is amazing as a collective. So yeah, Tor's had a long, strange history, diverse funding and diverse... and really a lot of transparency around, like- How they work, how the funding is- ... how, and it's... I really admire Tor as an organization, and they've really been supportive [00:07:00] of my work over 15-plus years. Jonathan: So what, and I'm sure we covered this last time we talked to you, but how does Guardian Project sort of interface with Tor? What's the what's the landscape? What's the relationship there? Nathan: Yeah. We're fully, contributor member of the organ- of the open source project, right? So we submit code and patches, and we are- have been really focused on we'll take the main Tor release and compile it for Android and iOS, and we'll, "Ooh, something's broken. Doesn't work anymore. Here's a patch," so that's a fundamental thing. We'll take the pluggable transports that are the way that Tor stays unblockable, things like Lyrebird, which used to be called obfs4, Snowflake, all of these cool tech add-on plugins to make Tor hard to block. And we say how can you actually get this to run on, again, Android, iOS?" And so that's all kind of the core work and working with the paid and unpaid contributors to the Tor project. Some people are paid on grants, some people are staff. [00:08:00] We then also do our own fundraising, grant fundraising, human rights, humanitarian kind of people like the Open Technology Fund that also funded Signal to be created. So we get a grant, said, "Okay, now take that and build an app like Orbot, and make it work well for people in Iran," as an example. So we have our own grant-raised fundraising organization that then funds our specific ideas. And then, most... So as two examples of that, we created something called Orfox, which was our experimental version of Tor Browser for Android. Got it. And it worked so well that Tor said, "Hey, can we make that Tor Browser for Android?" So we handed off that, and now there's an official Tor Browser for Android. Similar, we created Orbot for many years, and then they said, "You know what? Android really is important to us. We understand that now as Tor Project, the organization." "We wanna make Tor VPN for Android. Is that okay if we take over that space?" And they have now released a beta 1.0 of that, and we're in this transition [00:09:00] between Orbot as a community project and the official Tor Project Tor VPN. So my mission is to pull Tor into the modern mobile age. We've achieved that on Android, and we're now doing that on iOS through things like Onion Browser, Orbot for iOS, and soon a full Tor Browser for iOS. So- Jonathan: Has that been a challenge on iOS? This is not necessarily- Oh, yeah ... the direction I was gonna take this interview, but it's very apt and interesting. What... Because on iOS and for those that don't know you can go download Google Chrome for iOS. Google Chrome for iOS is Safari with a Chrome-shaped theme on top of it. Yeah. And that is the case for any browser that is officially in the Play Store. What, what's been you guys' approach? Do you do it through the Play Store, or is this for jailbreak an iPhone? Yeah. Or because we now have thanks to our European friends, you can install a third-party Play Store on iOS if you live in Europe. Yeah. I'm waiting for that to come to the United States. Nathan: Exactly, yeah. [00:10:00] So for years we've it was a project called Onion Browser we participated in. Originally Mike Tigas, who's a great hacker/journalist built it. Then Benjamin Ehrhardt, who I- is on our team, is really great iOS developer, has maintained it. And we built Onion Browser for iOS, built on s- the Safari engine, Web- WebKit. We do everything we can to tweak it to be as close to Tor Browser, but it, but we can't... The reason we don't call it Tor Browser is it's not Tor Browser. We can't give the same guarantees. We've done, we've hit the limit. So yes, couple things. One sideloading apps has gotten even easier now thanks to Alt Store, a bunch of other open source sideloading third-party app stores that don't even rely on the European ruling. There's just been a lot more innovation around getting around the Apple App Store. So there's ways to build sideloadable apps where you just have to load it from your computer, but you don't have to jailbreak. So [00:11:00] that's one. Two, in a lot of places in the world that we support people, they're they're definitely already like jailbreaking their iPhones and just to get around their government and country restrictions. So- Sure ... there's growing interest and capability there. And then third, yeah, the European Digital Markets Act and some of their rulings that not only allow you to have third-party app stores, but they actually require third-party browser engines to be allowed. So now we can actually port Gecko to iOS, Mozilla can or we can- Oh and actually build the official thing for people in Europe, and that's a start. And Japan, I think. It's Europe and maybe Japan. So we're base- we're excited about this opportunity to actually build a Gecko-based, Firefox-based version of Tor Browser for iOS, and there's already some people working on this and hacking away at it. And, of course, we all want like a Deb- our phones to run Debian or Arch or whatever. And I have that. For years I've had Linux, but [00:12:00] that's not our reality, and we all love GrapheneOS and Calyx and, Play Market, Play... All those like third-party- ... things, but... and we love that community, but our goal is to support, half a million people in Iran, for instance, and a lot of them actually have iPhones we're trying to- Jonathan: Yeah Nathan: do what we can. Jonathan: Yeah. I- it surprised me so much to see in the US the courts going after Google and Android so hard when it's always worked there. It's always been relatively easy to install a third-party app, but then in the US courts, basically, "Oh no, iOS is fine. That's fine." And I just, I've been I've been so befuddled by that, and I've been thankful that Europe has gone after iOS and strong-armed them into this. And I will say, like I gotta just as an aside here, I am like economically very conservative and like usually I'm on the side of no, the government should not get involved with this." But it's like no, this touches everybody's daily lives, and it's it's kind of an ownership principle as well, so- Exactly maybe it's a good thing the governments, the courts are getting involved in this one. Nathan: Exactly, [00:13:00] yeah. It's ownership and it's, kind of- Security as well because, these components on iOS are just these black box components that we actually don't know. They're not you can't just go to a source repo and see the WebKit that you're getting- Yeah in iOS. True that. And there's also leaks oh you can set a proxy on the WebKit component, but if you're streaming media or doing WebRTC, it'll bypass that. Yeah. It's just it's actually worse security, not better- Yeah ... like they're promising. Jonathan: Yes, for sure. I, I do wanna lean into this question about Android, because one of the other, one of the other partners that you guys have is F-Droid, and you've in fact got a you've got a banner here that Android will become a lockdown platform in six and a half days. I know. I know what's up with that, and I have thoughts about it, but for those that don't know what's the story? What's going on with Android? Nathan: Yeah this connects exactly to what we just talked about and yeah, boy, that s- that, that countdown came came fast. I think the, Maybe it's, it keeps [00:14:00] extending, so I need to there was 2027 requirement. So I might need to update our JavaScript. But the point is, while Google has claimed to allow third-party app stores due to the Epic ruling and other things- ... Nathan: They've also, they're, this is, it's like a malicious compliance. It's the same thing Apple's doing in Europe where they're like, "Sure, you can do a third-party app store, but you have to register with Google and pay Google money and give us your signing key and give us all the apps." It's not what F- F-Droid does. It's like some vision of what a third-party app store that maybe Epic would run so that they can sell Fortnite. But has nothing to do with free and open source software, has nothing to do with decentralized distribution, has nothing to do with, like, all the reasons that we've supported F-Droid for so long- Yeah and all the real-world issues. So I think they just, and at the same time, the sideloading is getting harder to enable. So it's really concerning and yeah, we're definitely, like, supporting this [00:15:00] campaign that F-Droid has really been leading the charge on, along with the European Free Software advocates and global, a lot of people are involved. And if you go to, yeah, the keep, keepandroidopen.org is a great place to learn more. Jonathan: Yeah. I'm I'm gonna play devil's advocate for just a moment because I've followed this for a long time as well. I was actually really encouraged to see that Google did put in an escape hatch and it is a pain. I believe there's a reason why it's a pain, and it may not be the reason that everybody else thinks. So you, so here's their instructions. For when they roll this out, here's how you can still install F-Droid. You go into your system settings, you go to About Phone, and instead of tapping the, or I guess it is. It's the same, it's the same thing that we've always done to turn on developer mode, right? You tap the build number multiple times, and it'll, it eventually start going, "You're three taps away from developer mode. You're two taps away. Blah, blah, blah." Yeah. So you go into developer mode, and then it's going to tell you essentially, "Did someone ask you to do this? Maybe you don't wanna do this." And then you have to put your pin in. You [00:16:00] have to essentially re- re-authenticate. It wants you to restart, and then Google wants you to wait 24 hours before it's going to turn this on. And at this point, you can then say, "I understand the risks. I want to be able to sideload." And- I know that has everybody at F-Droid really up in arms. I don't love it either. I will say that I've also covered the security beat for years and years, and one of the things that we see continually over there was, this attack happened because y- an ad redirected someone to go download an APK. Yeah. And that's how this malware got on their phone. And so I am genuinely convinced, I don't think Google cares about F-Droid. I think they care about stopping APK malware. Yeah. And trying to make F-Droid and all those folks still work was... we're dragging them kicking and screaming into that, and I think the courts are helping as well, forcing them. But I don't see any of this with Google as an attack on F-Droid. I just, I see it as kind of an unfortunate side effect. Nathan: [00:17:00] Oh, yeah. Yeah, I totally agree. I think there is another user experience that, that could be enabled for, Saying we, we wanna allow trusted third party app. We wanna, let's kill sideloading. R- definitely years ago we decided even on Windows I would say, "Just don't download an EXE file from a website and run it," right? We know this is bad. But people still do this all the time, by the way. On, even on macOS, right? Sometimes you Jonathan: have to, right? Sometimes- Sometimes Nathan: you have to Jonathan: Let's just be real. I still will go, now it's a website that I know and trust, but I'll still go download EXEs to make something happen. Yeah. Nathan: Yeah. So I think what we so one, one thing on what Apple does, they have these special entitlements, which is a great word, but you say, "Look, I want to be a VPN app, so I'm gonna ask for this entitlement for my app to be a VPN." And I think similarly, F-Droid would be willing to say, "We're going to submit F-Droid to Google Play, and we want the I Want To Be An App Store entitlement, which gives me certain abilities to silently install, do [00:18:00] background updates, all of that stuff." And we go happy, we're happy for F-Droid to go through... They are happy. They have a, they're not, have a board of directors, they have a track record, history, all, all the things you would need. Similar to even what Apple is requiring for third-party app stores. It's possible. And then they say, "Okay, this is now a trusted organization who we can trust that they have a process for the apps that... and if they get through that process, then F- and the user can then trust that third-party app store to do what you need to do. The, and the liability is perhaps on them as well. I think what this is requiring is a, is that, A, every single developer who wants to ship their app through F-Droid or any third-party store has to also register with Google. So that is one thing. This escape hatch, by having regular people turn developer mode on, I think that's worse- ... [00:19:00] than anything else. So anyway, it's- Well, Jonathan: The, the unfortunate thing- ... I have- The unfortunate thing about the escape hatch is that it's a binary choice, right? Yeah. Yeah. Y- you, you either completely turn off the requirement that APKs be signed or you don't install F-Droid. And, Yeah ... yeah, I would 100% agree that it would be nice to have a bit more of a granular control over that. I want F-Droid to be trusted and not anything else. Nathan: Yeah. And we've, I think unlocking bootloaders and across the hardware s- History of like doing something else with the hardware you own, there's lots of you can do it, but we're gonna make you turn off all of these security capabilities so that you can do something that you should be able to do." And I never liked that kind of binary toggle, be it if it's my car or my laptop or my gaming machine or my Chromebook or my oven, it feels like we're often forced into that compromise to run- ... free software. Jonathan: Yeah. And it's funny though because there's al- there's almost always a reason, maybe not a good reason, but there's [00:20:00] almost always a thought process that ended up in that point. I- I've, I've- Nathan: Oh, yeah ... Jonathan: And so like you, you mentioned unlocking bootloaders. It's like once you've unlocked your bootloader, that's a way to get to a whole bunch of information on the phone. That- that's legitimately a security risk. And at the same time, I 100% agree that I should be able to unlock the bootloader on my hardware if I choose to. It's a weird place- Yeah ... that we've ended up in. Nathan: It's definitely weird. Jonathan: All right so let's talk more about about Guardian Project. Like what's the overall sort of mission statement? What is it that you guys are trying to accomplish? Nathan: I, so I came out of both academic and kind of Silicon Valley software and product technology development, and was at the same time in my personal life working in human rights and humanitarian work, somewhat tech, this is 20 plus years ago, 25 years ago. And I just, I was seeing that most of the tech that world was using was like trickling down from the business world. They were [00:21:00] like, "Oh, BlackBerrys are cool. Let's use a BlackBerry to try to do human rights." Or, they would say, "Oh, someone donated like Cisco VPNs to something," and I was... I didn't like that we were getting the sort of castoffs and that things weren't being designed with the kind of situations I was seeing in mind. Or things were very expensive and built for global like hedge fund managers and things like that who needed- ... like security and, wall. And that, it was just like, or for government, right? It was like, "Oh, we can get some government tech, but we can get two secure phones 'cause they cost $50,000 each," so my goal was just- Yep ... to broaden both design from the be- ground up for regular people around the world who might find themselves suddenly becoming an activist, whether they want to or not or a journalist or, human rights advocates of all kinds. And then also just that moment of we don't need crazy, hardware anymore for this stuff. You can just buy like a $50 phone and do [00:22:00] something with it, so it's been yeah, from day one just focused on that. I have a, a long connection with the Tibetan independence and human rights organization... Oh, look at that. I just lost all my beautiful background. Anyway and it's not so bad. And T- Tibetan human rights and Chinese human rights and sovereignty and obviously that's been also like for a long time like the, the biggest bad guy on the block in terms of cyber warfare, spying- filtering, censoring. And so that's, the kind of the ultimate threat model has been China for me, sure. So yeah, that's what we've been doing, and through that we- I've worked on everything from Tor to open app stores to secure messaging through XMPP and Matrix to privacy preserving analytics because we saw a lot of people just dropping in like Google and Facebook analytics into their supposed secure stuff. And then [00:23:00] most recently, like internet shutdown technologies 'cause we've seen countries and places go from "Hey, we're going to filter a website" to, "We are turning off the whole internet." Yeah ... and so yeah, just along the way I've been doing what we can to help. Jonathan: Have you ever run into problems like US export controls getting in the way of trying to send some of these technologies various places? Nathan: Yes. And I have- Jonathan: No further comment about that? Is that the way that's gonna go? Nathan: No, I do everything we're- We do everything we're supposed to do we're required to do. The, the cool thing is increasingly we're not using any... Mostly we're not using any novel cryptography. So in fact- Right we're just using what's available on the platform. Sure. But I do everything I can to register and follow those restrictions and then also navigate. Like France is actually a hard place to distribute a cryptography app, for instance. And obviously- Yep ... there's restrictions in Iran and China and Russia, so I [00:24:00] think, the fact that we make... We don't profit. It's all free stuff. We're not a business selling stuff. Our code is open source, and we, people often are building and redistributing our code, or maybe they're downloading a binary, but we're just distributing that through the normal mechanisms that, either Apple or Google Play provide for us. We're just following the rules where we can. And, I'm also an ac- I am an activist. I've gone to jail. I've done scary things and put my life and people's life and, responsibly on the line and freedom on the line. This is real life for me. And why I do it is based on some, I don't know, how I was raised and the, and stories and inspiration of other people. And when you're doing tech and human rights, you're exposed to people that have given their lives or are willing to give their lives and freedom up every day. If I feel like I'm doing the right thing and I'm not harming people, then I do it and I think about it later in some ways. Jonathan: It's such a weird... It's a [00:25:00] weird time where France makes that list of countries where it's difficult to do human rights stuff. It's so bizarre. We're in a bizarre world. Nathan: Yeah. And, I'll say the other thing obviously is for so long our work was like- the early era of what I've done, 20 whatever, 20 years ago, was all about China. Over there in China, the Chinese firewall. And of course, increasingly, since Snowden, and even before then, since 9/11, of course- ... concerns here in the US about our privacy- Sure ... about surveillance. And it's both the sort of Snowden stuff, but also- ... we know, like, how horri- horribly compromised our mobile telephone operator company software is. We've, over and over again, we've heard about T-Mobile, AT&T, all these being compromised, all of our metadata, about who we call, when we... I don't trust the phone companies to keep my stuff safe, right? Yes. So- Jonathan: I have heard some stories. I... Nathan: Yeah. So- And that's, third countries I, I am, [00:26:00] I want people in the US to use all of this stuff, both to protect their rights of privacy that we have in our Constitution- and also to protect from the increasingly likely massive cyber attacks that could disable our country. Jonathan: Yeah. What a, w- what an interesting dichotomy. W- it, so what, i- is, speaking, thinking about here in the US, like, how successful has that been trying to get these problems and these tools into the mind of the average citizen? Nathan: Yeah, T- Tor and VPNs have had their moments obviously. Yeah, moments like Snowden raise a lot of awareness and we see huge spikes. I think we have so many users, a, a big section of users of things like Orbot and Tor in the US. Obviously, you can... signal is a huge success by any measure and, and- Absolutely Not our project, but we've, the database we worked on SQL Cipher, which is the database that Signal uses. That's a big part of our, proud of [00:27:00] our legacy is bringing that to Android and promoting that and basically telling Signal, "Stop writing your own secure database and start using SQL Cipher", and they finally did, which was great. And really just being an advocate for them and, so proud of... And, and they're an open source project that- ... we've contributed some other things that they're like the you can hide this Signal icon on your phone. You can change it to something else, and that's work, we contributed that and- Oh, nice you can also blur faces in the photos, and that came from work that we've done with other groups. Anyway, that's... But you can also then say this, the secondary effect of Signal and then encryption then on WhatsApp and Apple Messages and everything else has really been, like the vast majority of text messaging moving from unencrypted SMS to encrypted SMS- even if it's not all great, is hu- is a huge success I think in the US. ... In the recent years, like I've liter- literally done digital security trainings for retired mostly women at [00:28:00] churches. Who are like, "I am helping with XYZ cause," pick your cause, "and, I want my s- communications to be private. And I'm like, I'm happy to show up and do talk to the church ladies about digital security and privacy." And they care. Jonathan: I'm sure. Yeah. No, that's really interesting. I didn't expect that at all. Yeah. What are some of the other projects that you guys work with? I see multiple logos, some of which I don't recognize on your partners and collaborators. Yeah. Nathan: The, the... we've worked with Calyx off and on for a long time. They, And that's not up there, but in terms of, yeah, on our homepage WITNESS is Peter Gabriel, the musician years ago when he saw what happened with Rodney King speeding by the LAPD, and the video recording of that event- having a huge impact, he said, "What if we gave everyone in the world video cameras? Wouldn't that help with human rights?" And he's and WITNESS has been the leader in, like- ... what happens when we give rights defenders cameras. And we've worked with them on something called Proof Mode, and basically, it's now, like, [00:29:00] how we fight deep fake AI slop in the world, so that's awesome. We've been doing that forever with them. Tibet Action is a great partner who do all our work around Tibet and China with F-Droid. OkayThanks is an amazing design firm. If you're, if anyone's looking to help with branding, design, UX, they've been doing s- they- ... are in this security privacy world so deep, and know, like, how to make this stuff beautiful and come to life. And then yeah, SR2 is awesome UK-based, our, all of our ops and infrastructure, e- essentially, and they're like the, the guy behind the guy often in terms of run, actually keeping stuff running, and then the last one is Benjamin Erhard, who's our lead on iOS. We like to just lift up. We are a organization, but we also... The realities of what we do are such that- I'm- I describe it as like a movie studio or like Industrial Light & Magic often is my dream. But, [00:30:00] we get called like, "Hey, there's gonna be a movie." "We've got a couple years of money to make this film. We need you for the special effects." I say, "All right, I'm gonna put together my best crew," and then we pool in all these people. It's when you watch the credits of the movies and there's like a ton of people and with different names. That's what we do. Yeah. And so I put together a team to produce something with a certain amount of funding because we're not a company where I have 50 employees and- Yeah unlimited money. We I put together a squad for a heist, and we do it, Jonathan: yeah. Beautiful. What- tell me more about ProofMode, 'cause this is definitely something that we are seeing more and more on a daily basis. The, so the problem statement here is that weaponized outrage, and unfortunately weaponized outrage pays money because so many platforms now are paying you for what's the term they use? Basically people watching your thing and clicking on and interacting, right? Yeah. And so the more outrageous that you can make something, the more interactions you get with it, and therefore, the more money you make. And that's not [00:31:00] even at all covering the idea of there's governments out there and special interest groups that wanna put their thumb on the scale, and weaponized outrage is one of the ways to do it. And one of the easy ways to get to that is just make an AI video. Now, make an AI video that shows what you want it to or, selectively edit or there's a bunch of different ways to do it, but AI is becoming the go-to tool. And so what is ProofMode and how does it help? Nathan: Yeah, y- so years ago, we, it was an event in like 2010 at NYU, and it was related to the visual images on the internet. And back then, we just we realized there was two issues. One was, You could easily, there was like early cameras then that would like make you smile or would like manipulate thing. My niece who was, had this toy camera and I was like, it made you smile somehow. It like changed the pixels in your face. Like that's weird. That's unfortunate. But you were also seeing cases of Photoshop of imagery to add or remove convenient item, items- ... for a, [00:32:00] a certain message. So at this event we actually came with two ideas. One was Obscura, and we built an app called ObscuraCam that's been around, which was, "Hey, let's make it easy to redact and blur photos right on your phone." And that we could even use face detect and auto blur faces. So that came out of- Nice ... previous rounds of protests in Iran, saying let's just, how do we remove data from imagery and protect people? That was- ... and then we had In- InformaCam, which was how do we add, how do we digitally sign every photo, basically? Because PGP's hard. People don't know how to do hashing. Just take a photo and add a hash to every one. That was the initial. And then we said what if you take a photo, add a hash, notarize that hash on a third party and then grab a bunch of other sensor data, like all your GPS data and your phone data and all this stuff so you have this manifest, you notarize it, you timestamp it, and that is what a photo should now be in this modern world." So we've been working on that idea since then, and it was, I think just around [00:33:00] 2020, big movers and shakers in journalism, news, tech world started worrying about deepfakes. And that early, what they were starting to see and and this problem of easily increasing capabilities of things like Photoshop to, Magic Lasso and remove stuff and da. So yeah, so Witness really worked as our partner there with groups like Project Origin, BBC, Microsoft, Adobe, Google eventually, and this thing has emerged called the Coalition for Content Provenance and Authentication or C2PA. And it's like EXIF with digital signatures- ... it's the easiest way. And we've been a part of that, and we are- we developed the mobile SDKs for C2PA, C2PA Android and iOS, and then ProofMode is our flagship it's our Firefox of this world essentially. Yeah. It's a camera app, it's a verification tool, there's a server. It's all free and open source again, and- ... people use it to document war crime evidence. [00:34:00] People use it to document land theft in Latin America. People use it to document their catch, bass fishing catches for bass fishing contests. People use it in Europe to document like an Amazon delivery, but it's an independent organic fruit delivery to say, "We delivered a box of fruit to this school on this day," so that the government will pay them. So it's become a, the u- the idea is that it's a photo or video you can trust, and it has the receipts, and the receipts are digitally cryptographically backed, and it's pretty great. And we've found a way to, like- ... both work with the vision that Adobe and Google have. So this is built into Pixel now in some ways, right? It's built into Leicas, Nikons. That's great. They... Yeah, so they have a vision for C2PA that is one thing. We can do that v- version, but we also have a fully decentralized, anonymous, open source version of it, too- where you're not, like, submitting to s- [00:35:00] surveillance by Adobe and Google just because you want a photo trusted, right? So we're trying to balance... some people see this as as a global s- stamp on every photo, and it's gonna harm you because now every picture you take can be tracked back to you. And we don't want that. So we're navigating that. And so ProofMode, we really think is that balance of it's built with the same values where we build stuff like Orbot and Tor, but we're building a camera. Jonathan: Yeah. Yeah, interesting stuff. It looks like i- built into ProofMode is also some detection for AI? Nathan: So yeah, so our, we have a web-based tool called ProofCheck which you don't u- it's all a progressive web app. You don't upload your photos or... It's all in your browser. And so we do some detection. We're not, It's more like, ProofCheck's more like a Swiss Army knife, where we're just trying to provide you all the tools to see what metadata is there- Oh in any file, [00:36:00] like a media file. And then if we see something that looks like an AI d- fingerprint we'll let you know. Okay ... and then, but it's also if someone says, "Hey, I use ProofMode, here's a bunch of photos," you could drop it into ProofCheck and we'll tell you about 'em and do all the verification, okay. Again, it's this idea that normal people don't know how to verify a SHA-256 hash, and we wanna make it easy. Jonathan: It's true. It's it's ironic, sometimes tech people don't know how to actually verify- ... a SHA-256. Oh, the first six characters match, it must be good. Like- Nathan: Yeah. Well- Yeah, so we're- ... maybe Jonathan: not. Nathan: And the other, the like, another sort of hilarious offspring of this is we worked with Evan Henshaw-Plath, otherwise known as Rabble, who's a longtime hacker, worked on Twitter. And he created just something recently called Divine, which is a reboot of Vine, built on Nostr, which is another decentralized protocol. It's open source and it's uses ProofMode and C2PA underneath. So we wanted to build, it's Vine, no AI, just for humans. [00:37:00] So this is you may not be a fan of six-second videos, and I know it kind of fries my brain- ... but we're so excited that there's an app like this which is for whatever ridiculous you want, and we can help ensure that it's a human-only space. Jonathan: Yeah. Yeah, interesting. One of the other things looking at the Guardian Project homepage that stuck, that caught my eye, is Butterbox. Nathan: Yeah. So that a whole other branch of work and stories, ... and but I think I, I might have mentioned earlier is that increasingly We are not just seeing internet censorship and blocks in the world, we're seeing internet shutdowns, right? So around an election, they'll just shut down the internet, or there's some protest, internet will be shut down. Yes ... also, there's a lot of more severe weather events, we'll call them. And- ... we can do more to help people in those places with temporary servers, bring in some Starlinks. But we need little [00:38:00] boxes, little Raspberry Pis, for instance. But it doesn't have to be a Raspberry Pi. We can talk about that. So we've been around long enough that we've seen things like PirateBox, FreedomBox lots of ideas around... LibraryBox was an awesome one by a f- a colleague of mine, jason Griffey. How do you take a little router or thing and make it a server you can carry around and it can allow some people to connect and do stuff? Upload, download, share files. That's the idea. There's also a Internet in a Box, another great one. So our thing was we wanna be, make it... Butter is about, here's my butter towel, but make your life easier and smoother. There's a phrase in Spanish, "Como la mantequilla," like the b- easy like butter. Easy like butter. And we wanted to put an F-Droid App Store on there with apps that work without the internet, so that was a basic idea. We wanted a chat room. We wanted open street maps you could download. And so yeah, we've been working on this for a while and f- we've now recently we're [00:39:00] adding in Delta Chat, which is a really cool signal-like decentralized encrypted messenger. So you can have multiple Butterboxes and relay messages. We've been testing with things like Wi-Fi Halo for long range Wi-Fi, and also pairing one with a Starlink as a kind of a bridge into a Starlink server. And then, yeah, we've been working on Meshtastic pairing. So y- now you're carrying these things around in your bag with a battery. Yeah. And ch- because someone may not know how to... If you have a box at a relief center in a central place and people can come and just see a message board of stuff coming in off of the mesh and chat and different things, we're open to adding different technologies as long as they are focused on this, how do we help people when they're in trouble and how do we make something hard easier? Yeah. So yeah. Awesome ... and last thing, it's been a kind of a Raspberry Pi thing for a while, [00:40:00] Pis zero, 3, 4, all of that. The, the cool thing recently is we rebuilt everything on Debian, so now we have builds- Oh, yeah ... that are AMD64. Jonathan: Everything. Nathan: So everything You have it on Jonathan: everything now. Yeah. Nathan: Yeah. So you just, here's my old beautiful ThinkPad and a little router And this is now a Butterbox. And this was, like, so important for people in places that can't get this hardware. And have lots of l- old laptops around. And old laptops are great- Yeah ... for so many cool things. Yep. But we're also excited we have a lot of partners in Africa, both in Kenya, South A- southern Africa And there's... You can reliably get Pi 3s full Pi 3 kits there, and that's good enough for most people- Oh, yeah ... to use this with a small group. And so we're... Yeah, so Butterbox is really just trying to, we know the internet isn't always available, and can we do something that is straightforward and [00:41:00] simple for people to use? Jonathan: Yeah, absolutely. It brings to mind, so the little router you had hanging off of that was a GLI Net, and it runs openWRT. Yeah. And it makes me think, is there room for a Butterbox Mini that's right inside of openWRT? Nathan: I'm always asking what, thinking that. We've also thought about, an Android ROM or an app version- or a docker that runs in an Android phone. Sure. And yeah, we love, openWRT, those are... And I think Librarybox used to run on kind of openWRT type stuff. We've been pushing it because we're running a Matrix server and a Delta Chat server and... But I think a lot can be done on just the cheapest. Even this this is the bigger GLI Nets- ... Nathan: Can do a lot. Oh, yeah, they're actually really Jonathan: capable. Nathan: Yeah, so- Jonathan: Es- especially the, the ones that have the USB ports on them that you can, you can mount the file Nathan: system on. Yeah, exactly. Yeah. Jonathan: Yeah. Do a lot. Nathan: So we're, in some ways, Butter is not only a specific tech, but we also- help [00:42:00] communities prepare in different ways and just understand what they can do without the internet, right? And so people have become so dependent in the world on WhatsApp and Telegram and Signal and things and we, it doesn't have to be that way. Jonathan: So it's an interesting experience to to do a, like a war game, a exercise of what happens if you don't have the internet? I've heard of guys that'll do things like, okay, 24 hours and go out and just shut the power off. And then- Exactly ... how does it go for the next 24 hours? And at first it's real easy, and then you go to cook supper and it's oh no, I didn't, I was not as prepared for this as I thought I was. Or go turn your water off or turn your internet off. There's there's a bunch of these different scenarios that you can play through and it is, it's a l- real learning experience. How would we cope without this for a while? Nathan: Yeah, and I'm, the Meshtastic and Mesh Core and all these communities are interesting because, it's people just playing around, saying, I can buy this cheap gadget off of wherever and start dabbling [00:43:00] and say hello to I can reach people in New Hampshire it seems from he- from Boston, and that kinda makes me feel good- Yeah that there's this little like someone out there, and if something went down and and I've unfortunately had to live through 9/11 in New York, the blackout in New York. I've lived in India through a number of issues of blackouts. Even the Boston Marathon bombing the, which I was not far from. All your cell phones stop working, everything breaks. And and then now r- recently supporting people in Iran where there's been months and months without internet. And, or, one of our team members is from Mexico and she, there was a hurricane there a number of years ago, and she went back to assist them. And so yeah, this stuff is real and it's it's real and- It's educational and fun to think of, sure. And I think the other interesting thing we were working with MIT OpenCourseWare, and they distribute a four terabyte drive which has all of [00:44:00] their courseware ever. Wow. Jonathan: That's great. Nathan: And so you can take this four terabyte drive, plug it into a Butterbox- and you have, like, all of MIT anywhere in the world, you know, on about... it's So we've actually worked with communities to do that. And we became a real easy go-to- ... Nathan: For MIT to say, "Hey, this is a nice platform for a classroom," because we have chat and other stuff, too. And so that's the other part, is like people in Cuba are, have this whole weekly packet. They also distribute terabyte drives with Netflix and games and apps and stuff. And so sneakernets, especially at the terabyte level- ... Nathan: Are wild in terms of- Jonathan: Oh, yeah ... Nathan: if you actually think with that in mind what you can do, Jonathan: yeah, the, the old adage was the station wagon full of tapes hurtling down the road. A sneakernet of somebody with a one terabyte in their pocket actually has some reasonable bandwidth. Nathan: Truly, yeah. And, going back to, Delta Chat is really [00:45:00] interesting technology, by the way, because it's just email underneath. And so all the old tricks of sneakernetting mail spools or trickling mail spools over low bandwidth networks work. So it's not... Signal's amazing, but the infrastructure's controlled and there's all this kind of highfalutin stuff it's doing. Delta Chat is just email, but it has the same end-to-end guarantees. It basically made PGP super usable. But I love it because underneath we all know how to creatively handle mail spools. And so in fact, one of the modes we're working on for Butterbox is a s- is a sneakernet of mail, so you're just walking around a major city handing off drives, and the mail gets relayed through the drives being plugged into various Butterboxes. Jonathan: I was not familiar with Delta Chat, and I'm now very intrigued. I will have to learn more about this. Nathan: It's a good backup for Signal, if you ever need that. Jonathan: I wonder what's... i'm switching hats here for a moment. When they say low bandwidth, I wonder how low bandwidth they're talking about. Nathan: Yeah. We've wondered the same thing. Jonathan: Yeah, I mean- if you've played around with [00:46:00] Meshtastic, we're... We do low dial-up speeds, like 1K in some cases, right? Nathan: Yeah. Jonathan: Very low bandwidth. Nathan: I think... again, it's email. So we've all f- we've all- Synced our pop mail over a dial-up modem. Not all, but many of us a little older, and that was the experience. And it's- Yeah ... not so bad. We also, I mentioned Wi-Fi HaLow. We've done some work- Yeah ... with that. It's really interesting and promising, and it seems to be coming out of the hacker maker s- phase into something that is more of a normal product. The bandwidth you can get on, a, a one kilometer link is pretty good, Jonathan: yeah. Yeah, it's really, it really is pretty interesting. All right. What have we not talked about that I should ask you about? Nathan: I mentioned Matrix a bit. There's another thing that we are just starting to talk about more publicly. It's called Convene. It's on the Butterbox, actually. So we built our own Matrix client and it is browser-based, it's encrypted and it's at [00:47:00] letsconvene.im. And it's not... We've had to lock it down because we can't just offer it free to the world 'cause we don't have the money to run s- servers for everybody. Sure. But it's a great Matrix client to deploy if you're running your own Matrix infrastructure. It's built into Butterbox as the kind of Matrix chatroom, the web-based chatroom. It also, what I'm really excited, tying back to Proof Mode, it's kind of our demonstration app for a messaging app that also has photo verification in it. So that, you upload a photo or video. If there is provenance and metadata we pull that out. If something looks like AI or it's been edited or it's old, we flag that. So we're using it as a way to say, "Hey, we think all messaging apps should do this kind of thing," where you're flagging stuff around the photos being shared. But it's also a totally ephemeral... you don't have to have a Matrix account, you just click a link and join. So it, in a sense, works more like a Jitsi room or one of these temporary call [00:48:00] rooms. And it looks and works really well and, Okay ... you can find the code anywhere and, and it's I think for... We're big fans of the, of Matrix as a platform for messaging. And and again, being able to build it in, and we've tried a bunch of different things, but yeah we're excited for that. But overall, yeah, we really, we have a pretty good set of contributors on the Tor stuff. I think, Proof Mode and Butterbox, we're really excited. They're at kind of moments where we're seeing increased adoption, and the timing for both of these things is really needed and, whether you're interested in the Meshtastic pieces or- ... helping us support different hardware, or you're really into, yeah, wanting to build C2PA stuff into different things. A lot of what we do is, like, empowering other developers and acting as a home for developers who wanna, and technologists of all kinds, who want to help the world. So I think ultimately that is, our mission is this could be... used to be that I could [00:49:00] w- funding is tricky right now overall, I'll say. We had a pretty- Sure ... good run. ... I think obviously the current US administration and just the current funding environment's gotten really- ... complicated. So I'm not gonna say "Hey, everyone, come knock on our door and we can help make this your day job," but there's a lot of open source developers opportunities out there. We're a great place to learn and to improve your skills around this kind of tech. And also a great resume builder. And we love students, so I think that whole... I've done a lot of Google Summer of Code mentoring- ... and things like that. So yeah. Yeah. So we're always open to people that wanna get involved for whatever reason. And who Jonathan: knows? Do you think there's, do you think there's still a path for somebody that wants to make human rights technology their career? Nathan: Yes, absolu- absolutely. There's, whether it's the Open Technology Fund in Europe there's a group called The Prototype Fund- that f- basically funds new ideas by new people that, yeah, you do it based in Europe. And there's academic environments that kind [00:50:00] of support this work. I'm a affiliate at the Berkman Klein Center at Harvard which is always looking for mixture of tech and academic and research. And I think, the cybersecurity front is always a great career path, which- Oh, yeah. For sure ... which then you can do a lot of good in that path for lots of people and you can do pro bono work, and you can work in certs and other things that are really- ... helping the health of the internet. So yeah I think it, it still is, for sure. And we're just adjusting. There's been various booms. There was a early internet freedom boom that produced Signal, for instance- ... and our work. Then there was the crypto boom where all the Bitcoin folks were, like, handing out money because they had lots of money. Now that has come down. Yep. The boom we're in right now is an AI boom. Of course. And I think it's not really compatible with some of our values. All Jonathan: right, everybody, take a drink. He said it. Nathan: So most of the funding offers we get are like, "We'll give you a million dollars in [00:51:00] tokens," and I, I- Jonathan: I mean, that's not nothing. Are, so are you guys doing the LLM coding thing? Or do you stay away from that? Nathan: We mostly stay away from it. We definitely aren't like, yeah, one-shot vibe coding. I think adverse- the adversarial au- code audit stuff is really interesting. It's really great, yeah. That has been... Basically, we'll pay, we're human auditors. Right now we're getting 7A Security is doing an audit of Orbot- and we're paying them a good chunk of money to do that. And we love people to do that. We love Trail of Bits and Radically Open Security and Cure53, and there's all these really great auditors out there that have been doing it for years. And we wanna keep doing that. However if we have the ability to run a security-capable model as a lint check like every day, that's great, and I think if- So that, that is my current interest, and there's also, we see more AI popping up in pen testing tools and- Of [00:52:00] course ... scanning tools. So I'm comfortable with it there. I'm sure that just like our developers in the past would probably find code from Stack Overflow and copy and paste it into their work, they're probably- Oh, Jonathan: surely not. We never did that. Nathan: That's why we get audits, but I know that- ... yeah, some amount of our code is being written by Smart Code Complete in Android Studio because you can't, it's hard to avoid, sure. So yeah, it's an interesting time, and I think we're... we're also, there is some great research in algorithmic machine learning-generated anti-censorship-like protocols. Protocols that are using algorithmic evolution rather than a human having to figure out how to do it, we can actually build stuff that a human wouldn't think of in terms of getting through, a certain deep packet filter. Yeah, absolutely. So yeah, we're trying to approach this in the right way and be cautious of course, as always, so Jonathan: yeah. Yeah. I saw I saw the funniest thing. I think it was today, [00:53:00] earlier today I saw this. It was one of those, it was a PR review that was obviously made by AI, and at the end of the review, it finally got to the point to where it's like, "Green check mark. No issues found. Ready for human eyeballs." Nathan: Yeah. Yeah. And I know Tor specifically has got, when those first... this is not that long ago. A few months ago, when we ha- every, every open source project was getting this flood of- Jonathan: Yes ... Nathan: of vulnerability reports you needed machine assistance just to get through it all- Just, yes ... and try to figure out what was real. Jonathan: Yeah. So here's the crazy thing in just those few months that it's gone from the flood of bogus vulnerabilities to now a slightly less flood of real vulnerabilities that we are getting from AI. It has gotten so much better so quickly. Yeah. It's been amazing to watch, and terrifying, but amazing to watch at the same time. Nathan: Yeah, and it's, that's [00:54:00] definitely... Yeah it's a wonderful problem to have. I think a lot of us humans are still adjusting to how to deal with that scale and- Absolutely ... how to respond appropriately and, and I think we're, it's also r- yeah, it's a weird time because you will see some well-meaning person just vibe code, a human rights app and announce it to the world on, Hacker News, and you're just like, "Oh," like- it feels the speed at things, that things are happening, there's so many opportunities for things to go wrong or not have care- Right Especially in critical areas. So we're, there's a huge... I'm part of a number of groups that are like AI for democracy and AI for human rights, and there's great ideas, and people can whip up a really impressive prototype quickly. It just we're- Jonathan: You're in an industry, though, where oversights get people killed. Nathan: Yeah. Or lose elections, or, or in a way that shouldn't have happened, things like [00:55:00] that. Or- Yeah ... big gaps in There, there's a whole range of bad outcomes that can come from lack of care, and yeah, getting people killed is probably the worst. I think we are also evolving to say, what else is needed? So like I said, auditing and logging and, we've, with ProofMode, we wrote a whole National Science Foundation proposal with a UC Santa Cruz lab called the Explainable AI Lab, which does a really amazing work there, and we didn't get the funding unfortunately, but we're keeping at it. Basically saying, look, if you're allowing kind of machine processing of like science through a, a pipeline and you're not auditing every step of manipulation of that data or that sensor data- ... that image, the AI will just try to please you and will just make up stuff and add things and change stuff, and that we need notarization, just like with ProofMode photos along the way of like data sets and stuff so that- Oh, yeah Just like a Git history you can roll back the history and find who to blame. And so we're, that's a lot of what we're also talking about is the lack, [00:56:00] the kind of AI safety around, you know, transparency of logs and operations and provenance of the data that's being sent through because, we've clearly seen that in so many like court filings already where citations are just invented. Yeah. Jonathan: Yeah. For sure. All right. We are unfortunately, we could go longer. I would be thrilled to, but we are out of time. I gotta ask you a couple of final questions, and that is what is your favorite text editor and scripting language? You personally. Nathan: I still use plain old VI often and it's still just the muscle memory. Yep. I use, I just it's the, the muscle memory of my late teens and early 20s and anything that looks or feels like it is my favorite, so I'll stick with that. All right. And scripting language I really enjoy when I have the chance to write make files. Jonathan: Okay. Nathan: So it also... there's something that feels so good about when you [00:57:00] automate a build thing completely- This is true ... and a- and it's just it like, really, you feel responsible and grown up that you've taken something hack. And it's an interesting kind of scripting language. So yeah, I just- Yeah ... there's, so the great satisfaction I get in something like that, Jonathan: got it. That's fun. All right. N- Nathan, thank you so much for being here. It has been a blast. Real joy to talk to you. Nathan: Thank you so much. Yeah. Keep it up. Really also am a listener, and appreciate o- obviously this work needs so much continued advo- advocacy and highlighting of all the people putting in so much energy, so- Yeah thank you. Jonathan: Yeah, absolutely. All right. Wonderful conversation with Nathan. We've got some fun stuff coming up. We actually mentioned it. Next week we're talking to Trail of Bits, and then the week after that we're talking with Oliver Vernon of updatecli, and then after that is the Open Source Automation Development Lab. So we've got the next three weeks booked, and then the week after that, no show on the 22nd because I am out traveling. But hey, we hope everybody comes back next [00:58:00] week and for the next three shows. We plan to have a lot of fun. It'll be a blast. Appreciate everybody that is here today, whether you watch or listen or get us live or on the download. Thank you so much for being here, and we'll be back next week on Floss Week.
• 
  Episode 879 - Easy Like Butter 26.08.2026 1t 6min

  This week Jonathan chats with Nathan Freitas of the Guardian Project! What is going on at the forefront of Open Source and privacy advocacy? Where are we on a true Tor browser on iOS? And what's the ideal solution to the Android Lockdown issue? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 878 transcript 05.08.2026

  FLOSS-878 Jonathan Bennett: This week I'm talking with Jonathan Pallant about Rust and primarily in the embedded world, the Raspberry Pi Pico, the Nordic, Espressif, all of your favorite little MCUs. You can run Rust there and partially due to efforts by people like Jonathan. This is Floss Weekly, episode 878, recorded Tuesday, August the 4th. A tool with opinions Hey, folks. It's time for FOSS Weekly. That's the show about free, libre, and open source software. I am your host, Jonathan Bennett, and today, once again, we're gonna get rusty. Whereas last time though, when we talked to someone with Ferrous Systems, we were talking about Rust on the Linux desktop, in user space, Rust on general purpose computers, and today we're talking about Rust somewhere a little different. I've got Jonathan Palant with us today. He is also from Ferrous Systems. In fact, I think his day job is teaching Rust there, and I think also doing some programming work. But let's get him on and ask about all of the places where Rust runs that you may not have thought about. Hey, Jonathan. Welcome to the show. Jonathan Pallant: Great to be here. Thanks for having me. Jonathan Bennett: Yeah, it's good to have you back. Or no, you for the first time. Good to have Ferrous Systems back. The second interview with one of you guys in a couple of months now. All right, so this, this magical thing called Rust it runs on the desktop, it runs on the server, it also runs on the Raspberry Pi Pico? Jonathan Pallant: Yeah. It- So joys of having a systems programming language, right? You get a compiler, source code goes in, machine code comes out, right? So anything that runs the machine code should be able to run it. So yeah, Rust is really interesting in being able to scale from the very, very large. It is the language that powers Amazon Web Services, and Cloudflare, and Google, and a whole bunch of other stuff. But it scales down to, as you talked last time, desktop applications- ... things like Sudo written in Rust. But also scales down to the small, very small. How small is always an interesting question, but- Jonathan Bennett: Yes, it is. Jonathan Pallant: I have run Rust on ST Micro, Microcontroller, Arm Cortex-M4 32K flash, 4K RAM. As modern things go, that's pretty small. I'm not sure I'd be happy squeezing Rust-generated code onto like a Z80 or a 6502. That may be a bit too small, but certainly tens of K of flash, a few K of RAM, I think- ... is quite workable. Jonathan Bennett: I happen to have here the Seeed Studio Xiao board actually their next generation Xiao board. It is a Nordic nRF54. The word Xiao in Chinese means tiny, and it is an appropriate name, as you can see. T- Can we do Rust on a nRF as well, on, on Nordic? Jonathan Pallant: Yeah. So Nordic semi generally use Cortex-M4 M0 in the nRF51, M4 in the nRF52. I did a lot of work with their cellular modules. Don't know if you've ever used a 9160? Jonathan Bennett: I've not, Jonathan Pallant: but- the very small device that does LTE Cat M- ... so the, the IoT version of 4G, and that comes with a Cortex M33 that you get to play with, and then a secret second processor that runs the modem firmware- ... that you are absolutely not allowed to play with because cellular. Jonathan Bennett: Yeah. Okay. So the ... Boy, the, the embedded world is an interesting world. There's a large there's quite the gamut of different things that count as embedded, and sometimes the lines get a little blurred, right? Because you can take the Linux kernel and run it on an ESP32-S3. Now, it's not a fun experience, but you can do it. It'll boot. It can be made to, it can be made to happen. And so i- it's interesting to me that Rust goes all the way down to some of those very, very small targets. How does that work? Is there an operating system that a, a Rust binary is then running on top of? Is there some kind of real time operate OS? Do you guys run on Zephyr, or is, are you literally just writing Rust that runs on the bare metal? Jonathan Pallant: Yeah the output of the, the Rust compiler, it works like a C compiler. Source code goes in, machine code comes out. So the compiler doesn't really care whether- ... you have an operating system or not. You could absolutely be the first code that boots when the system comes out of reset. Jonathan Bennett: But I mean- Where- ... there's like the question of am I making system calls to a kernel? Am I telling the kernel to do, the, the multitasking stuff, or am I just always, is the binary always running on the Jonathan Pallant: CPU? And where that comes in is less in the, the language, but more in Rust's choice of standard library. People say, "Ah, the standard library," as if it's one thing. But as with all good things, it's actually three things in a trench coat. The, the top part of the standard library in Rust is the thing that knows about files and threads and networking, and that's the thing that's gonna be operating system specific. And it's basically the Rust equivalent of hash defines that say, "Hey, if you're on Windows, we're gonna call this Windows function, but if you're on POSIX, we're gonna call, open read and write." Sure. And that's what lib std is doing. It's this operating system abstraction layer. As I said, file system, networking, threading. Below that you've got the heap allocator and all of the heap allocated collections. They actually live in their own separate library. If you're a Windows, Mac, or Linux user, you may not be aware of that because it's hidden, and everything heap-related is all exposed at the top level anyway. But there is this lib alloc, so if you want a heap allocator, it lives there. And then you've got these fundamental Rust types, these things that no Rust program can live without, and that don't actually care about what operating system. If you've used Rust, you'll be familiar with the result and the option types. They're enums, they are generic, they need to be defined in a library somewhere. And things like that go into a library called lib core. So when people write bare metal Rust, so Rust running on a Cortex-M4 or a RISC-V processor with no operating system these people say, "Oh, I'm using Rust as no std." And what they mean is I'm using the Rust standard library, but I've basically switched off the top half and I'm only using lib core. So I'm only using the parts that are operating system agnostic. No file system, no threading, no networking, but you do get Unicode string handling, as I say, result and option types, sort of fundamental properties of integers and floats and things like that. Jonathan Bennett: It, does this work hand-in-hand with a like an SDK the published stuff that the vendor puts out? So like we, we mentioned the Nordic. Obviously Nordic writes... I mean they, they've written Zephyr support. They've also got their oh, I forget what they call it, their little shim libraries, almost like an OS. SoftDevice I think they call it. And so when we're in Arduino land with C and C++, we do a lot with SoftDevice on the Nordic devices. Do, do... When you write Rust, do you use that stuff on this, this sort of a device, or are you f- blaze your own path? Jonathan Pallant: People come at this from different places. I went to a trade show in London recently, and I went round and surveyed all the people there who had the booths down at the show in London, said how do you write embedded firmware?" And honestly, I was surprised at how often Zephyr comes up. A lot of people seem to be really into Zephyr. And if you wanna use Zephyr, absolutely, you can put Rust on top, build your little Rust applications, link them into the Zephyr kernel, and that's all gonna work great. I wouldn't describe myself as a purist- but I'm in this to push what Rust can do. A great analogy came up the other day about about, about AI usage, and it was like sometimes there are heavy things that need to be lifted, and sometimes this is your job, and you should use whatever machine you've got to lift the heavy things. And sometimes it's 'cause you're at the gym- ... and if you bring a forklift truck to the gym, you are defeating the point. You're cheating. When I write Rust, I'm at the gym, and I wanna do it the hard way because I wanna practice my Rust, but also I wanna show what Rust can do- ... and find if there are any corners where Rust doesn't work. 'Cause I wanna know about those- ... and maybe I can go to the Rust project and get them fixed, or, I can educate my customers about areas where Rust doesn't work. So- Yeah ... for me, going bare metal is always more interesting. Jamming random integers into random memory addresses, that's where I'm most comfortable. But if people wanna use like a FreeRTOS or a ThreadX or any number of real-time operating systems that are written in Rust- If that gets the job done, that's great. Jonathan Bennett: Yeah. I think that's actually a really good segue to ask the question of what do- what is your day job? You're not... Are you writing production code, or are you doing the the education thing there at Ferrous? Jonathan Pallant: So my I asked for the words senior embeddage engineer to be written on my business card- Okay ... because it covers a multitude of sins. Technically what I'm doing mostly is teaching. So I teach Rust. I am the lead trainer. I'm responsible for all of our training materials, which are all open source. Anyone listening can just go and find Ferrous Systems Rust training, read all my slides, do all my exercises. It's all creative commons. You can help yourself. But I can't only teach, one, because my head would melt. I did a day of teaching today. I've been teaching eight hours already, so I may- ... yeah, just dribble a little bit. But also, yeah, you need breaks. But also, if I only taught Rust I don't think I would get better at Rust. There is an a need to sharpen the knives and practice the tools. So in between trainings, I'm, like, picking random dev kits and going, "Great, let's do some Rust on this. Let's write some training material about it. Let's pick an entirely new platform nobody's ever written Rust before." I have in my hands a lovely NXP developer kit. S32Z2 for the people who don't have video. This is get this right, eight Cortex-R52 with 20 megs of RAM. Jonathan Bennett: Oh, wow. Jonathan Pallant: As microcontrollers go- It's huge ... this thing is cooking. But it can't run Linux because none of these processors have an MMU. These are all- ... Arm's real-time platform. So a couple years ago I was like, "Can you even do this in Rust?" Probably, but I don't find anyone online who's done it in Rust. So we called in a favor from NXP. We got the board. I would not personally wanna buy one of these out of my own pocket money. It's a little out of my fun budget. Yeah, and we got Rust running on it bare metal, dual-core lop step. We can run asynchronous Rust. We got the interrupt controller going. But I do that because it's interesting and because it ends up useful for customers and useful as training material, yeah, embedded systems engineer, as I said, covers a multitude of sins. Jonathan Bennett: Rust and embed- the embedded world as a whole is really a moving target as places like NXP and STMicro and Microchip, Nordic, and all the rest of Espressif, everybody, they're constantly pushing the envelope, releasing new things, g- gluing new weird chips together that we didn't think we would ever see, and then suddenly- you see it at a trade show or a talk. But it's a moving target, and if all you're doing was education, then, you're, you would get stale, right? So I think it's incredibly useful to stay hands-on with the technology. Jonathan Pallant: No, absolutely, and I try to make it out to Embedded World every year to run the booth, but also just to walk the floor and- go and see what's new and get a sense of how things are training changing. You go to Embedded World three or four years ago- most of the floor didn't know what Rust was. "Do you do Rust?" "No." "What's Rust? Tell me about Rust." And this, they'd never heard of it. And now they're like, "Yeah, no, we've heard of it. It feels like a thing we should be doing." But certainly I think these big silicon vendors are thinking about, "What is my next step? How would I introduce this without alienating all of my existing customers who like the CSDK? Is there a migration path? How... do these things feel the same or do they feel different?" Espressif, I gotta give shout-out to them for being far ahead here. They basically hired all the open source hobbyists who were doing Rust on their chips. Gave them a job and they have their own Rust team. We've seen Rust news out of Infineon as well. Yeah, and I hope we see more out of the other silicon vendors as well because if they don't do it, who will? I mean me, but my time is somewhat limited and there's only so many dev kits I can buy. And it's the open source community. It's random hobbyists who go, "Oh, I've got a random dev kit I got off Adafruit or whatever. Got a chip. Does this have Rust support? I don't know, let's write something. Let's put something together." And we have all this sort of flourishing ecosystem around STMicro, around Nordic, around Raspberry Pi. I personally did the Rust support for the 2350. That was fun 'cause they gave me pre-production chips, so we had Rust support ready to go on launch day. But these are all independent ecosystems with their own thing going on. Which is great if you just wanna get into Rust and have some fun. But if you are- on the hook to design the firmware for a product where you wanna make a million a month, I can see why you may not just want to pick up random packages online, and you may wanna talk to the silicon vendor or someone else- and get a bit more professional support. Jonathan Bennett: Because the way where madness lies, and I know because we've done this ourselves, is you end up coming along and forking about 100 different packages, that you've got your own version of the library because you have some fix, and the upstream library is dead, or you can't get ahold of the maintainer, or this or that. Yeah. Been there, done that. That's my life these days. How similar is it? We've thrown out a lot of different embedded architectures. So again, compare, like, Nordic to the Raspberry Pi Pico. Can you write one Rust program and get it to run on both places? Is there cross-platform driver support? So say I've got a an I2C-connected temperature probe. Can I write one Rust driver for that and run the same code on the Pico and the the, the Nordic? Jonathan Pallant: You can, and this is thanks to the work of a great bunch of people called the Rust Embedded Devices Working Group. So take you back to 2017. I'm just starting to get into Rust. I'm using a random TI Stellaris dev kit that I happened to have lying around, and I'm doing my own thing, and some other people are doing their own thing, and we find each other. And we're like, "We're C developers. We've written a lot of C code." "Rust looks fun, but we're very aware of how-" I describe the C embedded ecosystem as like islands. And these islands are all really quite far apart. Yeah. And if you live in Nordic island, and I don't know, you happen to run out of chips 'cause there's a chip shortage, and you wanna get over to- Jonathan Bennett: That would Jonathan Pallant: never happen ... ST Island, to get over to ST Island, those islands are quite far apart, and it's pretty difficult to write, say, a cross-platform driver that, that works across that. And we looked at that and we said, "What could we do in this fledging fledgling Rust embedded ecosystem to try to, for that not to happen?" And the Rust language mechanism that it offers for that is a thing called a trait. And if you're a C programmer, you can think of it like struct full of function pointers. That's how the Linux kernel does it. That's how the Linux kernel thinks one disk controller looks like another. They both, drivers, provide a struct with a function point of open and a function point of a close, and read block and write block. In Rust we call that a trait. So the Rust embedded working group published some traits, and there's a trait for I2C, and there's a trait for SPI, and a couple of other sort of common peripherals, GPIO- razor pin high or low. And then what someone needs to do is come along and say, "Oh, I would like Rust to be usable on i- my particular chip." NRF 52 for example. So you would write some Rust code, and you would implement the trait. So here's my I2C driver- Yep ... represents this peripheral, and it implements the trait. When I flip to the other side, someone's got- a CO2 sensor. Little I2C device, measure the air quality. I've got- And they just need an Jonathan Bennett: I squared- I've got one of those on the desk behind me waiting for me to either find or write a driver for it. Jonathan Pallant: Fantastic. And you just need an I2C bus to put it on. And what do I2C devices need to do? The driver needs to do a read maybe do a, a write and then a read. There's a couple of fairly basic operations. 7-bit addresses, 10-bit addresses, and that's all encoded in the trait. So if I wanna write a Rust driver for my little CO2 sensor, I am generic over some type T, where I don't care what type T is as long as it implements embedded hal colon I2C- some trait. And then a third person comes along, and they're like, "I may be the first person in the universe to use that CO2 sensor with that microcontroller." And they can compose it together. Yep. And in general, it should just work. You will... There's ef- efficiencies that you will lose because it's a g- sort of a very generic API- that's not very- ... chip specific. But the trade-off is, yeah, here's my random Adafruit LCD screen, here's my CO2 sensor. Plop them all on a board. Here's a, a, what we call a, a HAL implementation someone's written for this chip, and it should all just work together, and generally it does. Jonathan Bennett: Yeah, that's very cool. It's actually similar to the way the the Adafruit excuse me, the Arduino ecosystem has evolved. You've got kind of that, that same idea in mostly C code, some C++, but it's that same idea. We've got an abstraction of here's what a wire bus is going to look like. It's usually what you see it called. And then somebody else can come along and write a driver for it. Was that part of the inspiration? Jonathan Pallant: Yeah, I think we'd the people who formed the Embedded Work Group, we'd obviously used Arduino before, but we'd also used these kind of very vendor-specific SDKs. But you also get the, the classic embedded commercial vendors. They all have their own stacks and their, probably their own in-house operating system. It seems every embedded company's got their own operating system they've had since the '90s- Of course ... and that kind of stuff. So we've seen all of that. And what would a, like a utopian future look like? I'd love anybody to be able to pick up any MCU- and any device and just plug them together. Jonathan Bennett: Do you see vendors actually starting to write the the, that bottom half, that support layer for you guys? Jonathan Pallant: So we have seen Espressif do that. So they publish all of the, the, the drivers you need for their chips. For most of the other vendors, I think we've seen something from Infineon. But for most of the other vendors it's wait and see. Maybe throw the hobbyists a bone, some free hardware occasionally. But that's all mainly hobbyist-driven. I will shout out STMicro, who have done the other side. Surprised me. But if you've got your little STMicro sensors, 'cause obviously they don't just make MCUs- I believe they have all these sensing modules as well. They will give you Rust drivers for the sensors. Jonathan Bennett: Cool. Jonathan Pallant: Which you can just get from STMicro's GitHub and use, and then use those drivers with your other Rust code. Jonathan Bennett: I've been really impressed with STMicro's open source support. They were off my radar for the longest time. And then I started working with some of their hardware, and it's like, you ... 'Cause I'm from the Meshtastic project. That's my day job these days. As all of our listeners know, because I mention it at least, take a shot, if you're playing the drinking game. But so I got ahold of some STMicro hardware, and it's I wonder what it would be like to 'Cause, because it was gonna be able ... Get Ar- get the Arduino working on this, and then run our platform IO build on it, and then finally make Meshtastic work. And I was ready to strap in for the long haul, and it's oh, no, just go download it off of their GitHub and pick the board and it'll compile. It was easy. It was like a day's worth of work to make everything start working. It really surprised me how good their support was. Jonathan Pallant: Nice. I will shout out their tools. So I do use CubeMX quite a lot. Their MCUs have very complicated clock trees, so having a little graphical tool where I can just feed in the crystal frequency and go- Oh "Oh, I want a, like a times two," and configure the PLL. I don't need the C code that gets auto-generated out the back end, but it's very useful visually doing the clock tree. I also love the pin planner. Nordic don't need one, right? 'Cause you could put any peripheral on any pin. But ST you generally can't. On STM32 each pin has a set of three random functions. And you're like, "Great, I need UART5." I guess I can have it over there and over here, but oh, not here because I need th- that pin to be like a, an input interrupt, so I'll put it on this pin. And doing that with a pen and paper or a spreadsheet's horrible, so I do love the CubeMX pin planner- where I'm just like, "I need these peripherals. Just lay it all out on the, the 80 pin package, and I'll design the board around that." And then again, I throw away all the C code, and then I teach, I program Rust to f- to follow the kind of the, the pin map that we've got out the tool. Jonathan Bennett: Yeah. It's almost like a, a traveling salesman-esque problem. It's like it, it seems very simple, but then when you try to do it, it gets really complicated really fast. Jonathan Pallant: One, one of, one of those puzzles where there's one hole left- ... and one piece left. And this piece doesn't fit in that hole, and you're like, "Oh, I'm gonna have to take this all apart and redo this-" Yep "before, before this fits." Jonathan Bennett: Yep. Exactly. That's fun. So what what other interesting hardware have you run Rust on? We talked about a bunch of them, but what's your favorite, what's your favorite device to write Rust for? Jonathan Pallant: Ah, that's a really good question. I do love these kind of big real-time microcontrollers. But just 'cause they're so outside, like, where I'm normally using microcontrollers. Eight R52s is just an outrageous amount of power. So these are fun. Running the dual core lockstep actually turns out to be really easy. 'Cause the thing about having two cores locked together that do the same thing for safety purposes, so- Oh, okay if one of the cores gets hit with an electron and crashes, the other core does something different, and the hardware goes, "Oh, these cores do not agree. Let's reboot the system." I thought that was gonna be really tricky to program. But the thing is both cores do the same thing. So you program one core and the other one does exactly the same thing, but one clock cycle later. So you can effectively ignore it until then a bad event happens and they diverge, and the hardware goes, "Oh, oops, upset. You need to reset the system." So playing with this safety critical rated hardware- Interesting ... has been fun. Jonathan Bennett: Yeah. Jonathan Pallant: I also... I- I'm from Cambridge originally. Cambridge was the city I was born in. I've been hanging around with the Raspberry Pi people since 2012- ... with the original launch of the very first Raspberry Pi. I had one I ordered day one. I've got it signed by the team. Nice. So I have a soft spot for the Raspberry Pi stuff. I did enjoy doing the Rust port for the 2350. I think having RISC-V and Arm on the same silicon is just really fun. It's w- my first opportunity to write Rust for RISC-V. N- maybe the first Rust program that actually had to compile for both architectures, 'cause that's not normally a thing for bare metal. Indeed. You are either building an Arm program or a RISC-V program. So to suddenly end up where you want one program, talks to the UR, talks to the I2C, but it compiles for either architecture- With all the peripherals at the same memory addresses. I don't know, that was fun and unique. I'd never had to solve that one before. Jonathan Bennett: Yeah. Yeah, interesting. I, I have... Speaking of Raspberry Pi and their unique peripherals, I can't help but think about- ... the RP1, and I still, I want the little RP1 chip on a PCI Express board that I can slap into my desktop or put into a laptop. Yeah ... that's my dream. I asked the guys at I, I think it was Embedded World, actually. I, 'cause I was there this year. I don't think I ta- I don't think I bumped into you. I don't remember for sure when they- We Jonathan Pallant: did, 'cause that's how this podcast happened. Jonathan Bennett: Oh, that's right. Jonathan Pallant: Okay. 'Cause you came by and I said, "Yeah, I'd love to do that." And then Florian was like, "Oh, is that the guy from Floss Weekly?" "Yeah, we should give you his number. We should do that." Jonathan Bennett: Okay. Yeah, that makes sense. And so I've got your business card somewhere on my desk here underneath all of the DEF CON prep- ... that's just piled up. Anyway, I went to the Raspberry Pi guys, and I'm like, "Hey, you guys really need to make the RP1 available." And they're like, "We don't wanna make it available because we know our competitors will buy them and start putting them on their boards too." Good Jonathan Pallant: point. So you know they've got dual Cortex-M3s In the RP1 Jonathan Bennett: I don't know a whole lot about what's actually inside the Jonathan Pallant: RP1 Yeah, so it is actually a lot... have you ever clocked the numbering? So the, the RP2040, if you look at the chip, silk screened onto the chip, it simply says RP2. Jonathan Bennett: So- So I know that was the second board that they start- the second chip, excuse me, that they started designing in-house. Jonathan Pallant: Yeah. And Jonathan Bennett: The RP- See, so the Jonathan Pallant: RP1 was first- The RP1 and the RP2 was- Jonathan Bennett: And then there's the Jonathan Pallant: RP3 as well Do you remember what the RP3 was? Jonathan Bennett: Wasn't it the power supply? Jonathan Pallant: It is. If I remember right, it is the custom version of the Broadcom SOC that has SDRAM stacked on top So the Raspberry Pi Zero I think it's the 01, maybe the 02. Someone's gonna correct me. That's fine. One of the Raspberry Pi Zeros has the RAM sitting on top of the processor- ... a package on package, and that was the RP3. Jonathan Bennett: Oh, okay. Jonathan Pallant: Which according to the numbering, they started third, but was the one they managed to ship the first. RP2 came out in the middle, and then finally RP1 came out. But as I understand it, there are some Cortex M3s in the RP1, and we're like, "Can we program them?" And they're like, "Nope, absolutely not. You- ... that is not designed for general use." I can imagine they would not wanna deal with random people like me asking, "Why does it do this when I poke this register?" And he's like- ... "Don't poke that register. Just leave it alone." That's- ... that's Jonathan Bennett: not the kind of errata we want to write for this. Jonathan Pallant: No. That- I think that is a chip that they have enjoyed using. I hope to see it again on, on some other boards, but who knows what RP4's gonna be, right? Yeah. It's interesting to see what comes next. Jonathan Bennett: I just love the idea that you can have PCI Express and then GPIO and SPI and I2C, and I want that on a desktop. Do you know how- Yeah ... how amazing it would be to be able to do development and debugging work on hardware if I could have an accessible SPI bus on a desktop? Would be amazing. Jonathan Pallant: Yeah, without a s- without a stack of USB dongles all hanging off some- Ah ... crummy $10 USB hub that likes to reset. Yeah. Jonathan Bennett: Yep. Absolutely. But there, there are some chips out there that'll let you get SPI and GPIO. I2C, I think too. There's a Chinese chip called the CH341, which I've used a lot here recently. In, in fact, I met I met Nirav... Oh, I can't remember his last name, but Nirav, the CEO of Framework, and I gave him one of these guys. It's a Framework module, and it's got a CH341 in it that lets us talk to LoRa chips from on a desktop, which is a lot of fun. Very cool. No, no Rust- Do you have- No Rust code on, running on that yet, but maybe one of these days. Jonathan Pallant: Do you have a, like a box of boards you've bought and you thought, "That's gonna be fun one day"? So I ha- I have my guilty box. It's things like, "Ooh, that looks fun." There's a CH chip that has USB 3. And I'm like, "Microcontroller with SuperSpeed USB, that sounds great. I want some of that." So I bought one. It's on the pile with about 19 other boards- ... and maybe one day I'll get round to pulling it out and giving it a go. Jonathan Bennett: Yeah. So you note that when we started the show, I was able to just reach onto my desk and pull up a Seeed Xiao board, right? And that's not the only one that's there. That was just the one that was on top. Jonathan Pallant: Yeah. I just, yeah, there, there's a reason I just have random $800 no- NXP dev kits just kicking around. Yeah. Jonathan Bennett: Yes, absolutely. Yeah. I've got a, I've got a couple of FPGAs around here too. It's something I've really I've not been able to get into it yet, not with a ver- fervor at least, but the FPGAs with fully open source tool chains is really fascinating to me. Is that a realm that Rust gets to play in too? Can we write FPGA code in Rust, or? Jonathan Pallant: So I know there are people who like to do SoC design, and I think there's some that are built on Python where you write Python and out comes full FPGA SoC with core and bus and- and peripherals and so on. I don't know of anyone doing that in Rust, but it would not surprise me. Jonathan Bennett: Yeah. Jonathan Pallant: You can- I would love to get into the open source F- FPGA stuff. What I did do, shout out to Matt Venn on the Tiny ASIC thing, 'cause I did that a few years ago. And made... That I do have which I forget which run we were on, but I have the run that doesn't work unless you get the voltage precisely right. So technic- technically I have some ASICs with my code my designs in that will do a thing. I don't think I'm ever gonna get them to work, but it's cool. I can go, "Yeah my design is in that ASIC." Yeah. That was a lot of fun. Jonathan Bennett: Yeah. That's cool. A- and I guess you can quite easily take that a step to the side and put an SoC on an FPGA and then run Rust code on the Jonathan Pallant: SoC. Absolutely. That, that is one of the great things about RISC-V is you can just grab a core without paying- ... a license fees, drop it on. Rust has great support for RISC-V- ... 32 and 64-bit, so yeah that's a great way to get started, and that Tiny ASIC chip has a RISC-V core on it. Yeah, I should have given that a go. I should get some Rust code running on that. That definitely seems possible. I have also done Rust on PowerPC. I have a bunch of '90s Unix machines for fun because, I... you need a hobby, right? So my hobby- ... is Unix System Admin '98, which is the worst simulator game you've ever played. I- It's like installing H- HPUX Nine on some broken old PA-RISC machine is quite fun ... I've, Jonathan Bennett: I've played that game once. I, ... I had a customer that had an old SCO Unix server, and they're like, "The hardware is dying. We need to resurrect this because it's got our old legacy system." And so my, my solution was to get a copy of the hard drive and host SCO Unix inside a virtual machine on a Linux box. And I was pretty proud of that- Like- ... when that finally came up and worked. Jonathan Pallant: You're lucky it was x86, 'cause I have a PowerPC Macintosh, an UltraSPARC, and a HyperSPARC. I have a MIPS Silicon Graphics. That has run Rust code. I ha- was able to compile a Rust binary- Nice ... for MIPS, but the Rust the LLVM linker just makes ELF files, and it turns out IRIX- Just uses ELF files. Oh. So I just copied all the object code over to the SGI machine, used the SGI linker, and that works. Yeah, I've done Rust on SPARC as well. I know the European Space Agency, still rather fond of their SPARC processors- ... because they have the, the LEON3. SPARC was famously an open architecture. Anyone could make SPARC chips. And so the European Space Agency did, and there is an open source processor the LEON3 you can find online. It's a SPARC V8 32-bit. And so I did Rust, again, just as a fun thing to talk about. I have Rust code running on bare metal SPARC, and that means I ended up as the target maintainer. So Rust has these, these targets, these sort of operating system architecture pairs- ... or bare metal and architecture, and everyone technically needs a maintainer. Some of them don't. And I'm like, "Yeah, this would be funny if Rust could run on bare metal SPARC." They're like, "Yeah, okay, cool you can submit that as a pull request, but you're gonna have to put your name on it as the maintainer. And I'm like okay, that seems fine. I can't imagine anyone's gonna use this." And then I get messages from people going, "Yeah, we've taken your work, and we're now running Rust on RTEMS on Spark Leon3, and it's working great." Huh. By the way, we've observed this, so w- what do you think we can fix this?" And I'm like, "Cool. I didn't know anyone was gonna do this, but great." I'm glad to have, made that exist. It's a lot of fun. Jonathan Bennett: Yeah. Yeah. Y- you also mentioned MIPS in passing, and I've done- ... I've done some MIPS stuff mainly because in a previous life I did some OpenWrt stuff, and a lot of those routers surprisingly are running on MIPS. Yeah. And you would think, oh MIPS, we've left MIPS in the past now. All of this stuff is running on ARM, or they've gone to RISC-V. There's the Chinese Loongson, which is a MIPS-based architecture too. There's still some life left in MIPS and it might be interesting to run Rust there. Jonathan Pallant: Yeah, no, absolutely. I know MIPS, yeah, long and storied history. I think the latest version of the, the Loongson architecture has diverged away enough from MIPS that it now has the target string begins with a Loongson or whatever the name is rather than- ... mIPS. That makes sense. But there are still some MIPS targets in there. Yeah, it's fascinating to dive back and ap- apologies if this is gonna upset any old hands. I'm just gonna say branch delay slots. If you know what that is, you're quivering on the floor, but if you don't, it's fine. Don't worry about it. Never look up what a branch delay slot is. You won't enjoy it. Jonathan Bennett: I'm going to ignore your advice and look up what that is- ... because that's the sort of thing that I lose hours- Oh ... of time reading about. Jonathan Pallant: Yeah, so i- yeah, imagine a pipelined processor that is busy fetching, executing decoding and executing instructions. And if you hit a branch, you've already fetched the next instruction. So why don't you just execute it anyway and then do the branch? Jonathan Bennett: Oh, Jonathan Pallant: branch- So a branch delay slot means you always execute the instruction after the branch before you branch. Jonathan Bennett: That's what x86-64 processors do all the time, and they never ran into any problems from that. Never caused anybody problems. Yeah. No, that was a great idea. Jonathan Pallant: Just having, when you're handwriting assembly, just remembering that, yeah, you're gonna get one extra free instruction before the branch. Jonathan Bennett: Yeah that's fun. Okay. I understand now. I get it. We've talked about the very, very tiny embedded side of things. What about the, the big iron, like System 360 and some of those IBM which I guess a lot of those are just PowerPC, aren't they? Jonathan Pallant: Ah, so System 360 I guess originally starts in the 1960s. And it's really funny 'cause it's like a, a big technology project that ended up being massively over budget and massively late. You think that's a modern concept, no. People have been late delivering tech- Of course ... since the very first kind of computers. And I got into that kind of stuff Because I wanted to build my own computer. So I'm like, "Yeah, what is a computer? And can I turn a microcontroller into a computer?" And so you go back and go, "All right, traditionally, what was a computer?" You think of the movie The Italian Job. There's a, the big mainframe in the background- ... with all the spinny tapes. You look at it and go, "Yeah, '60s computer, spinny tapes, stuff is happening. Yeah, looks, looks fun." So I'm like what is it? What does it do? How does it work?" And computers really haven't changed much. They need some sort of terminal- ... where input can go in and output come out. They need some RAM as like some scratch space to store the stuff they're working on. They need a processor that can execute instructions, and they probably need some non-volatile storage to remember the things they've worked on, be that a spinning drum, a paper tape, a magnetic tape, or whatever. And actually, like a Raspberry Pi Pico, where, what do you got? Pico one, two Cortex M0s at 150 MHz. That absolutely wipes the floor with a Commodore Amiga- ... Atari ST. You'll absolutely spank something like, the Saturn, the PlayStation 1. It's probably not far off my Silicon Graphics O2. Peak 1996 desktop workstation. Obviously has a lot more RAM, but two 32-bit ARM cores at 150 MHz is pretty cooking. So why can't it be a computer? The answer is, A, it doesn't have enough RAM. So I wrote a little MS-DOS style operating system. It boots to a terminal. You can load programs off disk. You can run them. There's a graphical display that's all rendered in real time. The Raspberry Pi Pico's great at doing that. It's nice having a whole second core to run the video so the main core is uninterrupted. And so yeah, just distilling a machine down into what is the kind of smallest kind of computer. And yeah, those basic concepts, terminals with input and output. Your dev TTY- ... is named that because it was connected to a teletype, literally a typewriter on the end of a serial cable- where you could type and it would h- fire the hammers and print on the paper. Can you imagine that, running LS minus R on a machine where everything's printed on paper? Just the amount of paper you would get through. Obviously you can't use Vim. You don't have a screen. You're in Ed. You've got line based editing at that point. But- That is the fundamentals of a computer, and that is the same whether it's gonna be, you're doing a little operating system on a microcontroller or you're on a, a, a modern, big Linux system. I know there's good Rust support for Z Series, current IBM mainframes, PowerPC. We can still do SPARC64. And it's got a big iron systems. They're the same kind of concepts we had in the '60s, which is fun. Jonathan Bennett: One of the, one of the things we haven't touched on yet that I think we, we can't do a Rust show without talking about is the memory safety aspect of Rust. And I always used to say that I was a little unimpressed by the idea of let's say Rust in the kernel because The memory safety aspect of Rust just doesn't jive very well with writing device drivers, right? Because a device driver, by definition you're writing into uninitialized memory locations, and Rust won't let you Jonathan Pallant: do that. Random integer and random memory addresses. Jonathan Bennett: And so- That's Jonathan Pallant: how the machine works ... Jonathan Bennett: yeah, and the dr- the driver is just all gonna be unsafe Rust code, right? I I'm curious your take on that. I don't nec- I don't necessarily take that approach the, quite the same, but e- educate past me as to why there's still some some value in writing bare metal Rust when it's all unsafe anyways. Jonathan Pallant: Yeah, so I think that, I think there's kind of two, two aspects of that. One, actually the, the kernel's the very low layer part of the s- the system, and the bit that actually has to get a right to the memory, put the random number at the random memory location, is the very bottom of this stack of code. And everything above it doesn't have to do that because it's got the device driver to do it for you. Yeah. So if the device driver itself has the unsafe code in it, we can check that and validate that and manually review it and go, "Yes, this looks good. That's what the data sheet says we should do." And then above that, we can build a bunch of safe abstractions. A serial port driver. Is it okay to send some bytes over the serial port? Yeah, provided you've enabled it and configured the board rate generator. I can do all that in safe code. I can design an API that means it's impossible to send data on the UART until the board rate has been configured, 'cause I've got a type system that lets me do that and take ownership, and this is an unconfigured UART, and this is a configured one. And this one lets you send text, and this one only lets you set the board rate, and you're gonna have to do one before the other. So all those kinds of problems, I don't need to access random memory to do that because I'm building on top of this kind of low level that does. But I was also pointed to things that kernel maintainers have said. I think it was Greg KH at a recent Rust conference in the Netherlands. He said that introducing Rust into the Linux kernel has made the Linux kernel better, which was not a thing that he was expecting. But it's being forced to think about those Rust terms. I've got a value, and I'm giving you ownership of it. I'm calling a function, and I'm getting ownership of the thing back, and now it is mine. Or I'm calling a function, and I'm only letting you temporarily borrow it. Maybe you have a shared borrow, and you need to be aware that other people may be borrowing. Maybe you have an exclusive borrow. You can assume that nobody else is using this object right now, 'cause although you don't own it, you do have exclusive access to it. Those concepts exist in C The problem is the C language doesn't have the syntax and the terminology to describe them, so it goes into the comments. When you call this function, you get given a pointer to foo. You must call free on pointer to foo later because I've heap allocated it and it belongs to you. Or you must pass the pointer to foo into the foo destroy function later- ... and then you must not use the pointer again after you've done it. But it's all in the comments, not in the code, which means the kernel the, the compiler can't check it. Jonathan Bennett: The compiler- Jonathan Pallant: So the thing I Jonathan Bennett: think they- Yeah, the compiler will let you break all of those contracts- ' Jonathan Pallant: Cause it can't read it because it doesn't know about Jonathan Bennett: them. Yeah. Jonathan Pallant: It can't read the comments. It has no idea whether you're supposed to use this foo pointer and where you're not. Whereas in Rust you can. So when the kernel was forced to adapt their C APIs to make them usable to a driver written in Rust, the Rust people went do I own this? Is it mine? How do I destroy it? Where do I have to give it when I'm done? Are you just giving me a thing I gave you earlier, or is this a new unique thing you have created for me?" And I think that kind of understanding those concepts and going, "Oh, yeah, actually we could probably make this C API better in exporting the features you need for the Rust code," actually this becomes a better C API for all the C drivers as well- because these things are expressed better and it's kind of understanding of this kind of moving and ownership and borrowing. I, yeah it's fundamental to how computers work, but it's nice to have a language that lets you express it yeah, natively and not just in the comments. Jonathan Bennett: Yeah. One of the w- the early, early, this is a long time ago actually, but one of the criticisms I saw about all the different efforts to make Rust and C work together was that it was essentially making Rust under the hood in its memory management actually speak the C, sort of the C memory management language, and that I think s- I can't remember who it was that was writing about this, was really critical of that idea, and it's like we're taking Rust and we're making it almost C in order to make it work together. I've not heard anybody make that point in a long time though, so m- maybe that was just a a red herring a, a false road to chase down. But- Jonathan Pallant: I think it's, I think it's a fair observation because, yeah, if you want a Rust program to interface with a C library, or vice versa, a C program to interface with a Rust library Raw pointers and C compatible types are going to be involved. But the thing is for the Rust thing, whether it's the program or whether it's the library, that that C-ness only infects the edges, and you can wall it off and go, "95% this is Rust." We've got our unit tests, we've got our code coverage, we've got our Clippy tool giving us opinions. All of that is gonna work across almost all of this code base. But right at the edge, I'm like, "Okay, here is my heap allocated, my boxed type." I'm gonna convert it to a raw pointer and give it to the C library, and then later the C library's gonna give me the pointer back, and I'm gonna look at it and go, "Did you give me null? Did you give me a non-aligned pointer?" Does this actually look like a thing? And like, all right, it seems validly aligned. It's not zero. Let's assume it actually is the original Rust heap allocated thing that we had before, and you can convert back having done those checks. So most of your code is still very Rust, but there is this kind of C thing at the edge, which is only needed if you need to interact with it from C. If you're gonna interact with it from a Rust library, we just bypass that. Yeah. Just go straight into the Rust APIs. And I would say that one of the disadvantages of using the C compatible types There's a great feature in the Rust compiler. If you have a structure with three fields- you may assume the first field appears at the lowest memory address, appears first in memory. In Rust, that is not true. The order of structure fields is not defined. The compiler can change it between- ... compiler versions, and it allows it to remove padding because it can go, "Oh, you've got small thing, big thing, small thing. Let me put big thing first, two small things after-" "... and now I've made your structure smaller," which is great when it's Rust code calling Rust code. C compilers don't love that. They do rather assume the structure is laid out in the order you wrote it. So you just go, "Hey, Rust. This is a C-compatible structure. Don't do any shenanigans. Just put it in the order I put it." And then it's C-compatible, and Rust and C can just share the same memory and ha- have the same view of the same data. The reason you don't do that by default is it gives you padding, where previously, a Rust-native version might not. So it's a little bit of a price to pay with that, but it's generally not too bad, and I think it's worth it. Yeah. And we know there's a lot of great success cases, Firefox, Android, and so on- ... introducing bits of Rust into their code base and generally the trade-offs kind of work out. Jonathan Bennett: Yeah. Just a second ago you, you mentioned a Clippy tool, and I couldn't help but think of starting up Microsoft Word back in the day and little Clippy popping up. "It looks like you're writing a let," so do... Is there a Rust tool, "It looks like you're writing a system driver. Here, let me help you with it." Jonathan Pallant: Yeah, and that's literally why it's called Clippy. The Rust compiler- Jonathan Bennett: Oh ... Jonathan Pallant: will tell you when your code is wrong, right? There's errors and warnings, where a warning is, "This is probably wrong, but I'm not 100% sure." So it's about whether your code is correct. Clippy is merely a tool with opinions. And it's a great tool, and any kind of static analysis of a Rust program is probably gonna be built around Clippy. I can give you an example. You've got some kind of array, and you would like to iterate through all the items in the array. So you write for i in 0 to array.len. I'm doing my array indexing, I've got my integer, it goes from zero to len minus one, that's how the loop works, and I'm gonna index my array with my variable i. And that is not wrong, and the compiler will give you no warnings- ... 'cause that is all very valid Rust. If you give it to Clippy, it says, "It looks like you're trying to index through an array. Why don't you write for item in array instead of for i in 0 to len?" That's an opinion. The code you have is not wrong, but actually Clippy's opinions are generally pretty good, and you probably should have said for item in array instead, because it's probably gonna be faster, you can maybe skip a bounds check, you're not gonna accidentally go off the end, all that kind of good stuff. Jonathan Bennett: It's an opinionated style linter. Jonathan Pallant: Yeah, and highly configurable. So it has a default set of lints. But I know a lot of commercial customers have their own in-house rule set. You're probably familiar with the idea of a coding standard. Yes. Every company's got a C coding standard, so many different ways to write C. They're starting to get Rust coding standards, and they're usually expressed in the form of a set of Clippy lints- ... that you should have turned on. Would you believe the Society of Automotive Engineers, the SAE, have published a document which is basically a guide for writing Rust for high integrity systems? I think the document is JA 1020, 1020, and it's a document that you have to buy it. It's a commercial thing. But it includes a list of Clippy lints, but also rationale for why every single one of them should be turned on. It's al- almost like a sort of MISRA for Rust, which doesn't yet exist. But it's quite nice if you're writing Rust code to be able to have this thing and go, "Okay they've clearly thought about it a lot harder than I have, so I'm just gonna turn on their default set of rules," which is probably more than Clippy has on out of the box. And a lot of them are great, like every unsafe block should have a comment above it that begins with the word safety and then a colon. Like you should explain why you're telling the compiler, "Trust me, bro, I know what I'm doing." Because you may not know what you're doing. You do. So bit- Clippy can check that. Clippy can go, "Oh, here's an unsafe block without a safety comment above it." You should fix that. And turning on those kind of rules is a great idea. Jonathan Bennett: Yeah. Yeah. That's really good. All right boy, I think we're, I think we've been going at this close to an hour. I gotta ask, would you recommend somebody either just getting started in the programming world or even an old hand to go out and learn Rust and add that to their resume? Jonathan Pallant: I think it's really useful because I remember when I was learning Rust, I was a C developer. I've been doing this for gosh, I left university in 2004. Do not tell me how many years that is. I do not wanna think about it. But I spent my professional life writing C code- ... building all these kind of embedded products. And as I learnt Rust, I became a better C developer. And I write Rust 'cause I think it's a great tool to solve the kind of problems I have, but even if you don't actually want to write Rust full time, it's still fascinating to learn those concepts about ownership and borrowing, and it will make you rethink those C functions that return a pointer. And you're like, "Do I own it? Is it mine? Am I borrowing it? Is it a pointer I gave you earlier?" And having that new language to describe that, I think is- ... is super useful. I don't know, it's just super exciting. There's loads of new, fun stuff going on, loads of research in academia. That seem- they really seem to have taken Rust to heart. So yeah, it's it's a fun and exciting thing to get into, and even if you wanna remain a, a grizzly old C developer as I was, I still think it's a useful thing to dig into. Jonathan Bennett: Yeah, interesting. Very good. It- where should somebody go to, you mentioned that there's a, a lot of education material from Ferrous that's available. Where should somebody go to look for that? Jonathan Pallant: Yeah, so you can just find our website, ferrous-systems.com. You can also look at our GitHub, github.com/ferrous-systems. And you will find my training material and my exercises. They're all free of charge. Yeah, you can check them out and enjoy them. Jonathan Bennett: Yeah, very cool. And then before I let you go, I gotta ask a f- couple of final questions, and that is, what is your personal favorite text editor and scripting language? Jonathan Pallant: Ooh, that is a great question. I- I love Sublime Text. V- Codium, VS Codium seems to be the one that is mostly open on my computer, but you just cannot get past the speed of Sublime Text. And it introduced me to the world of multi-cursors, pressing Command+D to select multiple things at the same time. Which is still witchcraft as far as I'm concerned. And when you s- you just see it happen in front of you, that's amazing. I edited 39 things simultaneously in this file. So yep I love love Sublime Text. For scripting languages I've done a lot. I've done Perl and PHP and Python. So let's go with one that doesn't begin with P. Let's just say Tcl, 'cause I'm feeling obtuse. Jonathan Bennett: I'll Jonathan Pallant: have to go- Everything's a string. Why not? Jonathan Bennett: I'll have to go, I'll have to go look that up. I don't know what Tcl scripting is. Jonathan Pallant: Oh, yeah. Oh you're gonna have so much fun. Jonathan Bennett: Another rabbit hole for me to dive down after the show. Jonathan Pallant: Absolutely. Jonathan Bennett: Absolutely fun. All right. Thank you, Jonathan, for being here. It has been an absolute blast to talk with you. Jonathan Pallant: Yeah, it's been great. Thank you so much. Jonathan Bennett: All right. That is Jonathan Palant of Ferrous Systems talking about Rust on the other end of the spectrum. And so we do not have a guest yet for next week, I don't think, unless one is scheduled in the time that we've had the show. So if you want to be on the show or know a project that should be, you can drop us an email. You can get us at [email protected]. Let us know who we should be talking to. I will be at DEF CON later this week, and so if anybody is there, make sure to stop by and find me. I don't have a booth that I'm running this time. I'm just on the run, but find me and we can say hi, and we can chat for a bit. But yeah, we appreciate everybody that's here. Whether you watch or whether you listen, whether you get us live or on the download, thank you so much for being here, and we will be back on Floss Weekly. All right, cool. We had a bunch of fun show titles that got suggested too. Oh, Tickle TCL, I have heard of that. I'm not super familiar with it, but I... Yes. Some- someone linked it in Discord, and it's like, okay I know what TCL is. Three Things in a Trench Coat was a lot of fun. Source Code In, Machine Code Out. Sometimes There Are Heavy Things To Be Lifted, that one's long. A Tool With Opinions. I think I like either A Tool With Opinions or Three Things in a Trench Coat. Jonathan Pallant: I think if I was gonna pitch you an alternative, I'd say Don't Bring a Forklift To The Gym. Jonathan Bennett: Don't Bring a Forklift To The Gym. That one's pretty good too. Yeah. It's a little long. Jonathan Pallant: It is, yeah. Jonathan Bennett: See, I really like both Three Things in a Trench Coat and A Tool With Opinions. I think we'll probably go with A Tool With Opinions because it's also, it's a great snapshot of what Rust itself is, I think. Jonathan Pallant: It is, and the British people in your audience will assume it may be a description of your guest. Jonathan Bennett: Yes. That's great. That's brilliant. All right, we'll go with that one then. Jonathan Pallant: I'll live with that. That's fine. Jonathan Bennett: I may reference that pun somewhere in the show description too. That's great. Thank, thank you, sir, for being here. It's been a lot of fun. It's really been a blast. Jonathan Pallant: Yeah, my pleasure. It's been great. Jonathan Bennett: All right. See you next time. Let me get a title and tease recorded here. 878. Bye. 878, a tool with opinions. Yes, I had not thought about the the, the British-ism baked into that, but that's really good. All right. This week I talk with Jonathan Palant about Ferris systems and Rust in the embedded world. It's a lot of fun. You don't wanna miss it. This is Floss Weekly, episode 878, recorded Tuesday, August the 4th. A tool with opinions One of those should do it. Thank you everybody for being here, and thank you again, Jonathan. We'll see everybody next week. Bye. Jonathan Pallant: My pleasure. Bye.
• 
  Episode 878 - A Tool With Opinions 05.08.2026 1t

  This week Jonathan chats with Jonathan Pallant about embedded Rust! Learn about the growing Rust driver library, the different ways to build a Rust stack on an embedded device, and how the opinionated tooling can make you a better programmer!  You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 877 transcript 29.07.2026

  FLOSS-877 Jonathan: Hey folks, this week we're talking with Francois Proulx about smoked meat, and poutine, and Bagel, a trio of open source security tools that you don't wanna pass up on. This is Floss Weekly, episode 877, recorded Tuesday, July the 28th: RCE as a Service. It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we're revisiting a topic that we've talked about in the past, and in fact, revisiting a guest. We've got Francois Proulx of Boost Security. These guys are experts in supply chain security, and really one of the main things that we're talking about with this is GitHub CI, continuous integration. And so a lot of projects will have a CI set up to where you run automated tests on your code changes. Where this can get dangerous is if you allow the outside community, so untrusted people, to make pull requests, and then that CI engine runs against those incoming pull requests. And then you have this kind of strange problem, that you're running untrusted code inside your GitHub repository, and surely that doesn't ever go wrong. Yes, sometimes it goes wrong. We had Francois on the show about two years ago, and we talked about a project then called Poutine, which was a scanner, an automated scanner to take a look at a project's CI runs, and essentially give you an evaluation. Do we think this is safe or not? And in the time since then surely we've fixed this problem and it doesn't exist anymore, right? Unfortunately I don't think that's the case. I'm gonna go ahead and bring Francois on. He's the guy that has the answers, and we're gonna find out. So welcome, sir, to the show. It is great to have you back. Francois: Thank you. Jonathan: Yeah what, w- what, did I get the problem space right? And then what's changed in the last two years? Francois: Ver- very good introduction. Yeah, I think you got the, the takeaway from the last discussion, and I guess you, you did your own homework on your own projects and could actually see why the tool was created. Why I created it, why it's a, it's an issue that especially maintainers... this audience is both consumers, like users of open source, as well as maintainers. Jonathan: Yeah, absolutely. Francois: For the maintainers, I think my message was that they should give the tool a try so that they can protect their own end users, right? Jonathan: Yeah, and I think we're probably gonna pitch this in the context of GitHub, but this is not just a GitHub issue, is it? There's there's GitLab, there's Codeberg, there's glued together CI scripts that run on infrastructure that's not any of those three things. And I imagine you could have problems on any of those contexts, couldn't you? Francois: 100%. So effectively you know- Basically any source control system, be it GitHub, GitLab, or whatever- Yeah ... that can automatically trigger the execution of a CI environment upon the submission of new code changes, just the, you know, triggering on such an event, it they can have the same exact problem. So really the devil's in the details in terms of the configuration of what is the behavior that happens, like the, effectively the secret, or does the execution of the arbitrary code changes get hacks- access to secrets or not? If it's just arbitrary code execution of the new tests, for instance, or the new code potentially it can be done in a way that's perfectly by design and without clear risk. Of course, if you were to reuse the same CI environment, like the same machine is is to be reused across different runs, then there can be pollution across those. But assuming it's an ephemeral run runner environment it can be done safely. But then, yeah, like the, the way it gets configured and the various components that get pulled during execution is really what matters. Jonathan: Yeah. And I think part of the reason why this is such a sticky problem to solve is 'cause on one hand, obviously when you have a pull request from the outside, so like by definition you're talking about untrusted code. It's literally untrusted user-supplied data, and you wanna do a processing run on that. But- Historically, we've not been concerned about, and I'll just make this very generic, we've not been concerned about malicious code inside of make files, or malicious code inside of build scripts, or malicious code in all these various places, even in C and C++ headers, for that matter. You can do some interesting things just with preprocessor directives. And that's been a, historically a blind spot because oh, if you're compiling this code on your machine, surely you already trust it. And then you take this and you put it on the cloud, which is, as we all know, just somebody else's computer, and because it's on the cloud, it has to have access to some sort of, to, to some sort of secrets. It has to have access to your build environment and your accounts in the cloud to one extent or another. And there's multiple ways that this can go wrong, and I've seen firsthand someone will run a build, and part of the build is go grab all of the variables from that you can get ahold of, basically all your environment variables, and then send those off to somebody else's server, and then you lose your secrets, right? But then there's also even other problems, and I think GitLab had an issue here at one point where because you are... You're in this inside the runner you're validated. You can then potentially go and do things to the project, and so there's that danger as well. And this is not just theoretical anymore, is it? There's been real attacks pulled off with these sorts of problems. Francois: Yeah, absolutely. So that, that is exactly how as we publish open source tools, we write articles on the topic, pushing the boundary of the, the research in that area. And I speak at different conferences and, on podcasts and whatnot. But the, the last series of conference talks that I gave, it really starts with this fact that since we last talked, about two years ago- ... the the threat actors the, the bad guys have also been listening to the podcast or my talks- or seeing the ar- re-reading the articles and using the same tools that hopefully you guys, the defenders the, the maintainers- ... have been using, but they effectively flipped it or used defensive tools that were designed for defenders as an offensive tool for them to find zero-days effectively. Because those tools, they can find vulnerabilities in the the attack surface of the, the build pipelines, and then lead to poisoning the the well, poi-poisoning the open source release. And that's what we saw with worms, like in the fall of 2025. I'm sure all of you have heard about the different worms spreading on NPM and PyPI mostly, shyahlood, and there's been a number of iterations of those- Yes ... autonomously propagating malware in, in open source packages. And e-even in the spring of 2026, we saw another wave that was not necessarily using worms, but much more let's say, skilled threat actors that were r- that really understood the benefits that they could get by exploiting effec-effectively large open source projects and have downstream effects in large organizations, right? Jonathan: That's the, the classic supply chain attack, right? Particularly if you can pull that off without being spotted for a while. So then you can you get your tendrils into one project, and that gives you a l- a, a diving board to then get your tendrils somewhere else. And the longer someone can go without being spotted, the more of a mess that they can make. Francois: Yeah. But at the same time even if it's not very long, but everything is automated, they can, achieve the end goal- ... v- so quickly that no one can really stop something like that. Like- Right at best, it can be detected as it happens or as it has already happened. And if in some case we saw them pivoting multiple levels from the initial access vector, the initial point of entry- ... Francois: To steal secrets and then pivot down- downwards in the infrastructure, like cloud infrastructures- of large corporations namely. Jonathan: Yeah, absolutely. And then you, I think sort of tongue in cheek, but it's a point worth following up on. You mentioned the idea that, oh the, the attackers are listening to our podcasts and they're using our defensive tools, and that's the challenge anytime that you build like an auditor or a a red team tool that like, there's not a whole lot of difference between what an auditor does and what an attack tool does. Is that just the nature of the beast? Is that just the way that it works? Francois: Yeah, I think there's no... we're not in the 1990s anymore- ... where we try to keep the precious, elite tools behind the curtain, and we only give them to certain trusted people. The cat's out of the bag. Yeah. The techniques are well-known. And recreating most of those tools is not rocket science. Once you understand what you're looking for, you can probably just use grep, in many cases, it's just configuration mistakes or a couple specific things that you're gonna be looking in this case, in YAML files. For the most part, those CI/CD configuration will be in YAML files in specifically named folders. And yeah the simplest thing you could be using grep and highlight some things that you would then audit manually. So those tools are just doing that more efficiently, but ultimately that's, as you said it's that, that's what we have to wrangle with. The technique is known and by b- both defenders a- and attackers. Jonathan: Yeah, absolutely. We're not in the position where if your CI gets hit, you can just say, "Oh, it must have been a nation state that did it." No, it may have literally been a teenager in a basement. Yeah ... I blame, and again, it's not just a GitHub problem, but GitHub is the, the 800-pound gorilla in the room of of online Git repositories. And I blame GitHub for this, 'cause I've gone down this track a couple of times. You made me... I don't think that was the first time I'd heard of it, but you definitely made me more aware of this when we talked about it last time. And so I've gone and I've looked at some of the projects I've been involved in, and I've gone down the rabbit hole of do we do this right or not? And I've read through the GitHub documentation on things like, just for instance, the, the pull request event versus the pull request target event. And I will read through their documentation and come away with one feeling of "Okay, we need to do it this way." And then I go and read somebody else's blog, and they say no, you need to do it this way." And I still don't know if pull request or pull request target is the safer way to do a CI run. It's... I just, I wish there was a single answer somewhere that I could look to, and it's like, "Don't do this." Okay, I won't do that. Francois: Yeah. The, the s- the simple answer is pull request target is m- let's say- Is if you're to grab something, pull request target, it's not the only one, right? But if you're gonna flag that, like that's a clear thing where you need to stop and audit the rest of the code. It doesn't mean it's not is insecure kind of a Boolean flag, right? You m- you can use this in a safe way. It's just that it's a bit of a foot gun. Like you, you need to really understand how to use it in a safe way because it comes, great power, with great power comes great responsibility. It does grant access to the secrets during the execution of the runner. The big difference between pull request and pull request target, and specifically in the case of GitHub actions, is that one of them will not grant access to secrets, but it will i- it will run the latest version of the pipeline. So basically, this is in the case where you want your contributors to make- ... changes to the build pipeline- ... and you want them to be tested immediately, right? So that is let's say in the scenario of enterprise or like in a private repo or when you have direct maintainer contribution, this is fine because you want to test like directly the, the latest version of the build pipeline changes. But pull request target on the other hand, it effectively locks the version of the CI configuration to the one that's already been merged. So that's the biggest difference, is like one is untrusted build configuration, and the other one is trusted build configuration. But the big difference is that one doesn't have access to secret, the other one has ac- access to secret. If it's a trusted pipeline, it means that as long as there's no vulnerability in that pipeline configuration, everything is okay. Jonathan: But- But if your repo has a Python script in it that gets evaluated and executed during build time, which is quite common by the way, you have a problem all over again. Francois: Yeah, exactly. So in, in fact so much so that y- you were asking did the, is GitHub doing enough? And I would say in the past three, like four, basically since March more or less, there's been a big like... They've been, they've started to really move the needle. And that's in response, direct response to all the big things that happened- Yeah in the spring. So they've made a number of changes that actually in some cases are breaking changes. Like literally they decided to do the very non-Microsoft move of, okay, this is too bad. Too many people don't get it right, so we're gonna completely change the behavior to a more safe by default. Where you really need to explicitly go ahead and say, "Yeah, I know what I'm doing. Let me do it." As opposed to leaving foot go- foot guns, around. And so th- that, that's a pretty major change. Specifically to pull request target, they made a change that was made fully effective as of two weeks ago or so. But it still makes it possible to have the same kind of vulnerability as before, it's just that there's one less way that you're gonna do it, like without even knowing, right? ... It's at least a little bit more explicit on your part. Jonathan: And that's why everybody's CI runs started breaking randomly a couple of weeks ago. That, that solves one mystery. I, I think something that maybe not everybody realizes I have to remind myself of this every once in a while, is obviously Microsoft owns GitHub. GitHub is the owner or the, the trusted source of NPM. And so like all of this stuff comes back to the same group of people getting black eyes from it. And so it's not terribly surprising that they are very motivated to try to fix this. Francois: Yes. The, directly Microsoft for many of the things, like if you're a Azure cloud user, there's a number of components that are directly used in production in Azure, both from the infrastructure side as well as the end user side of their cloud services, like kind of cloud SDK, Azure SDK, and other things, right? That you want to interact with their backend that are hosted on GitHub, open source, and that use GitHub actions. Some of those components are deployed on NPM, so they have vested interest in ah, hardening the path to to Azure, right? Because from commercial perspective, ... Microsoft definitely has- Sure incentive. But it's for the benefit o- of everyone, because a- any, any improvement to this infrastructure is good for open source. Jonathan: Yeah. Absolutely. All right. So that's sort of the background. And last time we talked about Poutine, but now you've got a new tool, Smoked Meat. I just gotta say, I love the carnivore theme going on here. I 100% approve of that. But what is Smoked Meat? What does it bring to the game? Francois: Yeah. Okay. So Poutine was really a scanner. It's meant as a static code analysis scanner. It's meant to basically scan and find all those special YAML files and detect vulnerabilities, like in the misconfiguration in those YAML files. It's gonna say, "Oh, you have pull request target, and then you have X, Y, Z other conditions, preconditions for it to be considered almost certainly vulnerable, almost certainly exploitable." So there's some, caveats to that, like in practice, but it's a scanner and people would ra- run it. Since it was published it's now being used by many large organizations many big open source projects. And it's very lightweight, very easy to run at scale. You can scan a large organization with hundreds of repos thousands of pipelines in just a few seconds, and it's gonna spot those. Then you're responsible for, looking at those results and hopefully fixing the issues, Jonathan: Yeah. There, there's a- ... and Francois: on our side ... Jonathan: it's worth breaking in here and talking about there's a challenge when you run one of these static code analysis tools to then correctly understand the results, make the judgment call of either, "Yes, I think this is a real problem," or, "No, I don't think it is. I think this is a false, false positive." Or even like, "Yes, this could be a real problem, but we mitigate it in other ways." Yes. And so there's a real judgment call to be made there, and sometimes that's really a hard step to take. Francois: Yeah. And exactly, that's, I'll get to Smoked Meat, your, to answer your question in a moment. But a- ab- absolutely as you said, it's it's difficult, and we've been using this tool at scale to scan millions of open source projects. And in the past two years or so, we've made several thousands of coordinated disclosures to many projects like big and small. Like Small, tiny open source projects, as well as things that are maintained by Google, Python Software Foundation Red Hat. And so indeed, like some of those are pretty trivial, fairly easy to point to the bug and to the solution, the way to re-remediate. But other times it can be very difficult to confirm that it's exploitable- ... and then to find the way to address it. In some cases it's more architectural. You need to redesign the whole thing because you, you made like wrong assumptions on how to do that securely, and that's much more painful, right? Jonathan: Absolutely. Francois: But let me get to the answer to your question about Smoked Meat. So Poutine is a scanner, and Smoked Meat is a red team, like offensive tool. So the first one is gonna find the vulnerabilities, and in Smoked Meat I use Poutine as a library. So if you remember, Poutine is a Go, is written in Go. It's a common line tool, but here I just use it as a library, as part of a bigger project called Smoked Meat. Which is really, if you want to simplify it for those of you who may have heard about Metasploit. So Metasploit is an offensive tool, like a red team exploitation tool- Yeah ... where it's gonna scan to find vulnerabilities and then give you automated, like automatic craft payloads to test ex-exploitation and validate that indeed this given vulnerability can be exploited and then you can discover what's behind. Once you achieve remote code execution, you can, in some cases, like find credentials or discover the environment, do further reconnaissance and pivot. Part of Metasploit, there's this plugin called Meterpreter, which is effectively a small shell execution environment that will run remotely. Once you achieve the re-remote code execution, effectively you move from your laptop to somewhere further and you rinse and repeat. Like you execute as if you're sitting on that server observe what's around, and then continue. Smoked meat is this, has the same exact concept, but applied to the context of CI/CD. And the reason why Metasploit would not directly work in the context of CI/CD is because in almost all cases, CI/CD is running ephemeral environments. So the VMs that run your tests, it's typically torn down and completely deleted the minute the tests finish running. Then the next run or maybe parallel run of the test will use new VMs, so it's not like a persistent infrastructure like you would attack a cloud infrastructure typically. So yeah, so that's the gist of it Smoked Meat uses Poutine as a library. It scans, then it will automatically generate the corresponding payload for the different vulnerabilities and let you basically exploit it, get whatever secrets are running in the CI environment. And then re- rinse and repeat, discover further what, what's behind. In many cases it's very common, especially for a bit less for open source projects, but in a commercial enterprise environment, you will have a CI environment deploy to a production cluster, right? And then maybe this cluster is connected to something else. So that's that's really what Smoked Meat is all about. Jonathan: Yeah. You get a foothold onto maybe somebody's network or somebody's CI runs. Yeah. It's a real problem, and that's how you get into some of those messes with it's how you end up with things like we've been shipping this project or this product, these binaries for the past two years, and oh, they've had a backdoor on them the whole time," right? It's that it's that sort of supply chain problem that you can run into. Yeah. So super, super important. Does Smoked Meat work a, so if we wanted to use it inside of a Metasploit run, can you do that? Can you build- ... bu- build the stack on top of each other? Francois: Not really. It's really, it was designed as a turnkey solution, end-to-end, purely designed with the, the intricacies of CI/CD in mind. All the tricks that basic- It knows exactly what will happen when it lands in a GitHub Actions runner, the environment the tools that are a- accessible there, and the different services a- around it. And how to get access to the secrets and the different features that are inherent to GitHub Actions. So it's really meant to be used on its own for that specific type of red team engagement. Sure. I've actually created a capture-the-flag environment for people to experiment with that safely- That's great so that they can practice i- in, in a, an environment that It has a number of challenges. You start basically from a public repo, and once you achieve the first foothold, then you discover private repos, and then eventually you discover a cloud environment, and effectively you can hop through four or f- five or six pivot points a- and get quite deep. And you in, in the process, you exploit about five vulnerabilities in different CI/CD runners basically where you start from that public perspective initially. Jonathan: Yeah. That's actually really neat. I appreciate it when people do that with the tools, because it's very nice to be able to learn how to use tools without actually breaking the law. Francois: Exactly. That's Jonathan: So that's- ... that's really important. Francois: Yes. I mean- We don't, Jonathan: we don't wanna all become Kevin Mitnick, right? Francois: Yes. Researchers like, like me we all create those kind of environments to practice the proof of concept whe- before we disclose something. We try to reproduce that in a sandbox the separate environment where it tries to reproduce as many of the conditions of the real target. And, we can use that to take a video demo and say, "Oh look, if someone is- w- was gonna compromise that, that's how it would execute." And we can do that without doing it in the public repo where other people could notice that, "Oh, look at that, someone actually achieved poisoning the thing," and maybe the maintainer is on vacation, and someone else is gonna come back. So we need to do that ethically, and that, that's why this tool is released alongside an environment that people can practice and learn and then test it on their own projects again, maybe in a separate environment that just tries to reproduce the real setup and before they- they maybe apply the patches. Jonathan: Yeah. It brings to mind a potential problem, and I'm curious if you guys have seen this, 'cause obviously you've been using this tool, and you've used it to help find problems. What do you do if a project is maintained, or unmaintained, but it still has a, a CI vulnerability? Like, how do you even go about trying to fix that? Francois: I think it's the case y- that, that's a good point. And I guess you have to see un- unlike a vulnerability in the code itself, right? Where it would be subject of a, a CVE afterwards once it's patched and people know, okay, this, there's been a patch applied to this specific version- of, and so you should update to the latest version, which has addressed it. In this case, it's really an infrastructure. So it's not the code itself- ... where it is the YAML in the GitHub repo, but it is effectively a side door. It's another way to- ... get access to the repo, like as a s- because you achieve remote execu- execution in a privileged environment. Yeah, to your question if the project is completely unmaintained and the CI/CD is vulnerable I guess hard forking it is is the, what's necessary because otherwise it could be modified and in some cases it can be hard for someone to really determine that it's been compromised if there's no collaboration from the side of the, the maintainer. The, in terms of forensics, in terms of confirming that it's not been exploited, it can be difficult sometimes. Jonathan: Yeah. I'm thinking back to, of course, one of the most famous examples of this, this sort of problem. Not exactly CI related, but still close enough to talk about, the, the XZ hack where the, the backdoor was put into the XZ libs, and one of the things that made that more, more complicated is that Lassie Collin, unsung hero, absolutely the good guy in the story, but he was on vacation when all of this broke out, right? He, it took them several days to track him down and get him to fix things, and that made it all harder to clean up. And I'm just imagining I have seen, I've seen GitHub repositories that are still being used that nobody has touched for months or maybe years, and they're just running on autopilot. And it's not at all difficult to imagine a scenario where an attacker gets access to one of these through a CI run and pushes out a new version or rewrites a Git tag. Which by the way, if you have the option, you should really be referencing things by Git hashes and not Git tags. That's sort of a different story. I, it's, boy it's real challenge to try to deal with that. Do you guys have like contacts at GitHub to where you can- Yes ... you can clean something up if there's nobody at the wheel? Francois: No we do. So as I mentioned, we do coordinated disclosure with many projects. And we do have privilege like contacts, like we establish relationship with people at GitHub, but not just at GitHub. And in some cases, there... I've seen examples where the project was so high profile and the maintainer was overwhelmed or- ... for some reason, and the, the ecosystem impact was so broad that, yeah, like GitHub in some cases like put a freeze, like paused any new pushes to the repo. A- and some cases like reverted, but that's pretty rare. In... I haven't seen many examples of that. Maybe I can think of two or three at most in the past couple of years, where the maintainer was MIA and GitHub had to step in and freeze everything. Otherwise it was cascading. Like es- Yeah ... especially in the case of the worms, like in the fall, it was so bad, it was propagating so fast that they needed to do something a bit more drastic. Jonathan: Yeah, that's a big deal for the for the repository host to step in and make changes. That's outside of the normal social contract with how all of these hosting things work. And so it's gotta you gotta have a, a... It's gotta be a pretty drastic scenario. I think, so talking about XZ, I think that was one of those cases where GitHub hit the button and freeze this and turn it off, and nobody can get to the, the XZ repo anymore until things got fixed. Yeah, I think they did. Yeah. So we've talked about GitHub quite a bit, but this is not just a GitHub issue. Do you have sort of tales from the trenches from any of the other major hosts or anything outside of that, actually? I would love to- ... I would love to know if this is a proj- or problem that you see in just some maintainer's random glue scripts. Does it show up there, too? Francois: A- absolutely. A- again, like it's really a matter if we're talking about... first of all, there's kind of two main threat models that I have to to tell people about. So the first one is the one we just discussed, which is really the public open source project. People can contribute changes using pull requests, right? They will submit new code that is not yet reviewed, and if the CI is going to automatically trigger upon the submission of code changes, then depending on the configuration, we can lead to what we discussed. If the maintainer has been compromised already to some ex- to some way phishing- info stealer malware that ran on their computer, exfiltrated their credentials, or stole them through another, like a p- like basically a first attack- ... maybe a first CI compromise led to getting se- secrets credentials of a maintainer, and then use them to directly attack another project as the maintainer in this case. Like basically impersonate the maintainer, and that's... We've seen that. Also, the, the worms were really using that in some cases to different techniques to seed, to just bootstrap the the, the malware by phishing info stealer, and a CI compromise. And then after the, the worm would... in the, the case of direct automation, it would maybe get as I said, like some credentials of a maintainer, and then discover which projects they can directly attack. So with the second one, which is y- you're basically already the maintainer, , it's kinda hard to to stop that, right? It's not a vulnerability. It's just like the way that you would modify your own project. And we've seen that across any ecosystem. This is really not specific to any, any given platform. And the same thing, like if you steal like a PyPi publish token or NPM publish token, you're gonna publish directly and skip, s- skip touching the source control environment. And in fact, that's what most malware author would do because they, they want you to believe that the source code of the open source project is all good, but what's published on PyPi is completely different. Jonathan: Yeah, definitely a challenge. So what's the, i- d- does d- does Smoked Meat or Poutine, does it give any visibility into that situation? What, what can we do to even mitigate against th- that, that realm of problem? Francois: I- if your question is about a maintainer be- being directly- Right ... compromised, like their own credentials. So in, in between the time that we we came out with Poutine and before Smoked Meat, there's a third tool that I haven't mentioned that we released called Bagel. So all the food theme. Jonathan: Absolutely. Francois: And Bagel if you don't, didn't notice, like, all those food items, they're all kinda staple Montreal food because, we're a startup in Montreal, and Poutine is a Quebec- ... Francois: Food. Smoked Meat, if for those who don't know, who never came to Montreal, Smoked Meat is the way we refer to the, a special type of brisket smoked brisket with a marinade and, you know- Jonathan: Yeah Francois: it's just called Montreal style Smoked Meat. Jonathan: Oh, yeah. Francois: And Bagel, we have our own Montreal style bagels that is different than the New York style bagel. Got it. The point is that Bagel is all about looking at the posture of your laptop, your e- endpoint, where you write code, and trying to limit the blast radius, limit the impact of a malware running on your laptop- So that, for instance, take a very simple example. You have your SSH private key in the home folder without a passphrase, right? Malware runs, it's gonna get that, and the passphrase the, the SSH key is usable immediately by the attacker, versus you have a very long, complex passphrase. Even if they get that, they would need to crack the passphrase before they can use the SSH key, right? So things like that we would if you run Bagel, you're gonna find secrets lying around your laptop, maybe in your bash RC you load as environment variable some, some API keys to automatically push somewhere and, things like that. And then it comes with some kind of remedi- remediation guidance to tell people how to do that in a more s- safe manner. Because we know that, like- You're gonna click the wrong link, you're gonna download something- ... something will run on your laptop, and then- Yeah. That's true ... if you limit the amount of things that something running at a given moment has access to or there's at least some kind of user interaction, some kind of confirmation. Personally I'm not here to advocate for, I know that you're a big, like, framework la- laptop user. My colleague actually has that. But on, on a MacBook, I use Touch ID, which the secure enclave fingerprint-backed execution environment to store private keys. So my SSH key, for instance, requires me to put a fingerprint in interaction. So even if a malware runs, it would, the malware would need to trick me into putting my fingerprint exactly at the moment it wants to sign- ... using my private key. And then that means that the malware could not do that 1,000 times in one second. It would need to have me press 1,000 times with my finger, right? And the same thing applies to many other types of credential like a passkey, a YubiKey, and whatnot, things like that. They effectively move the sensitive key material off of the normal userland execution environment so that- ... malware running on it will not be able to at least propagate autonomously so quickly. Jonathan: It- Francois: So- Jonathan: It gives you more friction in your workflow, which you sometimes need. Francois: Exactly. Jonathan: I've talked about that- At least for Francois: those- Jonathan: Yeah, I've talked about that concept of friction when it comes to AI pull requests, but I've never quite thought of it in these terms, but that's really an interesting connection to make. Francois: Yeah. You want- So in fact, talking about AI and whatnot, ... we didn't do that We had to go Jonathan: there. We had to go there. Francois: Okay. Okay, yeah. How many minutes in we didn't use that acronym? But so Bagel is, one thing it is gonna find is if you use Codex Cloud Code, or whatever on your laptop- maybe you don't know, but it saves those conversations in a temporary folder- ... so that you can resume conversations. And very often, without you even knowing, maybe you never copy-pasted some secret, but in the process of Cloud Code reading logs to- Yeah ... for you to troubleshoot something in a production environment, it actually- It caches all kinds of stuff pulls in the context. Yeah. And those secrets are gonna be in that temp folder, so Bagel is gonna find them right away. There's even a feature to scrub it automatically, like- Oh, nice ... redact with, like a, a fake token. Jonathan: Yeah I imagine that part of the solution to this also is just to carefully scope your tokens so that, you create- Yeah a token for one use, and it only works for that, and it does not, it's not the keys to the kingdom for everything you have on GitHub- Yeah ... or every SSH, what have you. Francois: Yeah, basically, like, if, especially people who maintain open source projects typically everything is public, so read-only access to- things that are already public is not considered a sensitive secret. E- even if, go ahead and steal a token which just has access to a public repo, just read-only, big deal, right? And that's exactly the design consideration that GitHub has when it comes to pull request versus pull request target. The execution environment will get a secret that has read-only access to, in that case, a public repo. So the attacker gains no no further foothold. It's at the same level as it started initially. But so to your point, yes having different credentials, like a default credential that for instance, that SSH key, you can split your SSH key for pulling and pushing, but signing the commits is a different key material. You can do a SSH-based Git commit signing. So having two different keys, one that has the friction of the fingerprint authentication, the other one without, and then- ... when you do auditing after, afterward, you, you can confirm, okay, all my Git commits are signed. They're signed with the key that I expected and anything that deviates from that is, to be, Reviewed with suspect Jonathan: investigated maybe. Yeah. Yeah, that makes a lot of sense. So you opened Pandora's box. You talked about AI first and now I gotta ask about it. And i- in, in just in thinking about the... Of course, the world has changed significantly since we talked two years ago. I'm just kinda curious is there a, is there a strong argument to be made still that Poutine is the way to go to do these scans versus just, "Hey, Claude, take a look at my .github folder in this repo and tell me if there's a problem"? Francois: No, that's a good question, and in fact- We, when we do this kind of scanning of millions of open source projects, we leverage agentic like agentic approach. We do a hybrid approach. So basically we use Putine at scale because it's very efficient, very lightweight, no tokens needed. And we've tuned- And you don't pay $1,000 Jonathan: a month for the Francois: tokens to run it . Yeah, exactly. So we've tuned the rules in both... In, on one side, either it's very narrow but very high signal. It's like if it's gonna flag that, it's nearly 100% that it's exploitable. We can report it almost without looking. I'm not saying that, but it, it gets to the point where the rule, the deterministic rule from Putine is gonna highlight the, the, the low-hanging fruits. Then the other approach is we use Putine with effectively several preconditions that are not necessarily all proven to be l- lining, like all the ducks, not necessarily all in, in a row. But then using AI to more creatively combine and say, "Okay, yeah, it has this condition, but not that one. But given that it has this other thing, there might be like a, a less obvious way that static, like a deterministic scanner- ... would have skipped." So a hybrid approach is really what we've been using at scale. A- and also for those more advanced kind of multi, m- like chained exploit where you have a low risk thing combined with another one. That's the, the, the new thing that is now possible using Claude and others. Jonathan: Yeah, that makes a lot of sense. It is amazing to me how quickly we have gone from, daniel Stenberg was one of the first guys to really, to blow the whistle on this, because curl is one of the most installed projects anywhere. But people trying to game the trying to game the system and be able to find exploits to make money in, in, the, the various bug bounties, to u- using AI to do that and therefore just creating absolutely garbage reports. How quickly we've gone from that to AI is now a tool in every security researcher's toolbox, and it's finding real problems every day and helping us fix them. I it seems like I blinked and we went from A to B. Francois: Yeah. It does require ex- experts in front of the, the keyboard. But that's, there, the... I agree that there was a period where the experts were like non-believers and in the meantime, you had the non-experts that were just, like creating slop. And then people s- realized that, okay, if you use that smartly and you don't just trust the output right away, but you use it to augment your scanning, like vulnerability research it's quite amazing, yes. Jonathan: It, it seems like if you're using the latest models though, they are better about not giving you slop. Francois: Yes. No, absolutely. Jonathan: There's always, just like when a human finds something, there's gonna be a judgment call as to like how serious this is. Is this a real problem or was this part of the design? Is this an accepted trade-off? There's all of those things that honestly that make our job harder. If you could just, if there was just a magic rule set that, oh, this is a CVE and this isn't we would probably be out of jobs, right? Yeah. Francois: That being said, if we go back to the question related to did Git- GitHub fix the whole thing they are making... they published a roadmap with a number of changes to the environment where you can now set policies across your organization or specific repos that will turn off or basically prevent execution of workflows using certain risky features. So now you can effectively decide, like purposely decide that your policy is to effectively remove those footguns or explicitly allow some of them in specific cases that you did do your due diligence. So there's a lot that GitHub has done in the past two months or so, and there will be more in the next few months for those of you interested. Jonathan: I, in, in thinking about this more and having listened to your explanation of the two targets in GitHub runs, I don't understand why GitHub does not make a third target that is like a pull request target safe, where you take the safer of both of those two options. So you pin the .github folder so that you're running your known safe CI, and you also don't share the secrets with the run. This seems like an obvious fix. Francois: Yeah. No I understand. It's, yeah. At the same time- Jonathan: That would make life so much easier ... at the same time- We could just go set that for everything and be done with it. Francois: No, it's true. I think, It all comes down to is the code in the pull request, has it been remo- reviewed? Do you have execution o- of it before it gets reviewed? But every... As soon as it gets merged, it's merged, right? So there, it's trusted. So it's really this this, this line of clicking merge. But sometimes people review very quickly and merge things that they should not, Jonathan: Say it's not surely we're all extremely careful, and we read every line of code before things get merged. Surely. Oh, absolutely. All right. I think we've done a good job of covering the problem space and the tools that, that you guys have put together for this. If you look into your crystal ball and you think about what's coming in the next six months, we'll make it a little easier for you. I was gonna ask for the full year, but we know how much things can change in a year. Just the next six months of sort of this problem space, what what do you see coming? What are you watching as, as maybe the next trends? Francois: That's a good question. There, there's a number of things, as I mentioned that, that is happening that makes it harder for threat actors, much less obvious. But it doesn't mean that that we're, it- it's a done deal. Because ultimately CI/CD, I think I sh- personally I think we should all rename CI/CD RCE as a service, because it's really much easier to think of CI/CD as an execution environment running arbitrary code, right? And if you start to think about that- ... a- and design your environments like, But for the next six months yeah I don't know. I think the, the threat actors seem to the bad guys that have been very active in the past couple of months, they seem to have been a bit more quiet. I think they're feeling the heat. But if we remember what happened in the fall of 2025, I hope it's gonna be quieter. Jonathan: Yeah. Francois: But I don't know. Supply chain security as a whole there's so many angles. The build pipeline was a we- weak, weaker spot, I would say. Very easy attack surface to, to profile from the outside with Poutine or other tools. And the other angle, the maintainer side phishing, we're never gonna fix that limiting blast radius is something we can at least do. The, the, the package ecosystem like NPM and PyPi and others they've been doing quite a bit of work in that respect as well. So I, Jonathan: I will- Francois: I, I don't know ... Jonathan: I will throw in there that the idea of a, a malicious maintainer or a compromised maintainer, there, there are some tools out there that you can use. So we've got one repository that's set up to where I c- I don't think I can do a f- I'm the owner of it, and I don't think I can do a force push to it. And particularly if I do a pull request- ... I cannot approve and push my own pull request. There has to be a second set of eyes that go in there and gives it a review before it'll go in. So there are- Yes ... there are some tools that, I don't know that either of those are a magic bullet, but there's gonna be w- ways to at least to some extent mitigate that. And I think, if people are working with important code bases, that's definitely something they should take some time and investigate and see if they can get set up. If it's practical. Not every project has two dedicated maintainers, and I g- I get that. Francois: Absolutely. Jonathan: So Francois: that's Jonathan: that's a real challenge. Yep. E- as you look at at some of the attacks that have happened, do you get the idea that some of them are, like, big nation state actors, or do we think most of them are just essentially cyber criminals looking for a cash-in? Francois: There has been very few... when it comes to attribution, that's always you don't want to just, wave your attribution without a- as access to as many of forensics evidence. But yeah, I would say probably 98% of the cases that I've seen, it's clearly just people just having fun, ... Francois: Or with a intent of gaining some kind of, you know, like- A lot of them- ... money out of that ... a Jonathan: lot of them come down to stealing Bitcoin wallets, don't they? Francois: Yeah, absolutely. In many cases, if you're... And that's, that, that would go a bit closer to some nation state, which I will not name, but those interested in this topic, they know that there is one nation state threat actor that is very fond of crypto. And- Jonathan: the one that comes to my mind is not in the South and it's not in the West. Francois: Yes, it is. There's some compass if you were in a different... yes. And they, basically any open source project that is near, near or directly in the path of anything related to crypto infrastructure- Oh, yeah ... be it cryptocurrency wallet or some kind of backend- Oh, yeah ... it's dangerous. So- I Jonathan: mean, those guys just have a- ... you Francois: might be a collateral victim ... Jonathan: giant target painted on them. Yeah. There, there is... it is impossible for those guys to be too paranoid. Yeah. Francois: Yeah, absolutely. So basically, if you're gonna be using pretty much anything that some cryptocurrency infrastructure happens to use also you might unfortunately be a collateral victim of that, and it's kinda hard to know, right? If you go back to spring of 2025, C- Coinbase- ... was the actual intended victim for the TG Actions CI/CD compromise. So if you remember- ... there was TG Actions change files was a small GitHub Action component that many people were using. But the final victim was Coinbase. And yeah. But in that case, it was not, likely not a nation state threat actor, but still someone that was after Jonathan: cryptocurrency. Yeah. Yeah. That makes sense. So we talked a lot about the open source stuff. Tell us briefly about Boost Security. What do you guys do? Francois: Yeah. So we have a, ob- obviously like a commercial offering and we have the research angle that I've been talking about the different projects. So we have a application security posture management product that is in some cases like using the open source projects that we put out there and, put a layer that's more like for enterprise grade kind of policy and scaling of of all that. And yeah, w- we have an, a whole new set of offering that's a bit closer to when I mentioned Bagel, where it's gonna be looking for the laptop of a developer for enterprise users like they... We have a product specifically for that and it's quite on the mind of many security-minded people in, in, in medium and large organizations now- Absolutely ... because the laptop of developers has been effectively ignored in many cases, like by traditional EDR and things like that. And so that, that's a big angle that we're looking to address now. Jonathan: One of the reasons is because as a developer, it is such a pain to run almost any of the existing antivirus products. Like almost all of them are just terrible and will get in your day and ruin your... Get in your way and ruin your day. So most of us developers, me for one we're tech s- we're in this dangerous place where it's like, "Oh, we're tech savvy enough to keep ourselves out of trouble. I would... I know what I'm doing. I would never get my laptop compromised." Also followed with, the big company's antivirus project or, endpoint security pro- I know too much about it. I would never run that on my computer. And like- Yeah ... I get it because I'm right there too, but at the same time, that's a terrible and dangerous place to sit. Francois: Yeah, so I c- I can tease that we're gonna be coming out with something in the near future that people may... People interested in that area, they may want to look look at what w- we'll be announcing in the next couple of weeks. Jonathan: Yeah. I would, man, I would love for, I would love for there to be a really good endpoint security solution that was not insulting to run on my own machines. Maybe that's you guys. I would lo- I would love for that to be you guys. That would be so much fun. Parts of that open source? All of it open source? Francois: Part of it- Parts of it ... is definitely open source. The rest we'll see when the announcement, Sure ... comes, but Sure. Yeah, Jonathan: absolutely. I will make sure and watch out for that. That sounds that sounds really cool. Sounds interesting. I unfortunately, a lot of us, we get in situ- ourselves in situations where it's like because of whatever we're working on, we're required to have something running, and it's like, "Okay, fine. I guess I'll go install Malwarebytes. That's the one that I hate the least," or, something like that, and maybe not necessarily the right answer. Just because I hate it the least doesn't mean it's the best answer. Francois: Yeah the, the issue is that almost all of those EDRs or mal- antivirus, they were not designed to work on a developer laptop- where you write new code, and you compile it, and you run it five seconds after you, you wrote it. You modify a Python script, you execute. What's the difference between the one you just wrote five seconds ago and a malware that you downloaded where you Wget some Python script and then you execute it? Unless it's really need threading the needle, or you downloaded that from a third-party source, and then even then it... That happens all the time. That's what we do to build different things, right? It's really more about the behavior of, it's not normal for your SSH private key to be loaded in the memory of something that, it... There's a number of behavioral things that you can look at to, to at least either a- alert or block, right? Jonathan: Yeah, very interesting. Definitely looking forward to that. Is there anything that we didn't touch on that, that we should have? Francois: No I think that's pretty much that. Obviously, like Smoked Meat is a tool that's not for everyone. It's something that is more int- interesting for pen testers- It- ... red team operators and whatnot. But, Jonathan: I'm sure somewhere in the source code and/or on the website, there's a notice, "Do not run this. Do not break the law-" Yes with this tool. Do not run this on live code anywhere." Francois: For sure. But as a learning tool, like for even for people who think, "I'm no- never gonna be running that tool," we have a YouTube short YouTube video, three minutes- ... that people can see how it works and the type of damage that can be done if ex- exploitation of a CI/CD environment is to occur. And it can be used as just a, a, a learning moment for, for people to tell others that they should maybe be running poutine and look at auditing their CI environments. Jonathan: Yeah, absolutely. We do have a bit of a live audience today, and they've been going back and forth with the food puns, and Mashed Potato says, "It's not for vegetarians." I have to point out, it's the bagel. If you really- Yeah ... if you really want the vegan option, you've got the bagel. Francois: Technically poutine is not vegan, but it's vegetarian at least. There you Jonathan: go. I don't know. It sounds pretty good to me to put the smoked brisket on top of the fries and the gravy. That sounds really good. Just have it all together. All right. Thank you. Thank you so much for being here. I've gotta ask you a couple of questions before I let you go, and we asked them last time. I don't expect the answers to change, but we still, we gotta get them in. What, Francois, what is your personal favorite text editor and scripting language? Francois: Ah do people write code anymore, it's just like prompt code. Jonathan: I know. I know. Oh somebody one of these days is gonna tell me that "Oh yeah Claude Code is my preferred text editor." Yeah. I'm gonna have to resist the urge to strangle them through the screen. Francois: I was almost gonna say if you are to profile the number of times that I... The answer is VI. For me it's a VI guy, i've never installed Emacs on my box. But I do use Z- Zed as like instead of Visual Studio Code sometimes- Okay ... like for a lightweight more visual- ... like a Z- E-D. But yeah, VI guy. Jonathan: Yeah. And then you were gonna make the point I think that if you measured it by lines of code you write that something like Claude- Yeah might end up being the winner. Francois: Yeah, probably. Pr- well, Codex, but yeah. Jonathan: Okay. Yeah. That's... I would... To, to, to each their own, right? All of those, all of the different models have- Yeah ... their different wrinkles, and I think they're well-suited for different tasks, and we all get used to Francois: one. Especially for security tasks. I've had some... Yeah. Like the, Cla- Cla... I love Cla- Claude Code, but like for security professionals, it, it's- there's been some some some drama in the past two months or so. Jonathan: Sure. Francois: So we Jonathan: can see that. Although they do now have a some form you go fill out. One of my, one of my other maintainers did this, where it's like, "I promise I am a maintainer of this project. I promise I will use this responsibly. Please let me hack with it." And they will look at it and go, "Okay, fine," and give you m- more access. I don't know. There's... I'm sure there's still some guardrails, right? But they will give you a little bit more access. I- in my experience, Francois: that's not enough. I'm part of those, those those programs- Okay ... and even then the, the guardrails are a bit annoying for professionals, like security professionals. Jonathan: Sure. Francois: But- Jonathan: Ha- have you found have you found that running models locally lets you get around some of that? Is there a a case to be made for doing security research with local models- I- ... for that reason? Francois: I think so, but it's quite expensive. And quite... I think if to run any model that, that is even close to the, the frontier models, it's impractical for- Yeah. But I did, yeah, I did mess with some smaller models. They can be useful for a very specific a- scenarios, yes. Jonathan: Yeah. Yeah. Interesting. Very cool. Francois, thank you so much for being here. It's been an excellent conversation. The, the hour plus just absolutely flew by, and really had a lot of fun with it. Thank you, sir, for being here. Francois: Thank you. Jonathan: Yeah, I appreciate it. All right. That is Francois Proulx talking about Boost Security and their open source projects, Poutine, Bagel, and Smoked Meat. We've got a whole meal, Canada style, and that's a lot of fun. We do have some great shows coming up. Next week we're having Jonathan Polance of Ferris Systems, and then I know we've got quite a few that we have emailed and working on scheduling. But we do have some slots open, so if you have a project or you know of one that needs to be on the show, you can shoot us an email, [email protected], and we can get them scheduled up. We appreciate everybody that's here. Thank you for watching and listening, whether you get us live or on the download, and we will see you next week on Floss Weekly.
• 
  Episode 877 - RCE As a Service 29.07.2026 1t 7min

  This week Jonathan chats with Francois Proulx about SmokedMeat! That's the third in a trio of Open Source security tools from Boost Security, and this one is the red team tool to demonstrate vulnerabilities. Why are Continuous Integration vulnerabilities such a persistent problem, and what's on the horizon that may help? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 876 transcript 22.07.2026

  FLOSS-876 Jonathan: Hey folks, this week I talk with Michael Meeks about Collabora. They do a lot of things, but we're talking specifically about their office suite and how they pay the bills doing an open source project. This is Floss Weekly, episode 876, recorded Tuesday, July the 21st. There is no money fairy Hey, folks. It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and my voice is just a little extra raspy, a little extra bassy today, and that is because I've been at Open Sauce this past weekend. And in the infinite wisdom of the people putting the show together, they put our booth back to back with a guitar amp booth. So I've been speaking loudly to be heard over the sound of sick guitar riffs for about three days now, and my voice definitely took the hit. But that was a lot of fun. I'm back now, back in the saddle. Didn't miss any Floss Weeklies this time around, and we have got something really neat today. We're talking with Michael Meeks of Collabora, and one of our, one of our live listeners before we started the recording said, "Are we talking virtual reality, or are we talking office suites?" I think the answer is office suites, but apparently VR is part of the conversation. We'll get some information about that in here in just a moment. So our guest today, Michael Meeks he is CEO of Collabora, and he is, has served as the director of the Document Foundation. He's done all sorts of stuff over the years, including worked at Novell/SUSE, then part of doing lots and lots of Linux stuff over the years, and a, a whole lot more. Let's go ahead and bring him on. Michael, welcome absolutely to the show. It's good to have you here. Michael: Thanks so much, Jonathan. Pleasure to be here. Jonathan: So w- are we talking virtual reality, or are we talking office suites? Michael: That's a really good question. So Collabora, of course, is two things really. So it's a, a consultancy doing Linux enablement, open source stuff across the stacks- in multimedia, graphics all sorts of kernel enablement hardware stuff to get open source in at the bottom of the stack. So it's the obvious choice. There's always someone there that can help you get your hardware enabled, and so open source comes up from the bottom. And then of course, Collabora is also Collabora Office. It's always coming from the top of the software stack with these very, very leaf node- ... with all these dependencies. And so yeah there's two companies there. But we love to share the same brand and, work, work together. We have a mission, which is to make open source rock, and so we do that. Gotcha. And the hope is that in the end, the bottom and the top guys meet in the middle and shake hands and then everything is open source. That's the long-term plan, basically. But yeah- ... we're getting Jonathan: there. It's when they were digging the ch- the Channel, and they started at both ends, and finally they broke through and, shook hands through the rocks. Michael: Absolutely. Absolutely. H- So yeah, a, a breakthrough. Jonathan: How did it come about that there are these basically two separate companies that are under the same banner? Michael: Yeah, so I did a lot of work as you mentioned at SUSE Novell, as distinguished engineer there. And at some point, SUSE got really good management. And the good management realized that infrastructure software services and office productivity are pro- probably two separate things. All of these people are suffering in the Linux kernel at the bottom, toiling away. And, Linus says, "The interesting stuff's happening in user space. Up there. Go and do those things." And so there we were, writing Android apps and so on and people were a bit, aggrieved. And so fair enough, totally fair. We... They spun us out, they did it in a really good way Niels Brinkmann, Ral- Ralf Flakser. And they even gave us a, a small support contract for the first year or so to support their usage of LibreOffice, at the time. And yeah, just brilliant. So we got spun out, and that just makes decision-making so much easier. And helped me learn a whole load of, things about open source and business that seemed so simple back when I was employed and someone else was doing the selling and so on. And now, yeah, make... I think I was extremely naive, let's say. Yeah. To put it- I- Put it one way or another. I Jonathan: understand. Yeah. You got a very abrupt crash course- ... in herding cats, among Michael: other things. Absolutely. But of course, Collabora was a key part of that. So- ... Michael: I couldn't do it by myself. We took six other people from SUSE, and that's always an exciting ask, isn't it? "Please come and join my company with no customers, no... and no money and whatever." And so I think, Collabora played a really important role there, the parent company, in, in seeding that and providing some of that business experience. And my, my key partners there, Philippe Calaff at the time, Rob McQueen, really helpful- ... in terms of, just advising and providing infrastructure, finance and personnel and contractual basis and so on. So it was really useful. Jonathan: Yeah I had no idea until this very moment that Collabora was spun out of openSUSE. That is absolutely- Yeah, the office piece was- ... n- news to me ... the office piece. Yeah, interesting. Michael: Yeah. Jonathan: The... there's some news that everybody has been talking about this week, and that is that Valve and Collabora have released the official Arch Linux ARM64 port for the Steam Frame Michael: Awesome. Jonathan: And that is- Awesome ... that is something like what you're generally aware of that because you're the CEO- ... but otherwise you're just hands off. "Oh yeah, the guys are doing that. I don't have anything to do with it." Michael: So to be clear, I'm the CEO of the subsidiary that does office pieces. Philippe Calaff is the CEO of the, of the- Okay ... above. And it's a really, it's a great company. I think there's a lot of subsidiarity in those groups. Lots of groups that working, pushing out in their own direction. So yeah, I think sure this group is probably doing the valve bits there. But meanwhile, other people are reverse engineering GPU drivers and maintaining Mizar and improving. We- we're pushing in so many directions. Jonathan: Yeah, Michael: absolutely. So yeah, it's always great to see these things. Jonathan: Okay. Michael: We do so much good stuff, it's nice to be able to talk about it. Jonathan: Absolutely. So I feel like we've placed the landscape now and we kinda know the things out on the periphery that we're not gonna talk so much about. Let's talk about what Collabora, the office suite in Collabora Online is. And how did this come about? So is Collabora a fork of... the granddaddy of all of these is StarOffice, right? Michael: Yes. Ultimately yes. So if we go back to 2000 or before I was working with Sun Microsoft or Zimian. And Zimian, met with Sun and Marco Börries, who was the, the kid who created OpenOffice or StarOffice in his garage, and then built a big business around that in Hamburg. They kinda met together and decided how, to open source this thing and how best to do that. And so I was involved with that, before even it was open source, trying to think about component models and plugging things together. And if you remember component models, but the punchline is you don't need them. Open source is open, and so you don't need to have all of these special contracts. Just, mash the software together, improve it, and work, work together. But either way, back in the day we loved that. And so then OpenOffice, of course, lived for about a decade before LibreOffice came in. And of course, Oracle, the economics for the Oracle division there weren't, I think they say accretive to gross margin or something like that. Meaning you've gotta make an insane amount of profit, otherwise you're not welcome. And that's a great shame. And a lot of those Oracle people were being let go, redeployed in other places And so then LibreOffice came out of that. Jonathan: And Michael: that was great. It was funded by, I guess SUSE or Red Hat with help from Google. And, the Linux desktop was hot back in 2000. There, there were lots of people there, and we had big teams working on this. And yeah, and then of course, over time... So we founded LibreOffice, and over time some of that desktop excitement died away, let's say, or at least reduced in magnitude. And you then end up in a rather different world economically in, in terms of sustainability. And then of course, when SUSE effectively pulled the plug on this and said, we're stopping," that was, would've been devastating to The Document Foundation, LibreOffice. So we yeah, so hence- Gotcha Mortgaged the house, start the company, rescue as much as we can, and then grow up. And now I guess we're about 70, 70 people in the productivity office part of Collabora now. Jonathan: Yeah. There's yet another Office suite out there ONLYOFFICE. Michael: Oh, Jonathan: yeah. And I remember that there was some drama between ONLYOFFICE and Collabora back six months ago, nine months ago, something like that. And my, my comment- ... at the time was the whole story was just bizarre. What and obviously we're not here to throw stones at ONLYOFFICE, but I'm curious from your perspective, what's the what's the overview of all of that? Michael: Yeah. So at least from my perspective, we didn't have a huge amount of drama ourselves with ONLYOFFICE, but I think Nextcloud then took the ONLYOFFICE code base and then created a fork of that called, it's called EuroOffice. And, ... there's a lot of, There's a lot of interaction there that's quite, quite interesting. I, I'm- Jonathan: Entertaining for outsiders. Michael: Yeah, absolutely. There's a popcorn thing going on there. Yes. W- but, from our side, in terms of positive things we can say about, Collabora Office, and I'll just say a whole load of positive things. We have zero Russian comments in our source code. We have clean- ... licensing provenance. We know what our license is and- Absolutely ... have statements from everyone that, that contributed to it. Jonathan: That's Michael: such a huge deal. Jonathan: And we build around our... So- We've had a... I'm sorry to bre- to cut you off, but we've- Michael: No, go Jonathan: for it, yeah we've talked to several projects over the years, and it's like what license are you?" And it's complicated." One project in particular, portions of it were written by US government employees, therefore those portions are in the public domain. And portions of it were written by contractors, and no license was ever assigned to it, which technically means that those are, all rights reserved. And w- and we had the guys on the show, and they're like, "It's open source. We want everybody to use it." And we're like, "How does this work?" And they were "Yeah, it's hard." Oh my good- Yeah. So that is a, that is actually a really big deal that you have clean- Definitely ... provenance on your licensing. Michael: Yeah, completely. And I helped set up the structures there at the Document Foundation, and people would send license statements. Ultimately I think we use the Mozilla Public License, so the MPL 2.0- ... which has a number of advantages and disadvantages. So it's a weak-ish copyleft, right? So it's per module copyleft, and so we can be pretty sure that if you're contributing stuff or reusing that code, then you're accepting the, the MPL 2.0. But beyond that it allows you to ship binaries under whacked out licenses. As long as you don't harm people's rights in the so- source code you can sh- use IBM's proprietary license and ship the binaries under that. So that's really nice. We got a clear source binary split there for the for the code, and it works really quite well, I think. And we share that with LibreOffice and, ... yeah. Yeah. So there was a lot of license collecting and yeah, due diligence and so on there that's really very good, both at LibreOffice and Collabora Office, I think. Jonathan: So Collabora Office, and I'm trying to remember the exact details of this. Obviously your, even your T-shirt that you're wearing says Collabora Online. And I- It is ... I'm trying to remember which direction this went. You guys turned the office suite into a web-based version, and that was your- That's right was that your primary offering for a long time? Michael: That's our primary offering. Yeah. Okay. Has been our primary offering for 10 years now. So we're just about to celebrating the 10th anniversary, and we actually got our release on Thursday of the 26, so four enterprise release of that. So there's a whole load of excitement around that. Yeah. And the team are crazily, fixing and improving and documenting and writing press releases and so on. And I think- Yeah, absolutely ... you've seen some of those. So yeah, so we're super excited about that. And, but of course- We've done a lot of work on the user experience. It's much nicer to use, much more familiar. Lots of research and, study there to, to improve it. And so we thought why don't we bring that to the desktop as well?" And actually, that had already been there, so if you'd had an Android version, you have this responsive UI and it looks like a mobile phone, one-hand touch interface. Great. But were you to plug a big screen in, suddenly it would turn into the, the, the real Office Suite, full fat. Absolutely. And so that's why you could run it on a Chrome OS and you would get real document editing there. And so that was pretty cool. And so we sort, took that and then just brought it to Windows, Mac, and Linux natively, and that's Collabora Office. So yeah. Jonathan: What, what was the decision like back in the day to say, "We're gonna make a web version of this"? And what was the engineering challenge there? Michael: Yeah, so well, the engineering challenge is making it work in the browser is, was quite exciting at the time. Sure. There were a whole load of choices we had to make 10, 10 plus years ago. Jonathan: It was before, WebAssembly came along, made a lot of things easier. Michael: Oh, totally. Totally. Totally. So we still don't use WebAssembly, so still we run a lot of the code on the server side. And there are a lot of reasons for that. I think from a security perspective- ... you wanna enforce policy on the server side. Jonathan: Absolutely. Michael: So if you want a policy like you can't download it, or you can't print it, if the first thing you do is download the whole file to the client- ... and then have your office suite there, You have a problem, right? Yes. From a security perspective. Yes, Jonathan: absolutely. Michael: And of course, if the first thing you do is download eight million lines of JavaScript and then JIT them on your mobile phone, you also have a problem that not only did you lose the document it just, it's ruinously expensive in terms of memory, performance, startup- Absolutely ... user experience and so on. So we try and keep that, that fast rendering in the i- in the server there, and bring it as tiles to the client, and that's really good. And we're starting... We- we in- increasingly moving more and more stuff to the client. So of course, your cursor blinks and there's no network traffic, this is rendered- ... i- in the client. Your selections are overlays rendered on the client. Your objects as you move them have previews. Your dialogues are reconstr- reconstructed on the client using native Java, JavaScript, HTML widgets for accessibility reasons if nothing else, so that you can move through them and screen read and get that rich rich accessibility experience. And now we're gonna be moving what we call turbo layout to the client as well. So there's a whole load of work to move away from what we thought would be super easy, pixel rendering to vector rendering in the in the browser. And actually, it works extremely well. But we think we can do more with vector rendering. We can just render once independent of view zoom, and then have everyone use that. So there's some, there's efficiency wins. But then we can start to do what we call turbo layout. So if you look at how Collabora Office lays out text text layout seems simple. You just nail the next glyph along. What could be wrong, right? But it turns out there's squishing text and justification and hyphenation and balancing columns and footnotes and re- laying out pages, and it's very easy to end up in loops. So there's no space there's... We've got a footnote here. We've fitted the text in, but we can't fit the footnote in as well. So let's get rid of the footnote. Now we can fit more text in. But, and you can end up in these iterative situations where, yeah you just... Anyway, you fail. The grass- So we have- The grass Jonathan: is greener perpetually- ... on the other side of that decision. Michael: Yeah, absolutely. That, that sort of thing. So yeah, you just basically can't win, and at some point you have to give up and try and do your best you can. One of the things that means is that's can be quite a slow process in a large document, so we have a turbo layout where you type and it nails it on the screen as quickly as possible. Like i- in, in the most plausible way, and then it will slightly update it. In many cases, you never notice that or it doesn't really need updating. But we're starting now to look at bringing that into the browser itself so the keystroke latency is exactly nothing. It, a frame, maybe two frames. Then again, at 60 hertz, within two frames you can get, some, some pretty impressive round trik- trips. Frankfurt to, to London is easily, is within one or two frame delays anyway. So yeah, you you don't really have a big problem there. Jonathan: Yeah, interesting. What was the what was the impetus behind back in the day saying "We, we need a web version of this"? Michael: I think collaboration is the key thing. So being able to collaboratively ed- edit documents together is just so useful. And I guess, so not only collaboration, but also deploying this. Ease of deployment. So one of the things we've wrestled with for years is document standards, and, we, we do... A huge amount of our work is making Microsoft documents import and export and round trip beautifully. And actually, we've done tons of work in 2604 for that just to make that better. But turns out URLs are better standardized than Microsoft documents. And the spec you can print out and it doesn't break your back if you try and lift it, you don't need a, a wheelie thing. And so sending a URL to a friend and going, "Hey, what do you think of my document?" A- and then being able to live co-edit that, comment on it, and interact together is just so much richer and so much easier from a standardization perspective. And that then allows us to use open document format and keep your actual raw data in a really, future-proof consensus industry format rather than a, a documented dump of the Microsoft internals. So that's a good thing too. Jonathan: Yeah. So one of the, one of the things that we've talked about, and we've talked with for example, Simon Phipps about this, he's done a lot of work with this over the years- Sure ... is trying to convince particularly European governments not to mandate the use of Microsoft Office document format types. And it seems like some of that is finally taken in the form of what they g- what they're now calling digital sovereignty. Michael: Yeah. Jonathan: And that's obviously, it's something- ... that you guys have to be, like, tracking along with. And I have several questions about this, but first just what... W- where are you in this process and what do you see going on? Michael: Yeah. So I think we're at the, really at the center of the, the drive for digital sovereignty. I'm periodically more or less encouraged about where the open source world is going. At the moment, I'm super encouraged. I don't know. Some of the largest things in the news. W- when you see yourself being talked about in The Economist or whatever, it ma- it's really helpful. So for example- It's nice. It feels pretty good, yeah ... the International Criminal Court, we're helping these guys with, with Collabora to move away from some of their things that are happening. Geopolitical disaster areas- ... we, let's not go too deeply into them, but geo- geopolitical risk is back with a vengeance. And if you're a government, you've gotta continue governing. You've gotta serve your citizens e- even in, the, the h- these high-risk scenarios. And I think the balance of risk is now turning. So previously it was no one got fired for buying, IBM or whatever, right? And so people choose the low-risk option. But I think increasingly- There is a lot of correlated risk there. And so actually, the low-risk option is to make sure you have a plan B, a, a minimum of plan B, of some people so you can reset your hardware, reboot, and you're now able to collaborate on a, say, a Linux base with a free software office productivity suite, and support for that. Jonathan: Are we, Michael: are we to- and better ... are we Jonathan: to the point that we could say that no one gets fired for open source? Michael: I hope so. Yeah. Like- ... it's the obvious choice, right? And s- so I think the, the balance of risk is really moving. And actually, there's some really large groups. So Caisse des Dépôts of Fra- France is a big shareholder. The French government shares. They're saying, "Just give them the effing purchase order. Get... Do it. Let's not talk about it. We're ready." Yeah. The software's there. It's just a matter of political will. And I think if you look at, say, Schleswig-Holstein in, in, in northern Germany there so Dirk is the the leader of this migration You start to see what actually we all knew all along. There's nothing wrong with our software. The software is great. What's missing is the leadership to say we go this way, and maybe if you have a problem, we fix it." That's fine, but- Sure ... tell me about the things and we, with a goal for fixing them, not, as a way of making excuses to do not much. And and so it's just fantastic to see, and I think it's the CDU government there, so coming from the right, are saying, "This is the way to go." And I've been in a lot of more left, let's say, political drives for open source, which we talk about solidarity and working together, and that's fantastic. I love all of those things. But it's refreshing to hear someone quoting Schumpeter and Hayek and saying open source is the way to go on the other spectrum, and once you see... if we can get the whole political spectrum to realize that this is where we need to go then that's just deeply encouraging to me about that alone. The real Jonathan: danger, and I speak to this from a very American political viewpoint. But the real- Sure ... danger is that the left and the right realize that they agree on it b- ... and lose their minds and immediately go no, we have to differentiate ourselves from them." It's like, no. Oh, Michael: dear. So yeah, it's just really encouraging to see. I think 40,000 users in Schleswig-Holstein. Tens of thousands of them already migrated. The Austrian Armed Forces, a conservative type of institution. 16,000 users off Microsoft Office, and again, supported by Collabora and backed by, sponsoring development work there and buying licenses as well, so- ... one of the things from an economic perspective is to get that subscription revenue that effectively maintains and funds and improves the software for everyone. Jonathan: Wait. ' Michael: Cause we, we, we- Jonathan: You make money from open source? I didn't think that- ... was possible. Michael: Yeah. We like to say in our community, there is no money fairy. You know- ... we don't have some insanely wealthy banker or, backer. Like to be fair, we do have some insanely wealthy people who who out of the goodness of their hearts, invest in features and do things, but broadly, there's no one bankrolling it. It's organic. There's no VCs. It's a real company, Collabora and yeah we... I think my grandad said, "You pay as you go, and if you can't pay, you don't go." It's pretty simple. In a way. And so that's basically where we're at. So we try and, yeah, do this in a sustainable way for the long term. We, this is a multi-round battle we've been in for 25, 30 years. Yeah. And Jonathan: yeah. We can get back to digital sovereignty in a second, but I think it's- Sure ... just super interesting. What i- what is someone buying when they buy a Collabora l- online license? Michael: Sure. A lot. So- So we talked about security. I, there's a whole load more to say on what we do there. But- ... suffice to say that we have, some of the absolute experts in the industry looking at modeling and receiving threat things and generating fixes and improvements, often that are only in one level of our multilayered onion, but are still, a problem. So we ship updates, so we have signed updates and maintenance. We support the product for three, three-plus years. You can choose which of their annual releases you b- base off, and then you can deploy that with complete confidence that we'll be there and backing it and supporting it. And one of the most fun things I like is product management input. You can come into our conference call days, and you can talk to the engineers doing it. You can give your feedback. We rank that and show you what we're doing and you can get a real feel for how that's going. We also do consultancy, and on our product, we would expect you to buy the product first. So if you wanna drive a particular feature, then, that's fine. But you get a subscription first, and then, go wild and we'll work crazily together. What else? Bug filing. If you have a bug or an issue, a- any kind of thing, there's an, Eat as much as you can, support entitlement there. So if you have a problem, regression of any kind we'll fix that and then turn it around get binaries for you, and so on. So it's a whole load of different pieces intersecting there, a bit of a, a lasagna of of value for- ... for people. Jonathan: So if someone has a license and they run into a problem, can they actually call a phone number and talk to a real human about it? Michael: So I don't know that we like phone numbers as much, because we're quite distributed. But they can certainly email someone, and the developer will read that for sure and get back to them remarkably quickly. And- in some Jonathan: ways email is better, but Michael: Yeah, with logs and things, if you- ... if you've got a, if you've got a complicated problem. But, then we often get on a video conference like this. If it's a difficult problem, we'll just, video conference someone, they'll talk to you. Yeah, the world's expert in, their problem. Is it, I don't know, pivot table calculated field support or something? Or is it a- Yeah ... a, a sockets weirdness with some SSL variants, or TLS cock-up? And then we can help debug and, solve installation problems set-up problems, and occasionally even level three code problems too. Jonathan: Some of my, my... So I've done IT, one of my other hats that I wear, I've done IT support for years and years. Some of my most depressing moments is to find a bug on a customer's computer, go searching for the bug online, and find a thread on the Microsoft support forums where, th- it's one person seven years ago reported it. 50 other people are like, me too." And, you have a, "I'm a Microsoft certified support technician, have you tried turning it off and back on again?" And that's about all the help that you ever get, and it's like it's 2026 now, and this dang problem still exists. Thanks, Microsoft. Michael: Yeah. Yeah. Jonathan: So- Michael: It's really sad, isn't it? I think one of the things, and there's some quite nice an- a- analysis of that, but hopefully we build a culture in open source of excellence. That's the hope. Of getting down to the root causes of problems and fixing not only those, but the whole class of problems and all of the others that could be related and doing a good job. And of course that's time and and culture and is expensive, but in the end, my hope is it yields a better product. In their defense, Microsoft have so, such large deployments of paying customers that actually hardware failure does start to become a real thing. Jonathan: Yes. Michael: Yeah you want a lot of reports before you, You go, "Yeah, that, that could just be, cosmic rays." And I, actually, when you look at our, some of our analytics, there are crash reports there that you just... it's just not possible. Jonathan: It's go- it's gotta be bad RAM in some cases. Michael: Yeah. Either bad RAM or- Jonathan: And Michael: I- ... basic error in electron migration. We worked the CPU too hard, and it, Yeah. Jonathan: So in, in... I f- I could think of a few cases where I thought it was a bug in somebody's product, and I'm like, "You need to talk to..." One of them in, in fact, was Intuit with QuickBooks, and they would do one thing in particular and it would crash it. I'm like, "That has gotta be an Intuit bug. You need to talk to them." And the customer did it, called Intuit up, and it's like, "You need to replace your RAM." And they were right. That is exactly what it was. That's really interesting. And I was so s- I was so surprised, but yeah, it's, it is a real thing. Hardware failures sometimes look like software bugs- Yeah surprisingly. Michael: Yeah. We used to... I used to design hardware systems actually so we did a whole other thing. So one of the fu- fun things is DRAM refresh, and one of my colleagues created this little bit of embedded microprocessing, or like a, a bespoke microprocessor, an FPGA, to do a piece of software. And it worked fine until he increased the length of the program, and now it was two lines of RAM. And previously, just executing the program refreshed the RAM. Jonathan: And Michael: then subsequently, when it got longer, it stopped. And so you're looking at, he just forgot some RAS CAS cycles to to refresh his RAM all as well, but yeah, it just a little- That's- Jonathan: it's a little bit amusing. Yeah, that's hilarious. All right back to the digital sovereignty thing this, this- ... we have a question from the chat room that really ties in well with it, and it is "Tell us a little bit about the headache it is to support Microsoft's formats." Specifically, here's the question, "Does Microsoft intentionally twiddle things just to ruin your day?" Michael: Look I was involved in OpenXML standardization- Jonathan: And has your lawyer told you're not allowed to answer this question? Michael: So no, my lawyer hasn't told me that. But he probably should tell me to say nice things. But actually I'm gonna say nice things anyway out of the goodness of my heart. So I was involved in the OpenXML standardization- ... of the Microsoft formats back in the day, and it would be remiss of me... we complain about the size of the spec, but some of that is awful. We asked for more and more information, and they provided it. They did a straight-up job. And, whilst there are whole fields that aren't specified, like layout the layout of your document is based on an integer, which is the version of Word, right? There's one number in there. Change that, document looks different. Everything changes, yeah. Quite a lot changes. And the more complex the document, the more will change. And so yeah, there's a whole load of stuff there. But that's fair enough, and that's just software engineering, they wanna be able to improve their layout. And why not? And I must say that the people I dealt with there, Brian Jones, Tristan Sean Villaron, some, were great guys. Really friendly, really helpful. They did a, a great job of standardizing that. There are some things that are unfortunate. I think if I, if we look back in the day, I think the Visio file format, the binary file format, had an undocumented checksum of the whole file. And unless you could generate the right checksum, it would go, "This file's invalid," and that's fine. You can see there's always a reason for these things, right? Oh, Jonathan: Yeah. Michael: But I think this is gone now. I think the, the joy of the OpenXML or the XML formats, of course, then, is that they're much, much more open. And so obviously Open Document Format pushed them in that direction, but it's a good direction to go in. They've gone there. The, the slight sadness of the OpenXML story was that- we really were dumped all of this, this XML file format could not be changed. We could change the documentation about it, and we did a great job of that, so you could better document what was going on. But you couldn't go, "Wait a minute, this XML tag sucks a bit. Why don't we move that into an attribute? Ooh, there's a tree structure here that's represented with integer lists for parent nodes. What's going on?" "And we c- can we improve that and just, hey, we've got an XML, it's a tree structure, why don't we use that?" So none of that was possible. But even lots more was out there. So I, I think Microsoft is... plays quite carefully in many ways in the market and that's, appreciated and yeah I I have my concerns. There's a Competition Markets Authority investigation in the UK that we're we're involved with. But I think m- much of that's at the business level rather than the technical level. If that makes sense. Jonathan: Are we at the point now to where you can open up a Microsoft document in Collabora and it basically just works and looks right? Michael: Sure. Completely. Yeah. I do that all the time. Jonathan: Completely? I, I- That's quite a statement, actually. Michael: You say completely. There's always- Yeah. There's never 100% in what we're doing, right? I think encouragingly, sometimes we find bugs on the other side or we're like, "W- what is... we're broken. Let's fi- fix our problem." And they're like no, don't fix that. That's a bug in our side." We need to fix it over here rather than you learning to adapt to this, this bug in our side." Software has problems, and that's why you need support, and that's worth getting. But yeah. It's really extremely good. I routinely redline DOCX contracts with lawyers and sign them, a- as part of my day-to-day job. Sure. There aren't problems that I see. We do... We run the whole company on it, so our spreadsheets, our finance planning. We still get problems with spreadsheets, but it's fat finger errors and I got the wrong... it's hu- human error. One in 20 cells is supposed to contain an error when you analyze spreadsheets. So that's the problem. ... But yeah, c- compared to that we're acing it, Jonathan: yeah. Yeah. Michael: Yeah. Jonathan: So one of the, one of the other things that has been happening over in Microsoft world for a while is they are retiring SharePoint, or at least the, the online or the, the local premise version of SharePoint, right? Michael: Yeah, absolutely. Absolutely. So those people who cared most about their privacy or the, the confidentiality of their documents would typically run a SharePoint server on premise. And they'd embed in that their Office online viewer. So like Office 365's Microsoft Office, but actually on-premise. And so you could then collaborate around SharePoint behind your layers of VPNs and, anti-tank ditches or whatever. You could really be sure that was staying internal. And that's then being obsoleted by December 31st, that's going. And the good news there is, of course, that... there's two, two bits of good news. One, we have a solution here. And it's a drop-in solution. We use this WAPI-like protocol that's compatible with Microsoft's protocol. And so you can spin up your Kubernetes thing and just drop it in, and that's great. And then you can co-edit and view documents as before. Jonathan: Very cool. Michael: Totally on-premise. The second thing is that it's actually better. So one of... There are a whole load of things you can do in Collabora Online that you just can't do in that Office online editor. Say, for example, putting a chart in a word processing document Can't be done in Microsoft Office online Really? It's not there. It seems incredible. Check it out afterwards. Go and try and insert a chart, right? Now, maybe you can get a picture of a chart or maybe a bitmap off your clipboard if you- if you're particularly lucky. But actually getting a full fat, full feature chart, axes, numbers, data table, whatever, just, it's just not there. Jonathan: Oh. Michael: Which is quite extraordinary 'cause it hasn't been there for five, since its inception. It's not like it's improving. And so this leads people to suspect that the online Office products are kept deliberately low fat, low feature because they're an advert for a Windows Office fat client. Ultimately, that's their role, is to basically put something in this space- ... so that- Jonathan: I have noted, I, I have had customers over the years that use macOS, and they've always noted to me "Microsoft Office on Mac is just terrible." And I just give them this wry smile, and it's Microsoft isn't, th- they're not really a, th- they're probably not too interested in making it a really good experience on macOS, now are they? Similar thing. Michael: Yeah. Yeah. Jonathan: The in- the incentives just don't line up there. Michael: Business incentives are real, obviously. Yeah. Definitely. I think they do they try and do quite a good job o- on Mac. It's, compared with the browser version, the Mac thing is a, a, a thing of beauty and wonder. Jonathan: Sure. Michael: But obviously the real stuff is the, the Windows mainly fat client. And yeah, so that's- ... really quite interesting as a thing. Jonathan: Yeah. Michael: And of course it's different. So Collabora Office then is essentially feature parity with browser and desktop. It's the same thing. And so we're sharing code. It's a shared code base and building up. Jonathan: So does the Collabora Suite have an email client? Michael: No. Okay. One of the things that we try and avoid is boiling the ocean- ... as they call it, and Office is already quite a challenging space to get right. And we actually partner. So if you look at something like OpenDesk, which is the German government's s- digital sovereignty or- organization, or was it Zendesk? It's their product for digital sovereignty. They use a thing called Open Exchange, which is, a very mature product. Actually- ... was also spun out of SUSE in Poland i- in- in the past, I think Rafael Laguna and various others. And that's a really large, successful product that's used and white labeled in lots of hosting providers and so on. And they run Dovecot. They create a thing called Dovecot Pro, maintain that. So email is sorted there. Of course, lots of other... we've got a th- a partner called eGroupware doing that. Obviously, we partner with HCL Software, who have Domino. There, there's a whole load of different... We integrate with vast numbers of partners with different product features. And again, why would we want to compete with that? It's a difficult space. I was there. I helped write the Evolution client under Gnome- ... way back in the day. And it all looks easy until you, you find the iCal quirks and the whatever. Everyone starts. Jonathan: Yeah. One of the- we just- ... one of the real interesting stories over the last few years that I've looked at and seen and thought about is apparently there for a while, Mozilla was thinking about letting go of Thunderbird. And- Sure ... one of the, one of the suggestions was the, the Document Foundation would take it. Michael: Definitely. Yeah, Jonathan: yeah. And then Thunderbird would become part of LibreOffice. And that, that did not come about. And there's just this part of me that thinks maybe Thunderbird would be great as part of Collabora. Michael: So not for lack of trying. We really encouraged those guys to come to the Document Foundation. I think in retrospect, where you see where things have evolved there, I think it was it was a good choice not to come. Although- ... you can never s- see what, what could have happened. No one's ever told what would have happened. Of Jonathan: course. Michael: But, they seem to be doing really well, the Thunderbird guys, and- I'm pleased. I actually, I switched from Evolution to Thunderbird at that point, so I'd actually have some clue of what I was talking about, so I'm a Pine Evolution Thunderbird person. There you go. Who's evolved through that- Jonathan: There you Michael: go ... that axis. And, Yeah. I, I- I'm a secret Jonathan: I've, likewise, I've been thrilled to see that Thunderbird has really found their feet again at Mozilla, and they've- ... they've put that back together and it's becoming a more healthy project again. Michael: Yeah, loving it But yeah, that's we partner with people. So one of the things- Right I really wanna talk about is just, that we love to go to market not by ourselves, but with other people. And I guess that's the open source way at some point. Sure the, these, these smaller pieces you put together. So we need, file sync and share probably. We need someone to do storage for Collabra Online. And we need someone to do authentication, because you would think storing files is easy, and you would think aut- authentication is easy, and then you end up with a multilayer active directory SAML authentication system, blah, blah, blah. Yeah. And you have this object store and a metadata store and, scaling and making that thing work nicely. And so we love to work with, say Wire, that bought Piedra recently. They have a great product there. And Nextcloud, ownCloud, OpenCloud, this whole cluster of things called cloud that came out of that sort of trifurcation of companies there. Yeah. Jonathan: It was unfortunate to see Michael: And- Jonathan: I'm- Yeah ... i, so that was actually something I, that was in my mind to ask you about, is, the, the integration with the a- asterisk cloud providers, and whether Collabra has almost a competitor to that, or whether it makes more sense to run underneath one of those. Michael: It always makes sense to run underneath one of those. So our partners are not best thrilled if we compete with them, for a start, so that's basically... I mean- Sure ... some, some partners haven't got the memo. But generally I, from my perspective, I really don't wanna be competing with our partners. Now, some of them, if we're partnering with a chart rendering provider, then I think when you go into that it's pretty clear that they're gonna do charts, we're gonna do charts. There'll be some evolution of the product. But I think when there's a very clear functional difference, I think it's very helpful to, to, just have your bit, and I think that's a good faith way of partnering people, and we like to honor that. And build confidence in our partners that we're not coming after their business, that their customers stay their customers. We're not gonna go around them, so it's not just technology. It's also just business practice of getting those accounts and building a business for other people. So one of the things we love to do is give margin away to our partners. It sounds crazy but we really pay a, a high margin for the local guy who speaks the language, who owns the customer, hel- helps them, holds their hand, installs it, because we want their business to succeed as well. We really wanna grow a, a flourishing ecosystem of individuals and, mom and pop shops through to giant, multi-billion multinationals working with us to push open source- ... most effectively and get it out Jonathan: there. Yeah. So when someone goes and says I want a Collabra Online install," like how difficult is that to set up? Is this... I- is it, as simple as running Docker, or- Sure ... is there a- Absolutely ... a, Michael: Yeah ... Jonathan: five-hour course you have to go through to be able to get it working? Michael: No, that, no, that, that's it's that easy. And so what typically people do is that they get stuck in with code. So we have a, all of our software is 100% open source code. But we have this thing called Collabora Online Developer Edition, which has just the latest and greatest stuff in it. So if you're integrating, people typically play with that to start with, and yeah, it's just in the Docker repository. You Docker, up and there you are. They, you have that. And if you're lucky, you have a Helm chart for a bigger collection of things that does the storage and so on. And yeah, it's, it... I've heard people say, "It's easy, it just works," I think that's nice. And if you look at our dependencies, some of our competitors require a Postgres database, an NFS file share, or, a Redis, a RabbitMQ message bus, and then they run on top of that. We have none of that. We're basically a, a single Docker image, and yeah, you can s- scale that out sideways with a, a load balancer. So it's pretty easy to do. Jonathan: Inside that Docker image, what does provide things like 'cause obviously you still have the need for some of those services. There's probably a database down there somewhere. There's a file storage down there somewhere. Michael: Nope. Nope. So how- So basically what we talk to is we talk to our storage and our authentication provider. So we we get a URL. So there's some configuration. Obviously there's, there's some YAML somewhere that's gonna say, "You are allowed to talk to these servers and not these other servers." But ultimately we're gonna get a Whoopi URL coming in saying, "Hey, this file is over there on that server." We'll match that against the servers we're allowed, and if we like it, we'll get the file. We'll, we'll- fire up a actually container. So there's a, this highly secure container system there. And in the container there's no shell. There's actually only just some fonts. There used to be time zone information, but pretty much nothing, and your document. So if you break out of that containment in some way, and we're talking 8 million lines of C++, you end up holding only your document. "Yeah, whoo. I escaped. I managed to run some native code." I get only my document, and if I execute any kind of system call that we don't like the look of, we set comp BPF, so any time you try and P trace or kill or anything, you're just terminated straight out. There's no, you stop executing. So that model is really nice. And so then inside that jail when we wanna save, we put that back to the file file sync and share with that, that storage, and then we tear the jail down and that's pretty much it. So we're essentially stateless. Jonathan: Okay. Michael: What is the- Except for the state while you're editing, but- Jonathan: Sure. Sure. What are the file storage backends that you support? Michael: Oh, crikey. Now you're gonna just r- ruin me. So there's a very long list of these. So let's take some more exotic ones. I think I've talked about, HCL, Domino Workplace. There's Nextcloud, there's Zonecloud, there's OpenCloud, there's Pydio now owned by Wire. There's C-File there's eGroupware. But let's take, say, Moodle. So if you like doing a learning management stuff there's Moodle, and there's another one. There's the other big learning, open source learning management system, which now slips my mind. Both of them are using Collabora Online, and so then you can do classroom stuff. You can give people assignments, get them in market, and do it all inside the web there. Very Jonathan: cool. Michael: And because we're multi-tenant, you can actually do that for... The, the same instance can serve both your Moodle and your, Wire and also your, Py, all of these things on the same cluster, which is really nice aggregating that together. Yeah and I've forgotten we have about 230 partners, so I'm gonna get shot by, I don't know how many people afterwards. But I'm terribly sorry, a, a Jonathan: bunch of Michael: them. Jonathan: And so now- Michael: Yeah, a, a real bunch of Jonathan: them. It... But if somebody just wants to throw this on a Raspberry Pi, right? Sure. First off, does it fit? But then secondly, like I'm still trying to understand, like what storage backend do you use if you don't have anything at all, and it's like I just wanna use it on the Pi? Michael: Sure. So I think the Pi would probably provide something for you there, but OpenCloud, ownCloud, Nextcloud, one of those things is quite popular often. And you just grab one of those and then you embed us into that. Okay. And yeah, we do run on the Pi. That used to be a challenge. Actually, so Calabra did some of the original hardware enablement of the original Pi with the- Oh, really? ... the overlay masks- ... and the one-bit alpha, and the trying to make that perform really well. And they did, there were some miracles done I think with the original Pi. But when you look at a modern Pi, whoa, you know- Yeah ... it's this multi-core powerful GPU and a- again I'm hoping open source drivers left and right. So the world has totally changed- ... and that's something that actually can really run quite well now. Jonathan: The, Michael: the Jonathan: first one or two Raspberry Pis were quirky in multiple ways. Michael: Yes. Yeah, absolutely. One of the, one of the things I like particularly was that if you used one of these gaming mice, I actually have one of these, you know, that I... Just to test, with buttons everywhere and, you know- it, it gives you so many events coming out of the thing to try and keep up that, that it would just drown the poor Raspberry Pi's event interrupt handler. So like it would spend 30% of its time just trying to respond to mouse events and send them to people, which, yeah, it's not as cool as it can be. You're probably not what you want. And so yeah, we had to- Jonathan: Yeah. Michael: Yeah. Yeah, we had to dial that back a bit. Jonathan: Yeah. Michael: But yeah, these days of- it's no sweat at all, of course. There's lots of horsepower there. Jonathan: Yeah. Okay. And so you said there's a big release coming up in, like o- on Thursday. Michael: Yeah Jonathan: We're recording this on Tuesday. It goes live on Wednesday. It'll be the day after that. What what are you excited about there? What do you want, what do you want folks to look for in that new release? Michael: Yeah, sure. So one of the big things there is, of course AI. Now, lots of people are either fans or not fans of AI. A controversial minefield. I'm just... Let me just tap dance into it briefly and and we'll see what, what explodes. Obviously it's off by default, so lots of people don't like that. Yeah. Nothing is gonna leak your data anywhere unless you tell it to. And actually it's better than that. So the sidebar then will show you, and it kind of color-code it is this your dangerous off-premise whatever, or you can make it green or red or whatever to try and help the users understand- ... the, the implications of what they're doing. And then of course you can actually start to use the selected text as context, so that's nice. So instead of it being this kind of nailed on the side thing that can see the whole document, it can actually interact with bits of document that work works really nicely. And then there's lots of course, little demos. Of course, if you're writing text documents- Large language models turned out to be reasonably okay at language. And i- it makes a lot of sense for those kind of translation and make the tone more professional or make it a casual style. Th- this kind of thing is, summarizing and conciseness is really... Are really easy. It can do other things too, like looking for fraud in spreadsheets, so if you you know- Yeah ... numbers that are fraudulently added often don't follow the distribution you'd expect of sampled numbers. And so there's a whole load of things you can do with that's cool. Jonathan: Yeah. Michael: And yeah, and you can of course make pretty pictures and embed that. That's very important. Of Jonathan: course. Michael: In your presentations. Jonathan: Yeah. Does, does the c- the new Collaborator AI features, do they have the ability to use locally hosted AI? Michael: Of course. Of course. Yeah. And that, that's su- super easy because ultimately- these are basically a standardized protocol. You dump stuff down a, a text socket and you're done. So yeah, that, that's really nice. And there's also some MCP stuff coming- ... Michael: Which is really helped by this, are we talking about the security and the jails and the onion around that, right? You really don't want the AI playing inside your machine if you can avoid it, in a scary way. So having all of that sort of wrapping around that's really helpful. Whole load of other features obviously in there that, transcend the AI stuff. So better navigation, better search, a pretty UI s- document comparison, so side by side. We didn't do three up, which I think developers like, side by side document comparison, where there are- larger changes, if you've been offline for a long time, maybe when you come back. I was actually using Starlink on the in Mid-Atlantic the other day, and it's quite... Yeah, you can use Collabora Online really nicely over Starlink back to Ireland and, you're in the middle of Greenland looking at icebergs out the window and it's working just nicely, except for the laptop power, which, if you have a beefy laptop, you blow the fuse on the, Jonathan: Yes the Michael: thing. Jonathan: So I don't- I have 100% had this- Ah ... this experience. This is my- Too annoying ... this is my complaint, my one complaint with my Framework 16, is that it's got such a beefy power supply that it immediately, doo, turns off the the airplane breaker. Michael: Yeah, indeed. I'm trying with these, these xscale. So the hope is that if you get a small laptop with a with a big screen, and you put your glasses on- ... the, y- you're really winning. And so then you can get a lower power thing and li- life is good. But unfortunately my eyesight is, I'm like halfway to glory signpost here, and it's really just not very good. And so I needed to cut down some gl- anyway, we'll see. May- maybe that'll work out in the future. I hope so. Jonathan: Yeah, Michael: absolutely. Jonathan: It's funny. Michael: So yeah, lots of other things. Better calculated fields and pivot tables. Lots of interoperability functionality. We actually had a chance to do systematic interoperability work, so we have this huge, over a million test documents now I think. And actually just working through that and doing two kinds of things. So just making sure that stuff is preserved when it's roundtripped, but also just silly stuff like tweaking the order of attributes, so when you diff the XML to see what's different, you get many fewer changes there. And just making sure we don't do silly stuff. The whole load of- paper cuts there that would just cause grief on import that we've just got rid of for better round trip interop, and that's, it's really nice to see the fruit of hundreds of fixes there proactively trying to chase- things that could be problems. Now, of course, the, the test set is huge, and it's all bug documents with the worst documents in the world. We're really pushing the I guess you'd push envelopes, don't you? Something like that, you know, trying to find the the twisty corners. But it's- Right ... yeah, it's great to have a team that you can put on that for a while and make a real difference, so Jonathan: yeah. Yeah. I- ironically, that's something that AI is actually pretty good at, is to help find those weirdnesses in the code and write test cases for them. Michael: Yeah. I suppose so. Yeah, possibly. I, I like it for for the documents. What we're really focused on, though, I think is trying to train our staff because it turns out if you're looking at product for a 10X productivity win the, the key is to get your juniors and, give them the confidence and skills and experience to, to really to up, up level what they're doing there- I think. So that's that's where we're focused on that. Hey, I go to school- local schools and recruit. If you can get the best guy in the year, or best two people in the year- Jonathan: Yeah, Michael: absolutely ... that, that would be a guy and a gal, to to train them up, then we try and do that. Just because finding skills in our 35-year-old co- codebase is just not there. You get people sending you, "Oh, I'm a full stack developer. I know React and JavaScript and some Node." You're like, "Yeah." None of that is what we do. We, we have bespoke toolkits and huge C++ code bases, so we find, yeah, it's better to train people than to to hire, Jonathan: Yeah, interesting Michael: people's preconceptions. Yeah. Jonathan: Interesting. So in, in doing the development and the the maintenance on the code base, where you guys have had such an influx of users, and sort of users that are different than what you've had in the past in some ways at least. Sure ... have there been any really interesting bugs or findings that have shaken out from, like- Suddenly, the, the German minister is using Collabora. Are there any fun stories around that? Michael: Yeah, so surprisingly few which is annoying. And often they, some... So we had one with the Danish Ministry of Digitization, I think. So they're doing a pilot and their biggest problem that they had was that they had loads of styles in their documents, and they had style previews at the top, and they wanted, we were previewing all of the styles. And they wanted to filter this down to only a few. And so there's a feature in Microsoft Office that will effectively hide or not show, not make it so easy to use all of the styles they didn't want you to use, and then see the ones you want. And so this is a big deal for them, so we, we implement that, ship that, they're happy, life is good. And then for free. We're helping pilot these things. If you're looking for really odd bugs, some of the ones often come from automated systems. So some genius out there writes a script that generates OpenXML. Okay. So Microsoft Office generate OpenXML is one thing, but they basically generate this thing and they tweak it until it works in Microsoft Office. And that often involves not outputting any style information creating, they, they basically twist the thing to the nth And see what it will tolerate and generate the most minimal file. And so you then just have to know that if you don't have a font, it's Arial 12 point. And if you don't have a, a this and a that and the other, these are all of the... There are no page margins, there are no whatever. So all of those defaults that aren't actually written down anywhere, you then have to make sure that they match. So that's- Jonathan: It, it actually seems like that would be a really interesting place to set a fuzzer on and may- maybe even like an AI or an AI-guided fuzzer. Like- ... make minimal viable files that Microsoft Office will open, and then see what Collabora Office does when it tries to open them. Michael: Sure, sure. So actually I, we have some loop automation going on there that loads- ... all of these files, exports them to Microsoft formats, and then imports them to check that actually Microsoft is not complaining. So we have a big old loop automation doing that over hundreds and hundreds of thousands of files in the background. Jonathan: And then- Actually- And then you fall into a question, like a, a really important question I think, and that is do you want to be bug for bug compatible with Microsoft Office? Yes. Or at which point it's like Microsoft does this wrong, we're going to fix it. Michael: Yeah. So I think to some degree that depends whether Microsoft wants to be bug for bug compatible with themselves, right? Fair. And that's the sort of expectation. So there's this very famous Lotus 1-2-3 bug which has been inherited from Lotus 1-2-3. I, IBM created it, and it's a missing gap year. They, their leap year algorithm was wrong. And that sounds like... and so I don't know if you know how spreadsheets work, but basically you have this double precision number in a cell- ... Michael: And then you apply a format and it shows up as a date. But, a- and so you'd think we can fix the gap year algorithm. We'll just..." What would we do? We change the number, but the number could be a result of a formula. So then do you walk the formula tree and try and change the root numbers to fix the gap? It's just, it becomes- it becomes fundamentally impossible to fix, ultimately. Or to work out even in a spreadsheet where it is. And even today we support the old and the new way. The real leap years and the old leap years. So there's a setting there, and Microsoft has that. We have that. And, there are sheets that are still using that. And actually one of the things that you find most surprising is that when we look at some of, say, our macro support, so we support VBA macros to some limited extent in your browser running, from a XLSX file. Some of the things in those macros are the Office 95 to Office 97 portability shim. So in Office 97 they radically changed VB because b- that, that was before they'd forked the whole Visual Basic engine and put it in Office. So they were following the VB thing, and they didn't realize that the language would change and then the APIs would change, too. And so there's basically a magic object that is I forget the name of it, but, say VB 95 compat or something. And they rewrote those macros line by line to introduce this compat thing and have all of the old methods on it. And we still see those out there. So we have compatibility for that Office 95. Only some subset of tho- that, that helper class, which is weird that you would still see that in today's code. Jonathan: Yeah, so there is a... So you said that and it brought a memory to mind. MS Works. I, I have... Th- this, this kind of blows my mind. So years and years ago oh my goodness, I don't even want to tell you how many years ago, my family had a computer that did not have Microsoft Word on it, it had Microsoft Works. And we'd save documents in Microsoft Works. And Microsoft Word will not open those documents, but OpenOffice, LibreOffice, and I'm assuming Collabora Office does. Indeed. And it is of endless delight and hilarity to me that Collabora, and of course the rest of them, are more compatible with Microsoft products than Microsoft products are. Yeah. And I think so that's hilarious on one hand, but there's also a, a deeper thought here about Making that information of those documents accessible in the future and even just like the inf- the idea of the information preservation that's going on. Yeah. And so like you think about this, it's 100 years from now, are we going to be able to open any of these documents? And I think part of the conversation that needs to be part of that is like open source allows you to have a chance at preserving- Yeah ... that information. Yeah ... who- Microsoft may not exist in 100 years. ... Obviously Microsoft Word is not going to exist in the same format in 100 years, right? Yeah. And so th- this is not... I think about this with a lot of things, right? Even like the the preponderance of YouTube videos that are out there right now. Like YouTube ev- is eventually going to go away, and are we just going to lose... Like YouTube has become culturally important to the world. Are we just going to lose all of that? And so that's- Sure ... that's obviously, that's outside of the conversation we're having right now. But I think it is very important to think about that just for documents that like- Michael: Yeah ... Jonathan: it's important to think about the preservation for the future. Michael: Yeah, absolutely. And I think the sort of digital archeology almost is something. And there's a couple of guys there. So there's a chap called Friedrich Stroeher and Valek Filippov, who calls himself Frob. And these guys are just passionate about that. That's what they love doing. And they've found a niche there that they can serve in a beautiful way- Yeah ... and created those document liberation filters, and all kudos to them. And there's a project there to liberate documents they call it. But yeah, so we can import those Works files. Now, there's a security question there because the more import surface area you have, the more risk you have. So we tag those with what we call an exotic filter. So probably you're gonna get a pop-up going, "You do know this is a file that last was used, 20 years ago. This is the last file that was written with this, this format," right? And so hopefully that, that defers the, makes it much harder for a zero day in, if- if there is such a thing there. But then again, these guys are really passionate, and they fuzz their codes very heavily. There, there's a whole, there's a whole big fuzzing thing going on there. And actually it's really cool. Let me just tell you about that. So there's a thing called OSS-Fuzz. I don't know if you've seen that. The Google project. Yeah, absolutely. And that's run in part by a U- Oxford based UK company, I think called duh. Nope. I can't remember their name. That's terrible, isn't it? I how rude, let me see if I can look it up. Jonathan: I Michael: know OS- Anyway, Ada- ... OSS- ... AdaLogix. Okay. AdaLogix. Yeah. Google fund it. So Google fund the hardware, but I think A- AdaLogix contributes very heavily there. One of the things that they have is these huge multi-thousand node clusters basically do using evolutionary algorithms to hunt down bad code paths that, that take you to nasty places. And that's extremely effective. Before that started running, we would routinely get what looked like nation state reports on, there is this remarkably well-formed bug report that w- with a sample document that takes you to a problem. And after we got the cluster, that didn't happen. We caught most of those during development- which is the way to do it. One of the interesting things we found from the Mythos stuff, so AdaLogix actually has c- the... So it's a code coverage informed tool. So it's actually watching what code is executed- Absolutely ... and then trying to enlarge the coverage. So let me give you an example. We had a bug in our HTML filter for copy and pasting into spreadsheets. So there's a little custom parser there of tables And there's a bug in it. And so not only did it find the bug by walking the code wider and wider until it, it got it- ... Michael: But then it reversed the process to minimize that down to the smallest piece of HTML that would walk all of those paths and go there. So first it colors it all in red, it finds the problem, and then it colors it all, in gray again until it's just got that. And so normally file formats are binary, but with HTML you can see, like these are the 63 characters that minimally walk down the stack tra- trace to find your crash. Absolutely. And that's just such a beautiful thing. Yeah, it's very useful. And so we really love that. And I guess that's almost automating QA. Or you could argue it's like a, a precursor to AI stuff. Jonathan: Yeah. Michael: But the problem comes when you hit a crypto wall, right? Like you can't evolutionarily work your way through a signing. That, that ain't gonna work, right? And so one of the wonderful things that Ada Logics is doing is then applying Mythos to audit the code past that. Yeah. The bits of code they couldn't get to with their fuzzing. And we're getting quite a number of interesting more interesting reports from them which is great. Jonathan: Have you- ... have you found that they're fairly high-quality reports? Michael: Yeah, definitely. I Jonathan: mean- So that's something that people have found. Like historically over the last, let's say six months, it's gotten a lot better. But, Yeah ... you would ask AI to find you a vulnerability, and AI has n- this is one of the problems with it, still is one of the problems with it, honestly. AI has no capability of saying no or I can't do that. And so it- ... will just, historically it has just hallucinated a vulnerability for you. And then- Yeah ... someone that doesn't know what's going on will report that to a project, Michael: In, and in great detail with this, this mass of text that then someone competent has to read, which is the people you wanna protect. Look, all tools, all new tools have this problem. So if you- Sure ... if you look at the sort of what we've done with security, we did like W error. So cr- s- stop compiling when there's a warning. And of course there are a whole load of warnings that are, you need to instrument your source code to turn these off, because actually- Right they're not really errors, but at least there's a manual process of doing that. And then you add more and more warnings, all of which fail, and you repeat this process and eventually your W error clean at the highest level the compiler can do. And then you put, maybe lint tools and you put Coverity Scan on it. And, Coverity Scan comes up with loads of, you know- False Jonathan: positives ... Michael: yeah, there's a drop through into a case statement, and of course that was intended that you did this case and then the default after it. And it's, but you then put an annotation in saying, "I meant to drop through here." I know that it drops through. "Please don't warn me." Yes. Jonathan: Yeah, Michael: absolutely. And, and so at least half of the, the warnings coming out there are false positives, maybe more in, ace human-made static checking tools. So we, you expect that. And there's always a peak and a trough. Every new tool comes with a new set of problems, and then they fade away. I actually have a OWASP talk when I show the graph of all the generations of these these checkers coming in and the impact that has. Jonathan: But Michael: yeah. Yeah ... so yeah, I think doing good stuff interesting security pieces and yeah, just great to have that OSS Fuzz back up that. Jonathan: Yeah, absolutely. I've thought for the longest time that it would be super in- and it sounds like this is what they're doing, but it would be really interesting to have a fuzzer that is guided by AI it wants to look in- So as Michael: is, lib- libFuzz is like that. Yeah. So it's, I- as I understand it, it's looking at the c- the code that's executed. It's then i- inferring how to perturb the input to- ... increase code coverage, and it's watching the code execute as the data flows through it, and then perturbing the data, and it's doing that incredibly quickly. Yeah. So it forks and checkpoints and comes back, and it's... yeah. It looks like magic. It looks like intelligence, but there's not a, there's not an AI in it, it's a- ... it's a, no. So it's really cool. Jonathan: Yeah. Super cool. Michael: I love that. Jonathan: One of, one of the note I've got here, we're running out of time, but one- Yeah ... other note I've got here is that you you also have, in all of your copious spare time, you have a comic strip. How did that come about? Michael: I do. I do. Oh I've been, in the open source industry now for a very long time, and there's a lot of funny situations and scenarios and problems that people come across. And as one gets much older and, it, it might be useful to to pass on some of the the fun stories and- in a friendly way. So explaining what open source is we do that with, Oh, actually, I've got... what do I have? If I reach over here. Yes, pictures of beavers. If you've got the camera on, you can see this. Yeah. Building roads instead of building software. I think one of the tragedies as a software engineer is that we make beautiful things. We make castles in the air poetry that you can't see. There's a whole aesthetic around taste and structure and architecture that were at a cathedral, people would want to walk into it and go, "Wow, that's really good," or, "Oh, that's a bit dingy." Like- ... why didn't we paint, repaint the basement?" And it's just a shame because software engineers can communicate that to each other and see it. And so how do you articulate and share that? Anyway, we decided to do road-building as a more concrete way of seeing a- actual- ... talking about construction and- right luckily I studied a bit of civil engineering as well, so there's some- Oh, okay ... there's a whole lot of sort of overlaps there. Jonathan: Pull- Michael: pulling from- And show me the road instead of show me the code. Ah. It's, Jonathan: Show me the road ... yeah. I gotcha. I gotcha. So speaking of these sort of in-jokes, and this crosses over to bugs as well, one of my favorite stories about a bug, and you may have been involved with this, was the can't print on Tuesdays bug. Michael: Oh. Oh yeah. I think Bjorn fixed that. Yeah. I don't remember all the details, but yeah, some horror in the, the generated, output that. Yeah Jonathan: It, when you went to print, it generated a file, and because of the way the timestamp and the date stamp lined up when the computer then looked at it to print it, it interpreted it- Yeah as a different kind of file. Michael: Yeah. It's really not good, isn't it? And you... Yes, you can't make this stuff up, but it, you ... And actually it's quite interesting. You think Linux is by now a very polished desktop. It looks beautiful, and, i- it beats many other things. My... Just recent files is such a useful thing in your file picker. That's the hot place everyone goes, right? And you just still not... I don't think you see that in Windows still, which is just nuts. It's such an obvious and simple innovation, right? It's a beautiful desktop. Underneath it, though, when you start to print things, there really is a shell that's forking LPR and piping stuff to it still. And that's that's all fun. And so yeah if the file doesn't look like the file they think it's gonna be, then yeah- ... that can cause some problems. And yeah. And then of course it goes to CUPS at the back end, and there's some kind of reason- reasonable, IPP thing going on hopefully. But, Jonathan: yeah. So it it thought it was an Erlang JAM file. I had to go and found it, and went through and- ... and pipe- clicked through it. Yeah, this was what? Back in Ubuntu 9.10. Almost 20 years ago, this was a thing. Yeah. It's still, it's my... It is absolutely my favorite bug story because it, it well captures the seeming inanity of some bugs. And you're like, "How in the... There's no way. There's no way that could be a bug. There's no way it could work like this." And then you step through the details like, "Oh, okay. Yeah, I see it. I understand now how, this, this random, seemingly random collection of of things came together," and you can't print on Tuesdays. Michael: Yeah. And sometimes you need an expert to help you find those things and fix it properly. And that's yeah, so we love to be that- Jonathan: Yeah ... for people. Absolutely. Have you guys started using AI to try to find and fix those sorts of things? Is, are you, are Michael: you- you know- ... on the AI Jonathan: programmer bandwagon? Michael: As I say, I think by far the most effective thing we can do to improve productivity is just train and help our young people get involved with the code base. So that I think is, that's the 10X. And I'm always intrigued to hear of these stories of huge productivity wins. And so actually we do a whole lot of training along those lines. So we do tee-time trainings, and videos, and we look at the code structure, and we try and- try and teach people that a- as we go along. But yeah. So come and get involved, and watch a tee-time training and get stuck into the code. Jonathan: If somebody wants to if somebody wants to come and actually hack on the Collabora source code, where's a- Yeah ... where's a good place to start? Michael: Yeah, so if you go to collaboraoffice.org then there's whole sort of pictures of beaver mascots that'll make roads and all sorts of nice instructions to help people, Get involved. And of course, there's so many different platforms now. So there's Collabora Online on Linux, and then there's iOS, and Android, and Mac, and Windows and Linux. And there's instructions for each of those, and simple setup steps. And then we have a Matrix channel, so you can just jump in there and say hi, and get, get stuck in. Or come to our... we have weekly meetings, so just come to the meeting if you got stuck. And there's a video chat there with a whole load of engineers and mentors. People love to help you get going, really. Jonathan: Oh, very cool. Yeah. Awesome. Okay I've gotta ask you a couple final questions before I let you go. Sure. I do get emails if I forget this. You personally, what is your favorite text editor and scripting language? Michael: Oh, dear. So I get... I use the editor for middle-aged computer scientists, Emacs. Jonathan: Okay. Michael: It's probably a mistake. I should probably enter this advanced modern world of, automated everythings. But yeah for me that's where the key bindings feel like home. Sure. And the nervous tic is to bookmark where I'm at, so I just do Control-X, RB, one, every, too often. And then you say... what was the other thing? Client choice? ... scripting, Jonathan: scripting language. Michael: Scripting language. Grief. So because I'm extremely old, if I need to do text processing, I would use Perl, just because I understand the stupid regular expression syntax- ... and whatever in it. I- if it's gonna be s- small and quick. If it's bigger than that, I go for Python, just because my colleagues tell me I'm old and crusty, and they can't understand my Perl scripts. Which is fair enough. I'm completely with that. So yeah Python, I think, is is probably the right answer. Jonathan: I shall have to ping Randall Schwartz and let him know that we got someone that said Emacs, and their first answer was Perl. He will have a little celebration for that. Michael: Excellent. Excellent. Jonathan: Michael, thank you so much for being here. It has been a blast to talk about Collabora and all the various things going on there. Thank you so much. Michael: Absolute pleasure. Thank you. Jonathan: All right. That was Michael Meeks of Collabora. And we've got some fun stuff coming up. Next week we're talking smoked meat, and that is actually about CI security and not losing your all of your secrets to a malicious pull request. And then after that, we have Jonathan Polant of Ferrous Systems. We're gonna talk Rust yet again. And then I know we have some things working in the pipeline, but we've got some openings after that. You can drop us an email, [email protected], and we will get you on the schedule if you've got an open source project to talk about. Other than that, we just wanna say thank you to everyone that is here. We sure appreciate it. Whether you get us live or on the download, whether you listen or you watch we wanna say thank you. And we'll be back next week on Floss Weekly.
• 
  Episode 876 - There Is No Money Fairy 22.07.2026 1t 12min

  This week Jonathan chats with Michael Meeks about Collabora! What's the origin story in this consulting company, why do they have an outstanding office suite, and where is the world headed to accomplish digital sovereignty? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 875 transcript 15.07.2026

  FLOSS-875 Jonathan: This week we're talking with Neriman Gelva about Pewter. That is the operating system that runs literally in your browser. Maybe we're stretching the definition of operating system there just a little bit, but still. This is Floss Weekly, episode 875, recorded Tuesday, July the 14th. JavaScript as a systems language. It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we are talking about Pewter. That is the operating system, the desktop that lives on the cloud, on the internet, in JavaScript. I have questions about how this works. It's something like that. Can you locally host it? I think so, but I don't know for sure. I know that it's open source. That's why we have them on the show. And the guy behind it, the man himself, is Neriman Jelva. I met him at the Ubuntu Summit this year, and he did a couple of talks there. Actually, they had him do a lightning talk and a full-length talk all about the what you do when your open source project is a runaway hit. And I know a lot of people would say, "Oh, I wish I had that problem." It's actually a big deal. It's a tiring problem. It's a, there, there are things to deal with that maybe you don't think about. But Neriman is a programmer. He is an entrepreneur, and he's been doing this sort of stuff for about 30 years, so he knows what's going on and how it all works. But I think Pewter is a new ballgame even for him. It exploded in popularity. We even covered it. We talked about it over on the Untitled Linux Show as well as this cool thing that you may wanna check out. A lot of people thought it was cool. There's about 42,000 GitHub stars. I don't think he paid for any of those. 390 people have contributed. They've got over 400,000 downloads. And apparently, it's in 100,000 different web applications, which sort of talks to the the actual use case for Pewter. I could go on. I've played with it a little bit, but we do have Neriman here. And let's go ahead and bring him on, and we'll get the answers from him to all of these questions. So welcome to the show, sir. I'm so glad to have you here. Nariman: Thank you so much. Very excited to be here. Jonathan: Yeah, we were talking beforehand about... we were comparing notes about our travel schedule, and I was telling him about things that I've got coming up. But we met at the Ubuntu Summit in London. Are you based out of out of London? I'm in the middle of nowhere in the United States. I'm in southwest Oklahoma. I'm in flyover country. I wave at all the planes going by 'cause I know they got my buddies on them. Where are you based out of? Nariman: I'm in Vancouver, BC, Canada right now. Jonathan: Ah, our northern brethren, the Canadians. Nariman: Yes. Yes. And, but our team is from all over the place, basically. We have a couple people in the US. We have some people in Europe, Asia. Pretty much remote, like a true open source project. Even though we're a company, but... Jonathan: Hey it's allowed to be both. Open source developers- Yes ... need to pay rent and eat too it's okay to make money on these things. Okay, so let's start from the beginning. You- you've got a team now. You've got a business. You've got an open source project. That's not where any of this started, though. What was the very beginning? At what point did you go- A desktop on the internet would be cool. How did that come about? Nariman: Yeah, sure. So if you remember maybe you never had to pay attention to this part of the internet, but I would say between 10 and five years ago there was this period of time where everybody was proclaiming that the browser is the next operating system. It was like something people would say, and I really believe that because I remember the first days of the web. That's when I started writing my first line of code, and then, So I basically grew up with the web, and I saw the progression over time. So I really do believe that the browser is the next operating system, depending on how you define the operating system. So n- obviously not a Linux. That's not what we're talking about. Basically m- more about the user space. That's the discussion. We can talk about this more later on. But the real thing a- around this, the real thought around this was always that, hey, this is like... The browser is like the v- a virtual machine, that you can probably run any application in it. And more and more applications were moving to the web. But nobody really cracked the general purpose environment where you can just have any type of application, have an IPC, a desktop environment, and those kind of things. It's been tried many times before. In fact, I think the first try goes back to the late '90s. Imagine those browsers. Somebody tried to build a desktop environment. I think it m- it may have been literally called WebOS or something like that. So this has been tried many times in the past, where you have this operating system-looking or a desktop environment in the browser connected to the cloud. So there's been many attempts before, and I really wanted to take a stab at this, and that's why I was always passionate about it. And around five years ago peak COVID time, I was I was pretty disillusioned with building business sort of applications and that kind of industry, that kind of subsector of of the software industry. So I was just visiting my sister, and then I was on vacation, and then I just decided to do this. I was like, "I'm going to go after this." So started out as a hobby project, and then it took off later on. But the gist of it is people always wanted to do this. There's been many attempts at it, and then I think a lot of things came together a few years ago that made it actually possible. Jonathan: B- I jumped ahead by a question, and you answered the one that we need to ask. But let's make sure that we've gone over it. What is Pewter? What's the... what's the... is there a problem that it solves, or is it just a... s- start with this. Tell, tell us what exactly it is. What would you describe it as, someone that doesn't know, that's never touched it? Nariman: Yeah. So I would say Pewter is what we call the open source internet computer or internet operating system. If you go to Pewter... let's put the open source stuff aside for one second just- Sure just to explain how this thing works. If you go to pewter.com and create an account, you're basically given a very small cloud computer or a computer in the cloud- ... that you can use personally. Now- You might think, "Oh, we're giving you a server, or we're giving you a container." It's none of that. It's something that we built from scratch. It's a distributed file system with authentication database, everything attached to it. Anyway, you get this virtual computer, a cloud computer in the cloud that you can use for basically any purpose. You can put files in it. You can run applications on it. A lot of people use it to just r- use our office softwares. Just like a regular computer, it has very general purpose applications. Now, why would you want something like this? That, that goes back to the fact that, hey, operating systems in general I'm not talking about the kernel, but the user side of things, haven't really evolved in a very long time. We've been always device-bound, and anything with the cloud was always tacked onto the operating system. For example, operating systems don't support a cloud file system. It's always been, hey, I do something on my computer, and then it sync to the cloud. Or I'm using some website that does this for me. Or authentication is never universal. Maybe with Apple you have a little bit of that, with Google, but it's pretty fragment fragmented. So when you go into different applications, it's usually I create an account with this app, with that app. So the whole being sorry for the buzzwords, but cloud native or AI native thing- it, it never it doesn't really happen with the with the regular dominant operating systems. And so Pewter is an attempt at bringing that sort of modern paradigm to the basically desktop environment or Chrome-like interface, Chromebook-like interface. Now I could go on and talk about the use cases and what p- people are using it, but if you have any questions, I can answer those before. Jonathan: We will get into the use cases. I've, I've- Yeah ... gotta ask, though, and this maybe will lead on to that well. I'm going to intentionally undersell Pewter so that you can tell me where I'm wrong. It's just a, it's just a website that writes some JavaScript that looks like a desktop. That should be easy. Why did it take off? Nariman: That, that's a good question. So in the beginning, I think in the very beginning, Let's say four years ago when I put it on Reddit and it really blew up. I think at the time it was mostly because people were just shocked by the amount of effort h- that had gone into it, and the level of detail that was worked on. So ba- and back then, this is before AI. So I would say maybe five years ago. Jonathan: It's Nariman: in the dark times. But back then it was pretty difficult, right? It was really difficult to code like that. I had to sit down and manually code every single interaction. It was pretty difficult. So I think the very first time I put it out, people were just shocked by this, and then over time an application ecosystem formed around it. So we add applications. It started with very simple applications, like a Notepad application, and then a very simple camera application, and then all these APIs formed around it to make these applications better. So people started slowly, gradually using it as f- for file storage. For a file storage where you can edit text files. Believe it or not with something like a Dropbox or a Google Drive, you still can't edit plain text. So if you put the plain text in one of these services, you still can't edit it. You have to download it or install their local applications to do that. Yeah. So it started fill these small niches where you put your files there, and the interface is more advanced than a Dropbox or a Google Drive. You can double-click. You're familiar with this paradigm. I double-click on a text file, I can edit it, and then a photo editor was added so you could edit photos. But then the reason a few years ago it really took off was that we managed to port Visual Studio Code to Pewter. So we were- Interesting. Yeah ... basically the first yeah, the first company or project that managed to port it completely to the web and hook it up to the cloud. Now that's, that was a big hit, and it happened completely organically. But then again about a year ago coding basically died and nobody uses their code editor anymore. So that, that one- I- ... that one went down, but by the- Jonathan: I resent that. Nariman: You tell me, I've been coding since I was seven, so I... my identity is really tied to it. Jonathan: Yeah. Interesting. Nariman: Yeah. So it, it has started with basically a shock value sort of a thing- ... and then it, an ecosystem s- formed around it with applications and then with features. So now you can do a huge number of things with Pewter. It is really full of features. You can drop a directory on the desktop, you can right-click it and publish it as a website. It has zipping, compression. You basically get a lot of the things you get with a regular desktop, but it's happening in the cloud. So going from device to device, it's always there. Nothing changes. Your environment doesn't change. You can go from your- Laptop to your phone to your TV to your Tesla fridge. You, you name it. If you have a browser, you can use it basically. Jonathan: Yeah. So there's a ... To write all these apps, is there like an API that, that sort of resembles like a... Is it almost like a C standard lib or a, POSIX layer that you guys have written for JavaScript? Nariman: Yes and no. I, we were definitely inspired by it, but I think the JavaScript ecosystem in general, we had to move more towards maybe a Node.js-like API, but the concept is pretty much the same. We couldn't make it POSIX compliant. Maybe we'll end up doing it at some point. I did try it a few years ago, but it didn't really work out. We were almost there, but w- it didn't really work out. Now, we have our own basically standard, but it's close to typical APIs from Node.js or those kind of environments, more higher level. Jonathan: Yeah. So th- there's a, there's an obvious thought that I've had, and probably other people have asked you this too. But something that's really becoming popular and that people are doing a lot with is WebAssembly. And you've got some crazy things like taking existing apps and compiling them to WebAssembly. And so I'm, I've gotta ask, I'm so curious, has there been any work done with this in Pewter? Can I take an existing C++ app, compile it to WebAssembly, and somehow bootstrap that into this on the internet desktop interface? Nariman: Funny you ask, because we're doing a lot around that right now. In fact, tomorrow we're dropping a bomb when it comes to that. We're releasing a major port, a really major application. I don't wanna name it, but everybody knows it. We ported it to the web for the first time. Nice. And I think... And then we have two more, and they're progressively crazier. So yes and no again. The layer is not there yet, but we're working on it, and we have the proof of concept coming out. The first demo is coming out tomorrow. So very timely question actually. Nice. Jonathan: Yeah, so I was thinking about this at the Ubuntu Summit, and it's a potential end game of this, you could see there's several different ways this could go, but in the end, you could see Pewter being, like, almost a target that you could use CMake to compile something to. If you have a sort of a standard interface that would be, would map onto POSIX. You may not be POSIX compliant per se, but you can map it onto POSIX. And you could imagine just being able to write a prog- a cross-platform program, and one of the targets be Pewter via WebAssembly. And that's really interesting to think about. Nariman: Yes. Yes, for sure. And I think we're making incredible progress towards that. And this demo coming out tomorrow is a really great proof of concept. There, there's work to be done because there's a lot of file system calls and those sort of things- that we have to properly port and create a layer for. But what we're definitely making huge progress. Again, I think I keep, I I don't really bringing this up all the time, but I think AI is helping a lot with that as well. For sure. Otherwise, we would've needed s- we have a lot of contributors, but we would've needed way more than that to, to achieve some of this stuff. Jonathan: Yeah. So what what does the contributor base look like? There's hundreds, thousands of people working on it? What... How does that map? Nariman: No. No. We're at 390 right now, and there are ups and downs. So the way we typically do open source contributions, attract open source contributions, is that we go through cycles- where we create a lot of issues, we ask the community to help us out, and that's when things pick up, and then we have a merge cycle where we go through all of these and merge them back in and close all the issues. So right now we're not in one of those cycles, but when it happens we get a ton of contributions. I think by the end of this month we're gonna start another cycle like that. It's an odd, ... I would say it's an odd arrangement the way we do it, where we go, we call for contributions, and we create a lot of issues and we ask people to help out. But you didn't ask this question, but it's an answer to the burnout question, so that you w- you wanna manage your resources, especially you have limited resources, you wanna have focused times where you're only focused on the open source. Of course, we get contributions over time. We get contributions from some companies that are using Pewter, so there's ongoing things like that as well. But the bulk of contributions happen in cycles rather than consistently. Jonathan: Managed sprints with the community. Nariman: Yes. Yeah. I would call it that. That's a better way to call it actually. Jonathan: Yeah. Interesting. I'm looking at the App Store now and there's just a bunch. There's just a bunch of crazy stuff on here. There's games, productivity, lots of AI stuff of course, but has anybody ported has anybody ported Doom yet? Nariman: Oh, yeah. Oh, yeah. That, that, that was a thing. That was a thing back in the day. Yeah? Ev- everybody was obsessed with bringing Doom to it. But then again, Doom has been ported to the web, and Pewter is basically a web environment, so you don't have to do much to bring Doom to Pewter. Jonathan: Yeah, that's fair. That's fair. I remember you said that apparently there's been some interest in Pewter from the crypto community. Not the cryptography community, but the cryptocurrency guys. Nariman: Yes. So I think there is some parallels between Pewter and the ideas of a decentralized ownership and decentralized computing. So it's been it's being used by two crypto startups, and they've raised a bunch of money, and they're working... They basically forked Pewter, and they're building their own version of it, which is pretty awesome to see. So some of it comes back upstream obviously. And yeah, so they're building... One, one project which is interesting is that they're building a decentralized app store on top of Pewter. And so with NFTs, every time somebody clicks on an app or opens a file the creator of that app or file is supposed to get paid. I think it's a pretty cool idea. Jonathan: Interesting. Yeah. How does that work with the license? Isn't Pewter AGPL? Nariman: Yeah. It should be fine. As long as you open source it. Jonathan: Yeah. Nariman: Oh, Jonathan: that's true. Nariman: It should be fine. Jonathan: Yeah. A- AGPL is kryptonite to a lot of big companies like Google and Amazon. They're like- Nariman: Isn't that the point? Jonathan: I just find it interesting this startup, a crypto startup can could use AGPL and play nicely with it. I thought it would have been kryptonite to them too. Nariman: Yeah. But I really like it. I remember reading this article by Torvalds a long time ago, and he was saying GPL is the reason Linux is this big. I think he's being too humble, but it, it does make sense. It makes a lot of sense, right? We're going to give you a huge amount of resources we've put into R&D and IP creation. We're gonna give it to you. You can do whatever you want. Just- ... if you change it, just contribute it back and everybody wins. That's basically the idea. But AGPL as everybody probably knows, the audience being open source AGPL is basically the GPL of the cloud world, so that, the loopholes around cloud are basically closed up. Jonathan: Yeah, the GPL, if the code runs on somebody else's computer, you don't get any rights to it. The AGPL- Yeah ... basically says you still get rights to it, even if you use it on somebody else's computer. Nariman: Yeah. Jonathan: Yeah, it's, Now, does that mean that all of the apps have to be AGPL as well? Nariman: No. The SDK is Apache. Jonathan: Okay. Nariman: And the linking happens within an iframe, so it's not linked directly to our environment. No, you're safe. Your app is yours. Jonathan: You've done your homework. You understand how that works. Nariman: We have a, we have an incredible lawyer by the name of Heather Meeker, who's done all these licenses. She's very famous. She has written so many books about open source. In fact, she's the reason, she's the main reason I open source. She convinced me. So she's done she's done the open- the open core licenses for MongoDB, Elastic, if I'm not mistaken, all these big names. So she's incredible with this. And she guided us. Jonathan: Yeah. What so that was something I was gonna ask you about. When you first wrote this, it was all closed source, and you got talked into making it open source. I'm curious how that went, and also how it's going. What do you think of that decision now? Nariman: So ever since I put Pewter out, I would say one of the top three requests from the community was to open source it. So I was bombarded by it, comments, emails, DMs. Every- everybody just wanted this open source. I didn't, I- it wasn't a huge priority for me at the time in the beginning And because we ended up be- becoming a funded startup, there was also considerations around, hey, should we open source this as a business? And so I wanted to, even though, look, for many years I've been a programmer, ba- I basically grew up writing code, I still needed a proper, really logical business argument for it. I had to understand it really well- ... before open sourcing it. 'Cause I'd never done an open source company before, where the core product is literally open source, the entire thing. Yeah. So I wanted to understand this really well before I do it. I met, I ended up randomly meeting through friends, the founder of you might m- you may have heard of them, n8n the workflow automation software. So they are open core, but they're massive contributor base base. If you care about GitHub stars, I think they're around 200,000 now. So they're a really huge project. Yeah. Super close to the community, really doing it right. So the founder actually told me, "This is perfect for open sourcing." He gave me a number of reasons, and then he introduced me to the person I just named, Heather Meeker, whose whose entire career has been focused on open source. And then she's the one that gave me a bunch of really specific reasons why Pure should be open source. So for example, one argument is that, first of all, open source doesn't work for everything. For example, open source games, they're not a really huge thing, right? There are a number of open source games, but you never heard of a open source game that has, I don't know, 100 million users or something like that. Players. So some type of software is not really good for open source, even though I think everything should be open source. It's pretty awesome. But from a business perspective. But then again open source does really well with infrastructure or hyperhorizontal platforms, namely databases, programming languages, compilers- and and the best stuff of all, operating systems. You want that hyperhorizontal platform to be open source so that people can work easily on the long tail of applications of it. As an example- ... we would've never thought somebody would take Pure and extend it for crypto. These ideas wouldn't, would never come to us. So they took it, and they're improving Pure just because they have another type of use case for it. There's another really good reason for it. So in the beginning when we put Pure out and it went viral a couple of times, one of the biggest questions people had was around privacy. They're like- "oh, you're harvesting the data. You're trying to resell it." Because it was an open source and it was free, just, I just put it out. And because of the architecture, it was super cheap to run. So everybody was like, "What is this? If this is, if these are VMs or containers, how can he afford it if he's not selling the data," right? So it's, it makes sense. Even though I spent so much money on creating this incredible privacy policy, nobody cared. They were all suspicious. And then I remember Heather told me that the, your community is mostly developers, and you're basically front-loading the cost of building trust and branding with developers by releasing your IP to the world. So overnight, you're exchanging that with brand equity- ... and trust with developers. And she was right. The moment we open source it, when we open sourced it two years ago, never again did anyone ask me about privacy. It was the number one question before that, and the moment it became open source, nobody really worried about it. Jonathan: Yeah, absolutely. Nariman: So there were a couple other reasons, but I think these were really solid reasons. And there, there are some practical questions about how to open source. For example, you want to use common open source licenses. AGPL, MIT, BSD what have you. You don't wanna go out and create a custom license. You gotta be very brave to do that. The reason being people just don't know it, and then you add this thing called understanding tax to it, where now you have to understand what the license is. If it's a company, the lawyers are typically familiar with AGPL, GPL- ... all these famous ones. But then now you have a weird license that nobody understands, and everything just- Right screeches to a halt. Jonathan: If you're AGPL or GPL, there are literal cheat sheets where you just go, "Can I do this thing that I wanna do?" And it goes... Yes. Nariman: Yeah. True. True. Jonathan: Absolutely. So there's a- y- you were talking about trust a little bit, and I was gonna hammer on this, because that's the- that's the first thing that comes to my mind, is, somebody has a product, especially if it's free, and it's just out there and it does great things. I'm like, "Man, why would I trust this?" There's the old adage, right? It goes, "If something is free, you are the product." Nariman: Yeah. Jonathan: And that makes a lot of sen- especially the AI world that we live in now, that's also true, because usually if the thing is free, that means that your data that's going into it is being used to train an AI somewhere. And people might say, "Oh that's overly cynical." It- You may have seen Pokemon Go, right? Everybody was playing a few years ago, this was before COVID. I'm old enough to remember the Pokemon Go craze. Turns out that data has been used now to train AI of all those players' data. Oh, I didn't know anything. Nariman: Oh, Jonathan: wow. Yes. Yes. I forget- I never Nariman: played it, so Jonathan: I forget the exact details, but yeah, Poke- the Pokemon Go data has now been recaptured and where they had all that, the data downloaded, and it it's being used to, for training for AI now. Nariman: Oh, Jonathan: wow. Again the product was free, therefore the data that you generate is the product. And it's not that can't be a thing with open source. Obviously, there are companies that use data that... And sometimes that's not necessarily a problem, just so long as, everybody's upfront with it and you understand what's going on. But even just, would I be willing to put anything at all confidential into Pewter? If I can't see the source and I don't know what it's doing, then no, not at all. So I, I think just from the trustworthiness perspective open sourcing it was really the only option. I just, I can't see, I can't see the thing having a life outside of being open source just because of that. Nariman: You're right. You're right. And this answers the second part of your previous question- ... how is it going now after open sourcing? I would say it was an incredible decision. It, it couldn't be better than that. I think open sourcing Pewter was the exact right decision. I sh- I just should have done it earlier. So no I don't regret it even a bit. And I think it's given me so much personally and as a business. It's just Seriously, the best decision in the life of Puter, in the history of Puter. Jonathan: Yeah. Yeah. So I've... i'm real curious about a few things related to this. Do, does Puter have end-to-end encryption? Can... And again I'll be slightly cynical here. I'm doing it on purpose- Yeah ... since you can make the point. Can you look at my files if you want to? Nariman: Y- your files are encrypted at rest on S3. Okay. And if you really care about the privacy part, the open source Puter works really well, so you can put it on your own server and you can do that. But as much security and privacy as we get from AWS, and I think as an open source project and being a startup, we are pretty... I look at other projects and a lot of projects obviously have resource issues. You don't have enough volunteers and everything. But I think we- we've been pretty diligent when it comes to privacy and security. We have this really active bug bounty. We do, for example, buckets in different regions. We support this kind of stuff that is pretty much enterprise level, but we support it as an open source project. So we've gone out of our way to be privacy friendly. Now, end-to-end encryption, the way where your files are encrypted in the browser, we still don't support that, and it's mostly because we're not too good at it. We still have to figure out how to do it properly, but I think we're not too far from that. And our business model has nothing to do with people's data. That's one thing. So it's not oh, we're gonna sell ads to them based on their files or anything like that. So it's only in our interest to actually get deeper and deeper into this- Yeah because it's good for businesses and enterprise. Jonathan: Yeah. What is the business model? Nariman: Yeah, the business model is actually pretty simple. It's subscription based on the amount of resources that you have. So when you go create an account, you get a few megabytes of free storage, some AI credits. A lot... What we do, aside from AI and storage, we provide a lot of services under one API. There's a database there's hosting. There's, again, the AI and storage. So there's a lot happening. We have serverless workers, which is compute in the cloud. We have all of this as a simple API, and obviously if you consume, if you use a lot of apps, you're gonna keep consuming these resources, and at some point you'll, you're gonna end up upgrading. Jonathan: You're gonna grow out of the free tier. Nariman: Yes. So it has nothing to do with somebody analyzing somebody else's data to monetize this. It's none of that. It's basically pure cloud computing infrastructure for the regular user. Yeah. So it really does make sense for us to get deeper and deeper into privacy, security, and those type of things. Jonathan: Yeah, absolutely. Do you see a, Do you see a future where it might be, like, really useful for enterprise? Are you guys gonna have a, Oh, I for- I forget the terminology. There there's several different accreditations that you can get to where, you can use it with US government stuff and do you see that sort of thing in the future? Nariman: Oh, for sure, yes. Yes. And I think we're very well-positioned to do that because maybe out of luck in the very early days, I designed the architecture that is basically sharded to a to certain regions, meaning that even if you use pewter.com, which is the hosted version- ... you can, in theory, have all your data housed in a specific location. So that's really perfect, and it was baked into the architecture from five years ago- I see ... and it just grew with it. And that was by luck because I just saw the API and in the early days it was only S3. I saw the API that you can set the region. I'm like I'm gonna add this to our API." So that's basically what I did, and everything formed around it. So now that gives us so much advantage because this entire architecture and all these hundreds of thousands of lines of code has formed around that architecture that architecture choice. So we are incredibly well-positioned from that perspective. Also, being open source and having feature parity between o- the open source and the closed source version, proprietary version, the hosted version, all of these, and the fact that everything on Pewter is scoped to the user rather than being multi-tenant, that's also really incredible for the enterprise. It basically makes it enterprise ready right off the bat- ... from a technical perspective, but we still have to go get those certifications. I get that. Jonathan: Yeah. But that is something you guys are thinking about NIST certification and all that? Nariman: Yes. Jonathan: I s- definitely some potential there. Another hat that I wear, I've had to start thinking about these things, and some of that is a challenge to get right 'cause you've got to certify the entire desktop, the whole computer you're working on, and so- Yeah ... it seems like it could be a really interesting piece of the puzzle for businesses that need it, that are required to have it. Yeah. Nariman: Yeah. And I think in general, in the long term, it is, just from a business perspective, it might be a moat. I think some of these certifications actually enforce some good practice, so it's not just check- I see that- Jonathan: Some of them enforce some bad practice too. Nariman: Yes. True. Jonathan: Make it opt-in please. NIST certified you've got to use five-year-old encryption. You can't use the newest stuff, and there's- Nariman: Yes. Jonathan: There's some nasty stuff in there too. Oh, anyway so all of that is... Let's see. I asked you about the the WebAssembly, and you said you've got a f- you've got a fun announcement coming tomorrow. Hopefully, we can get that in the show notes, and people can go and check that out. It's pretty obvious why you used JavaScript, right? There wasn't a whole lot of other option outside of JavaScript. But you opted to use jQuery, and I'm curious about that. I have... So far I've managed to resist, and I've not written anything in jQuery. I don't know why I am stubborn and have chosen this hill to die on, but so far I have. Nothing particularly against it. I just, for whatever reason, it's no, I don't want to use jQuery in any of my stuff." What led to that decision to, to use it? Nariman: Yeah. So with the performance requirements of the desktop environment, we couldn't really use frameworks, front-end frameworks. So we had... Basically our choice was vanilla JavaScript meaning no li- no, no frameworks. And vanilla JavaScript is pretty nice. It's come a long way, it's... a long way. But I would say- It's still pretty difficult to work with vanilla Java scale- JavaScript at scale. So jQuery this library that was released, I think, in 2008, and it was built- That's a while ago for a completely different... a lot of stuff that jQuery did in the early days made it into the JavaScript, ... standard. Yeah. So we use jQuery as basically a helper library rather than a state management sort of a framework. But then again this sort of became a meme and I just ran with it. I'm like, "Okay, people make fun of it, but it gets so much attention." And some people actually defend it because it was such a wild choice building something this complicated in jQuery. But looking at other major very front-end heavy applications like OnlyOffice or Photopea, and I think even Google Docs they're- they don't really use any frameworks. They might use frameworks around the around the core for some of the basic UI stuff- Yeah ... but none of the major rendering or anything like that is happening using frameworks. It just, it's not performant enough for these purposes. So I use jQuery, and it- it's been incredible. We got really good feedback around the front-end code because it's easy to understand. Anyone who has any JavaScript experience can probably easily understand it, and you don't have to be tied to a certain framework to just read the code. So it did work out really well. And the desktop environment is very very stable, super stable. It never basically crashes. And it doesn't have performance issues. So far, we haven't heard complaints about it. It took many years to get here- ... but I think it was a right choice. But then again, something that has to be clarified, jQuery is a helper library that helps with with some of the stuff that we do with vanilla JavaScript. It's not, we're not managing the state using jQuery. That's not... We- we're managing the state using DOM, which is another crazy thing we do. But but jQuery is not, oh, jQuery versus React. It's not like that. Jonathan: Yeah. Interesting. You guys are using it more just to like backwards compatibility and to make sure things act the same on Firefox versus Chrome, that sort of thing, right? Nariman: That, that was the initial reason jQuery was made. To make make the code work consistently across different brow- browsers, because they weren't adhering to the standards perfectly- ... back in the day. But we use it because of SelectorsAPI. But that's not a problem anymore, Jonathan: right? Nariman: Yeah. You tell me. Have you written code for Safari? Jonathan: They still make Safari? Nariman: Oh my God. We have to. And- it's a, it's an incredible browser, but there is the stuff where- Yeah ... all of a sudden something breaks. And you go dig in, oh, it's not it's not... They haven't implemented the certain standards or whatever. Especially with the stuff that we push some of the really cutting edge that our team does, the ones that we are actually gonna start announcing, we do hit the wall pretty quickly with Safari a lot of times. Jonathan: Yeah. No, I'm not surprised. That makes sense. Yeah. I say that tongue in cheek, but Safari's dropped in popularity a lot. And then of course, if you're running iOS, your browser is Safari- Yeah ... whether you like it or not. Yes. I'm curious, have you tried any of the up and coming alternative browsers like Ladybird or what's the other one? Servo, I think is the other one. Nariman: Well, Ladybird I love Ladybird. I've been a sponsor for a very long time. Oh, very nice. Cool. I think it's an incredible project. I've talked to Andreas. Jonathan: Oh, Nariman: he's great. Yeah. And I once o- one of their team members used to work for Pewter as well. Jonathan: Okay. Nariman: So yes I do I do love them, and I think it's a great idea. The core of it is that Ladybird, I think could be the Linux of browsers, a really completely proper- open source license core that doesn't have any commercial interest. It's not tied to a certain company. And it's not corrupted. I love it. I love it, and I think it has incredible potential because if... Going back to what I said half an hour ago, if you go back to the idea that the browser is the new OS with some footnotes- Ladybird is very well positioned to be the kernel of that, basically. And in- incredible project. Jonathan: Yeah. Yeah. Super cool. Nariman: But does it work? We did port it to Wasm. It's a browser. It doesn't get more complicated than that. I think it's one of the most complicated pieces of software you can ever build. We did port it to Wasm just for fun a couple weeks ago, so you could run Ladybird in a browser. I mean- Oh ... just for fun, obviously. Not a lot of utility value, but we just did it for fun. Jonathan: That does get back to that thought, though, about WebAssembly as the new a new target to be able to compile anything to. I think that's a really intriguing idea. And- Yeah ... boy, I hope that can become a thing. There are so many things that would be really fascinating to be able to just run them inside of a browser instead of on the desktop. B- but does Pewter work inside Ladybird? Nariman: Yes. Really? To some degree. Yes. Surprisingly, yeah. Yes, Jonathan: asterisk. Nariman: We, we haven't tried it recently, but I remember Andreas actually did create a video about it. A lot of the things loaded, and I'm really surprised because we're doing some really crazy performance intensive stuff behind the scenes. And this was, I think, over a year ago, so I'm pretty sure it's way better now, way better. Yeah, I'm Jonathan: sure. The way Ladybird development works is one of the developers goes, "Hey, this website would be really cool to support." And then they go and they try to load it up in Ladybird, and they see all the things that are broken, and they'll just sort of- step down through the list of what they need to do to make that particular website work. And that's how the development works over on Ladybird right now. Yeah. And if you've got some people there that are interested in making Pewter work, then, yeah, sure. I'm sure they're working on the things that you need to bring it up. That's pretty intriguing that it works there on yet the, yet another browser. So is there better support for Pewter on Ladybird than there is on Safari? Nariman: No. No. Okay. I haven't been able to corrupt them. No. My sponsorship is not big enough to sway the interests of developers. Jonathan: Yeah. That's funny. That's funny. I- so interesting interesting segue here, talking about being corrupted. You've taken some venture capital money, right? How does that, how Nariman: does that work? Oh, boy. That's corruption now? Yes. We have. We've done we've done, I would say, two rounds of financing- Okay ... institutional investors. And yeah we were lucky to find investors that are super aligned with open source. They get it, they understand it. Open source com- it's called commercial open source software. So that's when you take an open source project and you have a commercial arm, it's called COSS or a commercial open source software. I think the understanding of it has been commoditized at this point. Investors get it. They understand it. They know how it works. They know what it means to be a commercial open source project. I think the success successes of the past the more recent ones, like MongoDB, Elastic- these sort of infrastructure offerings that became huge that really pushed investors to take open source more seriously. But historically- ... Nariman: I think open source was undervalued until we had these giant unicorns come out of it. So at this point, people just get it. A lot of VCs, they they are tracking every metric of repositories. They have indexes. They have they have software that tracks it to see how well they're doing and reach out to them. So it's become a very competitive s- space. So back in the day, I think it would've counted as semi corruption, but now it's just Jonathan: How is that process going though working with your in- your investors? Is it still going along pretty well? You guys are ma- making them happy and they're making you happy? Nariman: Yes. Yes. I think when it comes to open source, we w- again, we were just incredibly lucky to have investors that just get it. So being open source, I don't even remember being open source was a question at all in any maybe issue that we had with techno- whatever is happening on a day-to-day basis, it's never "Oh, maybe you shouldn't have open source." Like- ... not even once has that come up as far as I remember. So it's never about being open source is just the bedrock of everything we're doing, so we don't even go back to it, "Oh should we change the license or anything like that?" Yeah. No. Jonathan: Yeah. You've done quite a bit recently with with AI tooling, right? A bunch of the apps now have AI. You even mentioned that when someone signs up, they get some free AI credits to use. Nariman: Yep. Jonathan: What's been the what's been the special sauce that makes that work? Why is that so interesting inside of Pure? Nariman: Yeah. So if you go to our developer site the about s- SDK and everything, developer.pure.com or the API documentation, docs.pure.com we've explained this using illustrations and everything, but early 2025, we saw a surge in application. So first of all, when we open source in March 2024 application development really surged on Pure. We had issues at attracting developers, but open sourcing it really put us in a much better position, so we started getting developers organically. But I s- started seeing a surge in 2025, early 2025, and then we realized this is because of AI. So turns out- ... the architecture of Pure is really good for vibe coding and AI programming. Jonathan: I was just looking this up. So the term vibe coning, vibe coding was coined in February of 2025 by Andrej Karpathy. Nariman: Yes. That's when it started taking off, and I would say end of 2025 is when it just completely revolutionized- programming. Jonathan: JavaScript is probably really s- well-suited to the LLMs writing it, isn't it? Nariman: Yeah, I think not being typed or weakly typed, that contributes to it. And it's just, you don't need a very complicated build process- ... Nariman: To run a JavaScript program, basically, especially the browser-based ones. And again that's one reason Pure JS took off so much, because you add... lLMs, they don't need to MPM install anything, or they don't have- ... you just take that CDN URL, you put it in your HTML file, and then you see it in ChatGPT or or Claude. So that's why it took off. But yes, JavaScript is well-suited. Python, JavaScript. Yep. Jonathan: Yeah. You think about it, too. The entire training corpus of LLMs has JavaScript just sprinkled all throughout it because they're training on the internet. And so they're just seeing- ... JavaScript everywhere. Yes. And that's one, you might be able to make the you might be able to make the argument that JavaScript is the native language of LLMs just because there's so much of it on the internet. JavaScript and H- HTML, right? Nariman: Yeah, and I think there's another reason for this. So if you go to ChatGPT or Claude- ... but nowadays people use people use Call Code and Codex. They actually go through the installation process and download these tools. But a lot of people just use ChatGPT or Claude or Gemini, the web interface. Yeah. And if you ask the web interface to build an application for you, the only way it can actually show you something when they load the artifact or the preview panel the only way they can do it is by writing it in JavaScript. So they're biased, I think. They're biased towards building it in JavaScript so that the user can see it. Yeah. And I think that creates a vicious cycle or a flywheel. Jonathan: Yeah, for sure. One of, one of the guys I work with, he used, I think it was Claude, to put together like the b- a bare bones presentation, like PowerPoint presentation, and he got to looking around in the back end of that, and Claude built it in JavaScript first and then rendered it from JavaScript over to PowerPoint. Oh, wow. And did a really good job on it, too, which is fascinating. Like they're really good slides. Oh, wow. We've used them several times now. Just keep... we keep iterating on them. We'll go and we'll make changes by hand as needed. We'll have the LLM do changes, just depending upon what it is. But yeah, it's so- Was that Nariman: Claude Design, Jonathan: or? I think it was I think it was Claude. I don't for sure if it was Design or Chat. Probably Design. I don't know. It's been a little while ago that he did this, so I don't know if Design was a thing at the time. I think it probably was. It was probably in Design, but yeah. It's incredible. It's real fascinating to see. You've run multiple companies now. You said you're a serial entrepreneur. Is it particularly different doing an open source company? Nariman: Yeah, there, there are differences. We have a much larger community to deal with. And yeah, the, in terms of community and conflicting interests is a completely different ballgame. And from a business perspective also, the strategy is kinda interesting too. You have this hybrid strategy where, okay, you have the open source project and that has to stay open source and has has to have maximal feature parity with the proprietary version, and then you have this other product which is proprietary. You wanna make sure there's no conflict of interest between the two. Be very clear. So there, there is really differences. My other company, I did... I was pretty cl- pretty close to open source projects as well. For example I our company was doing very large scale data extraction from the web. So- ... Mixnode was a name, and it was massively scalable web crawling. So it was actually suited for AI and those kind of applications where you need billions of web pages of homogeneous data. So not scraping, nothing like that, just getting huge amounts of data to basically for machine learning for the most part. And I would, we were one of the seeders of Common Crawl, which all these LLMs are based on. So we would we would contribute to Common Crawl, and we had a lot of open source projects as well with Mixnode. But it was never the core strategy. W- looking back at it, we should have probably open sourced it. What were we thinking? But then this was a long time ago. But yeah, no, it is definitely different. There it's more complex to run an open source company, but I would say it's way more rewarding when it comes to software. Jonathan: Yeah. You get to see... I- if nothing else, you get to see your stuff used in places that you wouldn't have you wouldn't have imagined. Nariman: Yes, and I think that's incredibly important, not just because it's fun- ... which it is, but it's also very important that a lot of people take your creation, and they use it for things that you couldn't have imagined. So I've heard that somebody's using Pewter. They're trying to use it in a car in China. And so these are the types of things that I would never come up with them myself, right? Look at Linux. It's being used in the craziest places, and it makes sense 'cause you need that diversity of backgrounds of people who use it so that they can- ... take this, and they were like, "Oh, I have this idea. I could use this in my line of work for this purpose." And it wouldn't be possible without open source. Jonathan: Yeah. Have you, I don't mean this as a gotcha question, although it is. Have you seen Pewter pop up yet? Has it been used for things that you're like, "Oh, I really didn't wanna see our name there"? Nariman: Oh we are a very huge target for cyber attacks and those sort of things, and we have... I hope I'm not challenging people to attack us more, but we have pretty powerful anti-abuse systems in place. But there's been, people people were using it for phishing, that sort of stuff. Yeah. It's under control. We have a lot of systems in place right now. We are, again for a small company or with limited resources, I think we've done exceptionally well when it comes to that. We put a lot of effort and investment into it. But the really bad stuff, no. Thankfully, we've managed to prevent those. Jonathan: That's good. I I, so I'm involved in the Meshtastic project. Obviously you know that. That's what I was talking about at Avanti Summit. We did an interview one time with a reasonably well-known online magazine and, no, I think they have a print version too, and the the whole... It was like a 45-minute interview. The first 40 minutes were great, talking about the technology, and then they had the gotcha question at the end, which we knew it was coming. We figured that they would try to do this to us. But they had the gotcha question and they're like, "You know that Meshtastic is potentially being used for human trafficking across the Mexico border." And we're like, "Sure it is. Don't do things that are illegal with Meshtastic. Other than that, we're just like Signal. It's an open source project. You can use it for whatever you want." So we were ready for it. We knew there was gonna be a question like that coming, and there's some other things going on around the world that we keep, try to keep track of and, some of it's cool. Some of it, you're like, "Yeah I wouldn't, really didn't wanna see our name there." But anyway I think it's something that a lot of open source projects have to have to wrestle with and deal with, particularly ones, like Signal and Telegram, those sorts. They, they wrestle with this more than even than we do. But I'm always curious to hear, like, how that plays out. Nariman: Yeah. How can you even prevent it, right? If it's open source- ... Nariman: And you can Jonathan: take Nariman: it. Jonathan: Depending upon what it is, like what you're talking about, there's also the question of should you even try to prevent it? Or is this just... You know what I mean? There's a... And I guess when you're talking about a hosted platform like what you guys are doing, you do have some responsibility to prevent certain things. But there is, part of the open source definition, in fact, is, you don't make moral judgments with your licensing. Yeah. So like it's, it, you, doing open source you cannot say, "We consider this to be evil and therefore you're not allowed to do this with our software." And boy, that's unpopular with some people, but there's a really good reason that exists, and that is that, the world cannot agree perfectly on what it means to be evil or not. Nariman: Yeah. Jonathan: And that's really what that boils down to. Have you guys gotten- Nariman: haven't some of the craziest inventions come out of military, for example? Oh, sure. If there was a moral judgment against it, "Oh, you can't use this technology," or, you can do a certain thing," we wouldn't... Maybe we would have them anyway, but probably not as fast as coming from a military funding- basically, a research facility. Military funded. Jonathan: I'm trying to remember. I wanna say the microwave was actually invented because of radar, if I remember correctly. Nariman: Yeah. I Jonathan: think, Nariman: I think so. Jonathan: Do. Yeah, I'd have to, I'd have to look to find that out for sure, but I think that, I think it's all related to that. Have you guys... And you may not be able to comment on this and that's fine, but I'm just curious, have you guys gotten like subpoenas yet? "Turn over this person's data"? Nariman: I'd rather not comment on that. All right. Jonathan: That's fine. That's fine. Yeah, that's a, it's a, it's such a tricky thing to deal with, right? We've yeah we've seen a limited amount of, We have been contacted by law enforcement once, and that was very informal. But still we see that on the horizon of this could be a thing. Now the advantage over at Meshtastic is we don't have anything that we can give over. Nariman: Yeah, that's what I was wondering. What were they looking for? Jonathan: They were... I think in that case, they were just looking for information. We think this person may have a Meshtastic radio. What can you tell us about it?" We're like, "Not much, man. Here's the documentation." Nariman: Yeah. Yeah. They... they can ask the same thing from ChatGPT or something 'cause I know you're not tracking anything. Jonathan: Yeah, so the, Yeah, for sure. Apparently the the microwave oven. This sounds like it was written by probably Google's AI, but it may be accurate for sure. The microwave oven was invented in 1945 by Percy Spencer, an engineer at Raytheon who was working with electromagnetic waves. So yeah, Google- Google's AI overview. But still, that- that's- that matches the that matches the memory I have. He was testing out a radar magnetron and noticed that the candy bar in his pocket had melted. Nariman: Oh. Oh, wow, I didn't know this part. Yes. That's really cool. Jonathan: Yes. It's also terrifying. Nariman: It's yeah, it's like the story behind antibiotics where it was basically an accident. Jonathan: Basically an accident, yes. In the case of the microwave oven, though, he was microwaving himself accidentally, and that- Nariman: Yeah. ... Jonathan: that maybe was less good, but, I dunno. Oh, true. Fun stuff, though. Nariman: Things haven't improved much because- right now if I put something in the microwave, I have a cold basically outer layer with a super hot nuclear kernel of, ... spaghetti. Or the other way around. Actually it's really cold inside. Jonathan: Yeah, that's usually what happens. It's Nariman: raw outside. Jonathan: Yeah. So what, we talked a little bit about this, but what's the weirdest thing that you've seen somebody do with Pewter? Where has it surprised you the most to find a pop-up or to hear a, s- from a customer? Nariman: Ah, that's a good question. It's funny that I'm blanking because PeterJust RS CK has been used by 180,000 applications, and I'm blanking. I don't know which one is the craziest. I think in our showcase we have some really cool stuff. God. I'm blanking right now. I'll remember it though. Oh, Jonathan: you'll remember the- Nariman: I'll remember it Jonathan: in a few minutes ... yeah. Yeah. Yeah. If you think of it before, before the Hackaday article goes live tomorrow- Yeah ... you can send me a, send me an email. Okay. And we can include it in the show notes. That'll work. Sure ... the story of the Chinese car company sounds pretty fascinating. Nariman: Yes. If Jonathan: somebody wants to use it- Nariman: But it's not- ... for their self-driving car ... it's not really public, and I'm not sure when we're gonna hear back with a solid POC. But I think- Sure ... it should be soon. Jonathan: Yeah. Nariman: It was supposed to be more than six months ago, but then again, you know how- Oh, yes ... how scoping and... Jonathan: I know how it goes, yes. I- I've been catching up on some of my emails, and it's "Oh, this was sent two months ago. Sorry for the delay. Been really busy." Nariman: At some point you run out of excuses, yeah. There's nothing you can say. You can just start stop apologizing and just respond. Jonathan: Just do it. Yep. Just do it. And then I gotta ask you a couple of questions. First off, before I get to these, is there anything that we didn't talk about that we should have? Nariman: No, I think it's been pretty great. We touched upon the most important aspects of it. Jonathan: Okay. All right. Yeah I feel like we've done we've done a pretty good job taking a tour through the ecosystem. ... If somebody wants to get involved particularly host your own, where's the best place to go to? Nariman: Our GitHub page. And we've put a lot of effort into making it easy to run Puter. You can... As a de- for dev environment, you can just download it, clone it, and then npm install, npm start. There's nothing else you need. And then we have the curl to bash as for the- Oh, you remember that. Okay. Jonathan: Yes, I do. Nariman: Great. Curl to bash Jonathan: ometer. I... So I've complained about curl to bash for years. And when he started telling that story, I'm like, "Somebody else is on board." Bad. Don't do it. Nariman: Yeah. Yeah. But that's what people expect, right? There was so much... we internally, we argued too much about this, right? Because our better engineers meaning everybody aside from me, was against it, but I'm the one who was like, "No, we have to do it because everybody expects it." We ended up doing it because I pushed really hard, but I get it. I get it. It's not the most popular thing from, So- ... the engineering perspective ... Jonathan: I have thoughts here. And we probably ought to explain to folks what we're talking about, right? So- ... the question here is, like, how do you install software? And one of the popular ways to install software is that the software company just gives you a script. You download the script, you run the script locally, and it does all of the installation magic, right? And so you end up doing a curl piped into bash. And at the Ubuntu Summit we were talking about the, one of the keynote itself was from Shuttleworth, was, like... He had this, this- He called it to the curl to bash-ometer. And it was essentially like how many of the most popular applications is this the primary installation method? And, his chart basically w- y- you're going along and you have this, and it just goes up through the roof, right? It was it was pretty interesting to look at, and they were of course using that to pitch snaps as the alternative. And I have for the longest time made the argument that, unless you really trust the person publishing this script, you should not just run the script, and the fact that websites can get hacked means that you probably shouldn't just run the script anyway. You need to look at it and try to figure out what it's doing. And installing packages is better. But then of course there is the sort of the overhead of somebody like Pewter has to then package up the thing for RPMs, package it up for Debian, package... Make sure that package works on Ubuntu, package it up for this, package it up for that, and it's a lot of overhead work, right? And so there's this kind of back and forth of do you just write one script that does the install or do you do the packaging? And what I would actually say is you probably have to have a script that does the install, right? Like I don't think there's any getting away from that because that's how you would do... That's how you're gonna install it inside of a snap. That's how you're gonna do a Docker install, all of these things. But like it is also super useful to have then based on top of that, a snapshot that is installable without the script. And there's tooling to be able to do that so that you can not directly run the big ugly scary script on your production machine, which is really what we want to avoid, right? Nariman: Yeah. But it's the most convenient one, right? Absolutely. Sure. You're encapsulating the entire complexity of choosing what to download, what to run- ... and you just give them one script. So that's why I fought so hard for it. And I'm like, "Hey, it's our responsibility to make sure everything is safe." I know it's easier said than done. We have way too much experience with this security stuff to underestimate how much of an effort it is. Yeah. But at the end of the day I've always been har- fighting super hard to make it easy to run Pewter. And in the beginning, for the longest time, and I think one of the reasons Pewter used to go viral over and over again, is because for the longest time, if you went to pewter.com, you would just see the desktop. You didn't even have to create an account. It was just a desktop environment. It would do everything automatically for you. It would basically trust you, give you credits no matter what, just because I want it to be incredibly easy to use. The reason we ask for accounts now is actually over time, that became an issue where people got confused if they have an account or not. So it's not even because I want them to create an account. Anyway- ... but the idea, the core of the idea has always been it has to be incredibly easy. It has to be easy. Whether it's open source, whether it's hosted, you wanna make it, make this super convenient for the user. And I think curl to bash- with all its faults it is something people expect for the most part, and that it is a pretty convenient way to install. But then you might end up nuking your machine, so you gotta be careful. Jonathan: Yeah. Is there like a Docker image or a snap- Yeah ... or something like that people can use? Nariman: Yeah, we have a Docker image. Yeah. Jonathan: Okay. That's... See, that's perfect, right? It's the best of both worlds. You get both. Nariman: Yeah. Jonathan: There there's a... I think you probably... Most projects of this scale, you've gotta have an installer script. I think, like you said, that's just expected. Yeah. And then you also have a group of people, like me, that look at that script and go, "Psh, I'm not running that on my computer. I don't know what's in there." We're just a little bit paranoid. Yeah. For good reason, I would like to think, but still a little bit paranoid. And then, you've got the Docker image, you've got the other things to where, you can package it up, and people can do it without having to run the script. And so everybody's happy. Nariman: I think it's great. Being paranoid about this stuff is great actually. It keeps everything, th- these are the checks and balances around open source, right? And I think these sort of being "Oh, let me look at the script first," it actually, in a way, helps make the project better over the long term if the maintainers actually listen to the concerns. But the good thing is that now you can probably... you don't have to read the script. You can just drop it in chat GPT and be like- It's true ... "Hey, is this... How horrible is this?" Not if it's good or not, how horrible is it? It, Jonathan: it- Nariman: And then you can Jonathan: see- ... you know what, I'm about to run this script. Do you see any signs that it's gonna do anything malicious or otherwise cause problems? Yeah. Nariman: Yeah. Jonathan: That's actually a really good point. We have tooling now. Nariman: Yes. Jonathan: Yes, we have tooling, We have tools that make this easier. Yeah. Yeah, absolutely. I'm looking at your documentation to make sure that you have the the other stuff documented here. I don't see it. Maybe I'll make a pull request inside the self-hosting Puter. Oh, boy. Yeah. The one-liner recommended... I don't see the, I don't see the alternative. Nariman: Huh. Take a look. Jonathan: But I'm, I am of course very quickly skimming through this, so I may just be missing it. But yeah, I would love to see a a little alternative so that people... Oh, hey, look. Ha, even above the one-liner, there's the Docker Compose option. Okay. Yes. You're good. I just missed it. Nariman: Thank you. Jonathan: Very cool. All right. Again, two final questions I've gotta ask. People will send me emails if I forget this. F- what is your personal favorite text editor and scripting language? Nariman: Oh, boy. I wasn't... D- do I have to say Vim? Is that, You can Jonathan: say whatever you want to. I answered this with Nano most of the time. I'm not... You're not gonna lose geek points. We've had a few people tell Nariman: us- I do Vim I really do, but believe it or not I'm gonna sound horrible, but I do use the text editor on Puter a lot. Sure. So it was the first application I built on Puter- ... and it's been it's been improved over the years. I'm actually used to it. So- Jonathan: Yeah ... Nariman: but then again, I don't think that's what people wanted to hear. So I'm gonna say Puter. No, that's a Jonathan: perfectly valid answer. Okay ... I just I think yeah, Vim plus Puter, I think that's a perfectly valid answer. Can you run Vim inside of Puter? Nariman: Somebody did port it, actually. We actually took out the terminal because it was way too buggy, so it doesn't really... until we bring it back, I don't think- Or we can have Vim again I was Jonathan: g- I was getting ready to ask if there was Bash inside of Pewter Nariman: No we did, but it was way too buggy. We tried so hard. We tried to port Linux to Pewter. We got really far, but it was just not production ready. Jonathan: Right. Nariman: We're making more... As I said we- we're doing this release tomorrow. We're getting way more serious about porting things to Pewter really complicated software. And we're making progress. Again like I keep saying this over and over again, but I think AI is really helping us with this. Jonathan: Oh, sure. Nariman: So now with fewer resources, we actually have a chance at porting those really massive software to, or packages to Pewter. Jonathan: Yeah. That would be fun to see that come back. Oh, yes. Just as a command line purist, that would be fun to have it back. Yes. Nariman: Yes. Jonathan: And its scripting language. Nariman: Scripting language JavaScript. I was gonna say. Yeah, it has a special place in my heart. Jonathan: Yeah, of course. I suppose on Pewter it counts as a system language, right? Nariman: Yes. As crazy as it sounds when you say it out loud, it is true. Jonathan: Yeah, makes sense. All right. Hey, it's been a lot of fun. Thank you, man, for for coming on. And we... After after some big announcements, we will have you back here in a few months and talk about some changes- Sure ... that's changed. It sounds like things are moving pretty quick over in the Pewter world. Nariman: Yes. Thank you for having me on. Yeah. This was great. Jonathan: Yeah, it's been awesome. Thank you so much. It was good to see you Nariman: again. Yeah, likewise. Jonathan: Yeah. Nariman: Take care. Jonathan: All right. That is Neriman Selva talking about Pewter, a really fun project, and c- becoming quite the successful business, so good for them. Next week we have somebody else that I met at UbuNTU. We're talking Michael Meeks from Collabora, the open source office suite. And then after that, in two weeks, we are talking with Francois Proulx about Smoked Meat. That is all about security in places like GitHub, in your CI runs. And then we're having Jonathan Polance, a return from Ferrous Systems. We're gonna talk more about Rust and the things going on in that world. If you have a project or you know of a project that needs to be on the show, let us know. It's [email protected]. Give us an email there and we will get them scheduled. All right. Wanna say thank you to everyone that watches, that listens. Get, whether you get us live or on the download, and we will see you next week on Floss Weekly.
• 
  Episode 875 - JavaScript as a Systems Language 15.07.2026 1t 10min

  This week Jonathan chats with Nariman Jelveh about Puter! It's the project that takes the idea of the Browser-as-the-OS seriously. Why did a simulated desktop on the web take off, what the story of making it Open Source, and what's coming next? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 874 transcript 08.07.2026

  FLOSS-874 Jonathan: Hey folks. This week I'm talking with Andrea Gallo of RISC-V International. We talk all about everyone's favorite open ISA, the fact that there might be more RISC-V cores in the world than there are people, and lots more. You don't wanna miss it, so stay tuned. This is Floss Weekly, episode 874, recorded Tuesday, July the 7th. Really, we do PDFs It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we're gonna be a little risky today. We're talking about RISC-V. I, I had to make the pun. I know. I'm sorry. I'm a dad. I make dad jokes. Anyway, we're talking about RISC-V, the open ISA, the open ecosystem. And we've got somebody, we have somebody very special from RISC-V International and from various places. He does a lot of things. It's it's the man when you talk about RISC-V. We have we have Andrea Gallo. I think I got that did I pronounce that right, sir? Andrea: Perfect. Yeah, perfect. Jonathan: Okay. Excellent. Perfect. Excellent. And where I forget to ask this sometimes, but where are you speaking to us from? Where are you at in the world? Andrea: I'm just close to Milan, Italy. Jonathan: Oh. Oh, my goodness. You lucky dog. That's awesome. I'm in Southwest Oklahoma, which has its own charms. A business colleague of mine- Of course ... just the other day was driving through Oklahoma, and they stayed at a, he stayed at a little tiny town with his family in Oklahoma. And he's "Oklahoma's really nice." I tried to tell you it's really nice here. So Andrea, y- you are you're now the, let's see, the CEO at RISC-V International, and I have about three different questions that go along with that. And I think the first one is what's your background? Y- if I remember correctly, you're you've come from the technical side of things, not the- ... i, I don't mean to be derogatory here, but not the bean counter side of the house. What i- what has that been like to go from the very technical side of things to you're now CEO of this huge organization? Andrea: Oh it's a big honor and quite a challenge, of course. Yeah, I have a mixed background of hardware and software for- ... for, since the beginning I've always messed up because I always enjoyed mixing the hardware side and the software side- ... in many different roles. And I've al- always enjoyed facing customers as well. So I started as application engineer in my early professional days. But even before that I was writing little games that were published by one of those old style magazines where you had all the listings, and so I always enjoyed the public facing side as well. And I was also responsible for business development in a previous professional role. So I always enjoys the mix of both technical and customer facing. And, Jonathan: I was thinking as you were saying that, th- there, there's some combinations that c- give you superpowers. For example I've, one of the lawyers that I work with, she's also studying to be a CPA, and o- once you have both passed the bar and passed the CPA exam in the US you're a superhero in the legal world. And I think in the software hardware world, someone that has written firmware and dabbled with software, has done hardware designs being able to do both of those sort of gives you a superpower. And then on top of that, you also do the business side of things. No wonder they made you the CEO. Andrea: No, the real guys are the founders of RISC-V, and all the great minds on the technical steering committee that drive the work on RISC-V. Those are the real brains. I think at RISC-V International, we're just serving these big brains. That's our role, help them achieve what they are, they're really doing. Jonathan: So that's a, that's actually a really good segue. What is RISC-V International? How does it fit into the puzzle of what RISC-V is? Andrea: RISC-V International is the nonprofit association is the organization that hosts and coordinates the evolution of the RISC-V ISA. So we ensure that the evolution of the specifications is done in full coherence with the missions and standards set by the original founders of RISC-V and we ensure that there are no deviations along the path and that we ensure that the process follows the right policies and procedures, and it's transparent and collaborative. I think our role is to ensure that all the members that found the work at RISC-V, they collaborate in a constructive and positive environment. So we have RISC-V International to thank for things like RVA23? For example so we as a nonprofit organization, we are funded by all the commercial companies that design products based on RISC-V or IPs based on RISC-V. But we also benefit from the collaboration of nonprofit and academia institutions and some very skilled individuals. And we are uniformly distributed worldwide. I've been traveling a lot for the very r- RISC-V summits and for other industry events, and also visiting multiple academia. And I can tell you that it's it's so rewarding. It's such a pleasure to see that you can face the same level of technical competence and maturity in understanding and adopting RISC-V, the same level worldwide. From the US to Europe to Asia to Brazil all these students studying RISC-V at classes, at courses using RISC-V for their projects, for their master thesis. It's unbelievable. And even on the daily work at RISC-V in the technical working groups, when we... in the end we do PDF. If I oversimplify, what we do is that we deliver PDF. That's all we do. That's fair. That's what we do. PD- PDF. That's fair. So when we, on a daily basis on the technical working groups I see the conversations from all these great minds and I don't see one geography producing more or contributing more to the specifications and one geography consuming more. Not at all. I see the same level of interaction and collaboration across all. Do, Jonathan: do you think the open s- the open nature of the ISA is to to, to blame i- is what causes that? So many people know about it because it's open and it's taught in academia and it gets used everywhere. Andrea: First of all, just along the lines of I was saying we, we do PDF I would like to add that we focus on the standards. We like to make the reference to the USB or the Wi-Fi standards. Behind me I have some products that I designed many years ago that were, they were based on the USB specifications. And you can download the USB specifications from the website wherever you are, and the same is true for RISC-V. You can freely download the RISC-V ISA specifications wherever you are in the world, and then locally you would invest in designing your IPs or products locally where you are. So it's important because these, Reinforces the message that RISC-V is an industry standard. And this was also recognized by ISO last year. 2020- September 2025, the ISO/IEC, the joint technical committee recognized RISC-V International as a PAS submitter. PAS stands for publicly available specification. It was quite a long effort and helped us improve as well the way we work in tracking participation from all the members, in logging consensus from the members, in ratifying the specifications. And all this has been recognized as a transparent and and fully correct way of working, aligned to the ISO/IEC guidelines. So we got recognized as a, as somewhat as a standardization body. And the next step for us is to submit the RISC-V specification, the ISA manuals, to be evaluated and hopefully ratified as an ISO standard in the future. So I, I was just adding this as an important point for us that RISC-V is a standard. It is not an implementation. There are many implementations done by the members. Jonathan: I think that's a, I think that's an im- important point to make and to step into, and I was thinking about asking you the question, the leading question because the ISA is open, that means that all of the RISC-V hardware out there is open source hardware, right? Not quite. Andrea: I ... No, not quite, and I really like the question. Not quite. From the standard Then you can have all of business models. You can have a pure open source implementation done by academia or by companies that offer an open source implementation, and then offer commercial services to customize or productize it. Or you can have a fully closed commercial IP designs for licensing. All options are possible, from a pure open source academia to a fully closed commercial. But I also like the hybrid model. If you think about it companies in the open source space companies like Red Hat or Canonical or others, in a way they all invest in open source software. They sell you a subscription to their operating system distributions, and then they also offer you commercial services to integrate that into your s- software products. And this is this is an option for some of the RISC-V members, have an open source RISC-V implementation and commercial support services. Or fully commercial. So all the shades of colors are possible. Jonathan: And so when we talk about the RISC-V ISA that, that's essentially like the definition of the instruction set and like the definition of what each of those instructions are supposed to do, right? Andrea: Yes. Exactly. Okay. Exactly. Jonathan: Exactly. And so you can implement that. Someone can implement a f- fully to spec RISC-V implementation in an FPGA, in silicon. I'm sure- Yes ... someone has tried to do a minimum version of it in vacuum tubes or, people do crazy stuff just because they can. So there's a bunch of RISC-V all around us. Andrea: Yes. And even more on our website. We offer free training courses. One of them is one of my favorites. I took it during my interview process actually to get the job. And the train ... the course was Design your RISC-V CPU core, and it guides you from reading the ISA specifications- understanding them, designing the instruction decoder, designing the register file, and an example of a very simple single stage, single clock cycle e- execution unit. And you do everything in a browser. You get all the environment the system Verilog, the compiler, the emulators, and you can even run some simple assembly software to try it out. A- and this is a course that I recommend to all the hardware and software engineers, both. Jonathan: Yeah, that's very cool. I went through... In fact, we've interviewed him on the show. There's a similar idea called NAND to Tetris, and they start with the NAND gate, and it's "Okay, here's how you put two NAND gates together to get a NOT gate," and, all of those different things. Yes. And it gets all the way up to now program your Tetris game on your operating system that runs on the CPU that you built. Excellent. I don't remember if that uses the RISC-V ISA or not, it, but it's gonna be similar. I love those sorts of things, though, because you s- you start with the very basics and you build each layer up- Yes and so you get to the point to where you understand the whole stack. Andrea: Yes. W- And you appreciate the simplicity. I've, I worked on many different ISAs over, over the years, from CISC to RISC many different ones, and I really enjoyed looking at the hardware side of RISC-V, how simple it is from the op codes, the instructions, the bit fields, to write the instruction decoder. It's class. It's really classy. I think Krste Zaninović, Andrew Waterman, Yoon Soplee, Dave Patterson, they did something amazing with this RISC-V ISA. Jonathan: So it's RISC-V, which brings the question to mind, was there a RISC-IV? Andrea: Yes. Yes. Actually it all started with a RISC-I in the early '80s. Actually in the early '80s Dave Patterson wrote a paper making the case for RISC, reduced instruction set computing, and then the RISC-I was the very first chip as the very first proof of concept. And then there were a few other iterations done at UC Berkeley under the direction of Dave Patterson. And in 2010 the work started on the fifth generation, so the V in RISC-V, the V sta- is the Roman number for fifth generation. But we also joke that is also reminds V for vectors, because RISC-V has been designed since the beginning to have the most flexible vector extension design. Jonathan: Interesting. That and that's part of what RVA23 is about, isn't it? Is additional support for doing vector instructions? Andrea: RVA23 is a profile specification. Let's see. RISC-V is very modular. You have many extensions that add small features or security or performance, and they're all modular, and you can It's like ingredients in a recipe. And you can build your RISC-V microcontroller that is perfectly handcrafted for your specific workload or application or specific function. But then, yes, everyone may have a different recipe. RVA23 standardizes the non-differentiating bits- ... and sets the bar high for a high-performing application processor. So when you are when you own your design, you want to optimize it at best for your specific application that you fully own, hardware and software. Then RISC-V gives you this flexibility, and you can add your specific custom instructions for that specific function, the, and you will get the most optimized design. But if you want to run a standard operating system, like Linux or an RTOS, you are not going to port it Once for every single vendor. That would not be sustainable. That would lead to unnecessary fragmentation. And this is what RVA23 solves. RVA23 defines the minimum set of extensions that shall all be present as mandatory, as regional options, as growth options- that shall all be present in a design, in an RVA23 application processor. And this means that the operating system vendors can target RVA23 at the single build target, and you will get binary application compatibility. A- and this has been ... This has worked really well because Canonical immediately endorsed it. And Ubuntu- I was Jonathan: at, I was at the Ubuntu Summit at, the ... Where they, where they d- released that. They made that announcement, and then they- Yeah ... they had a f- ... Was it Space MIT? One of the, one of the companies had- Yes ... some hardware there. Yes. And I was like, "Ooh- Yes ... I can take this home with me, right?" And they're like, "No, you don't get to take that home." Andrea: No, but you will get access very soon- Jonathan: Yes ... to that one. Yes. Yes. I immediately sent- So it- I immediately sent a message to one of my developers. I'm like, "Hey, send an email to these guys and ask them for hardware." Yes. Andrea: No, th- that's exactly the point, and Ubuntu now runs on every RVA23 compliant design. And similarly, Nvidia disclosed ... Last year, they disclosed plans to port CUDA to RVA23. So you do the port once, you do the work once- ... and it works on all the RVA23 compliant application processors. Jonathan: Yeah, absolutely. What what are some of the challenges? You mentioned fragmentation, and that was something I was going to ask about. But i- and we can d- dive a little bit more into that if you want to, but I'm curious if there are some other challenges right now in the RISC-V ecosystem doing d- doing this development or even trying to work with the other companies out there. Andrea: I would say that we are working in expanding RISC-V from a horizontal technology. In deeply embedded microcontroller or security processors or AI specific schedulers, for example. We are expanding RISC-V into multiple different vertical industry verticals and so we, we introduced the concept of the special interest groups at RISC-V. So we started with the data center, with the automotive special interest group. End of 2025, we s- we launched a space interest group. The concept is that you invite the domain specific experts who come in with the domain specific requirements, and maybe they are new to RISC-V, so they need help. I remember in the early days two years ago, the questions was, for example if we want to design an ECU based on RISC-V, how do we run AUTOSAR on RISC-V? How do we achieve the right physical memory partitioning or protection? How do we run a trusted execution environment on RISC-V? And then we bring in the technical experts, and it's like matchmaking. And by mapping the requirements to the RISC-V technologies, then we come up with a gap analysis. We may identify low-hanging fruits for innovation. And we also write application notes that they explain how to configure RISC-V at best for each each vertical. So in the automotive space we're working on, first of all, functional safety. We have a white paper with a functional safety special interest group and experts in functional safety. Where we find that there are no gaps. And then in automotive in addition, how to configure RISC-V for an ECU microcontroller, or for a zonal controller, or for the central compute in automotive. Similarly in space I find it quite interesting and amusing. For me, SaaS means Software as a Service. But in the space world, for for our friends and colleagues and members i- working in space, SaaS means Satellite as a Service. Y- cloud in space. And you lease a an instance, and you run your workload on a satellite. In that case, you have requirements of software isolation, security, redundancy- Reliability, quality of service and you need to map them all. And the flexibility of RISC-V means that collectively, all together, we, we can ... Maybe we will do another RVA23 specific for space or for automotive or for Edge AI Jonathan: That, Andrea: that- Like specific configurations- ... for the various markets. Jonathan: That, that is actually a really interesting idea to try to tame some of that fragmentation, is to have y- a- additional specifications that say, "Okay, look- Andrea: Yeah ... Jonathan: to be able to be able to s- you know, for us to put our stamp of approval, RISC-V International says that this design is ready for space, you've gotta have this." And then, the next sort of logical connection on top of that is somebody like Canonical or whoever wants to says- ... " Jonathan: We can build a Linux image." We are going to port- we're gonna- Yes, exactly ... build an image for that. And so here is the- Yes ... Ubuntu space edition with the stamp- Exactly of approval on it. Yeah that's that's a fascinating idea. I look forward to that happening. Y- Andrea: The if I use I very much like the term by ETH in Zurich, the academia. And Professor Luca Benini. He gave an a very fascinating keynote at the last RISC-V Summit just a couple of weeks ago. His theme is focusing on domain-specific accelerators, DSA. I like the term domain-specific accelerators, and it means that they are building extensions or accelerators for specific workloads in specific domains. So in that case, i- it's not unness- unnecessary fragmentation. In that case, it is a real innovation. When you want to run standard operating system for a standard application in standard market, then you need the profile specifications like RVA23, and even more, we recently ratified the server platform. So going beyond. If I look at the hierarchy, you have the base ISA, you have all the extensions that are modular. Then you combine an, a subset of the extensions for an application processor profile like RVA23, or we're working on an RVM profile for microcontrollers. And then the next step in the hierarchy, which is a requirement from our members, for example, in the data center space our members realize that there are so many peripherals that are close to the CPU core. Timer, IOMMU the message system interrupts for the PCIe bus, all these core peripherals close to the core They need not be different. They shall be controlled, configured, accessed the same way. Then you can compete on the implementation, of course. Best performance better power management, et cetera, latency. That's the implementation side, but they all concur that they need the same specifications, not only for the ISA, but also for these core, core close peripherals. And so the server platform that we recently announced is based on RVA23- ... which means mandating support for hypervisor vectors and a few other critical extensions. And then the server platform adds specifications for ACPI tables, specifications for these peripherals that are non-differentiating, and then specifies the standard boot architecture for RISC-V. And all this leads to the data center distributions to run seamlessly on, on the RISC-V server platform chips. Y- Jonathan: you knew exactly where I was gonna go next. I was gonna ask about how you boot. I have... So I love embedded devices. I've got, probably 25 Raspberry Pis around the house, and I'm a sucker for buying new things. It's oh, it's an, it's a RISC-V board that you can run something on. Oh, sure, I'd love to having a RISC-V board I could run Linux on. Or, it's a, it's an Arm board that you can run Linux on. Oh, sure, that's different. I don't have one of those yet. And- ... the amount of time that I have spent fighting with trying to get those to boot, and going in and fiddling with the Linux kernel and, the the binary tables and all of that to try to get all the hardware to come up is just... And so when Arm had their system ServerReady, they called it first, and then SystemReady, and I think they used UEFI in that, Yes ... and things started to just work. It's oh, that... It's so nice. And, Yes ... so I was gonna ask, like on, on these s- server RISC-V boards, what does that look like? How do we just get something to boot without having to have a device specific image to flash on there? Andrea: Yes. That is exactly going back to a- adopting industry standards And at the same time, industry standards adding support for RISC-V. So the server platform specifies exactly the standard boot requirement architecture and boot requirement specifications. And it's, it leverages UEFI ACPI for servers. And this was possible because the latest ACPI specifications added support, official support for RISC-V ACPI tables. So y- you see it's coming from both ways meeting each other. Jonathan: Yeah, absolutely. I- is there some cross-pollination with the things that Arm is doing? I guess I should ask this first do you guys consider Arm to be one of your big competitors? It seems like that might be the case, but maybe not. Andrea: We are friendly to everyone. I think the market is growing so fast that there's room for everyone. And we are friendly and respectful to everyone as much as to all the regulations from every geography worldwide. I think that a healthy competition is is a great asset for everyone for the industry. Jonathan: A- and then is there with things like the, a- again, I mentioned Arm's ServerReady, and it sounds similar to the RISC-V project. Has there been some cross-pollination there? Have some of those additions to ACPI been helpful to, to, to both of those pro- projects? Andrea: I think we are all adopting the same industry standards coming from the UEFI Forum. So again- ... it's a great value of the open standards, 'cause you can you can download and study the ACPI specifications, the UEFI specifications and adopt them. And of course, those specifications, they receive feedback from all the implementations. So they receive also feedback from all the ISAs. So there's cross-pollination between the industry standards. Absolutely, yes. Yeah. Jonathan: One, one thing that I've seen that I find real fascinating is the ability of the RISC-V, the core ISA, to scale both up and down. And so we have essentially microcontrollers that are RISC-V all the way up to, now with, specifically with RV823, you've got eight core chips that are r- eight RISC-V cores in there, and you could legitimately run it as your primary computer. It might not be the most fun experience yet, but you could do it. It- Oh, it's coming ... it's, yeah. Yeah. And we're rapidly approaching that as a, a more reasonable thing to do. What's the secret sauce? What have you seen with the ability of RISC-V to scale up and down like that? How is that possible? Andrea: Because it's a single ISA and it's modular. If you, if we go back to the example of automotive. You have an ECU the elec- electric control unit. That must be a very tiny microcontroller, real-time, low power. Then you have the zonal controller where you may need some AI capabilities. And you have the central compute. You have AI extensions. You have GPUs. You have NPUs. So- Outside RISC-V, you have many different ISAs. You have a different 32-bit RISC ISA for the CPU, and a different 64-bit ISA for the central compute. And a different ISA maybe based on RISC-V for the GPU or the NPU. So you have many different ISAs RISC-V is a single one. The same single ISA, the same single instruction set can be used across all. You can configure RISC-V to be a 32-bit microcontroller for the ECU in a car- ... and focus on the real time extensions. It's the same ISA that you can configure to be 64 bit, but it's still the same ISA, the same instruction. It doesn't change- ... for the zonal controller, and you can add vector extensions, and in the future, matrix extensions to have AI inference inside the zonal controller pre- preprocessing data coming from the sensors. And the same ISA without change can be used in the central compute. Then you would add extensions like hypervisor support. So the same ISA, the same programming model, the same tooling can... The same expertise can be used across all. Jonathan: Yeah. That's... it's super cool to see. And so you see as a result of that, RISC-V pop up in maybe some unexpected places. I'm pretty sure Intel runs r- a little tiny RISC-V core as their a- as their sort of s- root of trust in most Intel processors. And I think I've read about hard drives that have RISC-V process. It's, we- we're part of Hackaday now, and so I remember there was a there was a particularly interesting story where someone says, "I installed Linux on my hard drive." He's no. I don't mean I wrote Linux to the hard drive. I meant, I hacked into it and I'm now running Linux on the RISC-V processor in my hard drive." Oh. Andrea: I love this. Thank you for sharing it. Jonathan: I'll see if I can dig that link up for you- Andrea: Yeah. Yes, please ... ' Jonathan: cause that one was particularly fun. Andrea: Yes, please. But- Yeah, RISC-V is popping up everywhere. As you say, it started in the deeply embedded microcontrollers from Western Digital from the early 2015 when the RISC-V Foundation was re- i- initially created. And Nvidia was also one of the first to adopt RISC-V in those years. And Nvidia has been shipping RISC-V in every single GPU between 10 and 40 RISC-V cores in a single GPU. In 2024 Nvidia were our guest speakers at the RISC-V Summit, and they publicly explained, and that's a very interesting keynote. They explained how they use RISC-V in many different ways inside a GPU for many different functions, and you can find that on YouTube, or we can provide the link later for- for your for your guest. And they evaluated that just in 2024, Nvidia shipped more than one billion RISC-V cores in the GPUs. Qualcomm in 2022 estimated that they had shipped about 600 million cores in wearables, and just a few weeks ago they updated the they've estimate to two billion cores in wearables. Infineon one year ago announced that their entire automotive microcontroller roadmap is based on RISC-V. Meta is using RISC-V in their Meta training and inference accelerator, the MTIA chip. Google is using RISC-V in the Coral NPU, which was also released as an open source project- ... last year. And then you have AI accelerators with hundreds of cores by Tenstorrent, Inspire Semi. You have the latest NPU by MIPS, all using RISC-V. And then we have the application processors coming this year Jonathan: There are d- do you think we've passed the point to where there are more RISC-V cores than there are people? This is my metric for when talking to a project that is really popular, and so I asked this, I asked Daniel Stenberg this, the guy behind Curl. I'm like, "Are there more Curl installs than there are people?" He's "Oh, definitely." Andrea: The SHD Group estimates that RISC-V will pass the 35 billion SOCs in 2031. And every SOC can have more, multiple RISC-V cores, multiple. Just as I said, the Nvidia GPU has between 10 and 40. So I think that we by large, we passed the number of humans on Earth already. Jonathan: That's, that is so wild to me. And of course with the with the AI boom and just the way all of that is growing I don't think any of this is slowing down. I think we're just continuing to accelerate. And you have you have serious contenders talking about, putting cores up in satellites and doing data centers up above us, and- Yes ... it's just it is incredible to watch. It, we are in a very interesting transformational moment in in human history right now. Andrea: Yeah. And you were also hinting at using it as a main computer- ... Andrea: Right? And yeah I'm happy to touch on that as well. Thanks to Deep Computing we have had RISC-V based laptops for the last few years. With the DC Roma laptop, the first generation, the second generation, and then Yuning from Deep Computing partnered with Framework the modular laptop company. Yes. The latest RISC-V based Framework m- motherboard, based on the SpaceMID K3- ... you sh- you should try it. Believe me, you should try it. You will get 4K YouTube video playback, very fast browsing office applications. So it's still a machine targeting developers in terms of price tag. It's m- but you should try it. We were really planning to purchase as soon as it's available in quantity, purchase a few for the RISC-V staff, and use it- ... at events, and eating our own dog food. Jonathan: Absolutely. I remember I reviewed, this has been a couple of years ago now, it was, Vision Five, I think Yes. I r- I reviewed that, and I set it up with a monitor and I s- I tried to use it as a desktop for just a little while, and that was not a great experience as a desktop user. But in my review, I made the statement that it's just good enough for developers to start actually doing development- and compiling their stuff on it. And so it's not gonna be great for running as your desktop, but it is gonna be transformational for the RISC-V ecosystem as a whole. And I think we're now seeing some of the results- Yes ... of that kind of work with, you know- Andrea: Yes ... Jonathan: players Andrea: Ubentu- We don't- ... and, Jonathan: and all Andrea: of that. Yes. And the latest RVA23 chips are bringing that level of performance. The SpaceMeet K3 is just the first one that has very good performance. It also running local LLM models. But there are so many other RVA23 silicon that, that are becoming available. This is the year of the RISC-V application processor, really. Jonathan: Maybe not the year of RISC-V desktops and laptops for everyone quite yet, but it ... do you ... W- when you allow yourself to dream, do you think about that one of these days Intel and AMD is gonna have to compete with RISC-V? Is that a future that you see? Andrea: Our goal is to ensure that RISC-V is the first or default choice for new designs Jonathan: That's fair Andrea: the default ISA for new designs. Jonathan: And that, that encompasses a lot. That is actually quite the quite the aspiration. Andrea: Yes. Jonathan: But on the other hand, so many of these devices have a RISC-V core in there. I suppose one might say that you're pretty close to that. Andrea: Thank you. Jonathan: That's fun. So what about ... And I touched on this briefly, the idea of developers getting ahold of RISC-V. Andrea: Yes. Jonathan: What does the sort of the ecosystem look like for the tool chain, the compiler, the developers, and I'm sure this is something that you guys have worked on quite a bit, is to- Yes to grow this ecosystem in the community. What does that look like? Andrea: Yes. We are focusing on the developers from multiple angles. On one side, we have been purchasing and shipping the latest development boards over the last few years. In 2024, we shipped more than 200 boards to key developers, including tool chain operating systems. In 2026 in collaboration with RISE, the RISC-V Software Ecosystem Project, we are setting up a build farm, a RISC-V based build farm with native GitHub runners in collaboration with the open source lab at the Oregon State University. We also have the first RISC-V based commercial cloud service provided by Scaleway in Europe so that's one, one axe is, one axis is hardware availability. The second one, as you mentioned, is the tool chain work. So we we are in close collaboration with the tool chain projects not only GCC and LLVM but of course, we also have many members who provide their own tool chains like Green Hills, IAR, TASKING, embikasm, they're all working on tool chains. So the point is that, for example, when we release the RVA... We ratified the RVA23 profile specification. Patches to add support for RVA23 in GCC and LLVM were already being reviewed by the maintainers. And then Rise is collecting funds to direct them to optimization work. So Rise has been funding work to optimize support for Python, LLVM, Go, and also projects like PyTorch and Iri or Llama CPP. We have been investing together with Rise to add RISC-V as an official ISA in the Yocto Project. In just f- we, we upgraded our membership to the top tier the platinum level in May 2025, and just five months later, RISC-V was the the third officially supported ISA. Today, Yocto supports x86 ARM, and RISC-V- as the three official architectures. So it means that for the developers, Yocto is not only an embedded Linux and a set of tools to optimize and build your embedded Linux distribution, but it also means that you have an LTS support for the file system that is using the kernel LTS, and most importantly, it's all using upstream Linux support. And this was possible thanks to the collaboration RISC-V International and Rise together. And it's a major achievement for us. And the third axe, so I mentioned hardware availability, the operating system, and tool chains, and the third one is the trainings and the developer experience. So continuously adding new training courses to our free catalog on our website. I mentioned the designer RISC-V CPU core, my favorite one, but we continuously add new trainings for real-time operating systems, for porting software from other ISA to RISC-V, or optimizing software for the RISC-V vector extensions Jonathan: Yeah, interesting. The the training that you guys provide, is that freely accessible? Can someone just- Andrea: Yes ... Jonathan: jump on the- Andrea: Just go on the riscv.org or RISC-V.org as you would spell it or type it. And then you have the community, and you have the resources and the full catalog over free online trainings. Jonathan: Wow. It sounds like that would actually be a, if someone was, say, in high school or starting college and really wants to break into the sort of MCU embedded environment- Yes that would be a really good place to get started. Yes. Put that on your resume that you've worked with RISC-V. Andrea: And we... we also have the mentorship program every year where we select multiple students and mentor them on… And they deliver real results. They do real work. Jonathan: Yeah, absolutely. What sorts of what sorts of work, what, where do they get em- embedded at when, in that internship program? What kind of work can someone expect to do? Andrea: It varies a lot. Some worked on enabling kernel CI on RISC-V and kernel testing. Others are working on tools that automate documentation and consolidate the, or get better organize the documentation for the various extensions, and generate programmers reference manual and so on. So it varies a lot. Jonathan: Yeah. Interesting. What what are some weird places or weird things that people have done with RISC-V that, that you're aware of? I feel like everyone that's in the tech industry has a story of some sort. I built this tool and someone used it for something completely different than what I intended it for. There, there's gotta be some oddball RISC-V use cases out there. Andrea: Oh, we in one of the last summits, we were offered one of those interactive teddy bears. That runs on RISC-V. That's pretty good. Yeah. That's pretty good. And more is the professional one is the humanoid that ran the marathon in China- Okay ... a couple of months ago. That was running on the Space Mid K3 chip. Oh, that's cool. RISC-V in space. RISC-V was on one of the last moon lunar lunar landers. Okay. And then today I was reading somebody designed a GPU using 8,000 RISC-V microcontrollers interconnected. Jonathan: Really? Andrea: Yes. Jonathan: That's impressive. Massively parallel indeed. Andrea: Oh, yes. Jonathan: A- and so l- what what do you see as you rub your crystal ball and think about what comes next for for RISC-V? What are some things that you're excited about for the future? What do you see coming down the pike? Andrea: In short term, all the RBA 23 application processors. SiFive, Andes, Tenstorrent, Akeana, Alibaba BOSC, Nuclei, Spacemi. There are so many that are coming up. At NextSilicon EPiC SEMI. I think at the last RISC-V Summit a few weeks ago, we announced more than 10 RISC-V application processors. Many of them have server-grade performance. So in short term, this is super exciting. Then ratification of the matrix extensions is something that is very important. The completing the work with ISO IEC I really want to get through and make sure that RISC-V, the RISC-V ISO manuals can be properly evaluated as a possible ISO standard. And the other thing I really want to continue investing in the idea of the developer workshops. W- this is something we started at the last RISC-V Summit North America in October twen- October last year, and then Summit Europe a few weeks ago. So at every RISC-V Summit, we're now dedicating one full day to hands-on labs. And just a couple of weeks ago the fee for this lab, the cost was, the price was 30 euro. So it's really just to ensure no-show, that's all. And we had a hardware track, a software track. We had labs running an open source RISC-V design where the authors broke the system very long. So while running some assembly test, suddenly you got an exception. And the developers had to find where in the system very long it was broken. That's alwa- Can you imagine that? That's always fun, to break your tooling. Can you imagine that? Yes. Lauterbach had something similar. They enabled 24 debug sessions, debug tools in parallel, independent. So we had multiple developers on each station, so 100 people. And Lauterbach demonstrated how you can do advanced trace and debugging on RISC-V. And then they ran a challenge. There was a software running with some memory leakage or some bug in it. And these 24 teams had to find it and fix it as quickly as possible. We had workshops on running ExecuTorch or Edge Impulse AI inference on RISC-V or running trusted execu- execution environments and mitigating risks from the threat models. So all these were hands-on labs. And this is something that we will continue investing, and just sharing ideas to organize the next workshop is super exciting. It's so much fun. So much fun when you see what can be done. Jonathan: Yeah. Have ... Some of my listeners hate it when I ask about this, but it is the world we live in. Ha- have the flavor and the way those workshops work, has that changed radically because of LLMs and AI in the last, say, six months to a year? Andrea: No. No, not yet. Oh. Not yet. Jonathan: Interesting. Andrea: Maybe in the future. Not yet. Jonathan: Oh. Andrea: Oh, one thing, yes. Do you- I think it may be used to generate more design verification tests from our members. But for these workshops we are really running them with a human brain. We want to challenge the developers. That's why they are joining these workshops. They want to learn hands-on. They don't want to just prompt the AI to do it for them. Correct. And I think it's super exciting. At times I would like to go back being a developer and being part of these workshops. Jonathan: Yeah. That, there, there's only so many things that we get to travel around and go to, and yeah. It's one of the downsides of being the CEO. You don't get to, you don't get to be as hands-on with some of the things that you really enjoy. I get that. I too am CEO of a company d- a small company, but still I have to balance those things too. I'm lucky that I get to write some code still, but I see a, maybe a time coming before too much longer that'll come to an end, and- Andrea: A- and that's why I enjoyed so much the interview process with Calista two years ago, when she interviewed me, and then the board interviewed me. And the founders of RISC-V. 'Cause that gave me the opportunity of studying all the ISA manuals, and even I had fun in taking some code in C and write my own RISC-V assembly. And I wa- I was good at arm assembly in the past. So I took this C code, I wrote it in arm assembly, then in RISC-V assembly, and I appreciated the differences. And then I took that course on designing a RISC-V core from the website. So both the software side of a RISC-V ISA and the hardware side, and I really enjoyed that, You- ... that interview process. Jonathan: You really understand RISC-V well. I don't know that I've ever talked to a CEO, somebody at the head of a company, that has the technical understanding of what he's working with that you do. That is quite impressive. Andrea: I better understand the rest then. Jonathan: Yeah, sometimes that is the challenge. All so I... Boy we've gone through a lot. Is there anything that we didn't cover that we should have? Is there anything that I didn't ask you about that you wanted to make sure and mention to folks? Andrea: The developer workshops and the laptop were really the things that I thought were maybe not on, on your mind, so I mentioned them. Jonathan: I must say, I have the I have the Deep Computing store page pulled up. Andrea: Oh, okay. Jonathan: I have Framework laptops. I really like the Framework laptops. And yes this particular main board looks very interesting. Don't tell my wife, I may be about to spend more money. The really funny thing about that is that she watches the show, so she's probably out there, muttering at me. Oh, so she knows already. She already knows. She's checking Andrea: your credit card statement now. Jonathan: Yes, of course. Don't look too close. So what, W- y- RISC-V is going to space. Do you foresee a big uptick there? I, so I went and looked f- for one thing. Yeah. I was interested. The the Ingenuity rover the helicopter running Linux on Mars. I, I thought "Was that RISC-V?" No, that one's running ARM. But, Yes Maybe the next helicopter will be RISC-V, right? Andrea: Oh, RISC-V was in the lunar lander last year And the interesting part is I attended a RISC-V in Space workshop organized by the European Space Agency and Geissler in 2025 last year. There were maybe 200 speakers in the room, 200 projects based on RISC-V. It was unbelievable. And we had another RISC-V workshop at the last summit just a few weeks ago. And we had representatives from the European Space Agency, from NASA, from Microchip, from Geissler E4 Computing, and they were all explaining why they need RISC-V, why they need an open standard, why Geissler moved from Spark to RISC-V now. Why they chose Spark in the first place, and now RISC-V. It's, it is so fascinating. Yeah. So fascinating. Jonathan: I I did have a thought earlier, and it just comes back to me. Has RISC-V had any problems with things like export controls? Andrea: No. Jonathan: I- is this because you're intentionally a European company, or is it because- Andrea: No everything is open? No. Jonathan: Or how have you- Because we- How have you avoided that? Andrea: Because we are extremely scrupulous and respectful of all the regulations worldwide. And because we are a standard. We are an industry standard. We do not produce any implementation, any reference implementation, any artifact that would be a technology that gets exported. We don't do that. We just do PDF. Remember? Remember? We just do PDF, like USB. Jonathan: I, I think this is going to be the show title, We Do PDF. We Andrea: Do PDF. Why not? No I'm s- I'm serious. We are laughing together, but I'm serious. Doing in- international open standards means that it's a standard. And the implementation, the technology gets developed locally where the members are. You can design a USB mouse or USB pen drive in Silicon Valley or in India or in Europe or in Brazil. And that will be a local design. But the standard is global, and the same is true for RISC-V. Yeah. No, that certainly makes sense. That- that's why this recognition from ISO/IEC is so important to us, that we got accepted as a PAS submitter, as a standardization body. It's very important that we are officially recognized as a standardization body. Jonathan: Have, has there been any has there been any patent problems with RISC-V over the years? Andrea: No, we don't do any patents. Jonathan: But I g- I guess the way that I would imagine that working is, you get an email from, and I'm just ... I'll pick one at random. You get an email from Spark, says, "Oh your ISA violates one of our software patents." Andrea: No. Jonathan: Nothing like that? Oh, lucky you guys. I think it probably helps that it's based on something that came out of academia back in the '80s. And so a lot of those things that someone would say, you could just point at and look, this guy came up with this in 1982. The patent on that's already expired." Yes. "Go away." Andrea: Actually, we are the fifth generation of the original one- That inspired all the other RISC-V designs. Jonathan: Yeah. That helps. That helps a lot, being, being the original. We did it first. Oh, that's great. All right. I do have a couple of final questions that I'm required to ask everybody, and I get in trouble if I don't. I get emails if I forget these. You personally, what's your favorite text editor and scripting language? Andrea: My favorite text editor was Zap Jonathan: Okay Andrea: That was a text editor that was, I was using when I was programming on an Acorn Archimedes. That was one of the first machines based on the ARM3 processor- ... and the RISC OS operating system. And at that time, I used the Zap editor. I loved it. I loved it. Jonathan: That's a throwback. Andrea: Yes. What do Jonathan: you use these days? Surely you don't run Zap in an emulator. Andrea: No. No, these days I just use a simple text editor available on on my laptop. Jonathan: Wh- which one, though? Do you go for VI or Nano or ... Andrea: ... w- Ah, let me check. No, the name is just Text Editor. Yes, Text Editor. Jonathan: Okay. Andrea: As simple as it is. Jonathan: As simple as it gets. Yes. A- and scripting language, if you have to, if you have to write some script code? Andrea: I was using the ... The one I used the most in the early days was the the MS-DOS Batch Jonathan: Yep, yep. I've have some experience with that from a long time ago. Andrea: I'm sorry i- I have some gray hair. Jonathan: Oh, don't look too close at me. I, I've- I- ... I am gaining some as well. I- It Andrea: may not... it may not sound that fascinating, Zap or Batch. Jonathan: I got my start programming with Microsoft QBasic. I'm old enough to say that. But I've have not done anything with the Acorn, with the Archimedes and or any of that, so I fit somewhere in there. Okay. All right. Andrea, it has been a blast to get to talk to you. Thank you so much for coming, and, Oh, my pleasure ... I'm glad we got to- Thank you so much Andrea: for inviting me. Thank Jonathan: you so much. Yeah, glad we got to talk about it. Really enjoyed it. Yeah, I did as well. Thank you so much, sir. Andrea: Thank you. Jonathan: All right. That was Andrea Gallo talking about RISC-V and RISC-V International, and boy, had a great conversation there. We've got some fun shows coming up. Next week we're talking with Neriman Jelva about Pewter. That is the desktop operating system that runs in your browser. I have questions about that. And then the week after that we're talking again with Michael Meeks of Collabora. And then the week after that we have a returning guest, Francois Proulx, about Smoked Meat, and that's all about security in your CI, like in places like GitHub. Very important. We do have some openings after that, so if you or your project you think should be on the show, you should let us know. Drop me an email. It's [email protected], and we'll get you scheduled up. Other than that, just wanna say thank you to everybody that's here, whether you watch or listen live or on the download, and we'll be back next week on Floss Weekly.
• 
  Episode 874 - Really, We Do PDFs 08.07.2026 1t

  This week Jonathan chats with Andrea Gallo about RISC-V! What does it mean for RISC-V to be an Open ISA? Where is RISC-V popping up, and what's the new frontier?  Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 873 transcript 01.07.2026

  FLOSS-873 Jonathan: This week, Aaron and Andy join me to talk about QNX. It's the other operating system that runs in a whole bunch of different places. It really has safety and real-time nailed down. There's just one little problem. This is FLOSS Weekly, episode 873, recorded Tuesday, June the 30th. Wait, that's not open source It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we're talking about a Unix or a Unix derivative or a Unix-like. I'm not sure which of those terms is technically correct. We're talking about QNX. It runs a whole lot of places. It used to run on your BlackBerry, and it runs a lot of places that you don't think about. Maybe not on your toaster, but maybe on your car. Anyway, we've got we've got Andy Green and we've got Aaron Bassett to talk about QNX, where the company is, where they're going, what changes are there, and why, maybe you should think about doing some development on it. Let's go ahead and bring them on. Guys, welcome. Welcome to the show. It is great to have you both here. Andy: Thanks so much. Really appreciate it. Jonathan: Thanks for having us. Yeah. And now, we gotta start with this. We have to start with this, right? So this is the show about open source, Floss Weekly, free, libre, open source software. We're not... We don't do dentistry here. We do open source stuff. QNX, at least like the core of it, the kernel of it it's closed source. And what are we even doing here, guys? Get get out. What's Andy: the conversation? Yeah. Why are we on this show? Jonathan: Why does this Andy: make sense? Yeah, no that, that's very fair. So I, we have had a kind of a really interesting and long evolving story with open source, and I think for some of your audience who may remember, we did go fully open source at one point in our history. And and that was like right around the time when we were acquired by Research In Motion for use in the BlackBerry, and they were thinking, "Hey this stuff is special sauce. We really don't want this to be, like-" "... our... one of our defining characteristics. We don't want to be fully out there because we wanna actually have some of that proprietary goodness for ourselves, for our own phones." So we clawed that back. And so that that is something that I think, a lot of us in QNX were not necessarily... we understood the business rationale for that and all that. But I think that there was always this feeling that, there's a lot of good and value that we get with working in the open source community and with the open source community, and we wanted to bring some of that back. And so I think relatively recently we've had this, So I guess QNX Everywhere has been a program that, that both Aaron and I are part of both sides of the, on the technical side and on the s- sort of more on the business side. And that's been around for about a year and a half. And that's been our start back to recognizing, l- look we need to be more involved with the community. We need to be more transparent. We need to open ourselves up a little bit more. So I'm not gonna make a commitment and say, "Hey, yeah, we're going all open source," or anything like that. We're not gonna, make that same mistake of making commitments and then pulling back on them. But what we are doing is we are trying to open up a lot of the pieces that we see don't really have impact on our business. Because I'll be frank, there's a few different things that are factors as to where- whether or not we do things open source or not. And one of the big ones is that a majority of our business is based on regulated industries. Things that either have functional safety components or cybersecurity components or things like that. And in a lot of cases there's this thing called SOUP, which is software of unknown providence that is an acronym that's used within those sort of, certification bodies or regulatory communities. And having open source software in a lot of cases makes things extremely difficult to pass through certification. Not because of the testing or anything like that, it's because you can't really guarantee where that software has come from, or you have a very difficult time tracing the line through of all the edits that have been made to it- from different contributing parties. So we have to be super careful and super cautious about things that are gonna impact our business. So we don't we're not gonna go and say, "Oh, yeah, hey, you're, we're gonna open source the kernel," because then anybody can contribute to it, and then we turn around and find out, oh, somebody put this really super clever hack in that- wasn't really intended, all that kind of stuff, right? But to that extent, though, we are doing things like, making sure that we're open sourcing all of our board support packages, open sourcing our development, our driver development kits- ... open sourcing pieces of our software stack that actually originally may have come from open source and we've made changes to it or things that we don't think are, impacting, functional safety or any of those kind of things. So keep your eye on that space because that's the one aspect and, but the other aspect is us working with the community because we get a lot of value out of working with open source and making sure that, like open source software runs on our platform or that we can use it or deploy it in, for either hobbyists or academia or, companies that are doing prototyping or proof of concept or R&D or any of that kind of stuff, right? So there's a lot of people that are playing with different open source things, and we wanna make sure that all those people are enabled. So there's like a lot of things that are going on within the company right now around open source, and we think we're good open source supporters, and we do recognize that, yes, we haven't thrown the kimono open for everything, but- but we're doing that in an intelligent way, right? Jonathan: Yeah. I do wanna touch on one, one thing that you mentioned, and this is something that we've seen several projects make moves in this space. But the idea of when you're open source, that means that you accept pull requests. And that is-- we've seen several projects come to the point where they say, "Look, we're open source. All the source is out there, but we do not accept pull requests," or, "We only accept pull requests from people that are already in the project." And, I think AI is leading to some of that. In some cases it's because of just very high code standard quality, code quality standards. I think in other cases, though, it's because of that legal question. And so there's a relatively famous blog post from DHH that talks about this, and it's like open source is a license. It's not a community management style. And I just think that's interesting in this particular context to think about that. And it goes all the way back to this idea of the cathedral and the bazaar, right? From ESR way back in the day. And you can do things that are free software and open and still, the monks in the cathedral are the ones that are slowly working away on it, and it's not just a bazaar where everybody can push code in. Andy: No, I agree. And I, and, and certainly I've seen things where it's yeah, but people are gonna have to stop taking software because of all the AI slop that's being generated and I think, our approach to that might be, like, even more conservative than- ... is strictly necessary. But I think that we're approaching this as a, we, we learned a lesson from what we've done before. We don't wanna actually, go a step far and then take a step back, right? So- ... so we're gonna do that i- in an intelligent way. I, the CEO of our company John Wall, has made a lot of comments about, how he feels that we, we should, open source as much as we humanly can. But there are gonna be, w- we have to still do that with all of the processes that we adhere to, with all of the regulations that we adhere to, all the rest of that stuff. Honestly, yes, we could immediately pivot to saying, "Oh, yeah we're gonna just do an open source methodology and modality to manage all that stuff." But kind of it's, I'll just say it's the path of least resistance to not do that- ... and to just say, "Okay, for those things that actually have critical components that we're j- we're gonna manage them the way that we always have," because we have processes that we know work that have been vetted. We've actually been audited on all that stuff. We don't have to re-audit everything and re- like reinvent the world when it comes to that, Jonathan: yeah. I, th- there's a lot more to talk about with this, but I do wanna bring Aaron in on the conversation. And you are- Absolutely ... you're part of the open source team, Aaron. What, so what are the bits of, in the company that, that you work on specifically? What's the open source story there? Aaron: Yeah, so primarily for myself, I work on the QNX developer desktop. So that is the self-hosted developer system that we've been making to allow Basically anyone who wants to use QNX, whether it's hobbyists, students, to get easier access to QNX. Primarily it's been cross-compiled, embedded system, right? So you've done Arduino development that, you cross-compile your thing, you copy it there, you flash a board, you go there. A lot of people don't m- know how to do that or it's something they've never learned, so it's really hard to get into, especially at hackathons. 48 hours, let's learn how to cross-compile and flash- ... an SD card and everything, right? That's something you just don't do. But part of that is, is I maintain the package manager that we have for that. How many... couple hundred or thousands of packages that we have now that are in there that we've built and distributed across that. So we touch a lot of stuff. Jonathan: Have you guys found that that AI is making a big change with the way that people interact with this whole system? Andy: Yes. Yeah. Yeah. Yeah, so I started this is s- cr- this job like just about a year ago. And when I first started doing it... So I've been in the industry a long time, but I came back to QNX after running my own company for a while. But in any case when I first did it, we were doing, we'd go to a hackathon, we'd be seeing somebody's project, and they'd be like I guess we used AI to do part of this," right? And hackathons now, it's "Yeah, no, we're 100% full in A- AI. We're- we- there's no bones about it. We're not being bashful about it. Like-" "... we just had Cloud Code do the whole damn thing." And it's like in the course of a year, like that's the perception- ... change that we've had, at least, from students, and maybe they're on the leading edge of that kind of thing. Sure. But I don't think they're the only ones. Jonathan: Yeah. So it, this is obvious. This is one of those observations that I made that's obvious in retrospect, and maybe I was late to this game, but open source makes AI work. And if you're if all of the things that you're doing is closed and not documented publicly, then AI is not gonna be able to do much with it. But when you put it all out there- ... on the internet, and it's part of the training data, then AI knows exactly how to work with your stuff. I- is that s- s- part of the consideration that QNX makes with, like, how much of this to publish and how much to make open? Aaron: Andy, I have an interesting observation on that. So- Andy: Yeah, Aaron: please go for Andy: it. Aaron: Yeah, so I have been playing on on my own time, on my own hardware what does AI on QNX look like? 'Cause, again, you go to these hackathons, and we actually were just at an AI hackathon- ... at Berkeley, and very surprisingly, there's next to no code that knows how to write a QNX Resource Manager, which is how you do devices and drivers. And I was able to, in an afternoon, ask it, "Hey, let's make a temp- like a memory file system," and it spit out some of like perfect QNX-isms, I'll call them. It knew exactly APIs I was able to use, how to do it, found the stuff that you're missing. But I was like what, like where did it learn this all?" There's no code out there that it's open source, like lots of it that doesn't do this. And it may just, it's just all from our documentation. We have really detailed technical documentation that's all public and free. And just by scraping that, it's actually able to do- to do things like that. I saw a lot of that at the AI hackathon. It was able to spit out things that looked like a professional QX, someone who's in QX development for years would write. Andy: Yeah. Yeah. And actually that so that's a really good point and I'll come back to, I think, your opening comment, Jonathan, that led us into this, which was I don't know if it's Unix or this or that or whatever." Yeah. And technically we're none of those things, but we are POSIX. So we're POSIX compliant- ... meaning that we use the same APIs essentially that Unix and Linux do for the most part. There's a little bit of stuff at the f- three signal layer that we differ for and Linux does and Unix does. But for the most part, that means that a lot of stuff runs very easily or without, minor stuff. And when you start playing with the hardware or dealing with, some of the minutiae of device drivers and stuff like that, yes, then you can start running into some things where we're different because of the nature of a microkernel or because of the nature of real time or whatever it is. But a lot of stuff will just work out of the box, and that's one of the really interesting things that, that we've been, like leveraging in terms of a lot of work that, that Aaron and team have been doing to, create all those things that are open source software but that runs on QNX. Jonathan: What, what does that process look like? So if someone has, let's just say, an open source project oh, I don't even know what a good one would be. Nano. All right? Text editor Nano. They, I wanna compile Nano for this QNX board. What does that look like? Can we use GCC? Does CMake work under QNX? How alien is the landscape for someone that's used to like Linux tools? Aaron: I am confident enough to say that if I gave you an SSH connection to my RPi that has QNX on it, if you didn't run uname, you probably would not be able to tell the difference. I'll get rid of the uname thing, 'cause you're probably like, "Oh, uname, hey, oh, it's a QNX system." It looks and feels like a Unix system. I can say the same thing about FreeBSD nowadays. You do SSH to FreeBSD, it, it feels and looks like everything you're used to. And that's one of the things that we really strive to with QNX Everywhere, is make the development system as close to as possible as FreeBSD or Linux, right? So for QNX Everywhere, we have the, we use the Clang compiler for self-hosted development. Obviously for cross compilation, QNX uses GCC with our QCC wrapper, which is, has all the safety stuff for it. But yeah, you just have, you have Clang, you can run Make. We have Mason. Actually just recently, about a couple months ago, you can run oh, what's the Java one called? Bazel. Oh. You can run Bazel on it, right? So we can... You can run everything on it. It's very familiar. Yeah. So- So but go back to your thing for Nano though. Nine times... you grab Git, you do your Git clone or grab it for release tarball. Start off with your standard configuration just like how you'd do it on Linux, and then start following the errors. Most of the time it's just telling it, "Hey, QNX, we are a system, we are a Unix system. Please respect us." But past that, all our headers look the same. There may be the odd header that you have to like, oops, it's not at sys something, it's just in the regular location. Most of the time you're not massively touching the code base to fundamentally rewrite it, 'cause like Andy said, it's POSIX compliant. Yeah. Nano is a very simple example obviously, 'cause, Nano's been around for a long time. I'm fairly certain it's already been ported to QNX, at least QNX 6.0 and probably QNX 4.0. I've found lots of remnants of QNX 4.0 and QNX 6.0- ... throughout a lot of old software Net- the Netscape libraries that power the web. You'll find QNX 4.0 patches left in there. It's like, "Oh, wow, that's a history lesson for you." Jonathan: It... And so you mentioned a package manager. Is there a package manager built into QNX that probably already has some of this stuff where you can just, you know, It's obviously it's not apt, it's not DNF, but, can I just install Nano if I want it on a QNX system? Aaron: On QNX Everywhere, yeah, 100% that's what we started with, funny enough. We started with a package manager before we went self-hosted like compiling on QNX. We actually use Alpine, so if Alpine Linux, APK tools, their package manager. If you've done any Docker environments on Linux, it's like the thing you use to make- Yes Docker environments a lot smaller. Standard. Yep. We just ported APK tools. We use their able package package builder system, so you know, if you've grabbed an Alpine Linux system before, it looks the exact same. It's the same tool. Andy: So that is one place where I do, I wanna just i- insert my, my kinda legal tiny print quotes. Yeah. Aaron: The Andy: legal version. Because this, that's all true on QNX Everywhere. QNX Everywhere is like our non-commercial version that anybody can download. But the stuff that we have for our commercial version, which is our SDP8 we don't have a package manager yet. So there's a lot of things that we're able to do I would say on the bleeding edge- that's in QNX Everywhere because it doesn't have to go through all the safety certs, because it doesn't have to, meet, like cybersecurity process, blah, blah, blah. So there's a lot of things that we can do there that takes us a while to do, and, in terms of moving those things over to product haven't been done yet. So the package manager doesn't exist in a commercial product. So if you're actually doing the free version, there's a l- few things that you can do that you can't do on the other side. But that's just the tiny print. That's- Jonathan: Yeah, and that's interesting. I think that's something to dig into here. I went to the Ubuntu Summit recently- ... and there was a representative from Nvidia, actually, that was there, and was talking about their plan to try to make, in this case it was the Linux kernel actually compliant with some of the safety regulations. And safety with a capital S, you could say. ... Because there's, there's regulations and there's tooling around how this works. I- I assume because of the places that QNX runs, that's already something that's baked into it from the beginning. But what does that process look like both in the production versions, but also in trying to pull more of this bleeding edge and more of the open source world in to QNX? Andy: Yeah. Yeah, that's a really good question. Like N- NVIDIA's a good example. Like we, ... NVIDIA has, I don't know, probably almost as many QNX developers as we have maybe. Because they, they do all of their own BSPs. So normally for a board support package, we would do the board support package. NVIDIA doesn't ... So they're like, "No. No, we're doing all of our own." It'll ... And so they're very proprietary about their IP and who touches it and who knows what. Yeah. But as a result of that and because they're in all the automotive applications and stuff like that, we're on a bunch of their hardware. Not all of their hardware- ... but we're on a lot of their hardware. And I think that they, like we're like their known partner for most of those sorts of engagements. I think that they probably want to expand that as much as possible to add Linux into the portfolio and things. I know that there was a big exercise done by, some car makers and others to try to basically do a safety certified version of Linux, and they spent a lot of year like two years and a ton of money on- ... doing that. And I think they eventually came to the conclusion that they, it really, that they didn't wanna do it. So what's necessary to do that? Honestly, I don't know. I haven't lived through that exercise. And to be perfectly honest despite the fact that I work with the colleagues that do that stuff is so incredibly boring. I like, I ... As soon as you start talking about "Oh, the safety this and that," like my brain just tunes it all out. So I really can't like- You know, knowledgeably speak about what is required to do it. I just know it's difficult and we've already done it. So there you go. Jonathan: Yeah, ... QNX is a real-time operating system, I think. Andy: Yes. Jonathan: Yeah. Yeah. That, that's- That's right ... that's one of those, that's one of those boxes you have to check to be able to be in certain places the actual drivetrain loop of a car or some of those things. Yeah, interesting. Andy: Yeah, it... it's funny though, but real time is a very flexible kind of a thing. For example you can use Linux with preempt RT patches and get a certain level of real time. But it depends on what it is that you need to do. What real time means f- for you, right? I think for a lot of our stuff that needs super, super critical real time we have customers that use us because, they need to be able to respond to an interrupt within 10 microseconds or less, and we can give them that. But that's super, super unusual. And in fact most things that I think you'd call real time, as long as you respond within a couple milliseconds, it's probably fine. And, Linux in the right environments can do that- ... most of the time. The problem is when the unusual cases happen, how do you manage that and things like that? Yeah. I think the bigger thing that we contribute is the reliability aspect because we are a microkernel and because there's very little trusted code that runs in kernel mode, and pretty much all of the rest of the system runs in user mode. So all the device drivers, file system stacks, USB stacks, networking all of the other components that you might consider kernel mode drivers like in Linux, they all run as applications for us. And so that way if they crash, doesn't bring down the system, right? So that tends to be something that has a very strong need across a lot of domains as opposed to real time, which is, I would say a lot of things in medical and robotics need real time, but y- and cars I guess too. Yeah, cars too, de- depending on what it is. But a lot of the car stuff doesn't need it, right? Jonathan: Yeah, and that's something else that's interesting. U- QNX runs on the head unit, but also maybe the engine management unit, and also maybe on a self-driving car in the loop to do the driving, right? You've got- ... you've got a whole bunch of different places and different scenarios where you can run and with different requirements on each of those. Andy: Yeah. I would say honestly though, we're not usually in the drive loop stuff because those are sometimes just microcontrollers that are bare metal. They have no OS, no nothing. They've been fine-tuned by the, the OEMs for a long time, and people just don't wanna really touch them, right? So y- like when it's that critical a thing, you tend to not wanna play with it a lot. But to your point though, we're in a lot of different applications in the car, like domain or zonal controllers tends to be like the new thing where you're trying to like do ECU consolidation by taking a whole bunch of different modules in the car and saying, "Hey, look, we've got, 20 different 16-bit or 32-bit micros here. Why don't we just put it all into one beefy 64-bit micro and just have it do everything that's there? Throw in a couple hypervisors or multiple different things, and we can manage all that stuff." And I th- that's the way that a lot of the automotive industry is going, and that sort of is a continuation of that is, more consolidation. It's leading to autonomous cars and all the rest, right? So yes, you're right. There, there's a lot of different places that we can play. Jonathan: Yeah. So I think you've probably... L- let me put it this way. I know the answer to this based on some things you're saying. But I think it's an interesting question all the same. Can you do crazy things like take the QNX kernel and run a Linux user space underneath? Or do it the other way around, take your QNX user space and run it with a Linux or a BSD kernel? Andy: I know we can run things on top of QNX because we have a hypervisor, and we do that all the time. I don't know about the other way around. That might be an Aaron question as to whether that's technically possible or not. I, Aaron: I guess it depends on what do you mean by user space? That's an interesting question. What's, what is Linux user space on QNX to you? Jonathan: So I've ... I am thinking of user space being everything but the kernel, and I think the answer to that technically is no, because it's such a different kernel architecture, being a microkernel. But I think it's probably interest- So maybe let's break this down into sub-questions. So what does the QNX init system look like? Can you run system D with... I don't know why you would Aaron: want to, but- Oh, okay. Yeah. So yeah, that's an interesting question. So based off that, so system D, no, 'cause that is just it needs C groups and everything. But that being said we're actually working on OpenRC right now for the QNX Everywhere system anyway, right? Yeah. We're using OpenRC 'cause it's just more around. We do have other init systems for the productized one it's called SLM, which is like a similar idea to OpenRC and all that. It serves the same purpose, but and things like that yeah, you can usually just get any anything that is running Linux that is not using specific Linux sys call. So if you pick something and you say, "Does this run on Linux? Does this run on FreeBSD? Maybe macOS 'cause they're all Unix systems," then yeah, it should just more than likely run on QNX, unless there's a very specific thing it needs implemented like C groups. Yeah, we're not gonna support C groups anytime soon. But- Jonathan: i- is Aaron: there Jonathan: been some work done to emulate those things that you don't actually support? Aaron: Yeah. Epoll is a great example. We actually have, ... So let's think. Epoll, timerfd, signalfd. There's one more I'm forgetting, but those sort of family of things. We actually have user space implementations of those. It will run actually as part of your application. It will spit up another thread that actually handles all the epoll stuff for your program. Nice. But yeah, we do try to actually have compatibility layers. That was actually really important when I did when I ported Java. Trying to write a, the, like an epoll style, like just doing like poll and select in Java would've been super slow, and I was able to just utilize our epoll user space implementation to accelerate the porting of it, 'cause it's good enough. Yeah. Andy: Interesting. One of the- Yeah. One of the things- Kind of to your- Go ahead. Oh, sorry. Go ahead, Jonathan. No, I was just gonna- I was just gonna say d- to that question that you were asking about like what's the boot process like. It's probably worth spending just a second or two to explain how QNX does it, which ... 'Cause it's very different from what a Linux or Windows world is like. And so basically when we start up, the only thing that's really running is the kernel. And you, as the system architect, and because we're an embedded system, there, there's always this concept that there is going to be somebody who's like planning out what the system does. So we don't really have a, out-of-the-box desktop model, other than some of the, QDD stuff that, that Aaron works on. But in order to do that, like you start with absolutely nothing. So if you want a disk driver, like you wanna actually see files on a disk, you start up a disk driver. If you wanna actually talk to a terminal, you start up like, like a serial port and, so there's ... You actually literally start every service that you want- ... in the order that you want them, and that's like one of the things that's used to get us super fast boot times too, right? So like- Sure ... in cases where hey the car starts, you don't wanna wait for three seconds while, like the whole thing is initializing. It's like you wanna actually start the car and go. And so that gives us, like the ability to get enough of the system up and running in, however many, 10, 15 milliseconds or whatever is required to do that. But because of that, it is super different from, yeah init D, R- RCA, all those kind of things, right? Jonathan: Yeah. Y- so Aaron mentioned something a minute ago, and that was his his Raspberry Pi. And I am- ... I am, I'm a Raspberry Pi geek. I've been a fan of- Nice ... of their stuff just as long as it's been around. I added up one time how many Raspberry Pis I had in my house and, lost track after a couple of dozen. What is the, what does the process look like to run QNX Everywhere on a Pi? How difficult is that? Aaron: I'm gonna say it's dead simple 'cause we just directly support it. That's the cheap answer. From a process, I guess probably look at what a process of supporting it would be. The nice thing is Andy kinda started off saying, "Oh, the kernel is the starter thing." There actually is one more thing above the kernel that is our, was the BSP. It's our startup code. Okay. So right above the ker- the kernel is not the thing that actually initial- initializes the CPU cores. It does- it doesn't actually do that level of initialization, the startup code does. So the startup code is the first thing that hits. So when you're flashing an RPi you have- we have a little syntax on how to say I want this startup code and then this kernel with these arguments. The startup code will run, do a- it'll actually set up the hardware. For instance, on the R- Raspberry Pi, it'll talk to the RP1 chip, initialize that, initialize the graphics core. And then from there it'll ... There's a bit of a proto- I'll call it a protocol for conversation. It does a protocol in memory effectively, and then it runs the kernel. The nice thing about that is the kernel you run on my Abund- oh, sorry, my, I say Abundu. My QEMU machine or this board or that board, it's all the exact same kernel. It's just the startup code that changes. Jonathan: Okay. Aaron: So actually our Raspberry Pi startup code is all open source under the Apache 2 license. Jonathan: Very cool. Aaron: Let's see if I can remember this off the top of my head. Github.com/qnx, and then it's BCM, the chip, Raspberry Pi. And they are actually the Raspberry Pi 4 as well. So all that startup code- ... is all open source which is an interesting reference point if you're like, "How do I boot up a Raspberry Pi from scratch and then have a ch- have a core that's initialized?" It's quite interesting to read. Andy: Yeah. I can give you the TLDR version is we have a quick start image for a Pi 4 and a Pi 5. So you can go on and download them and burn them onto a card and then boom, you boot it up. It's pretty simple. Jonathan: Now, the question: Do you have to register an account somewhere to be able to download those? Andy: Yes, you do have to register an account. Oh. Honest- I- so if y- oh my gosh. Y- you, ... So as the ecosystem director of the ecosystem kind of development and trying to get that thing off the ground and make sure that we could, have it a- available in the Raspberry Pi flasher, d- this and this, I'm like, "Look, guys, I don't want any addresses anywhere. I just wanna provide an image you can download." And they're like, "Okay, but w- how do we do export controls?" I'm like, "Oh, God, why are you ... legal guys are killing me on this." And, so yes, you still have to do an email address. I'm trying to get that as simple as possible. I really do wanna get it to just be able to do that, but, just make a, make up a Gmail thing, whatever. I don't care. It's fine. But you need one. Jonathan: Yeah. Export controls, that that's, ... And I get that. I have had to work a little bit with that over the years. They just ruin all the fun. Andy: Y- it, it does ruin all the fun. The, and then, it gets into all these conversations that are really oh, is it a mass market thing? It's if it's ... I'm like, oh, God, y- again, the lawyers start going on about all the details of everything. I'm like, "Okay, can we make this work? Yes? No. And can we make it as simple as possible? Yes or no? Yes? Okay, good. Let's do that." Jonathan: Yep. I- when someone does jump through the hoops, are there some limitations on what you're allowed to do with QNX Everywhere? Andy: Yes. You're not allowed to use it for high-risk applications. You basically take all of the liability of what you're gonna do. Jonathan: Oh, okay. Andy: You're also not allowed to use it for commercial applications. You can't build a Kinix thing and then start selling it, because that's a commercial application. Now, to be fair, we... That- that's the guideline that's in the QDL, the Kinix developer license- ... that we provide. When it comes to startups and proof of concept and, prototyping work and stuff like that, we know that those are, like, sometimes paid engagements. We know that you're building something and you're trying to actually sell a handful of them to see if there's any viability in it and all that stuff. So we really don't care about that. What we care about is look, you're building a product, you're selling millions of dollars worth of it, then that has to become a conversation- Yeah ... because then that engages with the business part of the organization. But if you're just using it to play around with then that's fine. Yeah. Yeah. Jonathan: Yeah, y- and again I'm gonna beat the open source drum because that's what I do. Andy: Yeah. Do Jonathan: it. Companies have solved this problem rather than saying no commercial, because that's, it- this is a thing that's been, like, a conversation in open source for a long time. And so you've got things like the business source license, which technically makes things source available and not open source. But one of the other solutions that I think is better is people use the the Affero public license, the AGPL, which is, that, that sort of closes the, "Oh, I'm running your open source code, but it's on my server, so I don't have to share my changes," right? And so I think there's a, there's an interesting approach to this where as a business you say, "Look, we're gonna open up all the code, but the license that we give it to everyone with is the AGPL, which means if you put it in your commercial project, you also have to release everything as open source. And if you want to put it in a commercial project that you have some closed bits in there, then you have to have a business conversation with us." And that's a, an- another way to go about it. It's an interesting approach. And for those of us that are the real open source enthusiasts that's the way we like to see it done. Andy: Yeah. No, that's fair. And I think, if you're coming at it certainly from the approach of y- you know everything open source and the infectious model of trying to encourage that ecosystem- ... I think that does make sense. Again, I think, we are approaching it and we're trying to actually make sure that, we're true to our word, that we don't you know- go out over our skis, so to speak. Maybe that's a Canadian analogy. I don't know if they use that term in Oklahoma very well. It translates okay. It translates okay. Yeah. Yeah. So like we're not gonna, we don't wanna kinda go all, all out like that then just realize, okay, that's like a terrible thing from a business model point of view- Sure because a lot of those things are done for companies that are fundamentally not the same sort of business that we are. I'm not gonna say that we couldn't ever get there. That's not where we're at right now. That's not what the current thinking is. Jonathan: Yeah. No. That's fair. It is interesting though. So the, one of the other sort of big open source projects that that you guys have worked with is is Eclipse, isn't it? Andy: Yeah. Jonathan: What's the- Yeah, we- ... what's the story with Eclipse? Why did so- so- ... why did someone at QNX say, "Let's give a lot of money to the Eclipse Foundation. Let's work on this source code"? Andy: I think we... So when we had... so we, we support Momentics IDE, which is our, development environment, and that's all based on Eclipse. And I think when we were trying to figure out what are we gonna do from, a software development standpoint?" That looked like really the best choice, and we decided that, th- that relationship was a good one to do. Now, that's not to say that we're not working with Eclipse on other stuff. So like- ... there's the various different automotive consortiums, and Eclipse has one of those, and we're working with them on that. But we also have seen like that Microsoft has taken over the IDE environment, with Mi- and so we're- Yeah, that's true ... we're embracing both worlds on that front. I think, like we're still big fans of Eclipse, to be quite honest, like our tooling on that side is a little long in the tooth. And we had a bunch of Eclipse experts who left the company to go start their own company, and so they started their own, UI framework company. And we were like, "Okay, we still have people to work on that, but like it just hasn't been as high of a priority. And so now it's okay, now there's a shiny new kid in town, so people are chasing that, so you know. I don't know. I don't know where that's gonna be. But yeah. Jonathan: Yeah. Aaron, do you do work on the Eclipse side of things? Aaron: No, I don't actually. I know quite a few people though that do that via some of our... They're not part of the open source team funny enough, but they're part of more of some of the services teams. UProtocol I believe is part of, under the Eclipse Foundation for automotive, and that's a big thing. I know we ha- we have a full-time dev who does internal stuff, but he's also like one of the lead maintainers of uProtocol. His name is escaping me off the top of my head, so I, ugh, apologize to him. But yeah, no, he's a... He works for us and he does a lot of stuff with uProtocol. So that's a proto- that's an automotive protocol that allows you kinda to, I don't wanna say it's like gRPC, but if you're trying to imagine in your head, it solves like the service discovery and protocol between how to like between cars and services and offline support and multiple languages. Jonathan: Yeah. Interesting what does it look like trying to build a community around QNX without the fully open source part? That's gotta be a, that's gotta be a challenge. You can't accept pull requests from your community, and that's one of the ... That's one of the best things about having a community. What- Andy: Yeah ... Jonathan: how does that work? Andy: I think it's probably important to think about what it is we offer and what the community means, right? So it's if you're building Linux like some Linux system, like how much of the code is actually stuff that you're changing in the kernel versus stuff that you're creating on a application side? And, so do you really need to be submitting pull requests to like, tinker with the kernel s- task scheduling- ... priorities and stuff like that? Probably not. Probably that's never gonna happen, right? There's been, thousands of people beating on it, and we have the same sort of dynamic, right? There are very few people who are interested or care about, the really gooey inside, details of how things get done. They just know, oh, hey if I use this software package on a QNX system, I can get this thing done. And yeah, there might be bugs in that and I have to get that fixed, or I want a feature request on that, but it's mostly at the application level or the library level. And that's, that stuff is all open, and that stuff is things that ride on top of our OS. So I think for the most part our interaction with building a community, if you can... I- if you're the kind of open source person that kind of says, "Oh, you know what? I'm not gonna touch it if it's not open source," we're probably not for you. If, like it's the same thing with Qt, I know Qt has had a on-again, off-again relationship there because, they have a commercial side, and it's yes, and they're a business just like us. Like- ... they have to make money they have to figure out a way to monetize what they're doing. But if you're gonna do stuff and you don't need to use that stuff and you can use their free stuff, then great. And we basically are in that same model. We just don't have a open source component to it. If you wanna use it and you wanna do your own projects with it, and you can deal with the fact that we're not, 100% morally within your open source framework, and you can do it for free though, then yeah, then it's actually cool and you can do all these kind of cool real-time things or, experiments with, your Raspberry Pi and, and- all of that kind of stuff. So yeah it's actually been okay. Jonathan: Yeah. Is there at least an avenue where someone can say, "Hey I've messed around with all of your open source stuff. I'm running into this bug. I think it's upstream into the stuff that I can't touch"? Can I go and report a bug- Oh, yeah. For sure ... to QNX as just a regular- 100% ... individual person? Andy: 100%. W- we would probably do that through our Discord. So we have a very active Discord in terms of, we have y- our chief architect is on there. He- I think he's addicted to Discord. He's on there- ... responding to people. People put on questions like, "Oh, how about blah, blah?" It'll be some simple question, and he'll come back with this crazy answer. I'm like, "Okay, Elad, do you... it's okay." So he gives all this detail and all this crazy stuff. So but we have a very active development community. And yeah if you had bugs there, we would track 'em down, submit internal JIRAs, get 'em fixed. Yep. Jonathan: Yeah. Very interesting. So I do to ask, and the question I normally ask is, what's the weirdest thing that you've seen somebody from the community do? And maybe that makes sense here as well. You, you've got a similar question in the rundown here, where they gave me some questions to ask. And it's the other way around. What's the weirdest thing that you guys have done? So maybe both of these are good questions. What's some, what's some- I don't know ... oddball projects? Andy: Aaron, you go first. Aaron: I, I have a couple, 'cause I'm quite free in the Discord. I've... People 'cause kinda going back to your question of building the community, there's a lot of people really enthusiasts who basically we have un- I'll say uncharted water. Maybe the, maybe not the right word, but there's a lot of things that if you wanted, like when somebody ported Kerberos to QNX 'cause they needed the libraries for something. Like- ... that's an interesting thing. How the heck did you get that to work? And then you'll... the software is there, it's just, like, how do you make it work on a new operating system? And people like that discovery, having something existing, working, and then porting it. I am gonna forget the guy's name, but there is some, this is not for QNX 8, but just someone who's having fun in the community. He took the Black, old BlackBerry Passport, so it's running QNX 6, and he's been doing op- he been doing open source development on it, like AI work and drivers. And it's really cool to see people take even some of the older hardware that, that was more, a little more open source. But taking some modern stuff and porting it over, and updating it, Has Jonathan: anybody, has anybody tried to port like KDE Aaron: to QNX? I have had some requests that, to do that, 'cause we have Qt 6 all ported over for self-hosted, so- Oh, ... it's been on my radar personally. But hey, if someone wants to go port the libplasma and see what that looks like, be my guest. I'd be very happy personally. I Jonathan: mean, if you've got, if you've got the Qt libraries, it might not be as much of a lift as I expected So it's, but it's a good p- Yeah, I mean- ... good Andy: portion of the work done ... and it's similarly like we like what is it, XFC or something that we run that runs on- Yeah Wayland that's on top of, QNX. So there's definitely good chunks of things that- Oh, yeah ... already exist. So I would recommend to anybody who might be serious into looking at this kind of stuff, is go to oss.qnx.com. And that's that's our open source portal. So that shows every package that we've ported and/or tested, where it lives, if we're the ones maintaining it or if it's being maintained on another server somewhere in GitHub or GitLab or wherever else. And what the status of it, what architectures that we know it works on, all that stuff. And there's some, I don't know, 2,000 some different packages that are there or y- somewhere around there. So it's actually, it's got a lot of stuff in there that's interesting to see if you, for example, if you're doing AI stuff and you wanna use TensorFlow or MediaPipe or any of those kind of things, that those are all ported. And you can just find what stuff's been ported and what stuff you've got access to. And those are all things that you can either get and, clone 'em onto your machine or, use the APK tool like Aaron was talking about and get access to 'em. So- ... makes it pretty simple. Jonathan: Yeah. I will tell you my most interesting QNX story, my only really QNX story. Okay. I've got a I've got a dbx DriveRack. It's a speaker management platform for doing pro audio. And turns out that I'm pretty sure it actually runs QNX inside of it. And I know this because this version of QNX, somebody accidentally left the debug port open and you can- Ask it to debug bash for a shell. Ask it to de- debug shell for you, and it will gladly do that. Which, means that you send it a single packet and you pop a shell, and you've got root access to it, which is fun. So I've crawled around inside of that, inside the QNX system inside of it which is yeah, it was quite an experience. That's been around for a long time. That is some- Oh, yeah ... old software. Aaron: Yes, I'm really curious what version it's running, 'cause it must be six or four if it's been around for a long time. Jonathan: Yeah. I, at one point I had the the actual CVE. I don't remember the CVE number that it was. I could I could probably find it again. I'm not... My Google fu is not quite good enough to pull it up right now. Andy: No, Google Flu is actually getting ruined too, because it's like you get results that are like, what that means nothing. Like what year did Biden die? Oh, 2039? Wait, what are you talking about? Jonathan: Yes. Just all hallucinations. Absolutely. Andy: Yeah. E- everything pure hallucination. Yeah, the Google Flu d- means nothing. Yeah. So Jonathan: I will tell you- Yeah, my- ... speaking of hallucinations, when I Googled for that- ... Google's AI told me that both QNX and DBX are owned by Harman. I'm like, "I don't know that's true." Andy: Was true. It was true actually for, so I was here during that period. So 2005, I think we were acquired by Harman, and I think Harman sold in 2010 to RIM, to Research In Motion. So there was a period of time when we were doing a lot of stuff with Harman yeah- It was probably during Jonathan: that time. I would... That would be very likely this is during that time that, that, Andy: that change was made. Probably was very likely during that time. Yeah. Yep. Yeah. And we were in like some guitar pedals and yeah, a bunch of cool applications, you Jonathan: know? QNX on a guitar pedal. That's- Yeah. I can't say much. Linux runs in some really weird places too, and, Andy: Yeah, Linux runs in weird places. And we're in things like space arms and nuclear power plants, so we're in l- we're in some pretty niche applications. And my crazy stories tend to come from my first stint around when I was at QNX as a FAE, and, finding out that it's oh yeah, this whatever it was, a coal plant or something like that, hadn't rebooted their system for 10 years, and then they tried to reboot it and it was running off of a floppy. The entire system was running off of... And the floppy wouldn't work. Yeah it hadn't been used in 10 years. So I'm, like, helping this guy who's frantically panicking as you know- ... the entire plant is brought down because they haven't even tried to run it. I'm like, l- on eBay looking for floppy. Okay, what is it? It's a model, something that, and I'm like, "Okay, I think I found one." That there's one in Arizona somewhere. We can get them to overnight it, Jonathan: like- Andy: Yeah, so- Jonathan: I- is- Yeah ... is QNX still a part of Research In Motion? Is that still the sort of corporate overlord? Andy: Research in Motion renamed to BlackBerry, and we still are a division of BlackBerry. So there's basically, there's two main divisions. So one is QNX and the other is Secure Comms. So if you think about what, what made the BlackBerry phone kinda all, secure and, its own very highly protected ecosystem, all that software, they've taken all of that stuff and turned that into a business. So like governments large corporations, institutions, all those kind of big company players that really value corporate security and those kind of things, they're customers of Secure Comm, so our sister company. And then QNX, we- we're doing the basically the same thing that we've always done. We just don't do it on phones, Jonathan: so they- they've- they've forked it. They've split the streams. That's right. You guys are no longer doing the phone stuff. Ah that's good to know. Andy: We're not... Yeah, basically as of whenever BlackBerry kinda folded up shop on the hardware business, then, we stopped sup- supporting the phones. But yeah, like we, even through all that whole period, we were still doing tons of automotive work, lots of robotics, industrial, all that kind of stuff. Jonathan: Yeah. Super interesting. What as you look into your crystal ball of the future, what what interesting things do you see coming for specifically for the, like the QNX ecosystem? What are some things people should be watching out for? Andy: Yeah, that's- This- That's actually a really good question ... Jonathan: this is where you can make your earth-shattering announcement that you're gonna open source it- Andy: Yeah ... if you want to. Yeah it probably won't be that earth-shattering. I like I- I'll tell you honestly, I think that one of the biggest challenges that, that I am trying to struggle with is just what are we doing with AI? And there's a lot of smart people in the company that are trying to figure that out, and we're looking at lots of different approaches to that. One is that we think we're, like, the ideal platform for people implementing physical AI systems because we provide all the safety and the real time and all of these kind of characteristics that you need. But we're also POSIX compliant, and so therefore, all of these AI things that people are experimenting with will run on our system. And like, when, Aaron had mentioned we had just gone to the Berkeley Hackathon, and yeah, that it was a AI-based hackathon, and we had people using our stuff to, implement, tank detecting or, looking for patterns outside cars or, doing whatever, doing games, all kinds of interesting applications to AI. So one of them is enabling AI, and that I think we have a relatively good handle on. But what I don't have a good handle on is what are we gonna do from a tooling standpoint to enable people to be better using AI with our platforms? And it's a little tricky for us because our entire brand is all about safety and security and reliability and stuff like that. You don't put us in a wind turbine 'cause you think, so you vibe coded what that is, and then it kind of- ... starts feeding the wrong current back into the, thing at the wrong phase and totally blows out the system or whatever, right? But there's all these serious consequences to this stuff. So- I think that there, there's that, but then there's the realization that it's okay people are gonna be using AI and we need to figure that out. And we're kinda undergoing that exercise ourselves of trying to understand, okay, how do we take AI and merge that with a lot of the lessons that we know about- functional safety and certification and process and all of those ways that you can use to kinda get a handle on a system. Yeah. And in some ways, it's very similar to just kinda hey, what if you hired a bunch of interns that are really eager, and you just set 'em at a task. How do you- Absolutely control their energy, right? Like, how do you vet that they're doing something right? And I think I- we're gonna come around to that, and I think the industry will come around to figuring that out. But- ... that's the biggest challenge, and I think that's the biggest place for us to kinda make a splash. But we're not quite there just yet. Jonathan: Yeah. Aaron, can we run Python and PyTorch on inside a QNX Everywhere? Aaron: Oh, of course, yeah. PyTorch. Let's see here, what's on my list? Llama CPP. I know we have Stable Diffusion, the PR up for that coming out in a minute or two again, it's one of those things. You, you name it, either we got it or tell me I don't and I'll make sure it works. Somebody's Andy: working on it, yes. I'll send you... Yeah, I'll send you a blog afterwards that- Okay ... that has all of the things that, that are AI related that we've got ported, and it's a list of a, I don't know couple, three dozen different packages that are in that space that, that would be useful. Actually, there's a couple blogs that I could send you that, that might be useful for that. Jonathan: Yeah. And something we briefly touched on this earlier, but I think it's interesting. Can I take one of the QNX images for... Obviously I've gotta make an account somewhere, which I'm not super happy about, but I use a burner Gmail address. It's fine. Can I take one of those images and run it a inside a virtual machine on my Linux machine? Play with it- Yes ... that way I don't have to, I don't have to dedicate any hardware to it. Yeah. And then does it work the other way? Yeah. Can I run a Linux VM inside of a, inside a QNX Hypervisor? Andy: Yeah, you can. Yeah. And a QNX Hypervisor is also part of the QNX Everywhere. The stuff you get for free is part, the Hypervisor is part of it, so you can do that too. Jonathan: Okay. Andy: Honestly, there's also a bunch of stuff about Graviton and running an AWS and, running in Azure as well. I- ... I'm not as familiar with that. A lot of the stuff that, that I've and Aaron do are, like, a lot of university stuff, but I do know that, people use our stuff to run in cloud instances as well. I would not- But if you start asking me any kind of detailed questions there, I'm gonna get way out of my depth instantly, Jonathan: I would not have thought of running QNX in the cloud. That's actually really interesting. Andy: I- It is interesting. It's, people use it for a scalability thing. And if they wanna have a whole bunch of developers work on the same kind of system across time zones and across the world or be able to scale up how many instances are running or not, there's a lot of reasons to be able to say, "Oh, hey, yeah, you wanna actually use this particular..." Let's say you're developing a, a digital instrument cluster or something like that, and you've got developers all around the world that wanna be working on it at the same time. It's easy to do that in a cloud instance, whereas it's tricky to do that with a physical piece of hardware, right? And plus you can get out ahead of, the needs. Like we're very much an embedded company. Almost all of our applications are embedded in some way, but that also means that you're tied to a piece of hardware. So as soon as you break that you can say, "Okay. Let's get this working in the cloud first, and then as soon as we get the hardware shipping, then we can start tying it." But otherwise you're waiting for hardware to come and it's "Okay. We can't run anything, so what are we gonna do?" I... it's, it just helps accelerate that whole thing. Jonathan: Yeah. I've gotta, I've gotta ask, and this is a meme-y, troll-y question, but Andy, did you inspire any- Awesome ... did you inspire any of the characters in the Blackberry movie? Andy: No. Jonathan: Were you around for that time period? Andy: I was around for that time period, but I- I actually, I left after that and then I came back. So yeah, but I was around during the kind of the acquisition and the phone sort of thing. Yeah some interesting times, that's for sure. Y- honestly, because I, at the time I was on the automotive side- ... and they had this really l- like this firewall of stuff. So it's like we knew that half the company was being hived off for something. And for a long time we didn't know what it was. We were like, "What the heck is going on? Are we being sold? Are we being bought? Are we..." We had no idea. And then it was then there was the announcement like, "Oh, okay, now it all makes sense." And that's why I couldn't talk to this guy for four months, yeah. Jonathan: That's hilarious. Yeah. Andy: Yeah, it was pretty weird. Jonathan: Yeah. Fun. All right, so before I let you go, it's been a, it's been a fun conversation. I gotta ask you, each of you though what's your favorite text editor and programming language? Let's go, Aaron: I'll, Andy: I'll go Aaron: first here ... Aaron's geo first. So I used to say VS Code, but because VS Code does not run on QNX, I am now a Neovim user. Jonathan: Ah, there you go. Aaron: For favorite language, I've been a JavaScript fun- I've worked in a betting company. I've been a JavaScript developer for a long time, but I've really come to appreciate Rust and those newer languages like Zig have been very interesting to see what they're doing in the space and modernizing embedded development. Jonathan: Yeah. Have you guys done anything with WebAssembly? I, that, that's something that really blew me away- Yes ... is that people are doing WebAssembly in, in embedded. And we're actually- Yeah ... in, in my company, Mesherastic, we're actually doing sort of that in reverse. We're taking some embedded code, compiling it to WebAssembly, and then running it in the browser or, on Android. Yeah. All kinds of crazy stuff. I'm just blown away by- We- ... by what you can do with WebAssembly. Andy: WebAssembly is pretty cool. We were talking with some people at Carnegie Mellon, and they were super into WebAssembly. And they were like what do you guys have for WebAssembly? What can we do there? 'Cause we, we think you got a really cool product, and we want to do all this WebAssembly stuff." And then we do support it as a outshoot of the fact that we have a browser, but we don't have ... we're not natively doing anything in it. So it's when they're talking about "Oh, can we do research projects with it and stuff?" We're like you can, but we're not ... We don't have a vested interest in it because we don't have any customers using it." But, we'll keep our finger on the pulse there, but yeah, I don't know. It is pretty interesting. Okay. So I'll answer those two questions with regard to favorite editors and stuff like that. Favorite editor that's like a weird sort of a question in my mind. I've got two answers to that- ... and you'll ... You'll probably say, "Oh my God, this guy." So one of my answers is VI. And okay, now please hear me out on this one because it just works everywhere. I can go anywhere, and I can do what I need to do with VI. You can tell I'm not- So it's like I'm not, like- You could tell that into a Jonathan: router and it'll be there Andy: Ex- exactly. Like I, I can always depend on it being there. Yeah. So that's But like for day-to-day use, oh God no. Like I used to know a guy who I worked with and like he did that with his main editor. And I'm like, okay, he was astounding. Like he knew all these colon commands and he could do crazy stuff. I was like, I would just watch him going, "Wow, if I knew like half of what you knew like that would be amazing." But I can get by. But Xcode, and I'm sorry, yes I do Mac and I'm, my next thing is not gonna surprise you, Swift. I love Swift. It like, it's just so well-designed. Like I just, I can't get over how nicely designed of a language it is. And now it really irritates me that the interfaces change like every flipping release. But other than that- that's a very Apple thing to do, Jonathan: I think. Andy: Yeah. They're just trying to make it perfect. So then of course why not break everyone's code every single time that they release something. So- Of course ... yeah. Jonathan: C- can you run Swift on QNX? Andy: No, but I want to. Aaron: Oh man, that would be amazing. I know, but the thing is that Swift is finally getting Linux better Linux support. Relatively recently. I remember there's a whole thing with Ladybird was looking at using Swift, but- ... it just wasn't there yet, I believe is what it was, if they get a little support Aaron, I had a couple of- ... maybe I'd love to do it ... kids come Andy: up to me and say, "Can we get Swift on, on QNX?" I Aaron: know. I- Andy: And I was like, "Yes! I found my people." I, Aaron: I love, every language we add to QNX just makes the ecosystem better and better, right? P- pick a, I don't care what language it is. Totally Andy: agree. Odin, Aaron: right? I take Odin, it's for 3D graphics and everything, but hey, people, auto- auto manufacturers need a good 3D programming language and do stuff like that. So yeah, every language you put there, let's do it. Yeah. It's fun. That's Jonathan: fun. Yeah. That's great. All right. Thank you, guys. Yeah, that was really cool. Thank you, guys, for coming on the show. Thank you for being patient with my questions and my badgering. It's been a lot of fun, though. Andy: It has. It's been an absolute pleasure. Thank you. Thank you so much. Jonathan: All right. Good to have both of you. That is Andy Green and Aaron Bassett from QNX, and it's been a bit of an oddball show, but that that's fine. We've had a lot of fun with it. We've got some fun stuff coming up as well. Next week we are scheduled to have Andrea Gallo. We're doing the complete opposite next week. Not only is the operating system open source but all the way down to the ISA, we're talking about RISC V. And then the week after that we're gonna talk about JavaScript and hopefully some WebAssembly because we've got Neriman Jelva of Pewter, somebody else I met at the Ubuntu Conference. And then after that we're talking with Michael Meeks of Collabora, and then we're having Francois Pol again, talking about smoked meat. That is security, specifically security on GitHub. A fun schedule coming up. You don't wanna miss it. And we will see everybody next week. Just wanna say thank you to everybody that watches, and yeah, we'll see you then
• 
  Episode 873 - Wait, That's Not Open Source! 01.07.2026 1t 3min

  This week Jonathan chats with Andy Gryc and Aaron Basset about QNX, and the interesting Open Source history and future of that embedded OS. Why does QNX Everywhere feel more open, and why do you need to register an account to download images? All that and more -- Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/