FLOSS Weekly

Epizodai

• Episode 873 transcript 01.07.2026

  FLOSS-873 Jonathan: This week, Aaron and Andy join me to talk about QNX. It's the other operating system that runs in a whole bunch of different places. It really has safety and real-time nailed down. There's just one little problem. This is FLOSS Weekly, episode 873, recorded Tuesday, June the 30th. Wait, that's not open source It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we're talking about a Unix or a Unix derivative or a Unix-like. I'm not sure which of those terms is technically correct. We're talking about QNX. It runs a whole lot of places. It used to run on your BlackBerry, and it runs a lot of places that you don't think about. Maybe not on your toaster, but maybe on your car. Anyway, we've got we've got Andy Green and we've got Aaron Bassett to talk about QNX, where the company is, where they're going, what changes are there, and why, maybe you should think about doing some development on it. Let's go ahead and bring them on. Guys, welcome. Welcome to the show. It is great to have you both here. Andy: Thanks so much. Really appreciate it. Jonathan: Thanks for having us. Yeah. And now, we gotta start with this. We have to start with this, right? So this is the show about open source, Floss Weekly, free, libre, open source software. We're not... We don't do dentistry here. We do open source stuff. QNX, at least like the core of it, the kernel of it it's closed source. And what are we even doing here, guys? Get get out. What's Andy: the conversation? Yeah. Why are we on this show? Jonathan: Why does this Andy: make sense? Yeah, no that, that's very fair. So I, we have had a kind of a really interesting and long evolving story with open source, and I think for some of your audience who may remember, we did go fully open source at one point in our history. And and that was like right around the time when we were acquired by Research In Motion for use in the BlackBerry, and they were thinking, "Hey this stuff is special sauce. We really don't want this to be, like-" "... our... one of our defining characteristics. We don't want to be fully out there because we wanna actually have some of that proprietary goodness for ourselves, for our own phones." So we clawed that back. And so that that is something that I think, a lot of us in QNX were not necessarily... we understood the business rationale for that and all that. But I think that there was always this feeling that, there's a lot of good and value that we get with working in the open source community and with the open source community, and we wanted to bring some of that back. And so I think relatively recently we've had this, So I guess QNX Everywhere has been a program that, that both Aaron and I are part of both sides of the, on the technical side and on the s- sort of more on the business side. And that's been around for about a year and a half. And that's been our start back to recognizing, l- look we need to be more involved with the community. We need to be more transparent. We need to open ourselves up a little bit more. So I'm not gonna make a commitment and say, "Hey, yeah, we're going all open source," or anything like that. We're not gonna, make that same mistake of making commitments and then pulling back on them. But what we are doing is we are trying to open up a lot of the pieces that we see don't really have impact on our business. Because I'll be frank, there's a few different things that are factors as to where- whether or not we do things open source or not. And one of the big ones is that a majority of our business is based on regulated industries. Things that either have functional safety components or cybersecurity components or things like that. And in a lot of cases there's this thing called SOUP, which is software of unknown providence that is an acronym that's used within those sort of, certification bodies or regulatory communities. And having open source software in a lot of cases makes things extremely difficult to pass through certification. Not because of the testing or anything like that, it's because you can't really guarantee where that software has come from, or you have a very difficult time tracing the line through of all the edits that have been made to it- from different contributing parties. So we have to be super careful and super cautious about things that are gonna impact our business. So we don't we're not gonna go and say, "Oh, yeah, hey, you're, we're gonna open source the kernel," because then anybody can contribute to it, and then we turn around and find out, oh, somebody put this really super clever hack in that- wasn't really intended, all that kind of stuff, right? But to that extent, though, we are doing things like, making sure that we're open sourcing all of our board support packages, open sourcing our development, our driver development kits- ... open sourcing pieces of our software stack that actually originally may have come from open source and we've made changes to it or things that we don't think are, impacting, functional safety or any of those kind of things. So keep your eye on that space because that's the one aspect and, but the other aspect is us working with the community because we get a lot of value out of working with open source and making sure that, like open source software runs on our platform or that we can use it or deploy it in, for either hobbyists or academia or, companies that are doing prototyping or proof of concept or R&D or any of that kind of stuff, right? So there's a lot of people that are playing with different open source things, and we wanna make sure that all those people are enabled. So there's like a lot of things that are going on within the company right now around open source, and we think we're good open source supporters, and we do recognize that, yes, we haven't thrown the kimono open for everything, but- but we're doing that in an intelligent way, right? Jonathan: Yeah. I do wanna touch on one, one thing that you mentioned, and this is something that we've seen several projects make moves in this space. But the idea of when you're open source, that means that you accept pull requests. And that is-- we've seen several projects come to the point where they say, "Look, we're open source. All the source is out there, but we do not accept pull requests," or, "We only accept pull requests from people that are already in the project." And, I think AI is leading to some of that. In some cases it's because of just very high code standard quality, code quality standards. I think in other cases, though, it's because of that legal question. And so there's a relatively famous blog post from DHH that talks about this, and it's like open source is a license. It's not a community management style. And I just think that's interesting in this particular context to think about that. And it goes all the way back to this idea of the cathedral and the bazaar, right? From ESR way back in the day. And you can do things that are free software and open and still, the monks in the cathedral are the ones that are slowly working away on it, and it's not just a bazaar where everybody can push code in. Andy: No, I agree. And I, and, and certainly I've seen things where it's yeah, but people are gonna have to stop taking software because of all the AI slop that's being generated and I think, our approach to that might be, like, even more conservative than- ... is strictly necessary. But I think that we're approaching this as a, we, we learned a lesson from what we've done before. We don't wanna actually, go a step far and then take a step back, right? So- ... so we're gonna do that i- in an intelligent way. I, the CEO of our company John Wall, has made a lot of comments about, how he feels that we, we should, open source as much as we humanly can. But there are gonna be, w- we have to still do that with all of the processes that we adhere to, with all of the regulations that we adhere to, all the rest of that stuff. Honestly, yes, we could immediately pivot to saying, "Oh, yeah we're gonna just do an open source methodology and modality to manage all that stuff." But kind of it's, I'll just say it's the path of least resistance to not do that- ... and to just say, "Okay, for those things that actually have critical components that we're j- we're gonna manage them the way that we always have," because we have processes that we know work that have been vetted. We've actually been audited on all that stuff. We don't have to re-audit everything and re- like reinvent the world when it comes to that, Jonathan: yeah. I, th- there's a lot more to talk about with this, but I do wanna bring Aaron in on the conversation. And you are- Absolutely ... you're part of the open source team, Aaron. What, so what are the bits of, in the company that, that you work on specifically? What's the open source story there? Aaron: Yeah, so primarily for myself, I work on the QNX developer desktop. So that is the self-hosted developer system that we've been making to allow Basically anyone who wants to use QNX, whether it's hobbyists, students, to get easier access to QNX. Primarily it's been cross-compiled, embedded system, right? So you've done Arduino development that, you cross-compile your thing, you copy it there, you flash a board, you go there. A lot of people don't m- know how to do that or it's something they've never learned, so it's really hard to get into, especially at hackathons. 48 hours, let's learn how to cross-compile and flash- ... an SD card and everything, right? That's something you just don't do. But part of that is, is I maintain the package manager that we have for that. How many... couple hundred or thousands of packages that we have now that are in there that we've built and distributed across that. So we touch a lot of stuff. Jonathan: Have you guys found that that AI is making a big change with the way that people interact with this whole system? Andy: Yes. Yeah. Yeah. Yeah, so I started this is s- cr- this job like just about a year ago. And when I first started doing it... So I've been in the industry a long time, but I came back to QNX after running my own company for a while. But in any case when I first did it, we were doing, we'd go to a hackathon, we'd be seeing somebody's project, and they'd be like I guess we used AI to do part of this," right? And hackathons now, it's "Yeah, no, we're 100% full in A- AI. We're- we- there's no bones about it. We're not being bashful about it. Like-" "... we just had Cloud Code do the whole damn thing." And it's like in the course of a year, like that's the perception- ... change that we've had, at least, from students, and maybe they're on the leading edge of that kind of thing. Sure. But I don't think they're the only ones. Jonathan: Yeah. So it, this is obvious. This is one of those observations that I made that's obvious in retrospect, and maybe I was late to this game, but open source makes AI work. And if you're if all of the things that you're doing is closed and not documented publicly, then AI is not gonna be able to do much with it. But when you put it all out there- ... on the internet, and it's part of the training data, then AI knows exactly how to work with your stuff. I- is that s- s- part of the consideration that QNX makes with, like, how much of this to publish and how much to make open? Aaron: Andy, I have an interesting observation on that. So- Andy: Yeah, Aaron: please go for Andy: it. Aaron: Yeah, so I have been playing on on my own time, on my own hardware what does AI on QNX look like? 'Cause, again, you go to these hackathons, and we actually were just at an AI hackathon- ... at Berkeley, and very surprisingly, there's next to no code that knows how to write a QNX Resource Manager, which is how you do devices and drivers. And I was able to, in an afternoon, ask it, "Hey, let's make a temp- like a memory file system," and it spit out some of like perfect QNX-isms, I'll call them. It knew exactly APIs I was able to use, how to do it, found the stuff that you're missing. But I was like what, like where did it learn this all?" There's no code out there that it's open source, like lots of it that doesn't do this. And it may just, it's just all from our documentation. We have really detailed technical documentation that's all public and free. And just by scraping that, it's actually able to do- to do things like that. I saw a lot of that at the AI hackathon. It was able to spit out things that looked like a professional QX, someone who's in QX development for years would write. Andy: Yeah. Yeah. And actually that so that's a really good point and I'll come back to, I think, your opening comment, Jonathan, that led us into this, which was I don't know if it's Unix or this or that or whatever." Yeah. And technically we're none of those things, but we are POSIX. So we're POSIX compliant- ... meaning that we use the same APIs essentially that Unix and Linux do for the most part. There's a little bit of stuff at the f- three signal layer that we differ for and Linux does and Unix does. But for the most part, that means that a lot of stuff runs very easily or without, minor stuff. And when you start playing with the hardware or dealing with, some of the minutiae of device drivers and stuff like that, yes, then you can start running into some things where we're different because of the nature of a microkernel or because of the nature of real time or whatever it is. But a lot of stuff will just work out of the box, and that's one of the really interesting things that, that we've been, like leveraging in terms of a lot of work that, that Aaron and team have been doing to, create all those things that are open source software but that runs on QNX. Jonathan: What, what does that process look like? So if someone has, let's just say, an open source project oh, I don't even know what a good one would be. Nano. All right? Text editor Nano. They, I wanna compile Nano for this QNX board. What does that look like? Can we use GCC? Does CMake work under QNX? How alien is the landscape for someone that's used to like Linux tools? Aaron: I am confident enough to say that if I gave you an SSH connection to my RPi that has QNX on it, if you didn't run uname, you probably would not be able to tell the difference. I'll get rid of the uname thing, 'cause you're probably like, "Oh, uname, hey, oh, it's a QNX system." It looks and feels like a Unix system. I can say the same thing about FreeBSD nowadays. You do SSH to FreeBSD, it, it feels and looks like everything you're used to. And that's one of the things that we really strive to with QNX Everywhere, is make the development system as close to as possible as FreeBSD or Linux, right? So for QNX Everywhere, we have the, we use the Clang compiler for self-hosted development. Obviously for cross compilation, QNX uses GCC with our QCC wrapper, which is, has all the safety stuff for it. But yeah, you just have, you have Clang, you can run Make. We have Mason. Actually just recently, about a couple months ago, you can run oh, what's the Java one called? Bazel. Oh. You can run Bazel on it, right? So we can... You can run everything on it. It's very familiar. Yeah. So- So but go back to your thing for Nano though. Nine times... you grab Git, you do your Git clone or grab it for release tarball. Start off with your standard configuration just like how you'd do it on Linux, and then start following the errors. Most of the time it's just telling it, "Hey, QNX, we are a system, we are a Unix system. Please respect us." But past that, all our headers look the same. There may be the odd header that you have to like, oops, it's not at sys something, it's just in the regular location. Most of the time you're not massively touching the code base to fundamentally rewrite it, 'cause like Andy said, it's POSIX compliant. Yeah. Nano is a very simple example obviously, 'cause, Nano's been around for a long time. I'm fairly certain it's already been ported to QNX, at least QNX 6.0 and probably QNX 4.0. I've found lots of remnants of QNX 4.0 and QNX 6.0- ... throughout a lot of old software Net- the Netscape libraries that power the web. You'll find QNX 4.0 patches left in there. It's like, "Oh, wow, that's a history lesson for you." Jonathan: It... And so you mentioned a package manager. Is there a package manager built into QNX that probably already has some of this stuff where you can just, you know, It's obviously it's not apt, it's not DNF, but, can I just install Nano if I want it on a QNX system? Aaron: On QNX Everywhere, yeah, 100% that's what we started with, funny enough. We started with a package manager before we went self-hosted like compiling on QNX. We actually use Alpine, so if Alpine Linux, APK tools, their package manager. If you've done any Docker environments on Linux, it's like the thing you use to make- Yes Docker environments a lot smaller. Standard. Yep. We just ported APK tools. We use their able package package builder system, so you know, if you've grabbed an Alpine Linux system before, it looks the exact same. It's the same tool. Andy: So that is one place where I do, I wanna just i- insert my, my kinda legal tiny print quotes. Yeah. Aaron: The Andy: legal version. Because this, that's all true on QNX Everywhere. QNX Everywhere is like our non-commercial version that anybody can download. But the stuff that we have for our commercial version, which is our SDP8 we don't have a package manager yet. So there's a lot of things that we're able to do I would say on the bleeding edge- that's in QNX Everywhere because it doesn't have to go through all the safety certs, because it doesn't have to, meet, like cybersecurity process, blah, blah, blah. So there's a lot of things that we can do there that takes us a while to do, and, in terms of moving those things over to product haven't been done yet. So the package manager doesn't exist in a commercial product. So if you're actually doing the free version, there's a l- few things that you can do that you can't do on the other side. But that's just the tiny print. That's- Jonathan: Yeah, and that's interesting. I think that's something to dig into here. I went to the Ubuntu Summit recently- ... and there was a representative from Nvidia, actually, that was there, and was talking about their plan to try to make, in this case it was the Linux kernel actually compliant with some of the safety regulations. And safety with a capital S, you could say. ... Because there's, there's regulations and there's tooling around how this works. I- I assume because of the places that QNX runs, that's already something that's baked into it from the beginning. But what does that process look like both in the production versions, but also in trying to pull more of this bleeding edge and more of the open source world in to QNX? Andy: Yeah. Yeah, that's a really good question. Like N- NVIDIA's a good example. Like we, ... NVIDIA has, I don't know, probably almost as many QNX developers as we have maybe. Because they, they do all of their own BSPs. So normally for a board support package, we would do the board support package. NVIDIA doesn't ... So they're like, "No. No, we're doing all of our own." It'll ... And so they're very proprietary about their IP and who touches it and who knows what. Yeah. But as a result of that and because they're in all the automotive applications and stuff like that, we're on a bunch of their hardware. Not all of their hardware- ... but we're on a lot of their hardware. And I think that they, like we're like their known partner for most of those sorts of engagements. I think that they probably want to expand that as much as possible to add Linux into the portfolio and things. I know that there was a big exercise done by, some car makers and others to try to basically do a safety certified version of Linux, and they spent a lot of year like two years and a ton of money on- ... doing that. And I think they eventually came to the conclusion that they, it really, that they didn't wanna do it. So what's necessary to do that? Honestly, I don't know. I haven't lived through that exercise. And to be perfectly honest despite the fact that I work with the colleagues that do that stuff is so incredibly boring. I like, I ... As soon as you start talking about "Oh, the safety this and that," like my brain just tunes it all out. So I really can't like- You know, knowledgeably speak about what is required to do it. I just know it's difficult and we've already done it. So there you go. Jonathan: Yeah, ... QNX is a real-time operating system, I think. Andy: Yes. Jonathan: Yeah. Yeah. That, that's- That's right ... that's one of those, that's one of those boxes you have to check to be able to be in certain places the actual drivetrain loop of a car or some of those things. Yeah, interesting. Andy: Yeah, it... it's funny though, but real time is a very flexible kind of a thing. For example you can use Linux with preempt RT patches and get a certain level of real time. But it depends on what it is that you need to do. What real time means f- for you, right? I think for a lot of our stuff that needs super, super critical real time we have customers that use us because, they need to be able to respond to an interrupt within 10 microseconds or less, and we can give them that. But that's super, super unusual. And in fact most things that I think you'd call real time, as long as you respond within a couple milliseconds, it's probably fine. And, Linux in the right environments can do that- ... most of the time. The problem is when the unusual cases happen, how do you manage that and things like that? Yeah. I think the bigger thing that we contribute is the reliability aspect because we are a microkernel and because there's very little trusted code that runs in kernel mode, and pretty much all of the rest of the system runs in user mode. So all the device drivers, file system stacks, USB stacks, networking all of the other components that you might consider kernel mode drivers like in Linux, they all run as applications for us. And so that way if they crash, doesn't bring down the system, right? So that tends to be something that has a very strong need across a lot of domains as opposed to real time, which is, I would say a lot of things in medical and robotics need real time, but y- and cars I guess too. Yeah, cars too, de- depending on what it is. But a lot of the car stuff doesn't need it, right? Jonathan: Yeah, and that's something else that's interesting. U- QNX runs on the head unit, but also maybe the engine management unit, and also maybe on a self-driving car in the loop to do the driving, right? You've got- ... you've got a whole bunch of different places and different scenarios where you can run and with different requirements on each of those. Andy: Yeah. I would say honestly though, we're not usually in the drive loop stuff because those are sometimes just microcontrollers that are bare metal. They have no OS, no nothing. They've been fine-tuned by the, the OEMs for a long time, and people just don't wanna really touch them, right? So y- like when it's that critical a thing, you tend to not wanna play with it a lot. But to your point though, we're in a lot of different applications in the car, like domain or zonal controllers tends to be like the new thing where you're trying to like do ECU consolidation by taking a whole bunch of different modules in the car and saying, "Hey, look, we've got, 20 different 16-bit or 32-bit micros here. Why don't we just put it all into one beefy 64-bit micro and just have it do everything that's there? Throw in a couple hypervisors or multiple different things, and we can manage all that stuff." And I th- that's the way that a lot of the automotive industry is going, and that sort of is a continuation of that is, more consolidation. It's leading to autonomous cars and all the rest, right? So yes, you're right. There, there's a lot of different places that we can play. Jonathan: Yeah. So I think you've probably... L- let me put it this way. I know the answer to this based on some things you're saying. But I think it's an interesting question all the same. Can you do crazy things like take the QNX kernel and run a Linux user space underneath? Or do it the other way around, take your QNX user space and run it with a Linux or a BSD kernel? Andy: I know we can run things on top of QNX because we have a hypervisor, and we do that all the time. I don't know about the other way around. That might be an Aaron question as to whether that's technically possible or not. I, Aaron: I guess it depends on what do you mean by user space? That's an interesting question. What's, what is Linux user space on QNX to you? Jonathan: So I've ... I am thinking of user space being everything but the kernel, and I think the answer to that technically is no, because it's such a different kernel architecture, being a microkernel. But I think it's probably interest- So maybe let's break this down into sub-questions. So what does the QNX init system look like? Can you run system D with... I don't know why you would Aaron: want to, but- Oh, okay. Yeah. So yeah, that's an interesting question. So based off that, so system D, no, 'cause that is just it needs C groups and everything. But that being said we're actually working on OpenRC right now for the QNX Everywhere system anyway, right? Yeah. We're using OpenRC 'cause it's just more around. We do have other init systems for the productized one it's called SLM, which is like a similar idea to OpenRC and all that. It serves the same purpose, but and things like that yeah, you can usually just get any anything that is running Linux that is not using specific Linux sys call. So if you pick something and you say, "Does this run on Linux? Does this run on FreeBSD? Maybe macOS 'cause they're all Unix systems," then yeah, it should just more than likely run on QNX, unless there's a very specific thing it needs implemented like C groups. Yeah, we're not gonna support C groups anytime soon. But- Jonathan: i- is Aaron: there Jonathan: been some work done to emulate those things that you don't actually support? Aaron: Yeah. Epoll is a great example. We actually have, ... So let's think. Epoll, timerfd, signalfd. There's one more I'm forgetting, but those sort of family of things. We actually have user space implementations of those. It will run actually as part of your application. It will spit up another thread that actually handles all the epoll stuff for your program. Nice. But yeah, we do try to actually have compatibility layers. That was actually really important when I did when I ported Java. Trying to write a, the, like an epoll style, like just doing like poll and select in Java would've been super slow, and I was able to just utilize our epoll user space implementation to accelerate the porting of it, 'cause it's good enough. Yeah. Andy: Interesting. One of the- Yeah. One of the things- Kind of to your- Go ahead. Oh, sorry. Go ahead, Jonathan. No, I was just gonna- I was just gonna say d- to that question that you were asking about like what's the boot process like. It's probably worth spending just a second or two to explain how QNX does it, which ... 'Cause it's very different from what a Linux or Windows world is like. And so basically when we start up, the only thing that's really running is the kernel. And you, as the system architect, and because we're an embedded system, there, there's always this concept that there is going to be somebody who's like planning out what the system does. So we don't really have a, out-of-the-box desktop model, other than some of the, QDD stuff that, that Aaron works on. But in order to do that, like you start with absolutely nothing. So if you want a disk driver, like you wanna actually see files on a disk, you start up a disk driver. If you wanna actually talk to a terminal, you start up like, like a serial port and, so there's ... You actually literally start every service that you want- ... in the order that you want them, and that's like one of the things that's used to get us super fast boot times too, right? So like- Sure ... in cases where hey the car starts, you don't wanna wait for three seconds while, like the whole thing is initializing. It's like you wanna actually start the car and go. And so that gives us, like the ability to get enough of the system up and running in, however many, 10, 15 milliseconds or whatever is required to do that. But because of that, it is super different from, yeah init D, R- RCA, all those kind of things, right? Jonathan: Yeah. Y- so Aaron mentioned something a minute ago, and that was his his Raspberry Pi. And I am- ... I am, I'm a Raspberry Pi geek. I've been a fan of- Nice ... of their stuff just as long as it's been around. I added up one time how many Raspberry Pis I had in my house and, lost track after a couple of dozen. What is the, what does the process look like to run QNX Everywhere on a Pi? How difficult is that? Aaron: I'm gonna say it's dead simple 'cause we just directly support it. That's the cheap answer. From a process, I guess probably look at what a process of supporting it would be. The nice thing is Andy kinda started off saying, "Oh, the kernel is the starter thing." There actually is one more thing above the kernel that is our, was the BSP. It's our startup code. Okay. So right above the ker- the kernel is not the thing that actually initial- initializes the CPU cores. It does- it doesn't actually do that level of initialization, the startup code does. So the startup code is the first thing that hits. So when you're flashing an RPi you have- we have a little syntax on how to say I want this startup code and then this kernel with these arguments. The startup code will run, do a- it'll actually set up the hardware. For instance, on the R- Raspberry Pi, it'll talk to the RP1 chip, initialize that, initialize the graphics core. And then from there it'll ... There's a bit of a proto- I'll call it a protocol for conversation. It does a protocol in memory effectively, and then it runs the kernel. The nice thing about that is the kernel you run on my Abund- oh, sorry, my, I say Abundu. My QEMU machine or this board or that board, it's all the exact same kernel. It's just the startup code that changes. Jonathan: Okay. Aaron: So actually our Raspberry Pi startup code is all open source under the Apache 2 license. Jonathan: Very cool. Aaron: Let's see if I can remember this off the top of my head. Github.com/qnx, and then it's BCM, the chip, Raspberry Pi. And they are actually the Raspberry Pi 4 as well. So all that startup code- ... is all open source which is an interesting reference point if you're like, "How do I boot up a Raspberry Pi from scratch and then have a ch- have a core that's initialized?" It's quite interesting to read. Andy: Yeah. I can give you the TLDR version is we have a quick start image for a Pi 4 and a Pi 5. So you can go on and download them and burn them onto a card and then boom, you boot it up. It's pretty simple. Jonathan: Now, the question: Do you have to register an account somewhere to be able to download those? Andy: Yes, you do have to register an account. Oh. Honest- I- so if y- oh my gosh. Y- you, ... So as the ecosystem director of the ecosystem kind of development and trying to get that thing off the ground and make sure that we could, have it a- available in the Raspberry Pi flasher, d- this and this, I'm like, "Look, guys, I don't want any addresses anywhere. I just wanna provide an image you can download." And they're like, "Okay, but w- how do we do export controls?" I'm like, "Oh, God, why are you ... legal guys are killing me on this." And, so yes, you still have to do an email address. I'm trying to get that as simple as possible. I really do wanna get it to just be able to do that, but, just make a, make up a Gmail thing, whatever. I don't care. It's fine. But you need one. Jonathan: Yeah. Export controls, that that's, ... And I get that. I have had to work a little bit with that over the years. They just ruin all the fun. Andy: Y- it, it does ruin all the fun. The, and then, it gets into all these conversations that are really oh, is it a mass market thing? It's if it's ... I'm like, oh, God, y- again, the lawyers start going on about all the details of everything. I'm like, "Okay, can we make this work? Yes? No. And can we make it as simple as possible? Yes or no? Yes? Okay, good. Let's do that." Jonathan: Yep. I- when someone does jump through the hoops, are there some limitations on what you're allowed to do with QNX Everywhere? Andy: Yes. You're not allowed to use it for high-risk applications. You basically take all of the liability of what you're gonna do. Jonathan: Oh, okay. Andy: You're also not allowed to use it for commercial applications. You can't build a Kinix thing and then start selling it, because that's a commercial application. Now, to be fair, we... That- that's the guideline that's in the QDL, the Kinix developer license- ... that we provide. When it comes to startups and proof of concept and, prototyping work and stuff like that, we know that those are, like, sometimes paid engagements. We know that you're building something and you're trying to actually sell a handful of them to see if there's any viability in it and all that stuff. So we really don't care about that. What we care about is look, you're building a product, you're selling millions of dollars worth of it, then that has to become a conversation- Yeah ... because then that engages with the business part of the organization. But if you're just using it to play around with then that's fine. Yeah. Yeah. Jonathan: Yeah, y- and again I'm gonna beat the open source drum because that's what I do. Andy: Yeah. Do Jonathan: it. Companies have solved this problem rather than saying no commercial, because that's, it- this is a thing that's been, like, a conversation in open source for a long time. And so you've got things like the business source license, which technically makes things source available and not open source. But one of the other solutions that I think is better is people use the the Affero public license, the AGPL, which is, that, that sort of closes the, "Oh, I'm running your open source code, but it's on my server, so I don't have to share my changes," right? And so I think there's a, there's an interesting approach to this where as a business you say, "Look, we're gonna open up all the code, but the license that we give it to everyone with is the AGPL, which means if you put it in your commercial project, you also have to release everything as open source. And if you want to put it in a commercial project that you have some closed bits in there, then you have to have a business conversation with us." And that's a, an- another way to go about it. It's an interesting approach. And for those of us that are the real open source enthusiasts that's the way we like to see it done. Andy: Yeah. No, that's fair. And I think, if you're coming at it certainly from the approach of y- you know everything open source and the infectious model of trying to encourage that ecosystem- ... I think that does make sense. Again, I think, we are approaching it and we're trying to actually make sure that, we're true to our word, that we don't you know- go out over our skis, so to speak. Maybe that's a Canadian analogy. I don't know if they use that term in Oklahoma very well. It translates okay. It translates okay. Yeah. Yeah. So like we're not gonna, we don't wanna kinda go all, all out like that then just realize, okay, that's like a terrible thing from a business model point of view- Sure because a lot of those things are done for companies that are fundamentally not the same sort of business that we are. I'm not gonna say that we couldn't ever get there. That's not where we're at right now. That's not what the current thinking is. Jonathan: Yeah. No. That's fair. It is interesting though. So the, one of the other sort of big open source projects that that you guys have worked with is is Eclipse, isn't it? Andy: Yeah. Jonathan: What's the- Yeah, we- ... what's the story with Eclipse? Why did so- so- ... why did someone at QNX say, "Let's give a lot of money to the Eclipse Foundation. Let's work on this source code"? Andy: I think we... So when we had... so we, we support Momentics IDE, which is our, development environment, and that's all based on Eclipse. And I think when we were trying to figure out what are we gonna do from, a software development standpoint?" That looked like really the best choice, and we decided that, th- that relationship was a good one to do. Now, that's not to say that we're not working with Eclipse on other stuff. So like- ... there's the various different automotive consortiums, and Eclipse has one of those, and we're working with them on that. But we also have seen like that Microsoft has taken over the IDE environment, with Mi- and so we're- Yeah, that's true ... we're embracing both worlds on that front. I think, like we're still big fans of Eclipse, to be quite honest, like our tooling on that side is a little long in the tooth. And we had a bunch of Eclipse experts who left the company to go start their own company, and so they started their own, UI framework company. And we were like, "Okay, we still have people to work on that, but like it just hasn't been as high of a priority. And so now it's okay, now there's a shiny new kid in town, so people are chasing that, so you know. I don't know. I don't know where that's gonna be. But yeah. Jonathan: Yeah. Aaron, do you do work on the Eclipse side of things? Aaron: No, I don't actually. I know quite a few people though that do that via some of our... They're not part of the open source team funny enough, but they're part of more of some of the services teams. UProtocol I believe is part of, under the Eclipse Foundation for automotive, and that's a big thing. I know we ha- we have a full-time dev who does internal stuff, but he's also like one of the lead maintainers of uProtocol. His name is escaping me off the top of my head, so I, ugh, apologize to him. But yeah, no, he's a... He works for us and he does a lot of stuff with uProtocol. So that's a proto- that's an automotive protocol that allows you kinda to, I don't wanna say it's like gRPC, but if you're trying to imagine in your head, it solves like the service discovery and protocol between how to like between cars and services and offline support and multiple languages. Jonathan: Yeah. Interesting what does it look like trying to build a community around QNX without the fully open source part? That's gotta be a, that's gotta be a challenge. You can't accept pull requests from your community, and that's one of the ... That's one of the best things about having a community. What- Andy: Yeah ... Jonathan: how does that work? Andy: I think it's probably important to think about what it is we offer and what the community means, right? So it's if you're building Linux like some Linux system, like how much of the code is actually stuff that you're changing in the kernel versus stuff that you're creating on a application side? And, so do you really need to be submitting pull requests to like, tinker with the kernel s- task scheduling- ... priorities and stuff like that? Probably not. Probably that's never gonna happen, right? There's been, thousands of people beating on it, and we have the same sort of dynamic, right? There are very few people who are interested or care about, the really gooey inside, details of how things get done. They just know, oh, hey if I use this software package on a QNX system, I can get this thing done. And yeah, there might be bugs in that and I have to get that fixed, or I want a feature request on that, but it's mostly at the application level or the library level. And that's, that stuff is all open, and that stuff is things that ride on top of our OS. So I think for the most part our interaction with building a community, if you can... I- if you're the kind of open source person that kind of says, "Oh, you know what? I'm not gonna touch it if it's not open source," we're probably not for you. If, like it's the same thing with Qt, I know Qt has had a on-again, off-again relationship there because, they have a commercial side, and it's yes, and they're a business just like us. Like- ... they have to make money they have to figure out a way to monetize what they're doing. But if you're gonna do stuff and you don't need to use that stuff and you can use their free stuff, then great. And we basically are in that same model. We just don't have a open source component to it. If you wanna use it and you wanna do your own projects with it, and you can deal with the fact that we're not, 100% morally within your open source framework, and you can do it for free though, then yeah, then it's actually cool and you can do all these kind of cool real-time things or, experiments with, your Raspberry Pi and, and- all of that kind of stuff. So yeah it's actually been okay. Jonathan: Yeah. Is there at least an avenue where someone can say, "Hey I've messed around with all of your open source stuff. I'm running into this bug. I think it's upstream into the stuff that I can't touch"? Can I go and report a bug- Oh, yeah. For sure ... to QNX as just a regular- 100% ... individual person? Andy: 100%. W- we would probably do that through our Discord. So we have a very active Discord in terms of, we have y- our chief architect is on there. He- I think he's addicted to Discord. He's on there- ... responding to people. People put on questions like, "Oh, how about blah, blah?" It'll be some simple question, and he'll come back with this crazy answer. I'm like, "Okay, Elad, do you... it's okay." So he gives all this detail and all this crazy stuff. So but we have a very active development community. And yeah if you had bugs there, we would track 'em down, submit internal JIRAs, get 'em fixed. Yep. Jonathan: Yeah. Very interesting. So I do to ask, and the question I normally ask is, what's the weirdest thing that you've seen somebody from the community do? And maybe that makes sense here as well. You, you've got a similar question in the rundown here, where they gave me some questions to ask. And it's the other way around. What's the weirdest thing that you guys have done? So maybe both of these are good questions. What's some, what's some- I don't know ... oddball projects? Andy: Aaron, you go first. Aaron: I, I have a couple, 'cause I'm quite free in the Discord. I've... People 'cause kinda going back to your question of building the community, there's a lot of people really enthusiasts who basically we have un- I'll say uncharted water. Maybe the, maybe not the right word, but there's a lot of things that if you wanted, like when somebody ported Kerberos to QNX 'cause they needed the libraries for something. Like- ... that's an interesting thing. How the heck did you get that to work? And then you'll... the software is there, it's just, like, how do you make it work on a new operating system? And people like that discovery, having something existing, working, and then porting it. I am gonna forget the guy's name, but there is some, this is not for QNX 8, but just someone who's having fun in the community. He took the Black, old BlackBerry Passport, so it's running QNX 6, and he's been doing op- he been doing open source development on it, like AI work and drivers. And it's really cool to see people take even some of the older hardware that, that was more, a little more open source. But taking some modern stuff and porting it over, and updating it, Has Jonathan: anybody, has anybody tried to port like KDE Aaron: to QNX? I have had some requests that, to do that, 'cause we have Qt 6 all ported over for self-hosted, so- Oh, ... it's been on my radar personally. But hey, if someone wants to go port the libplasma and see what that looks like, be my guest. I'd be very happy personally. I Jonathan: mean, if you've got, if you've got the Qt libraries, it might not be as much of a lift as I expected So it's, but it's a good p- Yeah, I mean- ... good Andy: portion of the work done ... and it's similarly like we like what is it, XFC or something that we run that runs on- Yeah Wayland that's on top of, QNX. So there's definitely good chunks of things that- Oh, yeah ... already exist. So I would recommend to anybody who might be serious into looking at this kind of stuff, is go to oss.qnx.com. And that's that's our open source portal. So that shows every package that we've ported and/or tested, where it lives, if we're the ones maintaining it or if it's being maintained on another server somewhere in GitHub or GitLab or wherever else. And what the status of it, what architectures that we know it works on, all that stuff. And there's some, I don't know, 2,000 some different packages that are there or y- somewhere around there. So it's actually, it's got a lot of stuff in there that's interesting to see if you, for example, if you're doing AI stuff and you wanna use TensorFlow or MediaPipe or any of those kind of things, that those are all ported. And you can just find what stuff's been ported and what stuff you've got access to. And those are all things that you can either get and, clone 'em onto your machine or, use the APK tool like Aaron was talking about and get access to 'em. So- ... makes it pretty simple. Jonathan: Yeah. I will tell you my most interesting QNX story, my only really QNX story. Okay. I've got a I've got a dbx DriveRack. It's a speaker management platform for doing pro audio. And turns out that I'm pretty sure it actually runs QNX inside of it. And I know this because this version of QNX, somebody accidentally left the debug port open and you can- Ask it to debug bash for a shell. Ask it to de- debug shell for you, and it will gladly do that. Which, means that you send it a single packet and you pop a shell, and you've got root access to it, which is fun. So I've crawled around inside of that, inside the QNX system inside of it which is yeah, it was quite an experience. That's been around for a long time. That is some- Oh, yeah ... old software. Aaron: Yes, I'm really curious what version it's running, 'cause it must be six or four if it's been around for a long time. Jonathan: Yeah. I, at one point I had the the actual CVE. I don't remember the CVE number that it was. I could I could probably find it again. I'm not... My Google fu is not quite good enough to pull it up right now. Andy: No, Google Flu is actually getting ruined too, because it's like you get results that are like, what that means nothing. Like what year did Biden die? Oh, 2039? Wait, what are you talking about? Jonathan: Yes. Just all hallucinations. Absolutely. Andy: Yeah. E- everything pure hallucination. Yeah, the Google Flu d- means nothing. Yeah. So Jonathan: I will tell you- Yeah, my- ... speaking of hallucinations, when I Googled for that- ... Google's AI told me that both QNX and DBX are owned by Harman. I'm like, "I don't know that's true." Andy: Was true. It was true actually for, so I was here during that period. So 2005, I think we were acquired by Harman, and I think Harman sold in 2010 to RIM, to Research In Motion. So there was a period of time when we were doing a lot of stuff with Harman yeah- It was probably during Jonathan: that time. I would... That would be very likely this is during that time that, that, Andy: that change was made. Probably was very likely during that time. Yeah. Yep. Yeah. And we were in like some guitar pedals and yeah, a bunch of cool applications, you Jonathan: know? QNX on a guitar pedal. That's- Yeah. I can't say much. Linux runs in some really weird places too, and, Andy: Yeah, Linux runs in weird places. And we're in things like space arms and nuclear power plants, so we're in l- we're in some pretty niche applications. And my crazy stories tend to come from my first stint around when I was at QNX as a FAE, and, finding out that it's oh yeah, this whatever it was, a coal plant or something like that, hadn't rebooted their system for 10 years, and then they tried to reboot it and it was running off of a floppy. The entire system was running off of... And the floppy wouldn't work. Yeah it hadn't been used in 10 years. So I'm, like, helping this guy who's frantically panicking as you know- ... the entire plant is brought down because they haven't even tried to run it. I'm like, l- on eBay looking for floppy. Okay, what is it? It's a model, something that, and I'm like, "Okay, I think I found one." That there's one in Arizona somewhere. We can get them to overnight it, Jonathan: like- Andy: Yeah, so- Jonathan: I- is- Yeah ... is QNX still a part of Research In Motion? Is that still the sort of corporate overlord? Andy: Research in Motion renamed to BlackBerry, and we still are a division of BlackBerry. So there's basically, there's two main divisions. So one is QNX and the other is Secure Comms. So if you think about what, what made the BlackBerry phone kinda all, secure and, its own very highly protected ecosystem, all that software, they've taken all of that stuff and turned that into a business. So like governments large corporations, institutions, all those kind of big company players that really value corporate security and those kind of things, they're customers of Secure Comm, so our sister company. And then QNX, we- we're doing the basically the same thing that we've always done. We just don't do it on phones, Jonathan: so they- they've- they've forked it. They've split the streams. That's right. You guys are no longer doing the phone stuff. Ah that's good to know. Andy: We're not... Yeah, basically as of whenever BlackBerry kinda folded up shop on the hardware business, then, we stopped sup- supporting the phones. But yeah, like we, even through all that whole period, we were still doing tons of automotive work, lots of robotics, industrial, all that kind of stuff. Jonathan: Yeah. Super interesting. What as you look into your crystal ball of the future, what what interesting things do you see coming for specifically for the, like the QNX ecosystem? What are some things people should be watching out for? Andy: Yeah, that's- This- That's actually a really good question ... Jonathan: this is where you can make your earth-shattering announcement that you're gonna open source it- Andy: Yeah ... if you want to. Yeah it probably won't be that earth-shattering. I like I- I'll tell you honestly, I think that one of the biggest challenges that, that I am trying to struggle with is just what are we doing with AI? And there's a lot of smart people in the company that are trying to figure that out, and we're looking at lots of different approaches to that. One is that we think we're, like, the ideal platform for people implementing physical AI systems because we provide all the safety and the real time and all of these kind of characteristics that you need. But we're also POSIX compliant, and so therefore, all of these AI things that people are experimenting with will run on our system. And like, when, Aaron had mentioned we had just gone to the Berkeley Hackathon, and yeah, that it was a AI-based hackathon, and we had people using our stuff to, implement, tank detecting or, looking for patterns outside cars or, doing whatever, doing games, all kinds of interesting applications to AI. So one of them is enabling AI, and that I think we have a relatively good handle on. But what I don't have a good handle on is what are we gonna do from a tooling standpoint to enable people to be better using AI with our platforms? And it's a little tricky for us because our entire brand is all about safety and security and reliability and stuff like that. You don't put us in a wind turbine 'cause you think, so you vibe coded what that is, and then it kind of- ... starts feeding the wrong current back into the, thing at the wrong phase and totally blows out the system or whatever, right? But there's all these serious consequences to this stuff. So- I think that there, there's that, but then there's the realization that it's okay people are gonna be using AI and we need to figure that out. And we're kinda undergoing that exercise ourselves of trying to understand, okay, how do we take AI and merge that with a lot of the lessons that we know about- functional safety and certification and process and all of those ways that you can use to kinda get a handle on a system. Yeah. And in some ways, it's very similar to just kinda hey, what if you hired a bunch of interns that are really eager, and you just set 'em at a task. How do you- Absolutely control their energy, right? Like, how do you vet that they're doing something right? And I think I- we're gonna come around to that, and I think the industry will come around to figuring that out. But- ... that's the biggest challenge, and I think that's the biggest place for us to kinda make a splash. But we're not quite there just yet. Jonathan: Yeah. Aaron, can we run Python and PyTorch on inside a QNX Everywhere? Aaron: Oh, of course, yeah. PyTorch. Let's see here, what's on my list? Llama CPP. I know we have Stable Diffusion, the PR up for that coming out in a minute or two again, it's one of those things. You, you name it, either we got it or tell me I don't and I'll make sure it works. Somebody's Andy: working on it, yes. I'll send you... Yeah, I'll send you a blog afterwards that- Okay ... that has all of the things that, that are AI related that we've got ported, and it's a list of a, I don't know couple, three dozen different packages that are in that space that, that would be useful. Actually, there's a couple blogs that I could send you that, that might be useful for that. Jonathan: Yeah. And something we briefly touched on this earlier, but I think it's interesting. Can I take one of the QNX images for... Obviously I've gotta make an account somewhere, which I'm not super happy about, but I use a burner Gmail address. It's fine. Can I take one of those images and run it a inside a virtual machine on my Linux machine? Play with it- Yes ... that way I don't have to, I don't have to dedicate any hardware to it. Yeah. And then does it work the other way? Yeah. Can I run a Linux VM inside of a, inside a QNX Hypervisor? Andy: Yeah, you can. Yeah. And a QNX Hypervisor is also part of the QNX Everywhere. The stuff you get for free is part, the Hypervisor is part of it, so you can do that too. Jonathan: Okay. Andy: Honestly, there's also a bunch of stuff about Graviton and running an AWS and, running in Azure as well. I- ... I'm not as familiar with that. A lot of the stuff that, that I've and Aaron do are, like, a lot of university stuff, but I do know that, people use our stuff to run in cloud instances as well. I would not- But if you start asking me any kind of detailed questions there, I'm gonna get way out of my depth instantly, Jonathan: I would not have thought of running QNX in the cloud. That's actually really interesting. Andy: I- It is interesting. It's, people use it for a scalability thing. And if they wanna have a whole bunch of developers work on the same kind of system across time zones and across the world or be able to scale up how many instances are running or not, there's a lot of reasons to be able to say, "Oh, hey, yeah, you wanna actually use this particular..." Let's say you're developing a, a digital instrument cluster or something like that, and you've got developers all around the world that wanna be working on it at the same time. It's easy to do that in a cloud instance, whereas it's tricky to do that with a physical piece of hardware, right? And plus you can get out ahead of, the needs. Like we're very much an embedded company. Almost all of our applications are embedded in some way, but that also means that you're tied to a piece of hardware. So as soon as you break that you can say, "Okay. Let's get this working in the cloud first, and then as soon as we get the hardware shipping, then we can start tying it." But otherwise you're waiting for hardware to come and it's "Okay. We can't run anything, so what are we gonna do?" I... it's, it just helps accelerate that whole thing. Jonathan: Yeah. I've gotta, I've gotta ask, and this is a meme-y, troll-y question, but Andy, did you inspire any- Awesome ... did you inspire any of the characters in the Blackberry movie? Andy: No. Jonathan: Were you around for that time period? Andy: I was around for that time period, but I- I actually, I left after that and then I came back. So yeah, but I was around during the kind of the acquisition and the phone sort of thing. Yeah some interesting times, that's for sure. Y- honestly, because I, at the time I was on the automotive side- ... and they had this really l- like this firewall of stuff. So it's like we knew that half the company was being hived off for something. And for a long time we didn't know what it was. We were like, "What the heck is going on? Are we being sold? Are we being bought? Are we..." We had no idea. And then it was then there was the announcement like, "Oh, okay, now it all makes sense." And that's why I couldn't talk to this guy for four months, yeah. Jonathan: That's hilarious. Yeah. Andy: Yeah, it was pretty weird. Jonathan: Yeah. Fun. All right, so before I let you go, it's been a, it's been a fun conversation. I gotta ask you, each of you though what's your favorite text editor and programming language? Let's go, Aaron: I'll, Andy: I'll go Aaron: first here ... Aaron's geo first. So I used to say VS Code, but because VS Code does not run on QNX, I am now a Neovim user. Jonathan: Ah, there you go. Aaron: For favorite language, I've been a JavaScript fun- I've worked in a betting company. I've been a JavaScript developer for a long time, but I've really come to appreciate Rust and those newer languages like Zig have been very interesting to see what they're doing in the space and modernizing embedded development. Jonathan: Yeah. Have you guys done anything with WebAssembly? I, that, that's something that really blew me away- Yes ... is that people are doing WebAssembly in, in embedded. And we're actually- Yeah ... in, in my company, Mesherastic, we're actually doing sort of that in reverse. We're taking some embedded code, compiling it to WebAssembly, and then running it in the browser or, on Android. Yeah. All kinds of crazy stuff. I'm just blown away by- We- ... by what you can do with WebAssembly. Andy: WebAssembly is pretty cool. We were talking with some people at Carnegie Mellon, and they were super into WebAssembly. And they were like what do you guys have for WebAssembly? What can we do there? 'Cause we, we think you got a really cool product, and we want to do all this WebAssembly stuff." And then we do support it as a outshoot of the fact that we have a browser, but we don't have ... we're not natively doing anything in it. So it's when they're talking about "Oh, can we do research projects with it and stuff?" We're like you can, but we're not ... We don't have a vested interest in it because we don't have any customers using it." But, we'll keep our finger on the pulse there, but yeah, I don't know. It is pretty interesting. Okay. So I'll answer those two questions with regard to favorite editors and stuff like that. Favorite editor that's like a weird sort of a question in my mind. I've got two answers to that- ... and you'll ... You'll probably say, "Oh my God, this guy." So one of my answers is VI. And okay, now please hear me out on this one because it just works everywhere. I can go anywhere, and I can do what I need to do with VI. You can tell I'm not- So it's like I'm not, like- You could tell that into a Jonathan: router and it'll be there Andy: Ex- exactly. Like I, I can always depend on it being there. Yeah. So that's But like for day-to-day use, oh God no. Like I used to know a guy who I worked with and like he did that with his main editor. And I'm like, okay, he was astounding. Like he knew all these colon commands and he could do crazy stuff. I was like, I would just watch him going, "Wow, if I knew like half of what you knew like that would be amazing." But I can get by. But Xcode, and I'm sorry, yes I do Mac and I'm, my next thing is not gonna surprise you, Swift. I love Swift. It like, it's just so well-designed. Like I just, I can't get over how nicely designed of a language it is. And now it really irritates me that the interfaces change like every flipping release. But other than that- that's a very Apple thing to do, Jonathan: I think. Andy: Yeah. They're just trying to make it perfect. So then of course why not break everyone's code every single time that they release something. So- Of course ... yeah. Jonathan: C- can you run Swift on QNX? Andy: No, but I want to. Aaron: Oh man, that would be amazing. I know, but the thing is that Swift is finally getting Linux better Linux support. Relatively recently. I remember there's a whole thing with Ladybird was looking at using Swift, but- ... it just wasn't there yet, I believe is what it was, if they get a little support Aaron, I had a couple of- ... maybe I'd love to do it ... kids come Andy: up to me and say, "Can we get Swift on, on QNX?" I Aaron: know. I- Andy: And I was like, "Yes! I found my people." I, Aaron: I love, every language we add to QNX just makes the ecosystem better and better, right? P- pick a, I don't care what language it is. Totally Andy: agree. Odin, Aaron: right? I take Odin, it's for 3D graphics and everything, but hey, people, auto- auto manufacturers need a good 3D programming language and do stuff like that. So yeah, every language you put there, let's do it. Yeah. It's fun. That's Jonathan: fun. Yeah. That's great. All right. Thank you, guys. Yeah, that was really cool. Thank you, guys, for coming on the show. Thank you for being patient with my questions and my badgering. It's been a lot of fun, though. Andy: It has. It's been an absolute pleasure. Thank you. Thank you so much. Jonathan: All right. Good to have both of you. That is Andy Green and Aaron Bassett from QNX, and it's been a bit of an oddball show, but that that's fine. We've had a lot of fun with it. We've got some fun stuff coming up as well. Next week we are scheduled to have Andrea Gallo. We're doing the complete opposite next week. Not only is the operating system open source but all the way down to the ISA, we're talking about RISC V. And then the week after that we're gonna talk about JavaScript and hopefully some WebAssembly because we've got Neriman Jelva of Pewter, somebody else I met at the Ubuntu Conference. And then after that we're talking with Michael Meeks of Collabora, and then we're having Francois Pol again, talking about smoked meat. That is security, specifically security on GitHub. A fun schedule coming up. You don't wanna miss it. And we will see everybody next week. Just wanna say thank you to everybody that watches, and yeah, we'll see you then
• Episode 873 - Wait, That's Not Open Source! 01.07.2026 1val 3min

  This week Jonathan chats with Andy Gryc and Aaron Basset about QNX, and the interesting Open Source history and future of that embedded OS. Why does QNX Everywhere feel more open, and why do you need to register an account to download images? All that and more -- Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• Episode 872 transcript 24.06.2026

  FLOSS-872 Jonathan: Hey folks, this week we're talking with Tris Willaker about open source and the law, including but not limited to topics about the GPL and legal cases, what AI means for lawyering, and the big court case over who the real Satoshi Nakamoto is. You don't wanna miss it, so stay tuned. This is Floss Weekly, episode 872, recorded Tuesday, June the 23rd. I'm not Satoshi It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today, oh, I've been looking forward to this one for a while. Today, we're gonna be talking with Tris Wilcher, and Tris is a friend of mine, a new friend of mine that I met at the Ubuntu Summit. And so you have to imagine that b- betw- at, they would do a couple of sessions, and then they would give us 15 or 30 minutes. And the, the tag on that was, "Go network. Go meet people." And I am an introvert at heart, so I have to work really hard during those times to go and meet people, but I was doing my best. So I start walking around and talking to people, and I walked into this conversation, and it's like there was one person there I was wanting to talk to, and that we all introduced ourselves, and Tris was the other person. I didn't know him yet. And Tris said, "Yes I'm a lawyer." And I went, "Wait a second. You're a lawyer, and you're at the Ubuntu Conference of your own volition? We need to talk. I need to know more about this." And so I got to talking to Tris about some of the things that he's done, some of the cases that he's been on that he is allowed to talk about. Sometimes in the legal world you're not allowed to talk about things, but there's some of these that happen in the open, and so you get to talk about them. And then just his other open source stuff. Tris was a real pleasure to get to talk to, and I said during that conversation, "I need to have you on Floss Weekly," and he agreed. So without any further ado, Tris, welcome to the show. Tris: Thank you so much for having me. It's a pleasure to be here. It's a sweltering day here in London- In the UK. We've been issued with a cataclysm warning of how hot it's gonna get, so you'll have to forgive me if I gasp. Jonathan: It was warm. It was warm when I was there s- surprisingly there, we Americans, we think of, sunny Britain. It's so far north, it's 70 degrees Fahrenheit all the time and just beautiful. It w- it was kinda warm while I was there. Tris: Yeah, it's a high fever s- temperature right now, so you'll have to forgive me if I gasp. Jonathan: Absolutely. All right. So Tris let's start with I think your background would be interesting. So how did you end up in this sort of interesting position where you're both a lawyer and a software engineer? Would you claim the title software engineer? At least canny. At least you understand what's going on. Tris: I do release my own open source software, so yeah, I think so. I've got no formal engineering background, so it might be a steep claim. But it's not like I'm vibe coding anything Sure So I'll accept it for the purposes of today. Jonathan: Sure. Tris: But yeah, my day job what I do day to day, I am a lawyer. Originally my background was in engineering. That's what I studied at university. It was not software engineering, but a m- mixed discipline. And then I converted to law and IP and tech law specifically, and that's where I've worked at the interface of law and tech for my entire career now. Jonathan: What, what led to that decision to, to switch from engineering to law, to legal? Tris: In fact, there's a picture just behind me on the wall which I drew when I was maybe four years old, and it says, "When I grow up, I wanna be a lawyer." And I fought against that for some time. But then I I couldn't resist the pull of it anymore, and I fell back into that. Jonathan: That's great. That is very cool. It is a, it is an interesting field, isn't it? The sort of... Because you've got over a thou- especially in England, a thousand years of history in the law, and then it's something that is at the same time so present and it affects every... Whether you realize it or not, it's got effects on every part of life, and- Yeah ... people that have devoted their careers to thinking about that. I definitely see the appeal. Tris: Yeah, you're absolutely right. A- and I think we're quite lucky as a generation to have liv- have lived through both the dawn of the internet and the dawn of AI- Yeah Whi- which have- Yes ... bookended my life to date. I don't think a lot of people get to live through two revolutions in one lifetime, right? So- ... being a lawyer in that context is very exciting because although our statutes our actual laws are old, our computer copyright law is from the '80s the law is all about keeping up. It's all about developing and reacting. And that's where I work. I work in disputes in courts in, in crisis situations, and we're always trying to think ahead, adapt to the changes. So yeah, I, I totally agree. Very interesting field and a wonderful partner for a tech interest. Jonathan: Yeah. V- not very many years ago I thought rather regularly that the internet is the next big revolution and it's equivalent to the printing press. And in my naivety I thought, "It'll be another thousand years before we see another revolution like that." Oh, how I was wrong. But we're accelerating. Tris: We're accelerating. Jonathan: Although I think probably, years from now looking back, people will probably see the internet and AI as basically being the same event. That- Oh, yeah ... being in the midst of it, they're very different, but history will probably see them as basically the same thing. Tris: One precipitated the other, for sure. Jonathan: Yeah, absolutely. All right. So you you, you do specifically copyright and IP, intellectual property law, that sort of thing? Tris: That's my specialty. And especially where that touches software and tech. It can be in the kind of case that we got chatting about when we were when we were- ... having our introduction at a printer conference or it can be somebody running off with source code and it's proliferated, it's been pushed out to I don't know, Git repos all over the place a- and controlling that, understanding the architecture. That sort of thing. And when there are new issues coming up, you get to learn a lot. I'm always learning from intelligent people who've come up with new concepts from businesses who've started new ways of doing things. A- and it's just a real joy to be in that world and, a- and help it progress if I can. Jonathan: I- is AI causing some ad- additional I don't wanna say headaches, but complexity in Tris: the legal world? Yeah. It's certainly causing a lot of complexity and headaches for for lawyers who don't understand it. I'm at Bird & Bird, a firm in London and EMEA w- where- Al- almost all we do is tech related in some way or another. And we act for a lot of AI companies, and we act in a lot of AI cases. There's been one big case which I can't talk too much about- Sure ... but is out there for and did catch some headlines. It was a case between Stability AI, w- who is well known for Stable Diffusion, and and Getty. And we acted for Stability, a- and we were victorious in that case. But it was a fascinating lesson in how your mental model of the law doesn't necessarily match up to reality and finding that out in real time. That was, I was lucky enough to be part of that team. It was run by my colleague Toby Bond who does a great deal of cutting edge AI work. Jonathan: I've seen some really interesting things with AI. I think I've mentioned this to you in another conversation, but there's a website. It's satire, but at the same time the point they raise is very interesting. It's Malus, M-A-L-U-S.S-H. And this website it's basically claiming to be clean room as a service. We will take an open source project, this is exactly what they claim to do. We'll take an open source project and we'll turn the AI loose on it, and we'll have one AI bot create a documentation for it. And then we'll take another AI bot and have it look at the documentation and create all new source code, and then you can put whatever license on it you want to. And they're basically making the claim that the, the modern rise of AI is going to... And I've heard people say this, "It's gonna kill open source." I'm... I think it's i- imminent demise is maybe a little overstated there. Have you had to look at things like this? Tris: Yeah. Yeah, absolutely. In fact, the Malus site we did chat about it. And the next, the very next day actually- ... a friend of mine sent it to me saying, "Oh, my God, have you seen this? It's crazy." And the reaction from the people who didn't take it as satire- ... was really alarmist. A- and I think what that tells you is that it is hitting a nerve and it is satire done as well as it can be, right? Yeah, absolutely. It does take a while for the concepts to land even once you process the joke. And I think that's for good reason because there's a philosophy about code and there's a philosophy about copyright. And where they interact it can be very difficult to draw the line. Now I don't think that the satirical recipe for a clean room is a complete recipe. There's a fair philosophy there. A- and I think probably done the idea of understanding a process and then clean room coding a process- Could work. But I'm not convinced that an AI pipeline is a foolproof method. Jonathan: That's probably fair. Yeah, I think I think there are some open questions just in general around what an AI does. Is it sufficiently transformative to break the copyright from the training data? And, there's some cases where you can get an AI to spit out, character for character some of its training data, and in that case, obviously it's not broken that, it's not transformative at all. And then w- the other really interesting thing is the output of AI is, at least in some jurisdictions, not considered copyrightable at all, which I don't know, that has some implications for businesses using it. One of, one of my trips recently, I sat beside a comic book artist, surprisingly. Really surprised me to have this conversation. I asked him "Are you guys using AI in your stuff?" He goes, "No, not at all, because we can't copyright it." Oh, that's interesting. Tris: That is interesting. I think if you're using code as a tool, then you probably don't care whether you can copyright it or not, if it's just an internal tool. But if you're if you're writing code as a product or as a service- ... then that is a pretty big issue. And it extends to all creative industries, right? Yeah. I don't think we should lose sight of the fact that software engineering is a creative industry. Jonathan: Absolutely. Speaking about people being creative in software engineering, There is one case in particular that you have been on top of. In fact, I think you were the technical lead there at Bird & Bird about this case. And that was when Craig Wright claimed to be Satoshi the mastermind of Bitcoin. And this was the story... the case that we got to talking about. And some of the things that, the stories you had from it were just so fascinating. I, I wanted to I wanted to go into this a little bit. Can you give us some background? I think most everybody knows what Bitcoin is, but starting from that point, what, what happened? What was this case about? Tris: Yeah, I'd be happy to set the scene. And before I do, I'll say that we're very lucky to be able to talk openly about this case. This was a case about showing the truth to the world. A- and everything was done, as you put it, in the open. We're also able to see the outcome of it from, in a very long and detailed judgment- ... resulting from the case. So with that in mind I think the best way to set the scene is probably to remind ourselves that open source is a community, and I feel every community has its myths and its folk heroes and even villains. And I think if Satoshi Nakamoto i- is the mythical hero of the story for sharing his invention with the world back in 2008- ... releasing it under the MIT license, and then instead of becoming its BDFL stewarding it for a couple of years and handing over to the community before vanishing into the background then, That Jonathan: vanish- that vanishing act has raised a lot of questions among the community. So I pitched the idea while we were talking about this, like I think Satoshi wasn't a real person, I think it was an intelligence agency, and I've heard other people say that. And as part of your work on this case you said probably not." Tris: I'm of the opinion that Satoshi was a single person. I was exposed to along with others in my team- ... we were lucky enough to be exposed to potentially as much information about Satoshi communications with Satoshi a- as anyone ever has been- ... who wasn't his closest confidant. A- and in order to be able to honestly ask to see that f- for the purposes of let's be honest, justice- I gave a commitment to each person that sent something to me to say that I would never use it to try and dox Satoshi, and I've kept true to that. I've never actually tried to investigate who the real Satoshi is. So while my opinion is that Satoshi's a real person, not an agency I have n- I have deliberately never indulged i- in the temptation to, to scrub about. Jonathan: Absolutely, and I wouldn't ask you to. Tris: No, of course ... but there is Jonathan: one person in particular that you can you can say with some confidence that is not Satoshi. Tris: So if w- there's actually only one person I'm aware of in the entire world who is by law proven to be not Satoshi. A- and that is, yeah, Craig Wright. So he's if anything- the villain of this story. He was certainly found by the court to be dishonest in many ways. But the core of that dishonesty was the methodical staking of his claim over the course of many years that he was Satoshi, that he was the creator of Bitcoin. A- and true to this myth he built up his persona in a lot of different ways. And to my mind, he was quite clever in that he focused on the narrative- ... on, on selling the dream. And I think from what I can tell he definitely convinced some people. He, he convinced people who funded him. He had a great deal of monetary backing. He started off by doing a kind of signing demonstration- ... to, to s- to journalists a- and even to one of the key Bitcoin maintainers where he dressed the whole i- idea of a, of a PGP signing process up into this dramatic event. A- and it was very theatrical, and I'm pretty sure it was faked because a- anyone who has used PGP will, will know that you can just sign something and it works or- ... or not sign something a- and you've got no proof, right? But he began to convince people which was fine until he escalated. And what he did was he moved from shelling a story fr- from the sort of reputational side of it to actively attacking others. A- and I don't mean verbally attacking or physically attacking. He, he did do certainly verbal attacking. But the really problematic ones were when he started to interfere with people's lives, with their rights by suing them personally. And he was suing people, right? Not just companies, but people, actual cryptographers, journalists, a- and open source developers who doubted him. He would take them to court to, to prove that they were wrong to do so or that they had libeled him. And with all of his financial backing it came to a head because he began to win, right? He began to- ... get judgments against people, that they had defamed him by the way they had called into doubt his claims. Jonathan: Yeah. I have a ... I wanna interrupt and ask something. He had financial backing. Tris: Yeah. Jonathan: Why would Satoshi need financial backing? Doesn't the man, whoever he is, or woman for that matter, I don't know own like 1,000 of the first minted bitcoins? Isn't Satoshi like unbelievably wealthy? Tris: Yeah. Why indeed? A- and there are many holes in the story. I think if, f- out of all of them, I feel like that is the most forgivable hole in the story, because you can imagine that i- if Satoshi who set up... Let's remember, Satoshi set up Bitcoin in its early days- where it was an unstable system and more vulnerable to a 51% attack. He did so by effectively sinking mining energy into it. A- and if he started to move those coins, which were effectively, as I see it, ballast in the system- ... Tris: Y- you can see that might be considered destabilizing. So perhaps of all of the of all of the questions, that is the most reasonable. A- and true to life those coins haven't moved, right? Jonathan: I remember at the time hearing critics of his say, "If the man wanted to prove he's Satoshi here's the real easy way to do it. Spend one of the first bitcoins, and then nobody would have anything to say about it." Tris: Just spend a sat. Jonathan: Yeah. Spend- Tris: A- and of course, there was this there was this principled response to that whi- which was enough. And this was really important, right? Because what Craig Wright was doing was selling the narrative. It was enough to sell the narrative- ... to say, "I, as a matter of principle, won't prove it cryptographically, because I want to prove it with different evidence." Okay he failed to do that but that was his story. Jonathan: That's a weird that's a weird place to stand on, but okay. Tris: Yeah. It... But if you've got no keys, then that's what you're left with. Jonathan: True, Tris: true. But you highlight a really good point, a- and the point you highlight I think implicitly is this: you don't need to prove you're Satoshi to benefit from this story. What you need to do is you need to prove that you're enough of a risk That people need to listen to you. So a developer may not believe you. Somebody who's got the technical savvy to understand how cryptography works may never believe you. But their boss may. The boardrooms of the companies which are handling these exchanges may. And it doesn't really matter whether you believe him or not because if he sues you, the judge may. And I see that as almost a form of social engineering, right? So social engineering works by taking power from a person at its core, right? By deceit. And I think if you can spin a yarn and get a judge to believe it, then you've effectively got the judge to award you that power just as in the same way an employee might give you a password into a corporate network. Jonathan: Yeah. Tris: It's an inter- it just resounds much more widely, right? Jonathan: Yeah. I suppose we see that, And of course this is my opinion on these, but we see that with things like junk software patents. And here in the US there was a problem for a while. One company was sending out letters to individual businesses over a patent that covered, I think it was scan-to-email. And so they were sending out- Oh, really? Yeah. It was terrible. They were sending out basically w- letters demanding a $10,000 patent license purchase for an- any company that they thought was using scan-to-email because you're infringing on our patent." And somebody finally- Tris: And let's say you receive one of those, right? What do you do as an open source developer? Y- you can choose to fight it on principle or you can choose to try and settle it or you can choose to pull your product. If you're a small business you might try and settle it. If you're a huge business you might fight it on principle. But if you're an open source developer you're gonna pull your product 'cause it is not worth it. The open source bargain is that you contribute and it doesn't come back on you, right? And whether that's more or less enforceable in different countries I'm not sure, but if you are faced with something that so dramatically changes that paradigm- ... then y- you're gonna just back out of maintaining. Pass it on to someone else, pull the software, archive it, whatever you do you're just gonna say, "No thanks. It's not worth it." And that's what was happening with Bitcoin. Some of the main contributors to Bitcoin were saying, "This guy is a litigation risk against us personally. It is too much. We are no longer going to be contributing to this system as much as we love it because we can't risk our livelihoods." So it's destabilizing and there's a chilling effect. Jonathan: Yeah. W- was there, and you may not be able to speculate on this, but like what do you think Wright's endgame was? Obviously he was trying to amass money and power, but like he he ran the risk of killing Bitcoin, and I ... it's just such a, it's such a weird, it's a weird thing for somebody to do to- Yeah ... to claim to be this and to, threaten all these people. I don- I don't understand it. Tris: I'm not sure there was an endgame. I'll never know, right? Yeah. And you're right to call it speculation but sitting here in my speculation chair I, I'm not sure that it ever would have ended. That there were many irons in the fire. I don't think you need to look far beyond power and money to get to the end of the story. But maybe it would have continued for a long time. A- and thankfully we'll never know, because I hope it has now ended if not in the way he predicted. Jonathan: So y- we talked about this idea of an open source contributor might just pack up shop and go home, and that, that was happening for some of the Bitcoin contributors. What what changed? Was there some big group that said, "On principle, we're going to stand up to this and stop it"? Tris: Yeah, and it is exactly that impact. So the Crypto Open Patent Alliance is an alliance of, B- blockchain ecosystem companies. N- not just in Bitcoin but Bitcoin is obviously a foundational technology. A- and these are by and large a group of competitors. Th- they're competitors in their industries in many ways- but they come together to form this alliance to share patents, which is why it's called the Open Patent Alliance- ... or although we always abbreviate it to COPA to share the patents among COPA members- ... a- and pr- promote stability in a system because this is a system built on open source- and these members are all about keeping the system going allowing everyone to benefit from it, a- and making it work i- in a way which it won't unless there is stabilizing a sort of stabilizing economy behind it. So these guys sat up and took notice because when you when you are starting to discourage people from contributing their intellectual effort to the foundational open source system, you're destabilizing the whole mountain that's built on top of it. And we've all seen that XKCD cartoon. We know what happens when that domino collapses. So the th- these guys stepped in, and they came and they spoke to us, and we put together a team at Bird & Bird, and together COPA and Bird & Bird we started this case to try and prove that Craig Wright's story was false- a- and to try and show the world that was the case. So it was actually led by my colleague Phil Sherrell, who heads up our our London headquarters a- and there were several of us. I will say that I will say that the guys on the other side had more than twice our number. So we often felt we often felt like we were underdogs in this fight. Jonathan: And Bird & Bird is not a small firm. That's probably not something you guys are used to, being the underdogs in a fight. Tris: Yeah. That's right. We are often pretty lean in how we handle things. So we like to put together a team where y- you know the ideal the dream team scenario, where everyone's got their place, and is an expert in their field, and we all integrate. Jonathan: Yep. Tris: And that worked really well in this case. It was an absolute delight. So I ended up leading on the technical side and the- ... forensic investigation, some of the technical factual investigations too, which we might come onto. My colleague Ning who was magic with the more open world factual i- investigations. And then before I joined the team Graham Smith who you may have heard of, he's been an internet lawyer for decades, since the dawn of the internet. A- and he actually created- And by, and Jonathan: by internet lawyer, you don't mean someone on the internet that thinks he's a lawyer. You mean an actual lawyer- I mean- that does internet law. Tris: The law of the internet. He's seen it all. Yes. ... and he's still very active. And he'd actually put together this amazing database, I think a SQL database- ... drawing all the threads of the factual stories together and cross-referencing to see where they didn't add up before the case. I- it was a really amazing team, and we were able to keep it quite lean by, by segregating everyone's specialties. Jonathan: Yeah that's super cool. What, what did the actual legal structure of the case look like? So like you can't just, from what I understand, you can't just sue someone because you think they said something that wasn't true. What was the actual- Yeah ... how did s- how did standing work? And and obviously for those of us in the US it's gonna be a little bit different because this is British law. It was in a, an English court. But how did that part of it work? Tris: It's a fascinating legal question, not one that most people usually touch on, so good one. A- and if I can try not to get too technical, you're absolutely right. You can't just sue someone because you don't like what they say. That's not how the law works. And the question of standing was very important. But in the end it came out all right because Craig Wright had embarked on a campaign of claiming copyright in the Bitcoin whitepaper itself. A- and as anyone in Bitcoin knows, it's a sort of mark of being part of the club that you host that whitepaper on your site- Jonathan: Ah ... Tris: to show that you're part of that ecosystem. So anyone who was hosting it, he was writing to them and complaining that they were infringing his copyright. So we checked with him, we wrote to him and we said "COPA's hosting the whitepaper. Are you saying they infringe your copyright?" And he said, "Yeah, I'm gonna sue you," at which point we, we were able to take it to court to prove- ... that he didn't own that copyright. A- and who owns the copyright on the whitepaper? Satoshi. A- and it boiled down to the same question. In fact it was a bitty, a bit of a tangled ball of wool. There were many threads altogether. He was suing open source developers and also the companies who happen to be COPA members. A- and it did coalesce into one huge case w- with one central factual issue. Wa- was, Jonathan: Was that a challenge to get the court to bring it all into one case rather than have to fight him 50 different times in 50 different cases? Tris: Yeah. I can't say exactly how the court would've approached it, but if I was the judge seeing this, I would've said, "That is a tangled mess. How can I shake it out?" A- and maybe that's what the judge did. It's certainly the effect of how the judge handled it. So we did coalesce the cases into one trial. Certain aspects were stayed because they were gonna fall away if the main case resolved the way we wanted it to. And it all came to a head after a few years in one big six-week trial which was an amazing trial for a few reasons. One is that it was held in the biggest courtroom in the building- With everyone piling in. And the seats at the back of the public gallery totally full. Two, because it was actually live-streamed, and i- in a, I, in a term which I've never heard of before or since, there were more than 1,000 people following the live stream daily. Which is pretty thick stuff- in a court of law if you're not a lawyer. Hats off to those guys and girls who dive in every day. Jonathan: Yeah, absolutely. Going into this, was there a part of you that thought maybe Craig Wright is Satoshi? Maybe he's Tris: Wright? Yeah. A- absolutely. In fact a- as a matter of principle. I'm a lawyer, so I don't have any skin in this game. I came to this totally cold. I don't have any Bitcoin. I don't know about the technology, and I'm learning from everybody around me. And I don't know the difference between Satoshi and not Satoshi at that point. You come to it with objectivity. You come to it open-minded. And you see the difficulties, but you can also see how they might be surmounted. And it's only as you begin to go through the evidence, the mountains of evidence that he dumped on us, a- and you start to see that everything that supports his claim turned out to be a forgery- that you start saying, "Yeah I'm beginning to make up my mind now." So it was a process. It was absolutely a process. Jonathan: How would you have handled it? So obviously you personally came to the conclusion, and I know this from talking to you, you personally came to the conclusion that Wright's not Satoshi and so that made this easier. How would you have handled it being on the side you were of the litigation if you had looked into the evidence and gone, "I think it's him. I think this is him"? Tris: Y- then you would have put the case as best you could, but you would never tell something to the court that was untrue, right? I- it's a bit like asking a criminal lawyer, how do you defend someone who you think is guilty? Yeah. You just have to be very careful. You give them the defense to which they're entitled- ... and and you do what is right and just, and you leave it for the judge to do the right thing. So luckily we weren't in that position. We didn't have to introspect too much. It was all outlooking because it became pretty clear, at least within our four walls- it became pretty clear to me who were the good guys pretty early on. Jonathan: Do you remember the moment or the piece of evidence that you looked at and went, "That's not him"? Tris: So we got his disclosure, I remember, and I did... The first thing I did was I just spun it all up on my Linux box- ... which I actually had to switch to j- just to get through the volumes of evidence that were being poured on us- and to be able to pull tools to analyze it. And I spun it up, and I just I just put some filters through it for the earliest documents that mentioned Satoshi. And I remember very clear, very clearly pulling up the first, I think it was a Microsoft Word document- ... opening up the internals of the document and seeing right then and there within a few moments that the internals of the document had all been sanitized away to appear from maybe 2007 But they had all of the editing artifacts, u- unfortunately for Craig, still compressed into the document itself. And it's all in the forensic reports. You can see the references to the Financial Times from 2016 and The Economist, and you're like, "Oh, yeah. That was prescient." Now, Satoshi was prescient, I think, but not that prescient. Jonathan: So when you found evidence that things were forged, that was pretty much the moment? Tris: That was an eye-opening moment. Very exciting. But actually the evidence came in many threads. It wasn't just forensic analysis. On that technical side, though, what really struck me was the extent to which everything was based on open source tooling. So we were there, and we were defending an open source industry against somebody who was suing open source developers o- on behalf of a patent openness client. So we're already into that field. But then we're receiving forgeries created with with LaTeX, with C++ with OpenOffice. A- and we're using Linux, and we're using open source tooling to be able to analyze them. And that was my moment when I ... when the whole open source bargain and community really clicked for me because you could just take these things, apply them, a- and get the result you needed. Maybe tweak them if you needed to 'cause of your specific use case. A- and I had to learn, I had to learn Python and I had to learn Bash scripting to get through all of this. So that was my learning curve. That's- And it was a delight. Yeah, Jonathan: that's great. One of, one of my ... Oh, I can't re- I can't think of his name, but one of my absolute heroes, there is a, there's a judge in the US that did the big Oracle Google case. Tris: Oh, yeah. Jonathan: ... Oh, what? I don't remember. S- Steven No. Oh, goodness, I can't remember his name I- Tris: is the name Alstrup or Alsop? Jonathan: That, that sounds right. Judge Alstrup. Yeah, anyway, he he presided over this big case about Android between, Google and Oracle, and at the end of it he's... One of the things that he said is, "I learned how to program to be able to understand what was going on in this case." And, he's... And then says the- there was, like, 11 s- lines of source code in question at the end, and he looks at it as part of his ruling, and he says, "There's only one way that you could write this. There's no other... you could ask a, a first-year engineering student to write it, and he would give you basically the exact same thing. So that's fair use." And I just... I love the fact that he learned to program to be able to try the case, and one of my, One, one of those stories about the law that I really enjoy, somebody Tris: doing it Jonathan: I think Tris: this is because we're in the information age and lawyers all around the world need to know that if you see things through the lens of the information age, if you start seeing your cases that way- then you become a better lawyer because you can use software tools to become a better lawyer. You can assist people who are using software more effectively. Yeah. And I th- I hope that everyone starts to do that. I once had a case a long time ago now- ... where somebody brought me in and they said, "This is a real problem. I know you've got... I know you've studied engineering. Maybe you can make a head, heads or tails of this because our client says that they never copied anything, but their code block is identical." And I looked at it and I was like, "This is just linear regression." "There is only one way that you can write this." It's your same point. I, and I do think that comes up again and again, actually. Jonathan: You actually... You see something similar not exactly the same thing, but something similar in music copyright. ... It's like the, "They copied this tune," and there's a, there's actually a project out there where a lawyer that understands music put it together and "Okay, six notes in a row. There's 12 notes. There's only, and I forget exactly how many, but it's there's only a few thousand ways that you can put this together. We're just going to write all of those tunes and copyright them all, and then put it in the public domain, and then we'll just be done with this idea of, "He stole my melody." It's six notes in a row. It's... You can't copyright that. Tris: I like that. I like that. Did you hear about how Ed Sheeran was in court and he brought his guitar with him, and he played through the various ways that you could put some chords together? It was a very effective- ... piece of personal advocacy. Jonathan: Yes. Yes, absolutely. Every l- how does that go? Every pop song for the last 30 years is the same four chords, something like that. Tris: Yeah. Jonathan: It's very much the same thing. It's the same idea in music theory as we would have in computer theory. "There's only one way to do this. Every song is basically the same chord progression." Tris: Yeah ... Jonathan: that's Tris: similar. It sounds good. Jonathan: Yeah. So one of the, one of the other neat stories that you had, and I'm probably jumping partway into this, but B- Bjorn Straustup, y- he was a, ... Y- you made personal communication with him. Yeah. And something about the story here was, like, you needed an expert witness, and you couldn't get one, and the only way- So it, it is the case- What's the What ... Remind me the story. Tris: So we ... W- Background is we'd gone through these 100 main documents that Wright said was ... his crown jewels, and we'd basically proven that every one of them was either unimportant or fake. A- and we'd served maybe 2,000 pages of expert evidence. There was a conspicuous absence of code in any of his documents. So he, as far as I can piece it together had read our manual of how to forge and how to be caught forging. A- and must have seen the whole seen the whole where there was no code, a- and tried to fill it, because the thing about source code is it's plain text. There's no metadata that goes with it. A- and that came very late in the case. We were already gearing up to trial. It was an extremely late disclosure, a- and I believe that it was late because he'd just finished forging it. In fact we were able to prove exactly the dates when it was all forged. But the the problem was that in order to prove that certain functions that were called on in that code- couldn't be correct to 2008 or 2007. It was necessary to un- understand things like what is the standard namespace? W- when were these things published? And it's very difficult to prove that without an expert who will educate the judge about what these things mean. So w- what do you do? You either go through a big procedural question of does everyone get evidence on both side? There wasn't time for that. Or you ask the man himself. So we decided we'll contact we'll contact the C++ developers. We contacted the person who wrote the library in question. And we contacted Professor Stroostrup and they very kindly responded, because open source is a community. And when people ask you about your software pe- people are generally in this community extremely helpful- ... at sharing knowledge, both technical and historical. And this happened again and again. So we had forgeries made in OpenOffice because that's the software Satoshi used. So we were able to contact the then maintainers of OpenOffice- ... who were able to give us their build logs and prove that the build that he'd used to forge it didn't exist at that time. And nor was it possible to predict the build hash o- of the software that, that was output by that compilation process. That was OpenOffice. The same happened in LaTeX because he then pivoted to, plain text LaTeX source. Which didn't end well for him either. But Professor Stroostrup a- and many others joined in to help us prove fact after fact. A- and these things were bricks in the wall that, that added up to a proof that each and every piece of evidence he relied on- Was false. Jonathan: I have a little note here that there, there was some interesting sleuthing done outside of the digital domain. Tris: Yeah, that's right. So what do you do if all of your digital forgeries are getting picked apart? Hand write some, right? W- we had this amazing ... A- and this c- this particular document actually went through three court cases I believe without being pulled apart, and it was very simple. It was a single sheet of A4 paper, and on it were written meeting minutes, handwritten, very terse, and they were on this form which was headed up minutes- ... prefilled table And with a single sheet of A4, handwritten, there's not much you can do- ... to prove that it's fake. So we thought about how do you go about this? A- and without going into the privilege detail the upshot was that as a result of an amazing teamwork of people all around the world y- across three or four continents, we were able to track down not just when this pad was printed and by whom, but we were able to follow the trail of companies that had been acquired one after another, and back up that stream, contacted the lady in Shenzhen, China, who had operated the printing press- which printed that particular pad of paper. Ah. And she'd kept her PDF proofs, and they still had all of their metadata intact, and we could show, As a group, we could show that this pad couldn't have existed before 2010, which was three or four years after he, he claimed it to be. It was the most magical moment when that landed. I think it was like 11 o'clock one night. I didn't sleep at all that night. It was so exciting. Jonathan: Yes. Oh, that's great. You do the little happy dance. Tris: Oh, yes. Jonathan: Is it... Because that's the sort of thing you don't know for sure until you get it in. This is a... i'm sure that was a long shot. You g- My guys must have considered that a long shot. Tris: It's totally a long shot ... maybe there's- But you're open-minded, right? You follow the truth because you'll either find out something which is true and is against you, in which case at least you found out the truth. Or you'll find out something which is true and for you. But if you're open-minded and follow the truth that is the right way to to approach these faction investigations. Jonathan: Absolutely. The the trial itself, you said it lasted six weeks? Tris: Yeah. Jonathan: How did that Tris: go? It was, it was pretty incredible. We started off with Craig Wright himself on the stand, as you say, for six days of cross-examination. And then that's a long time. That is a long time, yeah. Plenty of breaks. It wasn't inhumane. A- and he was- ... very good at telling his story. But I think it's fair to say that the people who went into that courtroom thinking that he was he was Satoshi, were able to convince themselves, "Wow, this guy s- has such a facility with answering questions, that he must be Satoshi." A- and the people who were maybe more open-minded could see the holes beginning to emerge at that point. And we went through all of the documents and every facet of his story and he clearly prepared. Excepting one case which I will dwell on because it makes a great story. So let's say that you... Just imagine that you are a person who's forged. W- whether you've done it with your own hands or you had help doing it, w- we'll never know but imagine you're that person a- and you've got an answer for everything, and then the one unexpected question comes, and you react on the fly, but you realize there's a hole in your story. What do you do? You go home that night and forge another document. And that is what seemed to happen during this trial. So a few days later, after that first week w- we had a situation where the lawyers on the other side said, "We... There's something we need to bring to the judge's attention." And they did and they did it in open court. They said, "These documents have just been disclosed to us." And they provided them, and of course we had to go through a whole another forensic investigation. By this time it was maybe our sixth. Because of the layered disclosures that we'd been having all the way around. A- and it was it was... The upshot of it was a document that he'd forged during trial So he had to be recalled once to answer for certain additional problems that had arisen, and then again right at the end, right at the end of the case, he was on a third time under oath to answer for this forgery that was made during trial. So it went right up to the wire. It was an incredible experience to be conducting a forensic a- examination back at the office while the trial was in progress. I hope I never find myself in that situation again. Jonathan: So it was established in a court of law that Craig Wright lied on the stand under oath? Tris: Yeah. Jonathan: Why is he not in a British jail right now? Tris: He, ... I don't know exactly where he is. I think he went to Thailand for a time. He was found to be in contempt of court and has not come back to the country since then. But I think the way the judge put it was like this. "If Dr. Wright's evidence was true, he would be a uniquely unfortunate individual, the victim of a very large number of unfortunate coincidences- ... all of which went against him- ... and/or the victim of a number of conspiracies against him." And then the judge took a break and says, "The true position is far simpler. It is, however, far from simple because Dr. Wright has lied so much over so many years that on certain points it can be difficult to pinpoint what actually happened." A- and he went on to find over the course of maybe 230 pages it's the length of a novella, Okay ... that Wright had lied and lied. A- and it was a very cathartic result. W- what had really happened was that all of Craig Wright's eggs were in this one basket by the end. He just had to convince this one person, this one man, that he was more likely, 51% likely- ... to be Satoshi. The archetypal 51% attack on Bitcoin you might say. And he picked the wrong mark because he was up before a judge who himself has an engineering degree and a long history of practicing in IP at the forefront of technology who really dived down into the detail, read every word- ... looked through, I'm certain looked through the source code line by line, really understood what was going on. A- and at the end of that six-week trial, the judge had been, He stands he-- I was gonna say sitting. You, we say a judge sits in court but this judge stands, and he'd been standing and listening to all the evidence silently throughout. And at the end of the case, it was his turn to speak. And the typical the typical conclusion of a case is that a judge will say, "Thank you for all of your submissions and evidence. I will consider it and deliver a written judgment." And that's what happened here. And then he paused, and he said, "But I have seen all of the evidence, and it is quite clear to me that I can make the following binding declarations." And he did. He declared right then and there, "Craig Wright is not Satoshi Nakamoto. He did not author the Bitcoin whitepaper. He did not author the Bitcoin software." A- and he delivered it, he brought it to an end right then and there with 1,000 people on the live stream, I think. A- and us all in the room. And I still get I still get goosebumps thinking about it now. Jonathan: That's great. That's a sort of once in a lifetime moment. Tris: Quite right. Jonathan: Yeah. That's awesome. Now, you you've done some other open source work that's in the sort of the legal sphere as well. So you do work at Bird & Bird in software. Yeah. Did they give their stamp of approval of, you did this on the clock, but you're allowed to open it, release it as open source? Tris: Oh, no my side project is my own and done off the clock. Okay. So I do make tooling internally- ... for n- neat jobs that need doing. I like to make self-contained tools- ... which can be run client side. A- and with a bit of JavaScript and just understanding how libraries work- ... combining your knowledge of how the legal process works with that technical capability allows you to make tools- a- and just deliver solutions, often very quickly. A- and it's often like document management or document manipulation. So I have a l- li- little library of functions that I can use. That's all great a- and helps us and is very very fun to, to work with my colleagues to to solve those problems. But one thing that I made for myself was a, o- originally Bash and then Python a- an implementation of a pipeline to get your documents from loose PDFs- ... to a bundle of documents that you can take to court. A- and what's what's great about this is most lawyers will give it to a trainee or a junior to do and they will deliver the documents- Okay ask for it to be done, and then receive it back the next day and never have to get their hands dirty. But that is the most soul-draining part of a young lawyer's job. It's awful. Ah. I- it is the bogeyman task that everyone tells stories about bundling late into the night. So I, I realized this was actually really useful. And I thought I've spent the last three years benefiting from open source tools myself. I've got the bug. Why don't I open source it?" So I did. I re-implemented it again in, in JavaScript- ... so that it runs client side. And taking advantage of some of the quite cool WASM r- recompilations that are available now. Oh, yeah. Especially for Move PDF. And with a bit of front-end magic, it allows people to create their court documents into a bundle which fulfills all of the court's requirements, and to do it in a- in about 30 seconds. And that saves time in two ways, right? So first way it saves time is if you're a lawyer and you know how to get it done, it's a quick process. The second way it makes a sa- a time saving for people and I find this very wholesome, is that if you don't have a lawyer, you can't afford reputat- representation- ... then you'd have to learn what the court requires first before you could start even doing your document preparation. And this pipeline does it for you. So all you do is throw your documents at it, get them in the right order, and it does all of the hyperlinking, cross-referencing, and all of that for you. So it is a time saver mainly for litigants in person. Also sometimes for lawyers, and I know some some major law firms in London definitely use it. So that, that's it. It's my way of giving back to the community because I don't have a great deal of time for pro bono work at the moment. And it's built on open source, so why not release it open source? Yeah. Jonathan: Absolutely. I imagine it also helps avoid those awkward conversations of I'm sorry, Your Honor, the table of contents says page five. That's actually on page seven." Tris: It's such a waste of time. And it's not just an admin problem, because it's an access to justice problem. Let's say you- ... haven't got a lawyer, and all you wanna do is go to this, possibly the most stressful place you'll ever find yourself. I can't think of many more. Perhaps hospital. But you're going to this stressful place, and you need to make your argument against a talented adversary. And if you can't communicate to the judge with the documents in front of you, if you're always losing your focus because you have to add three to the page numbers, you're not gonna be able to communicate effectively. A- and actually it should just help people not fall into that problem and I hope give them a better chance of conveying their case to the court. I can't improve their case for them, but hopefully it will help them do what they can do for themselves. Jonathan: Yeah. Yeah. That's, that's- Something I've thought about quite a bit is one of the problems with the legal system around the world, I think, is the barrier to entry. And a lot of times that's cost. But even just become- having your tooling together and your understanding together well enough to be able to present a case. We have this... The ideal is that anybody, no matter who you are, you can go to a court and you can present your case. And if you, if the, if right is on your side, you win. And unfortunately the reality is that oftentimes it's whoever has the most money that they could pay for fancy lawyers is the one that gets to win. And, Tris: Of course in the Copa case, if right's on your side, you're gonna lose. But the the barrier to entry problem is huge. Not just in terms of tooling, but in terms of c- conceptualizing. A- and actually the legal system is very similar to, to to software engineering. What it's attempting to do i- is create a highly precise form of words to convey a concept. And that is in essence w- what a coding language does. It, it applies a particular syntax to that problem. The language we use is English. But it's about conveying concepts with precision. A- and, and- ... if you squint at a contract next time you have to sign one and get your get your software engineer mindset on you'll notice that we have effectively imports function definitions in the main loop. I w- we don't call them that. We call them party names. We call them defined terms- Yeah ... and clauses. But they operate mechanically in exactly the same way. So I feel like there's a lot of shared mindset there. Yeah. And perhaps then of all people the people who, who suffer the least barrier to entry are software engineers. Jonathan: Yeah that's probably true if only because we are used to parsing through the difficult things and understanding how they work together. So one of the, one of the quotes that I like to use on the show and it's Simon Phipps, and usually he's talking about the importance of community when he says this. But he makes the statement that "Software licenses don't compile." And his point there is, you don't know exactly what the software license is going to mean until it be- appears before a judge. And, the judge has comments and some of it doesn't survive, some of it gets understood differently than you expected. Is that software license a contract or not? Some of those various questions that, that show up. It's interesting though to think about that in a way that is what this is an attempt to do. The law is a, an attempt to make s- these concepts almost compile. That's a rather intriguing thought. I wonder if someone could formalize that at some point and make it so that they do actually compile into a perfect meaning. Tris: Ethereum would say that smart contracts are the way. But the i- if licenses are your compiled programs, then lawyers are your CPUs. So what you're really trying to do is you- as I think out loud i- is you're trying to have something that reproduces no matter what processor it runs on in the same way. A- and a well-established license with well-established clauses- ... will be met with the same legal response, with the same legal advice, no matter which lawyer you go to, as long as- Right they know their job. I think that's the closest analog to compiling a license. I- if anything, the license is its own machine code i- in that paradigm. Jonathan: Tries to be. Do you follow some of the cases, like for example right now there's a quite notable case in the United States where Vizio is being sued by one of... Oh, I forget the exact name of the group, but one of the free software groups, basically saying, "The things in your TV are GPL. You need to release all of this source." And it's, it's notable because it's one of the first times that the GPL will go before a judge, and we get to see what the judge thinks of it. Tris: So that's, I think, a Software Freedom Conservancy. Jonathan: Yes. Yes. Tris: SFC. SFC. I had a chat with with those guys at one of the conferences. A- and, ... it's a fascinating debate. I am certainly not qualified to, to comment on the legal aspects of it. Sure. But I, I think I think there are not only the f- like, the fundamental questions of what does this license mean, but it also is... It's a big stand, isn't it, for the industry. We are going to enforce our licenses. We are not just gonna sit back a- and let you in, in the terms of SFC, abuse our license. And the outcome of that I potentially could change how people measure the risk balance. I don't know. E- especially since it's it's in an important court, isn't it? I don't know if it's federal court or Californian court, but it's a court that people are gonna be waking up and listening to. Jonathan: Hello Just going to look at what court that's actually in. I don't remember. Yeah, that's what I was gonna- I think it's, I think it's a California court for now. But the way these things tend to work though, US law is based on a lot of British law and the idea of common law, what the courts have decided over the years tends to become binding. And what a California court decides will get referenced by, a Florida court looking at it, or a federal court looking at it un- until someone else comes along, a higher court, and says no, you got it wrong. We're gonna change that." I, in a separate conversation I asked you something about whether you thought the the next version of the GPL was being worked on, and I'm reminded of that because depending upon how court cases like this turn out, that may be exactly how we get a GPL v3.1 or GPL v4. And I think- I Tris: actually have no insight into the question, but- Jonathan: I Tris: think- But I think you would... You're absolutely right that waiting for the outcome of a case like this would be a great thing to do if you were working on it. Jonathan: Absolutely. It may very well be one of the things that spurs on that, that next revision. Personally, I think AI is something that needs to be addressed in the next version of the GPL. Probably have a couple of different licenses, whether, you want people to be able to pull your work into an AI training des- set or not. Like how does that work with copyleft? How does AI training work with copyleft? That's an unanswered question in some ways. Tris: It's... In some ways I think it might be unanswerable. Be- because it, in many ways the core principle o- of open source is reproducibility, right? So does that mean that if you're gonna be looking for an AI model to be open source, you need to be able to reproduce, what, the weights? Or do you need to be able to reproduce the training from scratch? Where does that re- reproducibility line start and finish for something which is a totally different kind of software doing a totally different kind of task? Jonathan: Yeah. The OSI actually took that particular question on, and they now have definitions for what it means for an open sou- or a model a, an AI model to be considered an open source model. And they made some decisions about how exactly that would work, and people are not happy about it because it is, it's so new and it's so different from anything else that we've had to fight with before. So that, the, there are OSI definitions for what it means for a model's license to be, OSI approved. But people are not, people were not happy about Tris: it. It's a highly political question be- Sure ... because people have different opinions about what AI means- ... what it should be, and what openness means in that space. I thought it was a really good thing that they went ahead and published it. Yep. Be- because a lot of the questions were being asked in the- abstract. And this gives someone a framework against which to test their their thoughts and assumptions, right? I feel like it probably created as many opinions as it put to Jonathan: bed. Of course. Of course. I know, Tris, that you've got to hard out here in just a few minutes, so I think, and we've gone for over an hour. I think we can go ahead and wrap it up. It has been an absolute pleasure to have you on the show to talk about- Tris: My pleasure to join you ... Jonathan: things legal. We we'll have to do it again. Give it a few months, we'll have to have you back. It'll be a lot of fun. Love to. Maybe once the the Vizio and SFC case wraps up, we can have you back to talk about that. That could be interesting. But- Let's Tris: talk about it. Jonathan: Yeah, we'll do that. All right. Thank you so much for being here. I appreciate it so very much. Again, it was a real pleasure. Oh, I gotta ask. I gotta a- I will get emails if I don't ask. Okay. Favorite text editor and scripting language. Tris: Oh I love to script in Bash. That's my favorite that's my favorite scripting language, for sure. I have been experimenting in the last few weeks with Helix. I'm more of a Neovim kind of guy. And when I just wanna get a quick note together I find Kate, because I use Kubuntu. I find Kate to be extraordinarily fully featured. And very good for markdown with a preview. Jonathan: Helix is the post-modern editor and of course in their FAQ it's, "Why post-modern?" It's a joke. If Neovim- ... is the modern Vim, then Helix is post-modern. Tris: I have had trouble getting to grips with its reverse bindings- i'll be honest. But it does have a handy pop-up. Jonathan: Yeah. That's fun. All right, Tris. Again, thank you so much for being here. I appreciate it. It's been a blast. Tris: My pleasure. Thank you. Jonathan: All right. We've got some fun stuff coming up in the future. We've got Andy Rick and Aaron Bassett from QNX next week. We'll make sure and ask them why QNX is not open source. And then the week after that we've got Andrea Gallo of RISC-V. Looking forward to that one a lot as well. And then in a few weeks we're talking with Michael Meeks of Collabora. I believe I met him at the Ubuntu Summit. And then we have a returning guest, Francois Poulin of Smoked Meat, and that is yet another GitHub but related security tool. You don't wanna miss that one as well. We appreciate everybody that is here, whether you watch or listen, get us live, or on the download, and we will be back next week on Floss Weekly.
• Episode 872 - I'm Not Satoshi 24.06.2026 1val 7min

  This week Jonathan chats with Tristan Sherliker about the Craig Wright case, Open Source and the law, and Tristan's own Open Source project, BunTool. How did Open Source help win the day at the Bitcoin trial? And why is right now such an interesting time to be in the legal field? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• Episode 871 transcript 17.06.2026

  FLOSS-871 Jonathan: This week we're talking with Florian Gilcher of Ferrous Systems about Rust, Rust in the business, Rust in the kernel, the history of Rust, and a whole lot more. You don't wanna miss it, so stay tuned. This is Floss Weekly, episode 871, recorded Tuesday, June the 16th. Rust won't save you It's time for FOSS Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we're gonna get rusty. And that's because we have Florian Gilcher, the the man, the managing director, the co-founder behind Ferrous Systems. Ferrous Systems, Rust, you see what they did there? Anyway, he has been doing Rust since 2013, which I think is before Rust existed as it exists today. Wanted to make sure and ask about that. He's a co-founder of the Rust Foundation, been a core team representative, and done a lot of other really interesting things. We're mainly gonna talk about Ferrous Systems, but you can believe we'll get a lot of Rust questions in. I'll probably tease him a bit about rewriting things in Rust and the way the internet seems to be crazy about that. Without any further ado, let's let's bring him on. Florian, welcome to the show. Florian: Yeah, hello. How are you doing? Jonathan: I am great. It is good to have you here today. So you are, let's see, you're a German citizen. You're from Germany. Yeah. But you're not in Germany right now, are you? Florian: I'm currently in the UK at ACCU on C, which is the merger of the ACCU Conference and the C++ on C conference. So I'm currently over at the other side, essentially. I've been giving a Rust workshop there, but I have very good conversations with C++ folks all day long. Jonathan: Okay. That's a, this is an interesting topic to to touch on first. How well or poorly do you get received doing a Rust conference at a C++ convention? Florian: Oh, very well. Okay ... it's actually in subject. The ACCU reached out in 2016. That was right after Rust 1.0. And the Russell Winder, who ran the UC- ACCU conference back then actually had the stance of ACCU, which is the Association of C and C++ Users, I think. That's what it stands for in the UK. Thought about, "Okay, how do we deal with the fact that there's new systems programming languages coming up?" And their choice was, "Oh, it increases our relevance. They should be here, and they should be talking at this conference." So there's people from Swift here, people from Zig, and I think that's very much true. And the reception in the C++ space was generally good. Whenever something like that comes up, there's people who, there's naysayers on both sides. I know a lot of people on Rust that say C- C++ is gone now. And I think nothing could be further from the truth. Still very relevant. And I think most people will just be like, "Oh, ah, it's good that you're here. Let's talk. Let's exchange opinions. Let's see where all of these things are going." At a- Sounds so cliche, but yeah ... Jonathan: at a conference like that, you already have the subset of, C++ and C users- ... that can be together at the same conference without killing each other. You know- Yeah You're halfway there already, Florian: I was sitting at a table yesterday with someone from the C Standardization Committee and someone from the C++ Standardization Committee. I won't say anything they talked about, but we had a lot of fun. That's Jonathan: fun. Yeah. All right. So Ferris Systems, obviously it's a play on Rust. What is Ferris Systems and what do you guys do there? Florian: Okay, so Ferris Systems is one of the first companies that provided services for Rust. It actually comes out of... I was, just to answer that question, I was part of the Rust community before 1.0. 1.0 was May 15th, 2015. And I was part of, I w- I was doing training for Rust back then. I w- I had an old company, another company that was dealing with Ruby, also doing a lot of open source there, mostly applying it, less building it. And at some point someone just came up and said "Hey, could you do a Rust training maybe?" And so I said I could probably do this. I'm equipped for that as good as anyone else." Started doing that, and in ... Slowly started also offering Rust as a service, and in 2018 a number of larger clients came around. And so things like, "Hey, we're build- we're using Rust. We're a hyperscaler. We're using Rust internally." And I was like, maybe a second company makes sense at that point. Jonathan: Yeah. Florian: And on the other hand, so what I always wanted back then was also for the spread of the language to make sure that, for example, all of our training material is open source, because I wanted people to apply that and to spread the language. And yeah, that has turned into now a major operation. We're more than 20 people serving customers from, I always say from Japan to California which is- Jonathan: Everywhere ... Florian: a blessing and a curse sometimes. Yeah. Yeah. But, The Jonathan: time zones. Time zones are interesting ... Florian: yeah, that's what we do. Yeah, it's mostly the time zones that are the blessing and the curse, yeah. Jonathan: Yeah. Yeah ... do you guys have people in the various time zones? So like you're not the only one answering emails in the middle of the night? Florian: Yeah, this the first growth phase of the company was during the pandemic, so we're people call it remote first. We're, ... This, it sometimes takes a year before I see a new colleague personally. So yeah we're distributed mostly time zone-wise Berlin to Californian time zones- ... Florian: Mostly. Jonathan: I've got work colleagues, I've got work partners that I haven't met yet, and we've been doing our business for over a year now. It'll be, ... i'll meet a good portion of them at DEF CON for the first time, and then later this year- i'm going to Europe and I'll meet a- another, at least one hopefully he'll make it, another one at a conference there. It is, it's fascinating to have, people that you know so well, you have business relationships with, and never meet 'em in, never met 'em in person. We just know, we're just- internet friends. It's such a, it's such a wild thing. I- Go ahead ... Florian: it's the same in open source projects. I find that quite interesting also when I was part of, of- Absolutely ... a team lead there. It's you have such a wide diversity also not just of where people live, but also some people are just there for an hour a week. For some people it's their day job. So it's completely different to manage and a lot of different concerns to just get get under one hat. Jonathan: Yeah, absolutely. So Rust, Rust was a very- ... different language, and I alluded to this in my in my opening intro. Were you around for some of this transition from, and this is my understanding. You obviously, you can tell me that I'm wrong, and I will not hate you for it. But my understanding is that Rust started out as an academic language and, was, was- ... almost like a toy academic language at the very beginning. And then people said, "Oh, this is really interesting." And then, you have folks like Mozilla picked it up and said, "We could actually take this and turn it into something really useful as a system language." And so like the Rust 1.0 is quite a bit different from the Rust that the the academics were using before that. Is that accurate, first off? And second, were you around for some of that transition? Florian: I was around for the end of it- ... basically. So but some of the pretty big changes. So I was around when it was still three days later, the language could be completely different, where some syntax has changed, was added, was removed. So it, it's between, it's that. It always came out of Mozilla. It was a plaything of a Mozilla engineer for a while. And at some point, Mozilla Research picked it up as a proper project. And the interesting thing I found about Mozilla Research, and this is from an outsider's perspective, I was never part of Mozilla Research was that they had a very good vibe of we are doing research, but we are mostly doing research application in the sense of that Rust is based on a lot of things that came out of programming language research in the '90s, in the up to 25 or something like this, or a little bit later, plus some things on top. So things like the borrow checker and things like linear types that are the b- foundation of the type system had already been researched. No one has put that into a language yet. And so there was, before the 1.0 release, a huge group of people that were also using this language to try out things. There's a really good talk by Marjin Haverbeke at my first Rust conference as the closing talk. The Rust That Could Have Been, where he talks about all the features he has tried out that were all removed. So there's none of the features he tried out on the language were put into the language. And he went on stage and said, "And it's actually a better language for that." Yes that time existed, but at some point It, y- people found, okay, there's something here. We really want to industrialize that. So there was this phase of, And that was the time where I was mostly around. And that more as a community member than an actual project member from 2013 to 2015, where they figured out, "Okay, let's actually make this a systems programing language. Let's make it play in the C and C++ space, and let's actually release that." So the 1.0 Rust release is very imperfect in hindsight, but it is a release. And, Jonathan: Every, every 1.0 release is imperfect in hindsight. Florian: Exactly. And that was the thing that drew me in. There's this, they, this vibe coming up of, oh, we could actually do something here. Also, just to be clear, I think there was, it was the vibe in the room back then. It was not that surprising that Apple shortly later came out with Swift. So a number of people... That, that's what I meant, is like the, this observation that I gave earlier systems programming was becoming relevant again was very much in the air. So I always say Rust's success is not only the success of Rust, but also the success of making the topic relevant again. So it's also the ocean in which you swim. Yeah. Jonathan: Yeah. That's an interesting point. You mentioned, so I'm gonna ask you some questions just because Rust- Sure ... and I have Rust questions. Y- or maybe I'm gonna rant about things. Y- you mentioned making Rust play with C and C++, and there's a... I see it as a problem. Maybe this is bec- because I come at it from a different background. I don't think that Cargo works very well with integrating with C and C++, and I feel like maybe the kernel has had that problem, too. I've mused before that the worst decision the Rust guys ever made was packaging a, putting a package manager as part of the language. Tell me how I'm wrong. I Florian: say, so first of all I have a very long rant about package managers for on, on Linux distributions because I'm from the Ruby space. And we suffered so much with package managers that had their own opinions that were like just aggressively against what their community was doing. So if you think Cargo looks like Bundler, that's because it's actually from the same author. And I think some of that- some of that actually shows up in Rust a little. I think the good decision that Rust has made is that Cargo drives the compiler through its public interface. Okay. So you could, you can replace Cargo, and this is frequently being done. Google does build their Rust stuff with Bazel. Facebook has written their own build tool called Buck2 actually in Rust. So this is it is not uncommon that people replace Cargo. I think Cargo wants to own the world and wants to own the project is very much true, and if that's not what you want then then th- those stresses show up. Yeah. I d- I don't think it's an... it's an okay reading. I think though it has made Rust successful in the sense of it brings this mode of development to the systems programming world built by people who have actually had experience. It's very much a, it's very much lean, like some- something we took from the NPM and Ruby ecosystems, just to be clear. And that may sometimes feel a little bit off. This is an, like just one more sentence. This is another observation that p- I have about the Rust community. R- the Rust community has an interesting mix of systems programmers, but also people who went from scripting languages towards a systems programming language, and that sometimes leads to an interesting mix of concerns. Jonathan: Sure. Yeah, absolutely. Has Rust experienced any of the security problems with malware packages that NPM is famously having just about every day? And, now we see it in other places like the Arch User Repository is having quite a moment- ... with malicious packages. Has that happened in Cargo and the Cargo package backend yet? Florian: Let me put... just to be clear FOS systems used to operate or would be the on-call service provider for Cargo, so I need to make sure that I'm know what can share. I can definitely just say the, first of all of the package managers actually have forums in which they talk about these things and are observing- those attacks. Most of that is not happening in public for obvious reasons- Sure ... because it gives the attackers the up. Cargo d- has experienced some of those attempts. We have definitely also seen people just taking other people's code, republishing it ripping the open source license off, putting their own name on it, and all of these things. All of these things happen on crates.io as well. It- I don't think the situation on Crates.io is in any way that bad. I also don't think the situation on npm is that bad. Most of the time those things are smelled out and removed relatively quickly. You just need to be aware that you are using a live service. This is actually one of the things that we're offering to customers, is we're saying if you don't want to rely on a live service where these things may happen all the time we can offer you, for example, managed crates hosting, some ways of ingesting the crates that you want that are relevant in a semi-automated fashion. Jonathan: That's interesting ... but of course, Like a cur- a curated list of safe crates- Yeah ... and not- Yeah ... immediate updates and- Florian: Yeah ... Jonathan: some probably a combination of automated tooling and some human eyeballs to look at things. It's, it's- Yeah ... it's pretty interesting. Florian: And but that is, for example, a place where the more slow-rolling model of distributing these things with, for example, your Linux distribution definitely is not that, that much subject of you... I cannot hack Debian by just uploading a package somewhere. Someone needs to make an active move of integrating it into Debian, and that's already a first line of defense. Yeah. Jonathan: Yeah. N- not to say that it is a perfect line of defense. Yeah. Because XZ did happen. Florian: Yeah. Jonathan: Now that's, that is- Yeah essentially the same sort of attack, and it's just, it's a demonstration of how much more complicated it is. Florian: Yeah. Yeah, but you mis- ... So one of the things that I have as an issue sometimes is w- it's so easy to point to XZ, making it into Debian and saying, "Hey, the Debian model is broken." No, it isn't. It's a human system. Failures happen. There's a, this... if this doesn't happen all the time, I- I... Jonathan: Yeah, absolutely. Yeah. I would say that the fact that it, that only one pack- that we know of, only one package made it in and it got- ... caught that quickly is, if nothing else- ... it's a vindication of the Debian system. That their- Florian: Yes ... Jonathan: it works and their paranoia has some basis in reality. Florian: Yeah, definitely. Definitely. Oh. Yeah. Jonathan: All right. Yeah. What... So going back to our road map here, and we just, we touched base on it just a moment ago, but when someone is looking at Rust, what are some of the things that they're gonna worry about? What are some of the potential problems with Rewrite It in Rust? Florian: I don't think most people take a problematic stance at adopting Rust. They usually adopt Rust because they already have a reason with what they currently have. Jonathan: Right. Florian: You rarely make, you rarely migrate away from something that works for you. So they're more looking at what are the benefits than they're looking at what are the costs. In- interestingly- ... having been both in the growth phase of Ruby and in the growth phase of Rust 15 years ago people would be more like, "Come on, it's a nicer language. It has nicer syntax. Let's go." Nowadays, the questions you need to answer is will I break bank by, in the migration? With that, we've seen multiple companies actually go down in- ... between 2010, '10 and 2020 by just dying in, in a rewrite. Like this, it, it even became a topic, should you even do rewrites or not? So what are the benefits? And then it's usually operational concerns. How do we actually integrate the technology? The question, do we rewrite or do we not? W- what's the benefit? What's the disadvantages? If we're not rewriting, how do we integrate our old code base? Is that possible or not? And there's always also the question, I think that's something even even though my business is obviously in, in the Rust side of things we've had a number of clients where in the end our advice was probably- Don't do it you should first get someone... Yeah, you should first get someone to look at your current code base and maybe fix the issues that are in there. It will probably be more effective, ... in, in the immediate, And that's the other problem. If you're, if you already lost track of your software product, to be honest, the rewrites become more hazardous because you don't even know what they should do. You're just adding another problem to... It's a problem. And just to be clear, if you're in the services consultancy business you're never called when everything's in order. Jonathan: That's Indeed. Florian: And that, it, that's okay. There's like very often you have very successful companies who are like, "Okay we got successful overnight and we need help." So it's that, it's- Yeah ... it's- Oh, that's... Yeah ... I always take the stance of very few companies ended at the place where they are out of incompetence. Mostly it's competence, but also y- you know, you can only do that much in a day. Jonathan: Yeah. It's that, it's Florian: that- You only know about so many things. Jonathan: It's that combination of like just the inertia of what you've done, the fact that- something has snuck up on you, whether it's your growth or just the passage- ... of time has snuck up on you. And then suddenly you look down in your code base, it's "How did this happen? How did we get to this point? We need some help." I, I find it interesting, and maybe something like this will be the show title, but Rust won't necessarily save you. It's not gonna- ... fix all of your problems. Florian: Of course it will, but, but yeah it is it is something definitely where there's also, there was also a hazard when we grown Rust, and that was actually something that we, I was not only part of the core team, I was part of the community team and very much in lead there. And community team sounds so... You're basically the outside representatives. It, it always sounds like these are not the real core engineers of the project. Jonathan: Yeah. Florian: Quite the other thing is true, because you need to be so up to speed with so many technologies- ... because someone comes at you and says "How does Rust work on a GPU?" And I'm like, "I know nothing about GPUs, but maybe I should read up quickly." But there's always... you develop this feeling for, are people just there for the why, but do they have an actual problem? The other thing is that we always told people is, "Don't speak to people who currently don't perceive any problem. It will not be a useful discussion." But if someone says, "My current software stack, I really don't like it," then there you can make an offer. It sounds like marketing. I'm actually... I would actually open s- have open source projects to be more okay with doing marketing, but doing marketing in the sense of a friend of mine once said, "If you're not talking about a good thing you're doing, you're depriving people of the ability to use it." Jonathan: Yeah. Yeah, absolutely. Florian: And marketing on that level, not marketing in the sense of advertisement tracking and so on and so forth. And I used to work in advertisement. I know where the bad rep of advertisement marketing comes from, and it's very- Yes ... I g- I very much understand why people hate it. It's more like the go to conferences, go to a C++ conference, talk to people and be serious about your project, but also be serious about what they're doing. Jonathan: Yep, yep. Absolutely. Have you found any places where there's a, I don't wanna overstate this, but let's say friction or problems trying to make the business case and the business side of things work with the community side of things? Florian: You mean yet now company-wise or, Jonathan: So i- I mean it as an open-ended question, but you talk about being- both a core engineer and a community engineer and the- ... the difference between those two. And, either in that, those roles or now- ... into the business side of things. Florian: I would even say these things are always conflicting. I cannot just personally I can't, I cannot spend as much time at community events as I want because I- kinda have to, Jonathan: Maybe a Florian: business to run ... to run my business. Jonathan: Yeah. I see Florian: that. Yeah, but also needing to represent a business. This is actually one of the reasons why I why I left the Rust Project was- ... quite simply because I was in calls where people would ask me "What's your stance on this?" And "What's your stance on this as the director of the foundation? What's your stance as this, as the director of Ferrous Systems? And what's your stance on this as a core team member?" Because I was representing the Rust Project at the foundation, not my business. And that's a constant friction. It's a it's a... it's not that much of a problem. It's a conflict of interest. It's an open conflict of interest. Jonathan: Yeah. Florian: I don't think conflicts of interest are in any way a problem. It is you need to be in a meeting, and you would then need to say, "This is a business meeting between you and Ferrous Systems. I don't represent the Rust Project there, so the only opinion you can get out of me is- ... mine. I can tell you the stance of the Rust Project. The stance of the Rust Project is this one." And they sometimes diverge, and that's fine. The other problem is obviously on any pro- project I was experiencing that hard when I was running the Rust Allhands. You have... So the Rust Allhands the first ones were a one-week meeting where people finally met. And for everyone who was d- at who was contributing to Rust as a hobby, that was a week of vacation. Qui- quite literally they could go to their boss, and sometimes their boss would be like, "Okay, go there. It gives you five days for going." So they were like, "Finally, I have one week where I can only do Rust," and it was their first week where they could do that. And on the other side, you have people who are full-time employees and who are like, "Oh my God, that's the 70-hour week," right? And- Yes ... th- these Those are problems, but they're inherent, and I think you need to be very open very open about them. But that, for example, also meant, and you do need to find your way around this. The community mailing list that I was managing at the Rust project frequently got requests for trainings, which was my business. So the community team internally had a rule of we have a list of training providers. Training providers who are in the community team will not reply, and we will send out a structured list as, "Hey, by the way, these are all the people to call," so that I, that we don't snatch business out of the project just because we are at the right, because we have access to the right email address. Jonathan: Yeah. Florian: And the Rust project is actually quite good about having these things sorted. Yeah. Jonathan: It sounds like, and maybe you have been a part of this, but it sounds like there's some people involved in that project that understand the business side of things and understand the- potential conflict of interest and the- ... sane ways to handle that. ... Jonathan: It's k- it's refreshing actually that an open source- ... project that is also a business is not a train wreck. Florian: Yes. I actually- Thank you for that. That comes from also you may have observed that the Rust project has absolutely no problem with being political. And I don't want to be divisive there, and that sometimes leads to conflict, just to be clear. But that also means you have people there who say these things, like these are in a way political problems. Like we often put political problems more in the social side of things. But this thing like corporate politics and all of these things, and you have a lot of people who are actually well-versed in that. You may agree or disagree, but yeah. Yeah. Jonathan: Yeah. I've taken the stance over the years that an open source project should try to stay out of the politics and culture wars as much as possible. And I know like even that in and of itself is a political/culture war statement, and I acknowledge the irony of that. But that's just where I've come to with my projects. Like I'm not going to- Yeah ... put a statement out on what I think of any politician or... And I don't think- Yeah ... in, in my opinion, it's not the project's place to do that. I think it's a distraction. But at the same time- Florian: Yeah, but no and I didn't even want to to go there. That's a whole other discussion for a whole other - Right ... I just think like that sometimes I think people would wish those things wouldn't exist, and we could run a successful open source project that runs a new programming language without having corporations involved. And if, I've literally been in discussions where people express that view. Yes. And the problem I see there is if you're not set up for these companies have goals, and they have set these goals internally, and they are on your project because they want to fulfill those goals. And like that level of, hey, they're, this person might make that statement because it's their goal setting internally at their company, even if they personally disagree, that these effects you can't argue those effects away, and you need to deal with them. That's all I'm saying there. Sure. And the Rust project has been, from the beginning, very much aware of these effects. That's and I personally, that, that's my view. I actually feel that under a political mindset in a, in the sense of how I operate in my brain. That's all I want to say. Jonathan: No, that's fair. That's fair. Yeah. Let's talk for a minute about safe Rust. Florian: Yes. Jonathan: And and there's this feeling among some people that, "Oh Rust is safe. It's a memory safe language." Which, yeah. That's great, unless of course you use the unsafe keyword. ... But then you have in the regulatory structure and the safety, like safety engineering structure, something being safe has a different meaning. Is there Rust in cars? Does Rust run airplanes? In those serious positions of safety engineering, has Rust made a foothold there, and what does that look like? Florian: Yes. Mostly in what's called quality managed or a little bit of mid-safety at the moment. The overloaded term, like, of safety is obviously a problem. Rust says it's memory safe, and there's a very clear definition in the language what it actually considers memory safe. Jonathan: Right. Florian: Rust on the other hand also has a culture of correctness, I would say. People go to great lengths to use other facilities of language to make sure something's correct. We also have a strong type system. We have a type system that does not have that much baggage, because w- it's a new language. That's that comes with that fact. And we see a lot of interest in regulated industries. Like a lot of the things we're doing is automotive, but also a lot of, industrial safety. Where the definition of what safe software is it's also different. It's usually called functional safety. So in that front, it's more it's less the... So memory safety is a part of functional safety because it could impact the functional safety of my tool, but it's so much more. It is- ... even things like this plane will not take off at under 230 miles. And your software needs to deal with that. So a lot of outside requirements actually being put into software. The speed is completely random. I don't know what the normal takeoff speed of a 767 is. We're Jonathan: not aeronautical engineers. Don't quote us on that. Florian: Yeah. Jonathan: I went to the I went to the most recent Ubuntu Summit, and one of the speakers there was talking about trying to use the Linux kernel in that sort of- ... safety engineering sense. And the I don't remember what company he was representing. I wish I could remember that. But anyway he was talking about the lengths that they were going to try to re-engineer the kernel in such a way- ... that it could be considered safe for, like- ... automotive use or some of those industrial uses. And it was- ... it was eye-opening, like the lengths- ... that they were talking about having to go to, to- and again, not just memory safety, but, you also have real-time guarantees and other things that- ... that have to be considered. It's a lot- ... for these really regulated industries, but also the ones where there's a, there's engineering definitions of what it means for something to be safe. Florian: Yeah. So the interesting thing I like I have not spent most of my career in safety critical. And one of the things I liked when moving over to it is first of all, one of the things I find interesting is that there's a lot of things that would've been done for high safety software that are now table stakes. Big open source projects are practicing methods that are definitely on the level for safety. But the one thing I like in safety engineering is that the currency is the argument. You can basically do anything, like obviously not anything, but they're open to quite a lot of approaches. What they're also interested in is you also bring me the argument why you think this is safe, and what the safety standards actually say is, here's a number of basic techniques that we consider helping you towards that goal." So standards are not just checklists, but they're also a source of inspiration. Actually the the avionics standard, the DO-332 has a very good list of this is what can go wrong in dynamic memory allocation and the things that you should look at. Sadly, that standard costs I think $300 or something like that, so most people haven't read it. I can actually recommend reading it. It should be ... we could teach this at university. It's also not super magical in that sense. Yeah. Jonathan: It I went and looked it up. It was actually an Nvidia employee talking about ASIL B qualifications for Linux. And yeah. Very interesting. If somebody wants a quick primer on like this safety engineering stuff, that's actually a really good talk. He did it in just over 20 minutes. It's actually a really good talk- ... to check out to get an idea of what all is involved with that. I found- Yeah ... I found myself sitting there listening to it going like I I'm a software rebel. I tend to not believe the different "This software is secure." E- I've heard that before. And when he's talking about, "This software is safe and this is how we prove it," I'm like, "Sure it is, bud." And also "To be safe you have to do this and this," and then it's built into the spec. And it's my rebellious side comes out again and I'm like, "No, surely not." Florian: On the other hand, I've heard w- when I started in s- safety people were like, "Open source and safety is incompatible." "There's the bazaar out there. We do structured requirements management," and so on and so forth. And so my joy over the last five years is seeing that move very drastically. The Eclipse Foundation is now one of the biggest foundations in in open source in, for example, automotive. And I'm like sud- suddenly everyone's around. Suddenly there's the BMW, the Vectors, and all of these companies are around and they're like, "Okay, let's consider, like, how many of those open source things can we reuse?" And actually the one thing I want them, I want to convince them of more and more is then engage in those projects. Like- ... our open source projects would do better if they have a... I don't know if they would do better, but if it's a s- if it's a project that you find so high value that you wanna put it into a car, but you have these f- five things that are missing- the usual mode in open source is go engage, talk to the maintainer, see if they accept it get involved in the project. And I know a lot of people in those industries who are extremely sharp thinkers about particularly the where is my software deployed and what can happen in the physical world around it. So they, they have such a big mindset of it's not just my TCP/IP connections, it's actually what does the camera see? What can ... w- what's all of the possible permutations that my camera could see, and how does it need to react in that case? So yeah. In the other thing that you're saying though is also very much true. There's this kind of like safety standards are a catechism and this is what you need to follow, and you can't go left and right. And this is where open source definitely doesn't fit in because someone at the Rust project will say, "Hey, I'm a student somewhere in Ottawa, I'll just do whatever." Jonathan: Yeah. There, there's another... I forget who originally made this observation. But open source is a license. Yeah. Open source is not necessarily how you run your community. So like you can run, and in fact we're seeing projects do this now for various reasons. You can run an open source project and not accept- any outside contributions. For various reasons there are several projects that are taking that stance, which is wild because it's so different from the way open source has been run. But that is perfectly acceptable with the license. Florian: Some of our projects, like our we have our own Rust compiler distribution, and that one does only accept contributions from a limited list of people because we're going into liability- For these things. So we're actually encouraging people, if you want to change the Rust compiler, go to the Rust project. We will vet it later. So it's also, you were talking about conflicts earlier. For us it's a way of staying outside of like not to be a drag on the open source project as well. Or no, we are an open source, we are an open source project. We don't wanna be a drag on our upstream. And projects I think that's a good thing that projects are thinking more and more about their policies. And yeah, as you say, open source is just a license in the end. Yeah. Jonathan: What's the intersection been between Rust and the current AI craze that we're in? I ca- I say that slightly derogatorily. I've actually pretty much become a convert because it's just so powerful these days. I've written multiple new features by just saying, "Hey, Claude, do this for me." And it's "Oh, that code's actually not bad. Here, let me fix it in a couple of places." It's- astoundingly useful. Florian: Yeah. I think the Rust community is a little bit tired of sometimes being in the middle of those hype cycles. It was already in the middle of the blockchain hype cycle, and we were- ... we know where that went. And some people were incredibly aggressive about this. So- Yeah ... if you see some pushback just be aware that we just went through a five-year phase of people telling us, "If you don't do blockchain, you'll be out of business in the next three years." Or you'll be out of job. And now people are coming on, "If you don't do AI, you'll be out of this, out of job for the next five years." So there's part of that the other thing though is Rust is actually quite benefiting from that. There's a-- Because it seems like AI models seem to be quite good at working with strongly typed languages. We're a strongly typed language. We're in the hype field. We know that Rust is being used in multiple companies internally quite heavily, so there's something coming out of that. So it's also a funding source. The Rust project is- that's nice. Yep. The Rust project right now has struggles or struggles in the sense of they have a debate about what their LLM policy is. I won't say much on that because I'm- Sure ... I have a number of people who are having those debates, and they don't need someone else going on a podcast I get that, yeah ... stating their opinions. But what I can say is I have seen multiple people who were rather aggressively either saying it needs to be a hard ban or it n- no one needs a policy at all, and that just isn't feasible for that project. Yeah ... a project of that size or a community of that size needs to position itself. Internally for our safety critical products, we're currently not using any LLMs at all just for the reason that they are still faulty in ways. And in my experience with them, I find, I regularly find pretty outrageous bugs in the things they generate. And I don't want to be the first person who signed off a piece of software that using an L- an LLM hallucination has run a car into a tree. Yep. That's- Because that will be on me. But it is pretty much in the middle of it just like any other community as well. Here over the C++ folks are discussing that thing left and right as well. And but one of the things it, again, coming, circling back to the beginning it makes systems programming languages more relevant again because one of the reasons why people are switching to Rust or switching back to systems programming languages is not even the whole safety bits and all of these things. It's power consumption. These things burn, like every cycle you burn if you scale it up by a million is, Yeah. Yeah. Jonathan: It really Florian: makes a difference. And performance engineering gets interesting again, and this is where LLMs are actually pretty bad. Performant code? Not so much. I have no int- Yeah, Jonathan: that's- Yeah, Florian: I don't... Jonathan: Yeah ... it's an interesting observation. Yeah. We talk about LLMs making lasagna sometimes. They wanna be so, ... verbose and easy to understand it ends up with this really weird layered code. I've also observed that languages that are, like, open source, well-documented, and have a big corpus- out that's easy to get ahold of on the internet, those are the ones that tend to do really well in the LLMs. 'Cause obviously- ... it's got a lot of sources to pull from, and that kind of- ... describes Rust pretty well. So I imagine that the the modern LLMs do fairly well in writing Rust code just because they were able to train on so much of it. Florian: Yeah. And I used to do, I had a phase where I worked as a, in consulting for a full-text search, particular Elasticsearch. And the... I think a lot of people are very focused on the generative side of the whole thing. Jonathan: Right. Florian: Which you can consider problematic, and a lot of people are seeing that as an offense of basically you're replacing the human act of writing, which is I think, very cult- this is culturally pretty a pretty big lift. On the other hand, on the data retrieval side, I used to it I find it quite impressive and a big step up. On the indexing side of things, it's not that much more than a more sophisticated index. Very regressive view of things, but yeah. Yeah. Yeah. So I'm actually way more interested in the in the data res- retrieval- Aspects than in the generative aspects. Jonathan: Yeah, that's, that, that is something that's easy to easy to forget, that there's more to the modern inference than just using an LLM for generation. ... The techniques that have been developed, that there's they have more uses than just a really good chatbot. Yeah. Yeah. Are you mentioned briefly Rust on the GPU. Is that a thing that people- ... are doing? Can we use Rust to write the LLM and run it right on the ker- on the I guess as the kernel on the GPU? That's how these things work these days. Florian: That's a long way to think. Rust actually had a pretty active machine learning community very much in the beginning. Until Google came along with TensorFlow, which just kills off all the startups- Yeah ... because everyone says "We can't compete with Google." We can't Jonathan: compete with Google. Florian: Which I actually don't think is true, but, No, that's not at Jonathan: all, but Florian: in the investment market it has that effect. And since then there's, there have always been people who have built Vulcan wGPU implementations and all of these things. The, there's a number of approaches to actually run Rust on GPUs most of them by using a custom compiler backend. The Rust compiler actually is very well suited for plugging in an, in multiple code generation backends because it used to actually carry multiple code generation backends for quite a while. And but most of... And so there's an active community out there, but they run very shallow compiler forks effectively. So they build these backends but only on, for example, very specific versions of of the Rust compiler's nightly. But those things are usable. So there's a, there's an ongoing small community forming. The Rust project itself, this is actually something which, but that was a decision that I was part of at the beginning, did not invest much into that just for reasons of you need to focus on something that you can actually achieve and not have you- Sure things in different pots. It's up and coming, I would be say, I would say, but it's up and coming since about six, seven years. Yeah. Yeah. Jonathan: Have you followed any, any of the sort of Rust experiments like, and I can't remember the name of it, but the let's rewrite a kernel in Rust. Oh, what is that called? I can't remember what they call that project. It's got a, it's got a- It's multiple ... it's got a metal themed name I think, the one that I'm thinking of. Florian: Redux. Jonathan: Yes, Redux. That's it. Of course. Florian: Yeah. I've not followed Redux that much. I'm casually aware of a couple of people using it. But writing operating systems in Rust has been quite common. There's a number of microcontroller OSs, DockOS, which even has a commercial counterpart now called Oxidos. And all these things are coming along quite nicely. I think for most people a C based OS is currently okay. People are s- first starting to, for example, write drivers in Rust and things like that. It's also, it's one of those levels where Rust still has to answer quite a few questions. So it's more you need to know the compiler behavior. You're not always sure if that, if it's going to stay this way. You m- mostly will be on the nightly compiler because you want some unstable features and things like that. Those gaps are actually pretty quickly closing, so I would say in two to three years I would not be surprised if the Rust compiler's fully capable of writing a kernel on stable. Yeah. And yeah Jonathan: I know- Florian: These projects are coming along late. Jonathan: Yeah. Yeah. Yeah. I know that in places like the Linux kernel famously- Florian: Yeah Jonathan: Down in the depths of that, there's some really ugly things g- go on, like inline assembly and- ... all sorts of sort of dark magic hacks. Because hardware is hard. And and so I whenever somebody talks about "Let's build a kernel in Rust, it'll be memory safe," that's not how that works. You have to do so many questionable memory unsafe things to be able to talk to real hardware. That like I I've asked the question is it really even worth it to do the rewrite in Rust? Are you getting enough out of it? But that's just me. Florian: There's the second question is should we still work with operating systems from the '90s? W- the answer may be yes and no, but I think that's a m- are there other designs out there? And then we may write one of these new designs in Rust. We're seeing that in the microcontroller space. So- Sure ... people are actually actively writing new operating systems. The the thing is, I think it's also a misunderstanding of Rust. Hardware is fun- from the perspective of a compiler, fundamentally unsafe. Memory mapped registers behave randomly over time from the perspective of a compiler. They do things. They don't behave randomly. They obviously behave like the hardware is specified, hopefully. Jonathan: Sometimes. Some, sometimes they do. Florian: Yeah. I I... and then Rust's game is much more encapsulating all of these things. So the language gives you facilities to say, "Hey, by the way, this is this is a UART. This is how a UART behaves," and build a safe wrapper around it that is hard to misuse. Eh I would say the thing that pe- people forget about Rust is it als- is also a very strong e- encapsulation language. And that bit obviously So at the very bottom of the stack, the first question is: how does that hardware actually behave? And it's being answered in C and in Rust in the same way. I am actually, on the other hand, surprised of how many people nowadays, even if it's a nascent community, actually go and say, "We'll write it all in Rust because then it's all in the same language." So it's more of a... So that reduces the tool set we're, that we're using. I would've expected that we are, we've spent way more time in mixed land between C and Rust. Yeah. But those are... This ends up, again, as I said earlier we're now in a place where organizations that do actually spend money on these things and time, or even open source projects spend quite some time thinking about these things, thinking, thinking these things through before they actually do it. Yeah. Jonathan: Yeah. Are there any complications with Rust handling different iterations of an ISA and the different instructions that are available? And I'm thinking of things like, AVX-512 and the... For the longest time people thought, "Oh yeah, that's not very interesting. That's only for data science." And then someone came along and said, "Hey, you can do string compares with AVX-512." And some of those other things We can make things so much faster now. And getting that, getting support for that in, the C standard libs and the kernel, and then- ... being able to have one binary that can on the fly detect, oh, this processor does AVX-512. I should use this new advanced string comparison routine. I- is that sort of stuff possible in Rust too? Florian: Yeah, that is possible. This is something where Rust came after these concerns and C came up, so we can learn a lot from that. The, on the other hand with everything that touches ISA, we're based on LLVM, so- everything we inherit a lot of problems and a lot of benefits from LLVM. And if, for example, LLVM is not as well, as good in using some of these instructions as, say, GCC, you're out of luck. Or y- you can always hand code things not only in assembly, but also, for example, by using compiler intrinsics by saying, "Hey, please insert these instructions here and there." But- ... it becomes very detailed. Yeah. Jonathan: Yeah. Florian: Yeah. Jonathan: I was intrigued to see that same process happening in, in RISC-V, which we're gonna- ... we're gonna talk to some RISC-V people here coming up in the next few weeks. Cool. They, I think RV23 is what they call it, and that's their essentially next version of the ISA that's got- a bunch of hardware accelerated things. And, so with- ... with Ubuntu they've just said, "We're only gonna support RVA23 going forwards because it's so much better and it's the n- ... the next thing." Yeah it's just, it's real interesting to see the way that the different projects, the different languages have dealt with that. Florian: The benefit you have in an early stage, and I would still consider RISC-V even though it's, I know we're working on this for a decade now, it's still early stage is that usually communities across the board, even corporate communities still have a habit of, yes, we're betting on a new technology. There will be two to three migrations- ... Florian: To, to a better thing. So you're still in an environment where things were not... Like where it's totally okay to go to a manager and say, "You will probably have to do a compiler update in two years, and you will probably have to re-engineer some of your software in two years." But it's manageable. It's these, as long as you can say it's these and these and these things. And can give people a rough feeling. I g- I, I always say that you're always dealing with humans. It's not there's always this picture being painted of industry always wants everything to be stable. No, they also know that they need to change and that new technologies come with that. The biggest the biggest fear I have for RISC-V is they're going up against some pretty big behemoth that can basically just undercut them at every at every corner. So let's see how that works. Jonathan: Yeah. I think the advantage RISC-V has is that those behemoths are already using RISC-V in their technology. Like Intel- Florian: Yeah ... Jonathan: famously, the Intel management agent is a little RISC-V core. Yeah. And it would not be... And now one could make the argument whether or not Intel is the behemoth anymore, but RIS- RI- I think RISC-V is here to stay just because it's- Yeah it's in so many things. Florian: And just to be clear that I don't wanna spread any kind of FUD. It's just like these things that grind my gears. It's how- Sure ... are they approaching this problem? As you said, they've already been around for a decade. They're, as you say, they're here to stay. Also this is something I think the time where people who are really into these technologies, like the, there was a time in Rust for about two years where I knew where Rust was being adopted, like a number of larger companies that would later come out. And most of the time, even everyone in the Rust project knew. And you're sitting there and you're like, "This isn't very public news yet that we basically being used at all FAANG companies." And you're still you're at conferences where people's "Yeah, this is still this nascent language that is used nowhere." And you're like- I've seen- ... just cannot tell. And I- Yeah ... I probably think the same thing at RIS- is going on at RISC-V, maybe. Jonathan: Yeah, th- that's probably true. I've gotten to look just a little bit behind the curtain there and I've seen some things where- ... it's, a lot of hard drives for instance are running RISC-V cores. ... Intel is running RISC-V cores inside of... I I joke, it's not really a joke, we don't have any x86-64 processors anymore. None of them are native, and it has not been for a very long time. None of them are- Yeah ... natively x86-64. They're all something else and they're emulating x86-64 instructions, which is a wild place to be, but that's, that's fairly accurate. All right. Let's z- let me think. You got a conference. Let's talk about the conference. What is Oxidise? What's the story with that? Florian: Oh. Oh I c- I ran, I usually joke I sometimes accidentally run conferences. Jonathan: Just fall right Florian: into it. So Jonathan: I- Stumble into it. Florian: Yeah. I I had a Ruby conference called EuroCamp, and then later when I was starting in Rust, people were like, "We need a European Rust conference." I got going to run one. And that was RustFest. Which actually ran for 10 editions before the whole pandemic basically killed it off. Yeah ... it's been replaced by other events in Europe. The European conferencing scene does well. But at some point I figured out I actually want to have an event. And that was in parallel to RustFest. I want to have an event where we're talking more about things that people are actually doing with Rust, not necessarily, but also that often happens to be like what do you do in your day job with Rust? So what is the thing you're building? Or if you are, for example, building a major open source project what are you doing there? And I wanted to have a conference where it's less about, okay, this is what the language can do, this is what the type system does, and he is, he has a plan for making the Boa checker more powerful or something like that, like all of these things that are flying around. But more like having a place where I can, where s- someone who comes to me and says "I'm putting Rust on microsatellites." I'm like, "Then go to this conference, talk about microsatellites, talk about it." And I'm trying, I always try to encourage speakers as "Actually, don't hold back. Don't do an advertisement talk for your product." So it's not about that, but, "Hey, please talk about the actual full thing but that you're building. If it's a piece of hardware, bring the hardware, put it on, put it there, and then say, 'This was my actual experience in good or bad,' like proper in a proper engineering fashion. That was my experience of putting Rust on this thing." But not speculative, but more as a you share your experience or your current struggles with w- with Rust deployment. Summed up as the conference for applied Rust- ... in that sense. And I wanted to have that, and the, decided to basically run my own conference cycle and starting it at that time. Actually that was also hit by the pandemic in a weird way. It had multiple experimental editions over with s- some online formats. And nowadays- Virtual Jonathan: conferences Florian: and hybrid. Yeah ... it's a yearly thing that happens in Berlin. Yep. Yeah. Yeah. Jonathan: Cool. It's in, September? Florian: It's in September in Berlin. Jonathan: Yeah. I probably will not be able to swing making it out there. It's too far away from too far away from day job stuff. Florian: If you're going to Embedded World, we meet at Embedded World then. Jonathan: Yeah, absolutely. I don't remember if I ... I don't think I talked to you at Embedded World, although I may have. I talked to a bunch of people at Embedded World this past year. But I- ... I'm pretty sure that's how this conversation came to be, I got somebody's business card at Embedded World. I've joked- Yeah I need to print business cards that have Meshtastic on one side and Floss Weekly on the other. Yeah ... that way I can give it out and people know who I am. Florian: But it's the you'll end up in the open source and Rust corner. So we've actually managed to merge all of our booth. We have a cluster of companies there who are actually all household names in the open source- Jonathan: Yeah Florian: Scene. And Embedded World is actually, from that perspective, always interesting because embedded used to be that space where, yes, open source exists, and there's maybe this little bit of Yocto and things like that. But most of it, it was you buy, you it was very much a culture of proprietary software, proprietary- Right tools. You buy your BSP and things like that. And over the last five years, that has changed so much. Yeah. Especially there in Hall 4. There's so many open source companies- ... running around. Jonathan: It was very cool to walk around and just "Oh, I know that name. I know that name. I know that name. I'm gonna go talk to all of them and get business cards from all of them." Yeah. It was a lot of fun. Yeah ... some interesting people there too. I talked to QNX and it's- Yeah ... like I would love to have them on this sh- in fact, we have the guy from QNX on the show, and he's "You realize QNX is closed source, right?" I'm like, yeah, absolutely. But you guys do ... " They're a big part of Eclipse and, ... some other things that they're really big into. We'll have them on and talk about that. It, yeah, but it is, it was really fascinating to Florian: see. Are you going to have them on? Jonathan: I'm gonna have QNX on the show. Florian: Oh, very good. Jonathan: Yeah. I will ask him. I'll be like, "Hey, what's up with this? Why isn't it open source?" And I told him when I talked to him about at the booth. I'm like, "I'd like to have you, and this is what I want to do." I'm like, "Let's just lampshade it. Why in the world is QNX not open source?" I'm like, "I'll tell you up front I'm gonna ask the question and you can have a good response for it." He's "Okay, that's fine. We can do that." So- Florian: Oh, yeah ... Jonathan: look forward to that. All right. So- Yeah, Florian: great ... Jonathan: what's some weird things that people have done with Rust or something that people have asked you? And I, obviously you can't share every story, but some stories you can share. What are some weird things that you've done with Rust or that you see people do with Rust? Florian: Oh weird things. As we, for example, write a part of a writing Rust analyzer, the IDE, we know a l- a lot of Rust code that's incredibly weird. Yeah. But I think then weird things with Rust, Or Jonathan: s- or surprising things Florian: I'm just trying to figure out what's Jonathan: What you're allowed to talk about, right? Florian: Yeah. No not allowed to talk about it, but just trying, it ... there's fun place things all of, all the time. There, there was someone who built a compiler back-end that compiles to Redstone, which is that Minecraft, that computer built in Minecraft. Jonathan: Oh oh, I know what you're talking about now. Yeah, that's pretty good. That's a pretty good Florian: example. Things like that. Jonathan: Rust Florian: in Minecraft. We have people building all kinds of CLI tools. Someone over the pandemic thought they're going to rewrite g- new core utils, and suddenly they're successful things like that. A lot of these things aren't actually planned. I have, I've personally, I've started very much at the beginning, I've rewritten SL, the I don't know if SL? Jonathan: It's, it's- Florian: Most important ... Jonathan: I don't know if that's a core util, but yeah, that's, Florian: It's a, it's a- I'm trying to remember exactly what that- If you mistype LS, it runs a steam locomotive. Jonathan: That's right. Okay. That is not what I was thinking of, but yes, I know what you're talking about now. Florian: Yeah. But I have a patch in SL, which is my claim to fame in the open source community. It's not that Rust stuff. I have a patch in SL. The Rust community in general is pretty playful, especially on the embedded side of things. G- the the Hulks Robots team which we're supporting th- they're doing robot football. Like the interesting football, not the thing that's currently on TV. And one of their robots is built, Jonathan: Wait ... PyTorch. Which kind of football is the interesting football and which is the one that's currently on TV? Now wait Florian: a second. Oh you called it soccer, right? Jonathan: So we call f- the European football we refer to as soccer, and then there's American football- Florian: Yeah ... which is gridiron. No, so yeah, so it's a robot soccer team. Sorry. Jonathan: Okay. Florian: Yeah. So they have tiny robots and- I'm still trying to work out whether I'm Jonathan: offended or not. Florian: Yeah. It's all good. So they have a robot where the firmware is written in Rust. Oh, Jonathan: that's good. Florian: Things like that. We... I there's a number of things where people do... So there's a whole subset of where people are using Rust in medical software. But because they've done it very early, they've basically used it for prototyping. Jonathan: Okay. Florian: And then the and then have rewritten everything in C at the end of it be- because that's safety qualified, so it's safer suddenly because yeah. You paid for the tool, all of these things. Jonathan: Checks out. Florian: Ah, weird. Jonathan: I know we got Linux- Yeah ... on Mars. Was Rust part of the part of any of the Mars missions? Do you know? Florian: Yeah. Yeah. Jonathan: Did the helicopter run Rust? Florian: Exactly. Jonathan: Yeah. Florian: It's going everywhere, but the thing is I don't, I wouldn't say that there's also a ton of weird things that people are doing with C. Sure. And it's I think both of the communities in that f- on that front have a pretty similar vibe going on. And I find- ... I find especially the hardware communities to be, i- in the positive sense very hacky, ... and stuff. We're seeing Rust in the demo scene now. Yeah. Jonathan: Yeah, that's an interesting, that's an interesting one. I'm trying to remember if I've seen anybody port Rust back to the Commodore 64. Florian: Oh, I can totally see that. Jonathan: Yeah, that'd be fun. Florian: Probably. Oh we had someone who we have someone who maintains something they call Visual Rust, and it's, Of course they do ... it's V- Visual Studio 90 ... So it, it's Rust compatible with Windows 95 and Windows 98. That's great. Okay. And I just needed to check whether that's actually public. The fusion reactor they're building out of Boston, Fusion Commonwealth? There's some of their simulation software is in Rust and beta and on their GitHub. Very cool. So there was like fusion reactors in Rust, why not? Why not? ... people because Rust has some influence from Python, people are using Rust as a fast Python in a way. Jonathan: I can see that. Yeah, I can see that. Yeah. Python s- surprisingly is used by a lot of academics and researchers- ... like that, and it's been impressive the performance that they've gotten out of it. But I can see someone wanting to go to something that feels similar but gets a little bit to the next step on the performance. Yeah. All right. So is there anything, and this is another one of these hard questions. You gotta do some set math in your head. Is there anything that we didn't talk about that we should have? Anything I didn't ask that you wanted to cover? Florian: No, I think, ... I think I'm actually good. All right. With a good conversation running through everything. Yeah. You didn't ask about the rewriting that much, though. Jonathan: Oh, the rewrite it in Ru- that's just a running joke. It's obviously it's happening a little bit, especially things like uudutils and the- ... we talked about the replacement kernel where they're trying to rewrite it in Rust. Florian: I actually have something like we, we were part of the Sudoers rewrite project. And the one thing I said is like these things aren't often that competitive as they are. So first of all, the, one of the reasons why these things are getting rewritten is that because there's actually people asking for it. But one of the good things particularly about the sudo rewrite was we, first of all, before we started with this, we got in touch with the sudo maintainer, the sudo, the, like singular. That, that singular. Oh, Jonathan: that's a little terrifying, but okay. Florian: Yeah. And we were like, "Hey there's we got a request to rewrite sudo." And the response was basically, "Oh, more eyeballs on sudo is always good." So both of these projects are actually talking. And he also advised us on a lot of things where we can... this is one of the benefits of a rewrite. Sudoers does not support everything that sudo re- supports, and that's actually a conscious decision. Jonathan: Get rid of some of the cruft- Florian: Because- ... Jonathan: from over the last 30 years. Florian: It's not only cruft. It's if there is a platform where there's two or three users that really need it, sudo can't kick that feature out. We can because those people just don't move to Sudo RS. Jonathan: Right. Florian: Those people just use old Su- Sudo. And the, I think the most important thing there is w- that these rewrites are done with the maintainers. And the other thing about UUtils, it was really just someone was searching for a thing to do while they, they were at home one time and were like- I know, we Jonathan: did. We've interviewed him. We- Yeah ... we've had both UUtils and CoreUtils on the show. It's been- Yeah ... it's been a lot of fun. The- Yeah ... with the, with Sudo RS, have you guys done the same thing where you share a a test suite and, go back and forth between the two to make sure that, you have your test coverage and you do the things that you intend to do the same way- you pass the test- Yeah suite in the same way? Yeah. Florian: Oh, that's a good question. Now you caught me. I currently don't think so. I need to check that, though. Jonathan: Okay. Yeah. That's one of the, that's one of the really interesting bits of synergy between the two Core Util- CoreUtils and UUtils- ... is that there's they share... I don't know if they literally share the same code, but they have this shared database of test suite tests. Yeah. And it's one project will implement a new test, and then go to the other project and go, "Do you have coverage on this?" And if no then let's add that test suite here." And then, you know- they find all these weird edge cases because you're- ... re-implementing it, you find an edge case, and then you go to the other project- Yeah ... and "Were you aware of this edge case?" And sometimes the answer is no. There's been bugs fixed over in CoreUtils because UUtils was implementing an edge case. And, Yeah, as it should be ... it's been pretty neat to see the two projects coexist and both get better because of it. Florian: As it should be. Jonathan: Yeah, absolutely. Yeah. Okay. I gotta ask- Cool ... two final questions. I will get emails if I forget this. What is your favorite text editor and scripting language? Florian: Oh, let's let's start in the back. I used to be in p- president of the German Ruby Association, so it's obviously Ruby. That, So I always joke all languages that start with R-U are good. I had a fun, fun thing. O- one fun thing. I was actually in a way recruited to the Rust project by Steve Klabnik, who was previously one of the biggest contributors to Rails. So I I'm very thankful to the Ruby community. So that will be my favorite. Problem is most of the people in my company don't agree, so we're producing very little Ruby at Ferret Systems. And my first business was a Ruby business. Yeah, Jonathan: yeah. Florian: Text editor, stock VI. I used to do- AV server ... a lot of systems administration work. Exactly. Yep. AV server has a stock VI, so I'm, Jonathan: Even your router ... Florian: I don't cons- I I don't configure my text editors. And Emacs is, I don't have an I don't, I haven't even spent enough time with Emacs to even know if I should be doing it or not. Jonathan: That's so true of so many of us. "Oh, I've heard great things about Emacs. I don't have time to l- dive into it and learn it." Florian: Yeah. Yeah. I had a well-running Spacemacs configuration at some point, and I actually quite liked it. It just, Spacemacs starts for s- 30 seconds, which is way too much if all you wanna do is change a YAML file. And I will probably get an email about, "Yeah you should've configured it right," and that person is probably also Jonathan: correct. You're pro- yes, you're probably correct. All right. Yeah. Awesome. Hey, it has been a blast to get to talk to you, Florian. I appreciate you so much coming on the show and talking about Rust and Ferrous Systems and all the things going on there. Thank you so much. If somebody wants to find, if somebody wants to find you, find Ferrous Systems, or learn more about Rust- where are some places they should go to? Florian: Obviously our website. Send me an email first name.last name@ferrous-systems.com. Meet me at a conference. We got a list of conference places. In Berlin there's the Rust Meetup, which is super active. Okay. Mostly because I'm not leading the Rust Meetup anymore. If anyone he- here's running a meetup, the first thing you should do is find 10 people who run the meetup and be the backup because then this thing will go on for 10 years. And I'm always around the Rust Meetup. There you go ... so I'm attending all events. Jonathan: All right. Florian: Yeah. Jonathan: Yeah. Very cool. Florian: In general, if you're running a meetup, you're the backbone of the community. Thank you very much. Jonathan: There you go. Florian: You too, those that listen. Yeah. Jonathan: Yeah. All right. Awesome. Okay. Appreciate it so much. Okay. That was indeed Florian Gilcher, as we say in English. I don't think we've made it during the show, but he was telling me that he has trouble saying his name with the German pronunciation when he's talking English. It's one of those, it's one of those quirks that was telling him about my experience talking with German friends and hearing them say English things, and it's just, it's the languages, it's such a fun experience. We do have some really cool stuff coming up on the show in the future. We talked a little bit about that. Next week we're gonna talk with a new friend of mine, Trish Shurlooker, about FLOSS and legal issues. He is a lawyer. He is a lawyer that came to the Ubuntu Summit of his own accord, and so that was that was pretty incredible to get to meet, and he's got some great stories. And then the week after that, we're talking with Andy and Aaron from QNX. We're gonna ask them why it's not open source, see what they have to say. And then the week after that, I am super excited, we're gonna talk with Andrea Gallo, the president of the RISC-V Foundation, I think it's called. But very technical guy, and we're gonna talk about RISC-V and what it means for the ISA to be open source. We've got a, we've got a blank after that, and then coming up after that, we're talking with Michael Meeks of Collabora. And then we've got w- a returning guest, Francois talking about smoked meat. We talked po- poutine last time. We're back with smoked meat. I love the meat theme that they've got going on. Anyway a the schedule is filling up, so be sure to catch all of those. We appreciate everybody that's here, whether you watch or listen, get us live or on the download. Thank you very much, and we'll see you next week on FLOSS Weekly
• Episode 871 - Rust Won't Save You 17.06.2026 1val 15min

  This week Jonathan chats with Florian Gilcher about Rust and Ferrous Systems! How have we gotten here, what's coming next, and what's new in the Rust world? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• Episode 870 transcript 10.06.2026

  FLOSS-870 Jonathan: This week I talked with Alexander Neumann about Restic, the backup program written in Go that runs everywhere and most importantly will restore your backups. That's important. This is Floss Weekly, episode 870, recorded Tuesday, June the 9th, open source gardening It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett. Back after a break, after a respite, longer away than we really wanted to be because there was sickness in my family, that I had to travel all over the world. I've been to London, I've been all over the place, but we're back. And we've got a show for you, and we've got something really interesting. We're talking to Alexander Neumann about Restic. And I know that is a backup solution, and I know that it's not written in Rust or C. It's written in yet another interesting language, in Go, I think. And I've got questions. Like, why did we need yet another backup engine? What's wrong with the ones that are out there? And why should somebody use Restic? How does someone use Restic? I have questions, and I bet we have answers, 'cause we have Alexander right here. Welcome to the show, sir. Alexander: Yeah, thank you very much for having me. Jonathan: I am glad to finally get you. We, I talked about I was sick. My family was sick about a month ago. You, too, were on the schedule and then came down with a bug of some sort, so we had to reschedule it. It's good to finally have you here. Alexander: Yeah, it's an honor to be here. Jonathan: Yeah. So you are, first off, I forget this half of the time, but you're actually speaking to us from Germany, right? Alexander: Yes, that's right. Jonathan: All right. And I'm in- We say- ... southwestern Oklahoma, so we're, you know- ... it's almost nighttime there, and it's morning time here. Time zones are real. We don't believe in flat Earth around here. Alexander: Time zones. Time zones are my, my, my end boss. Ah. Because I always tend to mix up time zones, Jonathan: yes. And, the tools you would think that would make it so easy to get it right, Google Calendar and all those that do it all, those will get it wrong too sometimes. I found that out the hard way. No, I thought you were... No, I thought..." Yeah. And then but what really gets us is when I think the US does time change first, and then Europe does two weeks later. Sure. So we've got about two weeks there where nobody knows what time it is. It's the worst. Alexander: It's the same here in, in Europe because some countries do it earlier and some do it- ... later. It's not some like synchronized time zone all over Europe. We also have several ones because Europe as a whole is quite big. Jonathan: Yes. Understood. All so you do Restic and backups. Now, w- what's... I guess give me first the overview because there's a lot of things that can go into backups, and that can be handled- ... a lot of different ways. What's the overview of Restic? What does it do and how? Alexander: Yeah, may- maybe just a step back because at I'm working as a penetration tester in IT security as a day job. And in 2014, I was so fed up with all the different backup solutions that were out there because no- none of them seems to be suited to my use case. I wanted to make a backup of my home directory to a third-party server that I'm not the only root account at that server. So I wanted to have this backup location. It's not necessarily trustworthy. And so I looked at all the different backup solutions and all the bash scripts and Rsync and Rsnapshot scripts that I've written over the years- ... and decided that something new must be done about that. And so Restic was born, and at the time I didn't know that, but there was s- a software called Attic that worked in a similar way, but it, I did not discover that. And after starting the project, I also made a list of all the backup solutions that I stumbled across, and it turns out there are a lot of backup solutions. Oh, yeah. Yeah, quite a few. And I still get pull requests for including this or that backup program. And even after Restic was quite popular people started writing new backup solutions. But that's totally fine with me because it's not the backup that you want, it's the restore that you need. Restore is much, much more important than backup. Jonathan: Yeah, interesting. So you said you were in the you're in the security in security field, IT security field, and that brought to mind something funny I remember from several years ago. When ransomware really started to be a thing, and I saw advertisements a- and was it an advertisement or was it something in Windows saying you... essentially, "You need ransomware protection." And I remember thinking at the time "Isn't that just called a backup?" Alexander: Yeah, de- depending on some- sometimes it is. So in, in some engagements, we managed to compromise the backup service of some company, and then you could, in theory, delete the backups or tinker with the retention period so that they only have a backup for the last day or so. And if you then start deleting files and encrypting files, then usually the company has a very hard time restoring from that. Jonathan: Yeah that's something I've actually thought quite a bit about. I've done my own share of, writing rsync scripts and all of those different things. And I've thought about that quite a bit, because I have had a customer... i've, I wear many hats, and one of the hats that I've worn for a long time is a managed service provider. And I have had a customer get hit with ransomware on one desktop. Thankfully, we caught it before they were able to pivot into anything else and had a good backup solution. A couple of... Actually, we had two or three different backup solutions, and one of them was easy enough to be able to recover everything. But I've thought about that quite a bit. Like, how do you set a backup up so that even if an adversary gets into a machine, they can't, at least not trivially break your backup? And so there has to be an element of things have to... You either have to have another trusted machine that's gonna do this, or there has to be some sort of like you're, you pull instead of just push. And so we... This is a complete rabbit trail away from just talking about Restic. But it does bring you to the qu- lead into the question, does Restic have some of these protections? Can you set Restic up in such a way that it, it is indeed a ransomware protection? Alexander: Yes, you can do that. So what we started a few minutes ago with my special use case- ... that I would like to back up my files to a third-party server that is not necessarily trusted completely. At least I had to trust it to not lose my files. So this is the baseline. But apart from that, Restic is written in a specific way. We can get into the details in a bit, but you can set it up that you only... Restic is push, so you upload your new files that Restic creates to some third party storage provider, and there are providers out there, for example, Backblaze or sometimes S3 providers also provide this option that you cannot really delete files, but they are only hidden for some amount of time. For example, you can configure in your account that files are not deleted instantly, but instead they are just hidden and only deleted, really deleted after 60 days. So if you have this with ransomware scenario where your files are encrypted and then Restic only backups the encrypted files, and then the attackers even manage to gain access to the host running Restic and use the credentials stored there to delete the files. If you caught the adversary within these 60 days, then you can just easily restore the hidden or deleted files, and then restore from there. Jonathan: And if you didn't realize that your files were encrypted until 60 days, they probably weren't that important anyways. Alexander: Yes, and this is why we implemented our own storage backend that you can run locally yourself. And there you have the option to have append only backups, where you only can append new data, but you're not allowed and you cannot, from the protocol side of things, you cannot really delete files. Instead you would then do the backup pruning and removing of old backups from a different host that's, or even do it manually from time to time. Because at the end of the day, storage is very cheap these days. So then you can store 10 years of data there. That's not an issue. Jonathan: Storage is cheap until you have multiple terabytes, and then it gets really expensive. I know. I've paid the bills on S3 and various places. Yeah, that- Still have some of those I'm paying. Alexander: That, that reminds me of the of the time where we accidentally discovered a bug or a faulty hard drive within Backblaze's service because a user reported that his backup did not check out. So we have a check function within Restic- where it's, where it downloads some amount of data from the repository, and does some kind of self-check so you don't have to restore the whole backup in order to make sure that it's, will still work. Restic has the ability to do that itself. That's really neat. And it selects randomly some kind of files that it downloads and checks. And with Backblaze, there was always an error when some file was downloaded, and then we contacted Backblaze and they fixed the error in one of their storage pods. And yeah, that that was when we found a bug within Backblaze. Jonathan: Interesting. Yeah, that shouldn't be possible. That's the whole point of Backblaze, that's not possible. Okay boy, this is really fascinating. With Restic then, it, there's two, two components to it? Three components? So maybe- Yeah, yeah, that's, ... you've got one that's gonna run on some local machine, and then potentially another one that runs on a server somewhere. Is there a third component in between? What's the architecture look like? Alexander: Okay. The architecture is basically there is this single binary, it's a Go program, so it compiles to a statically linked single binary, and you can do 90% of ev- everything with it. You can do backups, you can do restore, you can do also like a fuse mount where you have run Restic mount, and then you get a file system simulated on your machine that is mounted in a directory. Perfect. And then you can use all your normal shell tools or whatever to explore the snapshots and the files that are in there. That's nice. And the data is only fetched on demand. That's interesting. So if you have several terabytes of data and only need a single text file from somewhere, like an SSH key or something like that- then it's really fast and really efficient. So this single binary does everything. It also does pruning and backup retention policy application and so on. Yeah. And there is a second component that can be either it's the Rest server, which is one of the storage backends that we support. And you can run it on some machine, and then do an HTP connection to this machine, and then have Restic push new data there and store it there. So you can run the service yourself on your local machine or on some Raspberry Pi in your basement or something like that, so that you can host it yourself. And the same protocol is also implemented by Rclone which is a very popular cloud s- R sync copy tool. It can also u- be used as a server for Restic, and in the backend it can store the data on all the backends that Rclone supports, and I think that's quite a lot. Jonathan: Yeah. It's gonna be a bunch. The server side, does it have does it have support for multiple tenants? Or does it have any, does it have any of the advanced tooling, to be able to go into a webpage and see this guy didn't back up for the last 30 days, I need to go to his office? I come at this from that sort of managed service provider mindset. Alexander: It has multi-user support, so you can have multiple users with different- ... access credentials, but you don't have some fancy web UI or so. We try very hard to limit the scope creep for Restic because this- Sure ... is a real issue once a project becomes popular. And yeah, I- For sure I, I must admit that Restic's popularity exceeded all my wildest expectations by several orders of magnitude because when I started in 2014, I was alone. I talked to my colleagues about the architecture, but I was implementing this thing. And then, yeah, several years later, we have almost 30,000 stars on Res- on GitHub. Oh, wow. So that's quite popular. And everybody would like to submit their feature, even with code, but we are the ones having to maintain this feature over time. So- Yes ... but the protocol that Restic uses for this REST server interface is is, it's documented in a separate document, and it's very easy to implement this. This is just a basic REST API get, put post, delete and that's it. And it's very easy to implement, and I'm aware that several other storage providers have implemented their own solutions that you can just use with Restic. Because at the end of the day the product goal is that more people have the option to restore when they need it. And backup should be as smooth as possible because backup is the chore that, what you do and so that you have it when you need it. But if backup is complicated or backup does take a long time, or it doesn't run every time, or you have to manually remember to do it, then you don't do it, and you don't have a backup at the end of the day. Jonathan: Yes. I- if it's a well-documented REST API, it sounds like it would be pretty easy to just vibe code a dashboard. And I say that- Oh, yeah ... I say that halfway jokingly, but I also acknowledge that we're at the point to where you could probably do that, and it would probably work and give you a reasonably good result, too Alexander: which is incredible. Yes, you could do that. A f- a friend of mine implemented the server side of this protocol, like 10 years ago or so in just under 100 lines of Ruby code. So the server side is not that complicated at all. Jonathan: And I assume it has it's credentialed, right? It's got some sort of login system or a token system to be able to even get in and upload- Yeah, it- download data and see things. Alexander: Yes, it's just username and password. But maybe we can start from the beginning at the end. At, in, in the beginning I thought about and discussed with my colleagues and friends, and what would we expect from such a personal backup tool to back up my home directory to some server? And we came to the conclusion that it must be secure. The backup s- storage location may not be trusted entirely, so I trust it enough to keep my files, but I don't trust it enough so that some other admin on that server decides may decide to have a look at the files that are yeah, lying there. Encryption is not optional, but- ... very integral part of what Restic is. And it must also be able to check whether the dep- backup has been tampered with, and th- that was also very important for me. And since restore is much more, yeah, much, much more important than backup itself, it mu- must be very easy to simulate a restore, because you don't really restore or test restore every week. But if you have an automated process, for example, if you every week, if you download 10% of the repository selected at random, and can have some kind of like self-check where it downloads the data and makes sure that it got the r- the right data, then you can be sufficiently sure that a restore would be possible. And this turned out to be a very good idea because over the years we have we have even have a label on, on GitHub issues for that. We uncovered a bunch of hardware issues with hardware doing wrong things. Usually the CPU is broken or the memory is bad from our users' machines, which they tend to not believe at first. But I've run into this myself twice already where I was wondering why a Restic check returned wrong data and returned error with a freshly bought machine. That was a very good idea. Jonathan: Interesting. Yeah. Time to go time to go pull out Memtest86 and make sure your memory's not- Exactly flipping bits and losing- Exactly ... losing bits. Yeah, no, I've, that, I've found that several times on fresh memory sticks. It seems like first it was a real problem there for a while. It may still be. Yeah. But there for a short period of time, it was surprisingly bad. I had a probably 25% fail rate on memory sticks I was buying. It was really- Alexander: Oh, Jonathan: That's Alexander: really high. Jonathan: Yeah. Okay I was gonna ask... Sometimes my brain wanders and I've, I completely forgot the thing that I was gonna ask. I really like the idea of doing this this random sampling pulling data back down. That's that's really fascinating to me. Oh, I know what I was gonna ask. So you have encryption is mandatory. How many emails have you gotten then from people out in the community saying, "Please help me find my encryption key"? Alexander: I think only once or one, one or two, something like that. Oh, Jonathan: wow. I was, I expected more than that. Alexander: Me, me too, but Rustik is very explicit. So the first step for, in order to be able to do backups with Rustik is you has to initialize a repository that's very similar to Git, because I like the way Git is built. And I took a lot of inspiration from that, and that turned out to be a very good idea. So you need to initialize the repository, and it asks for the password, and it very explicitly states that if you lose the password, you will not be able to access the data anymore. And this is coming from somebody who breaks systems and encryption protocols for a living, and I've been doing this for quite some time, and this is really, if you lose the password, then it's gone. ... What surprised me more was that there was a very very vocal group of people requesting that Rustik accepts an empty password, for example- ... or that there be an option for, to disable encryption because they don't need it, their hardware is trusted, and so on. But over time in my day job, I learned that it is very important to test all the code paths and be very sure in what you implement, because once there is a feature that is implemented, like having unencrypted repositories, maybe attackers can find a way to trick users into saving their backups without encryption. So in Rustik, there is no code path without a password. You'd have to supply one. And if you create a text file, a password.txt, and save it alongside the repository that's totally fine with me, but I would like to have Rustik not have a code path where it works without a password. Jonathan: Yeah, i- indeed. And I'm sure there's somewhere in the readme it says please take your phone out and take a snapshot of that .txt file. Don't send me an email. I won't be able to break into it." So you- Exactly ... you were inspired by Git. Does that mean- Not just ... I'm gonna be a little bit of a troll here. Does that mean that you're storing backups on blockchain? Alexander: No, but it's if you want to view a blockchain not as a concept to burn arbitrary amount of energy b- by using this, by computing the proof of work and- ... but rather see it as an append-only ledger, then some concepts within Rustik are like that. So once you have initialized the repository, then you can do the first backup, and there it deviates already from maybe existing backup software because the backup workflow is very highly optimized for that it's as smooth as possible. So you can just Rustik init, and you tell it where the repository is, which may be like a local directory somewhere. Yeah, you it asks for the password, and then you can directly start backuping things. It's just like Rustik backup, and then you pass a list of files or directories and and it will start saving the data. And in order to do that, and you can e- just after the backup has finished, you can just run it again, and it will take just a few seconds at most because it has a lot of optimization going on in detecting which files have changed. But also if you have the same files over and over again, or you have similar files with, for example an image of some some embedded device which is several gigabytes in size, and you have a second image of the same device after you've turned it on and configured it, then most of the data will be the same. And Restic will read the first file, split it into blocks, and only store these blocks. Then the blocks are identified by their SHA-256 sum. And this is similar to Git, but Git uses SHA-1 I chose the more modern hash algorithm. Jonathan: They're moving to SHA-256. That's a good thing. Alexander: And yeah, that's Then it stores the it stores the blocks identified by their by their hash, and then it stores the metadata in adjacent documents. So the repository format it uses it's very well documented. There is a separate design document that I wrote alongside the implementation, and there are even implementations that can write and read the same format because at the end of the day, you don't gain anything from being able to restore your backup if the backup program is not around anymore. That's a bit easier with Go binaries because they are statically linked, so you just need some Linux kernel or some- ... some Windows machine to, to run it. But the backup format also is quite well documented. Yeah. Yeah, so the first file is read. All the blobs are the file is split into blobs. The blobs are stored. Restic made a note which file consists of which blocks, and then it starts reading the the second file, and it sees almost the same blocks, and they are not stored again, so they are just stored once. And instead the second file will just reference the blobs from the first file. And yeah that's that What, what- The de-duplication ... Jonathan: that's cool. What size blocks do you use? Or is that c- probably user configurable? Alexander: No. Rustik has if I can if I can prevent it, Rustik does not have any configuration if it's not strictly necessary. For example, we have added compression. It was an, a feature added several years after Rustik's initial release, and we did it in a almost backwards compatible way so you can upgrade your repository and use all the data that's already there, because backwards compatibility is very important to us. But if we have the option to not make anything configurable and instead select a good default that works for most users or even, yeah, c- just have two or three configuration options that's a good idea. Because at the end of the day, most of our users will not be able to decide which encryption algorithm they want to use for their files, because they cannot really decide which one is best for their use case. Instead, we settled on just one encryption algorithm. And for the compression we use Zstandard. That's the one by Facebook. It was quite popular a few years ago. It's very fast and ha- has very good compression results, and this is the only compression algorithm that we support. It makes the software simpler. It does not have so many dependencies on different libraries. And for configurability we have three settings for compression. It's off, auto, and max, and that's it Because off when you have a very fast bandwidth to your storage location and don't care about compressing the data, you can just turn it off. Or auto is the default setting. It will compress data, and the Z standard is s- so optimized that you usually don't really yeah, don't really have any CP- additional CPU load by if you compress it. And then there's the max setting. That's for people who have very limited upload bandwidth, for example. If you have ... in a country where you have one or two megabits of upstream bandwidth but locally you have a lot of CPU power, then you can spend some time to really optimize the compression and really compress as much as possible. But that's it in terms of configuration options for compression. Jonathan: Yeah. And then I don't ... You may have said this and I just missed it. The actual block size when you f- when you carve files up, is that what, like 50 meg blocks and then it gets compressed or? Alexander: It's dynamic. So there are lower and upper bounds for blocks. Ah, okay. So the lower bound is obviously if you have a very small file of two or three bytes, then this is one block. But usually we try to have a block size between 512K and four megabytes, something around that. Small blocks, okay. And the algorithm that selects the block boundaries is a similar one to the one rsync uses. So if you have, for example, like a huge file and you have the same file with five bytes appended in front of it, then almost ... Then the ... If you have a static block size, then the first and second file will yield completely different blocks. And what Restic does instead is computes a sliding window over I think 64 bytes or something like that, and does a very fast hash computation that if the hash has some properties, then it will split the boundary the Split the file at that boundary and make a block. So it will recognize when the same block boundary is there again Jonathan: Oh, interesting. So it's almost got its own compression algorithm built into it. Alexander: Yes, the deduplication is hugely efficient. This is why we didn't at first have so much a focus on adding compression. But the compression itself also is, it's on a block level, so it's selects the blocks and only then compresses them. That's also quite efficient. But it was the most efficient thing from the compression that we got was that we, in the repository store, encrypted JSON documents because this was a serialization format that worked quite well for me when I started implementing it. And at the end of the day, these are text files, and text files are very well compressed with zstandard, for example. So this, Okay ... gave us a huge reduce in metadata s- size, for example. Jonathan: Yeah. Oh, that's clever. I like the sliding window. That's really clever. It's got its own built-in compression. That's- Alexander: Yeah, that was also- ... that's very neat ... i've studied computer science, and I was always fascinated from old papers from the '80s that are still relevant today. And there is the the content-defined chunking paper from 1983 or something like that from Rabin, and I managed to find a copy or a scan of the original paper, and it was all typewriter written and edits the small numbers edits with, by hand and so on. That was quite fun also implementing that. Jonathan: That's great. I like that. I like that a lot. So something you you referenced is the idea that you have people out there that, they have their pet feature that they would love to see in Rustic and you guys- ... you guys resist some of those. That's probably gotten a lot worse in the last six months, hasn't it? Alexander: Oh, yes. I'm a bit out of, I'm a bit out of the loop in recent developments because I don't have so much spare time anymore, and yeah. But this is, m- it was always the case that people tend to request features with very strong requests- because otherwise they will use some other backup program, and we just say, "Okay, then please do that. Just be sure to make backups." That's the important part. Jonathan: Exactly. Alexander: And the other thing is that we cannot, for example, we cannot really support all storage backends. And, ... people tend to like that their preferred storage backend is Build directly into Rustix, but then we defer them to just use our clone and use it as an adapter- which works quite well. But yeah, some features are, yeah, the discussions got a bit heated also because we as a project have limited resources in terms of engineering time and also in maintenance time. And keeping the project alive is the highest priority, and implementing features is somewhere below fixing bugs. So sometimes you just have to be very upfront about it and tell people, "Okay, this is a nice feature. I'm leaving the issue open so that other people will not request the same, but rather find this issue." But it is unlikely that it will be implemented in the next couple of years, for example. Jonathan: Yeah. Yeah, that makes sense. Have you gotten a lot of AI generated pull requests recently? Alexander: I've looked that up. There are sometimes people submitting pull requests that I, that are as I, I assisted and that's quite fine with us. So we will review them as other pull requests also. And usually people at the moment at least tend to be able to yeah, respond to questions about these pull requests. And I don't think we have some- sometimes we have pull requests that are obviously AI, and then sometimes re- reject them, and even when especially when people are not able to answer questions about that. Because then I could easily use Claude myself to, to code this pull request. So I don't need any people for that. Jonathan: I have my own Claude tokens, thank you. Alexander: Yeah. Jonathan: Thanks. Have you had any of the... This is something that blows me away every time it happens. Have you had any of them where you ask questions and the person on the other end is obviously just copying your question and pasting it into the AI, and then copying its response and pasting it back into the issue? Alexander: I've seen the reports by from the cURL project from Daniel. At this at... I don't think it has happened to us yet at least. But this is something f- for each feature that we get or for each bug fix that we get we need to do a thorough review. Because at the end of the day with the features that we integrate into Rustix, we have to maintain them indefinitely. Because Rustix as a project is only can only work if it's exists long time long term. And if people are even able to restore their backups that they did 10 years ago. So we'd be, usually we are very conservative in adding features or, for example, changing or extending the repository format. We had to do that when we added compression, and we did it in a backwards compatible way, which was it's technically not the most elegant solution. It's some kind of a hack. Sure ... but I liked it a lot because it lets people enable compression for newly uploaded data, but all the old data that is already stored in the repository m- remains valid, so they don't need to re-upload the data. And I think that was the right choice, but it was a very unpopular choice because the code looked not so elegant. Jonathan: I understand. The it's the e- the eternal the eternal war that happens within each of us as programmers. The, th- there, there's some things that are just you just have to do it. But it's like it, oh, it hurts to do it, but you have to do it. Alexander: And by the way, we have two more compression levels in the release that I published just a few minutes ago. There is also the fastest and better compression levels, so you have some n- not only these three, but now five compression levels. But apart from that most of Rustik is not configurable at all. Jonathan: So what about what about retention, data retention? Is that configurable, or is that just- Yes. Okay. Alexander: This was, that, that was a huge surprise for me because I thought when I started the project that getting the chunking algorithm and the crypto and the security right, this was the hardest thing. But it turned out it wasn't. Retention policy application is the worst because we would like to... what I had in mind when I started was, like you had, if you do one backup each day- ... and then after a while, you thin out these backups. So for las- last four weeks, for example, I have daily backups of my data, but for the backups that I did more than one month ago, I think just one backup per week is sufficient. I don't need any backup. And for six months ago, I maybe I only need one backup per month. And for more than a year ago, I'd only need one backup per year so that I can, I have a chance to discover very old data that has been deleted in the meantime. But apart from that, usually the more recent backups are the most important ones. Jonathan: Right. Alexander: So we implemented a retention policy that you can configure, and it turned out that people have very strong opinions about backup retention policies. And basically you can tell Rustik to keep the last 14 daily backups, the last six monthly backups, and the last f- like 10 yearly backups or something like that, and it will remove all the snapshots, all the points in time for which these policies do not apply anymore. So if you have yesterday I did two backups, and then I run the forget command and tell him, okay, to keep only one daily backup, it will remove the oldest daily backup for yesterday and keep the most recent one for yesterday. And then in a second step, it's called Rustik prune it will go through the repository and identify all data that is not referenced anymore, so it does some kind of garbage collection step. And there are also some heuristics that, for example, if you have several small blob small data blobs from small files bundled up together in a larger file, and if only one of them is not in use anymore, it will just keep the file as it is. So because it is not not a good idea to download all the files and re-upload them if only it will save five bytes or so. So this is something that you need to regularly do. And but you can also keep all snapshots forever. That's also an option if you have enough storage space. Jonathan: If you have infinite storage space, yes. Has there ever been like a serious bug in Rustik where data was lost? Alexander: I think at one point we had some kind of this bug, but we caught it early because we have a lot of unit tests- and a lot of integration tests. This is something that Rustik is written in Go, and Go makes it really easy to write unit tests, and it is also very fast, so we caught that. Usually what we have is people coming to us in the issues and say that, "Okay, Rustik reports that there is some kind of hash mish- mismatch." And usually it's some kind of hardware defect like memory or CPU or even storage in terms of the hard disk is dying and returning invalid data, for example. This is the most severe thing that I can remember, at least. Jonathan: Yeah. Interesting. And then you have unit tests. Do you do, fuzzing. Do you do fuzzing and coverage tests? I'm just curious. Alexander: We do, yeah we do coverage tests, and the Go test suite is already coverage-driven, so it tells you when there is new area that it uncovers. And Go, m- Go some, for some years, Go has had fuzzing testing integrated into the Go test command. I don't think we've... we've not enabled that for Rust yet. Jonathan: Okay. Interesting. Something that, and I don't know that this makes a whole lot of sense for Rustic just because of the kind of data that it's d- that it's processing and the way that it processes it, but I've thought for the longest time that AI-driven coverage testing would be really interesting. So y- you have a fuzzing harness, and then you watch the coverage on the back end, and basically you tell an you tell an LLM, "Come up with new, based on the corpus, come up with new data patterns to throw at this to try to exercise more of these portions of code." I've always thought that... I don't know of a project that's doing that. I'm sure somebody is. But I've, I don't know of a project that's doing that. But I've always thought that would be a really interesting approach to, to- Yeah, for, for- ... find bugs and security problems. Alexander: It is, but for Go you don't even need that. You don't need an LLM to do that. The Go test command has this integrated, and it also has means to modify the input, and it will automatically discover when new area of code has been covered. So this is really efficient. Especially for kinds like image parsers and so on, Jonathan: Yeah. Yeah. Super interesting. Now I haven't asked you this, but why Go? Sure- surely you could have rewritten that in Rust by now, or, pick your favorite language. Alexander: Yeah, sure. At the time of when I started Rustic it was 2014 and Go was out for some years then, and there was the Go 1.0 compatibility promise, so every Go code that you write will still be, yeah, can, you can still build it and run it with Go 1.X. And this was very appealing to me, and I wanted to start learning the language a bit more. I've used it for some toy projects before, but nothing serious. And then I started implementing it and came to like it very much because Go in, in contrast to other great languages such as Rust is optimized for readability. So it's very simple language. It does not have so many different im- language constructs. For example, you have to explicitly check every error, which becomes a bit repetitive, but yeah, you get used to it. And it's very easy to read, and also people who have never written Go before can grasp what the code is doing just by looking at it and not having, they, you don't have to study it for one or two years. And I've never really, unfortunately, I've never really got into programming with Rust because at the time when I started Rustic, then Rust was not really stable yet. And then I didn't have the time to really pick it up. Jonathan: Yeah, that's interesting you talk about being e- easy to understand, because I, for the longest time, and I still believe this to some extent, that, one programming language is going to be roughly equivalent to another in that I've learned how to program in C, I can go apply that to C++, and there's just a few things you gotta figure out with C++. Y- and, you can go apply that to PHP or Go or whatever, and the central concepts are gonna be the same. You just have to learn, the syntax that's different. And then I went and I looked at somebody's Rust code. I had the same idea. I went and look at Rust, and some of the things that Rust will let you do, I forget what it was but it's one of their, It's like where you do, it's almost like a ca- switch case, but it's- done in a very Rust sort of way. And I just, I remember- I think it's, Alexander: it's called pattern matching- Jonathan: Yes, pattern matching ... what you're talking about. That's what it is, yes. And I remember looking at that the first time and "I have no idea what this is doing. I'm gonna back away slowly and go back to my C++ code." Alexander: Yeah, what's also interesting for me at least, is that I, when I got more comfortable with Go, and that you have these concurrency primitives with channels and Go routines and so on. And it really gets you thinking about states and state machines. And having a few years of Go under my belt, I now write better Python code, for example. Sure. Because some concepts translate over to different programming languages. For example, the, in, in Go you have, most people write the Go code with a early out. So you have some- something that must be done five steps, and do step one if there's an error return. Do step two if there's an error return. And this is in contrast to Python code, which usually looks like a staircase. If that, and only then something happens. And I find Go code much easier to read because the happy path is on in, is left-aligned and you can follow it a lot better in a function. And you can also do that in Python, you just have to do it. Jonathan: Yeah you get into what Torvalds talks about with good taste in your code. And sometimes we get into bad habits and we write things that, from an objective standpoint are just terrible to look at. Yeah, it works, but you gotta sit there and stare at it for 20 minutes to understand what it's doing and all of the different, all of the different w- weird ways that it can fail or succeed. Yeah, absolutely. Alexander: Yeah, at the end of the day, a programming language is a tool to get a job done. In this case, make a backup and be able to restore. And and I'm not a fan of these culture wars of this programming language is better than the other. Just use the one that most suited to the job and that and it will be fine. Jonathan: Yeah. Yeah I would tend to agree with that. So w- I find it fascinating that you're thinking about such long-term backup and restore with Restic. It's Alexander: backup after all. Jonathan: And so I think about this sometimes with just I have floppy drives around. I do not have a floppy drive reader. I Alexander: have a a DVD burner lying around, but I've not read any CDs or DVDs in 15 years or so. Jonathan: Yeah there's that too. VHS, I have VHS tapes. I don't know that I have a working VCR. And you start thinking about these things, and it does, it brings to mind this idea of what, what of our media is going to become inaccessible here after a while? Sure. And so we've talked with we talked with some p- some folks, some projects over the years that are doing archiving work. One in particular was a group that was archiving source code, like off of GitHub, with the idea being that GitHub may not be around for- forever. There's another group that's archiving YouTube videos, because YouTube probably won't be around forever. I find that real fascinating to think about sort of those long-term archiving bits of work. And you mentioned 10 years, but in the back of your mind, how, like how long would you like this project to the project to be around, but also, like how old of data would you like to be able, for people to be able to restore with it? Alexander: That's an excellent question. For my personal data, which this all started with I'd like to be able to, like f- four or five years is usually sufficient. For the project that I do and the source code and so on. But I have an, a digital archive of all my paperwork with finances and so on at the moment stored in a paperless instance, and for this I'd like to be able to read for 10 years at least. But for Rustik, I think it sh- would be a good idea to be as backwards compatible as possible. So at the moment for Rustik is at the release 0.19.0, which has been released today. But it can still read all the repository formats that have been written by any released version of Rustik. So even the ones made in 2014. And this means that have, we have a bit of maintenance to do in terms of being able to read these old s- storage formats. But we haven't really changed the storage format since then. It's turned out to be quite robust, so the introduction of compression was- the biggest change overall. Jonathan: Do you have some of those old repositories snapshotted to be able to pull into your test suite? Alexander: Yes, in the test data. Okay. There is there, there are some old repositories archived as a tar. So we and we have like unit tests which try to restore data and exit it, and so on, so that we don't accidentally break backwards compatibility. Jonathan: That's excellent. I figured you would. You guys seem to be careful enough that would be definitely something you would do. Thinking about that idea of the age of backups, like- I've got digital photographs that are over 20 years old that I would hate to lose. And there are some things that I, I don't know exactly if I have a copy of because my backup system when I was a teenager was burning things to CDs and DVDs, and those don't last forever. Somewhere around here I've got some very old DVDs that every once in a while I look at. I'm like, "I wonder if there's any of those old photographs on there that I don't have anywhere else." Some recordings I made in college of classes and just all kinds of different stuff. So i- it's real fascinating to me to think about that idea of like how long is this going to exist and I could see this being useful to kids and grandkids in- Alexander: Oh, yes. Yeah ... Jonathan: 60 years, 80 years, 100 years. Who knows? But what is, what are computers even going to look like in 100 years? I- is there- Yeah ... going to be, like how many generations back are you gonna have to go like we do now. "Oh yeah, I've got I've got the big five and a half floppies." You have to have one computer to move it from a five and a half floppy to a three and a quarter, and then a second computer to be able to go from three and a quarter onto a USB drive. And then, then you can finally take that USB drive and plug it into a modern machine. So y- you, you're gonna have to have a 20-year-old computer to run Go, and a 40-year-old computer to run a Go 1.X. It's just it's wild to think about what things are gonna look like and- Yeah, definitely ... I guess also like the tail that our work is going to have and just that idea of, maybe my kids and grandkids are gonna run my code. That's just such a fascinating thought to me. Alexander: Yeah. I would like to, the, for the Rust project, I would like to, it, it to outlive me. That would be excellent. And I from the start I created an organization on GitHub so that it's not really tied to my person. I personally sign all the releases, but I'm not necessarily one who does the m- most of the work nowadays. And this is something that I would like to talk to about a bit like project organization. N- not meaning like how, who does the releases and how is code reviewed and so on, but I think it's very important to have a very positive attitude in the project and set the tone right from the beginning. Because there are these... i've been around for the late '90s, early 2000s when there was these huge flame wars on the Linux kernel mailing list and- ... sometimes, sometime later, years later, they realized that it's a bad idea and it drives people away, and this contempt culture is not something that, yeah, should, you should allow in a project. And then this was very important for me from the beginning, that it's the Rust project and the GitHub issues and also the Rust forum this is my small bit of the internet. This is my garden. And you either you behave or you're shown the way out. And I've, yeah, over, over time people from time to time people get into very heated discussions and sometimes I step in and say, "Okay, this is stop." And then I'm the bad guy saying, "Okay, stop. I will lock this issue now, and if you ever talk about it again, then please do it in a way so that you don't y- you don't drive other people away." And this is something that stuck with the project. Especially in the forum there are people- who stuck around for several years who- doing not s- not any coding, but who are helping other people with their backup issues. So somebody reports something that it's that their backup is strange or their retention policy does not work, and people in their own spare time step in and answer these questions, and I don't have to do that anymore personally, which is very great. Jonathan: Yeah, no that's interesting. I've seen projects take different shades of that. But I think some sort of, w- community management is really what it is. There has to be- ... some sort of community management because somehow the internet brings out the worst in people, and I think it's the pseudo-anonymous nature of it. There's not a, there's not a person in front of you that you're yelling at, so it's very easy to be mean when typing comments. Alexander: Yes. Jonathan: And yeah. Alexander: I also think some- sometimes people don't realize how their text is read by somebody else. And then I step in and point out that, "Okay, this can be read as very aggressive. Can you please tone it down, or can you formulate it in a, in the, in another way and not assume a bad intent but assume a positive intent?" And usually one or two remarks of these sorts is sufficient to bring the discussion back on track. Jonathan: Yeah. Yeah. That's good. Do you guys have a, do you guys have a policy on bringing culture war and politics into the project? Alexander: Not really, but I, th- this did not really come up. If I like Trump or not, that's shouldn't affect how the backup software is used, and Restic has a very liberal license. It's licensed as a two-clause BSD license, which is the most copyleft license I could think of. And I know that it's used in several different areas in, in, in Germany and all over the world where I don't have any control how they use it. But at the end of the day, if people can restore their backups, then I'm happy. So we don't have an explicit policy, but I Jonathan: don't Alexander: think- I actually- Jonathan: Yeah ... I actually think that is the way to go, and also what is intended by what it means for something to be open source. I've seen projects take very weird political stances before, and I always find it very weird. You end up chasing about half of your user base away just to, to wave whichever flag you wanna wave. So I am on Team It's an Open Source Project, and that's what it should be about. Alexander: Yeah. There, there are yeah, there are borders, of course. At the end of the day, as I already said it's our little garden- our little b- place on the internet, and if we don't like somebody or we, if we don't like something somebody said, then I, we will state it very clearly, and then maybe also hide posts on GitHub or hide posts in the forum because, yeah, that's not productive, and I- Absolutely ... like to be productive and help people. This is what it's all about. Th- this r- this reminds me of the the, some- someday somebody wrote a post on the forum and said, "Hey, did you know that CERN, the the nuclear research facility in, in, in Switzerland they use Restic, and they have the big instance, a- and I've discovered some slides here," and yeah, and then- this was very interesting, and d- and a day later somebody else showed up and they were, "Oh, yeah, I was the guy creating the slides, and we have a very big installation." And they have some 60K users or something like that. Oh, wow. And they have a, an automated Restic backup for each of them. So that's, that was really nice. Jonathan: Yeah, that's cool. That's always fun to see your work show up in, in unexpected places, but big places. Yeah. Alexander: Probably the most unexpected place where Restic was in the Arctic because I got a bug report. It was, like, a v- re- very long time ago. I got a bug report, and then at that time I still had time in the evenings and started fixing this bug, and said, "Okay, you can download the beta build from our beta builder software." And th- they responded back and said something like, "Okay, can you send me the patch, and can you show me the c- the commit? Then I can use the patch later on because our ship is leaving." And I said "Okay, but what ship? What are you from?" And then was from some American university. I forgot which one. And they had an Arctic expedition, and they used for saving their data they used a MinIO cluster, and they backed it up with Restic. And th- they said, "Okay I will have internet, but it will be very slow. It will be satellite internet over the Arctic. The bandwidth is very limited. Please, can you give me the patch and not the binary?" So that was really cool. Jonathan: Yeah. No, that's neat. That's really cool. Where is Restic at on the progress to being done? And so the... I g- I get the joke, and I say that on purpose, but at the same time there are some pieces of software out there, like- There's a there's a spreadsheet program that runs on the terminal. And it's not seen an update, like they've not written any new code for it for 10 years because it's like it's done. It does what it's supposed to do. The- there's no outstanding bugs. It's, it's done. It... Do you foresee a moment like that in Restic's future where it's like, "Okay we've fixed all of the problems. It does everything we want it to do. We... This is all of the backup, backends that it's gonna support. We're now in maintenance mode. It's pretty much done." Are you there? Do you see that coming? Alexander: Not really, to be honest. At some points we might be, we might declare that this is what version 1.0 and that's it. But usually with backup programs and computers in general, it's always more complicated, a lot more complicated than you think. So what's usually what I expect that we need to do and continue to do is fixing bugs and fixing weird issues because you cannot really, with such complex program, you cannot rule out that there are no, no bugs anymore, and people use it in unexpected ways. And this, I think, will continue for s- for quite some time. We have a lot of ideas what could be improved. For example, at the moment you can use Restic Restore to restore some some directory to some file path. You could also use diffuse mount, for example, to interactively browse within the in the snapshots and use your normal shell tools, for example. You can also... it would be very nice if Restic had a web UI, for example. If you can s- use like Restic Serve, and then it s- spins up an HDP server, and you can use your browser to browse on that because then you could run Restic where it's much closer to the data and just use HDP to access this data. And I s- I've started implementing this some, a few years ago, but it turns out that this is also quite complex, and this is something that would be nice to have. But it's not on the roadmap for the next releases. Jonathan: Yeah. Alexander: There is also something else that's is also very important. If you have like this, you have the backup step, which adds new data. You have the forget step, which applies the retention policy, and then we have the prune step where data is repacked in a safe way and re-uploaded and so that at the end of the day you can remove a few files from the backup storage location. And this last step needs an exclusive lock on the repository, which is not a really great design because all other clients that may access the repository at the same time, because you can backup from multiple machines at the same time into the same repository. They cannot really do a backup while the prune step is running. There are constructions which allow lock-less pruning, and this would be something that would be very nice to have for a 1.0. Jonathan: Yeah. I- is Restic, does Restic have a 2038 problem? Alexander: I don't think so. Because I was w- I was I was there when the year 2000 problem happened. So for example, for the timestamps, we used the ISO something string representation in adjacent documents, so there is no Unix timestamps. Jonathan: Yeah. For those that are not aware, 2038 is when we're gonna run out of y- seconds or mic- I think it's seconds in a 32 bits int seconds since the Unix epoch. Sometime in 2038 some subset of machines are gonna roll back around and think that it's 1970 again. Alexander: That'll be fun. Jonathan: Oh, yeah. That'll be great. Probably more There's a decent chance that we'll have more problems in 2038 than we did in 2020. Alexander: I also think so because at the 20 at the 2000, year 2000 there was a huge uproar and panic and people think the world will end. But it also led to a lot of people fixing stuff before the year 2000 showed up. So we got there yeah, without any m- major harm. Jonathan: Hey, we have still, the better part of a, more than a decade to panic. So yeah, we'll probably manage it. Probably manage some panic. Goodness. If you had to go over and do it again, or say Restic didn't exist, and yes, this is one of the, this is one of the pre-questions that he fed me, but it's an interesting one. Would you do it again? If Restic didn't exist, would you start the project? Alexander: To be honest, I would like to be in a situation where I don't have to do that again because there are so many great alternatives to Restic right now. There is Borg, for example, which is the fork of Attic, and there are other backup programs that also took quite a lot of inspiration from how Restic was built, and I think that's great. I think it's a very good idea to have options for people because not, nobody sometimes people don't like Go and then they use something written in Rust or Python or whatever. But if I were in the same situation 12 years ago I would start again, I think. Because at the end of the day, if nobody steps up and said, "Okay, I'll do it now," then nothing happens. Jonathan: Yeah, absolutely. I- is Restic anyone's day job? Is there like a business built around it? Alexander: I don't think so. Jonathan: Okay. Alexander: Because at the... We- we've started s- to get more requests for f- how can I donate to the project, but I'd rather spend time doing actual stuff than managing funds. So there is no as far as I know, there is nobody working on Restic full-time. We had this yeah, we have basically the project at the moment is mostly run by Michael, and Michael did a Ph- PhD. But unfortunately for us as a project, unfortunately he finished his PhD and also had a day job now. But there are people working on integrations for Restic. For example, there was Relica. Which was a, it was or is, I'm not really sure, a backup service that offers Restic in a nicer packaging. They added a web UI and so on around Restic, so they don't have to interact with the CLI binary at all. Because Restic explicitly supports that. There is a JSON output mode, so you can call Restic and read the JSON data back as JSON, and then use some kind of other tool to read that, and it was quite a nice service. And I suppose that there are more services like that, that we don't necessarily know. Jonathan: That, that seems like a natural evolution for that level of support and that long-term, Long-term maintenance for the project is, if companies, Backblaze obviously is one of the big ones that comes to mind, but in some of these smaller providers, they're using it for backups. It, they're, they are they're obviously interested in keeping it working and keeping things working well. And so you could imagine a few of those different companies putting a engineer in the, in the open source pot to make sure things are doing what they're supposed to do. Alexander: Yeah, but then you have to manage this this engineer. And so somebody has to guide them into implementing features and reviewing stuff. And this was also very hard for me, to be honest, to transition from somebody who writes code in the evening to somebody who's managing a project and doing code reviews. And oftentimes I'm, got very frustrated with reviewing code because it's so different than actually writing code. And sometimes it's even easier to just scrap everything that somebody spent a lot of time and submitted and- ... took the plunge and submitted code. It was, sometimes it's easier to just scrap it and write it myself. Yes. And this is the maintainer's conundrums. It's, there, there's no solution for it. You, at first you're a developer and then you get a manager and gardener and that, that's something different. Jonathan: In, yeah, indeed. It is. It is indeed. You've got the, you've got the interesting piece there that if as the gardener, if you do your job well enough, then your your sproutlings, those people sending you pull requests, they'll get better at it, and then eventually it'll be easier, you hope. Alexander: Yeah. Somebody, some- some time ago we had somebody who, yeah, submitted a lot of pull requests and had strong opinions and, but the code was good. And we didn't have the bandwidth to review everything. And at the end of the day, they got frustrated and then started a different project. And that was very unfortunate, but- Yeah yeah, that's the reality. We're all in doing this in our spare time for fun. If it stops being fun and becoming a chore, then yeah, I'd rather do something else. Jonathan: Sure. Ha- have you guys started using obviously Copilot has a reasonably good code review. Th- there's some other ones. CodeRabbit I think is another popular one for open source projects. Have you guys started using any of those for helping with code reviews? Alexander: No. Not yet. Because at the end of the day, the most hard the hardest things to review is not let, that the code is secure. This is very hard to do this with Go code, for example. But to make sure on an architectural level that it's a good idea- to actually do that. Also from a, like strategic standpoint, do we add another storage backend or not? This is not something that an AI could decide, and it's also not something that an AI could argue with somebody who submits this code. And there is, it's very important to have a human in the loop because this vision for the project, also this long-term vision, is not necessarily what people see when they want their storage backend to support, to be supported. Jonathan: Yeah. We have found in the projects that I'm in that those AI code reviews, they're really good at catching the dumb mistakes that a human might look over. And so sometimes that's just, a misspelling in a comment but also sometimes a little bit more subtle things. But a logic inversion where it's in, in this function you checked for this, but in this function you checked for something slightly different and you might wanna be aware of that. It is reasonably good at catching some of those things. Yes, indeed ... we found it to be pretty useful. All right. What what's the most surprising thing that you've heard of that someone has backed up with Restic? Is it, was it the Arctic expedition, or? Alexander: I think it was the Arctic expedition, because usually people don't tell me what they back up, which is totally fine. I don't really wanna know. But the amount of different, or the d- different people using Restic for so many different things. From time to time I get emails from people personal emails telling me that what's that Restic has saved their day or something like that. And this is really rewarding. But this Arctic expedition thing, that was quite something. Jonathan: Yep. You have a note here about an absurd interaction with someone you had concerning Restic. Alexander: Yes. This was one of the things that I mentioned earlier, where somebody was very sure that their hardware is not at fault. Jonathan: Oh, yes. Alexander: Because they tested Restic on a different machine and it has the same error, and at the end of the day it turned out that both machines were faulty in different ways. So sometimes Restic makes people very unhappy because they discover things that they- ... d- did not want to hear. Jonathan: Oh. Alexander: But it's a good idea to, to know that hardware is faulty even if they at first don't want to accept that. Jonathan: Yes. There are fun stories about the the various hardware bugs found in CPUs over the years. Alexander: Yeah. This is even a layer deeper, when the CPU itself is functioning correctly, but correctly means not really correctly. Jonathan: Yep. No I've heard of some of those. One of the, one of the very very early mini computer CPUs, you could put it into a loop that should last forever, and a couple of- ... a couple of engineers tried it. And, after about an hour it kicked itself out of the loop. Wasn't supposed to do it, but it did. And, they wrote in to the company that, who, DEC or whoever it was that made it, and it's "Ah, somebody found it." And the next, the next time they published the manual, there's a little asterisk there, "This command is not guaranteed to return." And then, you, we've had that in more modern processors too. There was a floating point error in one of the early Intel processors, and- Alexander: Oh, yeah, I remember. Yep. Jonathan: Yeah. All kind, all kinds of weird stuff that, we expect our computers to do what they say they're gonna do. That's the whole basis of this thing. And when it doesn't, it causes problems. Alexander: Yeah, this is something that, that engineering and computer science students at some point discovered, that floating point is not exact. But in terms of ab- absurd uses of Restic, there's also this ... I know of a company who has a lot of raw video material, and they use Restic to not only archive this, but also transfer it from location to a different location, because then they- they use a Restic repository, write all data there, and then transfer the files, because transferring a lot of smaller files that you can checksum before and after is much better than transferring a big file and only then discovering that something went wrong. Jonathan: Restic as a BitTorrent replacement. Alexander: Even that, because at the storage location, the repository is structured in a way that every file the file name for each file is the SHA-2 hash of its contents of the- ... encrypted version. So you can just use very simple tools to check at the storage location without knowing the password for the repository. You can check if bit rot has occurred and if something has changed. Jonathan: Oh, interesting. Yeah. That's a neat idea. All right. Is there anything that we didn't talk about that we should have? That's a Alexander: good guess. Oh, maybe a lot. One, one thing that also surprised me is that it's really important for a project to have good change logs. Because sometimes projects don't have any change logs or they paste the output of Git log into some GitHub release form or something like that. What we do instead is I've written a small tool to collect text files in a very simple format from a bunch of directories. And for each for each pull request that does something non-trivial, we require that there is a description within the change logs directory. And at first it's collected in the changelogs/unreleased directory. And whenever we do a release, we rename this directory so that then we can automatically generate well-written change logs from that. And this is it's not a Git log on the one hand, but it's not a complete explanation of what's going on. It's always a description of what is important for end users. So if I renamed variables or if I upgraded some library, that's not relevant. So it doesn't get a change log entry, but instead if some bug was discovered and fixed, or if compression was introduced, this is something that we want our users to to read about. And a- after using it for several years, I discovered that there are quite a lot of projects who use this simple program that I wrote some time ago to manage their change log. I think ownCloud is probably the most known one from that. Jonathan: Oh, cool. Alexander: But I think that's a really impor- important part of a project also. Jonathan: Yeah. I don't think I asked you what the what the encryption standard is that you use, actually. Alexander: Yeah. That's, it's AES 256. And the signature is made with Poly1305. And these were two of the encryption primitives that were available at the time, and that w- were sufficiently fast in Go. And so I had to make a trade-off between using something, a more standard construction, for example, AES GCM which is encryption and signature in, in one. Or building something myself. But since then, one of the Go crypto maintainers has had a look at the encryption protocol and did not find any issues with it. This is also something that I'd like to change and I would do different if I would, if I were to implement it again today. But now it's there and we have to support it. And it's fine as it is. Much more important is probably that we use a very intense key derivation function to get from the passwords to the encryption keys. And there are, like, different encryption keys for encryption and signing and so on. Jonathan: Yeah. AES 256 has really stood up the test of time in an impressive way. Yes. It's ... Yeah no, no real weaknesses in it have been found, and it's considered, quantum secure for the future and- Alexander: It's also very it's also much easier to do this to have this just attribute for a symmetric algorithm. The asymmetric algorithms always rely on some problem being mathematically hard. Yes. And if you discover either a new algorithm or if you have a different architecture, like quantum versus classic CPUs- which were not there when the protocol or the algorithm was designed, that's very hard to do. But for a symmetric algorithm that's much easier. Jonathan: Yes. Yes. I am wrestling with that right now- ... trying to figure out how to make an X25519 scheme quantum resistant at least, and not easy. Alexander: Indeed, yes. Jonathan: All right. I've got a couple questions that I've got to ask before we let you go. I think we have, I think we've gone through our hour already. And I will get emails if I don't ask you these. That is, what's your favorite text editor and scripting language? Alexander: My favorite text editor is Neovim. Jonathan: Okay. Alexander: Although I must admit I do use VS Code a lot these days. Sure. But with the Neovim extension. Jonathan: Okay. Alexander: And my favorite scripting language is probably Go, although that's technically not a scripting language, but you can, like- You can script with it ... use Go run and then- ... have a small file there. And it's usually I tend to ... Yeah if it's not Go, it's Python. Jonathan: No, that's fair. That's fair. All right. It has been an absolute blast to be able to talk with you, Alexander, to learn all about Restic. I'm ... Oh, I didn't ask. I'm gonna, I'm gonna cheat. I'm gonna throw one more question in here at the end because it was on my mind- Shoot and then ... Windows. Can you back up a Windows- Yes ... machine with Restic? Sure. How does Alexander: that work? We compile the binary for Windows and it just works. We don't have in the repository format, we have a adjacent document describing a folder of files, and if you have a sub-folder, then a different adjacent document is referenced. And so you don't have the slash versus backslash differentiation on the repository format. It's not ideal if you back up something on Windows and restore it on Linux or macOS, but it basically the basic functions work. You will get the content of the files. And some te- some things do not really translate well from one operating system to the other. ACLs, for example- ... on Windows. I'm not sure what the current implementation for that is, but there are some issues on macOS where file attributes are not really cannot be restored on other systems and so on. Interesting ... Windows was supported I think from one of the early releases. Somebody stepped up and did a port of Rustic to Windows, which is not so hard to do because Go just compiles- Yeah ... on Windows. Jonathan: Yeah. Very cool. All right. Appreciate the the last second answer there. Sure. Alexander, it's been a blast to get to talk to you. I appreciate it very much. Thank you for being on the show. Alexander: Yeah. Thank you for having me. Jonathan: Yeah, absolutely. All so that was Rustic, and was a lot of fun to get to talk about, and I've got some systems that maybe I need to start backing up with Rustic. Alexander: Please do. So we- You don't want back up, you want restore. Jonathan: There you go. Coming up in the the next few weeks, we've got Florian Gilcher of Ferris Systems talking, I think about something about Rust. And then we've got Tris Scherlicher talking about the legal system and Floss. Tris is a lawyer that I met at the Ubuntu Summit, and immediately he had great stories, and I immediately said, "Ah, we need to have you on the podcast." And then we're talking with a couple of guys from QNX, which that's gonna be interesting. And then we have at the 1st of July, the the first Tuesday in July, it's actually July 7th Andrea Gallo of RISC-V. He's actually the president of the RISC-V Foundation. And very much looking forward to that. And so a really neat lineup coming up over the next few weeks. I am very much looking forward to it. We appreciate everybody that's here, whether you get us live or on the download, those that watch and those that listen. And we'll be back next week for another Floss Weekly.
• Open Source Gardening 10.06.2026 1val 12min

  This week Jonathan chats with Alexander Neumann about Restic, a particularly compelling backup and restore solution written in Go. Why did the world need one more backup program? And what's Alexander's personal take on transitioning from programmer to maintainer? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• Episode 869 - Linux on Your Toaster 29.04.2026 34min

  This week Jonathan chats with Andrei, Mahir, and Praneeth, live on location at Texas Instruments! The team at TI has been working hard to provide really good Open Source support for Sitara processors, including upstreaming support to the mainline Linux kernel. We talk about the CI pipeline for these devices, the challenges of doing Open Source at a big company, and more. Check it out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• Episode 868 transcript 22.04.2026

  FLOSS-868 Jonathan: This is Floss Weekly, episode 868. Recorded Tuesday, April the 21st. Remove the Noodles. Hey folks, it's time for Floss Weekly. That's the show about Free Libre and open source software. I am your host, Jonathan Bennet. And today we're gonna talk about programming. We're gonna talk about, is it fair to call it vibe coding? See, that's become a prerogative. Pejorative, excuse me, and I don't know that it's entirely accurate. Hopefully not accurate for what people are doing in open source projects, right? Anyway, the tooling is getting better. The tooling is getting better around this. And one of the fun things is that there's a lot of open source tooling around this, and that is what we're talking about today. I'm chatting with Johannes Millan about parallel code. And something he calls super productivity. Let's go ahead and bring him on and we'll dive into it. Johannes, welcome to the show. Johannes: Yeah, hi Jonathan. Thank you very much for having me. Jonathan: I am it's good to finally have you, this has been scheduled for a month and a half now. Quite a while. Yeah. And yeah, it's good. It's good to have you here. And so your project is parallel code? Johannes: Exactly. Yeah. I've two big projects. I say one Super Productivity, which is an source product project I've been working on for I think, nine years now. And just recently I think two months ago I started to work on parallel code yeah, which is by comparing much smaller. But yeah, also at the moment I'm dividing my time equally for both projects. And so Jonathan: These are two very different projects, aren't they? Johannes: Yes, they are. Jonathan: Let's talk about first the older one which is, because there are some people that are gonna go AI and just check out. So we'll talk about the not AI stuff first and then yeah. And then we'll dive into the LLM coding and maybe how those two go together more than you'd think. So what is super productivity? Johannes: Yeah, so productivity is an open source to do Time Tracker app which I started many years ago because I'm working as a freelance programmer for, yeah, I dunno, 15 years now or something like that. And and for some project there was required to do jra time tracking. And yeah, as as most programmers I don't like to do repeatable stuff. And so I thought, oh, there must be a smarter way to use this, and this is how it all started. And yeah, I don't know, some somehow I stuck with it. It's like sometimes became a little bit of an obsession of mine and yeah, it was, even though there was no, never making money with it or something like that was never yeah, a focus of it. Yeah, somehow. I dunno. I use it myself every day. That's probably a big part. And yeah, I really enjoy tinkering with my tools, like to yeah to make my day a little easier for myself. And so that's probably the reason why I couldn't just drop it and yeah, I don't know. And then in, in the last year, that's probably worth mentioning it grew quite a lot. I think maybe also a little bit there also many new people on the project who contribute stuff who yeah, do testing, write back reports, and it's. It's really interesting how this changed. And but for the most part of the seven years, I think or for the nine years yeah, it has been mostly a solar project. Not totally, there were always seven people, but yeah, Jonathan: yeah. I've not met many open source developers that wouldn't say. They like to tinker with tools. That sort of seems to be something that's true of all of us. That's why we're here. We like tinkering with the tools. Johannes: Yeah. Jonathan: Okay, so this is a setup for you to answer this question because it's gonna sound a little mean, and I don't mean it that way, but it's just a time tracker. How hard can it be? Surely that was like, you programmed it in a day and it was done, right? Johannes: Yeah it's a good question because I think the first prototype back then was much worse tools and without AI and everything, it was done, I think in maybe a week or something like that. And I dunno, I can't really say what happened, but there is there, the complexity grew. I think one part of it is the integrations like that it's connected to, yeah. To other tools. GitHub, like Jira GitLab and many more. That's one part of the complexity. And the other part is like the more yeah, also maybe a little bit typical for sources. If people keep requesting stuff and then there gets added more and more. In hindsight, I'm not sure that was always a good decision, I have to say, because it's much harder on many levels to remove stuff than to edit. Like you Absolutely, Jonathan: yes. Johannes: You also, people get angry sometimes, if you remove stuff, yeah, it's there's definitely some tension there. Jonathan: I've found myself using, I'm sure you're familiar with XKCD. I think a lot of our audience is there. There. One of the XKCD comics is entitled Workflow, and it's a bug that was fixed in an imaginary program that holding the space bar was causing, unlimited CPU cycles of just spin and spin. And the guy's I finally fixed the bug and you had multiple users writing in and going, no, that broke my workflow. I've used that, I've used that comic multiple times in the last few days. Johannes: Yeah. Yeah. That's, yeah, there's something to it. I think for the most part, like it's really a great community and like people are really friendly and there are many people that are very understanding also of the limitations of the project. Sometimes I maybe I or there have been occasions like when I broke stuff, and it's usually people yes, stay friendly through it. Some maybe panic a little bit. But yeah it's really surprising in a way to see like that there, people coming together and to, yeah, to work on something which is not yet their own, so to say. And then, yeah, being just friendly and, good. How do you say? Good, good. Good welding or well intentioned about it. It's a really, was really a nice surprise again and again. Jonathan: Yeah. So using super productivity I'm seeing a term here that I'm not super familiar with. Pomodoro it, it works as a Pomodoro timer. Johannes: Yeah. That's what, is that one of the many features like Yeah. Pomodoro is a technique for for focus work or also an anti procrastination technique. It's pretty simple. Like you take like these red Pomodoro timers, this kitchen clock you have sometimes and then you set it to 25 minutes and then you try to focus as much as possible for those 25 minutes. And, yeah. And this helps some people. And this also was I think important feature for many, I don't use it as much myself, but this, yeah. Also I have always been a part of the project, like the two not only be a to do app, but to bring in productivity methods to play with them and to, yeah, to really make it easier to work and more, also more enjoyable. I think that's also a part of it. And yeah, now it's still there, the feature, it's a little bit extended. There's also like a flow time timer and you can yeah. Switch between these different methods. Yeah. It's also maybe explanation like why it grew recently to pay a little respect to it. Like last year, I think it was I also added a plugin system the idea is to have a strong core, but that people, because I think productivity is very much about experimenting with stuff and things, everyone is different and for everyone there different methods that work better and also it changes for everyone individually. Like maybe for some specific task or for some time in your life. I dunno, the pulmonary method works very well, but at another time, maybe you are facing different problems. And so I always think it's good to experiment with the staff and like with the plugin system it's, and also with wipe coding which is now pretty open to everyone everyone can adjust this tool. Yeah, at a very deep level. Which I think yeah, it could be an interesting direction like where the project is heading. So far, I think there are some 20 plugins or something like this. There's stuff like how to put it like yeah, it, it's a gamification of of you to do. Oh, Jonathan: right. Yes. Johannes: It's an experience system and yeah, I think it's really cool. I honestly, I don't use it myself because I don't need at the moment, but I think it's really cool, to have the option to do that. Jonathan: Yeah. Yeah. Interesting. I'm poking at the the web version right now which is pretty fascinating. I found a coffee counter. I may have to get that going. Johannes: Yes. Jonathan: This, so there's versions of super productivity for all of the desktops and also Android and iOS and running on the web. Is that all one code base? Johannes: Yes, it's electron based. Jonathan: Oh, okay. Johannes: Okay, Jonathan: gotcha. Johannes: Electron. And for mobile platforms, it's capacitor js and yeah. That's what makes it possible. It's still quite the hassle sometimes, I have to say absolutely. Especially on, on Linux, unfortunately. It's really, and it's also not the fun part, I have to say, like this very special knowledge involved and knowledge. You don't. Use for anything else. And then it just, someday it breaks again. It's, yeah. What Jonathan: I will, I'll tell you the secret that I have found to making this work is find somebody in your community that actually uses it on Linux and hand the packaging duties over to them. Johannes: Yeah. Yeah. That's probably, that would be a smart idea. Jonathan: That's the way to do it. Yeah. Not everybody can pull that off, but when you can, man, it's great. Johannes: Yeah. Yeah. I have to think about that. It's really a great idea. Yeah. I think actually, yeah, Jonathan: find somebody that actually uses it on that platform. All right. So we support on all these different platforms and this immediately brings to mind, and I know it's in here 'cause I've seen it being able to sync between those different platforms. 'cause it's not very useful to have a to-do list on your phone and have a different one on your desktop and a different one on your Windows desktop specifically if those don't talk to each other. So what's the answer here? Johannes: There's also a long story to it. For some reason many years back I decided I don't want to have a backend because honestly, also because I didn't want to deal with illegal stuff. I didn't want to do, deal with the risks of losing the data of other people. And so I decided to go with the how it's called. There's a specific term for it, I forgot. It's not self-hosted. Jonathan: Yeah, Johannes: I don't remember that one. But there's a specific term for these kind of apps. It's not many who do it like this, but so I decided people have Google Drive. It's not that complicated of it. The data's not so complicated. And why not just, make sync, file based. And yeah, and that's what I did. I think I started with Google Drive and then I added Dropbox and also file based sync. And in, in hindsight also, maybe that was not the best decision because also likewise with integrations and the many many channels where there is available this yeah, it was a third party getting involved and there were problems. And also using services which are not really meant for it cause problems. And also I had to cancel Google Drive at some point because they changed their policies and and also then I added web. D FFR support which opened, no, a whole other bunch of problems. But yeah, it's still, I think it works well for most people. But all these ongoing problems led to the decision to, to work on a dedicated backend service, which you can self host if you want, but that yeah, that's specialized on, on, on syncing the kind of data this application produces. It's called super sync just because of the, yeah. Super in it. Actually I'm not so happy about the name if I'm honest, but it is what it is and it doesn't make sense to change. It probably would've flagged something, some, something clever, more like maybe banana tasks or whatever. But yeah, here we are. It's Google Bill. So that's what it has going for it, I think. Jonathan: Yeah. There you go. So you're hosting super sync for right now. Johannes: Yeah. Jonathan: And I see a note here. This service is free for now, but will likely cost money in the future, Johannes: which Jonathan: having servers is not free. Somebody's gotta pay for it. So Johannes: it's unfortunately true. And also it's yeah, personally I would really like to work on the app full time. I'm doing it now, but I'm doing it from my savings. So this won't work forever, especially now that I have a family to sustain. This was like yeah, an idea, like how it could work. Like it's really tricky subject, I think like monetization for source projects especially especially if you want to be as ethical about it as possible. And but I think like providing this, the service which costs money, like I, as you mentioned, like the service are not for free. The legal risks you enter by having that are not free if you if you want. And so I think it's yeah, it's a fair. Fair compromise to go about it and nobody has to use it. I won't remove like the other thing options that are already there. But yeah I'm hoping that it's that it's, that it will become a source of income which is enough to sustain myself, but at this point, it's not yeah, it's not clear if it will happen. Jonathan: Yeah. Johannes: Yeah. Jonathan: No, that's understandable. All right. Super interesting. And then there's this looks like there's password based encryption. I am, as we talk, I am sitting here poking at this, trying to set up a little account for myself. So Johannes: Yeah it, like hands on it's a good thing. Yeah, exactly. Yeah, that's, it's another thing we see all sync providers, but with a new one it's mandatory. You do like end-to-end encryption, which means like you enter password locally and the data gets encrypted on your machine. And so I can't I, and also nobody who maybe, I dunno, HEXA server or something like that will be able to read the data. Given you don't have a, I don't know, a two letter password or something or someone use a quantum computer. I don't know how this will play out honestly. But yeah, it's, so it's should be pretty secure so you don't have to worry about yeah. Other people doing stuff you don't want with your data, which I think is a good thing. Also, I think yeah. Important point actually for many people. That's that you don't just send all your data to the cloud and that you Yeah. That, what happened with it. It's you don't have to be dogmatic about it, but I think yeah, at least having an idea what's happening with it is probably a good thing. Yeah. Jonathan: Yeah. Absolutely. And as I'm poking around at this, I'm seeing that, you have just your simple task list. You have a time tracker, but you can also set it up as a productivity suite. Johannes: Yes. Jonathan: And so what's the, what all do you get with the productivity suite that's, beyond just time tracking? Johannes: Yeah, there is there is a habit tracker, which is pulled in. You, I have to look at actually at the, yeah. Okay. Now you have you have a notes feature. You have all the integrations you have you have a planner, which is which is yeah, for planning obviously over time. And then you have a schedule which is like a timetable. So super predictivity you wouldn't build in feature time boxing, which you also don't have to use. But it's there. And if you time box your tasks, if you assign a certain estimation to them the schedule will automatically generate an overview, like a timetable how it would play out. That's also another thing. And you get a boards feature, which allows you to yeah, configure all kinds of boards like Kanban Eisenhower Matrix yeah, basically whatever you want. It's pretty flexible and yeah, lots of stuff. And like coming back to it like it's with productivity, I think you, you don't, I often use this thing, oh I don't think I'm using it to its full potential and maybe it gives me a little bit of anxiety or something like that. But it's really about having the options for experiment if, when you need it. So I think for many people, probably, like who never had a to-do list app before, like just doing this is probably. Will work very well. I think, like it will really change how you how you go about your day. But if you need something more specific or for example, you feel like I don't know where my time went that's, you also have options for that. And yeah, that's that's what I personally, or maybe also what explains this obsession a little bit or why I didn't get bored of the app is how I use it evolved a lot through the years and changed a lot. And yeah. And I'm, I am hoping to strike a balance that it's still like very accessible for, I don't know, non-power users or first time users. But there, yeah, there really are a lot of options if you wanna experiment. Jonathan: Yeah. I'm finding more and more, the ability to like connect it to a Google calendar. Yeah. The ability to pull tasks from GitHub and a bunch of other Yeah. Integrations here. But GitHub is the one that really is most interesting to me. I'm trying to, I'm trying to get some of these things set up because it looks really cool. I probably have to use this Johannes: yeah, try it out. I recommend it. Jonathan: My, my current time tracking solution is unfortunately just a website that I think is probably entirely closed source. Maybe not. Maybe they have some open source stuff. I don't know. I won't mention the name of it then, because I don't remember. But this looks like it's more featured than that is. Johannes: What's your reason for for doing tap tracking Jonathan: How you do it? So I have gone full time with the Mesic project as a programmer, but also a business person and just trying to. Trying to understand how much time I'm putting into that. Yeah is fairly important as some things develop and trying to track, which pieces of hardware I'm working on. And so that's what I'm working on. But the ability to it would be really interesting to be able to say, okay, let's pull in the GitHub issues that are assigned to me, and then, okay, I'm working on this one right now, I'm working on that one. And to automatically get that time tracking, like that's a cool idea. I could, yeah, I could get behind that. Johannes: Yeah, Jonathan: it'd be very neat. Like I've spent. I mentioned it before the show, I spent some time today. This is Lily Go T watch, S3 plus which is a cool little piece of hardware. It's, so it's got a lo radio in it and SP 32 S3 GPS IMU, all the normal stuff that you would expect out of a mastic device. And someone sent in a pull request, 'cause one of the buttons is wired up directly to a power control unit inside of there. And so we didn't have any, we couldn't observe the button and somebody sent in a pull request and said, Hey, here's how you observe the buttons that you can turn the screen on and off. And it's oh cool. So I pulled the pull request down and started testing it and the device crashed. It's oh, that's not cool. Come to find out the device crash was even without the pull request and we just hadn't noticed it. So this is why we have Alpha Software. So I've been, most of the morning I've been playing with that, trying to understand why and how we are crashing. I think I'm just about there, making progress on it, but yeah, that's been fun. Oh, anyway, let's see. It looks like things like the Google Calendar sync don't necessarily work from the web app. Johannes: I know. Yeah. With Google, the problem is yeah, it's something I started before my little break and it's actually a Google thing. You, I, it's a legal thing. I need to I dunno. I need to send them a video about the permissions and these kind of thing. And and yeah, hopefully I get to tomorrow and then it probably will take another week or two for them to check it, and then hopefully it'll work for everyone again. Jonathan: Yeah. Johannes: Yeah. Jonathan: Gotcha. Yeah. I'm I could share my screen and show you, I'm getting the message access blocked. This app's request is invalid. Yeah, Johannes: yeah. Yeah. It looks scary. Jonathan: It does. If you're not familiar with what's going on under the hood, alright, so that is super productivity, which is a very cool project in and of itself. And now I wanna pivot for a minute and talk about the new project, and that's parallel code and what is parallel code? Let's start there and then we'll talk about why it exists and what you're using it for. Johannes: Yeah. Probably code is wrapper for agent cl coding clients, like code codex, like Gemini, CI. And yeah, it also came to life because I saw a gap like when working with these tools more and more myself. I think. At some point I started was using them within my IDE and then they got more powerful and you could trust them a little bit more to run on. Yeah. To let them run on their own. And so it developed that I just had multiple terminals side by side to each other. And that Jonathan: describes the Linux experience for me, multiple terminals side by side. Johannes: Yeah. Which is a wonderful thing. There, I don't have anything bad to say about it, but I thought it's repetitive and as I said being a programmer, that's doesn't feel good. Sure. Jonathan: No, I gotcha. I gotcha. Johannes: So I thought yeah let's, it'll add, it's make this easier, like also this using GI workflows for it because it just makes sense to have isolated context when you're working on stuff in parallel and to add information, which I need all the time. Also not like with teamworks, you can also do all the stuff but you have to configure it again every time, or I dunno, maybe maybe you don't. But that's as far as I got with teamworks. And so I thought, yeah, it's I like the terminal, but there's also something to be said about the good user interface. I thought, yeah, it's not too complicated and I, I need it for myself. And so it started and yeah, and then I thought, yeah, maybe this is useful for other people too because I really thought it's pushing my, my, my workflow to another level really because yeah, with, I think with agent decoding now it's still, for some people it's still very controversial. Subject and rightfully but I think for most programmers it's already in reality and it's very exhausting way of coding, especially if you're doing stuff if you're doing multitasking it's not really what the human brain is built for and it's really exhausting. Like after you, you have to have a high concentration. And what connects to this or what is the reason for that is that you yeah, that these context switches are of hard. For example, like in my previous workflow, I was switching I had the terminals running side by side, and then maybe I had another bunch of terminals open side by side. And then I wanted to check the changes. And then I opened my IDE again. Then I wanted to check in the browser and open that. And, while there still is like this you're still doing this multitasking. It's a little bit softer because you don't have as harsh of a change with this parallel code because you have it on one screen and it's doesn't feel as disconnected as Yeah. Switching to another app does. And yeah, and yeah, basically I build around my own use case. There is there is like a changed file view there. You can fire up sub terminals like for example, to run your test suite, or you to just quickly check something with the command line. You can yes. Not it's not too crazy of a feature, but there's a notes panel, which is surprisingly useful because Jonathan: I find myself opening a a dedicated application just to make notes all the time. Like Kwright is usually I have usually multiple instances of Kwright to either, pop a quick note in or, to, I've got, I think I have two of them open right now, or one K right, with two tabs with a couple of crash outputs. I have no little, throwaway notes open all the time yeah. Is parallel code and IDE then, Johannes: oh, I dunno what the correct term is actually, it's, the soft base is so new in a way. I, yeah I dunno I dunno what the correct technical term is. Maybe I would call the wrap like, very pretty generic term, but it's rep for different, c Eyes. That's maybe, and it, Jonathan: so it, lets, I wanna make sure I have my understanding of this is right it's almost just a container for terminals, although there's obviously some more sparks in it than that. But as you first look at it, like looking at the screenshots here, it's just multiple terminal instances side by side. Yeah. But what this is doing is it's letting you load up your different coding agents, so you know, whether you're working with Gemini or Claude, or whichever ones you have access to. And so is the idea here that you would have, you have one task that you're trying to get done, and you want to give that same task to both Claude and Gemini to see how they approach it differently? Is that sort of, that also, do you use what I, I'm, I've only begun dipping my toes into the water of letting the AI right code. I've had good success with it so far. But I am not the I'm not the power user for this yet, so I'm still trying to understand like, what all can you do with it? What, so what does your workflow look like when you have a programming task to get done, or multiples, let's say? How do you use parallel code to make that happen? Johannes: Yeah. A reason for why multitasking is becoming a, has become a reality for me is because you wait for for the agents to finish, they're doing that thing, whatever they're doing. And so for me, like I'm an impatient person and I don't know, maybe I watched many YouTube videos or something like that, but I get distracted really fast. If I have this downtime in between I, oh, maybe I start surfing this and that and it's. Always requires like willpower to go back from that. So it's also another thing which exhausting. And so I started like opening up these terminals on different tasks at the same time. Sometimes you can do at work at, on the same stuff at the same time, but it's tricky because it gets messy. For example, you have two agents writing the same file and and then you can get a new weird, it could possibly Jonathan: go wrong. Johannes: Oh, the file change. Let me change it back. Yes. Oh, the file change, let me change. And, so it's for me personally, you can use it. There's also maybe I can come back to this later. There's also a feature for for letting agents do the same thing. But for me personally, it's mostly about this doing the same thing doing different things in Lia. So for example, there's it's also GitHub integration. So I just drop GitHub link from Backport, I drop it onto the interface and then and then it spins up and it analyzes. And so then, I dunno, this maybe takes five minutes and I can maybe open up five sessions and then I can check okay. It's set in that output and okay. We probably should proceed like this. And, that's how I use it and what's, what it, it doesn't do something completely new, but it does all those things I would usually do, do having another good work tree and branch for that. And all this is yeah, much. There's much less friction and and so you can work step by step at these things without hopefully getting yeah, losing track of what you're doing less because the UI is built around specific specifically that there's yeah, Jonathan: super interesting. And so I, one of the, one of the features that I saw here is that. Parallel code actually does isolation between agents so that you can have multiple agents working on the same code base at the same time without doing exactly what you described. Oh, something changed this file, lemme change it back. Johannes: Yeah. Jonathan: How does that work? What's the approach? Johannes: You have good work trees, so this what I usually do, but you can also, there's also a feature which if you need more isolation and also more security you can also use a docker container to have a completely isolated session that are the two options. You can also work on the main branch if you want. But like the how I use it the most often during the day is just having this good work trees alongside each other. Jonathan: I've never used get work tree, and I immediately feel that I need to look into this. I Google it more. Me neither. Johannes: Before Jonathan: I Google it, one of the first things I see is I was doing this and normally I would just stash it, but that gets really obtuse. Yes. Yeah. Yes. I have multiple STEs and sometimes you lose track of what's which one is which. Johannes: Yeah. Yeah. For good work trees, it's basically get copy copies, like your whole code base into another folder. So like you Yeah. Also like with your third party dependencies and everything. So you really have no conflict there if you're working on that. Nice. Jonathan: I'll have to look into this in and of itself, but back to the ais. So I'm still, I'm curious about the actual editing step. So I've, again I've dipped my toes into working with ais and letting them write code. I still find that I have to go in to some extent and clean it up. We talk about the facts that the AI likes to make lasagna. It makes too many layers. Like it'll do functions with, single line functions just to be able to get a prettier name on it. And it's no, I understand what you're trying to do, but man, that makes it difficult to actually read the code. I talked about needing to go back in after the AI and remove some of the noodles out of the lasagna to make the code better. How does that part work? Johannes: Yeah. That's it's an important thing and to still check the output. Probably people, my assumption would be that people don't do it as thoroughly anymore because of the sheer amount of code you, you have to deal with. It's for example, like before when working with a team and someone wrote a pull request and let's say there may be, there are 10 changes, the 10 lines changed I review. So really I can say something about that, but if if I see like he has written my coworker has written like a thousand lines of code, then maybe I'm maybe I'm looking at this like really quickly or Jonathan: sure, Johannes: yeah. Looking at this differently. But yeah, I think it's still important to do and what code adds. Also not like a brand new crazy concept, but it adapts the changed file tree you have. Which you would have also, like for example, if you do a pull request on GitHub and you can quickly go into this files. I dunno, or maybe a side question does, would it make sense to share my screen to show stuff hands on or, Jonathan: If you can do it inside of inside a video ninja, I can probably capture that. Yes. Johannes: Yeah. I hope this is. Is it big enough? I don't know. Can you see something? It's not too Jonathan: bad. Yeah, I'll get it pulled Johannes: up. I can make it a little bit bigger. Yeah, there. Like for, from the UI level, what parallel code does, there's just like this diff you. You can select stuff here and you can comment in. For example this is stupid. Oh, Jonathan: wow. Johannes: No, this is Jonathan: great. I, so the integration with all this stuff is a lot slicker than I expected it to be. To be able to just click on something and immediately go to the code and be able to write it feels a lot like working inside of the GitHub web actually. Johannes: Yeah. Yeah. That's the inspiration because like they're wisely the the professionals about it. Yeah, that's an adaption basically of what they do. You can also open like your local editor, if you click here you can also for example, you cannot just comment, but you can also ask please explain this to me. I don. No anything. Okay. And then it's syncing and and hopefully it at some point it, it gives you like these Oh. Jonathan: So when you do an ask it, it is an immediate query, but when you do comments Yes. You then gotta hit the button to send to agent. Okay. Johannes: Yes. What it does, it's just execute. Oh yeah, there you go. Oh these freelance, yeah. Jonathan: Small unit test for the scheduled component. Here's what Johannes: they do. Yeah. Who would've thought? Yeah. And yeah that I find it pretty handy feature. But as I said, like it's not something super innovative or something innovative. Yeah. And what I also like maybe can also share this. Sure. So I can. Just drop in GitHub links. Jonathan: Yeah, I did wanna see this. This sounded pretty cool. Johannes: Unfortunately. Maybe less Texas. This is a super productivity park. And then yeah, it prefilled, it, it auto to detect the project. It yeah, pre, so it figures out the branch, the issue. Yeah. Then you can choose which tool you want to use. Then you can decide if you want to use work tree. And you can also run it in a docker container. And yeah, Jonathan: you can skip all confirms if you really Johannes: feel Jonathan: brave. Johannes: Yeah. Then it does, its sing. And also another layer, like for this getting more better feel, or maybe let me start like this. Another thing I currently dislike about coding with AI is the feeling of losing control because you, you are at least, maybe there are people better than me who don't do it as much, but I am I'm not as thorough and as I was before ai. Because as I said, there is so much code you have and you delegate. You, you have to delegate. Also like part parts of the responsibility. But there layers to, to ate this and of course like 1, 1, 1 is testing and so what parly code does you can like create terminal shortcuts for, which could be basically everything. But here for this, what makes sense is like to have like the test command run, which runs the unit tests or the end-to-end tests. And so you have this all clustered in this block, which belongs to this task. So you don't have a terminal here, which belongs to this task and another terminal there, which also belongs to the same task. But you have it like a cluster that UI wise which hopefully makes it or which think does make it easier for the brain to yeah. To down feel as lost when doing these kind of things. Yeah, Jonathan: this is very interesting. What have you found is the most helpful for you? You talked about the difficulty in reviewing all of this code. What have you found that really helps to try to make sure that not only your work, but also pull requests that are vibe coded? How do you best make sure that the quality stays high? Johannes: Yeah. Yeah. There are multiple layers. I say the stuff or it's good to have layers which don't require your own mental capacity. I think like you can you have unit tests, you have end-to-end tests. You also have from the AI workflows there, I, for example, I have different skills for cloud code I'm using like, which. Then in turn fire up multiple agents and different models like my, my multiview task I, I'm using for most stuff is like, fire up five regular agents for different aspects. And then it it fires up also Codex and copilot. And so you have hopefully they're producing something useful. But usually I think it it helps to identify problems before, before I have to step in. So the idea is you try to reduce like the amount of stuff you need to think about yourself as much as possible. And so this means I hopefully when I start reviewing I has done the best it. So I, I just have to deal with the stuff which is left over after these obvious yeah. These layers beforehand. Jonathan: Do you ever do things like let one AI review another? This is something that we do quite a bit actually in the mesh astic project. 'cause Johannes: Yeah, Jonathan: all of that is run through GitHub and in GitHub there's the copilot reviews you can ask copilot for review. Johannes: Yeah. Jonathan: And we found that to be useful actually. Yeah. Like it, Johannes: yeah, Jonathan: it's it sometimes will be a little too nitpicky, but it usually finds really good stuff. I wrote a I wrote a pull request just the other day. I was doing notifications. We wanted, in the desktop app, we wanted notifications and I was feeling lazy that day. And so instead of using Lib notify, I just used the binary that comes with lib notify and just built the string and pushed it all out as a command. And, then did the poll request. I'm like, yeah, it works. And then we asked co-pilot to review it and co-pilot's you probably shouldn't use untrusted user strings on the command line like this. Yeah. And I went, oh, of course I can't do it that way. Johannes: Yeah, no, definitely. But I have to say, it's something there, there already people like on, on GitHub for the project who think who are thinking further about this and how to integrate it within the application. But for me at this point, this is something I try to figure out on the model level which means like for me most of the time using or creating cloud code skills for that, maybe I can quickly show like how this looks multi review. That's it. Yeah. Hope this is readable somehow. Yeah. Basically what I have is a file which defines like which other agents to, to start for this kind of review. So there's one specifically checking for correctness. Like a change really does what it's supposed to do. Then there's one for security, then there's one for architecture and design. Then there's one for exploring alternative approaches, and then the current one. Then there's one for performance. Then there's one for simplicity, and then there's one codex reviewer, like using another model, which just like a general review of everything and yeah. That's basically like how I, at the current time find it best to work because also yeah Phil philosophy of of the of the app is like not to not to be its own ecosystem. Like open how it's called open code is like where you have another layer, like its own model of how do have to structure all this AI stuff, but just to be a wrapper about the workflows you already have. For example, if this means if maybe in a two months time or something like I dunno, jet Brains maybe brings out a totally crazy new IDE just for agent decoding. Then you don't lose anything by working with power Layer code now, and this is also important I think for me at least, because the space is moving so crazy fast or has been moving crazy fast for the last year that it for me doesn't make much sense. Yeah. To redevelop the wheel and yeah. Jonathan: Yeah, absolutely. Let's see. I was gonna go a certain direction and of course I something dinged on my computer and I've completely squirreled and forgotten the the direction that I was going to go. Let's see. Oh, pushback from the community. I am super curious. Yeah. Like in, so you've got the existing project. Have you gotten pushback from the, have people said, ah, none of this AI on get off my lawn with your AI tooling? Johannes: Yeah. Jonathan: Yeah, because you've seen projects out there that are I've seen one project that have completely closed poll requests to anyone outside the organization. Yeah. I've seen other projects that are talking about going closed source because of the AI thing. Just some, some of it I think is really a step too far in, in some of those cases. But what's your experience been with that? Johannes: I, it's really hard to say I, from a, from moral standpoint, I. Don't have a, I don't have a clear opinion about it. Like there, but from how I personally work, like I was skeptical also in the beginning, like when the whole AI wave started. But it is currently, like for me, there's no doubt that it's more efficient at the moment. It's the most efficient tool to use while coding is using ai. And to come back to your question, like I, I don't have any, i've emotional connection to my code, of course, but I don't feel like I, I don't also, the license super productivity has, for example, is MIT I'm really okay with people doing what they want with it because yeah if somebody takes it and does something better with it I'm fine with it. I know that's probably not doing the subject justice because there are other aspects to it, but yeah. I Jonathan: Have Johannes: you resident, yeah. Have Jonathan: you, have you Johannes: resident think it makes much of a difference in a total, like if the models or new AI overloads have access to the code of super productivity or not, it doesn't make, probably want Sure. Make a big difference. Jonathan: Have you seen an uptick in low quality pull requests because of ai? Yeah. Johannes: Yeah. Jonathan: Is it pretty bad or just a little bit? Johannes: Just a little bit. Okay. I think, yeah, I think on the other hand, the stuff is better documented. It's more about also there's sometimes it's too much information because AI to produce so much text, like if you have a poor request, like with a long technical specification. And to be honest, I don't always read it very carefully. So Jonathan: yeah. Johannes: But yeah, on the other hand, like it's two to two sides to, on the other hand, like it enables people like, who are not programmers or who would never have the time to submit code to do it. Like to say, oh, there's a problem I'm seeing. And the very least it does for me is to. Yeah. To to get attention like that, oh, there is this problem. And even with the code, like it's horrible. I know about this. And and especially the part, like getting more people involved I think is a, it's generally a great thing even if it comes with some challenges. Jonathan: Yeah. The, some of the most interesting and terrible, but also interesting AI poll requests is where people ask the AI to do something that we didn't think was possible or we thought would be extremely difficult. And sometimes you've got people, I'm thinking of, I'm thinking of one poll request in particular where, Johannes: yeah, Jonathan: We had told people for the longest time, there's really not a way to do this. It would be such a huge lift, and somebody apparently had the tokens to burn on it and has what may or may not be working probably. Honestly, probably working at this point a poll request to make the thing happen. And in this case, it's running mesh tastic as a part of Zephyr, which, if you're in the embedded space, you know what I just said. And if you're not, then you know, that doesn't make any sense to you. But it was it's something a lot of people have asked for and it was gonna be a hairy a hairy thing. And it's like someone ignored the fact that we said that this would be really hard and just went out and said, I'm gonna do it. And made the AI do tortured the AI into doing it for them. And now you look at it, it's okay, first off, I had told someone offhandedly a couple of months ago. I'm like, if you actually wanna do this, here's the way to do it. And that's the way the AI approached it as well. And so that was interesting to me. But just like the fact that it I, it lets people challenge these, pronouncements is interesting. Sometimes it's a good thing, maybe sometimes not such a good thing because there, in some cases there were reasons the project didn't wanna do that. I don't know, it's just, it, it does, it's really, it's changed the game. It has, it's changed. Open source AI has changed open source in ways that I don't think we fully understand yet, but there's no putting that genie back in the bottle. Johannes: Yeah. Yeah, definitely. I totally agree that there, it changed a lot. Like for example, for before a year ago I really put a lot of effort into merging every pull request because I wanted. To get people involved. And like there, there haven't been so many contributions by other people. And these days I find myself more and more having to say no to stuff to say oh yeah, it's well done. It's totally, it's, yeah, it's great code. You did really well. And, but it's it's it's too much. It's it's a feature. It's too many features. Jonathan: You did a great job prompting your AI to write this code. Good Johannes: work. Yeah. But yeah, it's like, for me personally I still feel ownership for these things I'm doing. Of course it's losing something I dunno, back then when you wrote code by yourself it's it's satisfying. Like it's, there's a lot of beauty and well written code and a lot of satis, or at least I got a lot of satisfaction out of just key beautiful code, like absolutely clear concept behind it. And you can tap yourself on the back. Well done. Jonathan: Sure. Johannes: And this is different now. This is yeah, you, it's more, it's on a, more, on an architectural level maybe. And more on a, it's for super productivity, it's more on a product level that, that the decisions or the stuff I do this more about the decisions I you feel ownership to, and I think rightfully than for the Yeah. Intricate details of the code itself because you might not always operate on the level anymore. Jonathan: Yeah. Do you think there's a danger that so one of, one of the. One of the problems people suggested is that you have to treat, and I don't know if this is true anymore, things have developed so, so rapidly, but people said you have to treat the AI as though it's a junior program. Junior programmer may be an intern, and you have to be the senior programmer. And then there's this question that comes about, then how do we get new senior programmers? Yeah. This is a reasonable concern, but I think the question is, will we run out of senior programmers first or will AI get good to the point to where it can then also be the senior programmer? Johannes: Yeah. Jonathan: I'm not sure which of those is worse. Johannes: I also don't know, like for the time being I've settled for I just don't know because it's, yeah, it's all involving so fast and I think there are points to be made for, yeah, for things just change, but there's still need for programmers. I don't know. Yeah, I think a couple of days ago I read like that there may be I'm probably quoting the number wrong, but that there may be one, 1% of the global population or 1.3% of the global workforce for our programmers. And actually to me, if I think about that, I think that sounds like quite a lot actually. Do we need this much software? For so maybe there will be some sort of adjustment. There maybe will be a little bit less programmers. But at the moment, like from my own experience and how the past year working with AI has been, I still think like the technical decisions behind that's yeah, I may, people smarter than me said like that, that AI is not yet a good architect. And I hope it stays this way. Jonathan: Job security. Johannes: Yeah. Jonathan: Alright, so the big question I've gotta ask is, do you use, do dog food, this, do you use parallel code to work on parallel code and super productivity? And while you do do you use super productivity to track your time doing both of those things? Johannes: I do. I do. I do. That's the beauty of it. Jonathan: Yes. It's, Johannes: I've worked very many years as a programmer and yeah, to be honest, I. Apart from the code itself? For most, to most products I was involved with, I didn't feel a strong connection. I didn't, to be honest, I didn't for much of the stuff, I don't think it's, has been necessary to do at all. It's really, for me, it's really nice to work on, yeah. On, on, on my tools on things I'm using daily. And that's it's, I think it's also an effort for the project. I am, I'm thinking much differently about super productivity then I would for about some yeah. Freelance gig. Project. Jonathan: Sure. Are you still freelancing? Johannes: Yes. Jonathan: Yes. That makes sense. Yeah. Alright, we have we have hit the bottom of the hour. I've got two questions, two final questions I've gotta ask, I gotta ask this to everybody. What is your favorite text editor and scripting language? Johannes: Yeah, as my, it's, I don't know if it's the right answer because it's an IDE, but I have used JetBrains. Yeah, Intelli J Idea for many years and I really liked it. But to be honest, I haven't used it as much in the past months. Jonathan: I parallel code is almost an IDE. It blurs the line into being Johannes: it's my favorite and it's the best one. Jonathan: There you go. And then the scripting language. Johannes: Scripting language. Yeah, I think I still prefer type script just because I'm most comfortable with the language. There's some stuff I'm doing with Spanish and some stuff I'm doing with Python, and Python is very elegant. It's a beautiful language. But I'm I like to work with tools I'm comfortable with. So it's Cript. Jonathan: Yeah. Yeah. Very cool. Alright. If people wanna find out more about these two projects, where is the place to go? Johannes: Yeah. Probably the easiest way to find both is to go on, on, on my GitHub profile. There are both projects linked. Jonathan: Okay. Johannes: And yeah. And you find all the other stuff from there. Jonathan: Okay. We'll make sure and get some of those links in the show notes as well. Cool. Thank you Johannes, for being here. It's been a, it's been a delight to get to chat with you. Johannes: Yeah, likewise. Thank you. Thank you very much for having me as it has been an honor. Jonathan: Yeah, it's been a lot of fun. And I've got a couple of projects to check out here. One that I'm in the middle of checking out, and the other one I'm gonna have, I might talk myself into, doing some more AI work. I'm still tipping, dipping my toes into it, but it's fun. Alright that has been Johannes Milan talking about parallel code and super productivity, which I already have a super productivity account spun up here and got it installed. I'm gonna play with that some next week. If everything goes according to plan, we will not be live. We will instead be recording a show on location at a a particular place down in Texas. Hopefully we can make that work. And then it'll come to you, not those that watch live, but those that that get the download at the normal time, we hope. But it's been great to be back after missing a couple of weeks and yeah, we'll be back next week. All things all things go well. After that, the schedule is still looking a little bare. So if you wanna be on the show or know somebody that needs to be hit us up. Let us know. You can email us@flossathackaday.com or you can get ahold of me at the various places where I hang out. Appreciate everybody that's here. Those will get us live and on the download, and we will see you next week on Floss Weekly.
• Episode 868 - Remove the Noodles 22.04.2026 1val 6min

  This week Jonathan chats with Johannes Millan about Super Productivity and Parallel Code! Those are two very different projects, but both aiming for helping us get our work done. Super Productivity is a scheduling and time tracking suite, while Parallel Code is an almost-IDE for managing and isolating AI coding agents. This episode has something for everybody, so check it out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• Episode 866 transcript 25.03.2026

  FLOSS-866 Jonathan: This is Floss Weekly, episode 866, recorded Tuesday, March the 17th Breezy Box. Hey folks, it's time for Floss Weekly. That's the show about free Libre and open source software. I am your host, Jonathan Bennett, and today we have a real treat. You may remember I was off at Embedded World last week, did repping mesh tastic, but also learning about all kinds of fun hardware and software. And I had several people come up to our booth and say, I did this very cool thing. Don't you wanna look at it? My answer, of course, was always. Yes, I want to look at your very cool thing. I learned. I met some very interesting people on that on that trip. And one of the people that I met was Valentine, Daniel Chuk, who is, I told him he's crazy, but he's the right kind of crazy. He's our kind of crazy. So he is written an open source project called Breezy Box. That is essentially a terminal emulator that runs directly on the ESP 32. And he told me that he got bogged down into a side project, which was a C compiler that runs on the ESP 32. And that's when I called him crazy and fell in love with what he was doing. And so I got back to the states. Slept off a little bit of jet lag, and then we sent an email out to Valentine and we said, Hey, we need to have you on Floss Weekly. And he very graciously said, sure I'll do it. And so we've got him we've got him here. Welcome to the show, sir. Valentyn: Hey Jonathan. Thank you for the warm welcome from Berlin, Germany, and Jonathan: from Berlin. Yes. So embedded world was just a hop, skip and a jump away from you. You just, you jumped on the inner city express and got to enjoy high speed train travel. Valentyn: I actually enjoyed the slow state train travel on the occasion to use my deland car, which is my free train ticket. Jonathan: So a regional express rather than an inner city express? Valentyn: Yes. Jonathan: I'm learning. I spent some time in a Germany, almost a week in Germany, and I learned how the train and bus and subway system worked. We used that a lot to get around from to and from embedded world while it was there, so it was a lot of fun. I very much enjoyed it. Now, you're not a Berlin native though, are you? Valentyn: Yes. I come from Ukraine originally, but I've been living for 15 years here in Berlin, so by now I assimilated. Jonathan: Yeah, I understand. And so you've, one of the other things you told me, and we'll, we can get to this in a little bit more detail later in the show, but you've been working on a sort of an embedded development contract that's about to run out, that has run out? Valentyn: No, not really. I'm working mainly on JVM backend stuff. Oh, that's right. Like middleware. It's my main job and, embedded for me is a hobby project at the moment. The part about my contract running out is true. I wouldn't mind to enter this field more professionally, but either way it goes. I'm happy enough with the things I am doing right now as a free open source software. Jonathan: Alright, so let's lay out the story here. We're gonna talk about what exactly you did. You showed me a device that I think was the the inspiration for all of this. You built a device with your kids. Valentyn: Yes, that's true. Actually, the way it started I just thought it would be a cool project to try and build a game console with kids. And we built this thingy. Which we called Gamer Throne 3000 Uhhuh, which from usability perspective is almost like an old game Boy or something like that. And it's also SP 32. Yeah. Which that display, which is like cheap yellow display except it's red and very basic joystick pad. Jonathan: Okay. So you have this device that you put together, and then you said, I'm sure. I need some software for it. And is that kind of where all of the, is that where Breezy Box started, Valentyn: right? Yes. I was new to the topic and I wrote one Simple game and then I wanted to put another one there. And that's when the question enters your mind, how do I combine them together? And it was. Surprised to find out. Usually what people do is a custom firmware for that, like a custom menu that everybody makes. Okay. I also made that, but then I thought, okay, this is all a little. Cumbersome and I think it might be better. And especially for development, when I want to swap things in and out and try them and fix them and so on. Refreshing firmware is lengthy and not so much fun. So it's okay. Maybe it is an easier way and eventually I just slow. I just, I found a way to connect the wifi and to put files on there. And then I wanted some Shell and I could not easily find a readily available shell of E-S-P-I-D-F, which was my like ID and framework of choice. And I ended up writing my own. Jonathan: And that's what Breezy Box is, right? It's a, it's a play on busy box. Valentyn: Yes. But also I. It started some fun for kids, but then I thought, okay, I can also have some fun for myself because this whole ecosystem, it really brings back the memories like from the nineties when people were coding in C and assembly. And sometimes write in their own drivers. And thinking about how many kilobytes of memory they use, that sort of thing. And when you could actually know every register UCPU has and, think how sometimes know when it makes sense to do something with them more directly or explicitly. And I thought, okay, maybe I can make a retro mini PC for myself. And I thought, what would I do on that? And I thought. Yeah, maybe some retro gaming, but maybe also some retro coding like Turbo Seaside style. That's where that compiler came from. Jonathan: Okay, so now what does this run on? We said the SB 32, but there's a bunch of chips now that fall into that ESP 32 Valentyn: line. Yes. This. This one and the one you saw the expo was SB 32 S3, which has benefit of having both wifi and some PS ramm available like eight, eight megabytes of PS Ramm, which. It is slower than the fast ram on the cheap, but it still makes things easier, like what you can fit in there, even if you are talking about bigger screen and the screen buffer even. That gets tricky when you have a big screen. Jonathan: Yes. Eight, eight megabytes of PS RAM may not sound like a lot, but when you're talking about the embedded world in particular the earlier SB 32 chips, that's a lot of memory. That's a lot of ps. Ram you may have you, most chips, most embedded chips have less memory than that for their core memory. And so it's, it represents quite the quite the upgrade in, in addressable memory space. Okay. You got a shell working. Can you actually load apps within the shell? You can run applications. Valentyn: Yes. It was interesting to find out that actually as received now supports to a degree of binaries. So it's not exactly the same variety as Linux. But it's like object files you usually get from compile or something, relocatable code. And you can produce it with the fuel chain. But I also launched to produce it with my own compiler, which was also a fun journey I like to dig into such Jonathan: okay, Valentyn: Geeky things. Jonathan: Yeah. Oh yeah, of course. So you've mentioned this a couple of times and I've gotta ask about it. You wrote your own compiler, but this compiler is a little special. Where did, where does the compiler run? Valentyn: It actually runs on this very cheap and it can also compile itself, which was just a nice milestone I set for myself because. I was not going to implement the full C standard that would take forever. And when you start implementing one simple feature, then the other, you need to think where to stop. And I thought, okay, this as they call self hosting, when the compiler can compile itself is a nice milestone where you can say, okay, it is. Not really completely a useless toy, now you can actually build something that does something meaningful. So that's what I aimed for. Jonathan: How, Valentyn: and with that in mind, I just aimed to keep it as small as possible. So I ended up with 700 lines of code, Jonathan: 700 lines of code. How long did it take to build like the whole project, but also the compiler? How long have you been working on it all? Valentyn: The compiler was actually the longest part because it's a sidetrack, but it's all good because it's not like I had a special commercial goal and budget and the timeline. So the compiler took maybe about three weeks to write for something that kind of started to work, but then it also took twice that time to debug. To actually run on the device and produce something that runs, again, Jonathan: I, I find myself was challenging. That's actually pretty incredible. I find myself often saying that I can do a nearly infinite amount of work so long as it's not the work I'm supposed to be doing at a given time. And it sounds like this is very much one of those kind of projects. Yes. Okay. So what all can you do with this then? The breezy box and the C compiler? Like what's the limit? What kind of programs have you written for it? Valentyn: The limit is I. No, people try to actually implement more feature systems also for limited chips close to this, like something projects compatible with proper threads and stuff. I am just making what is practical within E-S-P-I-D-F without writing the full operating system. So the real operating system in my projects is pre RTOs is usually pIDF. And I just made the shelf for my convenience. And then I thought it would be cool to have several virtual like screens in it. So I made a virtual terminal subsystem, and I wanted to have a bigger screen. So I implemented something for that, like for the specific display module. I had some small driver. Some scaled graphics driver because I wanted to Port Celeste one, two weeks and some step by step just having fun and thinking what I can publish along the way. Jonathan: Yeah, that's fun. So when we talked, one of the things that I mentioned is that SSH would be cool, and I think in your email you said something about you've made progress with that. Valentyn: Yes. The only one, it's number one most often required feature. So I felt I was obliged to prioritize it and it actually works. There are actually many ways to do it. I cannot take credit. There are like five ways you can find on GitHub and Components repository. The thing is, each of them has their own frame of mind, their own way of doing things. It's not always straightforward to see how to integrate it in your project. So I did what was necessary to integrate it with box and it's not published yet, but it's fine and so on. The current state is the built in commands, work well and display in the SSH terminal. But some of the wineries I made for it, like BI for example, which would be nice to run that it currently is hard coded to run on the GFT screen, which is useless if you want into the device to do something there. So I will fix some malicious like that and then I will publish it. Jonathan: Okay, so is this the ability to SSH from a full-sized computer into Breezy Box? Can we go from Breezy Box into Valentyn: full-sized Jonathan: computer? Which direction? Valentyn: It's the direction to log into the device because I think that the main use case actually 'cause. I can imagine this actually being useful for people who are trying to debug something quickly. Or trying to diagnose some remote device and stuff like that. Jonathan: For sure. How much of the CPU and Ram does your software stack take up now? Like how Full how how crammed full is the ESB 32. Valentyn: It depends on features you enable across so that I do try to keep some of the iram free so I can be more flexible about loading some L binaries. And I think I usually have still about 150 kilobytes left there for PS rm. Usually most of it is actually free because even for the screen, I use the scaled mode and I don't use that much most of the time. The highest I went was when I made something to display photos and then I needed for resolution. And then that's several megabytes in PS R. Yeah. Understood. I And the binaries, most of the l binaries are about a few kilobytes. The compiler is 16 kilobytes when compiled with GCC as a. Game. Celeste Port is 42 Kilobytes. Jonathan: Yes. You showed me a, a Port of Celeste. That was actually very cool. That looked like a lot of fun. I've not run this yet. I need to get a device that you support and run. I'm, or I may have one. I may already have one. I don't know. But I definitely need to get this and do the install. I'm also looking on your GitHub and it looks like you've got a breezy term. Pulled out as a standalone library. Boy, that's interesting to me. Yes. What all can someone do with like breezy term and the other libraries that you've built around this? Valentyn: It's, again, it's a part of community feedback that I said, someone asked me, can I please have just the virtual terminal without pulling in the Bluetooth and wifi and everything else? I said, yes. Okay. Why not? As a site benefit, it also works on a bunch of other devices, including the C3, C five, like risk five devices. Nothing ties it to, three. What it provides is that the virtual buffer for the terminal and it's, it simplifies some things you want to do with text if you're doing anything like this. Jonathan: I I definitely have plans to see if I can use this in various places. It. It's such a, it's such a cool idea. It's so cool to be able to have a terminal on a little tiny handheld device that is such, so much less memory and everything than our big computers do. I, something about it just tickles all the right places. I like it a lot. And so you went with MIT license on everything? Valentyn: Yes. I just. I see no reason to restrict anyone doing anything with it. I just went with it as the most permissive common license that just keeps me out of trouble. It's just my minimal legal disclaimer. Jonathan: And so when you ran the demo for me, you had a wireless keyboard plugged into a device that you were using to type. Valentyn: Yeah, I used the Bluetooth keyboard. I also, by the way, extracted that as a separate component in a perceived repository because somehow there are standard components for the Bluetooth and for human interface devices. Somehow they kept failing in mysterious way with my keyboard. And I had to do some tweaking to make it work. And they thought, okay, maybe someone has the same issues, maybe someone can reuse it. And I also did not want to just have anonymous a blob of Bluetooth handling code in repository, which is about something completely different. But it also makes sense to extract it from that perspective. Jonathan: You've you've done this through everything, through the expressive IDF. So it how difficult do you think it would be for someone to use some of these things in say, platform IO or even Zephyr? Valentyn: Hard to say for me, when I come across some examples that use platform IO or Arduino, something else, sometimes I can borrow some ideas from there. Usually I just keep looking until I find something closer to the idea. So those are like separate realms. And that kind of sets also the scope in my head for like how I position this thing as well. I think it's, mostly for people who are already dealing mostly with E-S-P-A-D-F and just want some sort of terminal in there. Sure. Because in other environments, there are other solutions and they, some of them are probably also better than what I did. Jonathan: Yeah. Yeah. Interesting. Do you see a, obviously there's a future for this, just as a hobbyist level. It's fun if nothing else. Do you see a, do you see a future where maybe this gets commercialized or people want to use it as part of bigger projects? Valentyn: I. It's hard to predict. Anything can catch on if there are use cases and interested people and anything can get commercialized. I do not have a plan for that. Any master plan. But of course it's present to find so such positive reactions in people. It was actually much better than I expected. Jonathan: We, I think we put you we featured it on Hackaday, didn't we? This is exactly the sort of stuff that we'd love to to do on Hackaday. So I'm not surprised it landed there before, before even I heard of it. I don't get to read all of the Hackaday articles, although I do what I can try to get through as many of 'em as possible. Let's see. When did we cover this? Oh, you made the Hacka Day podcast too? February 6th. Oh, that's been a, that's been a bit back over a month ago. I'm assuming a lot has changed since then, since February 6th. Valentyn: A few things. Yes. Jonathan: Very cool. And then what about like other chips? So one of the ones that I've seen that's very popular right now is the ESP 32 P four. A very powerful chip. In fact, I've seen that's very Valentyn: impress. Yes. Jonathan: I've seen at least one one project that put a full on Linux kernel on the P four. Very Valentyn: interesting. Yes, people do that. Yes. That's why I'm both excited for its capabilities. And I am also questioning myself. Should I go there with this project? Because that one chip, I can actually run Linux more easily and maybe for someone who wants to show, maybe that's the better way, I'm not sure yet. Undecided. Jonathan: So one of the I don't know if I have one. Within Reach we did the the Hackaday folks did the Hackaday Communicator badge. Valentyn: I would definitely try to put it on C five list. Jonathan: I don't have one. I don't have one within reach. That's unfortunate. Anyway, there's a developer in the community that has been working on putting Linux on it and has gotten that working, but from what I understand, it's not a great experience. It doesn't run terribly fast. Whereas your project is very snappy. There is some overhead. There is some overhead that's putting that very graciously. You don't have a whole lot of overhead in yours. Valentyn: No I just keep it pragmatic. I am not right in an operating system. I am not even trying to do it prop properly. Multitasking. I did add that feature to have multiple virtual terminals, but right now they only work in a reasonable way when everything that trans in them is just waiting on input when switched out. For me, that's good enough. That gives me an opportunity, for example, to have a code in one terminal and some manual in another or something like that. Jonathan: Yeah. So what has the what has the response been? Obviously you showed it to me in an embedded world. I loved it. Folks at Hacket, I loved it. But have you gotten like poll requests from outside sources? Have people come along and said, I wanna port. Pearl to this Valentyn: pearl? Yes. Okay. You can absolutely put law on this with zero effort. There are existing components, especially for that. But yeah, that was not the core of your question. So yeah, the response was nothing short of amazing. I viewed this as a joy for myself with no commercial application, but people actually show a lot of interest. They, do not yet submit pool requests, but some people did submit the repositories like forks for their own boards. Some people started folks which may or may not be working yet with trying to port it to more devices. I think some devices are pretty obvious goals like computer or anything that has a screen and a keyboard. I would be curious to see that. And surprisingly, even the hardware manufacturers were pretty impressed. And I might get some devices to test and to port it as a result of that too, really to promis anything. But Jonathan: yeah, there was some interesting manufacturers there at Embedded World that when Valentine came and showed me this we said, oh, you should go show this to, i'm trying to, I'm trying to remember who all exactly we told you to go to. I know I know Ella Crow was one of them. Probably Ella Crow and Seed. They were both there. M five Stack was there. Rack Wireless was there. There's a quite a group of of vendors that I've worked with that we said, these guys might be interested in this as well. They can at least get you some samples to play with. So yeah, hopefully that works out and comes through. I know I am going to be I'm gonna be working a bit to see if I can bring at least some of the libraries over to platform io. I think that would be very interesting. Of course that's my development work is all around tastic and Platform IO is what we use, so that I'm biased towards that. People will come every once in a while. You guys should port to Zephyr okay, yeah, thanks. Get in line. Not terribly interested in that for various reasons, but, so what what do you see? What's on the horizon? What's coming next? What other, what wishlist things do you have? Valentyn: I do want to get it running on most of the mainstream as perceived jeeps, like c, C five, first of all. Possibly some variant with some data modified features on C3 and C six. And with relation to that, I might also continue, maybe work on the compiler bit just for feature parity because to me it would feel incomplete if on S3 I can show it with the c compiler working on device. And this five I cannot, so I have something in the works in the direction as well. Jonathan: Yes I imagine that the the bootstrapping is actually the difficult part there. Once you get the compiler actually compiled for the first time it'll basically just run itself, or I guess the output machine code is gonna be quite different between the two. And so that's gonna be a complete rewrite now that I think about it. If you're talking Valentyn: is gonna be different, and that's the back of compiler that needs to be completely different and. Jonathan: Yeah. Valentyn: Yeah. We'll see how it goes. Jonathan: We will, I will chat with you after the show about at least one other bit of hardware that I think would be really neat to run this on. I might be able to hook you up with the right people to get you a sample of it. So we'll see what we can do. Okay. So what I am a Linux geek, obviously. What about the one liner? Does your does Breezy Box have like the ability to pipe output from one command into another? So can you do the one liner with 15 Valentyn: different commands? Yes. It is a limited capability. I do not have a proper Linux pipes that requires proper thread, but I have some like. Basic redirect stuff into a file or even to the next command by a memory buffer. Something like that. So it will work only for small output because it does not run processes in parallel like Linux does or it on Linux, it would go line by line. No buffer by buffer. We would collect the full output and then run the next thing. But yeah, some basic piping is there in the show. Jonathan: Very cool. Excuse me. Goodness. Let's see. So of your, of your core U utils, how many of those things are actually supported now? Obviously like the LS is supported, I know CAT is supportive. Some of, there, there's some of the other, more Valentyn: esoteric. Yeah. It's about, a dozen commands at the moment. Mostly the basic stuff, as you said, a less cat and so on. I sort made a sort of based and tried to prioritize it, but at the moment I focus on the more basic things which are required to get this whole shell. To run on more devices and everything else is as needed. Also, it's something like small, easy onboarding tasks in case someone wants to add a merge request. Join the development of GitHub. It's good to have some easy starting tasks. Jonathan: Oh, absolutely. You I was gonna say that you need like your own name for in core Utils I don't know, nano u Utils, micro u utils, some there's probably some fun play on words that would describe like your U Utils package being extremely tiny. But I suppose that's what Breezy Box is. A, a play on busy box. So you're already, you've already got something that works, Valentyn: right? Yes. Actually Busy box I referred to it with this thing name, just to get the right associations in people's heads, what this thing is. Because it's easy to mistake it when you watch the video demo. Oh, did he write an operating system or did he write what now? Is it just a terminal? Is it, so it's this thing in between, which is what are those utilities in busy box as well. It's sometimes people call it user land. This, there's just no set terms. So I went by association, so I'm not re-implementing all the commands in Dbox. I just have this missing thing in between. Yep. And then you have a real file system. The file system. Yes. It's surprisingly for such a commonly used thing, it's surprisingly missing a few key elements somehow in E-S-P-A-D-F. Like they have the file systems separate file systems that you have on specific Mount Point, and they support, of course, directories and stuff, but they do not track the current working directory globally. So that's what one of the things that three box also add. And if you want, for example, to write a compiler, so you are interested in handling files and finding them and writing output somewhere. And you don't want it to be specific to a specific build on a specific device. You want people to just specify a pass like they used to. So with Pretty Box you can do that. Jonathan: Yeah, absolutely. Is it based on little Fs? I know that's one of the Valentyn: files little, I am using currently little F Affairs. There is nothing specific to it as far as I remember in my code. So you can use little f or FE file system or whatever is supported else in on the device. Jonathan: Okay. Very cool. Yeah, very cool. Let's see. DI, I mentioned ingest pearl, but you responded yeah. That's on the roadmap. Is that something you're honestly thinking about doing is some sort of Pearl implementation? Valentyn: It's not really on my road. My ledge. Okay. Experimented with different things and the tell from experience that LOA works well, but I'm not including it in my demo at the moment. Just a matter of peak features and how complex you want your project to be and how much memory you want to use, and for what. I can also tell from experience that a python is a bit harder to get running out of the full micro python. That wants to be a full firmware, but probably also possible for someone when someone is. More set on it and has more patience for it. Jonathan: Yeah, that makes sense. That makes sense. And as, as we like to say, pull requests are welcome. Somebody really wants to get in there. Valentyn: It, it doesn't have to run e everything and have everything even in one distribution. It's this whole approach was running of binaries that allows you to actually have the core and then just install the bits that you actually need, which makes sense with the limited resources on this chain. Jonathan: So if we have somebody that's listening and says, I want to get involved and do something, where would be a good place for somebody to start? Valentyn: The easiest way is to go to the demo repository. There is Breezy Box repository, which is for the actual module that is published on, and there is the Brey demo project on GitHub, which is the demo. All pieces together has a nice video of two videos by now, and has some instructions on how to proceed. A simple project that puts all together and makes it work. Ideally, of course it runs on one specific device is the easiest way, but basically you can run it on anything with ESP S3 with some success. You will have more fun if you have some sort of LCD screen as well. And even if you also have a Bluetooth keyboard. Jonathan: What's the what's the sort of the premier device? If somebody said, I wanna play with this, I'm gonna go buy something what what's the device to go buy to play with it? Valentyn: The easiest one to just on the repository and run is wave Share seven inch device, which is 1000 by 600 pixels. It's just the display module. I think it's about. 13 euros or $30 or something like that, maybe a bit more. Jonathan: Yeah, it looks 31 to 36, depending upon whether you get touch or not. And it's got the SB 32 S3 built into it, which is cool. It almost looks like a raspberry pie display, but of course that's not a raspberry pie on the backside of it. It's a raspberry pie display clone, I think. That's fine. Yeah, that's cool. And then you to flash it, what you just you clone the repository and you load it up in expressive IDF Valentyn: Right. Jonathan: And off to the races. Because of the way this works, if somebody wanted to write one of these elf binaries I guess it's not compiled in as a single program. So someone could write an elf binary using whatever license they wanted to and and load it, Valentyn: right? Yes, of course. Any license. And those are actually much easier to write. You write them basically as usual, console, absentee, you can also use all the platform specific bits, so everything in the framework can be exposed as a function to call. And you can write some standard C and you can use some specific bits and you can compile it on desktop. Jonathan: I, I don't think I've ever, I don't think I've ever messed with that. I'm now very intrigued by this. That's really something to be running Elf binaries on an embedded device. That's, yeah. I wonder if I could do that in platform io. Valentyn: It's like an open secret somehow. Apparently it's been available as a feature to some degree for two years, and it's not really widely adopted as far as I can see. Jonathan: Yeah, but I Valentyn: mean, it really, there are still works and there is this inconvenience with the file system and maybe that's where reason box some help. Jonathan: Yeah, those are things that can be worked around. But there's this issue that you have with some of these things. So if you have a GA GPL firmware. On an embedded device trying to add a, let's just say a proprietary plugin. So somebody wants to use this in a business case and they want to add some sort of proprietary bits to the code. You mash that up with the GPL, and suddenly everything is G ped and. Sometimes that does not work for businesses for various reasons. So this ability to load a, an actual separate elf is really pretty interesting. I'm surprised that hasn't taken off more than it has. I don't know. It's very possible that breezy box will be the gateway drug that gets all of us hooked on elfs, on SB 32. Valentyn: That remains to be seen. Jonathan: Yeah, I think there is, we I asked briefly about trying to commercialize and I don't know exactly what the model would be, but there's some really interesting things here. I could see some business really picking up on this, so hopefully. Between Hacka day's reach and floss weekly's reach, we will connect you with the right people and somebody, somebody will be interested and reach out. That would be, that'd be pretty cool. I would be, that would be very fun to be able to be a part of that. Valentyn: Sure. Thank you for support. Jonathan: Yeah. Anything any other projects on the horizon or any other features you're excited about coming? Valentyn: The running it on more devices running it on risk five and writing a compiler for risk five is like my, the most exciting thing at the moment. So have to think of anything else because I need to take it one step at a time. It's just a solo hobby project, if you remember. Things go slowly. Jonathan: Fellow developers want it. Valentyn: I definitely can see exception some requests. Jonathan: Yeah. Yeah, absolutely. Alright. Is there anything that I haven't asked you about that you wanted to let folks know about? Valentyn: We did cover a lot. Jonathan: We did, we got through my mind. We got through very quickly, and we actually have a few minutes left if there was something else to talk about. I'm not entirely sure because we did, we were very efficient getting through the coverage of the project. Valentyn: I could just recap just about the place of this thing of how far nine lesions go with it. As I said. There are many ways to build something like this. Even in low power world there are chips that run Linux on some five or 10 watts of energy and there are some much further developed terminal apps. Absolutely. But box, I think just feels a niche for someone who just is already settled in E-S-P-A-D-F specifically. They like it and they are doing something with it anyway, and they just see the use for them. Maybe it helps them in some way to have a show on it or maybe they just they also find it fun to play around with it. Or maybe they want to have a mini computer that is limited in resources on purpose. I find it fun. Maybe someone else finds it fun. When you actually think about those kilobytes of REM you use in your next game? Jonathan: Yes. Oh, now here's an interesting question for you. What about support for other peripherals? So can your applications talk over GPIO, for instance, Valentyn: GPIO? You can always do something with that is regardless of printed box and everything, it just. What other peripherals I was planning to add in my demos and just on my fun stuff myself, I wanted to add sound support because I'm also a little interested in music. It would be fun to make also some noises on this device. Maybe make some small, simpler or something. Jonathan: I, so that's the direction I was thinking about. I've done some with modular synthesis and I know it's always interesting to people to have these sort of, not analog, but in this case a low power device to control that. And can you do GPIO out? Can you do triggers out? Could you do midi out and control it from within Breezy Box? I think there's some interesting questions there. It's Valentyn: not a framework, it's just a helper. It is just the terminal subsystem and you can run whatever you want on it. Jonathan: I guess I'm thinking more about inside one of the c programs that gets compiled. Valentyn: What is Yes, that's also possible. And the API is just c function calls, so you can have anything that you have in your firmware, any function. You can also call from those elf binaries. Jonathan: Oh, okay. Can the elf binaries then talk to the expressive IDF? APIs. Ah, okay. That does make it a lot Valentyn: easier. You can call the PS RAM specific analog, for example, that's used a lot. You can reach for G PIOs. Jonathan: Okay. So you're not having to implement like every single thing. You're able to reuse some of the ES expressive tools inside of those elf binaries. Ah, that helps. That helps a lot. That makes a lot of this possible and, gives someone the ability to expand it into one of these additional sort of interfaces a whole lot easier. That's very cool. I'm glad that works. I've had to do supporting of, let's take this weird interface and make it work someplace where it's not intended to work with a completely different API. And sometimes that is a pain. I've done some work getting SPY and I two C on Linux to behave the same way that they behave inside Arduino and. That's pain and suffering sometimes. So I'm glad you don't have to do that. Alright. Excellent. It has been a lot of fun to chat with you to chat about this and again, all the best on trying to find that next contract and next line of work. If someone wants to reach out directly to you for that, what's the best way to do it? Valentyn: Best way to do it is, probably by email well@danielchip.com as my last name. I don't know if you have some company in text, you know my email. Feel free to share it. Jonathan: Okay. We can do that. We can Valentyn: list that. Oh. People can also reach MR through GitHub. I'm easy to find. I'm not hiding. Jonathan: I'm not hiding. There you go. Alright. Thank you. Thank you so much Valentine for being here. It was a blast to talk with you and I for sure will be keeping an eye on where this goes. I think I may have to go order some hardware and try it myself 'cause it's just cool. Valentyn: Thank you. It was a pleasure for me as well. Thank you for having me. Yeah. Yeah. Looking forward to talk to you. Jonathan: All right. Excellent. Alright, that was Valentine of the Breezy Box project. We, again, we did, we went through that really quickly, but it's a really cool project that I'm. Very interested in and hoping to be able to use some of the libraries and some other things. We'll see. We'll see if we can make that work. We do have a guest already scheduled for next week. We're talking with Milo Schwartz of Pangolin. That is a security company, I believe, and related to crowds sec, which we talked to a couple of weeks ago. And then we've got some open slots. So if you run an open source project or you know of one that we need to talk to, let us know. You can get us. At floss@hackaday.com. And then I also have a stack of business cards from Embedded World that I'm going to be reaching out to. So hopefully some names that people recognize that we will be talking to as well from there. But hey, we appreciate everybody that watches and listens whether you get us live or on the download, and we will be back next week on Floss Weekly.
• Episode 876 transcript 25.03.2026

  FLOSS-867 Jonathan: This is Floss Weekly, episode 867, recorded Tuesday, March the 24th. Pangolin, people can lie. Hey folks, it's Time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and today we're going to talk well. A little bit of IO ot. We're gonna talk security, we're gonna talk about something that claims to be a replacement for A VPN. Among other things we're gonna talk Pangolin, the open source solution for connectivity and updates and all sorts of, all sorts of other things. Um, I am not the expert on this. I don't know a whole lot about it yet, but I've got the guy I know, I know a guy, and I've got the guy here. Uh, we've got Milo Schwartz and he is the co-founder of a company called Faial and also behind Pangolin, software engineer by trade. He's done IOT and ot. I'm not sure what OT means in this instance. Anyway, we've got Milo here. Let's bring him on the show. Hey, welcome. Welcome. Milo: Hi, Jonathan. Jonathan: Hey, how's it Milo: going? Jonathan: Hey, it's great. Excited to here. It's great. It's good to have you here. Okay, so first off, I know what iot, the Internet of Things is. What's the OT sector? Milo: Yeah, that's a good question. Um, it's operational technologies. Oh, okay. Right. So it is informational technologies, OTs, operational technologies. Uh, and that tends to mean there's some physical thing out in the world, like, uh, maybe it's a, you know, camera, right? Mm-hmm. And you're connecting to it and then providing digital services on top of it. Jonathan: Gotcha. Milo: Yeah. Jonathan: Alright. So what, tell us the story starting, I guess, with Al Well, how, how did all of this come to be? Uh, what is this that came to be that we're talking about? Milo: Okay, so yeah. So it all started without a name, right? I think most, um. Uh, projects, um, uh, that start as a hobby, right? You like the name is the last thing you're worried about. Like, lemme build the thing first, lemme get it out there. Uh, so, Jonathan: or, or you pick the name first and you realize five years later that it was a terrible choice, but you're stuck with it. Milo: Exactly. That hurts a little little. That hits a Jonathan: little close to home. Yes, I know. Milo: Um, anyway, yeah. So I actually started this, um, project, this tunneling project, uh, about a year, a little over a year ago at this point, like maybe a year and a half ago. Okay. The fall of 2020, uh, excuse me, four. Um, I was just kind of looking at open source projects to build, uh, in the connectivity space. Um, and, uh. With Pangolin, um, we're like trying to find names, right? We wanted to, the one thing you do is you try to find a domain, right? What, what domain can I get that has a nice tld, like a.com or a.net, right? And there are very few like nouns available, unsurprisingly. Jonathan: Yeah. Milo: Yeah. So, uh, we, we were looking at just like what different, um. What's affordable, and then what can we use that gives us a nice space of names to name all of our products, because we wanted to be able to say, okay, this is the company name, and then the product name is X, Y, Z, and Fal is a, uh, classification of animals. It's any animal that tunnels is a, is a faial animal. Jonathan: Oh, nice. Milo: Yeah. So, Jonathan: oh, that's really, I like that. That's really clever. And you've also got the, the Foss as the beginning that everybody's gonna recognize Milo: e Exactly. It kind of was too good to be true. Um, so it, and the domain was like 2K, so it was within reach, you know, if, if we started to make some money from, uh, the, the project. Uh, so we, we were just like, let's, let's do it. Um, and we, it allowed us to pick cool names like Pangolin or Newt, uh, or Om, which are all different types of Faso animals for our products. Jonathan: I like it. You know, I've, I've been told that whenever you go to name an open source, a company in general, but open source company, open source project. When you think you've got your name picked, what you need to do is go punch that into Google and make sure that you're gonna be able to get your hit in like the top three results. Yes. And so I, I had, I interviewed one time a group that, uh, scarf was the name of their project I think. And it was super cool. It was like, you know, build your own API, I think, I think is what it was. But, uh, I went to Google for them and I could not find their project because, you know, scarf Team U and, and Amazon and Walmart and Target. Like, you get all those results instead. It's like no scarf programming and then you start getting results. Here's how you build a machine that you program that'll knit you a scarf. It's like, oh, I didn't want that either. So go do the research, make sure you can. You can, you can take over one of those top spots on Google. Milo: Well, you were talking about regretting the name later. I think that's our biggest regret is Pangolin. You know, you get all sorts of cute pictures of Pangolins that come up. Right. Even there's a documentary that came out around the same time we, uh, launched the project that dominated. Um, and, uh, actually in COVID too, there was the whole Pangolin COVID controversy. Mm-hmm. So that, that tends to, to bury us. But we're, we're working our way up the SEO ladder at the moment. Jonathan: Yes. Um, didn't, did Ubuntu use Pangolin as their animal for one of their releases too? Milo: Yeah, there's that, there's, there's the Ubuntu Pangolin, there's also, I think System 76 has a, a laptop code named Pangolin, so, Jonathan: oh, Milo: it, it pops up a lot. Jonathan: Yeah. Yeah. Good. Okay. So is, is Pangolin at this point kind of the, um, uh, the flagship product that you guys have? Milo: Yes. Yeah. Pangolin iss the flagship product. Uh, we actually, our, our domain now I mentioned forso.com. Uh, we, we do own that, but. Uh, everything now goes to pangolin.net. Okay. Um, as the main domain. Yeah. Jonathan: Got it. Trying to get that SEO juice. Uh, so what you mentioned tunneling, and this is something I've done a lot over the years in, in various ways. What's, let's start with this. What's like the, what's the problem that you needed to solve? Why, why did none of the other solutions for tunneling work and then what was the sort of differentiator with bangin? Milo: Yeah. So yeah, there's, there's a bunch of ways to tunnel. Um, we, we started as an open source CloudFlare tunnels alternative, so, Jonathan: okay. Milo: Um, what the type of CloudFlare tunneling, um, is, is it's outbound only tunneling, so it's not peer to peer necessarily. Um, and where this was born out of was my background in the iot space, um, which, uh, I don't know if you wanted to get into that now, but it's, yeah. We, we, uh, basically, you know, in the IOT space there's, there's, you put these devices oftentimes behind cellular networks mm-hmm. And they're behind CG app. Um. And the big thing there is how do you get, how do you expose something when you're behind a network that you don't control the firewall? Uh, and you just need to get access to some web panel for remote management, remote control, remote troubleshooting, whatever it may be. Jonathan: Mm-hmm. Milo: Um, so there really aren't any good well packaged like open source, um, reverse tunneling, reverse proxy tunneling, uh, solutions out there that have a, a nice little binary and then do forward off and let you manage your different sites. So we, we had looked for things to use at our, um, our former company, um, the iot company before starting, uh, penguin. And, uh, penguin was kind of the thing we wish we had at that time. So that's what we started working on that as a hobby project and then, and pushed it out. Um. Yeah, that, that's, that's kind of the, the high level, like what bread the, or what was the need and then what led us to, to building it Jonathan: that I I will just, I will just say quickly, that is a great way to pick a project and to even to start a business Yeah. Is to ask yourself like, what thing do I really wish existed? What would make my life easier? And can I build that? If the answer is yes, go do it. Make money. Milo: Yeah. Jonathan: Yeah. Well, Milo: and, and I'll be clear like there are, there, this is not a new idea. I mean, there, there is N Brock, there's FRP, right? Sure. Like there's things that do this. Um, there, the, the big one for us was the packaging. Um, we wanted a simple like binary or simple container that you could just chuck onto Chuck anywhere, right. And then have. Visibility into that network and then be able to define resources on, uh, on that network to then both do CloudFlare tunneling and then later, which we can get into. Uh, now we do peer-to-peer tunneling, so more of a traditional VPN. Jonathan: You know, if I were to try to solve this problem right now without thinking about Pangolin, I would, I would reach for something like wire guard or even open VPN I've used to open VPN in the past. What's, what's sort of the differentiation there? Milo: Yeah, so penguin's built on wire guard. Ah, Jonathan: okay. Milo: Yeah. Jonathan: Very cool. Milo: Yeah, it's, it's, uh, not smart to reinvent, um, uh, an excellent tunneling protocol on encryption protocol. It's not, why not do it? Right. Jonathan: So, so when, when, um, oh, I can't remember his name. We've interviewed him too. The, the dev behind wire guard. Um, anyway, Jason. Yeah, Jason, that's right. They, he talks about, um. And in some of the docs they talk about that wire guard does this one thing very, very well, but if you want to do, and then they've got this like DHCP over the, over, over wire guard or, um, all of your configuration stuff. It's like you need some other service on top of it. And so that's, is that sort of where Pangolin fits in it? It is the other service that gives you all of that configuration and, and automatic deployment stuff? Milo: Yeah, it's one of them. Um, yeah, I, I mean I think, uh, there's, there's all sorts of projects built on, on wire guard these days. I mean, famously tail scale dominates the mesh networking space at the moment. Um, I believe. CloudFlare even uses wire guard for, uh, so Jonathan: yeah, Milo: for some of their internal tunneling. Um, we used to use that our, our former company, um, for also like getting into devices. But yeah, the, the big thing with wire guard is it's very low level. Mm-hmm. Or you have to manage keys, you have to manage, um, uh, you know, it's not based on users. If you, if you wanted to like, bring on a new contractor to help you with something, you would have to generate new keys for them, send down SSH into Milo: the Jonathan: server and run the script and generate the keys. Right. That's a pain. I, I, I understand. Been there, done Milo: that. Oh yeah. And I, I've also set up countless, like you spin up a new cloud environment v PC or something, and you got, first thing you deploy is you play wire guard. Tell them to get in there. Jonathan: Yeah. Milo: Um, oh, Jonathan: absolutely. Okay. So, um, let's see. We, we, we've got, we've got this problem space. What was the, uh, what's, what's sort of the, you built, you built it on top of wire guard. What does the interface like, what's the, the actual step two process look like to actually use Pangolin? Milo: Yeah. Yeah. So we, we try to, um, sort of break things down outside. We try to abstract the network, right? Um, the whole part of the whole inspiration for Penguin was people these days are thinking about applications. They're not thinking about, uh, and identities. They're not thinking about ips, they're not thinking about overlapping subnets, et cetera. Um, so we wanted to boil access controls down to, um, sites, which are like your, these are your physical, physical networks. Mm-hmm. Um, or virtual networks, even if it's like a BPC. And then also resources and users. So resources are the applications, the things you wanna get access to, and users are the people, right? The, your personnel that need to get access to those resources. Um, so in terms of step-by-step process, it, it's very quite simple. You just deploy the site connector thing, um, which is, we call Newt, just back to the naming scheme. Uh, newt's a little salamander lookin creature. Jonathan: Mm-hmm. Milo: Um, and then that gives you that entry point into the network, hence why it's called a connector. You define resources for things that they don't necessarily have to be on the, the actual location of the site itself. It doesn't have to be on the, the wire guard pier. It could be in the address. It just has to be in the addressable range of the pier. Mm-hmm. Um, and you can, uh, then give your users access to those resources and the users when they, they can access to the web browser or they can connect in through, uh, client. And, um, you know, it doesn't matter if you have 150 sites and locations, you could have 150 connectors out there with, you know, 10 resources on each of those locations. And the users will be able to access all of them, um, at once without having to jump between sites. Jonathan: Yeah, very cool. Um, I've, I've seen, I've seen the need for this sort of thing firsthand. Uh, actually for a little while I was playing around with the idea of building something similar to this, uh, but then building a physical, a very small portable physical appliance. And the, the, the point here, the thought process here was one of the other hats that I've worn through the years of my career was a telephone service. And so there's been multiple times that I've rolled out to, you know, a hotel or what have you, and they've got, you know, they've got a big AVI switch or whatever, whatever brand, right? And the tech that I'm on the phone with at headquarters, he needs to be able to remote into it to fix something. And you, I mean, based on the fact that I'm there, you know, something is broken, so the, the backup modem is unplugged or somebody stopped paying for the, the POTS connection. Something has changed with the internet connection. And so they don't, they no longer have an IP address to dial directly in. And so they just want to be able to get some kind of connection into that phone system to be able to do their thing. And my thought process at the time was, it would be cool if I had just like this little router that it would, on one side, it would get an IP address from DHCP. Wire guard would connect it, go up, create the tunnel, and then on the other side you would have sort of this managed connection that you could plug into the switch. Yep. And everything would come up and they'd be able to jump in. It sounds like that's what you've done except in software. Have you thought about making a hardware version of this? Milo: Yes, we have thought about it. Yeah. So this comes back to, again, to the roots. So, um, uh, the iot world, it's very, and and the OT world, it's very popular for people to deploy edge devices. Mm-hmm. And there's this whole world of. Of, uh, manufacturers. There's Tonica, there's a Van Tech. Um, the GLI nets one that's a little bit less in the industrial space, but the whole idea is, let me just buy this device. I can plop it somewhere. Mm-hmm. Um, it, all it needs is an internet connection, and I can get in and I can access things. Um, so we have built a software, we built like a, we like to think of it as a hardware agnostic, um, version of what some of these providers provide. 'cause they're all, they're like tied. Tonica has a kind of a solution to this, but it's tied to their box. Um, so we, we, we are considering hardware. Um, uh, it's, I don't think it's our top priority at the moment because, uh, because of the whole hardware agnostic thing, right? Mm-hmm. That's like, like it's part of our edge at the moment. Um, but it, it's certainly open, uh, to consideration like selling a, a cheaper box. Uh, maybe, you know. Like a, a raspberry pie that's well packaged and then deployed something that's not quite as expensive as some of these hardened, like, edge devices that could be overkill for just simple access to a phone system. Jonathan: Yeah. The, the deal with that, and this is what I ran into, is, was the chicken and egg problem, right? So like there's the, there's the engineering requirements to make the thing, you gotta put some money and some time and some effort into it, and you've got it. But then that's not any sales. You've gotta also have somebody on the other side that says, yes, yes, we like that idea. We will pay you money for it. Right? But to get that sort of commitment, sometimes you've gotta have the device in hand that you can show them. So that was, that was where I got stuck, is trying to Milo: Yeah. Hardware's hard. Jonathan: Yes. Milo: Hardware's very hard. Uh, yes. For that reason. You have to manage inventory, uh, things break. It just adds a lot more variables. Jonathan: Yep. Milo: Um, I'm definitely much more of a software guy, um, at the, again, former company and mainly doing software. There's someone else doing the hardware. Um, but, uh, I, I like. I like seeing how they can, how they can marry. Jonathan: Absolutely. Milo: It's, it's, it's more exciting. Yeah. Yeah. Jonathan: Yeah. Agreed. Um, so you've got a note here about, uh, Nat Traversal and corporate firewalls, which this is something that I also have fought with multiple times over the years. Yeah. Milo: Yeah. Jonathan: Uh, have you seen that our, our corporations starting to try to block, uh, wire guard? Is that something you're running into? Milo: Uh, actually yes. Uh, there, there is a bit of that, um, not necessarily blocking wire guard, but blocking just, just very hard nats where they're doing, uh, maybe port randomization or they, uh, are blocking all like, or UDP traffic, which obviously makes it very hard to, to handle the tunneling. Jonathan: Yep. Milo: Um, yeah. Jonathan: Yeah. This, this was something that. Uh, another, another project that I was involved in years ago was a, uh, a port knocker. It was a FW kn, that's how we said it. Uh, or wha kn. Um, it was a, it was a port knocker that actually used real encryption and authentication inside of it, rather than just the list of ports. Uh, but that was something we ran into is that, you know, all of the, we were gonna send outgoing UDP packets and all of these corporate firewalls would block them. And it's like, no. Yeah, that's kind of a downer, isn't it? Um, you guys, and, and I'm afraid that at some point the corporate guys are gonna realize that wire guard exists. And that it is a potential avenue for exfiltration, which that's something that those guys, you, you have to, once you get into like a certain, um, a certain level of business or working with a certain type of data, you, you, you're required to care about this, right? So it's not, um, it's not entirely a ridiculous concern, but they, they care a lot about things like exfiltration. They don't want a disgruntled employee or a spy or what have you to be able to send data off the network. And, uh, my fear is that at some point, these corporate guys are gonna realize that wire guard exists and they're gonna start looking for it and blocking it. Milo: Yeah. I, I, I don't doubt it. I mean, I know, um, uh, many public wifi networks, uh, like famously schools will block VPN connections because they want to. Track and block. Mm-hmm. Uh, internally and a v VPN's a way to circumvent that, but that's a, it's a different type of V vpn that's more of a, a nor Nord VPN or PIA that they're, they're going after and less of a remote access. Jonathan: Right. Well, but I mean, the packets kind of all look the same. Milo: Exactly. Right. Yeah. There and there's ways to get around it. I mean, so there's, um, uh, there's on like the, the nor VPN side, there's ways to wrap packets in, uh, like an SSL wrapper, so that way it appears to just be normal web traffic. Jonathan: Mm-hmm. Milo: Um, or proxy it through some soc server. And then, um, it also appears to be web traffic. And, and I think those are pretty successful. 'cause at a certain point, the, uh, the firewall has to make a, a guess. Right. And it's, I guess how, how lenient do you wanna be on that guess? Jonathan: I, I'm trying to remember. I've, I've done some reading about technology developed for like the Chinese great firewall. Yeah, people trying to tunnel, um, either, uh, either just VPNs or in some cases they were trying to tunnel, um, tour through it. And some really interesting technologies people have come up with, like hiding tour traffic inside of. Uh, telephone, like VoIP calls. Mm-hmm. Um, yeah, one in one in particular, like you would create a Skype call at first, and then you would, you would enter, exchange the internal traffic to be, instead of Skype, it would now be tour. Um, do you, have you guys sort of borrowed any ideas from, from the stuff going on there? Milo: Not really. So, uh, in, in terms of natural universal, I'll go back to the, the original question. Uh, we're, we're mostly doing, um, mostly doing port hole punching. So the classic it, it's what again, tail scale does this. Mm-hmm. Uh, zero tier does this without wire guard, but you know, you fire packets from both directions, both the site and the, the user's client out to a central coordination server. Mm-hmm. Just fire a big stream of just random U DP packets. Basically, you look at the two ports that were opened on the, the two gnats, and then you just, you point them at each other, um, to do that peer-to-peer. So that's, uh, in terms of nat reversal, that's uh. Really the extent that we go there, uh, really is it like a stunt server like Jonathan: you would see in sip? So essentially what that is, Milo: uh, it's very similar. It's actually, so it's our, we, we did not follow the stunt protocol because, uh, we, we thought we wanted to be special. So Jonathan: you said Milo: it not Jonathan: be Milo: Yeah, yeah. No, we, we, uh, we thought it would be more, more fun and just a little painful to, to develop our own. So we have our own protocol, but effectively if you understand stun, you understand turn it basically works the exact same way. Jonathan: Okay. Milo: Yeah. Jonathan: I mean there's probably some advantage to not being identifiable as a stun server and doing turn Milo: right. Jonathan: But, uh, it's, that's interesting. Uh, Milo: yeah. And, and sorry to interrupt, but there's, there's also the, um, the outbound tunnels. So the, the peer-to-peer thing makes a lot of sense for a VPN connection mm-hmm. Where the person's running a, a client to be able to connect in. But when in situations where you aren't able to run a client, um, and you just have a web browser. Then you, that's where the reverse proxy comes in. Mm-hmm. Um, because you can just establish an outbound only tunnel from the site connector to that coordination server in the cloud. All traffic will hit the coordination server basically as a front door or a gateway and then go down. If it's allowed traffic, it will go down the right tunnel to the right network. Jonathan: Yeah. Very cool. And so you guys, you, you mentioned this briefly, but you have like the idea of sites and resources to where you can have one device sitting on, uh, a network somewhere, but then what you have sort of a, a, a little cloud of IPS around it that are accessible. Milo: Yeah, basically. Um, so I guess one another difference between Pangolin and wire guard is, uh, you know, wire guard's all based on peers. Jonathan: Mm-hmm. Milo: Um, so you, you get access to the pier and what's on the pier, uh, unless you set up proper routing. So what we've done to sort of flatten the network. Is, um, we've added a proxy, um, in front or I guess at the end of the wire tunnel. Okay. So traffic can come outta the wire guard tunnel and then it gets proxied out to another destination. So this is built into our, our new connector. Uh, and what that lets you do is yes, you, you can define different resources in the addressable range of the connector, um, that, uh, you can then get to and you can actually, um, refer to them or, or, you know, your, your users can connect to them via the familiar land addresses when connected into the network, um, even if they're on overlapping or, or d completely different physical networks. Mm-hmm. Uh, which is a difference between us and something like a, uh, again, a tail scale or traditional traditional wire guard where you, you would just be addressing nodes within a network. Jonathan: Mm. The other advantage I would think is that if you just set this up with wire guard and the routing, then you can get to all of the ports. And it sounds like you guys, you, you give some administrative control to say, you know, you get to talk to port 4, 4 3 of this device and that's it. Milo: Exactly. Yeah. Yeah. So you, you define a resource and you say exactly which ports are available on this resource. Uh, you could block all TCP block, all UDP, you give a range, et cetera. Um, and then you set users of roles on that, on that resource. And, and yeah, of course you can do this with, you can throw a, a wire server somewhere and you get access to everything on that network. Um, but you, you can't be connected to multiple at the same time easily. Um, traditionally it's, it's either, you know, you pick a site, um, and, and that's your entry point. So Penguin lets you sort of link together different networks in a, kind of, in a way. Jonathan: Yeah. Very cool. So you guys, you needed this tool, you built this tool, you decided to open source the tool. What did, what did that conversation, what did that decision look like to open source? It? Was it just always the plan or was there at some point you had this business discussion like, this makes more sense if we release it under the no GPL or the MIT or whatever. Milo: I think business came second or third or fourth. I don't, I don't even know. It didn't come first. Um, so we, I was, I always wanted to run an open source project because just been being involved in this space, um, for a long time. And, uh, most of my hobbies have tied back to the home lab in some way. Jonathan: Mm-hmm. Milo: Um, I think like many people in this community, so I, I was deploying servers and, and deploying image, you know, all that, all the good stuff that you would expect next cloud. And I saw like. Now I kind of like looked up to some of these maintainers as, as like role models a little bit and was like, I, I wanted to be one of them. Mm-hmm. Um, I also liked the idea of actually having people use my software. Um, uh, again, coming from that other company, we, we were selling to a very select group of people. So we, we, we made money, but on very few people. So it, it was just less gratifying. Yeah. Uh, you get less feedback. Um, so we wanted, knowing all of that, we wanted to, um, create our own open source project. We wanted to grow naturally, um, which is the ideal scenario. People share word of mouth. You get all sorts of people from all over the place using it, and that all feeds back into the projects. You now can, the community can basically pull it up into, um, a place where it's beneficial to everyone and gives you ideas, right. For free. Mm-hmm. Right. I mean, people get very passionate about open source software, um, Jonathan: to def Definitely, Milo: yes. Yeah. Yeah. That, that was, that was the arc of it. It was the, the intention from the beginning was to open source it and we launched it on Reddit. Sorry. Jonathan: Okay. So you said you, you'd always dreamed of being a dev a, uh, a maintainer. You always wanted to have an open source project. You always wanted to have a lot of users that were using it. Now that you have those things Milo: Yes. Jonathan: Do you regret it? I don't regret it to, for those that have, have not been in this position, um, there are downsides to having, being the guy at the top of an open source project and having users that will sometimes ask for ridiculous things and get mad at you for no reason. And like, there, there are, uh, it's, it's not all sunshine and roses. There are some thorns with running an open source project. Milo: Yes. No, it's open source. Uh, well, I like to say it's, yeah, it's a very, very sharp double-edged sword. Jonathan: Yes. Milo: Um, yeah, you, it's, it's a double-edged sword. People are very opinionated, um, about software. Uh, and they are, you know, they wanna share it, right? So I think the, the biggest problem is, is simply, is, is simply just managing, um, so much mm-hmm. Conversation. It, it becomes like a, a community management problem, right? Once you get to a certain scale. Jonathan: Yep. Milo: Uh, where do you have, how do you triage issues? Like how do you. Manage support threads. Where do you manage support threads? Mm-hmm. Uh, how do you block spammers who are like coming in and are trying to take advantage of people? Um, oh, like now the founding team can't manage all the support. Do we need to pay someone for support? Like, there, there, there's so many, so many things there, um, that are make open source very hard. Jonathan: Mm-hmm. Milo: Once you reach a certain scale. Jonathan: Yeah. Absolutely. Um, how about the, how about the pull request side of things? Have you guys, are you guys being innu in, in inundated with, uh, slot pull requests like everybody else? Milo: Yes. Yes. Jonathan: What, what, Milo: unfortunately, Jonathan: what, what have you done about that? Do you have a solution yet? Or are you still waiting through it like the rest of us? Milo: We're waiting through it, so we make people disclose, but there's, you know, people can lie. Right. It's, it's an honor Jonathan: system. People can lie show title right there. Yeah. People can lie. Milo: Exactly. Yeah. So we have a disclosure on the pull request and, and all that really ends up doing is it makes us just be heck of a lot more skeptical when looking at it. Yeah. But we still have to scan for, um, scan for ourselves. Uh, we try to write a, a big sort of con contribution guide, contributors guide that includes best practices and even areas in the code to look where. Mm-hmm. Um, if someone were to vibe code, they, you know, they could probably just take that document and it would be a good source of context. Right. For the, for the LLM, um, which would I improve to a small degree? Maybe the, the types of pull requests we get. Yeah. Um, but ultimately we still have to, um. You know, search for security errors. Uh, the big one for us is making sure, you know, the LLMs stick to ux UI patterns that we follow. 'cause they, they like to kind of go off on their own and like create their own components and, and ways of doing things. But that's not good user experience and, and our whole like thing is user experience. So, yeah. Um, yeah. Answer the question. Jonathan: Yeah. Yeah. Very good. Um, I, I pulled up the, uh, the Pangolin, um, GitHub repo here, and I see you've actually got a couple of licenses and I was curious about this. Um, at the very least I see the, the GNU, uh, A GPL, the aero general public license. Um, but I also see that some, some files are also, uh, the faial commercial license. How do those two play and interact together? Milo: Yeah. So, um, you know. As a business, you need to make money, right? So, um, uh, there's, there's a few ways you can do that. You have to, uh, sell something that is of valuable to people. So at some point we decided we had to create features that are, uh, considered paid features. Jonathan: Okay. Milo: Uh, right. And these are licensed, not very GPL. However, um, we, we were very careful about this 'cause this is very touchy subject for the open source community as I'm aware. So, um, we, we did two things. One, um, the open source, we build two versions of the application. One is fully built and distributed as a GPL three. So all the proprietary code is stripped out. Okay. Um, before it's distributed. And two, the part that is, um, licensed under the enterprise edition is free for, um, anyone using it for personal use and for non-commercial use under a certain threshold. Okay. Uh, 'cause we wanted to make it as accessible as possible to people. Um, and so far I think this has been working pretty well. Mm-hmm. Um, it, you know, it. Uh, enables access to everyone. Um, the people who are, you know, need, um, it to be licensed a certain way for distribution under certain open source, um, restrictions, can use the H-H-G-P-L version. And people who either don't care, want the extra features and are are hobbyists from home laborers like I was, um, when I started, can get access for free. And then the businesses who are getting real value out of the product, um, can pay us a little bit of money. Right. So we can fund development Yes. And fund all that support that I mentioned earlier. Jonathan: Yes. Milo: No, I, I, Jonathan: I am, um, very sympathetic to that. Somebody's gotta pay some money because the programmers have to be able to eat and pay their rent. I, I'm right there with you. I, I totally get that. Yes. Um, ha have you found that, uh, so one of, lemme back up a little bit. One of the, one of the real concerns that open source projects have had, and I think this has sort of died down some, but therefore a while it was the big deal was. A company like Amazon coming along and selling your product, selling the exact same services that you're selling for half the cost because they're Amazon and they can, um, and that, so that had to be like one of the things that you were thinking about. Um, but have you found, as I have that the A GPL is just, uh, terrifyingly effective at keeping someone like Amazon away? Milo: Yes, I think so. Um, I think we're, yeah, like, I mean, I, I saw the whole Redis famously. That's, that's the one that kind of shook us, or even WordPress I think is having all sorts of issues Jonathan: at the moment. WordPress has Yeah. Is kind of fighting between a couple. The, the, the WordPress company itself, and then WP Engine is the name of the other group. I've not, I've not looked up and seen, seen where that is recently, but it is kind of the same deal. Right? It's an outside company offering some of the same services, right. And competing against the guys that run the open source project. Milo: Yeah, I, yeah. I mean, um, do you know what the licensing model for WordPress is? Is it. I licensees. Jonathan: Um, I think the core is all open source and then they make most of their money by selling hosted services, I think. Milo: Hmm. Jonathan: I, I don't, I don't know all of the details, but I know that's at least part of it. Milo: Yeah. Yeah. I think A GPL is, is quite effective at doing this. Mm-hmm. Um, it, it is one of the more restrictive licenses, that's for sure. Uh, prevents, you know, uh, it's copy left. Of course if you, if you modify the source code and then you incorporate it in your product, or even if you access stuff over a network, that's the, the big one with, with A GPL and interact with the software. Right. Um, Jonathan: it, it closes the, it closes the, what do they call it? The cloud loophole, the web loophole of the, of the GPL. Yeah. So I, I, we should clear off the spot real quick and explain this. So like the GPL says that when you have it, when you have something licensed under the GPL and you give that binary to someone else, that person has the right to come to you and say, I want the source code that goes with the binary. Well, when you're providing a service on a server up in the cloud, you're not actually giving the binary to someone, you're just giving them the service. And so the GPL doesn't have the same teeth. In that case, a user, you know, I use Gmail. I know that Gmail has a bunch of GPL code in it, but I can't go to Google and say, Hey, give me all of your Gmail code. They will laugh at me. Um, because it's GPL, but the way it's written, I don't have a binary, so I don't have that right. The A GPL was the solution for that and, and it was come, it was written by the same folks as the GP L's, from the gnu. It's from the Free Software Foundation and it basically says not only do you have the right to get that source code, if you're given a binary, you essentially have that right to get the source code if you are a user of a service running this code. And so it does exactly this. If, if, you know, if, if some part of Gmail was licensed under the A GPL, I could then absolutely do that. I could write to Google and say, Hey, I am a user of your service under the A GPL. Please send me a copy of all of your source code. And the license would require to do that. And because of that, Google and Amazon and all of those big cloud companies have very, very strict regulations that no one in their company is to touch a GPL code. Like, don't even look at it for too long. Um, you know, I imagine in their, in their programming cubicles, they must have like the little eyewash stations. Like if you look at at TVL code for too long, come run to the eyewash station and wash your eyeballs out. Um, it's almost, it's almost that serious, uh, because man, it would, it would poison all of their stuff in, in the way that it's addicted to, right? And so that's just sort of the, the, the tension between the two. And so what a lot of companies have done is very similar to what you're doing. You have, uh, either a, a, a tiered approach with code, or you have a dual license with a, uh, uh, A CLA so that, you know, you're open source, that you're giving out to everyone. You can run with a GPL and nobody cares. It's fine. But if it's Amazon, well they gotta pay you the big bucks to be able to do it, which works out pretty well. I mean, like, that's a fairly, it's a fairly ideal solution for everybody. Milo: Yeah, no, I think so. I recommend the approach to anyone. Um. Of course you need to do a license if you ever do wanna do a commercial license. 'cause if people contribute code, you have to, um, you know, maintain the right kind of copyright. Jonathan: Mm-hmm. Milo: Uh, over that to redistribute it. Um, yeah. Jonathan: Do, do you guys have a contributor license agreement? Milo: Yeah, we, we have ACL a. We were very careful from the, careful from the beginning. Um, yeah. You, you asked about open source if that was our Yeah. First intention it was, but we always knew if this was going to become a company. Um, that we needed to be transparent at the beginning. 'cause we, we've seen all the problems of like, changing licenses and, and making Yeah, like, right, like people contribute their, their time and their code and then you take the copyright away from them. Um, 'cause technically they own that when they, they. Contribute. Jonathan: Mm-hmm. Milo: So we, we, um, even though the commercial side wasn't figured out at the beginning, we added a CLA just in case. Um, and there's pros and cons to that. I mean, obviously it protects the maintainers, um, uh, and the project, but it also means you, you, you get less contributions. 'cause some people don't wanna contribute if they have to sign ACL A. Jonathan: Uh, something else that we found, I don't know if you guys have found this, some people can't legally cannot contribute with a CLA because they have signed an agreement with their company that all of that code, you know, hobbyist or whatever belongs to the upstream company. And so we've got some guys from a US company that would love to be able to contribute code to the, the one of the projects I'm involved in. But we too have ACL A, um, for, for somewhat similar reasons. Ours is, ours is more, uh, I, for, I forget the name of it. Um, it's not so much about commercial use, but it's more about being able to fix a problem if there's ever a, an issue found with the license. Um, but anyway, we've had a couple of people that says, I've been talking to legal for months about this and we still can't figure out if I'm able to write this code. Sorry. Do that. I think Milo: that's, that's the big downside for sure. Um, yeah. Is is just concerns, I mean you mentioned it both on the Google right? They won't, they won't touch any A GPL software. Jonathan: Yeah, Milo: I think it's, um, fortunately I think there's, like, it matters more for dependencies of software, right? Something like a library that you're going to integrate into your code. Right. And a little bit less for fully packaged ecosystems, which I think we fall a bit more into 'cause the software gets deployed standalone, um, and it's less interlinked with things, um, which I think helps us a little bit. Jonathan: Yeah, yeah, absolutely. Um. I just one more comment on this, on the idea of CLA and the business use case. You know, there's been, obviously there's been some real blowback in communities when, when businesses have done this, they've taken something and they've added either added A CLA or they've used a CLA to re-license code. Um, but I think if a project is started from the beginning and it's made clear that, look, this is open source, but it's also gonna be a business, we're going to have a CLA on it. This is why, this is what the CLA allows us to do. Like most people get that, and particularly when it's. Stated from the beginning. I find most users are, are pretty open to that. They may not like it necessarily, but you know, they're not gonna Milo: Right. Jonathan: They're less likely to call. You mean words on Reddit as a result? Milo: Yeah, I know the, the name of, I mean, the whole name of the open source game is transparency. So, um, right. Part of it's the whole idea get access to the source code, so you need to be transparent on every front. Um, and there's actually, you know, another inspiration of ours, but we haven't, um, we don't have the time these days to really go deep into this area, is companies like GitLab, uh, or Fleet dm, which are, have built this model around an open source company too, where all of their processes are, are fully. Available on their website, even their sales processes, you know, you can see how they, they go after clients and stuff. Um, and as a, uh, uh, you know, as a customer right, of, of a company like that, um, I think that's as almost as almost more important than sometimes than seeing some of seeing the source code because you can see how they, they do, how they do things, how they operate their business, and do you trust them? Do you, for lack of a better word, vibe with that? Jonathan: Yeah. Milo: You know, Jonathan: I will say that there are some business models where that just does not work. Milo: Yeah, of course. Jonathan: I've got, I've got an email sitting in my inbox right now that I'm a little scared of that is marked not for, you know, do not distribute, do not blah, blah, blah. It's like, uh, I would not, I would not be able to do that at all with that particular, um, that particular contract. But Milo: yeah, no, I mean, there's levels to it, of course, right? Like. Um, I think, I think the big ones for me are roadmap, right? Roadmap is, is not necessarily, doesn't need to be held super close to, to the heart. Sure. Uh, make that public fleet dm. I really like how they, um, do sprint. I think they, they do their sprint meetings, like their standup calls with the company. All of those are live streams so everyone can see. And I always thought that was super neat. Um, because usually that's internal. Like you have no insight into what the company's meetings look like. Obviously you can't, you know, you can't have a confidential meeting on, on, on the public internet, and I'm sure they're not. They're picking and choosing. Jonathan: Yes. Milo: But it, it makes you trust them more because they feel a little bit better Jonathan: about it. Milo: Yeah. Jonathan: Yeah, absolutely. So what, what does it look like to, to self-host Pangolin? Milo: Yeah. So, um, uh, pangolin started fully self-hosted. Um, so open source, fully self-hosted. We didn't have a cloud. We had no idea how we're gonna build a cloud. The system's very complicated as you could imagine, with tunneling and relay servers and coordination servers. And when you're dealing with the reverse proxy now there's like, um, ways to sync certificates and ways to, you know, you have to move DNS and, and all that type of stuff. Yeah. So the first version of pendulum that is self-hosted was just a stack of three containers. There's a, the penguin container, which is the sort of business logic. Jonathan: Mm-hmm. Milo: Um, all the penguin specific stuff. There's this other container called gerbil, uh, which is, uh, another animal named another Faso animal that handles all the tunneling. The idea there is that. You know, gerbils, they, they move through. Like if you go, ever go to like a PetSmart or a Petco and you see all the tubes. Jonathan: Yep, yep, yep. Milo: Yeah, that was the idea there. 'cause like Gerbil is building that whole network. Um, and then finally we use traffic under the hood as our reverse proxy. So it's actually handling, um, the routing and, uh, certificate generation with ryt and all that good stuff. So all that stuff gets deployed on a stack onto VPS and, and boom, there's your simple self-hosted single penguin server. Okay. Uh, but then you can graduate up, um, to what we call penguin clustering or multi-region deployments. Mm-hmm. Um, depending on where, which one you fall into, um, this is where maybe you, you want high availability and fallback. If, if one of. One goes down where you want to put a point of presence on the east coast and a point of presence on the west coast. Um, and this is where it gets hard because if a tunnel disconnects Yeah, tunnel disconnects. Like from one of those points of presence, it connects to another one. Well, now DNS was pointing to the first point of presence and how do you get it to the, to the other one? And the same with the certificate. The certificate was being served from one pop, served from another one. Um, and this is what CloudFlare does all day, every day. Yeah. With billions of dollars. Right. So we were trying to figure out how do we do it with like. Jonathan: Thousands Milo: of Jonathan: dollars at best. Yes. Milo: Right. I Jonathan: understand. Milo: Um, so I think we came up with a pretty good solution. Um, there's, you know, a couple extra components you run. We have a DNS server, we have a certificate sinker. We could, like if when traffic, when the tunnels connect, disconnect from one pop to the other one. If DNS is still pointing to the first pop, we use like an SNI proxy to actually send like traffic over to the second pop, uh, and then go back, like kinda goes out one pop and then into the next one. Huh. So that's all can be fully self-hosted, obviously. I keep saying we, we also use this like in our cloud platform because we did eventually build a cloud platform, and that has to be like globally distributed and all that stuff that comes with the cloud platform. Jonathan: So, so you guys were, you guys were containers from the very beginning. You are, as they say, cloud native. Milo: Exactly. Yes, yes, yes. Jonathan: If somebody says, I don't like those dang containers, is there a way to host it? Is there a way to get it running without containers? Milo: Uh, technically yes, but it might be kind of hard. Yeah. Yeah. We, I, I don't think we have any official documentation on it, but I think you could reverse engineer it if you wanted to. Jonathan: Yeah, yeah, Milo: yeah. Jonathan: Makes sense. Milo: Uh, it's Jonathan: funny. Milo: Yeah. But we started a docker, um, and graduated app, I think as many do into like Kubernetes. Uh, the Real, real Men territory of, uh, cloud Native. Jonathan: That's hilarious. Yeah. I used to, I used to be that guy for a while. I was like, ah, I don't like containers. Show me how to actually run the code. And then one day it's like, okay, fine. I'll give this container thing a try. And I was like, oh, okay. Yeah. That's pretty cool. I guess I'll use it. Milo: Yeah. Yeah. It's, um, there's pros and cons for sure. I, I, um, I guess we haven't received many requests for it, so we haven't like, put, uh, any real, real effort into documenting the process. Jonathan: Can, can you scale it down all the way to run the, the Pangolin self hosted on something like a Raspberry Pi? Milo: Yeah. Yeah. People are doing that for sure. Yeah. Uh, I mean, people, um, the way we started is, oh, you don't wanna use CloudFlare tunnels? Use Pangolin. How can I do it cheaply? Let me get a very cheap VPS, like one gig Ram one, one or two C bcps, right? And, and deploy it. Um, that maps down to a small raspberry pie. As well. Jonathan: Yeah. Milo: Yeah. Jonathan: About the same. Alright, so I've got a note here that you guys have 20,000 stars on your GitHub repository, which is a bunch. Um, Milo: yes, Jonathan: I am, I am tempted to ask if they're all real or not because there there was, there was some news stories about, uh, Milo: oh, really? Jonathan: Uh, yeah. Um, Milo: what Jonathan: happened there? I, I don't remember who it was that, and this, I'm, I'm, I'm not being serious here. Um, there, there was a few projects that some investigations showed up that maybe some of those stars were paid for and, and clicked to buy, you know, bought accounts of some sort. Uh, but I, I'm, I'm not making that allegation. Not at all. It was, it was, uh, just an off the cuff joke. Um, but that, that is a bunch, that's a lot of people that saw this and went, wow, that's really cool. Um, what's it, what's it like to have that many fans? That many people watchers in the community. Milo: Yeah. Yeah. Like, I mean, like I said before, there's a lot of people using the software, which makes us very excited. Like we, we've pushed bugs out by accident before. Right. I think it's inevitable. And you're rem reminded by how many people use the software. 'cause you get inundated with, uh, I wanna like, you know, like, people reporting the book, right? Jonathan: Mm-hmm. Milo: So, um, it just, it just that many stars just bring scale, uh, stargazers are, uh, you know, they're a bit of a vanity metric for sure. Um, Jonathan: yeah. Milo: And, uh, all it really means is, you know, there's, there's some projects that will get tons of stars and then disappear and not have any real usage. Jonathan: Mm-hmm. Milo: Uh, I feel like we've seen that a lot more recently with some of these viral like. AI products. Jonathan: Yep. Milo: Um, but I think generally, like on average, more stars correlates to more usage, so that's why it's like an important, important metric to see if a project is, has adoption, um, et cetera. Yeah. So, uh, no, we did not pay for stars, but our, uh, unless you consider like yeah, we, we, we've exclusively just posted on Reddit, posted on Hacker News and, uh, our slash self posted has been where we, we've, uh, pretty much gotten all the stars and then people have shared it, you know, among themselves, which I think leads to a network effect. Jonathan: Yeah, for sure. I wonder if I should start making that one of my standard questions. Have you paid for any of your sars on GitLab? Milo: Yeah. I mean, I wonder, I wonder what the cost per star is. Jonathan: I mean, it can't be much, but. Yeah. Still. Um, interesting. So in part of your background, you actually mentioned doing IOT and OT stuff. Uh, you, you did traffic light infrastructure. Yeah. And so I'm curious if, if the circle has been squared, does Pangolin do any traffic light connectivity now? Milo: Not quite traffic lights yet. Okay. Uh, but the, the industry is, um, very close. Like everyone's kind of doing the same thing that I used to do at this traffic company. Jonathan: Mm-hmm. Milo: Uh, you, I mentioned earlier, you know, you, you deploy an edge device somewhere to control physical hardware and for traffic lights. Every traffic cabinet has a traffic controller. These things are built in like the, the eighties and nineties. They're, they're, they're old, old pieces. They're older than me. Jonathan: Right. Milo: They're like, like, uh, these, Jonathan: they're similar to some of the, the phone mainframes that I've worked on. Yes. Milo: Right. Exactly. And, and, um. They like all the traffic lights, connect to the controllers and the controllers then have a management panel that you use the Edge device to get access to. So that's basically what I was doing at this other company. Mm-hmm. Um, as well as many other things. But, you know, we, we have, we don't have any current customers in the traffic industry, but we have lots of people, you know, in industrials right? Where they're doing the same thing. There's A-A-P-L-C or some sort, there's some sort of like, um, you know, oil mining contraption, right? Mm-hmm. Or like a, uh, BMS like Building Management Systems is another one where they're linking all these different pumps and like fans into a central sort of base station and then, um, getting access to it with an edge device. Um, so it it has all come Yes, full circle basically. Um, uh, I think part of the reason it's a bit of a self-fulfilling prophecy when you're part of an industry and then. You build a product, it's very easy to sell, you know, and make a, a value proposition to that other industry. Sure. Because you know how to talk to them, you know, what the pain points they experience are. Um, the big one, the big pain point, right, that comes up a lot in these iot fleets is provisioning, right? How do you provision these devices? Mm-hmm. Contain our site connector, contain any other software, do it on a repeatable basis, and then actually push it out into the field and manage it. So there's, there's management involved with that. Um, and existing tunneling solutions like tail scale, um. Are are not really built for this in mind. Right, right. So we, we have some extra pieces in there that make it very easy. Uh, like a, a site just needs an ID and a secret, and then we have like a provisioning key, which you can use to, uh, you know, map to your devices and then it will pull down e femoral keys for, for the, the device at the point of provisioning as it comes off the conveyor belt and they get pushed out into the field. So there's like little things like that that we do to optimize, um, that, um, kind of bring us back into the, the OT or IO OT sometimes called IIOT, which is industrial iot. Jonathan: Got Milo: it. Space Jonathan: learning all kinds of acronyms today. Yeah, yeah, yeah. Um, is, is the, it's the traffic light industry and all of that. Are they still doing like raw IP provisioning in some cases? This is something I, again, I've seen in the, the, the phone system world is, uh, you know, okay, well here's your phone cabinet and just give it this public IP address. Oftentimes no firewall in front of it or anything. It's like, Milo: ah, yeah, Jonathan: I'm sure it'll be fine. It's not always fine. Milo: Oh, yeah, no, I mean, there's, um, it's crazy in the, we've heard this from people using Penguin in the building management space, uh, but also my experience in the traffic space, there was all sorts of just open ports for, um, public IP addresses. You could, if you were scanning, you could find traffic controllers and get to the front panel. Um, and it is terrifying because mm-hmm. All this industrial stuff, all this industrial hardware and software is super old stuff. So it's, who knows what kind of vulnerabilities are in there, um, that can be exploited. So part of the goal, the push right now in these industries is to close down ports, um, keep everything off the internet. And in fact, even sometimes, let's not let people connect in. Uh, like a VPN 'cause that extends like, you know, that, that to, uh, actually extends the network, uh, boundary a little bit. Right? So they only wanna do browser based access Jonathan: just to keep it isolation. Yes. So one of the, one of the most hilarious things that we used to do is, so when you're in, is this specific to a hotel, um, you would have your hotel management system and then you'd have your phone system and they would need to be able to talk to each other, but for similar security reasons. It couldn't be an IP connection, and so you would have two separate pieces of hardware. Each of them were IP to their respective board, and then in between the two, it was Uart serial connections to get these two pieces of hardware to talk to each other for security reasons. Yeah. It, it never ceased to amuse me whenever we set one of those up. Uh, very, very similar idea. So does, does Pangolin sort of, it's a VPN under the hood, but with all of the, uh, the admin stuff you have on top of it, does it sort of fulfill this need to not be a VPN? Milo: Yeah. Yeah. So the, the browser based access is what really gets, is really applicable to that industry. It gets them excited. Jonathan: Mm-hmm. Milo: Um, because. Again, all these things are, all these industrial systems are just running a, like a web panel. Um, it's just a, a website you go to usually on the local network. Mm-hmm. You go to an IP address, call port, whatever. Right. And you click around Jonathan: hopefully, hopefully not over the public internet, Milo: right? Not well. So Jonathan: office Milo: it is. Jonathan: But Milo: yeah, they would just open up that port and, and go to the web panel right from anywhere. Um, but with hangin they can just drop in the site connector. Mm-hmm. They can define the resources on, on that network. Um, and then the browser based access uses the outbound tunnel. So there's no client software You even have to run. Contractors don't have to run, like install some new software on their computer that you then have to manage. They just go to their browser and can access the thing. And then it's also protected with a layer of authentication in front where you could give and grant temporary access as needed. Um, yeah. Jonathan: You know, I hadn't been thinking of this in, in terms of this, but I could actually really use this. I've got Linux servers that I admin remotely, and I'm, I have a, they held together with duct tape and bubble gum wire guard installed that maybe I should go and just re replace with a self hosted Pangolin. That would probably work. Milo: Yeah. Let us know. Jonathan: Um, Milo: see proof of concept. Jonathan: Yeah, absolutely. Oh, that's, that's actually, that's actually a really interesting thought. Uh, one of my to-do lists is to see if I can go find an old wire guard server and prop it back up because it's got keys on it that I otherwise don't have copies of. If I end up having to just redo that, I will probably take a look at Pangolin and just see if I can do it that way. Milo: Yeah, do it. Yeah. If you have a mix of, uh, public and private resources, uh, I think it, it could be a good mix. Yeah. Um, absolutely Simple access is, is all or four. Jonathan: Very cool. Um, alright, it, we're getting close back to the bottom of the hour. Is there anything we didn't talk about that I didn't ask you about that I should have? Milo: Oof. Good question. Um, Jonathan: this was hard to figure out. You gotta do some set math in your head. Milo: Yeah, yeah, yeah. Um, I think we, we kind of, we kind of covered most, most everything. Um, yeah. So I think, I think that was good. That's pretty comprehensive. Jonathan: Okay. I did just, I did just see something I want to ask you about, and this is not necessarily an open source thing, although it kind of is. Um, you guys are part of Y Combinator. Milo: Oh yeah. Jonathan: Yes. How, how has that experience And so this is, this is sort of me taking off my floss weekly hat. Yeah. Putting on my businessman hat. Um, although there's definitely overlap here. How has, how has that been? How has that experience been? Has it worked out fairly well? Milo: Yes, it has. So, well, and you're reminding me of this. Yes. This whole quote we didn't talk about, like, my co-founder is also my brother, so we can talk about that. Jonathan: Oh, well, I'm sure that's interesting sometimes. Milo: Yeah. So, um, yeah, Y Combinator is, is a force like, you know, it's, it's an institution. It's pretty big. Um, it right now, you know, it's, I think we're in a weird spot. We were like the only. non-AI company right in, in our, in our batch. Good, Jonathan: good for you guys. Milo: Yeah. Oh my goodness. We, we, we we're proud of ourselves. Yeah. Yes. Jonathan: You should be. Milo: Uh, we were the, we were the, for, for most people, we were the boring company. Um, Jonathan: I have, I have learned, I I will let you, sorry to cut you off, but man, Milo: no Jonathan: worries. I've learned you're doing business, that there is really something to be said for boring. Yeah. Like, like you, you review a contract and the contract is boring, man, that's a good thing. You have a meeting and the meeting is boring. Like you're doing something right. If your meetings are, are somewhat boring, you, you want the, anyway, that little, little money rant. Go ahead, continue on. Milo: No, it's true. It's like all, all the big companies are. Often boring companies, if you like, if you cut out like the, you know, the top three or four, like massive companies right now, or even if you do, like, you pick Nvidia, I mean, the core Nvidia is just building these chips and like selling these chips. It's not exactly all that sexy. Jonathan: Yeah, Milo: yeah, Jonathan: absolutely. Milo: Uh, but yeah, know the Y Combinator experience was great. We, we actually, um, we did it primarily because we wanted to quit our other jobs, so we were like, okay, um, what are our options? Uh, you know, we could find a new job. Um, we could, we could just quit, you know? Yeah. Uh, we need to pay ourselves some money. So, um, a quick, a quick thing to do is shoot your shot on Y Combinator and, and they'll, they'll give you some investment to build a business. Jonathan: Yeah. Milo: Um, and, and, uh, we. Applied and got in. I think they liked, you know, that we had a ton of adoption. Um, right. Because that's a metric of do you have something that people want, which is their, the whole pitch with Y Combinators, can you build something that people actually want? Jonathan: Mm-hmm. Milo: Um, and I think we did that. So, um, that was about, that was last summer. That was summer 2025, actually just had our reunion. Um, they do reunions every now and then with the, the batch mates. So, uh, that was this past weekend actually. So it's funny timing. Jonathan: Uh, one of the, one of the things that we sometimes see in on Floss Weekly and where I do some other coverage of like the Linux and open source space is that. Sometimes venture capital is just, uh, toxic to open source projects. Yeah. Um, I guess Y Combinator is technically vc, but you, Milo: yes. Jonathan: It seems like you've had a very different experience than some of the other VC horror stories. Milo: Yeah. No, you have to, you have to pick and choose who you get into bed with, uh, a hundred percent. Um, you don't want to, uh, it's true. Yeah. You don't want to be dealing with the wrong venture capitalists. They, uh. So Y Combinator is because they're so big, they fund all sorts of stuff. Right. And they're very hands off. Um, they, they funded GitLab, right? Massive open source company. Docker came out of, out of Y Combinator. Jonathan: Yeah. Milo: Um, so there's, uh, it can be controversial, but at the end of the day, it comes down to, I think the founders and the, the roots of the founders. Are they gonna stick to, um, the ethos of what, you know, started the company? Uh, I think. GitLab has done that very well, in my opinion. Um, they're a massive co they're a public company, right. They've taken on tons of venture capital. Jonathan: Yeah. Milo: Uh, but they're able to provide both a really good enterprise suite of products that are paid and free open source software that anyone can deploy and self-host. Mm-hmm. So, um, yeah, it's just, I think you just have to, it's like dodging a minefield in venture capital. You just don't want to, you just don't wanna take money from the wrong person. Jonathan: Yeah. Do you, do you, do you feel like you guys have, uh, turned the corner and turned it into a profit profitable business? Milo: Uh, not, not yet. No. I mean, we're, we're still not profitable. We're focused on, um, growth at the moment it's still more adoption. Uh, but we have, you know, I think the seeds that are, are leading to will be profitable probably sometime next year. Jonathan: Oh good. Yeah. Milo: Yeah. Jonathan: That's, that's excellent. I'm, I am, I'm super glad to hear that Milo: human. Sorry, hu. Human cost is very expensive, right? You, you have to, Jonathan: I am aware. Yes. Am Milo: aware. Right? Yeah. So Jonathan: you, I I've, I've recently gone full-time with my, I call it a startup. That's not entirely accurate, but that's the easiest way to describe it. I've recently gone full-time with my startup. Uh, it's, yes, yes, Milo: yes. No, it's, it's very, very expensive. Especially if you hire more than just the founders. Uh, right. You can pay the founders less. Yes. Because you're, you're taking on the risk. You're, you, there's equity to be had, but um, when you hire someone, you know, they, they are taking on sometimes even more risk. 'cause like Jonathan: they're taking on a different type of risk. Milo: Yeah. Like opportunity, cost risk almost. Right. Exactly. They could be working somewhere else. Um, so that's, yeah. And, and, and part of the goal of raising venture capital is to. Hire more people up front to, so you can develop software as an investment in, in your later business. Mm-hmm. So what that means is high labor costs now to be able to sell more later. Um, and that's kind of, kind of where we're at the moment. Yeah. Jonathan: Yeah. Very cool. Alright, uh, I've gotta ask you a couple of final questions before I let you go. I will get emails complaining if I forget this. What is your personal favorite text editor and scripting language? It's a longstanding tradition of the show. I gotta ask Milo: te so text editor and scripting language? Jonathan: Yeah. Milo: Okay. So Neo Vim Jonathan: Okay. Milo: Is the text editor. Yeah. So any of them. Um, I switched to that two or three years ago from VS. Code. I tried Z for a bit. Jonathan: Okay. Milo: Um, have you used Z? Jonathan: I've not, but I, I'm aware of it. I don't know what it is. Milo: Okay. Yeah, it's, it's uh, it's like vs code but runs on a game engine kind of, so it runs at high refresh rate. It's a very nerdy thing to like mm-hmm. But um, yeah, I know Neo Vim is my go-to. And then scripting language. It's hard. It's a hard choice between, I mean. I would say go, probably go is like the best, my favorite overall language. But it's not great for scripting. Like, you know, bash is, is what I think more of a scripting language. Sure. But I, I wouldn't say I, I'm not really super fluent in Bash uh, Python's also a great scripting language. I'm pretty fluent in that, but I, uh, I like to pick up go where possible. Jonathan: Sure. Yeah. Makes sense. Milo: Yeah. Jonathan: You know, the answer, the answer to that question used to be, you know, either VI or emax. Mm-hmm. And uh, then on the scripts and language side, you know, it was pretty much either Pearl or Bash way back in the day. And the, this show has roots far enough back in the day that we've gotten a lot of those answers. And one of the things that I find fascinating is to see how people's answers have changed over time. Uh, we'll, we'll even have some people on the show that we've had on years ago, like in some cases, somebody we had on a decade ago and you know, they used to be a big emax fan and now they're like, ah, I pretty much use VS. Code all the time. Milo: Yeah. It's Jonathan: funny. Milo: Well, there, it's like, sometimes it's like that curve, right? Where you're like, on one end you're using. Like VS. Code and then you're in the middle and you're trying to like Z and all these different projects and you just go back to VS. Code because it's, it's like the best overall. Jonathan: It's the one that works. Milo: Solution least amount of headaches kind of thing. Jonathan: Yeah, absolutely. Milo: Yeah, Jonathan: time. Time is a flat circle man. Milo: Exactly. Jonathan: Alright, thank you so much for being here. It has been a blast to, to get to talk with Milo and talk about Pangolin and Faial, the I, ot, ot, and just all kinds of fun stuff. A lot of stuff from my background and things I'm working on now too. So I thank you very much. I appreciate it. Milo: Yeah. Thanks for having me. Jonathan: All right. Uh, we do plan on having a show next week. I don't think we have a guest scheduled yet, although I've got some business cards from Embedded World. I'm gonna start sending emails out to hopefully later today. So plan on being back next week. If you run a project, you know, somebody that does let us know so that we can give 'em an interview, uh, floss@hackaday.com or come jump into the Discord, let us know, get us in touch. We will get you on the schedule and chat about your project. Um, if you want to find more of me, there is of course, the Untitled Linux Show over at twit and you can come check out what we're doing at Mesh Tastic. Other than that, thank you. Appreciate everybody that watches and listens whether you get us live or on the download, and we'll be back next week on Floss Weekly.
• Episode 867 - Pangolin: People Can Lie 25.03.2026 1val 3min

  This week Jonathan chats with Milo Schwartz about Pangolin, the Open Source tunneling solution. Why do we need something other than Wireguard, and how does Pangolin fix IoT and IT problems? And most importantly, how do you run your own self-hosted Pangolin install? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• Episode 866 - BreezyBox and Embedded Compilers 18.03.2026 47min

  This week Jonathan chats with Valentyn Danylchuk about BreezyBox. That's the ESP32 shell and toolkit that gives you a console and compiler right on an ESP32 device. What was the inspiration for this impressive project? And what direction is it heading? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• Episode 865 transcript 04.03.2026

  FLOSS-865 Jonathan: Hey folks. This week I talked with Philip Humo about Crowds sec. That is an open source security company and their flagship product is a web application firewall that puts the open source into it in a couple of different ways. It's really fascinating and you don't wanna miss it. Since stay tuned, this is Floss Weekly episode 865, recorded Tuesday, March the third, multiplayer firewall. It is time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and today we're talking well open source. Of course, we're also talking security and probably ai. I know our audience, some of our audience. Sick of talking about ai and I am too. But at the same time, it's the world that we live in right now. It is the bubble. So we've gotta do some coverage of it. Today I'm talking with Philip Huo up the guy behind Crowds sec and a bunch of other things he just told me in the pre-brief that he has literally written the book or a book on ai. Doing offensive security stuff. And these are all things that I am super interested in, care very deeply about and I know a lot of our listeners are too. Without any further ado I'm gonna bring pen, bring Philip on, and we are going to dive into it. Philip, welcome to the show. Philippe: Hi, Jonathan. Thank you for having me. Jonathan: Yeah, it is great. It is great to have you here. And it's it's really interesting to look at some of the things that you've had your fingers in. You've been doing cyber cybersecurity since 1999, which is a long time now. Yeah. Don't think too hard about how many years that's been, but it's been a long time. And you've got a note here that you also like to crack business models on top of security. I'm fascinated to to hear about that. But let's start with a background. How did you get into. All of this what was your introduction to open source and cybersecurity? How did those things come together for you? Philippe: Yeah, so the inception moment for cybersecurity was when I was in my engineering school. So I met a guy and he told me his name on screen was JDI Sector one. And I was like, wow, okay. Wait. The name rings a bell. What I've, why does it he says I've been cracking games on Atari St. When I was a kid. Wait, I was playing to those games. But you are my age. So when I was playing and I was 11, you were, I know. It is I'm one year younger than you are. I was 10. So you were cracking games being 10 yeah. Okay. I wanna do what you're doing now. Now I'm interesting. Show me the rabbit hole. Jonathan: Yeah. Philippe: And then this guy introduced me to security or what it was back in the day. So we used it for stupid stuff, dredging on girls and trying to find the name of this beautiful lady we saw in a party, whatever. Nothing bad really, but also there were no law framework around it. So with a bit of crap, nothing reprehensible, nothing that could get you in a court nowadays, or probably everything could get you in a court nowadays. But what I mean by that is it was just a free space and we enjoyed the time and we hone our skills and then I became a pentest quite obviously. Jonathan: Yeah. And then when did you make the connection with that and open source? Because I know that this is part of your, this is part of your background and the two have some natural overlap, but it's not something that everyone thinks about. I'm curious where this connection came. Philippe: Yeah, so it, it dates back from the days where we wanted to have the proper waf. And we found known, and with my CTO Tebow we're like, okay, that, let's develop one ourselves. So we used nix as a base and we developed what's, what was called back in the days xi which stands for NIX and TX and a SQL injection. And it used we used it a lot and it was very robust, very efficient. And we were like, okay, you know what? It's just a tool for us. It's not about making money about the tool, it's about giving the tool to the community so they can contribute rules. Because the problem with the WAF is not really writing the engine. The problem with the WAF is having the rules and updating them consistently. So in many way, open source is helping you by having a community. Now, what I tell to other fellow members of either business or cybersecurity or fast is you have probably 10 reason, which would be bad reason to go the false way. And one or two that would be the right reason to go the fast way. So think twice before building a business of a fast for your personal interest. Fine. Do whatever you want. For a business, think a lot about what you're doing because it's not so of use and what it is. It is. Jonathan: So this is a tangent, but what's really interesting to think about there is you have a lot of these businesses and open source folks that come at this from the opposite direction, and that is, they start with the open source project. And I fall into this boat too with some of the things I do. You start with the open source project people love it. They make a GPL, whatever license, and then suddenly you realize. Yes, I am spending the equivalent of a full-time job trying to keep up with this. I've gotta make money at it somehow. And so some of the place people get themselves in trouble is they then take their open source project, try to make it fit into some kind of a commercial commercial arrangement. And, people run into problems every 'cause. You have to do that very carefully so you don't destroy the open source project trying to make, commercialize it. Philippe: Yeah, so it's exactly the problem we've been facing, and it's exactly why this was the hardest business model ever to crack for me, because we knew about those hiccups and those problems early in the days already in the inception of crowds. Sec, Elasticsearch already had the problem with Amazon, and people either rigging you out of your value. Or not contributing, abusing, or you have to try to fit a model where you make some things premium, but it's just features you've developed and you try to hide. Jonathan: Yeah. Philippe: And the committee tries to replicate it because it's a useful feature. So then you get in a argument with your community. All of these are bad models, period. Yeah. The way we want, we saw it in crowds Sec, was, okay, look, we reverse engineered this. The base of the company is to have signals data about what's happening over the world map, what IP are used by cyber criminals, where, when, and how. So that was the end game. And this is something we can monetize. Now, how to do that, we need to provide a software that would be useful to the masses. And that the masses would be able to contribute to, because we don't have connections to all kind of systems. If you wanna pass as 400 logs from IB. I don't have an S 400, but maybe someone in the bank wants to do the scenario and the passers and the rules around it, and that's fine. We just need to host this. So the tool is a mean to an end. We develop proper grade open source software with 18 million euros. In funding, we give it for free. In return, people are using it and we capture signals. We give back signals to the community so that everyone defend themselves, but also we sell those signals under the guise of block lace and CTI data to businesses, to banks, to large corporation, governments, cybersecurity companies and all that jazz. Jonathan: Yeah. So when you said waf, that's web application firewall. Give it, give us the quick rundown on what a WAF is. Philippe: Yeah, so a waf, a KAA web nowadays is web application firewall, web API firewall, whatever you wanna name it. So the long story short is we used to pass logs back in the days is what called IDS Intru Detection System. Something would appear in the logs and you say, Hey. Probably Jonathan didn't forget his password 10 times in a row, or he would've pressed, I forgot my password button eventually. Except if he's dead drunk, but he usually runs on coffee, so it's unlikely. All of these he knows me, a rule like, yeah, if Jonathan misses his password 10 times in a row, there's something wrong about it. So that would be brute force, for example. Or if we see Jonathan email tried in several different places, that would be credential stuffing, stuff like that. So this is visible in the locks now. Sometimes when you interact with a web application, you don't see all those parameters. You don't see log splittings, anything. It just happens. So to land into the database directly or into the user interface directly, and then you blind to whatever attack is sent. So you need to passe the URL, the parameters that are sent during the get and the post action to have a kind of granularity about what's happening. And that's exactly what WAF is doing. It's analyzing the parameters of get posts in HTP queries. Jonathan: That usually involves peeling off the H-T-D-P-S layer, doesn't it? Philippe: Yeah, at some point you supposedly are behind it already. You supposedly are filtering it when the web server is receiving the transaction, not be above that because otherwise you would be blind to everything basically. So WAF tend to either do many in the middle by themselves. By having, the proper certificates to peer into what's happening, or you just deal with this on the server side at the htp at the, sorry, at the application layer so that you see what's transacted. Jonathan: Got it. Got it. Is crowd, let's see what I'm trying to figure out the naming here. Is Crowds Sex, the name of the waf. Philippe: So Crowds SEC is the name of the company. We're an open source editor and the business model is as follow. So we provide this ideas and WAF security engine for free to the masses. Today, probably 200,000 users and 400,000 machines are using crowds sec. Either as an e, d IDS or reading logs, or either as a WAF reading HTP streams or both. And they protect themselves using these two for free. The scenarios are for free. The virtual patch at CB detection, all the rules, everything is for free. You can use it, download it, install it. We don't have any any crew in it. And that's fine. Every time your machine is blocking an attack, this attack is shared with us, with our API endpoint. You can disability it if you don't want an telemetry, but you get in return also the ips that are doing the same stuff somewhere else. So say you are a hospital. Hopkins, right? I think they're using it, but I'm not sure. There are a lot of hospitals using crowd, but, so say a hospital in breast in France is being attacked. It's blocked, but all other hospitals all around the world using crowds will get that. IP. Attack this hospital. Jonathan: Oh, Philippe: interesting. Not the name of the hospital obviously, but attacked the hospital. And they should also protect themselves, not only against a bad behavior, but also against a bad reputation of this IP address that is particularly nasty against hospitals. That's just an example. But you see the point, Jonathan: it's it's like the shared spam spam Philippe: list. Jonathan: Yep. Philippe: Yep. It's a bit of a spam house, but directed toward HD towards servers that are not specifically mail servers. Right. You can think about it as a ways of firewalls, like a giant multiplayer firewall where when you see an IP address. Multiplayer Jonathan: firewall. Philippe: We call them Pokemons, by the way, internet. We don't have the right to do because obviously an nten would be pissed, but we call them Pokemons these IP addresses because we have cars about them, name, which is the IP address, what they do in life, what is the superpower, what is the threat they are exposing and so forth. Jonathan: Huh? This is very cool. One of the, one of the other things that I don't know that you guys are in the right place to do this, but when I first heard about this I thought about the internet back scatter detection. I've heard people call that like a, an internet telescope. Do you get to do any of that? And that is essentially, that's where someone makes a TCP connection with a random. TCP or UDP with a random source address. And so then you hear the cac, I believe, if I remember that correctly. Is it the A or the cac, one or the other? The second part of the TCP handshake or the UDP response comes back to this random address. If it happens to be a security guy like Philippe that owns that address, he can then log that packet that comes in and you find out some information about it. Do you guys get to do that as well? Philippe: No. Because mostly, yeah, so I see what you mean. Merit did the oil oil project. Yeah. So practice scanning. Is not something we allow ourself to do for many reasons. But long story short, it could be illegal in certain regions. And it's not bringing a lot of qualified information for us because what we're interested is not so much what is about the internet. As to which IP are used by cyber criminals to do what at a given moment in time. And this map is a real time map. It's changing consistently. And they don't host in services. So if we would ping back at them, the only thing we would know is that eventually they host a C two or something like this. Jonathan: Yeah, Philippe: but it's none of our concern in the sense that we are dealing with ingress ips, not egress ips. Just the ity for the listeners is that egress ips are the one you are going toward. So usually they're associated with the domain name because humans are very bad. I remembering numbers. So this is where you would find potentially phishing pages and stuff like that. We deal with. Ingress ip, so ips that are coming from the internet towards your services. And most of times, 90% or 80% of times the guys have proper SecOps. So separate their egress ips from their ingress ip. So what we are capturing are ingress ips, and in this direction doing proactive fingerprinting would not bring much to the product. Jonathan: Yeah. So you're not looking for the. Windows, 2000 servers that got infected with Code Red and are still out there. You're looking for, you're looking for IP addresses that are assigned to, oh, what's the term they use? Bulletproof servers and tour exit nodes. That's the sort of thing you guys are looking for. Philippe: Yeah. All of this interests us. For example, which IP are used to do scanning? Injecting credential or brute forcing or exploiting cvs or trying to temper with your video ip, by the way, it's crazy. I've never imagined that it would be so much all of these. And or DDoSing on the layer seven, everything that goes in ingress. So maybe your old school Windows server is still a nest for people to use them as attacking machine. But most of times the ones that are very dangerous are the ones that are very specialized and that are very recent. Jonathan: Yeah. So you guys have a lot of information about things like botnet size. Do you have some to insight into things like that? Do you get to watch botnet? I'm reminded of the XKCD comic where it talks about where someone made a, an aquarium out of viruses running in virtual machines and watching them grow. You guys get to do that with the internet. You'll get to watch these various botnets grow and start making attacks. Philippe: Yeah. So sometimes, yeah, we see some of them being exploited in the white. Like the tic one was very famous because it took down a lot of targets. So we saw the exploitation spree and then, all those machine being used later on. What we know about this layer seven DDoS is they have two type of of two flavors, let's say. So one is an orchestrated one like DDoS and this kind of services where people have a master node that is sending them a j query. A Jason package. Where they will find which target to to attack, what request to make so that it hurts the performances of the machine on the other end the most. And then all of those machines that are. Voluntarily botnets, voluntary botnets, sorry. Answer the call and do this request one after the other. And they're the more private ones. So recently exploited, usually at scale think rows or cameras or IOT devices in a thing like this. And then all of a sudden they raise and they target someone, something in CloudFlare cease. 10 gigabyte, 10 terabytes of bandwidth. Jonathan: One of the, one of the notes that you have in the prep work here is how is it that that crowds SEC can compete with CloudFlare? Seeing one third of the internet traffic, CloudFlare is just so huge. They are the, the 800 pound gorilla in this space. What's the secret? What's the special sauce that allows you guys to even be in the same conversation? Philippe: Yeah, so good one. First of all, it's super easy to pinpoint Cloudflare's. So if you want to dodge them, you dodge them. So if you have a super secret CV you wanna use, you'd rather not use it on Akamai or CloudFlare because you're really likely to be bused super quickly. The second thing is CloudFlare, Microsoft, Google, whomever, they offer extremely low diversity. Think about it, that you know exactly the address where they live, and they all live in the same country. To put it short on the internet. Literation now Crowd SEC is spread across hundred 80 countries and 6,000 autonomous system. Microsoft is one autonomous system. So if they would wanna have such a diversity, they would have to go in so many countries, in so many different clouds and industrialize the deployment of so many machines to listen to the internet. They would be nearly unfeasible. This is where it ties nicely with open source because. You're incentivized to use the software and in return you're meshing the network. And I know you know a lot about meshing, so this is what we do. We are meshed a lot more than CloudFlare. Jonathan: Yeah. Interesting. That, that is it's super interesting to learn about crowds sec. And we, I feel like we could go down that rabbit hole deeper, but I actually wanna ask you about some other things because this is, again, this is something I care a lot about and it's the. The realm that you work in all the time. When it comes to security, and particularly in open source, I'm just curious what you think. How are we doing? Are we okay? Are we making progress or are we losing ground on the security war? What's your feeling on this? Philippe: No, I think we are winning this battle in the open source. Open source, did they have a lifting of pretty much any other product in the world? If you look at Oracle, most of the product will have a license where at some point they say, Hey, we have open source somewhere who honors. He's gonna develop another open ssl. You were speaking about xkcd cartoons. I love them as well. Yeah. Look at the one speaking about this flow of open ssl. It's three guys incon or wherever they are. Developing this on their own time. They did stellar job for decades. They didn't ask for anything. Everybody is using it. If two more you would be coding an open Thel library, like you would probably have tens of bugs that they to deal with already in the past. And by the way, when they send rovers on Mars, they don't run it on windows. Sorry to say. So you know Indeed. Yes. Of course we dominate this space. Now the problem is people putting proprietary interfaces above really good open source code. Now, think about let's not name them Fortinet. Oh, sorry. Oh, not politically correct. My bad. Can you scratch that? Yeah. It's not live, huh? Yeah. So they take a very good piece of kernel, right? Which is safe, stable, ruthlessly efficient, and so on. And then they put a nice GII above, which gets. A security issue every other day. Yeah. So yeah, let's not ruin the efforts by having some commercial product above that is seemingly rendering a service, but actually creating holes. Because nowadays your biggest risk in any security environment's, probably a firewall, Jonathan: goodness. That's probably true, but goodness. I enjoy reading the Watchtower blog posts about the various things that they find. Yeah, me too. In endlessly entertaining. Yeah. Philippe: Yeah, them and yeah, we are, we're quite buddies. We know each other and we exchange data and we discuss. Yeah. Yeah. Watchtower is super cool. I love the fact that they release on Friday so that the people are like, oh no. Oh no. Don't do that. Jonathan: Not on Friday. That's great. Okay. I'm sorry everybody, but I've gotta ask how is AI changing the game? Philippe: Yeah. I thought as well that it was a flu or people were over, reacting or whatever. It's not, it's definitely's not, yeah. Think about it like this and it's very important, very pragmatic. Take a decent LLM, make it agent. Tell it to scan all the surface of, I dunno, Siemens, four oh thousand employees, whatever. And try. With RAG to enrich these data and say, Hey, every employee you find their social networks and scan their social networks and find the one that are disgruntled, the one that are upset at the company. Okay. You fund what? Out of a million employee, you find what? A hundred. Maybe well make an offer to them for 0.5 Bitcoin, gimme your credentials. All right. That's just a very. Very super strictly simple way of using an Netherland m. You and I and everyone else know that it'll work. Not only it'll work, it'll be goddamn efficiently. Yeah. So in 15 seconds, I can turn my voice, change my appearance, be credible, wave my for my face, and so on. I have sound like a woman, like a Asian person or whatever. So yes, it changed everything. Like at President Attack, it used to be ridiculous. Nowadays it's super dangerous again. Yes. When they try to exploit vulnerability automatically, should I say, they go extremely fast. They find zero days. And when you have a scanner, a stupid scanner, it'll make every brute force at 10 point by one 2000 try to eventually find something. A human will narrow down to 50 trials before finding something. And you look at an AI doing this and you don't know how things and goes here. And in five. Attempts you met and you're like, what the hell happened? Remember a second? So yes this is not bullshit. This is not overstated. This is a real current, extremely dangerous situation. And it's just the very beginning because states are harnessing and training those models like we saw with the US DOD lately, having this fault fraud with tropic. Jonathan: So that, that's interesting. This is a different take than a lot of people are publicly talking about. I hear people talking about LLMs, the security in the aspect of, we're gonna, we're gonna get the AI to find the CVEs and then go out and automatically exploit it. And I've followed for a while now the various research groups, some of them have had success with that, but it is. A lot more complicated than just pulling up Claude Bot and saying, find me a CVE in this code. Philippe: I can give you a recipe if you want. A simple one. Alright, so let's say you host a platform of capture the flag. So capture the flag for the alliance is a game for rat teamers to try and test their skills. And so you get some vulnerable, but pretty well hidden servers that, that have some vulnerability, but sometimes it's very delicate to find and trigger, right? And those teams are competing against each other. What do they do? They have logs of what the best hackers on earth have been doing to breach into those servers. Now feed those logs into an AI to teach them how to exploit vulnerabilities. It'll learn from the best, except it'll execute in seconds. Now this is realistic. Now this is being done as you speak. Jonathan: Have you Philippe: sorry. Jonathan: No, that's fine. I'm. Thinking of a different tangent here. I assume you followed Daniel Stenberg at Curl and his rantings on some of these topics as well. Dan Daniel Stenberg, we've interviewed him on this show. He's a great guy. He is the, I think the lead architect behind Curl and he made the brilliant decision to put his email address into the Crow license. And so he. Gets a lot of fun emails. Philippe: Ghetto attention. Yeah. Jonathan: Yes, it's great. But they ran, in fact they were one of the early open source projects, I think to really have a good bug bounty. And they've run it for years and years, and they have the problem that their noise to signal and noise ratio has just gotten terrible on that incoming bug bounty because of LLMs, because it's so easy to say, write me what looks like a good CVE, the person. Reporting. It doesn't understand what they're doing. They send it in. The LLM does a good enough job at making it look reasonable that somebody has to sit there and read through it for sometimes an hour to figure out, no, this is just garbage. And he's actually, they curl has closed down their Bug bounty program as a result. And I, I've heard some people that. Make a good argument. Basically say this is AI is probably going to kill bug bounties overall. I'm curious, do you have a thought on that? Philippe: Yeah we oriented our decision in the other way. We're using a famous bug bounty platform that everyone knows about around the world and we want them because it's human based and specifically when. This bug monthly program is open, and if you report a bug, you have to go through this platform because we know that they're not gonna allow stupid mindless LM subscription and and, posting. Yeah. Jonathan: You guys are using a project where it kicks out and doesn't allow people to use the AI slop stuff. Philippe: Yes. Jonathan: How, as AI gets better, how do we continue to stay ahead of that detecting what is. AI garbage. And what is real vulnerabilities? Philippe: We don't, I don't think it's a battle or we can win. Jonathan: Okay. So does that, we don't, that kind of implies that this is going to kill bug bounties. Philippe: Yes, somewhat, yes. And why not after all? But the problem will be the way we perceive it is a following. All those AI so far, they don't have their they don't have limbs, they don't have legs, they don't have hands and they're limited to a virtual machine somewhere over the internet. So even though cloud bots and all of those have human rental body programs and so on, I don't think it's gonna go further than that. It's extremely funny, don't get me wrong. But it, they will have to interact through digital highways, so through IP addresses. That's why I think IP addresses are still and will the most important, intelligence and what you want to accept and what you don't wanna accept. Jonathan: How big of a problem is I PV six for you guys? Philippe: Mostly not but it's non zero. Okay. Let get two things straight. U DP is a problem because obviously you dunno, don't know who the sender, so we cannot rely on udp spotting. Period. IPV six is another problem entirely because there an infin amount of IP instead looking at three something billion ip, we're looking at trillion. The way we see it is that. Contrary to IPV four, this was forged in the proper way, this more or less properly engineered. So the ipv six blocks are attributed to very specific operators, and they're tagged. So we know whose operator it belongs to pretty precisely. And it's then fragmented so that the users have their personal network that is pretty large, but it's still one user. So we can ban the whole slash the whole subnet. We don't have to go like one by one ip. By the way. Sometimes when we see, I dunno, 20 ips being rogue in a slash 24 IP four address space, we can block the whole 24 less, because it's unlikely that. 220 other are good guys. So yes, we treated by blocks in this case, Jonathan: I suppose. I suppose that logic works in IPV six, just like it does IPV four. You're just talking about bigger sparser blocks. Philippe: Yes. And also remember that if you burn your IPV V six, it's a big problem because you have to change and get another chunk. And that's not so easy because then your operator can get notified and ask why and how and so on. Again, they are pretty well tagged. IPV four is much more of a far west space where something is somewhere we don't know exactly. Transit 10 times, and changing hands every other day. By the way, the database of ip, we, we deal with both, right? But we have probably 5% of ips that are IPV six compared to 95% that were ipv four. And as you highlighted, I finished my engineering school in 1999, which is quite a while ago. And we were told already back then that we should get ready for IPV six. It's coming. It's coming tomorrow. Yeah, sure. 30 years later. It's not, but hey, no, it's really coming because fair is fair. The 5G network is changing everything because it's it's a real IPV six network. C Gnet is reaching an end in so many ways. But the problem we're seeing more than IPV six is residential proxies. Those are pest, those are real pests. Jonathan: Yeah. Philippe: And people are doing crap with that. Jonathan: Yeah that residential proxies, that's where essentially somebody's desktop is part of a botnet. Philippe: Yeah, they ran their, so what they try to do, users is trying to offset the cost of their ISP by renting their landline to anyone willing to pay for it. On principle smart, the problem is the following. And I got a very precise example during the 7th of October, attacks on Israel. The, there were DDoS. To shut down the siren network that is alerting and the mobile alerts that are alerting civilians to rush toward shelters because missiles are gonna rain on the country, right? So this is a system that is forever in place in Israel because they are pretty used to get rockets, whatever. I'm not in politics here. I'm just stating the facts, right? Not saying good, bad, whatever. Yeah. When they get a DDoS from external sources, they ask, what did we know about the IP addresses that were doing this? And we return a few slices of ips because we want the civilians, whatever the politics of their government, we want civilians to be protected. So it was a legitimate case. Okay. So we gave them a bit of addresses and they were like no your source is inaccurate. Sorry, what? We have zero force for six years running. Tell me what's inaccurate. A lot of those IP addresses are actually a Israelis and obviously a Israelis would never do a DDoS on a, yeah, sure. Have you ever heard about residential proxies? No. Let me enlighten you. And then all of a sudden they realize that whomever is behind those attacks, and I'm not gonna state any name, whomever was behind the attack, were using residential proxies that were located in Israel. Guess why? Because it's than. And that's just one case. Residential proxies are PEs because blocking a IP of a person is more complicated than blocking an IP from a data center wrench or a c gnet wrench for plenty of reasons. But in Europe, mostly because it's considered a private data, a private information about yourself. Jonathan: That is a pain for you guys, isn't it? That IP Philippe: addresses there? Yes. Yes. Jonathan: I, Philippe: yes. Jonathan: Yeah I think we've talked about this here on the show, but one of my longstanding annoyances is when politicians that have no clue what they're talking about, go out and write laws about things. And we have this problem in the United States. Europe seems to have this problem in spades as well, and I. I am thankful for guys like Simon Phipps, who, I don't agree with him politically on everything, but one of the things that he tries very hard to do is to go out and actually help those politicians that are writing those laws have a bit of a clue. And it's a real problem because you end up with things like this. IP addresses are considered personal data, and therefore, trying to solve any of these problems becomes exponentially more difficult. Philippe: Yeah, we found a middle ground with the authorities, which is acceptable because their point is not to forbid entirely the use of this long story short, GDPR states that you should keep the minimum amount of data on the minimum amount of time for precise reason, blah, blah, blah, blah. And give the person an opportunity to correct this information. So what we did is we said, okay, look, we are supposedly entitled to keep it for a year because we are a service security company doing this for the greater good. But what we'll do is we'll keep it only for three months. And we'll blur the IP address and the timestamp because it's a combination of both. That is considered a problem for the GDPR commission. So what we said is okay, we'll blur the range, so the IP in range, so say it's an IP address and we have we'll say, hey, between zero and 2 55 in this range, in this last 24, we have an IP address that has been aggressive. After three months, we'll blur in this range, and it was aggressive at 1234 and 56 seconds. Okay? It was aggressive between 12 and 1:00 PM and that's enough by blurring those two dimensions to make it an anonymized IP address, and it's no longer a personal data. Jonathan: That's handy. I'm, I am glad to hear that you were able to make that work. I've been in projects and it's hilarious to me. So you mentioned the requirement that people can go in and correct this, and one of the things is people can go in and delete their own data. The thing that gets me is. Sometimes that requirement that people be able to delete their own data forces the company behind it to collect more data about the person to be able to handle the deletion request. Philippe: True. That's true. Jonathan: So it's like we over in Meh Tastic land, we run an MQTT server. We don't want to collect any of that data. We just want to take it and hand it back off to the network where someone is trying to use this public MQTT server. But because technically things go into the logs, we now have this. Potential burden where no, you've gotta have an account for everyone so that they can log in and ask you to delete their data. And it's but we don't keep the, it's such Philippe: a, another thing that is funny is technically a cybercriminal can come to our website and remove these IP address from the block list. Actually we have to. But we found a way around it, which is increasing penalty. So you've been in the block list and you say it's not correct. And okay, fine. Show us it's not correct. Okay. Eventually you find proof, which is hard, but let's say, and we remove you from the block list. If your IP is back, instead of being there for hours, you'll be there for four days. And then four weeks. And the four month is before you can remove yourself. So yes, you can remove yourself once, but you won't be able to remove yourself every time you want. So that's all you know. You have to find the middle ground, and if ever the authorities come to us and say, Hey. It's not the spirit of the law. Yeah. Guess what? The guys on the other end are not exactly in the spirit of the law either. Jonathan: Yeah. It's important that people can remove themselves because we run servers that get popped sometimes and you don't want your IP address. I've had this happen. I run a server, I've had this happen where my WordPress site got a malicious let's see, what was it? It was a PHP script that got dropped because there's a vulnerability in one of the plugins years ago, and then somebody started sending emails, spam emails out from it. And the next thing you know, you get the ding that, hey, you are on the spam house block list. You gotta go clean it up first and then go make the request to spam house to get off of it. It's the same thing with this stuff you can get, you can. Part of, become part of a botnet, unintentionally. You figure it out, you, you wipe it off your server and, but you still have your IP addresses. You'd really like to be able to use those IP addresses still. Philippe: What we do in this matter, because it's an interesting point you're making here. What we do in this matter is we have a forget policy, if the network, what we call the network pressure because you are not reported anymore by people around the world, we consider that this IP is clean, at least for now. So we move it from the block list. Now, if your IP is seen again, you get a fast track ticket back into the block list because you've been. Seen before because we want to avoid the people that go on off, on, off with their IP addresses just to, cool down. What we want though is also to forget people and forgive people that had several compromised once and they don't want to end up into an block is forever. A lot of services are doing this online and it's not something you can really. Use as a defense mechanism because some IP addresses in some databases are in age of driving a car and drinking booze. They are 21-year-old and what's the point? And as well, if you give me a China Telecom is scanning the internet. Yes bro. For the last 40 years consistently. What is the value of that? We know it it's zero value. What I want to know is like what happened the last five minutes? When the Try, try, guys tried to export, react to Shell, that is of value. Jonathan: Yeah. Do are there any IP addresses that have like special notes beside them? One of the, you mentioned scanning the internet and I can't help but think of Nate Graham who talks about, doing the entire IPV four block. So do you have a post-it note, the equivalent of post-it note in your database, like this IP address is Nate Graham, he's fine. Or. A little bit more seriously, this block of IP addresses is Amazon AWS. We don't really wanna block those. Philippe: Okay. So on the first point, we have a fun, something inside. We call it, we call them Christmas trees because, they're scanning for everything constantly and so on forever. So they are Christmas trees because the webpage describing this Pokemon is just three page long, but just for the cvs, they're for. As for the second part of the question. We, AWS in particular is really clean about that because when they see that any IP is doing crap, it's taken down 24 to 48 hours after they're the fastest. Google comes second with 72 hours and Azure five days, something like this. So those guys are really serious about, taking down the IP addresses that are dangerous. Now, do we block them? Yes. Because they still deserve to be blocked, and our client don't care where the men is coming from. They wanna protect themselves. So it would be too easy for someone to, pop a credit card SPO a hundred AWS instances and start doing crap. No, we block them. But to be honest usually 48 hours after they are, they're taken down by AWS itself. Jonathan: Does AWS detect that stuff internally or do you guys send over a report to 'em? Philippe: No. They do it by themselves. We have some collaboration with some mssp, some cloud and so on, and we're getting closer to AWS and Azure and others, but honestly, they did it by themselves already way before us. Jonathan: That makes sense for their own internal business. Like Amazon really does not wanna be known as the closest thing to a Bulletproof server in the us. That would just not be a good market position for them. Philippe: No. And some of them don't care. Look at Digital Ocean. Oh, sorry. Not particularly. Correct. Or vh. Oh, damn. I told the name. Sorry about that. You cut it, you, you know the procedure well, so those guys don't care. The literal policies, as long as someone is paying for the machine, we don't care. Jonathan: I, Philippe: I, I, and I had a discussion with policy, so what do you see for 30 days, three weeks? Three. Whatever. But their, I as number autonomous system, which is long story short, it's the, it's the big iron. It's big iron Jonathan: routers. Yeah. Philippe: Yeah. It's in the group of ip. They are dedicated they are known to be dangerous. So they are blacklisted in many places. And AWS as you and the others, they don't want that, obviously. Jonathan: Yeah. Interesting. That the whole. The whole bit about not what you guys do, but the broader thought about IP address and banning them. And so I, I had to move servers not terribly long ago. Got had to get a new block of IP addresses from the new host and, the block of IP addresses they gave me was banned in a bunch of places. It's let's try to figure out why. And I actually had a conversation with somebody at the Department of Defense is where I ended up. I made phone calls and asked questions and this is where I ended up. And he goes lemme look. And he goes, are you out of Iran? No, I'm in the middle of the United States. Your IP addresses say that you're out of Iran. A, it was a block of IP addresses that had originally been assigned to some company in Iran. And they were given to me and I had to get ahold of my provider and be like, this is not gonna work. Please gimme a different IP addresses. Philippe: No, that's not cool. That's really not cool. So the first thing is. What it tells us is you have to be a dynamic system. You cannot have a map and get this map and think that it'll be true forever. The rotation rate that we see in our own data lake is around 6%. So the ips are, were nefarious yesterday. 6% of them will be clear tomorrow and 6% of them will be new tomorrow. So yeah, it's a very dynamic landscape and you're right to highlight that. The second thing is if you want to help the listeners on craft sec net, you can look for an IP address and check if this IP address is known to be malicious already. So if you get a new IP address before signing in, check that the IP address is clean, Jonathan: yeah. Very cool. Philippe: It's free. Jonathan: Yeah it's funny too, things like IP address geolocation. Used to be, it used to be fairly fairly easy and pretty reliable that, oh, hey look, this IP address, it's probably right here. But because, and I think a big part of the reason for this is the IPV four exhaustion, and now we've got so many of these that are just in churn or companies are releasing them back to the is it the IETF? Whichever the, or the internet standards and numbering group, whichever group it is that controls the ipv four and they'll hand them back out. And so now you know, you get an IP address. It says on the ticket that it's from Iran and no, it's a server in Dallas, we promise. And it happens all the time. Philippe: Yeah. And it makes no sense. And to be honest about this geolocation stuff, I had even another kind of conversations with some clients and were like, we give them the targeted country. Because we know what business is sitting where, because in their own interest, they tell us, Hey, we are based in Germany and we are pharmaceutical industry, whatever. So with graph ai, we can give them more IP addresses to block and to protect themselves back. So it's in their own interest to tell the truth. So we can tell this IP address is attracted toward Germany, Belgium, France, whatever, United States and so on, because most of its time it's attacking these countries. And then they come up with another question, where is it coming from? I have no clue, dude. I really literally have no clue. To a dice, look at the result, multiply it by 10 and see if Mars and the moon are aligned. And maybe it's coming from Russia. What do I know? And what do you wanna know? Oh, if Russia is attacking us. Yeah, among others, and maybe the guy is just bouncing off Russia, but coming from wherever else. Australia, what do you know? It's like the people doing attribution. The attribution game is so weird to me. Oh, those, this is a group APT 38. Yeah, sure bro. You do this because you wanna sell or pedal your plush choice during the black hat. But the reality when you discuss with the guys is they have no freaking clue. Jonathan: Yeah. Philippe: Because it looks or those IP were seen before you being used by the same group. The guys have alliances. They subcontract, they ran, they change groups from one day to another. It's a dynamic landscape as well. So what you thought could be apt 38 could be a teenager in Great Britain piece with his teacher. We have no freaking clue. And by the way, because you know it's coming from AP 38, you're gonna react differently. No, not really. You're gonna block the attack because it's not because it's coming from North Korea or from a teenager in England that you're gonna let it go through. It makes no sense. Jonathan: Yes. Philippe: Sorry. No, Jonathan: no. Philippe: Don't get me started on that. Jonathan: No, that's fine. I've, I have I've long had a undercurrent of skepticism about the attribution thing. Philippe: Yeah, you're right too, Jonathan: because there at some level. I'm trying to remember one of the. One of the examples and again without getting overly political, Country One has invaded Country two, and country two just had this attack and all the ips came from Russia. It's but Country three could have done that very trivially and it would've made a lot of sense for like this. The guys at the NSA are smart. They no xz. So the XZ thing people were all trying to figure out, the, jing ta, or whatever the dude's name was the imaginary developer that, that implanted the bug. People were all trying to figure out like, what country was he from? Was he from Russia, was he from North Korea? And there's a whole bunch of different signs that people were looking at and they didn't all point the same direction. But the one that I found the most interesting was when you looked at like when the commit timestamps Philippe: Yeah. Jonathan: Lined up. Yeah. It looked like somebody working a nine to five job on the US Eastern seaboard. I don't know. So like I could imagine the NSA doing that, but I could also imagine that the guys in the Kremlin going, Hey, you know what would be really funny is if we made all these commits. During nine to five on the sea, Jonathan: I dunno which one it was. I, I, Philippe: I have no Jonathan: idea. Philippe: There is a counterargument to this, which was given to me by a very smart guy at threat cautioned. So he told me, Philipp, if they are using a adversary as the main key in their thinking is because everything else is. Equally as inconsistent as this. So look, IP addresses are changing constantly. The commits are constantly changing the timings and stuff. The DNS are changing constantly. The code is changing constantly. So what they know or what they think they know is this kind of group is targeting specifically banks and insurance and cryptocurrencies, right? So it's really likely that if someone is working in this field, it's less a reset. So they take it the other way around. They say, okay, let's say it's less risk and let's try to have a cloud of information around what this adversary could be using. Which I understand. Fair is fair. I wouldn't take IP address as a primary key in my CTI either it doesn't make sense. It's not a better indication. Is more actionable. But it doesn't encompass who is the adversary. So yeah, it's a complicated field and I don't wanna put to cast a shadow on them. Those guys are doing serious job. We have friends at Mandi and I don't wanna piss them off, but yeah, it's a really complicated thing to say, Hey, these are Russian, this is this group, or whatever. Jonathan: Yeah. So on the flip side of that we've thrown a little bit of shade, but on the flip side, you do occasionally read the stories of guys getting arrested. We track them down through, and I don't even know all of the ways that these guys get tracked down. I'm sure some of these techniques probably are, secret squirrel stuff. But we found them, we tracked the Bitcoin. It ended up in this guy's wallet and we went to, he traveled through France and so we picked him up and we went to Ukraine and we arrested him. And so I good for them for actually being able to catch some of these ne do wells that have. Philippe: Absolutely done encryption wear on Jonathan: everybody's computers. Philippe: Absolutely. And I've seen also things that are unmistakably from one adversary because it's contextual. So for example, a lot of Ukrainian institutions are using our systems to protect themselves. Guess what? We can suspect that the people are taking their infrastructure are unlikely to be Spanish Jonathan: right Philippe: now. What do I care? It's not my problem per se. I'm here to protect their infrastructure whom whomever is attacking. And by the way, same applies for their addresses because in this country, there are also people that are not supportive of this war. Sure. So if they run a business and they're being attacked by the rest of the world because you, they want to take down this country, they also deserve protection. Yeah, it's a complicated field. It's an ever evolving field, and having certainty is the worst thing you can do. Having a very lively system a network effect that is capable of being very reactive on this threat is, I think, the right direction to go. Jonathan: Yeah. Alright. So if you look in your crystal ball. What what are you tracking for? What's coming next? We talked a little bit about ai, but aside from that, or maybe there's not, aside from that, what what are the things that you think is gonna really shake up the open source and security world over the next 24 months? Philippe: The open source world. What I see in this social network for ai, I find it fascinating. It's absolutely funny but also, eye-opening. I look at this with a lot of interest to be honest. Yeah. I don't know what's gonna emerge from that. But definitely something at very least a lesson for kind for the rest. I really know. Yeah. Plus dominating the world in cybersecurity because as it's, is the only serious way of getting these things done because we need to interact with each other. I think if I had to make one prediction specifically, I would say MCP is going to be a massive change. I really believe this, believe in this, and I've been discussing with the guys of Rat Canary, very smart people. And we have the same vision with our CTOs and all the community. You will get one LLM, which would be your natural ui, natural LAN language UI to your job. And then you will have a lot of CPS connected to various products that help you render the service you're supposed to render. And then you'll prompt it. And harmonize your work environment around one lm connected to several cps. Jonathan: Yeah. Interesting. I can definitely see people taking advantage of that. Alright. Is there anything I didn't ask you about that we should talk about? We should briefly cover. So Philippe: I, I have to get back Jonathan. Jonathan: It's been fun. We'll have you on again. Yeah, for sure. Philippe: Likewise. Jonathan: Yeah. So let me ask you, let me ask you this. I got a couple questions I will get, I get hate emails if I don't ask you this. What's your favorite text auditorium scripting language? Philippe: Oh, my text tutorial is definitely visual code. I've been a MX fan forever, but at some point visual code is so useful. Jonathan: Yeah. Yeah, it Philippe: really is. And the second one was what? Jonathan: Scripting language. Philippe: Ah, I would say Python is my go-to. I know it's doesn't sound so elaborate, but I love vibe coding in Python. I love to code in Python back in the days and vibe coding, it's super easy, super accurate. So yeah, when I did to do something in my home automation system, it's either this or. Jonathan: A security guy that likes vibe coding. Oh my goodness. Philippe: Yeah. Yeah. But I'm not the one coding the product. The guy coding the product are doing Golan. And from what they tell me is LMS are really not suited to do Golan yet. So they do everything by hand. Jonathan: Interesting. Yeah. Good to know. Alright, Philippe, thank you so much for being here. It has been a blast to get to chat with you. I've very much enjoyed it. And we will. I to have you back on here after a while. It's been a lot of fun. Philippe: Thank you, Jonathan. Have a good day. Yeah. Have yourself. Jonathan: Yep. Have a good evening. Appreciate it. Alright. That was Philippe, hu Huo. I believe is how his last name was said of crowds Sec, a really interesting company and open source project and idea around doing a web application firewall that then shares the, not the user's telemetry data, but the attacker's telemetry data back with crowds sec. And that is the thing that they have turned into a product, which is really a cool idea. I wish I had thought of that. I would be better off, I would be a wealthier man if I was the one that came up with that, because that's a really clever idea. Anyway. Next week we will not have a show because I will be at Embedded World. I will be looking for guests there though. So get ready for a lot of open source embedded projects on the show after that. But we will be back in a couple of weeks and we will see you then. We appreciate everybody that watches and listens whether they get us live or on the download. And we'll see you in two weeks on Floss Weekly.
• Episode 865 - Multiplayer Firewalls 04.03.2026 55min

  This week Jonathan chats with Philippe Humeau about Crowdsec! That company created a Web Application Firewall as on Open Source project, and now runs it as a Multiplayer Firewall. What does that mean, and how has it worked out as a business concept? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• Episode 864 transcript 11.02.2026

  FLOSS-864 Jonathan: This week I chat with William Sch, the author of the Lenox Command Line, a book all about how to get stuff done in Lenox, in the Shell. It's a lot of fun. You don't wanna miss it. So stay tuned. This is Floss Weekly, episode eight a hundred and sixty four, recorded Tuesday, February the 10th. Work hard, save money. Retire early. Hey folks, it's time for Floss Weekly. That's the show about Free Libre and open source software. I'm your host, Jonathan Bennett, and today we're gonna lean into that open source software, the Linux side of things, and the command line. And that's because we've got Bill, we've got Bill. Oh my goodness, what's his last name? Shots Bill Sch. That's right. I had the emails up, I have the email up, I have the bio up and it just has his first name of the email and it doesn't have his name of the top of the bio. That's all right. Bill Shots the author of the book, the Lennox Command Line, which was something of an online document first, and it's become a modern classic. It's been around for quite a while. We were comparing notes before the show started about which came first, whether it was Floss Weekly or the book. And it looks like Floss Weekly came before the internet edition. Of the Lennox command line, but it, it's it hearkens from the same era. And so looking very much forward to this conversation. Without any further ado, I will go ahead and bring it the man himself on Hayville. Welcome, welcome to the show. Bill: It's great to be here. Thank you. Jonathan: It's good to have you today. So the folks at No Star Press reached out to me and they said, Hey, we've got this book. Do you know about it? I said, I've not read that book, but I'm interested. And they told me what it's about. I said, yeah, that's definitely down, down my alley. I don't know how, I wasn't aware of the book. I've probably come across the document online in looking for things. But now we get to talk to the man. So tell us the story. Let's start at the beginning. At what point did you say, Hey, I'm going to write this huge document of command, essentially command line tips. What did, what was the origin story of that Bill: i've been a Linux user since 1996, and previous to that I had been using Mark Williams Coherent operating system, which was a a small model Unix that you could run on PCs. And by small model, it means that the largest program you could run was 60 4K. And I was truly impressed with what you could do in 60 4K. The I had Usenet news readers, I had email clients UUCP to move stuff back and forth. And I used this system for several years, starting in the, I guess in the late eighties. And the. When I discovered Linux I was really blown away about how much further along it was than coherent, and it was a lot slicker had a lot more features. It was a much richer environment, and of course it was free. The coherent was 99 bucks. And over the years, I've been mostly a Linux hobbyist, and though I did work in a Solaris shop as a leading a team of system administrators for a bunch of years. But I. After getting involved in this and using a lot of free and open source software for years and years, I felt like I wanted to give back. And so I created this website called Linux command.org, which I put up in 2000. And I wrote a series of tutorials dealing with mostly the ba shell and the canoe core utils. And then I had a second part of it where I talked shell script. And the site was, got to be fairly popular. And an editor from a textbook publisher reached out to me, I guess in, probably in 2006 or 2007, and wanted to know if I wanted to write a Shell scripting book. They had a series of books dealing with Linux, and she sent me one of the books in the series and I read through it and I was really unimpressed with their approach. It was very dry and it wasn't very, I didn't like the structure of it. So I wrote an outline for a book of my own, which expanded on what I had done in Linux command org. And set it off to them with proposal to do a book. And then I just never heard back from them. So I had the outline written and it was pretty extensive, and I sat down and just decided to write the book and just self-publish it and give it away. For two years roughly, I wrote a page a day until I had a book. The the book. Was out for, I guess it came out in 2009. And then in the fall of 2010, an editor from No Starch reached out to me and asked me if I wanted to commercially publish it. And after a long series of negotiations, over a publishing agreement I, I did a deal with them and they took the internet edition and converted it into a printed book. The part of my publishing agreement with no starch is that I get to continue making internet editions, and I've done seven of them so far. The seventh one is, comes out coinciding with the third print edition. The books are essentially the same with a couple of minor differences, but nothing very significant. They're the same book. And I have the no starch edition editions have sold about 150,000 copies, and my download count as of a few weeks ago was 1.7 million of the free ones. Jonathan: Wow, that's Bill: great. So I have a, roughly about a 10 to one ratio between, I give away 10 and sell one. And and it's been very successful. It's, if I have an info card on Google search now, and I'm I describe myself as slightly famous. Jonathan: I I do the same. I've almost a celebrity I think is what I've said a few times about myself. That's funny. That's impressive. Did you expect to sell that many? Like I, I wouldn't have thought that a book about Linux command line that you can get for free on the internet would sell that many. That's a surprise to me. Bill: A lot of people wanna a print a book and and I have to give credit to the no starch people, they really know how to do marketing. They're very good at it and they're very good publishers. I've liked working with them a lot. They've helped me a lot in making the book better. And they're really like, pleasant to work with and they are very helpful. Jonathan: I have found in my work at Hackaday that good editors are amazing to work with. And yeah it's really something. And if you've not done, if people out there listening have not done this, if they've not worked with an editor, it really is a good experience when you find a good one that'll work with you. Sometimes you have to, I don't know if you've seen this, writing these books I have with some of the columns that I've done over the years. Sometimes you have to leave your ego at the door just a little bit and be able to, swallow your pride and accept the feedback your editor is giving you. But it always makes for a better end product. And it's really, it really is something to work with a good one. I've enjoyed it. Bill: Yes, it very much is. Jonathan: Yeah. Bill: So if there are any aspiring authors out there that have an idea for a good tech book, then use no starch. They're very good. Jonathan: Yes. We interviewed I don't remember his name, but we interviewed a fellow about a couple years ago, I think it was, that wrote a book about how to write tech books. And I think his recommendation, 'cause I, I did the interview and I immediately thought, Hey, I could do that. And his recommendation, I believe was to go through no search press, talk to them about being potentially the publishers, because they do a lot in this area. Sort of the tech stuff and the educational stuff. I quickly realized that I had enough on my plate, and in fact, I had too much on my plate. But yeah, that was my game plan. If I was gonna do it, is I would start putting it together and then reach out to no starch because they do, they seem like great people to work with. Bill: Yeah. If I was to suggest the methodology for writing a book the first thing you gotta do is you gotta think about a good outline for it. The sequence in my book is very important, and that's why I think it's achieved the popularity that it has, that I have a, I wanted to teach the foundational skills of the command line, and I do it layer by layer. Going through a series of essentially tutorials. And I try to build on what they've already learned and do more and introduce additional tools. And then finally when I have so much stuff that they can't remember it all, then I'd start talking about shell scripting. So they don't have to anymore. Jonathan: Put it all in file and then you can forget about it. Bill: Yeah, exactly. And absolutely it's funny, the book pretty much embodies everything that I know. So don't ask me a question outside the scope of the book 'cause I just don't know anything else. Jonathan: There is a, you're talking about that and there is a video of Yeah. I'm not sure if this is Dennis Richie or Ken Thompson, but it's like from the at and t archives and it's him, he's got it. Dress shirt with a sweater on over it. And he's leaned back and he's got a keyboard in his lap that he's poked. Bill: Oh, that's that's Brian Kagan Jonathan: Ah, Kagan. Okay. Bill: I know that video. Jonathan: Yes. So anyway he's talking about this thing that they've put together called Eunuch and it's so let's say that I've written a document and I wanna spell check it, and normally you would have to use this big spell check program. He's but in Unix we can just, and he goes through this. We take the document, we sort it, and then we do a unique, and then we run it through our spell check program. And, you use pipes to to redirect one command into the next. And I. A about once a year, and I'm due another one about once a year. I sit down and I watch that and I just marinated it because it's so great hearing from one of the original guys working on this. And it embodies a lot about what I think is really cool about doing bash scripting and making one-liners and Linux and Unix. But obviously we're primarily talking Linux here. And that's just, I don't know. It makes me feel good to watch it. And I think your book has the same vibe to it. What parts of it I've seen? Bill: It's I try to teach both technique and context to a lot of the command line stuff. The Unix has a very rich heritage of command line tools And programming concepts. And so I'm a traditionalist. I want people to be. Read when it comes to Unix and to know how to do things in it that maybe not the most popular things anymore. I have a chapter in the book on printing, which, hardly anybody does anymore. And the command line print tools I have a lot of stuff about text formatting. Because those are so essential to the, the history of Unix. And so I try to teach everybody what I think they ought to know. Not necessarily like the most fashionable things that are popular today, like containers and virtual machines and stuff like that. But I want him to know the fundamentals of the Unix experience. And to have an appreciation for, the giants on whose shoulders we stand. Jonathan: Yeah. Yeah, absolutely. How much do you in the book do you get into I'm thinking of Said and a those are probably the two biggest ones, a lot of tools like that are just about topics for books and unto themselves. Bill: I I have a chapter on on like I said about text formatting and and text editing. And I cover said to some extent including little said scripts and stuff. I don't cover a in the book though, I mention it. I have a separate on my website. I have another section in there besides, the books and the existing tutorials. I have another set called The Adventures. And on which I'm basing my second book adventures with the Lenox Command Line. Jonathan: Oh, cool. Bill: And I have a big chapter on a in there. And I was excited to see recently that there's a new edition of the, a Programming Language came out a year or so ago finally after the original version came out in 1985 which is really the definitive book on a Which is a skinny little book. I used to, I got it free with a do based version of a, that I got back in the late eighties. And, that book is fantastically expensive of the first edition. They were charging like $70 a copy for this thing. And it's it's like about a three, three eighths of an inch thick. And I was just horrified at what that publisher was getting away with. But recently they've come out with a new edition of the book by the original authors. They're still breathing and yeah, it's much more sensibly priced. So I recommend people get that if they're interested in a, get the, a programming language book. Jonathan: Is it worthwhile still to learn A instead? Bill: I think so. Particularly a, I think a is a really fun language to work in. It's very c like, so if you're, if you're c curious you could certainly learn a lot from using a and a is really good at certain kinds of problem domains. Anything where you have to deal with tabular data. It's really, it's the best way to do it. Jonathan: Yeah. Yeah. Boy, I'm thinking of a lot of different directions I want to go with this. You mainly concentrate on Bash, right? So do you give any treatment to like the fish shell or going in the other direction? Any treatment to just working with sh out of something like I guess most machine will have an sh but also out of like busy box where you don't have all of the Bash niceties works. Bill: I covered that a little bit, but I don't talk about ZSH and I don't talk about fish. And it's very bash centric, but I think I point out in places where this is a batch and it, it won't work in dash or in busy box. Jonathan: Yeah busy box is honestly the one that I have dealt with the most. That's because I've got some background doing open WRT stuff. So I've spent some time inside routers on the command line. And that's an experience that's, you get an education real quick about things that you didn't realize were BS schisms. I imagine you have the same, you would have the same experience working on like a Mac machine, which I've done very little of. But they use their own shell over there too. Bill: Yeah. They're using ZSH now. Jonathan: Okay. I Bill: think. Yeah. 'cause they were using bash version three because they wouldn't go beyond that because of the changes to the they're very. Proprietary, shall we say? Jonathan: Yes. Bill: And one of the reasons I'm a, an Apple free shop here, Jonathan: same they finally convinced me to get an iPad for doing testing. But other than that same I have avoided drinking that particular bit of Kool-Aid. I can't say much. Google is just about as bad in some ways, but Bill: It has definitely gotten worse. I was very deeply into the Google ecosystem. But if you've been following Cory Drow in his in ification talks, that it's inevitable. Jonathan: Yeah. Yeah. And I've got some, I've got a friend that does a lot with the Android Open source project, and, over on the Google side, you've got they're moving things out of A OSP into oh, what do they call that? The thing the place. Google Services Place. Google Place Services, I think. Services, yeah. Yeah. And so that move things over there make it harder and harder to have a decent Android experience with just the open source stuff. That's a tangent, although it is one of the things that always amazes and amuses and delights me as well, is being able to install something like turx over on Android and have a full bash experience there. Now, typing on a touch pad is terrible, but Right. Being able to have a full bash inside your phone is pretty great. Bill: Yeah, I've got I've got a, what is it a $79 Walmart Android tablet, which has term on it. And and I use a Bluetooth keyboard with it. Jonathan: Yeah. Bill: And it's pretty funny. It's running, running Vim inside your Android tablet. Jonathan: Yes. It's great. Be able to SSH from it out to the world. Yeah. Yeah. I like it. Do you cover SSH That's, actually, that's an interesting question. I always consider SSH to be like a superpower. I always feel like I'm in the matrix when I'm SSH from one computer to a second one, and then from there to a third one. Bill: Yeah I cover that in the book. Jonathan: Do you dive into any of the SSH tricks like port tunneling and the shortcut to be able to pull up the, in the in session menu? Bill: I don't talk about in session menus, but I do talk about board tunneling a little bit. It's a little, in the weeds for the book, but I used to talk about how to send X through SSH how to do tunneling that way. But way that's become irrelevant and I think I removed that in the new edition because in the Wayland world we don't do that anymore. Jonathan: You can, there's stuff like way pipe, but it is different. This is true. Yeah. This is true. Alright. So what's the first thing when someone picks up the book and wants to get started, what's like the first thing that you try to teach them? What do you think what do you find to be the most sort of fundamental of the things you try to teach in the book? Bill: The very first thing I talk about in the book is there's this thing called the shell, and you have a thing called a term terminal emulator and go find it in the menus and then launch it, and then type some gibberish into it and see what happens. That's the very first thing I do. Jonathan: That's great. And so do you get to, do you get to talk to folks that that kind of use this the book or the website as their introduction to the shell? Do you get feedback from people? Bill: Oh, yeah. I get a lot of really kind emails from people who are deeply appreciative of the fact that I given the book away and that I get a certain percentage of people who say the book changed my life. That it put me on a course to go into cybersecurity or yeah. Or development and so on. Jonathan: Yeah. That Bill: 1, 1, 1 email I'm particularly fond of, and I brag about this all the time, so please forgive me. Jonathan: No, that's fine. Bill: Which is the first sentence of the email was you are a global treasure and that will really brighten your debt. Jonathan: Yes. Bill: When you get one of those. Jonathan: Yes. Yep. That's really cool. I was gonna go, I was gonna go a particular direction with that question. Let's see if I can remember what it was. Do you have a sense then of like, how many people are reading the book just to brush up on their existing user Linux experience or, do people find the command line through this book or through the through the website? I'm just curious like what the, I would say what the onboarding experience is. Bill: I would say both that. There's a bunch of people who will read portions of the book and not read all of it. Because I get reports from users or from readers reporting like typos and so on. And I, the most reports come in the first few chapters of the book and nothing ever gets reported about the latter chapters. People read probably the first part of the book, which deals with the basic command line stuff that you do. Typing file copying, wandering, the file system, navigation. Processes, permissions and stuff like that. They, the basic stuff, the later parts of the book talk about all kinds of tools that you can use on the command line. And then the final part of the book is deals with scripting. And in the middle I talk about the environment and how to configure it and how to use Vim to prepare you for all the things that are gonna happen. Most of the reports I get are for people who, who have read the first few chapters of the book, which covers, the very fundamental basic things you need to know how to do. Jonathan: I'm curious, I remember where I was actually thinking about going with that question, and that is are people still generally interested in the command line, or is the gooey taking over, and I guess this is an old debate, but I'm curious your thoughts on it. I've talked to some people that it's I wanna learn about computers, but man, don't make me go into that black window with the white text. I'm scared of that. Bill: In the introduction of the book, I talk a little bit about this and I have this little story that I tell which is that. If I might quote myself here, if I. In the movies where you see the super hacker, the guy who can break into the ultra secure military computer in 30 seconds, you ever notice how he never touches a mouse? And it's because filmmakers know that humans instinctively know that the only way to get something done on a computer is by typing at a keyboard. Jonathan: Oh, that's great. It's funny. That's an interesting tangent. But it's funny, you've got some movies that the hacker is just, spamming at the keyboard and then magical things happen with the screen and it's ridiculous. I really enjoy though those few movies where they work really hard to get it right. There's a scene from, I think the second matrix movie where they used real command line tools ironically the Second Tron movie the one that's, I don't know, I guess a decade old now, which makes me feel old. They did the same thing. They used real command line tools and made it look very futuristic. But it was actually just, I think it was running inside of emax. Okay. And then another scene was basically a Solaris shell. I enjoy that kind of stuff. I think it's really cool when film in particular, when they actually take the trouble to get it right and not just be ridiculous. Bill: I noticed that the social network. Movie was very accurate about about command line stuff and using KDE and so on. It was, I was very impressed that usually in movies, accuracy is secondary to, cinematic experience. Jonathan: Yeah, that's a good way to put, that's a very charitable way to say, to put that. So you talked about VI and Vim. Is that the only text ary that gets love in the book, or do you hold people by the hand? Bill: I talk a little bit. I talk about Nano, but I discount it. You know Jonathan: what, come now. Bill: Yeah. One of the, one of the jokes in the book that gets probably the most reaction is I have a, at the beginning of the. The chapter on Vim, which is called a Gentle Introduction to vi. I talk about the reasons why you should use it and I go through the fact that it's almost always available, it's lightweight, it's fast. You never have to lift your hands from the keyboard and you don't want other Linux and U Unix users to think you're a pansy, Jonathan: oh, that's great. I like it there. There's so many of those fun. I guess they were quite serious Holy Wars at one point, but now they've just become in jokes. And things that we all laugh about. Bill: Like EMAX versus vi. Jonathan: Yes. Yes. Although ironically we're seeing our own generation of holy wars with things around like whether the rust language is good or not, whether it belongs in the kernel and whether you should use rust or go, or c plus. And people do get quite passionate about some of those things I've seen. Do you dive into any of the other scripting languages We mentioned a briefly, but do you have a intro to any sort of intro to Python or any of those things? Bill: I have a chapter on compiling software. Because I have an exercise in there where you download a tar ball from the canoe project and and expand the thing, do dot configure, and then and then, run, make and build it. And it's a very simple exercise with a pretty small code base. And I talk in the beginning of that chapter of. How things are, I talk about interpreted languages and I talk a little bit about and to some extent it's an interpreted language. And I talk about languages and I talk about assembler. Just give somebody an introduction of what does the the programming ecosystem look like in the most broad, categories. Jonathan: Do you cover how to get out of dependency hell at all in that Bill: no, there is no way out of dependency. Jonathan: That's, that is actually true. It's sad but true. I remember one of my first experiences trying to compile something. I didn't really understand how the package manager worked at that point. And so I was trying to compile something and I didn't have. I, oh, this has been years and years ago, and I don't remember like what package it was I was missing some fundamental package and me and my ignorance, I just went and downloaded that tar ball and started trying to build that first. And, I ended up accidentally trying to do my own Linux from scratch and, finally someone pointed out to me, you have a package manager and you can install dev packages from the package manager. It's I would've appreciated someone holding my hand for that first experience. Bill: Yeah. When I started out in Linux, Linux was hard to use. E Jonathan: everything was Linux from scratch back then. Bill: Yeah. Yeah. Pretty much. You really haven't lived until you've edited mode lines in an x config file to get your monitor work. Jonathan: I know what you're talking about. I, I don't know if I've ever had to do, it. Seems like I've had some device that had a mode line problem. I think it was a phone. I think it was some weird old phone that I was trying to run Turex on and I ended up having to do an edit there to make something work. So yes, I, but that is an esoteric thing, like there are a very limited number of people that know how to edit those mode line vials, Bill: Oh it's hairy. I dunno how. To figure out what you put in that thing. But I know where you could get the information and put it in. The guys who figured out, this is the video timing that you need to worry about. That's a that's hairy stuff. Jonathan: Yes. Bill: The people who work on VLC or work on Fmm Pig, those guys, those people are deities. I'm concerned Wizard. Jonathan: Yes. Yes. Very much Wizard level. Alright, so we do have a live audience and one of them just asked something and so those of you playing the drinking game at home, take a drink. We're about to ask about ai. Quippy says, this is where Gemini and Claude code come in very handy. Bill: I hate that stuff. Jonathan: I'm not surprised, but I'm curious why? Bill: I would say that it presents to. Basic problems. One is three actually, it it's not reliable. Some recent studies have shown that programmers who use it end up spending more time checking it and fixing it than they do getting any productivity gain. Jonathan: Yeah. Bill: Secondly, I think it makes code that's opaque and you would, might end up with a lot of technical debt and inability to maintain the code in the long term. And third, by having them do it, you don't learn anything. So it, it de-skill you, which I don't think is economically valuable to, to the individual. Jonathan: Yeah. So one thing you mentioned in your bio is that you've ridden three waves in your career. Bill: Yeah. Jonathan: I would say, and I think you, you mentioned this e the even there that the current ai craze is yet another wave. Tell us about those three waves. I'm curious. Bill: Okay. When I first started doing computer work I started doing it in comp in college when I was an industrial design student. I, I did a computer graphics course in the experimental studios department, and they had an IBM system three 70 with a CalCom plotter on it. And they had written a specialized programming language called splat, the simplified programming language for artists, which you would sit down at the key punch machine, punch cards, drop the deck into the reader, and then come back in a few hours and see if your job had run. Jonathan: Oh, Bill: that was my, my college level work in computers. I got my first computer in 1978, the year after I graduated, and it was a TS 80 model one, and I proceeded to write some software, art software on it. And I thought that in the first few days that I'd had that computer, I'd used more computer time than I had entire of my college. Jonathan: Probably accurate. Bill: Yeah, because they used to charge by the minute. CPU time on the mainframe. Jonathan: Yeah. Bill: I was teaching at the time I was teaching photography and mechanical drawing at a private school, a high school. And I left there and got a job at a little computer graphics company near not far from where I lived. In Tacoma Park, Maryland. And this was at the beginning of the PC era. This was in 1985. And I rode that wave really well. The the PC explosion. The second wave I rode was the the early internet wave around two Jonathan: thousand.com. Bill: Yeah. And I was working at that same company. What was I doing then? Actually, I had left there, gone to work on the FBI contract that I did for four years. With, in the Solaris shop. And then I came back, I got rere, I got recruited back to the company. I had been and went through the.com bubble. That was a very strange time because it was really hard to hire people. I was, I was in management and it you couldn't hire people for love and their money 'cause everybody wanted to be in a company that was about to do an IPO. I got included in the Red Hat ipo. Oh, okay. Nice. Because I had been file, I had been filing bugs on their son on their Spark version of Red Hat Linux. And I got invited into that IPO. Jonathan: Nice. Bill: And the third wave is, the Linux wave where, I have made some money from writing a book. And I've done that. And the next one is probably ai, though I think AI is gonna go through a big shift soon, in my opinion, that all this data center construction is gonna blow up in everybody's face because nobody's figured out a good way of paying for all of it. If you spend a trillion dollars, you need to have some serious revenue stream coming outta it. Nobody's done it yet. And I think that they're gonna have a, they're gonna have a terrible reckoning over the finances of all this capital expenditure they've been doing. Jonathan: Yeah. One of the, one of the observations we've made is that each of these could be described as a bubble, but Oh Bill: yeah. But Jonathan: each of them as a bubble. When the bubble bursts it it still changed the world. And I don't think that's gonna be, that's gonna be the same thing with ai. I've got another friend who says they're not bubbles, they're balloons and some of the air gets led out of the balloon, but it's still around. Yeah that's an interesting way to put it. That's a fair analogy Bill: as well. Yeah. 'cause the technology will persist, but the businesses that try to exploit it will not Jonathan: some, so Yeah, some of them will definitely go away. Bill: Yeah, Jonathan: for sure. I saw, I don't remember if it was in your bio or over on the website, but I saw a bit of a advice that you gave to people that want to be programmers, and some of it was a bit mercenary, but I thought it was really interesting. Bill: Do Jonathan: you remember that? Bill: Oh, yeah. Yeah. I tell people to work hard, save your money, and retire early. I, I retired when I was 50. Jonathan: Nice. Bill: And I've been retired for a long time and and it gave me a chance to do what I really wanted to do, which is to write this book and make the world a little better than the way I found it. Yeah and I say that, programming is mostly a young person's game. When you get old like in your forties and fifties, nobody wants to hire you anymore because they can't make you work 80 hours a week. And 'cause you might have, like a family or something. So you need to have an exit strategy to get outta programming and get into something else. And also if you're in a situation where you're working for somebody, a rich guy and making him try and making 'em richer, make sure you get your cut. Jonathan: Yeah that's the one I think that's actually fairly important for young programmers to, to think about. When sometimes when you're a programmer, you make really good money. I've seen this firsthand. Programmers make really good money. And it's tempting to, it's tempted to buy the big house and get the big mortgage, but that programming job may not last forever. Things change. And yeah, I think that advice to be a little bit mercenary is pretty important. At least think through those things. Bill: Yeah. Take care of yourself. Jonathan: Yeah. Yeah, for sure. Let's see you have the adventures in Linux book is coming. Do you have a sort of an idea of when that's gonna be a real thing when it's be gonna become a real boy? Bill: It's it's available now as a PDF as you can download. There's 14 chapters in it. I wrote most of them during lockdown. And I'm I originally planned it to be about 30 or 35 chapters, like the, like my current book. And I may eventually, if I live long enough to finish it, I will. 'cause I'm getting on in age and I don't know how much more of this I've got in me. Jonathan: Sure. At what point is it gonna get a physical copy? Is it gonna get published too? Bill: No starch has not reached out to me to commercially publish this book yet. Jonathan: Yet. Yeah. Yeah. Hey, guys. If you're listening hopefully it sounds like that would be a fun follow on. I will have to order at least to the one I would like to have it on my bookshelf. I was thinking, I've got a large books, obviously the camera can't see it, but over here on this side of the screen, I've got a large set of bookshelves and I've been thinking this whole time. I'm pretty sure I have at least one no starch press book, and I haven't seen it yet, but I think it's up there somewhere. If not, I need to order a few, because I do like them as a publisher. I need to pick some up. Bill: Oh, I've got a copy of my new one right here. Yeah. Jonathan: Nice. Is it a it's a soft cover. Bill: Yeah. Jonathan: Yeah. Very cool. Bill: It's about 600 pages. Jonathan: Oh, wow. Bill: It's a big Hummer. Jonathan: Lot of book. Bill: Yeah. A lot of book. Jonathan: Yeah. So you've been doing the programming thing the command line thing for 30 years now. What what's changed? And I have a story here too, but I want to get your thoughts first. Bill: First off, I wanna say it doesn't change very much that one of the things I talk about in the introduction of the book is the fact that, if you learn this skill now, 10 years from now, it'll still work, which is very much unlike many other things in the computing. I would say the shell and the core utils, which is the bulk of the book. Hardly change at all. And when they do, which you have instead have, and when they do Jonathan: change, it's considered a bug. Bill: Yeah. And what you do have is you have certain tools that come in and out of fashion. For example, nano is gains features and it's become more popular over the years compared to the way it was when it was really just the clone of the the PCO editor from the Pine email client. And in the current, in this new edition of the book, I. Also I change from w get to Curl because those are things that are, are more in fashion now than before. I've also, in the new edition I have, I deemphasized optical Media since that's gone out of fashion. Jonathan: I'm a little sad about that one, but it is true. Bill: Yes. I still have DVD burners and I use them because I've got computers that, you have to, if you wanna install Linux on, you have to do it from disc. Jonathan: Yep. Bill: Yes. This machine I'm talking to today is this is a framework framework 13 laptop. And which I like a lot. It's the first new computer I bought myself in about 10 years. And I've got a, it's sitting on top of a couple of other laptops. One of 'em is a circa 2003 IBM ThinkPad. Which is happily running Deion 12. Even though this computer is, it's over 20 years old. Jonathan: Yeah. Bill: And I just love the fact that it does that, but I'm very sad by the saddened by the fact that support for 32 bit machines is vanishing from Linux distributions. Debian was like the last one, and they've just thir Debbie and 13 does not support 32 bid anymore. Jonathan: Yep. I imagine we'll start seeing some specialized distros specifically to support the older hardware with the 32 bit stuff. It's gonna be, it's gonna be around in the kernel for forever, We still the, did they finally re remove the I 3 86 support, or is it still in there? I don't remember which, but very recently that's been a conversation in the colonel. Bill: I think that has I think it's gone away certainly in Debian, I dunno about the Colonel proper. Jonathan: I, so my story with this is I worked at a doctor's office one time. I run a small IT shop. Have for years. And I went into a doctor's office and they were doing a transition from their old system to their new system, but they said, we still need the old system to work. But they were having some hardware problems with it. And I got in there and I took a look and it was an old SCO Unix box. It's oh, I know what this is. I actually took it home, copied the, did a bite for bite copy of the hard drive, and I got SCO Unix running inside of a virtual machine on Linux. And then set it up to and this is funny too, because the the, you talked about optical. I set it up to where every night that Linux box would check and see is there a blank DVD in the drive. And if there is, it would do a, like a full disc backup of the SCO system onto that DVD with all of their data on it. I was pretty proud of making that work. But the fun thing with that is working, 'cause obviously I had to do some work inside SCO to make it work inside the virtual environment. And so many things were so familiar because it's all based on the same shell. It's all Unix. But there were certain niceties that I was so used to on an inside of Bash and on the Linux shell, like even tab auto completion and the ability to press the up button to get into history to repeat something. It's like those didn't work in the old SCO shell. And it's oh, this was the old times that people talk about where you had to walk uphill both ways in the snow. Bill: That's right. Yeah. Coherent was like that too. It didn't have command line editing like that. And yeah, it lacked a lot of the niceties that I fell in love with when I started using ux. Jonathan: Yeah, for sure. There were a lot of things like one of the, one of the other things about SCO Unix is I don't think it had modules. So if you needed to change what you wanted to load at boot time, you recompiled the kernel. And so that I have recompiled the SCO eunuch kernel in that way. And, that's, that is something that's probably pretty rare for someone my age. I'm almost 40, so I'm not young anymore, but I'm young enough that I have no, I had no business doing that at the time. Bill: I think I think the BSD Linux, Unix is are frequently have kernel compiling done. Jonathan: Yeah, that's probably true. That's an interesting segue. How much does your book apply to the bsds? Bill: In so far as you're using canoe tools on it lots. But otherwise no, Jonathan: that sounds about right. That's been my experience working on them. Okay. So one of the things I found, one of the other things I do is the Untitled Linux show, and we have the command line tip section at the very end where each of the hosts will pick for the week, a command line tip. And we've been doing that for years now, and the Pickens are getting a little slim, but we've still not run out because there are so many tools on the command line and more being written every day. What has your experience been with that? Surely there's always a sort of a punch list of, I suppose I could add this to the book and you've gotta cut it off somewhere or you said it's 600 pages. Did you include all of them? Bill: Oh yeah. Pretty much everything I know is in that book. And. I have a very extensive, bash RC file. And and I have a another file that I source on all my machines because I've got a lot of computers in here. That sets up the, all the environment for the my network at home. A lot of the stuff I do is very specialized for my particular administration needs. So I don't know that I have a lot of really general tips for people. One of the ones I talk about in that bio you have is a very silly one, which is to simply set the environment variable for man man width equals 75, and then man put it in an alias. And so man will always come out with, nicely margined text when you look at it. So if you're using a giant terminal window, you don't have 400 character wide things in it, which makes it hard to read. Jonathan: Yes. Yeah, that definitely makes sense. What has the process been like of updating the, both the online version but also the print version from one addition to the next? Bill: When I start to feel that the text is becoming dated I undertake a revision to it. The the third print edition is a combination of the sixth and the seventh internet editions. The sixth edition added the 40 pages that have been added to the current book. And I changed I added a bunch of new commands that I hadn and covered before, like I put in nice and re, nice and no hub which were in the processes chapter. I switched over, like I said before from F disc to parted for storage stuff. And I added, chapter and. How you have to build this this ever increasingly long pipeline to to filter out the words that don't qualify for what your clues are. Because my wife, she plays all the New York Times Word games. So I've written scripts that facilitate those, though she won't use them, of course not. I've got a, I've got an interesting script. I wish more people would play with and use on my website. It's called New Script, and it's a script template generator where you answer a series of prompts of what's the script called? What's a do indicate what the command line options are so it can build a help function for it. And it will flesh out the script with all the traps and air handling and Jonathan: it's vibe coating without the LLM behind it, Bill: pretty much, it fleshes out the parts to make a nice script. One that's well written and it puts in a command line, parser in it and builds one. And it, it will help you, write a script that will be a lot better and it'll do it a lot faster. And I use it all the time. Whenever I start a new script, I just fire that thing up and just describe what I'm doing and then fill in the template with all the actual live logic. Jonathan: I, I will plug this, I will mention this on our next ULS, the UN Title Linux show. That is a great command line tip to let folks know about. I'll get you, I'll get you some more beta testers. Okay. That's fun. What about speaking? We talked a little bit about Holy Wars. There is another Holy War and it is germane to the book. What about system D? You cover some system D administration stuff. Bill: I, in the current edition of the book, I've pulled out references to the Knit scripts in Etsy. I don't cover that anymore because it's gone. Hardly anybody use, there's no mainstream distribution that doesn't use System D now. So I I talk a little bit about System D, but I don't try to talk about how to administer it Jonathan: outside of the scope. Does that one irritate you or is that just the way it is? Bill: I don't mind it. I think it does some very interesting things and it solves a bunch of sequencing problems that you have during boot up. And handling services. I have no aversion to it. Jonathan: That's fair. That's about my conclusion as well, although I don't like the fact that it's taken over DNS. That, that Bill: one. Now the thing I really don't like right now is I don't like these universal package formats, snap, flat pack and app image. I do not like them. Particularly I particularly don't like Snap and I, and the reason why is my main box where I do most of the writing and I did all the work for the book on it is is a a system 76 rattel. From 2015 and it's, I'm running lts version OFTU on it. The Libre office snap on that thing is ungodly slow. I had to put the deb the Deb file on instead. That it was just, the book is written as a one huge monolithic 600 page Libre office document, and it's easier to do that way. I, the adventures book that I've written so far, I did with with separate chapters and a master document, and it's built, from separate components and it's really unwieldy in Libre office to do it that way. Working at a big monolith is much easier actually. And in the deb version of of Libre office it's fast. I can go up and down that thing, edit and reformat pages and stuff like quick. But the Snap Boy, it is dog slow. And, it's just one of the reasons I don't like that thing. Plus what Snap does to your device table and you see all these loops and there you have to filter that stuff out if you wanna look at it. It's just annoying and I, it doesn't do me any good, so I don't like it. And and I have a rant about it in the book about the universal pro package formats that they're good for proprietary software vendors who wanna keep it all bundled up, but it's I don't think it's doing the users any favors by gobbling up disc space and memory. Jonathan: Yeah I get that. That is definitely the direction a buntu is going. Are you considering a jump over to something like the Fedora side? Bill: This, this framework laptop I'm on right now is running for doth 43 with k with KD plasma which I like a lot. And I think I'm going to probably move to plasma generally. I've got a Debian box in here that's running that, but my main box is still running a abuntu. I don't mind the user interface in Abuntu at all. I think it's fine. I don't like playing gno. It's too, it's, it requires too much clicking to get anything done. But the the changes that Ubuntu has made have is fine. I don't have any problem with it. I've used a lot of desktop environments in my time. And they, they don't bother me. I don't freak out about them. Jonathan: Have you gone and tried the new cosmic desktop yet from Bill: 6, 7, 6? No, I have not. Jonathan: That one does look interesting. If nothing else, I appreciate their audacity to be willing to try it, I suppose start the whole thing from basically from scratch. Bill: Yeah. I'm, I'm a little concerned about that distribution, about its viability, doing a distribution is a big deal, especially doing a good one because you gotta have a lot of infrastructure behind it, and you've gotta have, people looking at security, you've gotta have proper technique on how to do quality assurance on it. And, it's challenging. Everybody's brothers come out with a distribution. There's hundreds and hundreds of them, but how many of 'em have any long-term viability? I think Fedora does because IBM's behind that one though. They'll probably pull the rug out from under soon. Jonathan: Oh, I hope not. I don't know where I'll go if Fedora goes away. Bill: Yeah. And canonical you'll go to Red Hat Enterprise Linux and you'll pay for it. The canonical, it seems to be. Still. Okay. But Linux Mint, I like that distribution. My, my wife's computer has that on it and Cinnamon. Which is plasma light. And it's all right. I like plasma more, but plasma it's I wonder about Linux mins learn long-term viability, basically because of how do they fund this thing? Jonathan: Yeah. I've had that thought too. Yep. I've got a new computer I've been playing with actually I like quite a bit. I wonder if you saw, it's the Argon 41 up. This is not sponsored. I paid for this thing myself. I just really like it. And it's got the cute little logo on the screen. But one of the ing things about this is the whole laptop. It's a raspberry pi C five. And then it's the whole laptop shell built around that. And it's it's like the same size as a framework 13. It it reminds me of my frame, my wife's framework 13. So we'll let's point out what's going on here. My wife has the framework 13, and I will occasionally steal it. And she gets tired of that. And so when I pick that thing up, it's it's just powerful enough to do everything I need it to do and I can SSH into one of the bigger machines to compile or whatever. And I've been using that as that grab and go machine to have on my lap while I'm putting a kid to sleep or something. And it's worked rather well, but it's a raspberry pie, CM five and I'm I put raspberry pi os on it. And I'm running KDE on it and I am actually really surprised how well KDE runs on there. I expected it to be a sort of a problem and it's not. So I don't know that k all that to say, I dunno that KDE needs a lightweight version. It's pretty lightweight these days itself. Bill: Yeah. I was impressed by that too, that I have the Debian machine Debian 13 I have sitting over here is a dual core intel machine. From a long time ago. Very slow machine with I think it's maxed out at four gigs of ram. And it runs KD just fine. Yeah. I was very impressed 'cause most of the old machines that I have, my 32 bit machines I use XFCE on them. But plasma really impressed me about how zippy it is on small, slow hardware. Jonathan: Yeah, I think they've really worked hard on that. Alright. We have hit the bottom of the hour. It has gone by very fast. It's been a blast to talk to you. Is there anything we didn't talk about that you wanna let want to let folks know about? Bill: Let me think. Oh, yes. If you would like to support my work please buy a copy of the print book. And preferably buy it directly from no starch press. Jonathan: I was going to ask. Bill: Yeah so we get to not fund any fascistic. Middlemen. Okay. Jonathan: You could leave the adjective out of there. You don't wanna fund any billionaire middleman. You want people to do the work to get the money as much as possible. Yep. Is there is there an audio book version of it? Can I get it on Audible? No, Bill: not yet. Jonathan: Not yet. Bill: No. No. I think that somebody has done like an ai voice version of the book. I've seen crazy stuff on YouTube. And it's, yeah, I think somebody may have done it, but it's that would be insane. I don't know how you express the command line, verbally Yeah. In any sort of way that would make sense to anybody. Jonathan: Yeah, I agree. I'm not sure that would work. Okay, so we talked about this a little bit, but I've gotta ask final questions. What's your favorite text editor and scripting language? Bill: Oh bash Shell's my favorite scripting language and and Vim is my favorite tech set. Jonathan: I figured that's what it would be. Alright. Thank you. Thank you so much for being here. It has been a blast to, to get to talk to you, sir. And thank you. I's been a lot of fun. Bill: Okay, thank you for having me. Jonathan: All right. That is bill Chuchu sch shots. I'm terrible with names. Bill sch, goodness gracious. And appreciate him very much being here, talking about his book, which I think I'm gonna order as soon as we get done here. Don't tell my wife, although she listens to the show, so she already knows. She probably already knew that I was gonna order it. It's, it'll be all right. It'll be our little secret. Anyway we do not yet have a guest for next week, so if you know of somebody that needs to be on the show, let us know. You can shoot us an email@flossathackaday.com and we will get 'em scheduled and yeah, we appreciate it. Other than that, thank you so much to everyone that's here, those that watch and listen whether you get us live or on the download, and we'll be back next week's another floss weekly.
• Episode 864 - Work Hard, Save Money, Retire Early 11.02.2026 1val 3min

  This week Jonathan chats with Bill Shotts about The Linux Command Line! That's Bill's book published by No Starch Press, all about how to make your way around the Linux command line! Bill has had quite a career doing Unix administration, and has thoughts on the current state of technology. Listen to find out more! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• Episode 863 transcript 04.02.2026

  FLOSS-863 Jonathan: Hey folks, this week I'm talking with Ola Schulte and Lars au about Open Cast. That's the project for recording lectures, uploading them, and, uh, coming soon, even a video portal built into it. You don't wanna miss it, so stay tuned. This is Floss Weekly episode 863, recorded Tuesday, February the third, opencast. That code is there for a reason. It is Time for Floss Weekly. That's the show about Freed Libre and open source software. I'm your host, Jonathan Bennett. And today, well, today we are having a revisit from a project from quite a while ago. We are talking Opencast, and that is, well, it's all about video and casting, but not, I don't think in the same way that OBS is or Video Ninja. No. Open Cast is more about, uh, in its open source video management, but it's more about academic institutions in places like that. So we're talking sort of a, a, a bigger, maybe more formal solution. Um, but today we've got a couple of the experts with us. We have Lars Ki, who's been an Opencast developer since 2009, and we've got Olaf Schulte. Who is responsible for video management at ET Zurich Switzerland and has been supporting the Open Cast project ever since 2007, and he joined the board in 2011. So we have a couple of real experts to, first off, tell us what exactly Open Cast is and then get into all of the nitty gritty details. So to start with, welcome to both of you guys. Appreciate you being here. Lars: Thank you, Jonathan, for having us. Jonathan: Yes. And so Lars: thanks for having us. Jonathan: Before I forget, Lars: it's nice to be back. Jonathan: Yeah, it's good to have you back. Before I forget, where are you guys talking to us from? I'm in, I'm in the middle of us. I'm in flyover country in Oklahoma. I don't think you guys are, are very close by me. Olaf: I'm in, uh, Zurich, Switzerland. Mm-hmm. Where it's, uh, getting dark over time. So you will see that the lights are going down here over time. Jonathan: Yes. Olaf: Still in my office though. Jonathan: I'm sorry. Lars: Yeah, I'm, I'm at home already, so, uh, I'm visiting for here from, uh, Oberg in Northwestern Germany. Uh, yeah, it's, it's also getting dark here. It's also actually, uh, beginning to snow right now. Jonathan: Oh fine. We just got done with some snow here in the us. Uh, for, for the, the sake of everyone listening, who is, who is Lars and who is Olaf? Lars: So, I'm Lars. Olaf: And I'm Olaf. Jonathan: Alright, very good. Makes sense. Okay, so what is Opencast? I gave, I gave my, uh, my best impression of knowing what it's about, uh, but I am not the expert here. Um, it's up to you. Which one of you want to take this question, but, uh, what, like if someone says, okay, lemme put it this way, what's the problem that somebody has that they go opencast, that's the solution to it. Lars: So, uh, I guess I take it, uh, yeah, actually quite close. When you say it's, uh, it's about open video and it's about streaming and it's about video recordings. Uh, but all related, or mostly related, at least to higher education. So you wouldn't actually run opencast at home to, I don't know, have your, uh, DVD collection, uh, to stream somewhere or something like that. If you want to do that. Technically you can, but just don't, uh, it, it's not fun. Um, however, if you run, run a school and, uh, I dunno, you have. Uh, a lot of videos, a lot of people recording stuff. You may even have lecture halls where non-technical personnel wants to be recorded. Uh, then Opencast is all about this automation. It's also, uh, there is, uh, since we last spoke, uh, development of a new video portal. So, uh, it's also about kind of a YouTube for your campus project. Uh, so larger scale, all of that, uh, and lots of automation. So if you want a story to, to imagine how this works, uh, imagine you are a lecturer, lecturer of, I don't know, ancient Greek mythologies. Something completely unrelated to technology. So you have no idea about technical stuff. You don't, uh, know, for example, how to record your lectures yourself. Um, what you want is to just tell someone, well, please record me. And of course. You could send out people who then set up a camera and stuff like that, but that's actually a lot of work if you have a lot of lecturers, and that's extremely costly. Uh, instead what Opencast provides is that you have an integration in, for example, uh, your room media. So whatever goes on your presentation, uh, can be automatically recorded by capture devices. Mm-hmm. Um, and then as a lecturer, I just have to tick a box. When I schedule my, my courses and say, you know, I want to be recorded. I enter the lecture room, the camera turns on automatically. Uh, Sam's being recorded automatically. And then even the video processing and stuff like that is mostly automatic. Um, of course people can interact with that. So there's a manual video edit mode as, for example, uh. But to the most part, people don't have to. And if you look, uh, at schools, say our school, for example, has about 50 to 60,000 videos recorded in Opencast, uh, which you don't want to do manually. Uh, and we are far from being the largest school when it comes to recording. Jonathan: Mm-hmm. So, boy, there's a, there's a whole bunch of different directions that I wanna go with this all at once. We last talked to you guys, I think in 2019. Lars: Exactly. Yeah. Jonathan: There's been some things that's happened since then that are probably important. Uh, I know a lot of the, in the US at least, a lot of high schools and even younger schools went, did a lot of virtual learning as a result of, of COVID and those things. Uh, did that, did that impact the project? Was that a, a big deal for you guys? Olaf: Absolutely. It, it was a, a major impact, uh, on our project. And on, on the requests we had with respect to, to whatever form virtual learning can take. And that totally depends on, on the school you are at and the school's policies towards COVID and after COVID mm-hmm. Towards the, uh, new normal we found. So what happened first is that the number of, uh, recordings overall and the number of videos, sorry for the ambulance. Jonathan: That's all right. Olaf: Uh, the number of recordings overall increased, so people doing virtual, uh, lectures, uh, everything could be recorded and people had to start and think about what to do with these recordings. And as Lars indicated, the number of recordings suddenly increased from a couple of thousands to 10 of thousands, which is something that you, you have to organize in a, in a video management system of some way or another. Um, take the live streaming, for example. Um, most universities didn't do live streaming before COVID, and suddenly it became a necessity in order to reach students. Mm-hmm. Uh, which is also something that Opencast was, was able to do before. And we actually, um, dealt with that part of the technology during COVID in order to increase or to decrease the latency, for example, because of the, the need for interaction in the, in the virtual learning scenario we had in most schools. So, uh, I think that many schools saw the necessity to, to, to find a solution, uh, for all the videos they produced. If they had a video management systems, let alone, whether that's Opencast or a commercial solution, um, they certainly had to find a way to deal with a large number of videos afterwards, uh, how to store them and how to pay for storage in case you have a commercial system. Because obviously the more videos you have in a commercial video systems, the more you have to pay for that, which in turn raises the question of whether you want to pay for older recordings being stored away and not being watched. Uh, that was pretty interesting to see how, how differently schools were dealing with that. Um, we certainly benefited from, from that situation in, in that we were able to sort of, um, develop new features. Uh, Lars can talk to that better than I can. Uh, we have, uh, a web-based do it yourself, uh, solution for, for recording. Uh, called Opencast Studio where teachers can sit at home, which is what you did at the time, uh, and record their lectures in order for, for students to see that, uh, something we didn't have. So I think it's fair to say that overall Opencast, uh, as a, as a product and as a project benefited from, from COVID. It's not a nice thing to say, but it's a pact. Jonathan: It's, it's the reality of the situation. Some, yeah, there was, there were some industries and products that really took a hit, took a hit as a result, and others did very, very well. Um, what, uh, you mentioned the, kind of the dichotomy between, uh, an enterprise, a paid solution and Opencast, is there crossover there as well? If someone says, I, I want to use this neat open source project, but I don't want to be the cis admin for it. What's the, what's the solution there look like? What, Olaf: so, um. If you, if you are a school and deciding on whether to, uh, either buy a commercial system or use opencast, uh, you don't have to run that, uh, for yourself, OnPrem, per se. So there are different options. Uh, when we started out and when we first met in 2019, I think it's fair to say that uh, most of the open cost systems we saw were, were running OnPrem and had a system administrator, uh, at that local universities. But also due to COVID and also due to the increasing demand of not, of, of schools that were not as huge as, as the schools, uh, initially involved in Opencast, there were different models. So, for example, we now have a number of, of service providers who can run opencast, uh, for you in the cloud so that you can use it as a school without that system administrator sitting by your desk. Uh, also we saw a number of projects where one institution was providing services towards other, uh, higher education institutions in the sense of one institution having that system administrator and other smaller schools benefiting from that. Mm-hmm. Um, so there is actually a, a number of settings in which you can run open costs without a system administrator. Jonathan: Yeah. Uh, have, have any of those things allowed the project to bring on some developers full-time? Is there, what, what is the, what is the funding model of all of this look like, I suppose is the question there, maybe, maybe that's a, maybe that's a question for, um, uh, for Lars. Lars: I can certainly answer that. Sure. So, um, now I'm, I'm actually working directly for a university. Uh, back then when the growth started during COVID, I was working for, uh, a nonprofit that, uh, was actually helping schools with running a incar and with mostly running this on premise. Um, and I can tell you that we started out with, uh, I think about six engineers. Uh, at the end of COVID there were about 20 engineers. Um, and most of them were actually, uh, working on, on a cost. Uh, to be fair, most of that work went, uh, towards the, the customers. Yeah, setting up is up locally and solving local problems. But a lot of that actually goes also in the direction of the project. And what I, for example, found pretty cool back then with my previous employer that they have for every contract, uh, a margin of 20% of all time spent on this project actually goes to the, um, the open source product directly. So, uh, that's time that is being spent on the project directly. Um, and the other thing is that, I mean, lots of, uh, the customers essentially here are our schools. Mm-hmm. Uh, and they don't really compete in that area so much. So, uh, lots of them contributing back as well. So if you actually look at, at Opencast now, there's still. A number of, uh, of, uh, full-time employees working at Opencast. Uh, we looked earlier, uh, when we were waiting here and we have 18 committers right now. So that's core committers with commit rights directly produce projects. Mm-hmm. Which is still active. Um, and yeah, I think that's quite a lot. And most of them are employed somewhere to work on, on the project actually. So, uh, I, I said right at the beginning that not a lot of people would run this at home. So we don't get the, a lot of individual contributors who say, oh yes, I want, I want to do that. We actually get this sometimes 'cause students find a bug and they want to fix it, uh, which is pretty nice. Yeah. But most of the contributions actually come from people employed by, uh, a school or something like that, or by one of these service providers. Jonathan: Yeah. Very cool. Olaf: But let me just add that, um, this certainly sounds quite good when you talk about 18 committers, for example, but you have to keep in mind that they are employed, uh, at an institution. So we are not organizing them as a project without sort of coordinating the, the universities as well. And what, what can always happen is that universities, for example, have local needs that their developers have to look at first. So for example, during COVID, some schools suddenly started to look into certain features that weren't that interesting for Opencast in, in general. So they were missing from the project at the time, or are missing today because of other, uh, things that have to happen, uh, at their, uh, particular school. So it's, it's very much a coordinated effort. Uh, there's not a lot of money flowing around. We do have some, some sponsor. Uh, institutions as well as, uh, commercial sponsors. And I think it's fair to say that we only have one, uh, developer who's actually the, the quality assurance manager also for the project who is sort of employed by the project. But the other, uh, uh, 17 committers then basically are coming from higher institution, uh, higher education institutions, uh, committed to Opencast, but to a certain level. So there's no one who can call on them and tell them what to do. You always have to coordinate them and find something that is interesting enough for those 18 institutions. Actually, it's a smaller number for those institutions who are then willing to, to, um, make their developers work for Opencast at that particular time, which is sometimes quite difficult, but I think that's something that other open source projects also will face. Jonathan: Yeah, absolutely. You, you, you have to ask nicely. Olaf: You have to, you have to, uh, ask nicely and you also have to present them the advantages, which is very easy if you, for example, if you build a new video portal, everyone is very excited and wants to have that video portal right now. But if you go to the back end of your system and want to fix something, which is totally unattractive in terms of decision makers at universities, right? Well, I'm not sure we can afford that. Uh, can we look at, into this next year? It's very much, uh, uh, a thing you have to convince them of. Jonathan: Yes, yes, I understand that. Um, okay, so let's, let's see here. Uh, I'm curious how this actually works. Um, the, the, the idea of automatically recording, uh, in lecture halls. And when you first said that the thing that. Just immediately came to mind is like a raspberry pie strapped to the ceiling running off of POE with a webcam hanging off of it. And it's like that, that's probably exactly what you do in many of these cases, isn't it? Lars: You know, it's a, it's actually quite funny because when the first raspberry pie came out, uh, I was back then still student, uh, and I thought, well cannot build capture device from this. Uh, actually that's not true. Uh, when the first camera module came out mm-hmm. Not the first berry pie. Uh, and yes, I did that. Uh, and some schools actually tried employing that. Uh, but in the end, most nowadays actually buy their devices from, from vendors. There are big technology companies like external electronics, uh, like Epi Fund, like, uh, Adina, aec, and so on and so forth, where you can actually buy devices compatible, uh, with Opencast. And the way it works, uh, to oversimplify the whole thing is you have essentially a calendar for, for each room, uh, where you make entries and basically just say, okay, uh, from 9:00 AM to 10:00 AM there will be an event here. Uh, here's the name and select, uh, and so on. So you add some metadata and the capture device will then look at this schedule, um, and see, okay, uh, I need to start recording now. Um. There can be more integrations. Like for example, uh, we have displays showing people that the recording starts right now, we even have, uh, PT Z cameras. So like they can turn and on, uh, and they usually turned towards, uh, the wall so people don't feel like we were watching them all day. Uh, and, and they, they have to then, uh, go and turn to the direction of select trends, things like that. You, you can have more integration in that. But essentially if you boil it down to very, very simple how it works, uh, there's a schedule. Uh, the device looks at the schedule, records that, and this all happens then locally intellectual. So it can even log loose, the network connection or whatever. And afterward it will then automatically upload this to central server or. Mostly it's, it's a server cluster, uh, where then a workflow happens, which is fully customizable, um, which is great and bad at the same time because you can do very weird things then there if you really want to. Um, but, but also, uh, it, it's great because you can do stuff like automatic subtitling if you want to. You can do picture and picture videos or you can have some separate streams, uh, and so on and so forth. You can integrate your, uh, your editor where you have a manual editing step. Mm-hmm. Um, but unless you have such a manual editing step, all of this will run automatically and then it will publish the video to somewhere. And for a lot of people that is nowadays either the learning management system, which is. Is essentially a, a website where, uh, schools have their, uh, entries for courses and and so on. So all the course information are out one place and Opencast can speak to the most well-known systems and have an integration in that so that students actually have one place to find all material related, uh, to, to course including the video. Um. Or the alternative to that is we talked earlier, very briefly about Berra. So the, the video portal, uh, we have essential YouTube for your campus, so that's also alternative. However, there, there are more modules and, uh, things change from time to time, say mm-hmm. As an old time, uh, opencast, uh, developer, I still remember that Opencast could, uh, at the beginning actually produce DVDs. Uh, so you could theoretically burn that. No, no one was ever using that. It was an idea. Um, Jonathan: yeah. Lars: And, and later on people came up with different, uh, ideas where you want to publish something like that. But the most well used are either the learning management system or the video portal. Especially for if you have public videos, which schools, uh, quite often have. Jonathan: Yeah. Yeah. Very cool. Um, is it possible to use something like OBS to feed video in. I know some of the places I've seen, not exactly the education realm, but lectures and things like that, people will have a computer at the back running OBS and doing the recording there. What, what does it look like to integrate that with opencast? Lars: So, the funny thing is we actually wrote, uh, an OBS plugin, what, at one point, uh, it's not actively maintained anymore. Okay. Um, because yeah, most people just buy these devices, uh, and those people who do manual recordings, and a lot of people actually use OBS for manual recordings. Mm-hmm. If they want to have a little bit more functionality, and that's some, that's a project we recommend, uh, then usually we just tell them, yeah, it's the end. Just, just upload, just upload it afterwards. And so you can just do that. Um, but, uh, yeah, there are different implementations for, uh, these capture devices. They're also open source implementations, so you can just, uh, do that. If you want to and, and takes that and, um, it's actually not hard. It's, it's essentially an a few HEP requests to, to make if you want to build something, uh, which is special for you. Um, so whatever you, you need is, is a chancellor. You what you can count. Jonathan: Yeah. Yeah. Interesting. So you, you talked about, I usually wait for the end of the show for this, but I'm gonna do it now. You talked about that you can, with this automated system for your video editing, you can do weird things. And it sounds like there are stories there, and I, what I usually will ask at the end of the show is, what's the weirdest thing you've seen somebody do with this project? And it, it's just a, it's a perfect time to lay it out there. What are, what are some of the weird things that you've seen people do? Lars: I mean, I, I think, uh, if you're talking about the weirdest thing I've seen, then it's actually talk, which should be online somewhere, uh, of one developer who proved that the workflow engine in Opencast is touring complete. Uh, so, so you can actually build programs within the workflow engine, which yeah, you, you shouldn't, but, uh, you can. Um, yeah. And, and other than that, uh, I dunno, I think what's very impressive is always if you have these really, really big installations, uh, things like that, um, when you say, uh, weird things, then you sometimes find local integrations or workarounds for really interesting stuff because we still had this hardware lying around, which we had to integrate. And, uh, we, we have been using that for years. So we still want to continue using that. Uh. And so you, you write an integration which then pushes stuff to, I don't know, a file folder, which is then automatically being picked up by, by Opencast passed with some weird regular expressions. You can find all of that. Uh, but yeah. Jonathan: Yeah. Is, is there somebody out there Lars: using that? That's not how you, what you should do, Jonathan: that's not what you should do. Right. Is there somebody out there using open cast to record their fish tank or, you know, that sort of thing? Olaf: Haven't come across them yet. Jonathan: Haven't come. How about, how about like, uh, churches? Like, it almost seems like somebody, if they wanted to, could use, use open cast to do service recordings for churches? Lars: Uh, actually, so, uh, looking few years back, we had the, uh, the RPM repositories, a package repository where we, uh, could register and there were actually. Number of churches who did register for that. Uh, they are not active in the community, so we don't ever see them. But uh, we know that there are some. Jonathan: Yeah. Yeah. It's Olaf: a, it's a, it's a big thing in the US isn't it? Because we, we haven't seen that in Europe. I mean, there are churches these days using in general video management systems or live streaming to do that, but that's coming from the US isn't it? Jonathan: I, I think, I think it is. Um, I don't, I'm not sure exactly why that's a cultural phenomenon here and not other places. I know it did, it did become a lot more popular during COVID. Um, but, uh, yeah, it's, it's interesting. But it's, I mean, it's something that I've, I've been the guy sitting in the back running the computer before, so that's why it comes to mind to ask. Olaf: Ah, okay. Yeah. Makes sense. But we're, we're basically, we're, we're all about educational institutions, uh, admittedly, and, and we do have, um, seen some other use cases. For example, that there are more towards the video management side of things where lecture capture is not essential. So for example, the whole idea of having an archive where the original file sits is something that, that some people find attractive, think about libraries and stuff. But it's all the, the total majority is in an, uh, educational setting. I think that might change over time when, when the video portal sort of, uh, gets better known. Because that is also, or could also be interesting to smaller institutions, uh, NGOs or stuff, uh, who have videos and don't want to store them on YouTube, right? Or need them locally, uh, under their own control. But as far as opencast as a lecture capture system is concerned, obviously the pace is set by, by, uh, universities mainly. Jonathan: Yeah. Uh, that, I think that is an interesting thought. Um, one of the problems that I've seen with people that don't want to use YouTube, what they really don't like so that it can't control the ads. And so to, to put it into that same sort of religious service setting, because again, this is what I have a bit of experience with. It's like we're going to stream our service to YouTube, but the advertisement that someone's going to get in the midroll of the service is, and you know, it's just something completely off in left field. And it's like, that's not the experience that we want for our church goers. Olaf: And the same goes for your school, doesn't it? Jonathan: Yes. Olaf: I mean, ly we, we do have, we do have other systems here, uh, in, in Europe, uh, this, uh, who pursue this idea of YouTube for your campus. And one of their main selling, uh, arguments is that it's ad free and that it's under your control. What, what people can actually see. Jonathan: Right. Olaf: So I, I'm sort of surprised that some schools are not bothered with that, but, um. It still makes a difference whether you can afford to look into open source alternatives. In general, it, I'm not saying that it's completely, uh, accessible in the, in the way that, uh, YouTube and other commercial, uh, opportunities are available, but sort of the threshold has become definitely lower compared to when we last spoke. Uh, as I said, the necessity to have a system administrator who could grasp what what Opencast can do was, was, uh, obvious at the time and, and this, it has changed during COVID significantly. So smaller institutions can also afford to look into this and no one has to go for YouTube when they, when they want live streaming. Jonathan: Mm-hmm. What, what does the process look like of setting up an opencast instance? Like what did, I'm assuming this runs on a Linux server somewhere, but like what, walk me through sort of, kind of the, uh, the, the steps in the technologies that make it work. Lars: So if you, uh, just want the bare bone open cost. Yeah. Take any Linux server. Uh, we have, uh, Dian packages. We have, uh, so with official support Dian, you going to, uh, all of the Red Hat flavors? So we have RPMs or you can just run the containers, contain images we have. So if you want to run these in Kubernetes or in Docker or something like that, all works. Um, and if you just wanna start out, a single machine works just fine. Uh, you said, uh, there is a video I would there showing you how to set up an entire open cost system in half an hour. Um, so, but that can be done. I did that live. Um, and it's essentially just, yeah, at the repository, install packages at the database to that. You are up and go, uh, and, and can, can run open costs. Uh, what's more interesting is obviously if you want to scale it or if you want some integration, um, let's just assume that you want integration in your, in your rooms. Uh, we talked about that earlier. So the. Options you have is either you buy something or you build it yourself. Um, if you want to build it yourself, you need any form of video capture cards. Uh, I mean, they are quite cheap nowadays. Um, so you have these USB dongles, which you can use, um, if you want to use that. That's another question, because if we have liability, uh, so, but there are also more expensive options, I would say if you, if you want to spend the money, uh, and then you would install the, the cap trading software on that brings it to your lecture hall. Uh, or the, again, the alternative is to buy something from a, from a vendor and put that in. Um, that's integration into rooms of course. Then there's also the question about how you want, uh, or where you want your videos to end up. If you just set up single server, you get video links out of that, you get play out of that, you can, uh, take. Supplier and integrated it, I dunno, paste it to your WordPress. Uh, no problem. However, uh, you have to decide if, if you have an LMS. Um, we have for some of the larger LMS just native plugins where you can just install that. Let's say you run Moodle, you install the plugin for open cost, you enter the, uh, UL of your open cost and credential and, and that's it, essentially. Jonathan: Mm-hmm. Lars: Um, the more you want to integrate, of course, the more complex it goes. Um, the other side, as I said earlier, is, uh, scaling. Um, if you are a small school, you have a few videos a day. A single service is just fine. Um, if you do have more. So, I dunno, we have, uh, sometimes. 12, 14, 20 recordings a day. And these recordings are not five minutes. These are usually two hours. Uh, so there's actually a lot going on. And, um, yeah, if you want to process all this video, um, what you want to do is, uh, is to scale. And that's relatively easy with saving cost. You, you can, instead of setting up a all in one system, just set up a number of systems. And there are different flavors like the admin node, uh, the presentation note where the video ends up, uh, and the player is still delivered from. And then there are work nodes and you can essentially have an arbitrary number of worker nodes depending on how many videos you want to process. Um, but you don't have to start with setting up, I don't know, 50 machines with all your workers because just that small, uh, and you can always scale up later. Jonathan: What, what's the underlying technology that makes this work? Is it a, you know, does it run on top of Apache? Are we talking about a big Java server thing, or, I hope not. Lars: Yes. It's, uh, so, uh, I'm sorry about that. Uh, but yes, it's actually an, uh, enterprise Java stack from, uh, I mean, open Cars is, is quite old. Uh, so we started, uh, 2008 or nine, I think was a 1.0 release. Um, that's, that's a while ago. Uh, and the underlying technology has, uh, shifted slightly over the years. But essentially it's, uh, Java, OSDI app. Um. With actually I think about 180 or even more, um, services. So it's a microservice architecture. Mm-hmm. But yes, it's, it's Java, so is dry. And, um, with such an actually sometimes quite old infrastructure, it's, it's actually interesting. Sometimes though, uh, since I was around for a long time, you see the new developers coming in and looking at that and ask why the hell is, is it working like this? Olaf: I can, I can Lars: tell you makes huge sense at Olaf: all. I can tell you why it is LAR because at the time when we were trying to get some funding, there were two words which had to be included into the funding documentation, and that was OSGI and microservices and that was the thing at the time. So. Like service oriented architecture. If you haven't got these buzzwords in your, in your funding, uh, purpose, then you won't get the money at the time. Jonathan: So how are you integrating AI into your function, your, your program now? Olaf: Um, like everyone else? Uh, we're trying our best. Jonathan: That's, Olaf: no, it's, it's, Jonathan: that's about half a joke because that's the buzzword. Yeah, Olaf: it is. Jonathan: But I'm also curious if it's something you're looking at. Olaf: I mean, it's pretty obvious because, um, when you look at these two hour videos, there's a lot of metadata in there. You can work with it, it's. Basically, um, we will see, um, a presentation in these videos. So that's words, that's content you can use over time. Jonathan: Mm-hmm. Olaf: Like, uh, metadata over time. And then there's obviously the, um, spoken word which is, uh, transcribed and can be contextualized with the content. And that's the ideal basis, I would assume, for any AI system to work with, which is why, for example, the use of that particular video to create summaries or to create shorter versions of that video has been, um, presented as a, as a use case in, in the Opencast context. We have also set up a small video portal where you can do quizzes, uh, based on the, on the, um, video content. Obviously, like the AI driven tutor. Uh, the university is not able to pay for anymore because they invested into ai. Um, and that's the most obvious things when it comes to, to learning. I think that in general, our metadata should go into the whole ecosystem of AI learning because the metadata is so interesting. The connection between the slides being shown and the, the, the, the speaker talking is, is good metadata to use Jonathan: for, for Olaf: training. Um, for training for example. Yeah. Okay. Um, on the other end, there's the back end of Opencast and Lar and I were just talking about that. If you imagine that recording we're doing is two hours, but we're starting a couple of minutes before the lecture actually starts in order not to miss the start. Jonathan: Right. Olaf: And there might also be a break in the record in the lecture, which we want also to be edited. Mm. And we want the end, which is also longer than, uh, the actual recording to be edited also automatically. And we are currently using FM PEG for that, if I'm not mistaken, which is Jonathan: every, everything is FFM peg. Even if Correct. Even if you weren't using ffm peg, it would still be f fm peg doing the work. Olaf: Correct. And we're happy to use every, uh, FFM pick, obviously. Yes. Olaf: Um, but obviously, we presume, and we will sort of, uh, have a, a test case for that as well in the near future, that with ai it's easier to do that. And last you can, you can tell them what you think might be done in that context in order to better detect when the actual lecture starts, when there is a break, and when the actual lecture finishes. Lars: Yeah. Uh, to, to tell you a little bit about this. Uh, so what is AI is also a good question because lots of things we are talking about right now. You could also, you would have called the media analysis five years ago, or 10 years ago. Um, but, uh, for example, uh, to take what OLA just suggested is, uh, that we automatic, uh, have automatic subtitles for, for everything. And so instead of going and, and using just the audio levels, and that's what we used with, uh, FFMP filters before, um, and say, okay, the, the audio became quieter here. Um, let's just cut this away. That doesn't always work reliable because people are. Running around in the lecture hall, it's before the, the lecture actually starts and, and talk and whatever. Mm-hmm. However, uh, if you actually look at the transcription, so the, the subtitle, which you can do automatically now, um, it actually comes a lot easier to detect, okay, here we have coherent speech and before we don't. So we can cut kind of automatically there. And the second thing is what we usually do. So we could just say, okay, let's cut this completely automatically. Um, what, what we can also do, and what most often actually do is to just generate suggestions, um, where someone can actually take a look at this. Just say, oh yeah, that looks fine. Mm-hmm. Go on. Um, and, uh, yeah, if, if someone detects it, uh, but it's not fine because I don't know if they have a private conversation at the beginning or whatever, um, which can happen because people forget. Uh, yeah, that's, then they can still cut that away. Jonathan: That's actually an interesting question that, that I thought about earlier, and I wasn't sure whether it applied or not, but there are, we, we've entered a realm where there are multiple new privacy laws that, uh, you know, I'm an, I, I am, obviously, I'm an open source and a privacy enthusiast, but I would say that some of these privacy laws are insane, um, because of the things that they attempt to require businesses to do. Um, like for example, the right to be forgotten. I, I, I have, I have problems with that, but anyway, that's not what we're here to talk about. Um, but do you guys find that, uh. That your project runs into any of those privacy laws, like in unexpected ways. So for example, this idea of you accidentally caught a private conversation at the beginning of the lecture, I imagine that becomes even more legally interesting if those are minors that you're recording. Uh, and, and does any of that, um, does any of that liability stick to the project or is that all in the realm of the, uh, the universities or whoever is, is implementing, you know, actually running the system? Lars: So, uh, two things here. The good thing is that, uh, a core. Base of the committers actually comes from Germany. So we had very strict privacy laws before the other rest of the world had very strict privacy laws. Jonathan: Hopefully you had Lars: enough Jonathan: time to get them right. Ever Lars: closer. And it was quite, quite interesting early in the project when, uh, someone from North America implemented something and we just said, sorry, but we can't deploy that, but that won't work for us. Yeah. Um, so, uh, with, with that in mind, we actually have a lot of. Things, uh, to, to make sure that, um, something like that doesn't happen. But at the end of the day, it's up to the institutions to, to work around that. And for example, for just to, to get away from the project, to go to our institution and how we run that. We are working very closely with our privacy officer and we have things like, there is recording sign at the door. If you enter that, that, you know, okay, this can be recorded. If you, uh, book yourself intercourse, which will be recorded, you get notified in advance. Um, and also I said earlier that there are different integrations, like the cameras turning away or, uh, there is a display which actually shows you that the next recording will begin in. 10 minutes, uh, and then it turns red and you, uh, see that okay, the recording starts right now. Uh, so there are lots of things you can actually do and, and depending on your needs and, uh, what you hash out with your privacy officer, essentially, uh, you can, can work with that quite well. Jonathan: Yeah, I, I appreciate that you guys have, uh, worked to sort of get that right and take the, obviously what's legally required, but also it sounds like some common sense things that should have been there. Uh, I've, I've seen instances where open source projects are required to collect more data about people in order to be compliant with the privacy laws. Like you've gotta be able to delete, uh, everything about a person. They have to be able to send these deletion requests and it's like, well, we didn't have anything that we could have deleted, but in order to comply with the law, we're going to have to collect this information to be able to delete. I know it sounds silly, but. It's things like that actually do happen, uh, because lawyers and laws and the people writing the laws don't necessarily talk to the people that will have to live underneath them. So, uh, I'm, I'm glad that that is, uh, something that you guys have already been aware of, worked out, and, uh, it sounds like have some, some reasonable solutions in place. Uh, I've got a, I've got a note here to ask about technological issues with being mature open source software. Is that, is that mostly just you're running in Java or is there more to that? Olaf: There's more to come. Jonathan: There's more to come. What, Lars: what I said earlier that it, it's interesting to be, uh, the old guy and, and people ask you why this insane thing happened, and the reason is just. Because, uh, I mentioned earlier that open cost could create DVD images. Mm-hmm. No one needs that anymore, but the had ability built in and then later on people stacked misuse. Let's just label it like that for something else. Right. And, uh, it drifted farther and farther away from, from where we are. We Jonathan: can't, we can't remove the DVD burning code because it's going to break the new AI thing that we just added. Yes. I'm very aware Lars: with how that works. Essentially that, and I think we are not necessarily bad at, uh, throwing old code away. Uh, as a developer. I really love doing that. That's also always so nice. Yes. Just throws this big old thing away and have something shining in you. Um, Jonathan: yes. Lars: But uh, yeah. Right now, for example, we are half. Uh, definitely a, a bigger discussion about, uh, how to restructure, for example, data model and restructure. Data model means a lot of things needs to change. Uh, but on the other hand, this is something that is kind of required. I mean, it's not necessary required to get the functionality because the functionality there. But, uh, if you look at, uh, technology and how that changed, for example, storage technology, uh, is that nowadays S3 is, is a storage standard basically that everyone, uh, has and, uh, that becomes deployed more and more even in, in local instance if they want to use that. And integration in opencast is, let's just say a little bit sketchy. You, you can use that, but you have some certain drawbacks because of decisions that were made. 15 years ago. Um, and that's something, uh, we, we need to change and uh, something like that can come up from time to time. Uh, another would be we have an os GI based Java os GI based, and we used some spring components back in the day. They work pretty nicely with, with each other. Uh, at some point, spring project decided to not use OSGI anymore, which is quite challenging if one of your core libraries doesn't work with one of your other core libraries anymore. And you can work around that, but sometimes you just have to, to, uh, yeah, redo something in, in the backend that no one wants to see. Um, fortunately for this specific instance, we actually got state funding for that, uh, which is really nice. So we have a few years to, to get this right. And we have a few people working on that, uh, which. Uh, I really, uh, appreciate. 'cause something like this is really hard to sell to, uh, yeah. A single institutions or even a couple of institutions because no one sees that anything happens at the end. You have the exact same thing. Now it will make everything in the future easier, but that's not what the decision makers necessarily see. So, uh, yeah, working with technical depth is always challenging and, uh, working with, with mid code base, which is quite out at this time, sometimes it's quite interesting. Jonathan: Yes. Lars: Um, but at the same time, it's also good to, we have, uh, new committers that, that are relatively re or is it relatively recently joined program. Olaf: Mm-hmm. Lars: Um, and we have, yeah, I would, folks like us, um, who have been with the project for quite a long time. It's good to have both because you see that new ideas coming and they want to change something. Um, at the same time, you have people who actually can explain, yeah, that's there for this reason that they expect 10 years. Jonathan: It's not a good reason, but there is a reason. Yeah. I I Sometimes Lars: it's even good reason. Jonathan: Well, sure. Um, Lars: but yeah, that, there's definitely a reason, Jonathan: yes, Lars: if it's good or bad, but no one knows. Jonathan: Uh, I've, I've re I've recently, um, been able to start a bit of new employment working on a code base, and, uh, I also set my own, uh, I, I basically set my own priorities for it, which means that I've got to go in and kill some of that old tech debt. Like, you know, all of these devices have to have this particular pin defined. We don't do anything with that pen, but if it's not set compilation, breaks. Let's figure out why and actually fix that, and was able to do that with a couple of different, different things. And it's so satisfying to finally send the pull request in and get it, get it merged. Lars: Yeah, exactly. Jonathan: Yeah, it's great. Um, so you talked about having new committers and we, I asked you about adding AI on purpose, but the other side of this that I've gotta ask about is, have you seen the, the AI slop problem? Have any of your new committers been getting a little overeager with the, uh, LLMs in writing their pull requests? Lars: Uh, fortunately I would say for pull, not necessarily, uh, they still. At least seem good so that they actually, uh, seem to do exactly what they should do. Uh, I mean, at least it, it's not worse than before this box always happen. Uh, but we have a review process in my, in that, so, uh, I I don't think that we've really seen, uh, anything worse with that. Where we actually did see that is that we have a way of reporting security issues and stuff like that. Uh, and we get a lot more notices in that. And a lot of these notices are just Yeah, the bogus. Jonathan: Yeah. Lars: A AI slot. So they're very well written. Um, and essentially at the end of the day, it's something like. Uh, yeah, we can reproduce all the ai, uh, all the, uh, API calls to this use default password, which is at bed at Opencast. And you look at this and you think, well, yeah, that's our demonstration. So, which is reset on a daily basis. So people are supposed to be able to use that. They're supposed to, uh, use the, uh, the default credentials, which are Right, even written on, on the landing page if you, if you go to that server. Uh, and it's a surprising number of that, what, what we actually got. Jonathan: Yeah. Um, is there actually a bug bounty for the project, or is this just people looking to farm essentially? Reputation? Lars: Uh, it's actually just people farming some reputation. We don't have a bug bounty program for that. Jonathan: Yeah. Um, yeah. Uh, something I've, I've seen, um, people begin to talk about is that AI will probably kill bug bounties in the long term. Um, it just, it's not gonna be, it's not gonna be sustainable. I've actually seen at least one project go so far as to say not just, not just issues, but to close pull requests from anyone outside the project, because they were getting so much of this. They put out a statement and just said, we're not gonna, at least, I, I think the way they put it was, until GitHub gives us better tools to deal with this, we're just not going to accept poor requests from anyone outside the organization. And it's like, that's kind of scary that we've hit that point. Um, and I'm, I'm really glad to hear that you guys are not seeing that. And I think maybe the fact that you're in such a, a niche, such an academic niche has sort of sheltered you from some of that, but hopefully it's not coming to you because it is a pain to deal with. Lars: Yeah. If you take a look at any of the talks from the main coal developer, for example, uh, it, I don't think it's actually nice to be in his shoes for the last. Two or three years. If you look at back bounties and things like that, he spent lot of time on that. Uh, his talks are very good in, in that direction if you're interested. Uh, but, uh, it's not nice that, uh, how he had to work with that. And he actually killed his Back Bounty program, I think a few weeks ago. Jonathan: Yeah. The Dan Steinberg, we've actually had him on the show before. Um, and yeah, I, I saw that, that they, they were one of the ones that, that they killed the, the bug Bounty for, you know, curl is in the running for the most installed software in the world. Right. Because it's in so many different places. And, uh, Dan, Dan did something. He, he is, he's humored about it now, but he put his email address into the curl license and, uh, you know, this is, he, he gets emails from people completely off the wall, and I, I bet he's now getting emails from. LLM agents that are completely off the wall for similar reasons, you know, but he talks about like if somebody's in their car and they're poking through the infotainment system and they're looking for help, they're looking for an email address. Well, one of the places that they might be able to find an email address is in the, you know, the, the, the terms of service, the licenses, the curl license, his email address is gonna be in there, in the license. And so he will get, please unlock my car for me. Poor guy. Yeah. Yeah. Super interesting. Um, so on the other side, do you know, are your developers actually making appropriate use of AI to write some of these new features? Like, I, I, I'm not ashamed to admit, I've got a, uh, through the, the, the, through the project that I mentioned just a minute ago, I've got a copilot. Uh. I dunno if it's copilot Pro or whatever they call it. We've got the, the copilot subscription and you know, I use it not a whole lot, but I use it some, like the, um, the code completion engine is the one that I found useful because it's like I, well either I need to go search on exactly how that function call goes, or I could just start typing it and copilot will go. This is probably what you mean. Yes. That's what I mean, hit the tablet like, so is is that something the, the developers there're making use of? Is Lars: it, I can't tell you if, if, uh, on, on the person by person basis if they using AI or not. That's, that's up to them. So we had a, a reasonable discussion about, uh, what we expect from people if say UCI and, and things like that. Um, but other than that they can, and to be quite honest with you, I mean. Some of them will definitely, uh, use ai. Mm-hmm. Some of them will absolutely not use ai. Yes. Uh, I, I, I mean, it's a very controversial thing and, uh, at the end of the day for us, it's just okay. Uh, if it seems no, if, if it goes through a review, uh, and it it works, yeah. That, that's okay. Uh, we had some additional discussions about, for example, if you, uh, are not just using the auto complete for, for small things, but actually have some larger code written by AI or something like that, you should label that. Um, and that just makes it easier for everyone else. Um, but, uh, do I know any pull requests which was written with ai? Yes, I do. Uh, I'm absolutely certain about that because we did play around with stuff and we had things. Well, we asked copilot to try and fix something, which at least one time worked well. Other times it didn't. Jonathan: It worked once. Lars: So, yeah, so having fun with that. Definitely. So people need to know how this works, I think. Um, and then they have to decide if we wanted to use it or not. Um, and hopefully they will use it responsible. I mean, yeah, I'm, I'm probably a little bit too made for that and they won't use it responsible, but I, I hope they will. Jonathan: Right. Yeah. I think that's where we're all at. We hope our people use it responsibly. Uh, alright. I, I've got a note here that apparently there are some organizational challenges that have popped up and I'm curious what, what that is. What, what are, what are some of the challenges that as a, as an organization you guys have faced? Olaf: I think you should add, what are the challenges you face as a small organization? Mm. Because with the number of committers you can manage, imagine that everything else that has to be organized is organized by a small number of people as well. Yes. And most of them come from, from Europe these days. Uh, from, from German speaking countries in particular. Mm-hmm. Now you have to remember that we originate from the us So uc, Berkeley started the whole open cast thing. Oh, okay. But over time there were, uh, less institutions in the US interested in this, which is somewhat interesting with respect to the fact that the open source alternatives seem to be more popular in Europe these days in general. So for example, if you think learning management systems, I think that Moodle, for example, is very popular here as an open source. Um, LMS, uh, and not so much in the, in the Northern American, uh, uh, uh, area and the US in particular. And that's the same for Opencast. So we had, uh, committers and contributors and users, adopters as we call them from the, uh, states, uh, in the, in the early days, but they sort of dropped off. Uh, it was actually. Pretty difficult to sell an open source video management systems to universities because you had that whole system of procurement and, and you had to provide a certain level of service, right? Which was difficult for a, for a small project to actually realize and submit, like something, uh, comparable to what the commercial competitors can commit as a paper is difficult, so yada, yada, yada. Uh, we basically, uh, are not, uh, that present in North America anymore. And as a consequence, everything else happens in Europe, which makes financial and organizational transactions are very difficult. So we are based with Aerio, which is, uh, a foundation in, in New Jersey. That's for historical reason. Mm-hmm. It's a very good, um, umbrella organization for us. So there are a number of small open source projects in, in, uh, aerio, which basically all share the same idea of providing open source software for, for higher institu educational institutions, which is a perfect fit for us. Problem is that they are all, or Aperio is sitting in the US Jonathan: right. Olaf: So if we have our conference in, in, uh, the Europe, uh, say Spain, and we have a conference dinner, the payment has to go via the US in order to pay the restaurant in Spain, which is quite difficult and painful and expensive as you can imagine. Yes. So over time we decided. We are becoming or have become a more European project and that we sort of had to, to move on and, and sort of rebuild that in our organizational structure. Mm-hmm. Which is why we, we left or are about to leave aerial, uh, which is still a perfect fit for us when it comes to the whole idea of, of open source in, in education. Uh, but we're now moving to Linux Foundation Europe because, uh, it seems to be, uh, a fit as you heard in, in a technol technological sense. And they also have smaller educational product under their hood, uh, still allowing us via Linux Foundation to serve North America in case that's something we want to do. So that's what we're doing this year, basically from an organizational perspective. Jonathan: And that's your, that, that's your fiscal host, right? That's essentially who, that's, yes. So you, you guys don't, for those that don't know. Most open source projects, and I'm gonna speak at this from a US perspective because I, I know the business law here much better than Europe, but most open source projects, you know, they may have some funds in, in often cases, not enough funds to really be worth trying to build their own, you know, LLC uh, trying to become an actual nonprofit is very difficult in the US specifically because, and I've, I've been told this from another, uh, another open source project writing code is not considered to be charitable activities. And so that makes it really challenging to try to get nonprofit status for an open source project in the us. So, because of all of these, what we have is these things called fiscal hosts, which is like an umbrella organization. They hold the, you know, the actual like, um, they are the LLC or you know, maybe there'll be a subs or a C corp or, or a, a trade corp in, in many cases. Um, and then the individual process projects will just. Exist underneath them and, and they sort of get the coverage for the, all of the, the business sort of things and, you know, the whole bank accounts in lieu of the project, that sort of thing. So, referred to it as a fiscal host. I, I, I thought everybody in open source knew this. And then I had a chat with a buddy that's been doing open source for years and years, and he's like, what's a fiscal host? Everybody does not know this. Okay. Noted. Olaf: Yeah, but the, the situation to describe is exactly what happened to us. So, so when we looked into this, we also thought that we might establish something on our own, which basically turned out to be impossible at the, uh, at the size we are. So there are illegal entities in, in Luxembourg and Belgium in particular here in Europe, which some of the larger institutions are sort of using, and Linux Foundation Europe as well. Mm-hmm. Uh, but that's not something a small project like ours can basically use. You either have to pay too much or you have to organize too much and know too much about the legal entities and the resulting financial, transactional organizational implications in order to deal with that. And that's why this whole idea of having an umbrella institution for smaller open source projects makes. Perfect sense for me. 'cause then there's someone who makes sure that our license is good, our trademarks are good if you have them. Uh, the financial situation is good. Someone is taking care of that. Someone is looking into, uh, other fiscal activities and so on. It's just a shame that there is that sort of difference between the North American and the European sphere, which we're leaning towards Europe these days. Jonathan: Yeah. In in, in the US at least, it is almost trivial to set up an LLC. Like someone that knows what they're doing can get one going in, in most states in about 30 minutes. I mean, literally it's, you can get something up that fast to be able to go to a bank and set up a bank account. Uh, I imagine, I imagine in Europe it's probably a little bit more complicated, maybe a little, little higher barrier for entry. Uh, one, one could, one could disagree. Which system, which approach to that is the better one? And probably just depends upon the, the scenario you're talking about, but it is a bit different. Olaf: Yeah. We, we actually did ponder that as well, but decided against that too because. All in all, uh, the size of the project doesn't allow for us to, to spend too much, uh, effort and resources into organization. And that's why we think that, uh, being part of a larger organization makes perfect sense for, for us as a small open source project. Jonathan: Mm-hmm. Yeah, absolutely. Most, well, I dunno about most, a lot of open source projects take this path of using one of these fiscal hosts. Um, it's, it's very popular for the reasons that we've talked about, right? You, you've, you, you almost have to have something like once you hit a certain size, once there's money involved, you've gotta have something, um, just to keep ev keep everybody outta trouble. So that makes, uh, makes sense to me. Uh, Lars: yeah, I mean, um, you, you don't, you have to have something like that. And honestly, the people involved with the project usually don't want to spend all of their time dealing with money related things, right. And more related things and, and so on. I mean, that's no fun. Jonathan: Yeah, to be able to do it in-house, you've almost gotta be big enough to have full-time employees, right? And so you've gotta have, you can have somebody like me that sort of knows what they're doing and can do it full-time. Um, and even, even at that, the project that I'm in, we still have a fiscal host for the open source side of things. So, you know, it's just, it's a, it's a lot to have to deal with. Alright, let's, uh, so we talked about one of the plans that you guys have for 2026, and that is coming under the Lennox Foundation, Europe as your umbrella organization. What else is coming in 2026 and in the future? Olaf: So before last talks about the various project he's involved in. I'm just gonna mention that, that video portal again, because it's, it's pretty important for us in a, in a strategic sense in that we're still moving towards getting open cast into the direction of, uh, medium sized and small sized institutions and maybe even institutions beyond, uh, the, uh, universities, uh, we are currently, uh, serving. Um, so that's one of the main focuses for us from a, from an organizational, from a marketing perspective, to put, uh, the video portal to Vera in, in, in the front space in order to make people realize, well, I'm using this video portal and there's a open cast beneath, which I don't see. Um, and that's pretty, pretty interesting for, for those smaller institutions. So that's from a board perspective, we're having a a, a board to sort of, um, share the, uh, strategic perspectives of Opencast. That's pretty important for us at the moment. And last, can talk to, uh, a number of the other projects we are looking into for the next couple of months and years. Jonathan: Mm-hmm. Is, is the, sorry, just before we go to that. Is, is Tabera the name of the video portal? Olaf: Yes. And it seems to be a portal in Japanese. Jonathan: Ah, okay. I, I wonder where that came from. So I, I've, I've navigated to tobia.open cast.org and I'm seeing lots of, uh, lots of cat videos, which the internet is for, right? Olaf: It's supposed to be for CAT videos. Yes, that's correct. That's why we have the internet. That's the, if you stumble upon a dog video, that's my dog. Actually, Jonathan: this is, this is interesting. Um, I could see, I could see this. The video portal 'cause it looks good. I could see this taking on life outside of just the, uh, even just the educational sphere. Um, 'cause it, businesses wanna be able to put videos up somewhere and I don't know that any of the existing solutions are really great. So, I mean, this, this could have some legs for you. Let's see where that goes. Alright, sorry to interrupt. Continue. I Lars: kind of, kind of hope, uh, that, that happens at least for smaller schools and so on, uh, by the, by, uh, to be at 11, that's the test server. If you want to actually see if you are conference recording, so a kind of production to be, you can go to, uh, explore opencast.org. Olaf: But he can, he can upload his CAT videos on the other demo server. Jonathan: There you go. Lars: Exactly that. That's, uh, what opencast is all about. CAT videos. Um, actually we have internal channel called Open Cats, uh, as. As a twist of two, uh, characters. Jonathan: Nice. Lars: Um, yeah, going for, for a reasonable additional thing to happen in 2026. Uh, we already talked about, uh, backend infrastructure, data model stuff. Uh, really boring, but really interesting. Uh, sorry. Really important. Um, Jonathan: interesting Lars: to, interesting to a very strong Jonathan: group of people. Right, Lars: exactly. Um, so that's hopefully happening. Um, but we also will hopefully see a few other things. For example, uh, we have this editor with. Briefly talked about, this is not only for for video editing, but it also has things like subtitle, editing and so on. And there will, uh, be a few extensions to that. One is that there will be a new chapter editor. Uh, the other thing is, uh, that want to have, uh, an interactive video editor. What that essentially means is that you can bring quizzes or additional information or something like that, uh, into your video. Um, and you can have, I dunno, your video pause at some point and ask, uh, you, you know, um, in, we just talked about ancient Greek and, uh, what is the name of, I don't know, uh, the, the capital of Sparta at that time, which hopefully anyone can, can then answer. Um, yeah. And. Things like that is something that, uh, hopefully, uh, will improve, uh, things a little bit towards a learning experience, um, related to, to these educational videos. Um, there's also things when, when you look at hardware and coding and things like that, um, essentially this is a side effect of, um, of AI because there are now lots of GPUs at, at institutions. Um, and well, you can can use them if we just lose Jonathan. Jonathan: No, we, I can. I've got you guys still. Olaf: He's there. I can see him. Jonathan: Okay. Lars: Oh, good. Olaf: Go ahead. Lars: Um, Olaf: you've got GPUs lying around in Osburg, obviously. Lars: Oh, we, we got some GPUs lying around in Austin Road. Yeah. Um, yeah, so that's definitely something that's happening in 2026. Uh, hopefully. Um, and then there are some, Jonathan: oh, now I've, now I've lost him, I think. Olaf: Yep. Jonathan: Yeah, I lost him for a second. Let's see. Now we've got him back. Uh, give me that last 30 seconds again. Lars: Okay. Okay, uh, let, let's just start with, um, the AI integration then. Um, so what we've seen in the last few years that, uh, people deploy a lot of, uh, GPUs. And so what we can do now is actually also reuse these resources, the GPV resources for things like hardware, ENC coding and things like that. So that's, that's also interesting. Uh, and of course, and what I've already talked about this very briefly, um, there will be some form of a AI integration. Um, I'm almost certain that that will happen in some form or another. So I think that's the broadest strokes of 2026, uh, for Opencast. Jonathan: Yep. Now, I've, I've gotta say, you sent me down a rabbit hole because you said the capital of Sparta and anybody should know that. And I went, wasn't Sparta City State? Isn't smart, the capital, it's, yes. Wikipedia agrees with me. I'm not losing my mind. Trick question. That's, that's terrible. Lars: Hopefully Olaf: everyone knows that that's the best interactive video you can have with a trick question, Jonathan: I suppose, makes you go and look up the article on it and read about it. Yeah, it's fun. Alright, um, yeah, let's see. We're, we're getting close to the end of our time and, uh, it has been a, a super fascinating conversation. So let me ask you guys this, is there anything that we didn't talk about that we should have? I know we've talked about a bunch of different things. You gotta think back through all of it, but is there anything you, you, anything burning that you really wanna get out? Lars: Um, maybe very briefly, if you happen to have, uh, be in the United Kingdom, uh, there will be the international Summit, uh, at the end of this months. So if, if you want to be us in Paris, uh, 24th, 27th of, uh, February in Manchester. Jonathan: Ah, very cool. Yeah, I will be, I will be in Germany just a couple of weeks after that. Uh, I'm gonna be at an embedded world there in Nuremberg, so very fun. Uh, I don't know if anybody that knows me is gonna be close enough to try to do any sort of meetup, but I'm letting folks know that I'll be there. Wait, you can wave at me as I go over. Right. Uh. All right, so let's see. I already asked you guys what the weird things were that people did, and you told me nobody was using it to watch their fish, which I was a little sad about Somebody out there. You need to set this up just to have an aquarium, live aquarium cam. Uh, but what now I, I do have a couple of final questions. I am required to ask each of you, and I do get in trouble if I don't. You guys probably remember these. Uh, let's, let's start with Lars, and that is, what's your favorite text editor and scripting language? Lars: So, I definitely said Wim last time, and I think I still go with Wim because I actually use it quite a lot still. Uh, I, I do use other stuff I now, so it's not remotely, uh, since back then, but, uh, I, I still use it quite a lot. So let's go with that. Jonathan: Alright. Lars: Uh, and for the scripting language, it's probably Python. Um, I actually use quite a lot. Jonathan: Yeah, that's a very, very common, very common answer these days. Uh. Alright, and uh, Olaf to you as well, same two questions. Text editor and scripting language. Well, Olaf: what I thought that you might have realized by now that I'm not able to tell you what my scripting language is because I'm obviously not, uh, a developer and not into the technical side of things here at Opencast, which is why my text editor still is Microsoft Word because we are a Microsoft school here at it Zurich and not proud of it. Jonathan: Well, okay, I guess we'll take it. Uh, gentlemen, thank you so much for being here. It has been a blast to get to talk to you both again after what, seven, six and a half, seven years, something like that. And, uh, you know, give it another decade and we'll have you back to talk about everything that's changed since then. Olaf: Thank you very much for that. Alright. Jonathan: Thank you guys for Olaf: having us. Jonathan: Yep. Appreciate it. Olaf: Bye. Jonathan: Alright, so that was open cast and uh, as we said, very academic focus, but particularly I, I, I went and looked at that video portal and very interesting. I, I really do have a feeling that that'll be interesting to, uh, other people outside the academic world. Uh, maybe not the fact that it's all in the, in Java backend, but who's to say Lars: the video portal is actually written in Rust? Olaf: That's, that's something I was thinking about. Jonathan: So I'm gonna have to do some edits anyways, which we'll cut this in. Tell me about this, Olaf: because, because the young, the young folks have taken over open cast and they decided to build in rust. Jonathan: You should have written it in rust. Was that, was that sort of a, a stressful or controversial thing or was everybody sort of okay with it? Olaf: I think it got unnoticed because we started the project during the pandemic and everyone was busy with something else. Jonathan: Shh. Don't tell the old guys that we're doing it Rust. Olaf: Exactly. Jonathan: Uh, that's Olaf: great. No, actually it, it, it's, it was actually part of our policy, wasn't it? To, to be open towards new technologies and languages in order to attract new developers. I'm not talking young, but new developers. Is that correct? I know. I remember the discussion we had Lars: looking at different new, so new projects or parts actually used a new thing. So we have a lot of front end written as a React app, for example, or something like that. So that's actually somewhat new. And back then the original was vanilla, JavaScript and jQuery and things like that, that at some point got replaced with Angular that at some point got replaced with React. So we are actually somewhat good with replacing things like that. But looking at the core, it's just so hard to just say, you know what, this hundreds of thousands of lines of code just. Let us throw that away and, and replace that. Yeah, that's pretty hard. So that's probably staying Java for the foreseeable future. Jonathan: Yeah. Lars: Uh, but for site projects, yeah, it's, it's different languages. Jonathan: Uh, anyway, was a great conversation. Appreciate it with the guys. Uh, next week. We are open next week, so if you know of a project that we need to talk to or you have one, let us know. You can jump on Discord or you can even email us, floss@hackaday.com and uh, we will get that scheduled. Other than that, just appreciate everybody being here. Thanks for those that watch and those that download, and we will see you next week on Floss Weekly.