FLOSS Weekly

Episodes

• 
  Episode 878 transcript 05.08.2026

  FLOSS-878 Jonathan Bennett: This week I'm talking with Jonathan Pallant about Rust and primarily in the embedded world, the Raspberry Pi Pico, the Nordic, Espressif, all of your favorite little MCUs. You can run Rust there and partially due to efforts by people like Jonathan. This is Floss Weekly, episode 878, recorded Tuesday, August the 4th. A tool with opinions Hey, folks. It's time for FOSS Weekly. That's the show about free, libre, and open source software. I am your host, Jonathan Bennett, and today, once again, we're gonna get rusty. Whereas last time though, when we talked to someone with Ferrous Systems, we were talking about Rust on the Linux desktop, in user space, Rust on general purpose computers, and today we're talking about Rust somewhere a little different. I've got Jonathan Palant with us today. He is also from Ferrous Systems. In fact, I think his day job is teaching Rust there, and I think also doing some programming work. But let's get him on and ask about all of the places where Rust runs that you may not have thought about. Hey, Jonathan. Welcome to the show. Jonathan Pallant: Great to be here. Thanks for having me. Jonathan Bennett: Yeah, it's good to have you back. Or no, you for the first time. Good to have Ferrous Systems back. The second interview with one of you guys in a couple of months now. All right, so this, this magical thing called Rust it runs on the desktop, it runs on the server, it also runs on the Raspberry Pi Pico? Jonathan Pallant: Yeah. It- So joys of having a systems programming language, right? You get a compiler, source code goes in, machine code comes out, right? So anything that runs the machine code should be able to run it. So yeah, Rust is really interesting in being able to scale from the very, very large. It is the language that powers Amazon Web Services, and Cloudflare, and Google, and a whole bunch of other stuff. But it scales down to, as you talked last time, desktop applications- ... things like Sudo written in Rust. But also scales down to the small, very small. How small is always an interesting question, but- Jonathan Bennett: Yes, it is. Jonathan Pallant: I have run Rust on ST Micro, Microcontroller, Arm Cortex-M4 32K flash, 4K RAM. As modern things go, that's pretty small. I'm not sure I'd be happy squeezing Rust-generated code onto like a Z80 or a 6502. That may be a bit too small, but certainly tens of K of flash, a few K of RAM, I think- ... is quite workable. Jonathan Bennett: I happen to have here the Seeed Studio Xiao board actually their next generation Xiao board. It is a Nordic nRF54. The word Xiao in Chinese means tiny, and it is an appropriate name, as you can see. T- Can we do Rust on a nRF as well, on, on Nordic? Jonathan Pallant: Yeah. So Nordic semi generally use Cortex-M4 M0 in the nRF51, M4 in the nRF52. I did a lot of work with their cellular modules. Don't know if you've ever used a 9160? Jonathan Bennett: I've not, Jonathan Pallant: but- the very small device that does LTE Cat M- ... so the, the IoT version of 4G, and that comes with a Cortex M33 that you get to play with, and then a secret second processor that runs the modem firmware- ... that you are absolutely not allowed to play with because cellular. Jonathan Bennett: Yeah. Okay. So the ... Boy, the, the embedded world is an interesting world. There's a large there's quite the gamut of different things that count as embedded, and sometimes the lines get a little blurred, right? Because you can take the Linux kernel and run it on an ESP32-S3. Now, it's not a fun experience, but you can do it. It'll boot. It can be made to, it can be made to happen. And so i- it's interesting to me that Rust goes all the way down to some of those very, very small targets. How does that work? Is there an operating system that a, a Rust binary is then running on top of? Is there some kind of real time operate OS? Do you guys run on Zephyr, or is, are you literally just writing Rust that runs on the bare metal? Jonathan Pallant: Yeah the output of the, the Rust compiler, it works like a C compiler. Source code goes in, machine code comes out. So the compiler doesn't really care whether- ... you have an operating system or not. You could absolutely be the first code that boots when the system comes out of reset. Jonathan Bennett: But I mean- Where- ... there's like the question of am I making system calls to a kernel? Am I telling the kernel to do, the, the multitasking stuff, or am I just always, is the binary always running on the Jonathan Pallant: CPU? And where that comes in is less in the, the language, but more in Rust's choice of standard library. People say, "Ah, the standard library," as if it's one thing. But as with all good things, it's actually three things in a trench coat. The, the top part of the standard library in Rust is the thing that knows about files and threads and networking, and that's the thing that's gonna be operating system specific. And it's basically the Rust equivalent of hash defines that say, "Hey, if you're on Windows, we're gonna call this Windows function, but if you're on POSIX, we're gonna call, open read and write." Sure. And that's what lib std is doing. It's this operating system abstraction layer. As I said, file system, networking, threading. Below that you've got the heap allocator and all of the heap allocated collections. They actually live in their own separate library. If you're a Windows, Mac, or Linux user, you may not be aware of that because it's hidden, and everything heap-related is all exposed at the top level anyway. But there is this lib alloc, so if you want a heap allocator, it lives there. And then you've got these fundamental Rust types, these things that no Rust program can live without, and that don't actually care about what operating system. If you've used Rust, you'll be familiar with the result and the option types. They're enums, they are generic, they need to be defined in a library somewhere. And things like that go into a library called lib core. So when people write bare metal Rust, so Rust running on a Cortex-M4 or a RISC-V processor with no operating system these people say, "Oh, I'm using Rust as no std." And what they mean is I'm using the Rust standard library, but I've basically switched off the top half and I'm only using lib core. So I'm only using the parts that are operating system agnostic. No file system, no threading, no networking, but you do get Unicode string handling, as I say, result and option types, sort of fundamental properties of integers and floats and things like that. Jonathan Bennett: It, does this work hand-in-hand with a like an SDK the published stuff that the vendor puts out? So like we, we mentioned the Nordic. Obviously Nordic writes... I mean they, they've written Zephyr support. They've also got their oh, I forget what they call it, their little shim libraries, almost like an OS. SoftDevice I think they call it. And so when we're in Arduino land with C and C++, we do a lot with SoftDevice on the Nordic devices. Do, do... When you write Rust, do you use that stuff on this, this sort of a device, or are you f- blaze your own path? Jonathan Pallant: People come at this from different places. I went to a trade show in London recently, and I went round and surveyed all the people there who had the booths down at the show in London, said how do you write embedded firmware?" And honestly, I was surprised at how often Zephyr comes up. A lot of people seem to be really into Zephyr. And if you wanna use Zephyr, absolutely, you can put Rust on top, build your little Rust applications, link them into the Zephyr kernel, and that's all gonna work great. I wouldn't describe myself as a purist- but I'm in this to push what Rust can do. A great analogy came up the other day about about, about AI usage, and it was like sometimes there are heavy things that need to be lifted, and sometimes this is your job, and you should use whatever machine you've got to lift the heavy things. And sometimes it's 'cause you're at the gym- ... and if you bring a forklift truck to the gym, you are defeating the point. You're cheating. When I write Rust, I'm at the gym, and I wanna do it the hard way because I wanna practice my Rust, but also I wanna show what Rust can do- ... and find if there are any corners where Rust doesn't work. 'Cause I wanna know about those- ... and maybe I can go to the Rust project and get them fixed, or, I can educate my customers about areas where Rust doesn't work. So- Yeah ... for me, going bare metal is always more interesting. Jamming random integers into random memory addresses, that's where I'm most comfortable. But if people wanna use like a FreeRTOS or a ThreadX or any number of real-time operating systems that are written in Rust- If that gets the job done, that's great. Jonathan Bennett: Yeah. I think that's actually a really good segue to ask the question of what do- what is your day job? You're not... Are you writing production code, or are you doing the the education thing there at Ferrous? Jonathan Pallant: So my I asked for the words senior embeddage engineer to be written on my business card- Okay ... because it covers a multitude of sins. Technically what I'm doing mostly is teaching. So I teach Rust. I am the lead trainer. I'm responsible for all of our training materials, which are all open source. Anyone listening can just go and find Ferrous Systems Rust training, read all my slides, do all my exercises. It's all creative commons. You can help yourself. But I can't only teach, one, because my head would melt. I did a day of teaching today. I've been teaching eight hours already, so I may- ... yeah, just dribble a little bit. But also, yeah, you need breaks. But also, if I only taught Rust I don't think I would get better at Rust. There is an a need to sharpen the knives and practice the tools. So in between trainings, I'm, like, picking random dev kits and going, "Great, let's do some Rust on this. Let's write some training material about it. Let's pick an entirely new platform nobody's ever written Rust before." I have in my hands a lovely NXP developer kit. S32Z2 for the people who don't have video. This is get this right, eight Cortex-R52 with 20 megs of RAM. Jonathan Bennett: Oh, wow. Jonathan Pallant: As microcontrollers go- It's huge ... this thing is cooking. But it can't run Linux because none of these processors have an MMU. These are all- ... Arm's real-time platform. So a couple years ago I was like, "Can you even do this in Rust?" Probably, but I don't find anyone online who's done it in Rust. So we called in a favor from NXP. We got the board. I would not personally wanna buy one of these out of my own pocket money. It's a little out of my fun budget. Yeah, and we got Rust running on it bare metal, dual-core lop step. We can run asynchronous Rust. We got the interrupt controller going. But I do that because it's interesting and because it ends up useful for customers and useful as training material, yeah, embedded systems engineer, as I said, covers a multitude of sins. Jonathan Bennett: Rust and embed- the embedded world as a whole is really a moving target as places like NXP and STMicro and Microchip, Nordic, and all the rest of Espressif, everybody, they're constantly pushing the envelope, releasing new things, g- gluing new weird chips together that we didn't think we would ever see, and then suddenly- you see it at a trade show or a talk. But it's a moving target, and if all you're doing was education, then, you're, you would get stale, right? So I think it's incredibly useful to stay hands-on with the technology. Jonathan Pallant: No, absolutely, and I try to make it out to Embedded World every year to run the booth, but also just to walk the floor and- go and see what's new and get a sense of how things are training changing. You go to Embedded World three or four years ago- most of the floor didn't know what Rust was. "Do you do Rust?" "No." "What's Rust? Tell me about Rust." And this, they'd never heard of it. And now they're like, "Yeah, no, we've heard of it. It feels like a thing we should be doing." But certainly I think these big silicon vendors are thinking about, "What is my next step? How would I introduce this without alienating all of my existing customers who like the CSDK? Is there a migration path? How... do these things feel the same or do they feel different?" Espressif, I gotta give shout-out to them for being far ahead here. They basically hired all the open source hobbyists who were doing Rust on their chips. Gave them a job and they have their own Rust team. We've seen Rust news out of Infineon as well. Yeah, and I hope we see more out of the other silicon vendors as well because if they don't do it, who will? I mean me, but my time is somewhat limited and there's only so many dev kits I can buy. And it's the open source community. It's random hobbyists who go, "Oh, I've got a random dev kit I got off Adafruit or whatever. Got a chip. Does this have Rust support? I don't know, let's write something. Let's put something together." And we have all this sort of flourishing ecosystem around STMicro, around Nordic, around Raspberry Pi. I personally did the Rust support for the 2350. That was fun 'cause they gave me pre-production chips, so we had Rust support ready to go on launch day. But these are all independent ecosystems with their own thing going on. Which is great if you just wanna get into Rust and have some fun. But if you are- on the hook to design the firmware for a product where you wanna make a million a month, I can see why you may not just want to pick up random packages online, and you may wanna talk to the silicon vendor or someone else- and get a bit more professional support. Jonathan Bennett: Because the way where madness lies, and I know because we've done this ourselves, is you end up coming along and forking about 100 different packages, that you've got your own version of the library because you have some fix, and the upstream library is dead, or you can't get ahold of the maintainer, or this or that. Yeah. Been there, done that. That's my life these days. How similar is it? We've thrown out a lot of different embedded architectures. So again, compare, like, Nordic to the Raspberry Pi Pico. Can you write one Rust program and get it to run on both places? Is there cross-platform driver support? So say I've got a an I2C-connected temperature probe. Can I write one Rust driver for that and run the same code on the Pico and the the, the Nordic? Jonathan Pallant: You can, and this is thanks to the work of a great bunch of people called the Rust Embedded Devices Working Group. So take you back to 2017. I'm just starting to get into Rust. I'm using a random TI Stellaris dev kit that I happened to have lying around, and I'm doing my own thing, and some other people are doing their own thing, and we find each other. And we're like, "We're C developers. We've written a lot of C code." "Rust looks fun, but we're very aware of how-" I describe the C embedded ecosystem as like islands. And these islands are all really quite far apart. Yeah. And if you live in Nordic island, and I don't know, you happen to run out of chips 'cause there's a chip shortage, and you wanna get over to- Jonathan Bennett: That would Jonathan Pallant: never happen ... ST Island, to get over to ST Island, those islands are quite far apart, and it's pretty difficult to write, say, a cross-platform driver that, that works across that. And we looked at that and we said, "What could we do in this fledging fledgling Rust embedded ecosystem to try to, for that not to happen?" And the Rust language mechanism that it offers for that is a thing called a trait. And if you're a C programmer, you can think of it like struct full of function pointers. That's how the Linux kernel does it. That's how the Linux kernel thinks one disk controller looks like another. They both, drivers, provide a struct with a function point of open and a function point of a close, and read block and write block. In Rust we call that a trait. So the Rust embedded working group published some traits, and there's a trait for I2C, and there's a trait for SPI, and a couple of other sort of common peripherals, GPIO- razor pin high or low. And then what someone needs to do is come along and say, "Oh, I would like Rust to be usable on i- my particular chip." NRF 52 for example. So you would write some Rust code, and you would implement the trait. So here's my I2C driver- Yep ... represents this peripheral, and it implements the trait. When I flip to the other side, someone's got- a CO2 sensor. Little I2C device, measure the air quality. I've got- And they just need an Jonathan Bennett: I squared- I've got one of those on the desk behind me waiting for me to either find or write a driver for it. Jonathan Pallant: Fantastic. And you just need an I2C bus to put it on. And what do I2C devices need to do? The driver needs to do a read maybe do a, a write and then a read. There's a couple of fairly basic operations. 7-bit addresses, 10-bit addresses, and that's all encoded in the trait. So if I wanna write a Rust driver for my little CO2 sensor, I am generic over some type T, where I don't care what type T is as long as it implements embedded hal colon I2C- some trait. And then a third person comes along, and they're like, "I may be the first person in the universe to use that CO2 sensor with that microcontroller." And they can compose it together. Yep. And in general, it should just work. You will... There's ef- efficiencies that you will lose because it's a g- sort of a very generic API- that's not very- ... chip specific. But the trade-off is, yeah, here's my random Adafruit LCD screen, here's my CO2 sensor. Plop them all on a board. Here's a, a, what we call a, a HAL implementation someone's written for this chip, and it should all just work together, and generally it does. Jonathan Bennett: Yeah, that's very cool. It's actually similar to the way the the Adafruit excuse me, the Arduino ecosystem has evolved. You've got kind of that, that same idea in mostly C code, some C++, but it's that same idea. We've got an abstraction of here's what a wire bus is going to look like. It's usually what you see it called. And then somebody else can come along and write a driver for it. Was that part of the inspiration? Jonathan Pallant: Yeah, I think we'd the people who formed the Embedded Work Group, we'd obviously used Arduino before, but we'd also used these kind of very vendor-specific SDKs. But you also get the, the classic embedded commercial vendors. They all have their own stacks and their, probably their own in-house operating system. It seems every embedded company's got their own operating system they've had since the '90s- Of course ... and that kind of stuff. So we've seen all of that. And what would a, like a utopian future look like? I'd love anybody to be able to pick up any MCU- and any device and just plug them together. Jonathan Bennett: Do you see vendors actually starting to write the the, that bottom half, that support layer for you guys? Jonathan Pallant: So we have seen Espressif do that. So they publish all of the, the, the drivers you need for their chips. For most of the other vendors, I think we've seen something from Infineon. But for most of the other vendors it's wait and see. Maybe throw the hobbyists a bone, some free hardware occasionally. But that's all mainly hobbyist-driven. I will shout out STMicro, who have done the other side. Surprised me. But if you've got your little STMicro sensors, 'cause obviously they don't just make MCUs- I believe they have all these sensing modules as well. They will give you Rust drivers for the sensors. Jonathan Bennett: Cool. Jonathan Pallant: Which you can just get from STMicro's GitHub and use, and then use those drivers with your other Rust code. Jonathan Bennett: I've been really impressed with STMicro's open source support. They were off my radar for the longest time. And then I started working with some of their hardware, and it's like, you ... 'Cause I'm from the Meshtastic project. That's my day job these days. As all of our listeners know, because I mention it at least, take a shot, if you're playing the drinking game. But so I got ahold of some STMicro hardware, and it's I wonder what it would be like to 'Cause, because it was gonna be able ... Get Ar- get the Arduino working on this, and then run our platform IO build on it, and then finally make Meshtastic work. And I was ready to strap in for the long haul, and it's oh, no, just go download it off of their GitHub and pick the board and it'll compile. It was easy. It was like a day's worth of work to make everything start working. It really surprised me how good their support was. Jonathan Pallant: Nice. I will shout out their tools. So I do use CubeMX quite a lot. Their MCUs have very complicated clock trees, so having a little graphical tool where I can just feed in the crystal frequency and go- Oh "Oh, I want a, like a times two," and configure the PLL. I don't need the C code that gets auto-generated out the back end, but it's very useful visually doing the clock tree. I also love the pin planner. Nordic don't need one, right? 'Cause you could put any peripheral on any pin. But ST you generally can't. On STM32 each pin has a set of three random functions. And you're like, "Great, I need UART5." I guess I can have it over there and over here, but oh, not here because I need th- that pin to be like a, an input interrupt, so I'll put it on this pin. And doing that with a pen and paper or a spreadsheet's horrible, so I do love the CubeMX pin planner- where I'm just like, "I need these peripherals. Just lay it all out on the, the 80 pin package, and I'll design the board around that." And then again, I throw away all the C code, and then I teach, I program Rust to f- to follow the kind of the, the pin map that we've got out the tool. Jonathan Bennett: Yeah. It's almost like a, a traveling salesman-esque problem. It's like it, it seems very simple, but then when you try to do it, it gets really complicated really fast. Jonathan Pallant: One, one of, one of those puzzles where there's one hole left- ... and one piece left. And this piece doesn't fit in that hole, and you're like, "Oh, I'm gonna have to take this all apart and redo this-" Yep "before, before this fits." Jonathan Bennett: Yep. Exactly. That's fun. So what what other interesting hardware have you run Rust on? We talked about a bunch of them, but what's your favorite, what's your favorite device to write Rust for? Jonathan Pallant: Ah, that's a really good question. I do love these kind of big real-time microcontrollers. But just 'cause they're so outside, like, where I'm normally using microcontrollers. Eight R52s is just an outrageous amount of power. So these are fun. Running the dual core lockstep actually turns out to be really easy. 'Cause the thing about having two cores locked together that do the same thing for safety purposes, so- Oh, okay if one of the cores gets hit with an electron and crashes, the other core does something different, and the hardware goes, "Oh, these cores do not agree. Let's reboot the system." I thought that was gonna be really tricky to program. But the thing is both cores do the same thing. So you program one core and the other one does exactly the same thing, but one clock cycle later. So you can effectively ignore it until then a bad event happens and they diverge, and the hardware goes, "Oh, oops, upset. You need to reset the system." So playing with this safety critical rated hardware- Interesting ... has been fun. Jonathan Bennett: Yeah. Jonathan Pallant: I also... I- I'm from Cambridge originally. Cambridge was the city I was born in. I've been hanging around with the Raspberry Pi people since 2012- ... with the original launch of the very first Raspberry Pi. I had one I ordered day one. I've got it signed by the team. Nice. So I have a soft spot for the Raspberry Pi stuff. I did enjoy doing the Rust port for the 2350. I think having RISC-V and Arm on the same silicon is just really fun. It's w- my first opportunity to write Rust for RISC-V. N- maybe the first Rust program that actually had to compile for both architectures, 'cause that's not normally a thing for bare metal. Indeed. You are either building an Arm program or a RISC-V program. So to suddenly end up where you want one program, talks to the UR, talks to the I2C, but it compiles for either architecture- With all the peripherals at the same memory addresses. I don't know, that was fun and unique. I'd never had to solve that one before. Jonathan Bennett: Yeah. Yeah, interesting. I, I have... Speaking of Raspberry Pi and their unique peripherals, I can't help but think about- ... the RP1, and I still, I want the little RP1 chip on a PCI Express board that I can slap into my desktop or put into a laptop. Yeah ... that's my dream. I asked the guys at I, I think it was Embedded World, actually. I, 'cause I was there this year. I don't think I ta- I don't think I bumped into you. I don't remember for sure when they- We Jonathan Pallant: did, 'cause that's how this podcast happened. Jonathan Bennett: Oh, that's right. Jonathan Pallant: Okay. 'Cause you came by and I said, "Yeah, I'd love to do that." And then Florian was like, "Oh, is that the guy from Floss Weekly?" "Yeah, we should give you his number. We should do that." Jonathan Bennett: Okay. Yeah, that makes sense. And so I've got your business card somewhere on my desk here underneath all of the DEF CON prep- ... that's just piled up. Anyway, I went to the Raspberry Pi guys, and I'm like, "Hey, you guys really need to make the RP1 available." And they're like, "We don't wanna make it available because we know our competitors will buy them and start putting them on their boards too." Good Jonathan Pallant: point. So you know they've got dual Cortex-M3s In the RP1 Jonathan Bennett: I don't know a whole lot about what's actually inside the Jonathan Pallant: RP1 Yeah, so it is actually a lot... have you ever clocked the numbering? So the, the RP2040, if you look at the chip, silk screened onto the chip, it simply says RP2. Jonathan Bennett: So- So I know that was the second board that they start- the second chip, excuse me, that they started designing in-house. Jonathan Pallant: Yeah. And Jonathan Bennett: The RP- See, so the Jonathan Pallant: RP1 was first- The RP1 and the RP2 was- Jonathan Bennett: And then there's the Jonathan Pallant: RP3 as well Do you remember what the RP3 was? Jonathan Bennett: Wasn't it the power supply? Jonathan Pallant: It is. If I remember right, it is the custom version of the Broadcom SOC that has SDRAM stacked on top So the Raspberry Pi Zero I think it's the 01, maybe the 02. Someone's gonna correct me. That's fine. One of the Raspberry Pi Zeros has the RAM sitting on top of the processor- ... a package on package, and that was the RP3. Jonathan Bennett: Oh, okay. Jonathan Pallant: Which according to the numbering, they started third, but was the one they managed to ship the first. RP2 came out in the middle, and then finally RP1 came out. But as I understand it, there are some Cortex M3s in the RP1, and we're like, "Can we program them?" And they're like, "Nope, absolutely not. You- ... that is not designed for general use." I can imagine they would not wanna deal with random people like me asking, "Why does it do this when I poke this register?" And he's like- ... "Don't poke that register. Just leave it alone." That's- ... that's Jonathan Bennett: not the kind of errata we want to write for this. Jonathan Pallant: No. That- I think that is a chip that they have enjoyed using. I hope to see it again on, on some other boards, but who knows what RP4's gonna be, right? Yeah. It's interesting to see what comes next. Jonathan Bennett: I just love the idea that you can have PCI Express and then GPIO and SPI and I2C, and I want that on a desktop. Do you know how- Yeah ... how amazing it would be to be able to do development and debugging work on hardware if I could have an accessible SPI bus on a desktop? Would be amazing. Jonathan Pallant: Yeah, without a s- without a stack of USB dongles all hanging off some- Ah ... crummy $10 USB hub that likes to reset. Yeah. Jonathan Bennett: Yep. Absolutely. But there, there are some chips out there that'll let you get SPI and GPIO. I2C, I think too. There's a Chinese chip called the CH341, which I've used a lot here recently. In, in fact, I met I met Nirav... Oh, I can't remember his last name, but Nirav, the CEO of Framework, and I gave him one of these guys. It's a Framework module, and it's got a CH341 in it that lets us talk to LoRa chips from on a desktop, which is a lot of fun. Very cool. No, no Rust- Do you have- No Rust code on, running on that yet, but maybe one of these days. Jonathan Pallant: Do you have a, like a box of boards you've bought and you thought, "That's gonna be fun one day"? So I ha- I have my guilty box. It's things like, "Ooh, that looks fun." There's a CH chip that has USB 3. And I'm like, "Microcontroller with SuperSpeed USB, that sounds great. I want some of that." So I bought one. It's on the pile with about 19 other boards- ... and maybe one day I'll get round to pulling it out and giving it a go. Jonathan Bennett: Yeah. So you note that when we started the show, I was able to just reach onto my desk and pull up a Seeed Xiao board, right? And that's not the only one that's there. That was just the one that was on top. Jonathan Pallant: Yeah. I just, yeah, there, there's a reason I just have random $800 no- NXP dev kits just kicking around. Yeah. Jonathan Bennett: Yes, absolutely. Yeah. I've got a, I've got a couple of FPGAs around here too. It's something I've really I've not been able to get into it yet, not with a ver- fervor at least, but the FPGAs with fully open source tool chains is really fascinating to me. Is that a realm that Rust gets to play in too? Can we write FPGA code in Rust, or? Jonathan Pallant: So I know there are people who like to do SoC design, and I think there's some that are built on Python where you write Python and out comes full FPGA SoC with core and bus and- and peripherals and so on. I don't know of anyone doing that in Rust, but it would not surprise me. Jonathan Bennett: Yeah. Jonathan Pallant: You can- I would love to get into the open source F- FPGA stuff. What I did do, shout out to Matt Venn on the Tiny ASIC thing, 'cause I did that a few years ago. And made... That I do have which I forget which run we were on, but I have the run that doesn't work unless you get the voltage precisely right. So technic- technically I have some ASICs with my code my designs in that will do a thing. I don't think I'm ever gonna get them to work, but it's cool. I can go, "Yeah my design is in that ASIC." Yeah. That was a lot of fun. Jonathan Bennett: Yeah. That's cool. A- and I guess you can quite easily take that a step to the side and put an SoC on an FPGA and then run Rust code on the Jonathan Pallant: SoC. Absolutely. That, that is one of the great things about RISC-V is you can just grab a core without paying- ... a license fees, drop it on. Rust has great support for RISC-V- ... 32 and 64-bit, so yeah that's a great way to get started, and that Tiny ASIC chip has a RISC-V core on it. Yeah, I should have given that a go. I should get some Rust code running on that. That definitely seems possible. I have also done Rust on PowerPC. I have a bunch of '90s Unix machines for fun because, I... you need a hobby, right? So my hobby- ... is Unix System Admin '98, which is the worst simulator game you've ever played. I- It's like installing H- HPUX Nine on some broken old PA-RISC machine is quite fun ... I've, Jonathan Bennett: I've played that game once. I, ... I had a customer that had an old SCO Unix server, and they're like, "The hardware is dying. We need to resurrect this because it's got our old legacy system." And so my, my solution was to get a copy of the hard drive and host SCO Unix inside a virtual machine on a Linux box. And I was pretty proud of that- Like- ... when that finally came up and worked. Jonathan Pallant: You're lucky it was x86, 'cause I have a PowerPC Macintosh, an UltraSPARC, and a HyperSPARC. I have a MIPS Silicon Graphics. That has run Rust code. I ha- was able to compile a Rust binary- Nice ... for MIPS, but the Rust the LLVM linker just makes ELF files, and it turns out IRIX- Just uses ELF files. Oh. So I just copied all the object code over to the SGI machine, used the SGI linker, and that works. Yeah, I've done Rust on SPARC as well. I know the European Space Agency, still rather fond of their SPARC processors- ... because they have the, the LEON3. SPARC was famously an open architecture. Anyone could make SPARC chips. And so the European Space Agency did, and there is an open source processor the LEON3 you can find online. It's a SPARC V8 32-bit. And so I did Rust, again, just as a fun thing to talk about. I have Rust code running on bare metal SPARC, and that means I ended up as the target maintainer. So Rust has these, these targets, these sort of operating system architecture pairs- ... or bare metal and architecture, and everyone technically needs a maintainer. Some of them don't. And I'm like, "Yeah, this would be funny if Rust could run on bare metal SPARC." They're like, "Yeah, okay, cool you can submit that as a pull request, but you're gonna have to put your name on it as the maintainer. And I'm like okay, that seems fine. I can't imagine anyone's gonna use this." And then I get messages from people going, "Yeah, we've taken your work, and we're now running Rust on RTEMS on Spark Leon3, and it's working great." Huh. By the way, we've observed this, so w- what do you think we can fix this?" And I'm like, "Cool. I didn't know anyone was gonna do this, but great." I'm glad to have, made that exist. It's a lot of fun. Jonathan Bennett: Yeah. Yeah. Y- you also mentioned MIPS in passing, and I've done- ... I've done some MIPS stuff mainly because in a previous life I did some OpenWrt stuff, and a lot of those routers surprisingly are running on MIPS. Yeah. And you would think, oh MIPS, we've left MIPS in the past now. All of this stuff is running on ARM, or they've gone to RISC-V. There's the Chinese Loongson, which is a MIPS-based architecture too. There's still some life left in MIPS and it might be interesting to run Rust there. Jonathan Pallant: Yeah, no, absolutely. I know MIPS, yeah, long and storied history. I think the latest version of the, the Loongson architecture has diverged away enough from MIPS that it now has the target string begins with a Loongson or whatever the name is rather than- ... mIPS. That makes sense. But there are still some MIPS targets in there. Yeah, it's fascinating to dive back and ap- apologies if this is gonna upset any old hands. I'm just gonna say branch delay slots. If you know what that is, you're quivering on the floor, but if you don't, it's fine. Don't worry about it. Never look up what a branch delay slot is. You won't enjoy it. Jonathan Bennett: I'm going to ignore your advice and look up what that is- ... because that's the sort of thing that I lose hours- Oh ... of time reading about. Jonathan Pallant: Yeah, so i- yeah, imagine a pipelined processor that is busy fetching, executing decoding and executing instructions. And if you hit a branch, you've already fetched the next instruction. So why don't you just execute it anyway and then do the branch? Jonathan Bennett: Oh, Jonathan Pallant: branch- So a branch delay slot means you always execute the instruction after the branch before you branch. Jonathan Bennett: That's what x86-64 processors do all the time, and they never ran into any problems from that. Never caused anybody problems. Yeah. No, that was a great idea. Jonathan Pallant: Just having, when you're handwriting assembly, just remembering that, yeah, you're gonna get one extra free instruction before the branch. Jonathan Bennett: Yeah that's fun. Okay. I understand now. I get it. We've talked about the very, very tiny embedded side of things. What about the, the big iron, like System 360 and some of those IBM which I guess a lot of those are just PowerPC, aren't they? Jonathan Pallant: Ah, so System 360 I guess originally starts in the 1960s. And it's really funny 'cause it's like a, a big technology project that ended up being massively over budget and massively late. You think that's a modern concept, no. People have been late delivering tech- Of course ... since the very first kind of computers. And I got into that kind of stuff Because I wanted to build my own computer. So I'm like, "Yeah, what is a computer? And can I turn a microcontroller into a computer?" And so you go back and go, "All right, traditionally, what was a computer?" You think of the movie The Italian Job. There's a, the big mainframe in the background- ... with all the spinny tapes. You look at it and go, "Yeah, '60s computer, spinny tapes, stuff is happening. Yeah, looks, looks fun." So I'm like what is it? What does it do? How does it work?" And computers really haven't changed much. They need some sort of terminal- ... where input can go in and output come out. They need some RAM as like some scratch space to store the stuff they're working on. They need a processor that can execute instructions, and they probably need some non-volatile storage to remember the things they've worked on, be that a spinning drum, a paper tape, a magnetic tape, or whatever. And actually, like a Raspberry Pi Pico, where, what do you got? Pico one, two Cortex M0s at 150 MHz. That absolutely wipes the floor with a Commodore Amiga- ... Atari ST. You'll absolutely spank something like, the Saturn, the PlayStation 1. It's probably not far off my Silicon Graphics O2. Peak 1996 desktop workstation. Obviously has a lot more RAM, but two 32-bit ARM cores at 150 MHz is pretty cooking. So why can't it be a computer? The answer is, A, it doesn't have enough RAM. So I wrote a little MS-DOS style operating system. It boots to a terminal. You can load programs off disk. You can run them. There's a graphical display that's all rendered in real time. The Raspberry Pi Pico's great at doing that. It's nice having a whole second core to run the video so the main core is uninterrupted. And so yeah, just distilling a machine down into what is the kind of smallest kind of computer. And yeah, those basic concepts, terminals with input and output. Your dev TTY- ... is named that because it was connected to a teletype, literally a typewriter on the end of a serial cable- where you could type and it would h- fire the hammers and print on the paper. Can you imagine that, running LS minus R on a machine where everything's printed on paper? Just the amount of paper you would get through. Obviously you can't use Vim. You don't have a screen. You're in Ed. You've got line based editing at that point. But- That is the fundamentals of a computer, and that is the same whether it's gonna be, you're doing a little operating system on a microcontroller or you're on a, a, a modern, big Linux system. I know there's good Rust support for Z Series, current IBM mainframes, PowerPC. We can still do SPARC64. And it's got a big iron systems. They're the same kind of concepts we had in the '60s, which is fun. Jonathan Bennett: One of the, one of the things we haven't touched on yet that I think we, we can't do a Rust show without talking about is the memory safety aspect of Rust. And I always used to say that I was a little unimpressed by the idea of let's say Rust in the kernel because The memory safety aspect of Rust just doesn't jive very well with writing device drivers, right? Because a device driver, by definition you're writing into uninitialized memory locations, and Rust won't let you Jonathan Pallant: do that. Random integer and random memory addresses. Jonathan Bennett: And so- That's Jonathan Pallant: how the machine works ... Jonathan Bennett: yeah, and the dr- the driver is just all gonna be unsafe Rust code, right? I I'm curious your take on that. I don't nec- I don't necessarily take that approach the, quite the same, but e- educate past me as to why there's still some some value in writing bare metal Rust when it's all unsafe anyways. Jonathan Pallant: Yeah, so I think that, I think there's kind of two, two aspects of that. One, actually the, the kernel's the very low layer part of the s- the system, and the bit that actually has to get a right to the memory, put the random number at the random memory location, is the very bottom of this stack of code. And everything above it doesn't have to do that because it's got the device driver to do it for you. Yeah. So if the device driver itself has the unsafe code in it, we can check that and validate that and manually review it and go, "Yes, this looks good. That's what the data sheet says we should do." And then above that, we can build a bunch of safe abstractions. A serial port driver. Is it okay to send some bytes over the serial port? Yeah, provided you've enabled it and configured the board rate generator. I can do all that in safe code. I can design an API that means it's impossible to send data on the UART until the board rate has been configured, 'cause I've got a type system that lets me do that and take ownership, and this is an unconfigured UART, and this is a configured one. And this one lets you send text, and this one only lets you set the board rate, and you're gonna have to do one before the other. So all those kinds of problems, I don't need to access random memory to do that because I'm building on top of this kind of low level that does. But I was also pointed to things that kernel maintainers have said. I think it was Greg KH at a recent Rust conference in the Netherlands. He said that introducing Rust into the Linux kernel has made the Linux kernel better, which was not a thing that he was expecting. But it's being forced to think about those Rust terms. I've got a value, and I'm giving you ownership of it. I'm calling a function, and I'm getting ownership of the thing back, and now it is mine. Or I'm calling a function, and I'm only letting you temporarily borrow it. Maybe you have a shared borrow, and you need to be aware that other people may be borrowing. Maybe you have an exclusive borrow. You can assume that nobody else is using this object right now, 'cause although you don't own it, you do have exclusive access to it. Those concepts exist in C The problem is the C language doesn't have the syntax and the terminology to describe them, so it goes into the comments. When you call this function, you get given a pointer to foo. You must call free on pointer to foo later because I've heap allocated it and it belongs to you. Or you must pass the pointer to foo into the foo destroy function later- ... and then you must not use the pointer again after you've done it. But it's all in the comments, not in the code, which means the kernel the, the compiler can't check it. Jonathan Bennett: The compiler- Jonathan Pallant: So the thing I Jonathan Bennett: think they- Yeah, the compiler will let you break all of those contracts- ' Jonathan Pallant: Cause it can't read it because it doesn't know about Jonathan Bennett: them. Yeah. Jonathan Pallant: It can't read the comments. It has no idea whether you're supposed to use this foo pointer and where you're not. Whereas in Rust you can. So when the kernel was forced to adapt their C APIs to make them usable to a driver written in Rust, the Rust people went do I own this? Is it mine? How do I destroy it? Where do I have to give it when I'm done? Are you just giving me a thing I gave you earlier, or is this a new unique thing you have created for me?" And I think that kind of understanding those concepts and going, "Oh, yeah, actually we could probably make this C API better in exporting the features you need for the Rust code," actually this becomes a better C API for all the C drivers as well- because these things are expressed better and it's kind of understanding of this kind of moving and ownership and borrowing. I, yeah it's fundamental to how computers work, but it's nice to have a language that lets you express it yeah, natively and not just in the comments. Jonathan Bennett: Yeah. One of the w- the early, early, this is a long time ago actually, but one of the criticisms I saw about all the different efforts to make Rust and C work together was that it was essentially making Rust under the hood in its memory management actually speak the C, sort of the C memory management language, and that I think s- I can't remember who it was that was writing about this, was really critical of that idea, and it's like we're taking Rust and we're making it almost C in order to make it work together. I've not heard anybody make that point in a long time though, so m- maybe that was just a a red herring a, a false road to chase down. But- Jonathan Pallant: I think it's, I think it's a fair observation because, yeah, if you want a Rust program to interface with a C library, or vice versa, a C program to interface with a Rust library Raw pointers and C compatible types are going to be involved. But the thing is for the Rust thing, whether it's the program or whether it's the library, that that C-ness only infects the edges, and you can wall it off and go, "95% this is Rust." We've got our unit tests, we've got our code coverage, we've got our Clippy tool giving us opinions. All of that is gonna work across almost all of this code base. But right at the edge, I'm like, "Okay, here is my heap allocated, my boxed type." I'm gonna convert it to a raw pointer and give it to the C library, and then later the C library's gonna give me the pointer back, and I'm gonna look at it and go, "Did you give me null? Did you give me a non-aligned pointer?" Does this actually look like a thing? And like, all right, it seems validly aligned. It's not zero. Let's assume it actually is the original Rust heap allocated thing that we had before, and you can convert back having done those checks. So most of your code is still very Rust, but there is this kind of C thing at the edge, which is only needed if you need to interact with it from C. If you're gonna interact with it from a Rust library, we just bypass that. Yeah. Just go straight into the Rust APIs. And I would say that one of the disadvantages of using the C compatible types There's a great feature in the Rust compiler. If you have a structure with three fields- you may assume the first field appears at the lowest memory address, appears first in memory. In Rust, that is not true. The order of structure fields is not defined. The compiler can change it between- ... compiler versions, and it allows it to remove padding because it can go, "Oh, you've got small thing, big thing, small thing. Let me put big thing first, two small things after-" "... and now I've made your structure smaller," which is great when it's Rust code calling Rust code. C compilers don't love that. They do rather assume the structure is laid out in the order you wrote it. So you just go, "Hey, Rust. This is a C-compatible structure. Don't do any shenanigans. Just put it in the order I put it." And then it's C-compatible, and Rust and C can just share the same memory and ha- have the same view of the same data. The reason you don't do that by default is it gives you padding, where previously, a Rust-native version might not. So it's a little bit of a price to pay with that, but it's generally not too bad, and I think it's worth it. Yeah. And we know there's a lot of great success cases, Firefox, Android, and so on- ... introducing bits of Rust into their code base and generally the trade-offs kind of work out. Jonathan Bennett: Yeah. Just a second ago you, you mentioned a Clippy tool, and I couldn't help but think of starting up Microsoft Word back in the day and little Clippy popping up. "It looks like you're writing a let," so do... Is there a Rust tool, "It looks like you're writing a system driver. Here, let me help you with it." Jonathan Pallant: Yeah, and that's literally why it's called Clippy. The Rust compiler- Jonathan Bennett: Oh ... Jonathan Pallant: will tell you when your code is wrong, right? There's errors and warnings, where a warning is, "This is probably wrong, but I'm not 100% sure." So it's about whether your code is correct. Clippy is merely a tool with opinions. And it's a great tool, and any kind of static analysis of a Rust program is probably gonna be built around Clippy. I can give you an example. You've got some kind of array, and you would like to iterate through all the items in the array. So you write for i in 0 to array.len. I'm doing my array indexing, I've got my integer, it goes from zero to len minus one, that's how the loop works, and I'm gonna index my array with my variable i. And that is not wrong, and the compiler will give you no warnings- ... 'cause that is all very valid Rust. If you give it to Clippy, it says, "It looks like you're trying to index through an array. Why don't you write for item in array instead of for i in 0 to len?" That's an opinion. The code you have is not wrong, but actually Clippy's opinions are generally pretty good, and you probably should have said for item in array instead, because it's probably gonna be faster, you can maybe skip a bounds check, you're not gonna accidentally go off the end, all that kind of good stuff. Jonathan Bennett: It's an opinionated style linter. Jonathan Pallant: Yeah, and highly configurable. So it has a default set of lints. But I know a lot of commercial customers have their own in-house rule set. You're probably familiar with the idea of a coding standard. Yes. Every company's got a C coding standard, so many different ways to write C. They're starting to get Rust coding standards, and they're usually expressed in the form of a set of Clippy lints- ... that you should have turned on. Would you believe the Society of Automotive Engineers, the SAE, have published a document which is basically a guide for writing Rust for high integrity systems? I think the document is JA 1020, 1020, and it's a document that you have to buy it. It's a commercial thing. But it includes a list of Clippy lints, but also rationale for why every single one of them should be turned on. It's al- almost like a sort of MISRA for Rust, which doesn't yet exist. But it's quite nice if you're writing Rust code to be able to have this thing and go, "Okay they've clearly thought about it a lot harder than I have, so I'm just gonna turn on their default set of rules," which is probably more than Clippy has on out of the box. And a lot of them are great, like every unsafe block should have a comment above it that begins with the word safety and then a colon. Like you should explain why you're telling the compiler, "Trust me, bro, I know what I'm doing." Because you may not know what you're doing. You do. So bit- Clippy can check that. Clippy can go, "Oh, here's an unsafe block without a safety comment above it." You should fix that. And turning on those kind of rules is a great idea. Jonathan Bennett: Yeah. Yeah. That's really good. All right boy, I think we're, I think we've been going at this close to an hour. I gotta ask, would you recommend somebody either just getting started in the programming world or even an old hand to go out and learn Rust and add that to their resume? Jonathan Pallant: I think it's really useful because I remember when I was learning Rust, I was a C developer. I've been doing this for gosh, I left university in 2004. Do not tell me how many years that is. I do not wanna think about it. But I spent my professional life writing C code- ... building all these kind of embedded products. And as I learnt Rust, I became a better C developer. And I write Rust 'cause I think it's a great tool to solve the kind of problems I have, but even if you don't actually want to write Rust full time, it's still fascinating to learn those concepts about ownership and borrowing, and it will make you rethink those C functions that return a pointer. And you're like, "Do I own it? Is it mine? Am I borrowing it? Is it a pointer I gave you earlier?" And having that new language to describe that, I think is- ... is super useful. I don't know, it's just super exciting. There's loads of new, fun stuff going on, loads of research in academia. That seem- they really seem to have taken Rust to heart. So yeah, it's it's a fun and exciting thing to get into, and even if you wanna remain a, a grizzly old C developer as I was, I still think it's a useful thing to dig into. Jonathan Bennett: Yeah, interesting. Very good. It- where should somebody go to, you mentioned that there's a, a lot of education material from Ferrous that's available. Where should somebody go to look for that? Jonathan Pallant: Yeah, so you can just find our website, ferrous-systems.com. You can also look at our GitHub, github.com/ferrous-systems. And you will find my training material and my exercises. They're all free of charge. Yeah, you can check them out and enjoy them. Jonathan Bennett: Yeah, very cool. And then before I let you go, I gotta ask a f- couple of final questions, and that is, what is your personal favorite text editor and scripting language? Jonathan Pallant: Ooh, that is a great question. I- I love Sublime Text. V- Codium, VS Codium seems to be the one that is mostly open on my computer, but you just cannot get past the speed of Sublime Text. And it introduced me to the world of multi-cursors, pressing Command+D to select multiple things at the same time. Which is still witchcraft as far as I'm concerned. And when you s- you just see it happen in front of you, that's amazing. I edited 39 things simultaneously in this file. So yep I love love Sublime Text. For scripting languages I've done a lot. I've done Perl and PHP and Python. So let's go with one that doesn't begin with P. Let's just say Tcl, 'cause I'm feeling obtuse. Jonathan Bennett: I'll Jonathan Pallant: have to go- Everything's a string. Why not? Jonathan Bennett: I'll have to go, I'll have to go look that up. I don't know what Tcl scripting is. Jonathan Pallant: Oh, yeah. Oh you're gonna have so much fun. Jonathan Bennett: Another rabbit hole for me to dive down after the show. Jonathan Pallant: Absolutely. Jonathan Bennett: Absolutely fun. All right. Thank you, Jonathan, for being here. It has been an absolute blast to talk with you. Jonathan Pallant: Yeah, it's been great. Thank you so much. Jonathan Bennett: All right. That is Jonathan Palant of Ferrous Systems talking about Rust on the other end of the spectrum. And so we do not have a guest yet for next week, I don't think, unless one is scheduled in the time that we've had the show. So if you want to be on the show or know a project that should be, you can drop us an email. You can get us at floss@hackaday.com. Let us know who we should be talking to. I will be at DEF CON later this week, and so if anybody is there, make sure to stop by and find me. I don't have a booth that I'm running this time. I'm just on the run, but find me and we can say hi, and we can chat for a bit. But yeah, we appreciate everybody that's here. Whether you watch or whether you listen, whether you get us live or on the download, thank you so much for being here, and we will be back on Floss Weekly. All right, cool. We had a bunch of fun show titles that got suggested too. Oh, Tickle TCL, I have heard of that. I'm not super familiar with it, but I... Yes. Some- someone linked it in Discord, and it's like, okay I know what TCL is. Three Things in a Trench Coat was a lot of fun. Source Code In, Machine Code Out. Sometimes There Are Heavy Things To Be Lifted, that one's long. A Tool With Opinions. I think I like either A Tool With Opinions or Three Things in a Trench Coat. Jonathan Pallant: I think if I was gonna pitch you an alternative, I'd say Don't Bring a Forklift To The Gym. Jonathan Bennett: Don't Bring a Forklift To The Gym. That one's pretty good too. Yeah. It's a little long. Jonathan Pallant: It is, yeah. Jonathan Bennett: See, I really like both Three Things in a Trench Coat and A Tool With Opinions. I think we'll probably go with A Tool With Opinions because it's also, it's a great snapshot of what Rust itself is, I think. Jonathan Pallant: It is, and the British people in your audience will assume it may be a description of your guest. Jonathan Bennett: Yes. That's great. That's brilliant. All right, we'll go with that one then. Jonathan Pallant: I'll live with that. That's fine. Jonathan Bennett: I may reference that pun somewhere in the show description too. That's great. Thank, thank you, sir, for being here. It's been a lot of fun. It's really been a blast. Jonathan Pallant: Yeah, my pleasure. It's been great. Jonathan Bennett: All right. See you next time. Let me get a title and tease recorded here. 878. Bye. 878, a tool with opinions. Yes, I had not thought about the the, the British-ism baked into that, but that's really good. All right. This week I talk with Jonathan Palant about Ferris systems and Rust in the embedded world. It's a lot of fun. You don't wanna miss it. This is Floss Weekly, episode 878, recorded Tuesday, August the 4th. A tool with opinions One of those should do it. Thank you everybody for being here, and thank you again, Jonathan. We'll see everybody next week. Bye. Jonathan Pallant: My pleasure. Bye.
• 
  Episode 878 - A Tool With Opinions 05.08.2026 1h

  This week Jonathan chats with Jonathan Pallant about embedded Rust! Learn about the growing Rust driver library, the different ways to build a Rust stack on an embedded device, and how the opinionated tooling can make you a better programmer!  You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 877 transcript 29.07.2026

  FLOSS-877 Jonathan: Hey folks, this week we're talking with Francois Proulx about smoked meat, and poutine, and Bagel, a trio of open source security tools that you don't wanna pass up on. This is Floss Weekly, episode 877, recorded Tuesday, July the 28th: RCE as a Service. It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we're revisiting a topic that we've talked about in the past, and in fact, revisiting a guest. We've got Francois Proulx of Boost Security. These guys are experts in supply chain security, and really one of the main things that we're talking about with this is GitHub CI, continuous integration. And so a lot of projects will have a CI set up to where you run automated tests on your code changes. Where this can get dangerous is if you allow the outside community, so untrusted people, to make pull requests, and then that CI engine runs against those incoming pull requests. And then you have this kind of strange problem, that you're running untrusted code inside your GitHub repository, and surely that doesn't ever go wrong. Yes, sometimes it goes wrong. We had Francois on the show about two years ago, and we talked about a project then called Poutine, which was a scanner, an automated scanner to take a look at a project's CI runs, and essentially give you an evaluation. Do we think this is safe or not? And in the time since then surely we've fixed this problem and it doesn't exist anymore, right? Unfortunately I don't think that's the case. I'm gonna go ahead and bring Francois on. He's the guy that has the answers, and we're gonna find out. So welcome, sir, to the show. It is great to have you back. Francois: Thank you. Jonathan: Yeah what, w- what, did I get the problem space right? And then what's changed in the last two years? Francois: Ver- very good introduction. Yeah, I think you got the, the takeaway from the last discussion, and I guess you, you did your own homework on your own projects and could actually see why the tool was created. Why I created it, why it's a, it's an issue that especially maintainers... this audience is both consumers, like users of open source, as well as maintainers. Jonathan: Yeah, absolutely. Francois: For the maintainers, I think my message was that they should give the tool a try so that they can protect their own end users, right? Jonathan: Yeah, and I think we're probably gonna pitch this in the context of GitHub, but this is not just a GitHub issue, is it? There's there's GitLab, there's Codeberg, there's glued together CI scripts that run on infrastructure that's not any of those three things. And I imagine you could have problems on any of those contexts, couldn't you? Francois: 100%. So effectively you know- Basically any source control system, be it GitHub, GitLab, or whatever- Yeah ... that can automatically trigger the execution of a CI environment upon the submission of new code changes, just the, you know, triggering on such an event, it they can have the same exact problem. So really the devil's in the details in terms of the configuration of what is the behavior that happens, like the, effectively the secret, or does the execution of the arbitrary code changes get hacks- access to secrets or not? If it's just arbitrary code execution of the new tests, for instance, or the new code potentially it can be done in a way that's perfectly by design and without clear risk. Of course, if you were to reuse the same CI environment, like the same machine is is to be reused across different runs, then there can be pollution across those. But assuming it's an ephemeral run runner environment it can be done safely. But then, yeah, like the, the way it gets configured and the various components that get pulled during execution is really what matters. Jonathan: Yeah. And I think part of the reason why this is such a sticky problem to solve is 'cause on one hand, obviously when you have a pull request from the outside, so like by definition you're talking about untrusted code. It's literally untrusted user-supplied data, and you wanna do a processing run on that. But- Historically, we've not been concerned about, and I'll just make this very generic, we've not been concerned about malicious code inside of make files, or malicious code inside of build scripts, or malicious code in all these various places, even in C and C++ headers, for that matter. You can do some interesting things just with preprocessor directives. And that's been a, historically a blind spot because oh, if you're compiling this code on your machine, surely you already trust it. And then you take this and you put it on the cloud, which is, as we all know, just somebody else's computer, and because it's on the cloud, it has to have access to some sort of, to, to some sort of secrets. It has to have access to your build environment and your accounts in the cloud to one extent or another. And there's multiple ways that this can go wrong, and I've seen firsthand someone will run a build, and part of the build is go grab all of the variables from that you can get ahold of, basically all your environment variables, and then send those off to somebody else's server, and then you lose your secrets, right? But then there's also even other problems, and I think GitLab had an issue here at one point where because you are... You're in this inside the runner you're validated. You can then potentially go and do things to the project, and so there's that danger as well. And this is not just theoretical anymore, is it? There's been real attacks pulled off with these sorts of problems. Francois: Yeah, absolutely. So that, that is exactly how as we publish open source tools, we write articles on the topic, pushing the boundary of the, the research in that area. And I speak at different conferences and, on podcasts and whatnot. But the, the last series of conference talks that I gave, it really starts with this fact that since we last talked, about two years ago- ... the the threat actors the, the bad guys have also been listening to the podcast or my talks- or seeing the ar- re-reading the articles and using the same tools that hopefully you guys, the defenders the, the maintainers- ... have been using, but they effectively flipped it or used defensive tools that were designed for defenders as an offensive tool for them to find zero-days effectively. Because those tools, they can find vulnerabilities in the the attack surface of the, the build pipelines, and then lead to poisoning the the well, poi-poisoning the open source release. And that's what we saw with worms, like in the fall of 2025. I'm sure all of you have heard about the different worms spreading on NPM and PyPI mostly, shyahlood, and there's been a number of iterations of those- Yes ... autonomously propagating malware in, in open source packages. And e-even in the spring of 2026, we saw another wave that was not necessarily using worms, but much more let's say, skilled threat actors that were r- that really understood the benefits that they could get by exploiting effec-effectively large open source projects and have downstream effects in large organizations, right? Jonathan: That's the, the classic supply chain attack, right? Particularly if you can pull that off without being spotted for a while. So then you can you get your tendrils into one project, and that gives you a l- a, a diving board to then get your tendrils somewhere else. And the longer someone can go without being spotted, the more of a mess that they can make. Francois: Yeah. But at the same time even if it's not very long, but everything is automated, they can, achieve the end goal- ... v- so quickly that no one can really stop something like that. Like- Right at best, it can be detected as it happens or as it has already happened. And if in some case we saw them pivoting multiple levels from the initial access vector, the initial point of entry- ... Francois: To steal secrets and then pivot down- downwards in the infrastructure, like cloud infrastructures- of large corporations namely. Jonathan: Yeah, absolutely. And then you, I think sort of tongue in cheek, but it's a point worth following up on. You mentioned the idea that, oh the, the attackers are listening to our podcasts and they're using our defensive tools, and that's the challenge anytime that you build like an auditor or a a red team tool that like, there's not a whole lot of difference between what an auditor does and what an attack tool does. Is that just the nature of the beast? Is that just the way that it works? Francois: Yeah, I think there's no... we're not in the 1990s anymore- ... where we try to keep the precious, elite tools behind the curtain, and we only give them to certain trusted people. The cat's out of the bag. Yeah. The techniques are well-known. And recreating most of those tools is not rocket science. Once you understand what you're looking for, you can probably just use grep, in many cases, it's just configuration mistakes or a couple specific things that you're gonna be looking in this case, in YAML files. For the most part, those CI/CD configuration will be in YAML files in specifically named folders. And yeah the simplest thing you could be using grep and highlight some things that you would then audit manually. So those tools are just doing that more efficiently, but ultimately that's, as you said it's that, that's what we have to wrangle with. The technique is known and by b- both defenders a- and attackers. Jonathan: Yeah, absolutely. We're not in the position where if your CI gets hit, you can just say, "Oh, it must have been a nation state that did it." No, it may have literally been a teenager in a basement. Yeah ... I blame, and again, it's not just a GitHub problem, but GitHub is the, the 800-pound gorilla in the room of of online Git repositories. And I blame GitHub for this, 'cause I've gone down this track a couple of times. You made me... I don't think that was the first time I'd heard of it, but you definitely made me more aware of this when we talked about it last time. And so I've gone and I've looked at some of the projects I've been involved in, and I've gone down the rabbit hole of do we do this right or not? And I've read through the GitHub documentation on things like, just for instance, the, the pull request event versus the pull request target event. And I will read through their documentation and come away with one feeling of "Okay, we need to do it this way." And then I go and read somebody else's blog, and they say no, you need to do it this way." And I still don't know if pull request or pull request target is the safer way to do a CI run. It's... I just, I wish there was a single answer somewhere that I could look to, and it's like, "Don't do this." Okay, I won't do that. Francois: Yeah. The, the s- the simple answer is pull request target is m- let's say- Is if you're to grab something, pull request target, it's not the only one, right? But if you're gonna flag that, like that's a clear thing where you need to stop and audit the rest of the code. It doesn't mean it's not is insecure kind of a Boolean flag, right? You m- you can use this in a safe way. It's just that it's a bit of a foot gun. Like you, you need to really understand how to use it in a safe way because it comes, great power, with great power comes great responsibility. It does grant access to the secrets during the execution of the runner. The big difference between pull request and pull request target, and specifically in the case of GitHub actions, is that one of them will not grant access to secrets, but it will i- it will run the latest version of the pipeline. So basically, this is in the case where you want your contributors to make- ... changes to the build pipeline- ... and you want them to be tested immediately, right? So that is let's say in the scenario of enterprise or like in a private repo or when you have direct maintainer contribution, this is fine because you want to test like directly the, the latest version of the build pipeline changes. But pull request target on the other hand, it effectively locks the version of the CI configuration to the one that's already been merged. So that's the biggest difference, is like one is untrusted build configuration, and the other one is trusted build configuration. But the big difference is that one doesn't have access to secret, the other one has ac- access to secret. If it's a trusted pipeline, it means that as long as there's no vulnerability in that pipeline configuration, everything is okay. Jonathan: But- But if your repo has a Python script in it that gets evaluated and executed during build time, which is quite common by the way, you have a problem all over again. Francois: Yeah, exactly. So in, in fact so much so that y- you were asking did the, is GitHub doing enough? And I would say in the past three, like four, basically since March more or less, there's been a big like... They've been, they've started to really move the needle. And that's in response, direct response to all the big things that happened- Yeah in the spring. So they've made a number of changes that actually in some cases are breaking changes. Like literally they decided to do the very non-Microsoft move of, okay, this is too bad. Too many people don't get it right, so we're gonna completely change the behavior to a more safe by default. Where you really need to explicitly go ahead and say, "Yeah, I know what I'm doing. Let me do it." As opposed to leaving foot go- foot guns, around. And so th- that, that's a pretty major change. Specifically to pull request target, they made a change that was made fully effective as of two weeks ago or so. But it still makes it possible to have the same kind of vulnerability as before, it's just that there's one less way that you're gonna do it, like without even knowing, right? ... It's at least a little bit more explicit on your part. Jonathan: And that's why everybody's CI runs started breaking randomly a couple of weeks ago. That, that solves one mystery. I, I think something that maybe not everybody realizes I have to remind myself of this every once in a while, is obviously Microsoft owns GitHub. GitHub is the owner or the, the trusted source of NPM. And so like all of this stuff comes back to the same group of people getting black eyes from it. And so it's not terribly surprising that they are very motivated to try to fix this. Francois: Yes. The, directly Microsoft for many of the things, like if you're a Azure cloud user, there's a number of components that are directly used in production in Azure, both from the infrastructure side as well as the end user side of their cloud services, like kind of cloud SDK, Azure SDK, and other things, right? That you want to interact with their backend that are hosted on GitHub, open source, and that use GitHub actions. Some of those components are deployed on NPM, so they have vested interest in ah, hardening the path to to Azure, right? Because from commercial perspective, ... Microsoft definitely has- Sure incentive. But it's for the benefit o- of everyone, because a- any, any improvement to this infrastructure is good for open source. Jonathan: Yeah. Absolutely. All right. So that's sort of the background. And last time we talked about Poutine, but now you've got a new tool, Smoked Meat. I just gotta say, I love the carnivore theme going on here. I 100% approve of that. But what is Smoked Meat? What does it bring to the game? Francois: Yeah. Okay. So Poutine was really a scanner. It's meant as a static code analysis scanner. It's meant to basically scan and find all those special YAML files and detect vulnerabilities, like in the misconfiguration in those YAML files. It's gonna say, "Oh, you have pull request target, and then you have X, Y, Z other conditions, preconditions for it to be considered almost certainly vulnerable, almost certainly exploitable." So there's some, caveats to that, like in practice, but it's a scanner and people would ra- run it. Since it was published it's now being used by many large organizations many big open source projects. And it's very lightweight, very easy to run at scale. You can scan a large organization with hundreds of repos thousands of pipelines in just a few seconds, and it's gonna spot those. Then you're responsible for, looking at those results and hopefully fixing the issues, Jonathan: Yeah. There, there's a- ... and Francois: on our side ... Jonathan: it's worth breaking in here and talking about there's a challenge when you run one of these static code analysis tools to then correctly understand the results, make the judgment call of either, "Yes, I think this is a real problem," or, "No, I don't think it is. I think this is a false, false positive." Or even like, "Yes, this could be a real problem, but we mitigate it in other ways." Yes. And so there's a real judgment call to be made there, and sometimes that's really a hard step to take. Francois: Yeah. And exactly, that's, I'll get to Smoked Meat, your, to answer your question in a moment. But a- ab- absolutely as you said, it's it's difficult, and we've been using this tool at scale to scan millions of open source projects. And in the past two years or so, we've made several thousands of coordinated disclosures to many projects like big and small. Like Small, tiny open source projects, as well as things that are maintained by Google, Python Software Foundation Red Hat. And so indeed, like some of those are pretty trivial, fairly easy to point to the bug and to the solution, the way to re-remediate. But other times it can be very difficult to confirm that it's exploitable- ... and then to find the way to address it. In some cases it's more architectural. You need to redesign the whole thing because you, you made like wrong assumptions on how to do that securely, and that's much more painful, right? Jonathan: Absolutely. Francois: But let me get to the answer to your question about Smoked Meat. So Poutine is a scanner, and Smoked Meat is a red team, like offensive tool. So the first one is gonna find the vulnerabilities, and in Smoked Meat I use Poutine as a library. So if you remember, Poutine is a Go, is written in Go. It's a common line tool, but here I just use it as a library, as part of a bigger project called Smoked Meat. Which is really, if you want to simplify it for those of you who may have heard about Metasploit. So Metasploit is an offensive tool, like a red team exploitation tool- Yeah ... where it's gonna scan to find vulnerabilities and then give you automated, like automatic craft payloads to test ex-exploitation and validate that indeed this given vulnerability can be exploited and then you can discover what's behind. Once you achieve remote code execution, you can, in some cases, like find credentials or discover the environment, do further reconnaissance and pivot. Part of Metasploit, there's this plugin called Meterpreter, which is effectively a small shell execution environment that will run remotely. Once you achieve the re-remote code execution, effectively you move from your laptop to somewhere further and you rinse and repeat. Like you execute as if you're sitting on that server observe what's around, and then continue. Smoked meat is this, has the same exact concept, but applied to the context of CI/CD. And the reason why Metasploit would not directly work in the context of CI/CD is because in almost all cases, CI/CD is running ephemeral environments. So the VMs that run your tests, it's typically torn down and completely deleted the minute the tests finish running. Then the next run or maybe parallel run of the test will use new VMs, so it's not like a persistent infrastructure like you would attack a cloud infrastructure typically. So yeah, so that's the gist of it Smoked Meat uses Poutine as a library. It scans, then it will automatically generate the corresponding payload for the different vulnerabilities and let you basically exploit it, get whatever secrets are running in the CI environment. And then re- rinse and repeat, discover further what, what's behind. In many cases it's very common, especially for a bit less for open source projects, but in a commercial enterprise environment, you will have a CI environment deploy to a production cluster, right? And then maybe this cluster is connected to something else. So that's that's really what Smoked Meat is all about. Jonathan: Yeah. You get a foothold onto maybe somebody's network or somebody's CI runs. Yeah. It's a real problem, and that's how you get into some of those messes with it's how you end up with things like we've been shipping this project or this product, these binaries for the past two years, and oh, they've had a backdoor on them the whole time," right? It's that it's that sort of supply chain problem that you can run into. Yeah. So super, super important. Does Smoked Meat work a, so if we wanted to use it inside of a Metasploit run, can you do that? Can you build- ... bu- build the stack on top of each other? Francois: Not really. It's really, it was designed as a turnkey solution, end-to-end, purely designed with the, the intricacies of CI/CD in mind. All the tricks that basic- It knows exactly what will happen when it lands in a GitHub Actions runner, the environment the tools that are a- accessible there, and the different services a- around it. And how to get access to the secrets and the different features that are inherent to GitHub Actions. So it's really meant to be used on its own for that specific type of red team engagement. Sure. I've actually created a capture-the-flag environment for people to experiment with that safely- That's great so that they can practice i- in, in a, an environment that It has a number of challenges. You start basically from a public repo, and once you achieve the first foothold, then you discover private repos, and then eventually you discover a cloud environment, and effectively you can hop through four or f- five or six pivot points a- and get quite deep. And you in, in the process, you exploit about five vulnerabilities in different CI/CD runners basically where you start from that public perspective initially. Jonathan: Yeah. That's actually really neat. I appreciate it when people do that with the tools, because it's very nice to be able to learn how to use tools without actually breaking the law. Francois: Exactly. That's Jonathan: So that's- ... that's really important. Francois: Yes. I mean- We don't, Jonathan: we don't wanna all become Kevin Mitnick, right? Francois: Yes. Researchers like, like me we all create those kind of environments to practice the proof of concept whe- before we disclose something. We try to reproduce that in a sandbox the separate environment where it tries to reproduce as many of the conditions of the real target. And, we can use that to take a video demo and say, "Oh look, if someone is- w- was gonna compromise that, that's how it would execute." And we can do that without doing it in the public repo where other people could notice that, "Oh, look at that, someone actually achieved poisoning the thing," and maybe the maintainer is on vacation, and someone else is gonna come back. So we need to do that ethically, and that, that's why this tool is released alongside an environment that people can practice and learn and then test it on their own projects again, maybe in a separate environment that just tries to reproduce the real setup and before they- they maybe apply the patches. Jonathan: Yeah. It brings to mind a potential problem, and I'm curious if you guys have seen this, 'cause obviously you've been using this tool, and you've used it to help find problems. What do you do if a project is maintained, or unmaintained, but it still has a, a CI vulnerability? Like, how do you even go about trying to fix that? Francois: I think it's the case y- that, that's a good point. And I guess you have to see un- unlike a vulnerability in the code itself, right? Where it would be subject of a, a CVE afterwards once it's patched and people know, okay, this, there's been a patch applied to this specific version- of, and so you should update to the latest version, which has addressed it. In this case, it's really an infrastructure. So it's not the code itself- ... where it is the YAML in the GitHub repo, but it is effectively a side door. It's another way to- ... get access to the repo, like as a s- because you achieve remote execu- execution in a privileged environment. Yeah, to your question if the project is completely unmaintained and the CI/CD is vulnerable I guess hard forking it is is the, what's necessary because otherwise it could be modified and in some cases it can be hard for someone to really determine that it's been compromised if there's no collaboration from the side of the, the maintainer. The, in terms of forensics, in terms of confirming that it's not been exploited, it can be difficult sometimes. Jonathan: Yeah. I'm thinking back to, of course, one of the most famous examples of this, this sort of problem. Not exactly CI related, but still close enough to talk about, the, the XZ hack where the, the backdoor was put into the XZ libs, and one of the things that made that more, more complicated is that Lassie Collin, unsung hero, absolutely the good guy in the story, but he was on vacation when all of this broke out, right? He, it took them several days to track him down and get him to fix things, and that made it all harder to clean up. And I'm just imagining I have seen, I've seen GitHub repositories that are still being used that nobody has touched for months or maybe years, and they're just running on autopilot. And it's not at all difficult to imagine a scenario where an attacker gets access to one of these through a CI run and pushes out a new version or rewrites a Git tag. Which by the way, if you have the option, you should really be referencing things by Git hashes and not Git tags. That's sort of a different story. I, it's, boy it's real challenge to try to deal with that. Do you guys have like contacts at GitHub to where you can- Yes ... you can clean something up if there's nobody at the wheel? Francois: No we do. So as I mentioned, we do coordinated disclosure with many projects. And we do have privilege like contacts, like we establish relationship with people at GitHub, but not just at GitHub. And in some cases, there... I've seen examples where the project was so high profile and the maintainer was overwhelmed or- ... for some reason, and the, the ecosystem impact was so broad that, yeah, like GitHub in some cases like put a freeze, like paused any new pushes to the repo. A- and some cases like reverted, but that's pretty rare. In... I haven't seen many examples of that. Maybe I can think of two or three at most in the past couple of years, where the maintainer was MIA and GitHub had to step in and freeze everything. Otherwise it was cascading. Like es- Yeah ... especially in the case of the worms, like in the fall, it was so bad, it was propagating so fast that they needed to do something a bit more drastic. Jonathan: Yeah, that's a big deal for the for the repository host to step in and make changes. That's outside of the normal social contract with how all of these hosting things work. And so it's gotta you gotta have a, a... It's gotta be a pretty drastic scenario. I think, so talking about XZ, I think that was one of those cases where GitHub hit the button and freeze this and turn it off, and nobody can get to the, the XZ repo anymore until things got fixed. Yeah, I think they did. Yeah. So we've talked about GitHub quite a bit, but this is not just a GitHub issue. Do you have sort of tales from the trenches from any of the other major hosts or anything outside of that, actually? I would love to- ... I would love to know if this is a proj- or problem that you see in just some maintainer's random glue scripts. Does it show up there, too? Francois: A- absolutely. A- again, like it's really a matter if we're talking about... first of all, there's kind of two main threat models that I have to to tell people about. So the first one is the one we just discussed, which is really the public open source project. People can contribute changes using pull requests, right? They will submit new code that is not yet reviewed, and if the CI is going to automatically trigger upon the submission of code changes, then depending on the configuration, we can lead to what we discussed. If the maintainer has been compromised already to some ex- to some way phishing- info stealer malware that ran on their computer, exfiltrated their credentials, or stole them through another, like a p- like basically a first attack- ... maybe a first CI compromise led to getting se- secrets credentials of a maintainer, and then use them to directly attack another project as the maintainer in this case. Like basically impersonate the maintainer, and that's... We've seen that. Also, the, the worms were really using that in some cases to different techniques to seed, to just bootstrap the the, the malware by phishing info stealer, and a CI compromise. And then after the, the worm would... in the, the case of direct automation, it would maybe get as I said, like some credentials of a maintainer, and then discover which projects they can directly attack. So with the second one, which is y- you're basically already the maintainer, , it's kinda hard to to stop that, right? It's not a vulnerability. It's just like the way that you would modify your own project. And we've seen that across any ecosystem. This is really not specific to any, any given platform. And the same thing, like if you steal like a PyPi publish token or NPM publish token, you're gonna publish directly and skip, s- skip touching the source control environment. And in fact, that's what most malware author would do because they, they want you to believe that the source code of the open source project is all good, but what's published on PyPi is completely different. Jonathan: Yeah, definitely a challenge. So what's the, i- d- does d- does Smoked Meat or Poutine, does it give any visibility into that situation? What, what can we do to even mitigate against th- that, that realm of problem? Francois: I- if your question is about a maintainer be- being directly- Right ... compromised, like their own credentials. So in, in between the time that we we came out with Poutine and before Smoked Meat, there's a third tool that I haven't mentioned that we released called Bagel. So all the food theme. Jonathan: Absolutely. Francois: And Bagel if you don't, didn't notice, like, all those food items, they're all kinda staple Montreal food because, we're a startup in Montreal, and Poutine is a Quebec- ... Francois: Food. Smoked Meat, if for those who don't know, who never came to Montreal, Smoked Meat is the way we refer to the, a special type of brisket smoked brisket with a marinade and, you know- Jonathan: Yeah Francois: it's just called Montreal style Smoked Meat. Jonathan: Oh, yeah. Francois: And Bagel, we have our own Montreal style bagels that is different than the New York style bagel. Got it. The point is that Bagel is all about looking at the posture of your laptop, your e- endpoint, where you write code, and trying to limit the blast radius, limit the impact of a malware running on your laptop- So that, for instance, take a very simple example. You have your SSH private key in the home folder without a passphrase, right? Malware runs, it's gonna get that, and the passphrase the, the SSH key is usable immediately by the attacker, versus you have a very long, complex passphrase. Even if they get that, they would need to crack the passphrase before they can use the SSH key, right? So things like that we would if you run Bagel, you're gonna find secrets lying around your laptop, maybe in your bash RC you load as environment variable some, some API keys to automatically push somewhere and, things like that. And then it comes with some kind of remedi- remediation guidance to tell people how to do that in a more s- safe manner. Because we know that, like- You're gonna click the wrong link, you're gonna download something- ... something will run on your laptop, and then- Yeah. That's true ... if you limit the amount of things that something running at a given moment has access to or there's at least some kind of user interaction, some kind of confirmation. Personally I'm not here to advocate for, I know that you're a big, like, framework la- laptop user. My colleague actually has that. But on, on a MacBook, I use Touch ID, which the secure enclave fingerprint-backed execution environment to store private keys. So my SSH key, for instance, requires me to put a fingerprint in interaction. So even if a malware runs, it would, the malware would need to trick me into putting my fingerprint exactly at the moment it wants to sign- ... using my private key. And then that means that the malware could not do that 1,000 times in one second. It would need to have me press 1,000 times with my finger, right? And the same thing applies to many other types of credential like a passkey, a YubiKey, and whatnot, things like that. They effectively move the sensitive key material off of the normal userland execution environment so that- ... malware running on it will not be able to at least propagate autonomously so quickly. Jonathan: It- Francois: So- Jonathan: It gives you more friction in your workflow, which you sometimes need. Francois: Exactly. Jonathan: I've talked about that- At least for Francois: those- Jonathan: Yeah, I've talked about that concept of friction when it comes to AI pull requests, but I've never quite thought of it in these terms, but that's really an interesting connection to make. Francois: Yeah. You want- So in fact, talking about AI and whatnot, ... we didn't do that We had to go Jonathan: there. We had to go there. Francois: Okay. Okay, yeah. How many minutes in we didn't use that acronym? But so Bagel is, one thing it is gonna find is if you use Codex Cloud Code, or whatever on your laptop- maybe you don't know, but it saves those conversations in a temporary folder- ... so that you can resume conversations. And very often, without you even knowing, maybe you never copy-pasted some secret, but in the process of Cloud Code reading logs to- Yeah ... for you to troubleshoot something in a production environment, it actually- It caches all kinds of stuff pulls in the context. Yeah. And those secrets are gonna be in that temp folder, so Bagel is gonna find them right away. There's even a feature to scrub it automatically, like- Oh, nice ... redact with, like a, a fake token. Jonathan: Yeah I imagine that part of the solution to this also is just to carefully scope your tokens so that, you create- Yeah a token for one use, and it only works for that, and it does not, it's not the keys to the kingdom for everything you have on GitHub- Yeah ... or every SSH, what have you. Francois: Yeah, basically, like, if, especially people who maintain open source projects typically everything is public, so read-only access to- things that are already public is not considered a sensitive secret. E- even if, go ahead and steal a token which just has access to a public repo, just read-only, big deal, right? And that's exactly the design consideration that GitHub has when it comes to pull request versus pull request target. The execution environment will get a secret that has read-only access to, in that case, a public repo. So the attacker gains no no further foothold. It's at the same level as it started initially. But so to your point, yes having different credentials, like a default credential that for instance, that SSH key, you can split your SSH key for pulling and pushing, but signing the commits is a different key material. You can do a SSH-based Git commit signing. So having two different keys, one that has the friction of the fingerprint authentication, the other one without, and then- ... when you do auditing after, afterward, you, you can confirm, okay, all my Git commits are signed. They're signed with the key that I expected and anything that deviates from that is, to be, Reviewed with suspect Jonathan: investigated maybe. Yeah. Yeah, that makes a lot of sense. So you opened Pandora's box. You talked about AI first and now I gotta ask about it. And i- in, in just in thinking about the... Of course, the world has changed significantly since we talked two years ago. I'm just kinda curious is there a, is there a strong argument to be made still that Poutine is the way to go to do these scans versus just, "Hey, Claude, take a look at my .github folder in this repo and tell me if there's a problem"? Francois: No, that's a good question, and in fact- We, when we do this kind of scanning of millions of open source projects, we leverage agentic like agentic approach. We do a hybrid approach. So basically we use Putine at scale because it's very efficient, very lightweight, no tokens needed. And we've tuned- And you don't pay $1,000 Jonathan: a month for the Francois: tokens to run it . Yeah, exactly. So we've tuned the rules in both... In, on one side, either it's very narrow but very high signal. It's like if it's gonna flag that, it's nearly 100% that it's exploitable. We can report it almost without looking. I'm not saying that, but it, it gets to the point where the rule, the deterministic rule from Putine is gonna highlight the, the, the low-hanging fruits. Then the other approach is we use Putine with effectively several preconditions that are not necessarily all proven to be l- lining, like all the ducks, not necessarily all in, in a row. But then using AI to more creatively combine and say, "Okay, yeah, it has this condition, but not that one. But given that it has this other thing, there might be like a, a less obvious way that static, like a deterministic scanner- ... would have skipped." So a hybrid approach is really what we've been using at scale. A- and also for those more advanced kind of multi, m- like chained exploit where you have a low risk thing combined with another one. That's the, the, the new thing that is now possible using Claude and others. Jonathan: Yeah, that makes a lot of sense. It is amazing to me how quickly we have gone from, daniel Stenberg was one of the first guys to really, to blow the whistle on this, because curl is one of the most installed projects anywhere. But people trying to game the trying to game the system and be able to find exploits to make money in, in, the, the various bug bounties, to u- using AI to do that and therefore just creating absolutely garbage reports. How quickly we've gone from that to AI is now a tool in every security researcher's toolbox, and it's finding real problems every day and helping us fix them. I it seems like I blinked and we went from A to B. Francois: Yeah. It does require ex- experts in front of the, the keyboard. But that's, there, the... I agree that there was a period where the experts were like non-believers and in the meantime, you had the non-experts that were just, like creating slop. And then people s- realized that, okay, if you use that smartly and you don't just trust the output right away, but you use it to augment your scanning, like vulnerability research it's quite amazing, yes. Jonathan: It, it seems like if you're using the latest models though, they are better about not giving you slop. Francois: Yes. No, absolutely. Jonathan: There's always, just like when a human finds something, there's gonna be a judgment call as to like how serious this is. Is this a real problem or was this part of the design? Is this an accepted trade-off? There's all of those things that honestly that make our job harder. If you could just, if there was just a magic rule set that, oh, this is a CVE and this isn't we would probably be out of jobs, right? Yeah. Francois: That being said, if we go back to the question related to did Git- GitHub fix the whole thing they are making... they published a roadmap with a number of changes to the environment where you can now set policies across your organization or specific repos that will turn off or basically prevent execution of workflows using certain risky features. So now you can effectively decide, like purposely decide that your policy is to effectively remove those footguns or explicitly allow some of them in specific cases that you did do your due diligence. So there's a lot that GitHub has done in the past two months or so, and there will be more in the next few months for those of you interested. Jonathan: I, in, in thinking about this more and having listened to your explanation of the two targets in GitHub runs, I don't understand why GitHub does not make a third target that is like a pull request target safe, where you take the safer of both of those two options. So you pin the .github folder so that you're running your known safe CI, and you also don't share the secrets with the run. This seems like an obvious fix. Francois: Yeah. No I understand. It's, yeah. At the same time- Jonathan: That would make life so much easier ... at the same time- We could just go set that for everything and be done with it. Francois: No, it's true. I think, It all comes down to is the code in the pull request, has it been remo- reviewed? Do you have execution o- of it before it gets reviewed? But every... As soon as it gets merged, it's merged, right? So there, it's trusted. So it's really this this, this line of clicking merge. But sometimes people review very quickly and merge things that they should not, Jonathan: Say it's not surely we're all extremely careful, and we read every line of code before things get merged. Surely. Oh, absolutely. All right. I think we've done a good job of covering the problem space and the tools that, that you guys have put together for this. If you look into your crystal ball and you think about what's coming in the next six months, we'll make it a little easier for you. I was gonna ask for the full year, but we know how much things can change in a year. Just the next six months of sort of this problem space, what what do you see coming? What are you watching as, as maybe the next trends? Francois: That's a good question. There, there's a number of things, as I mentioned that, that is happening that makes it harder for threat actors, much less obvious. But it doesn't mean that that we're, it- it's a done deal. Because ultimately CI/CD, I think I sh- personally I think we should all rename CI/CD RCE as a service, because it's really much easier to think of CI/CD as an execution environment running arbitrary code, right? And if you start to think about that- ... a- and design your environments like, But for the next six months yeah I don't know. I think the, the threat actors seem to the bad guys that have been very active in the past couple of months, they seem to have been a bit more quiet. I think they're feeling the heat. But if we remember what happened in the fall of 2025, I hope it's gonna be quieter. Jonathan: Yeah. Francois: But I don't know. Supply chain security as a whole there's so many angles. The build pipeline was a we- weak, weaker spot, I would say. Very easy attack surface to, to profile from the outside with Poutine or other tools. And the other angle, the maintainer side phishing, we're never gonna fix that limiting blast radius is something we can at least do. The, the, the package ecosystem like NPM and PyPi and others they've been doing quite a bit of work in that respect as well. So I, Jonathan: I will- Francois: I, I don't know ... Jonathan: I will throw in there that the idea of a, a malicious maintainer or a compromised maintainer, there, there are some tools out there that you can use. So we've got one repository that's set up to where I c- I don't think I can do a f- I'm the owner of it, and I don't think I can do a force push to it. And particularly if I do a pull request- ... I cannot approve and push my own pull request. There has to be a second set of eyes that go in there and gives it a review before it'll go in. So there are- Yes ... there are some tools that, I don't know that either of those are a magic bullet, but there's gonna be w- ways to at least to some extent mitigate that. And I think, if people are working with important code bases, that's definitely something they should take some time and investigate and see if they can get set up. If it's practical. Not every project has two dedicated maintainers, and I g- I get that. Francois: Absolutely. Jonathan: So Francois: that's Jonathan: that's a real challenge. Yep. E- as you look at at some of the attacks that have happened, do you get the idea that some of them are, like, big nation state actors, or do we think most of them are just essentially cyber criminals looking for a cash-in? Francois: There has been very few... when it comes to attribution, that's always you don't want to just, wave your attribution without a- as access to as many of forensics evidence. But yeah, I would say probably 98% of the cases that I've seen, it's clearly just people just having fun, ... Francois: Or with a intent of gaining some kind of, you know, like- A lot of them- ... money out of that ... a Jonathan: lot of them come down to stealing Bitcoin wallets, don't they? Francois: Yeah, absolutely. In many cases, if you're... And that's, that, that would go a bit closer to some nation state, which I will not name, but those interested in this topic, they know that there is one nation state threat actor that is very fond of crypto. And- Jonathan: the one that comes to my mind is not in the South and it's not in the West. Francois: Yes, it is. There's some compass if you were in a different... yes. And they, basically any open source project that is near, near or directly in the path of anything related to crypto infrastructure- Oh, yeah ... be it cryptocurrency wallet or some kind of backend- Oh, yeah ... it's dangerous. So- I Jonathan: mean, those guys just have a- ... you Francois: might be a collateral victim ... Jonathan: giant target painted on them. Yeah. There, there is... it is impossible for those guys to be too paranoid. Yeah. Francois: Yeah, absolutely. So basically, if you're gonna be using pretty much anything that some cryptocurrency infrastructure happens to use also you might unfortunately be a collateral victim of that, and it's kinda hard to know, right? If you go back to spring of 2025, C- Coinbase- ... was the actual intended victim for the TG Actions CI/CD compromise. So if you remember- ... there was TG Actions change files was a small GitHub Action component that many people were using. But the final victim was Coinbase. And yeah. But in that case, it was not, likely not a nation state threat actor, but still someone that was after Jonathan: cryptocurrency. Yeah. Yeah. That makes sense. So we talked a lot about the open source stuff. Tell us briefly about Boost Security. What do you guys do? Francois: Yeah. So we have a, ob- obviously like a commercial offering and we have the research angle that I've been talking about the different projects. So we have a application security posture management product that is in some cases like using the open source projects that we put out there and, put a layer that's more like for enterprise grade kind of policy and scaling of of all that. And yeah, w- we have an, a whole new set of offering that's a bit closer to when I mentioned Bagel, where it's gonna be looking for the laptop of a developer for enterprise users like they... We have a product specifically for that and it's quite on the mind of many security-minded people in, in, in medium and large organizations now- Absolutely ... because the laptop of developers has been effectively ignored in many cases, like by traditional EDR and things like that. And so that, that's a big angle that we're looking to address now. Jonathan: One of the reasons is because as a developer, it is such a pain to run almost any of the existing antivirus products. Like almost all of them are just terrible and will get in your day and ruin your... Get in your way and ruin your day. So most of us developers, me for one we're tech s- we're in this dangerous place where it's like, "Oh, we're tech savvy enough to keep ourselves out of trouble. I would... I know what I'm doing. I would never get my laptop compromised." Also followed with, the big company's antivirus project or, endpoint security pro- I know too much about it. I would never run that on my computer. And like- Yeah ... I get it because I'm right there too, but at the same time, that's a terrible and dangerous place to sit. Francois: Yeah, so I c- I can tease that we're gonna be coming out with something in the near future that people may... People interested in that area, they may want to look look at what w- we'll be announcing in the next couple of weeks. Jonathan: Yeah. I would, man, I would love for, I would love for there to be a really good endpoint security solution that was not insulting to run on my own machines. Maybe that's you guys. I would lo- I would love for that to be you guys. That would be so much fun. Parts of that open source? All of it open source? Francois: Part of it- Parts of it ... is definitely open source. The rest we'll see when the announcement, Sure ... comes, but Sure. Yeah, Jonathan: absolutely. I will make sure and watch out for that. That sounds that sounds really cool. Sounds interesting. I unfortunately, a lot of us, we get in situ- ourselves in situations where it's like because of whatever we're working on, we're required to have something running, and it's like, "Okay, fine. I guess I'll go install Malwarebytes. That's the one that I hate the least," or, something like that, and maybe not necessarily the right answer. Just because I hate it the least doesn't mean it's the best answer. Francois: Yeah the, the issue is that almost all of those EDRs or mal- antivirus, they were not designed to work on a developer laptop- where you write new code, and you compile it, and you run it five seconds after you, you wrote it. You modify a Python script, you execute. What's the difference between the one you just wrote five seconds ago and a malware that you downloaded where you Wget some Python script and then you execute it? Unless it's really need threading the needle, or you downloaded that from a third-party source, and then even then it... That happens all the time. That's what we do to build different things, right? It's really more about the behavior of, it's not normal for your SSH private key to be loaded in the memory of something that, it... There's a number of behavioral things that you can look at to, to at least either a- alert or block, right? Jonathan: Yeah, very interesting. Definitely looking forward to that. Is there anything that we didn't touch on that, that we should have? Francois: No I think that's pretty much that. Obviously, like Smoked Meat is a tool that's not for everyone. It's something that is more int- interesting for pen testers- It- ... red team operators and whatnot. But, Jonathan: I'm sure somewhere in the source code and/or on the website, there's a notice, "Do not run this. Do not break the law-" Yes with this tool. Do not run this on live code anywhere." Francois: For sure. But as a learning tool, like for even for people who think, "I'm no- never gonna be running that tool," we have a YouTube short YouTube video, three minutes- ... that people can see how it works and the type of damage that can be done if ex- exploitation of a CI/CD environment is to occur. And it can be used as just a, a, a learning moment for, for people to tell others that they should maybe be running poutine and look at auditing their CI environments. Jonathan: Yeah, absolutely. We do have a bit of a live audience today, and they've been going back and forth with the food puns, and Mashed Potato says, "It's not for vegetarians." I have to point out, it's the bagel. If you really- Yeah ... if you really want the vegan option, you've got the bagel. Francois: Technically poutine is not vegan, but it's vegetarian at least. There you Jonathan: go. I don't know. It sounds pretty good to me to put the smoked brisket on top of the fries and the gravy. That sounds really good. Just have it all together. All right. Thank you. Thank you so much for being here. I've gotta ask you a couple of questions before I let you go, and we asked them last time. I don't expect the answers to change, but we still, we gotta get them in. What, Francois, what is your personal favorite text editor and scripting language? Francois: Ah do people write code anymore, it's just like prompt code. Jonathan: I know. I know. Oh somebody one of these days is gonna tell me that "Oh yeah Claude Code is my preferred text editor." Yeah. I'm gonna have to resist the urge to strangle them through the screen. Francois: I was almost gonna say if you are to profile the number of times that I... The answer is VI. For me it's a VI guy, i've never installed Emacs on my box. But I do use Z- Zed as like instead of Visual Studio Code sometimes- Okay ... like for a lightweight more visual- ... like a Z- E-D. But yeah, VI guy. Jonathan: Yeah. And then you were gonna make the point I think that if you measured it by lines of code you write that something like Claude- Yeah might end up being the winner. Francois: Yeah, probably. Pr- well, Codex, but yeah. Jonathan: Okay. Yeah. That's... I would... To, to, to each their own, right? All of those, all of the different models have- Yeah ... their different wrinkles, and I think they're well-suited for different tasks, and we all get used to Francois: one. Especially for security tasks. I've had some... Yeah. Like the, Cla- Cla... I love Cla- Claude Code, but like for security professionals, it, it's- there's been some some some drama in the past two months or so. Jonathan: Sure. Francois: So we Jonathan: can see that. Although they do now have a some form you go fill out. One of my, one of my other maintainers did this, where it's like, "I promise I am a maintainer of this project. I promise I will use this responsibly. Please let me hack with it." And they will look at it and go, "Okay, fine," and give you m- more access. I don't know. There's... I'm sure there's still some guardrails, right? But they will give you a little bit more access. I- in my experience, Francois: that's not enough. I'm part of those, those those programs- Okay ... and even then the, the guardrails are a bit annoying for professionals, like security professionals. Jonathan: Sure. Francois: But- Jonathan: Ha- have you found have you found that running models locally lets you get around some of that? Is there a a case to be made for doing security research with local models- I- ... for that reason? Francois: I think so, but it's quite expensive. And quite... I think if to run any model that, that is even close to the, the frontier models, it's impractical for- Yeah. But I did, yeah, I did mess with some smaller models. They can be useful for a very specific a- scenarios, yes. Jonathan: Yeah. Yeah. Interesting. Very cool. Francois, thank you so much for being here. It's been an excellent conversation. The, the hour plus just absolutely flew by, and really had a lot of fun with it. Thank you, sir, for being here. Francois: Thank you. Jonathan: Yeah, I appreciate it. All right. That is Francois Proulx talking about Boost Security and their open source projects, Poutine, Bagel, and Smoked Meat. We've got a whole meal, Canada style, and that's a lot of fun. We do have some great shows coming up. Next week we're having Jonathan Polance of Ferris Systems, and then I know we've got quite a few that we have emailed and working on scheduling. But we do have some slots open, so if you have a project or you know of one that needs to be on the show, you can shoot us an email, floss@hackaday.com, and we can get them scheduled up. We appreciate everybody that's here. Thank you for watching and listening, whether you get us live or on the download, and we will see you next week on Floss Weekly.
• 
  Episode 877 - RCE As a Service 29.07.2026 1h 7m

  This week Jonathan chats with Francois Proulx about SmokedMeat! That's the third in a trio of Open Source security tools from Boost Security, and this one is the red team tool to demonstrate vulnerabilities. Why are Continuous Integration vulnerabilities such a persistent problem, and what's on the horizon that may help? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 876 transcript 22.07.2026

  FLOSS-876 Jonathan: Hey folks, this week I talk with Michael Meeks about Collabora. They do a lot of things, but we're talking specifically about their office suite and how they pay the bills doing an open source project. This is Floss Weekly, episode 876, recorded Tuesday, July the 21st. There is no money fairy Hey, folks. It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and my voice is just a little extra raspy, a little extra bassy today, and that is because I've been at Open Sauce this past weekend. And in the infinite wisdom of the people putting the show together, they put our booth back to back with a guitar amp booth. So I've been speaking loudly to be heard over the sound of sick guitar riffs for about three days now, and my voice definitely took the hit. But that was a lot of fun. I'm back now, back in the saddle. Didn't miss any Floss Weeklies this time around, and we have got something really neat today. We're talking with Michael Meeks of Collabora, and one of our, one of our live listeners before we started the recording said, "Are we talking virtual reality, or are we talking office suites?" I think the answer is office suites, but apparently VR is part of the conversation. We'll get some information about that in here in just a moment. So our guest today, Michael Meeks he is CEO of Collabora, and he is, has served as the director of the Document Foundation. He's done all sorts of stuff over the years, including worked at Novell/SUSE, then part of doing lots and lots of Linux stuff over the years, and a, a whole lot more. Let's go ahead and bring him on. Michael, welcome absolutely to the show. It's good to have you here. Michael: Thanks so much, Jonathan. Pleasure to be here. Jonathan: So w- are we talking virtual reality, or are we talking office suites? Michael: That's a really good question. So Collabora, of course, is two things really. So it's a, a consultancy doing Linux enablement, open source stuff across the stacks- in multimedia, graphics all sorts of kernel enablement hardware stuff to get open source in at the bottom of the stack. So it's the obvious choice. There's always someone there that can help you get your hardware enabled, and so open source comes up from the bottom. And then of course, Collabora is also Collabora Office. It's always coming from the top of the software stack with these very, very leaf node- ... with all these dependencies. And so yeah there's two companies there. But we love to share the same brand and, work, work together. We have a mission, which is to make open source rock, and so we do that. Gotcha. And the hope is that in the end, the bottom and the top guys meet in the middle and shake hands and then everything is open source. That's the long-term plan, basically. But yeah- ... we're getting Jonathan: there. It's when they were digging the ch- the Channel, and they started at both ends, and finally they broke through and, shook hands through the rocks. Michael: Absolutely. Absolutely. H- So yeah, a, a breakthrough. Jonathan: How did it come about that there are these basically two separate companies that are under the same banner? Michael: Yeah, so I did a lot of work as you mentioned at SUSE Novell, as distinguished engineer there. And at some point, SUSE got really good management. And the good management realized that infrastructure software services and office productivity are pro- probably two separate things. All of these people are suffering in the Linux kernel at the bottom, toiling away. And, Linus says, "The interesting stuff's happening in user space. Up there. Go and do those things." And so there we were, writing Android apps and so on and people were a bit, aggrieved. And so fair enough, totally fair. We... They spun us out, they did it in a really good way Niels Brinkmann, Ral- Ralf Flakser. And they even gave us a, a small support contract for the first year or so to support their usage of LibreOffice, at the time. And yeah, just brilliant. So we got spun out, and that just makes decision-making so much easier. And helped me learn a whole load of, things about open source and business that seemed so simple back when I was employed and someone else was doing the selling and so on. And now, yeah, make... I think I was extremely naive, let's say. Yeah. To put it- I- Put it one way or another. I Jonathan: understand. Yeah. You got a very abrupt crash course- ... in herding cats, among Michael: other things. Absolutely. But of course, Collabora was a key part of that. So- ... Michael: I couldn't do it by myself. We took six other people from SUSE, and that's always an exciting ask, isn't it? "Please come and join my company with no customers, no... and no money and whatever." And so I think, Collabora played a really important role there, the parent company, in, in seeding that and providing some of that business experience. And my, my key partners there, Philippe Calaff at the time, Rob McQueen, really helpful- ... in terms of, just advising and providing infrastructure, finance and personnel and contractual basis and so on. So it was really useful. Jonathan: Yeah I had no idea until this very moment that Collabora was spun out of openSUSE. That is absolutely- Yeah, the office piece was- ... n- news to me ... the office piece. Yeah, interesting. Michael: Yeah. Jonathan: The... there's some news that everybody has been talking about this week, and that is that Valve and Collabora have released the official Arch Linux ARM64 port for the Steam Frame Michael: Awesome. Jonathan: And that is- Awesome ... that is something like what you're generally aware of that because you're the CEO- ... but otherwise you're just hands off. "Oh yeah, the guys are doing that. I don't have anything to do with it." Michael: So to be clear, I'm the CEO of the subsidiary that does office pieces. Philippe Calaff is the CEO of the, of the- Okay ... above. And it's a really, it's a great company. I think there's a lot of subsidiarity in those groups. Lots of groups that working, pushing out in their own direction. So yeah, I think sure this group is probably doing the valve bits there. But meanwhile, other people are reverse engineering GPU drivers and maintaining Mizar and improving. We- we're pushing in so many directions. Jonathan: Yeah, Michael: absolutely. So yeah, it's always great to see these things. Jonathan: Okay. Michael: We do so much good stuff, it's nice to be able to talk about it. Jonathan: Absolutely. So I feel like we've placed the landscape now and we kinda know the things out on the periphery that we're not gonna talk so much about. Let's talk about what Collabora, the office suite in Collabora Online is. And how did this come about? So is Collabora a fork of... the granddaddy of all of these is StarOffice, right? Michael: Yes. Ultimately yes. So if we go back to 2000 or before I was working with Sun Microsoft or Zimian. And Zimian, met with Sun and Marco Börries, who was the, the kid who created OpenOffice or StarOffice in his garage, and then built a big business around that in Hamburg. They kinda met together and decided how, to open source this thing and how best to do that. And so I was involved with that, before even it was open source, trying to think about component models and plugging things together. And if you remember component models, but the punchline is you don't need them. Open source is open, and so you don't need to have all of these special contracts. Just, mash the software together, improve it, and work, work together. But either way, back in the day we loved that. And so then OpenOffice, of course, lived for about a decade before LibreOffice came in. And of course, Oracle, the economics for the Oracle division there weren't, I think they say accretive to gross margin or something like that. Meaning you've gotta make an insane amount of profit, otherwise you're not welcome. And that's a great shame. And a lot of those Oracle people were being let go, redeployed in other places And so then LibreOffice came out of that. Jonathan: And Michael: that was great. It was funded by, I guess SUSE or Red Hat with help from Google. And, the Linux desktop was hot back in 2000. There, there were lots of people there, and we had big teams working on this. And yeah, and then of course, over time... So we founded LibreOffice, and over time some of that desktop excitement died away, let's say, or at least reduced in magnitude. And you then end up in a rather different world economically in, in terms of sustainability. And then of course, when SUSE effectively pulled the plug on this and said, we're stopping," that was, would've been devastating to The Document Foundation, LibreOffice. So we yeah, so hence- Gotcha Mortgaged the house, start the company, rescue as much as we can, and then grow up. And now I guess we're about 70, 70 people in the productivity office part of Collabora now. Jonathan: Yeah. There's yet another Office suite out there ONLYOFFICE. Michael: Oh, Jonathan: yeah. And I remember that there was some drama between ONLYOFFICE and Collabora back six months ago, nine months ago, something like that. And my, my comment- ... at the time was the whole story was just bizarre. What and obviously we're not here to throw stones at ONLYOFFICE, but I'm curious from your perspective, what's the what's the overview of all of that? Michael: Yeah. So at least from my perspective, we didn't have a huge amount of drama ourselves with ONLYOFFICE, but I think Nextcloud then took the ONLYOFFICE code base and then created a fork of that called, it's called EuroOffice. And, ... there's a lot of, There's a lot of interaction there that's quite, quite interesting. I, I'm- Jonathan: Entertaining for outsiders. Michael: Yeah, absolutely. There's a popcorn thing going on there. Yes. W- but, from our side, in terms of positive things we can say about, Collabora Office, and I'll just say a whole load of positive things. We have zero Russian comments in our source code. We have clean- ... licensing provenance. We know what our license is and- Absolutely ... have statements from everyone that, that contributed to it. Jonathan: That's Michael: such a huge deal. Jonathan: And we build around our... So- We've had a... I'm sorry to bre- to cut you off, but we've- Michael: No, go Jonathan: for it, yeah we've talked to several projects over the years, and it's like what license are you?" And it's complicated." One project in particular, portions of it were written by US government employees, therefore those portions are in the public domain. And portions of it were written by contractors, and no license was ever assigned to it, which technically means that those are, all rights reserved. And w- and we had the guys on the show, and they're like, "It's open source. We want everybody to use it." And we're like, "How does this work?" And they were "Yeah, it's hard." Oh my good- Yeah. So that is a, that is actually a really big deal that you have clean- Definitely ... provenance on your licensing. Michael: Yeah, completely. And I helped set up the structures there at the Document Foundation, and people would send license statements. Ultimately I think we use the Mozilla Public License, so the MPL 2.0- ... which has a number of advantages and disadvantages. So it's a weak-ish copyleft, right? So it's per module copyleft, and so we can be pretty sure that if you're contributing stuff or reusing that code, then you're accepting the, the MPL 2.0. But beyond that it allows you to ship binaries under whacked out licenses. As long as you don't harm people's rights in the so- source code you can sh- use IBM's proprietary license and ship the binaries under that. So that's really nice. We got a clear source binary split there for the for the code, and it works really quite well, I think. And we share that with LibreOffice and, ... yeah. Yeah. So there was a lot of license collecting and yeah, due diligence and so on there that's really very good, both at LibreOffice and Collabora Office, I think. Jonathan: So Collabora Office, and I'm trying to remember the exact details of this. Obviously your, even your T-shirt that you're wearing says Collabora Online. And I- It is ... I'm trying to remember which direction this went. You guys turned the office suite into a web-based version, and that was your- That's right was that your primary offering for a long time? Michael: That's our primary offering. Yeah. Okay. Has been our primary offering for 10 years now. So we're just about to celebrating the 10th anniversary, and we actually got our release on Thursday of the 26, so four enterprise release of that. So there's a whole load of excitement around that. Yeah. And the team are crazily, fixing and improving and documenting and writing press releases and so on. And I think- Yeah, absolutely ... you've seen some of those. So yeah, so we're super excited about that. And, but of course- We've done a lot of work on the user experience. It's much nicer to use, much more familiar. Lots of research and, study there to, to improve it. And so we thought why don't we bring that to the desktop as well?" And actually, that had already been there, so if you'd had an Android version, you have this responsive UI and it looks like a mobile phone, one-hand touch interface. Great. But were you to plug a big screen in, suddenly it would turn into the, the, the real Office Suite, full fat. Absolutely. And so that's why you could run it on a Chrome OS and you would get real document editing there. And so that was pretty cool. And so we sort, took that and then just brought it to Windows, Mac, and Linux natively, and that's Collabora Office. So yeah. Jonathan: What, what was the decision like back in the day to say, "We're gonna make a web version of this"? And what was the engineering challenge there? Michael: Yeah, so well, the engineering challenge is making it work in the browser is, was quite exciting at the time. Sure. There were a whole load of choices we had to make 10, 10 plus years ago. Jonathan: It was before, WebAssembly came along, made a lot of things easier. Michael: Oh, totally. Totally. Totally. So we still don't use WebAssembly, so still we run a lot of the code on the server side. And there are a lot of reasons for that. I think from a security perspective- ... you wanna enforce policy on the server side. Jonathan: Absolutely. Michael: So if you want a policy like you can't download it, or you can't print it, if the first thing you do is download the whole file to the client- ... and then have your office suite there, You have a problem, right? Yes. From a security perspective. Yes, Jonathan: absolutely. Michael: And of course, if the first thing you do is download eight million lines of JavaScript and then JIT them on your mobile phone, you also have a problem that not only did you lose the document it just, it's ruinously expensive in terms of memory, performance, startup- Absolutely ... user experience and so on. So we try and keep that, that fast rendering in the i- in the server there, and bring it as tiles to the client, and that's really good. And we're starting... We- we in- increasingly moving more and more stuff to the client. So of course, your cursor blinks and there's no network traffic, this is rendered- ... i- in the client. Your selections are overlays rendered on the client. Your objects as you move them have previews. Your dialogues are reconstr- reconstructed on the client using native Java, JavaScript, HTML widgets for accessibility reasons if nothing else, so that you can move through them and screen read and get that rich rich accessibility experience. And now we're gonna be moving what we call turbo layout to the client as well. So there's a whole load of work to move away from what we thought would be super easy, pixel rendering to vector rendering in the in the browser. And actually, it works extremely well. But we think we can do more with vector rendering. We can just render once independent of view zoom, and then have everyone use that. So there's some, there's efficiency wins. But then we can start to do what we call turbo layout. So if you look at how Collabora Office lays out text text layout seems simple. You just nail the next glyph along. What could be wrong, right? But it turns out there's squishing text and justification and hyphenation and balancing columns and footnotes and re- laying out pages, and it's very easy to end up in loops. So there's no space there's... We've got a footnote here. We've fitted the text in, but we can't fit the footnote in as well. So let's get rid of the footnote. Now we can fit more text in. But, and you can end up in these iterative situations where, yeah you just... Anyway, you fail. The grass- So we have- The grass Jonathan: is greener perpetually- ... on the other side of that decision. Michael: Yeah, absolutely. That, that sort of thing. So yeah, you just basically can't win, and at some point you have to give up and try and do your best you can. One of the things that means is that's can be quite a slow process in a large document, so we have a turbo layout where you type and it nails it on the screen as quickly as possible. Like i- in, in the most plausible way, and then it will slightly update it. In many cases, you never notice that or it doesn't really need updating. But we're starting now to look at bringing that into the browser itself so the keystroke latency is exactly nothing. It, a frame, maybe two frames. Then again, at 60 hertz, within two frames you can get, some, some pretty impressive round trik- trips. Frankfurt to, to London is easily, is within one or two frame delays anyway. So yeah, you you don't really have a big problem there. Jonathan: Yeah, interesting. What was the what was the impetus behind back in the day saying "We, we need a web version of this"? Michael: I think collaboration is the key thing. So being able to collaboratively ed- edit documents together is just so useful. And I guess, so not only collaboration, but also deploying this. Ease of deployment. So one of the things we've wrestled with for years is document standards, and, we, we do... A huge amount of our work is making Microsoft documents import and export and round trip beautifully. And actually, we've done tons of work in 2604 for that just to make that better. But turns out URLs are better standardized than Microsoft documents. And the spec you can print out and it doesn't break your back if you try and lift it, you don't need a, a wheelie thing. And so sending a URL to a friend and going, "Hey, what do you think of my document?" A- and then being able to live co-edit that, comment on it, and interact together is just so much richer and so much easier from a standardization perspective. And that then allows us to use open document format and keep your actual raw data in a really, future-proof consensus industry format rather than a, a documented dump of the Microsoft internals. So that's a good thing too. Jonathan: Yeah. So one of the, one of the things that we've talked about, and we've talked with for example, Simon Phipps about this, he's done a lot of work with this over the years- Sure ... is trying to convince particularly European governments not to mandate the use of Microsoft Office document format types. And it seems like some of that is finally taken in the form of what they g- what they're now calling digital sovereignty. Michael: Yeah. Jonathan: And that's obviously, it's something- ... that you guys have to be, like, tracking along with. And I have several questions about this, but first just what... W- where are you in this process and what do you see going on? Michael: Yeah. So I think we're at the, really at the center of the, the drive for digital sovereignty. I'm periodically more or less encouraged about where the open source world is going. At the moment, I'm super encouraged. I don't know. Some of the largest things in the news. W- when you see yourself being talked about in The Economist or whatever, it ma- it's really helpful. So for example- It's nice. It feels pretty good, yeah ... the International Criminal Court, we're helping these guys with, with Collabora to move away from some of their things that are happening. Geopolitical disaster areas- ... we, let's not go too deeply into them, but geo- geopolitical risk is back with a vengeance. And if you're a government, you've gotta continue governing. You've gotta serve your citizens e- even in, the, the h- these high-risk scenarios. And I think the balance of risk is now turning. So previously it was no one got fired for buying, IBM or whatever, right? And so people choose the low-risk option. But I think increasingly- There is a lot of correlated risk there. And so actually, the low-risk option is to make sure you have a plan B, a, a minimum of plan B, of some people so you can reset your hardware, reboot, and you're now able to collaborate on a, say, a Linux base with a free software office productivity suite, and support for that. Jonathan: Are we, Michael: are we to- and better ... are we Jonathan: to the point that we could say that no one gets fired for open source? Michael: I hope so. Yeah. Like- ... it's the obvious choice, right? And s- so I think the, the balance of risk is really moving. And actually, there's some really large groups. So Caisse des Dépôts of Fra- France is a big shareholder. The French government shares. They're saying, "Just give them the effing purchase order. Get... Do it. Let's not talk about it. We're ready." Yeah. The software's there. It's just a matter of political will. And I think if you look at, say, Schleswig-Holstein in, in, in northern Germany there so Dirk is the the leader of this migration You start to see what actually we all knew all along. There's nothing wrong with our software. The software is great. What's missing is the leadership to say we go this way, and maybe if you have a problem, we fix it." That's fine, but- Sure ... tell me about the things and we, with a goal for fixing them, not, as a way of making excuses to do not much. And and so it's just fantastic to see, and I think it's the CDU government there, so coming from the right, are saying, "This is the way to go." And I've been in a lot of more left, let's say, political drives for open source, which we talk about solidarity and working together, and that's fantastic. I love all of those things. But it's refreshing to hear someone quoting Schumpeter and Hayek and saying open source is the way to go on the other spectrum, and once you see... if we can get the whole political spectrum to realize that this is where we need to go then that's just deeply encouraging to me about that alone. The real Jonathan: danger, and I speak to this from a very American political viewpoint. But the real- Sure ... danger is that the left and the right realize that they agree on it b- ... and lose their minds and immediately go no, we have to differentiate ourselves from them." It's like, no. Oh, Michael: dear. So yeah, it's just really encouraging to see. I think 40,000 users in Schleswig-Holstein. Tens of thousands of them already migrated. The Austrian Armed Forces, a conservative type of institution. 16,000 users off Microsoft Office, and again, supported by Collabora and backed by, sponsoring development work there and buying licenses as well, so- ... one of the things from an economic perspective is to get that subscription revenue that effectively maintains and funds and improves the software for everyone. Jonathan: Wait. ' Michael: Cause we, we, we- Jonathan: You make money from open source? I didn't think that- ... was possible. Michael: Yeah. We like to say in our community, there is no money fairy. You know- ... we don't have some insanely wealthy banker or, backer. Like to be fair, we do have some insanely wealthy people who who out of the goodness of their hearts, invest in features and do things, but broadly, there's no one bankrolling it. It's organic. There's no VCs. It's a real company, Collabora and yeah we... I think my grandad said, "You pay as you go, and if you can't pay, you don't go." It's pretty simple. In a way. And so that's basically where we're at. So we try and, yeah, do this in a sustainable way for the long term. We, this is a multi-round battle we've been in for 25, 30 years. Yeah. And Jonathan: yeah. We can get back to digital sovereignty in a second, but I think it's- Sure ... just super interesting. What i- what is someone buying when they buy a Collabora l- online license? Michael: Sure. A lot. So- So we talked about security. I, there's a whole load more to say on what we do there. But- ... suffice to say that we have, some of the absolute experts in the industry looking at modeling and receiving threat things and generating fixes and improvements, often that are only in one level of our multilayered onion, but are still, a problem. So we ship updates, so we have signed updates and maintenance. We support the product for three, three-plus years. You can choose which of their annual releases you b- base off, and then you can deploy that with complete confidence that we'll be there and backing it and supporting it. And one of the most fun things I like is product management input. You can come into our conference call days, and you can talk to the engineers doing it. You can give your feedback. We rank that and show you what we're doing and you can get a real feel for how that's going. We also do consultancy, and on our product, we would expect you to buy the product first. So if you wanna drive a particular feature, then, that's fine. But you get a subscription first, and then, go wild and we'll work crazily together. What else? Bug filing. If you have a bug or an issue, a- any kind of thing, there's an, Eat as much as you can, support entitlement there. So if you have a problem, regression of any kind we'll fix that and then turn it around get binaries for you, and so on. So it's a whole load of different pieces intersecting there, a bit of a, a lasagna of of value for- ... for people. Jonathan: So if someone has a license and they run into a problem, can they actually call a phone number and talk to a real human about it? Michael: So I don't know that we like phone numbers as much, because we're quite distributed. But they can certainly email someone, and the developer will read that for sure and get back to them remarkably quickly. And- in some Jonathan: ways email is better, but Michael: Yeah, with logs and things, if you- ... if you've got a, if you've got a complicated problem. But, then we often get on a video conference like this. If it's a difficult problem, we'll just, video conference someone, they'll talk to you. Yeah, the world's expert in, their problem. Is it, I don't know, pivot table calculated field support or something? Or is it a- Yeah ... a, a sockets weirdness with some SSL variants, or TLS cock-up? And then we can help debug and, solve installation problems set-up problems, and occasionally even level three code problems too. Jonathan: Some of my, my... So I've done IT, one of my other hats that I wear, I've done IT support for years and years. Some of my most depressing moments is to find a bug on a customer's computer, go searching for the bug online, and find a thread on the Microsoft support forums where, th- it's one person seven years ago reported it. 50 other people are like, me too." And, you have a, "I'm a Microsoft certified support technician, have you tried turning it off and back on again?" And that's about all the help that you ever get, and it's like it's 2026 now, and this dang problem still exists. Thanks, Microsoft. Michael: Yeah. Yeah. Jonathan: So- Michael: It's really sad, isn't it? I think one of the things, and there's some quite nice an- a- analysis of that, but hopefully we build a culture in open source of excellence. That's the hope. Of getting down to the root causes of problems and fixing not only those, but the whole class of problems and all of the others that could be related and doing a good job. And of course that's time and and culture and is expensive, but in the end, my hope is it yields a better product. In their defense, Microsoft have so, such large deployments of paying customers that actually hardware failure does start to become a real thing. Jonathan: Yes. Michael: Yeah you want a lot of reports before you, You go, "Yeah, that, that could just be, cosmic rays." And I, actually, when you look at our, some of our analytics, there are crash reports there that you just... it's just not possible. Jonathan: It's go- it's gotta be bad RAM in some cases. Michael: Yeah. Either bad RAM or- Jonathan: And Michael: I- ... basic error in electron migration. We worked the CPU too hard, and it, Yeah. Jonathan: So in, in... I f- I could think of a few cases where I thought it was a bug in somebody's product, and I'm like, "You need to talk to..." One of them in, in fact, was Intuit with QuickBooks, and they would do one thing in particular and it would crash it. I'm like, "That has gotta be an Intuit bug. You need to talk to them." And the customer did it, called Intuit up, and it's like, "You need to replace your RAM." And they were right. That is exactly what it was. That's really interesting. And I was so s- I was so surprised, but yeah, it's, it is a real thing. Hardware failures sometimes look like software bugs- Yeah surprisingly. Michael: Yeah. We used to... I used to design hardware systems actually so we did a whole other thing. So one of the fu- fun things is DRAM refresh, and one of my colleagues created this little bit of embedded microprocessing, or like a, a bespoke microprocessor, an FPGA, to do a piece of software. And it worked fine until he increased the length of the program, and now it was two lines of RAM. And previously, just executing the program refreshed the RAM. Jonathan: And Michael: then subsequently, when it got longer, it stopped. And so you're looking at, he just forgot some RAS CAS cycles to to refresh his RAM all as well, but yeah, it just a little- That's- Jonathan: it's a little bit amusing. Yeah, that's hilarious. All right back to the digital sovereignty thing this, this- ... we have a question from the chat room that really ties in well with it, and it is "Tell us a little bit about the headache it is to support Microsoft's formats." Specifically, here's the question, "Does Microsoft intentionally twiddle things just to ruin your day?" Michael: Look I was involved in OpenXML standardization- Jonathan: And has your lawyer told you're not allowed to answer this question? Michael: So no, my lawyer hasn't told me that. But he probably should tell me to say nice things. But actually I'm gonna say nice things anyway out of the goodness of my heart. So I was involved in the OpenXML standardization- ... of the Microsoft formats back in the day, and it would be remiss of me... we complain about the size of the spec, but some of that is awful. We asked for more and more information, and they provided it. They did a straight-up job. And, whilst there are whole fields that aren't specified, like layout the layout of your document is based on an integer, which is the version of Word, right? There's one number in there. Change that, document looks different. Everything changes, yeah. Quite a lot changes. And the more complex the document, the more will change. And so yeah, there's a whole load of stuff there. But that's fair enough, and that's just software engineering, they wanna be able to improve their layout. And why not? And I must say that the people I dealt with there, Brian Jones, Tristan Sean Villaron, some, were great guys. Really friendly, really helpful. They did a, a great job of standardizing that. There are some things that are unfortunate. I think if I, if we look back in the day, I think the Visio file format, the binary file format, had an undocumented checksum of the whole file. And unless you could generate the right checksum, it would go, "This file's invalid," and that's fine. You can see there's always a reason for these things, right? Oh, Jonathan: Yeah. Michael: But I think this is gone now. I think the, the joy of the OpenXML or the XML formats, of course, then, is that they're much, much more open. And so obviously Open Document Format pushed them in that direction, but it's a good direction to go in. They've gone there. The, the slight sadness of the OpenXML story was that- we really were dumped all of this, this XML file format could not be changed. We could change the documentation about it, and we did a great job of that, so you could better document what was going on. But you couldn't go, "Wait a minute, this XML tag sucks a bit. Why don't we move that into an attribute? Ooh, there's a tree structure here that's represented with integer lists for parent nodes. What's going on?" "And we c- can we improve that and just, hey, we've got an XML, it's a tree structure, why don't we use that?" So none of that was possible. But even lots more was out there. So I, I think Microsoft is... plays quite carefully in many ways in the market and that's, appreciated and yeah I I have my concerns. There's a Competition Markets Authority investigation in the UK that we're we're involved with. But I think m- much of that's at the business level rather than the technical level. If that makes sense. Jonathan: Are we at the point now to where you can open up a Microsoft document in Collabora and it basically just works and looks right? Michael: Sure. Completely. Yeah. I do that all the time. Jonathan: Completely? I, I- That's quite a statement, actually. Michael: You say completely. There's always- Yeah. There's never 100% in what we're doing, right? I think encouragingly, sometimes we find bugs on the other side or we're like, "W- what is... we're broken. Let's fi- fix our problem." And they're like no, don't fix that. That's a bug in our side." We need to fix it over here rather than you learning to adapt to this, this bug in our side." Software has problems, and that's why you need support, and that's worth getting. But yeah. It's really extremely good. I routinely redline DOCX contracts with lawyers and sign them, a- as part of my day-to-day job. Sure. There aren't problems that I see. We do... We run the whole company on it, so our spreadsheets, our finance planning. We still get problems with spreadsheets, but it's fat finger errors and I got the wrong... it's hu- human error. One in 20 cells is supposed to contain an error when you analyze spreadsheets. So that's the problem. ... But yeah, c- compared to that we're acing it, Jonathan: yeah. Yeah. Michael: Yeah. Jonathan: So one of the, one of the other things that has been happening over in Microsoft world for a while is they are retiring SharePoint, or at least the, the online or the, the local premise version of SharePoint, right? Michael: Yeah, absolutely. Absolutely. So those people who cared most about their privacy or the, the confidentiality of their documents would typically run a SharePoint server on premise. And they'd embed in that their Office online viewer. So like Office 365's Microsoft Office, but actually on-premise. And so you could then collaborate around SharePoint behind your layers of VPNs and, anti-tank ditches or whatever. You could really be sure that was staying internal. And that's then being obsoleted by December 31st, that's going. And the good news there is, of course, that... there's two, two bits of good news. One, we have a solution here. And it's a drop-in solution. We use this WAPI-like protocol that's compatible with Microsoft's protocol. And so you can spin up your Kubernetes thing and just drop it in, and that's great. And then you can co-edit and view documents as before. Jonathan: Very cool. Michael: Totally on-premise. The second thing is that it's actually better. So one of... There are a whole load of things you can do in Collabora Online that you just can't do in that Office online editor. Say, for example, putting a chart in a word processing document Can't be done in Microsoft Office online Really? It's not there. It seems incredible. Check it out afterwards. Go and try and insert a chart, right? Now, maybe you can get a picture of a chart or maybe a bitmap off your clipboard if you- if you're particularly lucky. But actually getting a full fat, full feature chart, axes, numbers, data table, whatever, just, it's just not there. Jonathan: Oh. Michael: Which is quite extraordinary 'cause it hasn't been there for five, since its inception. It's not like it's improving. And so this leads people to suspect that the online Office products are kept deliberately low fat, low feature because they're an advert for a Windows Office fat client. Ultimately, that's their role, is to basically put something in this space- ... so that- Jonathan: I have noted, I, I have had customers over the years that use macOS, and they've always noted to me "Microsoft Office on Mac is just terrible." And I just give them this wry smile, and it's Microsoft isn't, th- they're not really a, th- they're probably not too interested in making it a really good experience on macOS, now are they? Similar thing. Michael: Yeah. Yeah. Jonathan: The in- the incentives just don't line up there. Michael: Business incentives are real, obviously. Yeah. Definitely. I think they do they try and do quite a good job o- on Mac. It's, compared with the browser version, the Mac thing is a, a, a thing of beauty and wonder. Jonathan: Sure. Michael: But obviously the real stuff is the, the Windows mainly fat client. And yeah, so that's- ... really quite interesting as a thing. Jonathan: Yeah. Michael: And of course it's different. So Collabora Office then is essentially feature parity with browser and desktop. It's the same thing. And so we're sharing code. It's a shared code base and building up. Jonathan: So does the Collabora Suite have an email client? Michael: No. Okay. One of the things that we try and avoid is boiling the ocean- ... as they call it, and Office is already quite a challenging space to get right. And we actually partner. So if you look at something like OpenDesk, which is the German government's s- digital sovereignty or- organization, or was it Zendesk? It's their product for digital sovereignty. They use a thing called Open Exchange, which is, a very mature product. Actually- ... was also spun out of SUSE in Poland i- in- in the past, I think Rafael Laguna and various others. And that's a really large, successful product that's used and white labeled in lots of hosting providers and so on. And they run Dovecot. They create a thing called Dovecot Pro, maintain that. So email is sorted there. Of course, lots of other... we've got a th- a partner called eGroupware doing that. Obviously, we partner with HCL Software, who have Domino. There, there's a whole load of different... We integrate with vast numbers of partners with different product features. And again, why would we want to compete with that? It's a difficult space. I was there. I helped write the Evolution client under Gnome- ... way back in the day. And it all looks easy until you, you find the iCal quirks and the whatever. Everyone starts. Jonathan: Yeah. One of the- we just- ... one of the real interesting stories over the last few years that I've looked at and seen and thought about is apparently there for a while, Mozilla was thinking about letting go of Thunderbird. And- Sure ... one of the, one of the suggestions was the, the Document Foundation would take it. Michael: Definitely. Yeah, Jonathan: yeah. And then Thunderbird would become part of LibreOffice. And that, that did not come about. And there's just this part of me that thinks maybe Thunderbird would be great as part of Collabora. Michael: So not for lack of trying. We really encouraged those guys to come to the Document Foundation. I think in retrospect, where you see where things have evolved there, I think it was it was a good choice not to come. Although- ... you can never s- see what, what could have happened. No one's ever told what would have happened. Of Jonathan: course. Michael: But, they seem to be doing really well, the Thunderbird guys, and- I'm pleased. I actually, I switched from Evolution to Thunderbird at that point, so I'd actually have some clue of what I was talking about, so I'm a Pine Evolution Thunderbird person. There you go. Who's evolved through that- Jonathan: There you Michael: go ... that axis. And, Yeah. I, I- I'm a secret Jonathan: I've, likewise, I've been thrilled to see that Thunderbird has really found their feet again at Mozilla, and they've- ... they've put that back together and it's becoming a more healthy project again. Michael: Yeah, loving it But yeah, that's we partner with people. So one of the things- Right I really wanna talk about is just, that we love to go to market not by ourselves, but with other people. And I guess that's the open source way at some point. Sure the, these, these smaller pieces you put together. So we need, file sync and share probably. We need someone to do storage for Collabra Online. And we need someone to do authentication, because you would think storing files is easy, and you would think aut- authentication is easy, and then you end up with a multilayer active directory SAML authentication system, blah, blah, blah. Yeah. And you have this object store and a metadata store and, scaling and making that thing work nicely. And so we love to work with, say Wire, that bought Piedra recently. They have a great product there. And Nextcloud, ownCloud, OpenCloud, this whole cluster of things called cloud that came out of that sort of trifurcation of companies there. Yeah. Jonathan: It was unfortunate to see Michael: And- Jonathan: I'm- Yeah ... i, so that was actually something I, that was in my mind to ask you about, is, the, the integration with the a- asterisk cloud providers, and whether Collabra has almost a competitor to that, or whether it makes more sense to run underneath one of those. Michael: It always makes sense to run underneath one of those. So our partners are not best thrilled if we compete with them, for a start, so that's basically... I mean- Sure ... some, some partners haven't got the memo. But generally I, from my perspective, I really don't wanna be competing with our partners. Now, some of them, if we're partnering with a chart rendering provider, then I think when you go into that it's pretty clear that they're gonna do charts, we're gonna do charts. There'll be some evolution of the product. But I think when there's a very clear functional difference, I think it's very helpful to, to, just have your bit, and I think that's a good faith way of partnering people, and we like to honor that. And build confidence in our partners that we're not coming after their business, that their customers stay their customers. We're not gonna go around them, so it's not just technology. It's also just business practice of getting those accounts and building a business for other people. So one of the things we love to do is give margin away to our partners. It sounds crazy but we really pay a, a high margin for the local guy who speaks the language, who owns the customer, hel- helps them, holds their hand, installs it, because we want their business to succeed as well. We really wanna grow a, a flourishing ecosystem of individuals and, mom and pop shops through to giant, multi-billion multinationals working with us to push open source- ... most effectively and get it out Jonathan: there. Yeah. So when someone goes and says I want a Collabra Online install," like how difficult is that to set up? Is this... I- is it, as simple as running Docker, or- Sure ... is there a- Absolutely ... a, Michael: Yeah ... Jonathan: five-hour course you have to go through to be able to get it working? Michael: No, that, no, that, that's it's that easy. And so what typically people do is that they get stuck in with code. So we have a, all of our software is 100% open source code. But we have this thing called Collabora Online Developer Edition, which has just the latest and greatest stuff in it. So if you're integrating, people typically play with that to start with, and yeah, it's just in the Docker repository. You Docker, up and there you are. They, you have that. And if you're lucky, you have a Helm chart for a bigger collection of things that does the storage and so on. And yeah, it's, it... I've heard people say, "It's easy, it just works," I think that's nice. And if you look at our dependencies, some of our competitors require a Postgres database, an NFS file share, or, a Redis, a RabbitMQ message bus, and then they run on top of that. We have none of that. We're basically a, a single Docker image, and yeah, you can s- scale that out sideways with a, a load balancer. So it's pretty easy to do. Jonathan: Inside that Docker image, what does provide things like 'cause obviously you still have the need for some of those services. There's probably a database down there somewhere. There's a file storage down there somewhere. Michael: Nope. Nope. So how- So basically what we talk to is we talk to our storage and our authentication provider. So we we get a URL. So there's some configuration. Obviously there's, there's some YAML somewhere that's gonna say, "You are allowed to talk to these servers and not these other servers." But ultimately we're gonna get a Whoopi URL coming in saying, "Hey, this file is over there on that server." We'll match that against the servers we're allowed, and if we like it, we'll get the file. We'll, we'll- fire up a actually container. So there's a, this highly secure container system there. And in the container there's no shell. There's actually only just some fonts. There used to be time zone information, but pretty much nothing, and your document. So if you break out of that containment in some way, and we're talking 8 million lines of C++, you end up holding only your document. "Yeah, whoo. I escaped. I managed to run some native code." I get only my document, and if I execute any kind of system call that we don't like the look of, we set comp BPF, so any time you try and P trace or kill or anything, you're just terminated straight out. There's no, you stop executing. So that model is really nice. And so then inside that jail when we wanna save, we put that back to the file file sync and share with that, that storage, and then we tear the jail down and that's pretty much it. So we're essentially stateless. Jonathan: Okay. Michael: What is the- Except for the state while you're editing, but- Jonathan: Sure. Sure. What are the file storage backends that you support? Michael: Oh, crikey. Now you're gonna just r- ruin me. So there's a very long list of these. So let's take some more exotic ones. I think I've talked about, HCL, Domino Workplace. There's Nextcloud, there's Zonecloud, there's OpenCloud, there's Pydio now owned by Wire. There's C-File there's eGroupware. But let's take, say, Moodle. So if you like doing a learning management stuff there's Moodle, and there's another one. There's the other big learning, open source learning management system, which now slips my mind. Both of them are using Collabora Online, and so then you can do classroom stuff. You can give people assignments, get them in market, and do it all inside the web there. Very Jonathan: cool. Michael: And because we're multi-tenant, you can actually do that for... The, the same instance can serve both your Moodle and your, Wire and also your, Py, all of these things on the same cluster, which is really nice aggregating that together. Yeah and I've forgotten we have about 230 partners, so I'm gonna get shot by, I don't know how many people afterwards. But I'm terribly sorry, a, a Jonathan: bunch of Michael: them. Jonathan: And so now- Michael: Yeah, a, a real bunch of Jonathan: them. It... But if somebody just wants to throw this on a Raspberry Pi, right? Sure. First off, does it fit? But then secondly, like I'm still trying to understand, like what storage backend do you use if you don't have anything at all, and it's like I just wanna use it on the Pi? Michael: Sure. So I think the Pi would probably provide something for you there, but OpenCloud, ownCloud, Nextcloud, one of those things is quite popular often. And you just grab one of those and then you embed us into that. Okay. And yeah, we do run on the Pi. That used to be a challenge. Actually, so Calabra did some of the original hardware enablement of the original Pi with the- Oh, really? ... the overlay masks- ... and the one-bit alpha, and the trying to make that perform really well. And they did, there were some miracles done I think with the original Pi. But when you look at a modern Pi, whoa, you know- Yeah ... it's this multi-core powerful GPU and a- again I'm hoping open source drivers left and right. So the world has totally changed- ... and that's something that actually can really run quite well now. Jonathan: The, Michael: the Jonathan: first one or two Raspberry Pis were quirky in multiple ways. Michael: Yes. Yeah, absolutely. One of the, one of the things I like particularly was that if you used one of these gaming mice, I actually have one of these, you know, that I... Just to test, with buttons everywhere and, you know- it, it gives you so many events coming out of the thing to try and keep up that, that it would just drown the poor Raspberry Pi's event interrupt handler. So like it would spend 30% of its time just trying to respond to mouse events and send them to people, which, yeah, it's not as cool as it can be. You're probably not what you want. And so yeah, we had to- Jonathan: Yeah. Michael: Yeah. Yeah, we had to dial that back a bit. Jonathan: Yeah. Michael: But yeah, these days of- it's no sweat at all, of course. There's lots of horsepower there. Jonathan: Yeah. Okay. And so you said there's a big release coming up in, like o- on Thursday. Michael: Yeah Jonathan: We're recording this on Tuesday. It goes live on Wednesday. It'll be the day after that. What what are you excited about there? What do you want, what do you want folks to look for in that new release? Michael: Yeah, sure. So one of the big things there is, of course AI. Now, lots of people are either fans or not fans of AI. A controversial minefield. I'm just... Let me just tap dance into it briefly and and we'll see what, what explodes. Obviously it's off by default, so lots of people don't like that. Yeah. Nothing is gonna leak your data anywhere unless you tell it to. And actually it's better than that. So the sidebar then will show you, and it kind of color-code it is this your dangerous off-premise whatever, or you can make it green or red or whatever to try and help the users understand- ... the, the implications of what they're doing. And then of course you can actually start to use the selected text as context, so that's nice. So instead of it being this kind of nailed on the side thing that can see the whole document, it can actually interact with bits of document that work works really nicely. And then there's lots of course, little demos. Of course, if you're writing text documents- Large language models turned out to be reasonably okay at language. And i- it makes a lot of sense for those kind of translation and make the tone more professional or make it a casual style. Th- this kind of thing is, summarizing and conciseness is really... Are really easy. It can do other things too, like looking for fraud in spreadsheets, so if you you know- Yeah ... numbers that are fraudulently added often don't follow the distribution you'd expect of sampled numbers. And so there's a whole load of things you can do with that's cool. Jonathan: Yeah. Michael: And yeah, and you can of course make pretty pictures and embed that. That's very important. Of Jonathan: course. Michael: In your presentations. Jonathan: Yeah. Does, does the c- the new Collaborator AI features, do they have the ability to use locally hosted AI? Michael: Of course. Of course. Yeah. And that, that's su- super easy because ultimately- these are basically a standardized protocol. You dump stuff down a, a text socket and you're done. So yeah, that, that's really nice. And there's also some MCP stuff coming- ... Michael: Which is really helped by this, are we talking about the security and the jails and the onion around that, right? You really don't want the AI playing inside your machine if you can avoid it, in a scary way. So having all of that sort of wrapping around that's really helpful. Whole load of other features obviously in there that, transcend the AI stuff. So better navigation, better search, a pretty UI s- document comparison, so side by side. We didn't do three up, which I think developers like, side by side document comparison, where there are- larger changes, if you've been offline for a long time, maybe when you come back. I was actually using Starlink on the in Mid-Atlantic the other day, and it's quite... Yeah, you can use Collabora Online really nicely over Starlink back to Ireland and, you're in the middle of Greenland looking at icebergs out the window and it's working just nicely, except for the laptop power, which, if you have a beefy laptop, you blow the fuse on the, Jonathan: Yes the Michael: thing. Jonathan: So I don't- I have 100% had this- Ah ... this experience. This is my- Too annoying ... this is my complaint, my one complaint with my Framework 16, is that it's got such a beefy power supply that it immediately, doo, turns off the the airplane breaker. Michael: Yeah, indeed. I'm trying with these, these xscale. So the hope is that if you get a small laptop with a with a big screen, and you put your glasses on- ... the, y- you're really winning. And so then you can get a lower power thing and li- life is good. But unfortunately my eyesight is, I'm like halfway to glory signpost here, and it's really just not very good. And so I needed to cut down some gl- anyway, we'll see. May- maybe that'll work out in the future. I hope so. Jonathan: Yeah, Michael: absolutely. Jonathan: It's funny. Michael: So yeah, lots of other things. Better calculated fields and pivot tables. Lots of interoperability functionality. We actually had a chance to do systematic interoperability work, so we have this huge, over a million test documents now I think. And actually just working through that and doing two kinds of things. So just making sure that stuff is preserved when it's roundtripped, but also just silly stuff like tweaking the order of attributes, so when you diff the XML to see what's different, you get many fewer changes there. And just making sure we don't do silly stuff. The whole load of- paper cuts there that would just cause grief on import that we've just got rid of for better round trip interop, and that's, it's really nice to see the fruit of hundreds of fixes there proactively trying to chase- things that could be problems. Now, of course, the, the test set is huge, and it's all bug documents with the worst documents in the world. We're really pushing the I guess you'd push envelopes, don't you? Something like that, you know, trying to find the the twisty corners. But it's- Right ... yeah, it's great to have a team that you can put on that for a while and make a real difference, so Jonathan: yeah. Yeah. I- ironically, that's something that AI is actually pretty good at, is to help find those weirdnesses in the code and write test cases for them. Michael: Yeah. I suppose so. Yeah, possibly. I, I like it for for the documents. What we're really focused on, though, I think is trying to train our staff because it turns out if you're looking at product for a 10X productivity win the, the key is to get your juniors and, give them the confidence and skills and experience to, to really to up, up level what they're doing there- I think. So that's that's where we're focused on that. Hey, I go to school- local schools and recruit. If you can get the best guy in the year, or best two people in the year- Jonathan: Yeah, Michael: absolutely ... that, that would be a guy and a gal, to to train them up, then we try and do that. Just because finding skills in our 35-year-old co- codebase is just not there. You get people sending you, "Oh, I'm a full stack developer. I know React and JavaScript and some Node." You're like, "Yeah." None of that is what we do. We, we have bespoke toolkits and huge C++ code bases, so we find, yeah, it's better to train people than to to hire, Jonathan: Yeah, interesting Michael: people's preconceptions. Yeah. Jonathan: Interesting. So in, in doing the development and the the maintenance on the code base, where you guys have had such an influx of users, and sort of users that are different than what you've had in the past in some ways at least. Sure ... have there been any really interesting bugs or findings that have shaken out from, like- Suddenly, the, the German minister is using Collabora. Are there any fun stories around that? Michael: Yeah, so surprisingly few which is annoying. And often they, some... So we had one with the Danish Ministry of Digitization, I think. So they're doing a pilot and their biggest problem that they had was that they had loads of styles in their documents, and they had style previews at the top, and they wanted, we were previewing all of the styles. And they wanted to filter this down to only a few. And so there's a feature in Microsoft Office that will effectively hide or not show, not make it so easy to use all of the styles they didn't want you to use, and then see the ones you want. And so this is a big deal for them, so we, we implement that, ship that, they're happy, life is good. And then for free. We're helping pilot these things. If you're looking for really odd bugs, some of the ones often come from automated systems. So some genius out there writes a script that generates OpenXML. Okay. So Microsoft Office generate OpenXML is one thing, but they basically generate this thing and they tweak it until it works in Microsoft Office. And that often involves not outputting any style information creating, they, they basically twist the thing to the nth And see what it will tolerate and generate the most minimal file. And so you then just have to know that if you don't have a font, it's Arial 12 point. And if you don't have a, a this and a that and the other, these are all of the... There are no page margins, there are no whatever. So all of those defaults that aren't actually written down anywhere, you then have to make sure that they match. So that's- Jonathan: It, it actually seems like that would be a really interesting place to set a fuzzer on and may- maybe even like an AI or an AI-guided fuzzer. Like- ... make minimal viable files that Microsoft Office will open, and then see what Collabora Office does when it tries to open them. Michael: Sure, sure. So actually I, we have some loop automation going on there that loads- ... all of these files, exports them to Microsoft formats, and then imports them to check that actually Microsoft is not complaining. So we have a big old loop automation doing that over hundreds and hundreds of thousands of files in the background. Jonathan: And then- Actually- And then you fall into a question, like a, a really important question I think, and that is do you want to be bug for bug compatible with Microsoft Office? Yes. Or at which point it's like Microsoft does this wrong, we're going to fix it. Michael: Yeah. So I think to some degree that depends whether Microsoft wants to be bug for bug compatible with themselves, right? Fair. And that's the sort of expectation. So there's this very famous Lotus 1-2-3 bug which has been inherited from Lotus 1-2-3. I, IBM created it, and it's a missing gap year. They, their leap year algorithm was wrong. And that sounds like... and so I don't know if you know how spreadsheets work, but basically you have this double precision number in a cell- ... Michael: And then you apply a format and it shows up as a date. But, a- and so you'd think we can fix the gap year algorithm. We'll just..." What would we do? We change the number, but the number could be a result of a formula. So then do you walk the formula tree and try and change the root numbers to fix the gap? It's just, it becomes- it becomes fundamentally impossible to fix, ultimately. Or to work out even in a spreadsheet where it is. And even today we support the old and the new way. The real leap years and the old leap years. So there's a setting there, and Microsoft has that. We have that. And, there are sheets that are still using that. And actually one of the things that you find most surprising is that when we look at some of, say, our macro support, so we support VBA macros to some limited extent in your browser running, from a XLSX file. Some of the things in those macros are the Office 95 to Office 97 portability shim. So in Office 97 they radically changed VB because b- that, that was before they'd forked the whole Visual Basic engine and put it in Office. So they were following the VB thing, and they didn't realize that the language would change and then the APIs would change, too. And so there's basically a magic object that is I forget the name of it, but, say VB 95 compat or something. And they rewrote those macros line by line to introduce this compat thing and have all of the old methods on it. And we still see those out there. So we have compatibility for that Office 95. Only some subset of tho- that, that helper class, which is weird that you would still see that in today's code. Jonathan: Yeah, so there is a... So you said that and it brought a memory to mind. MS Works. I, I have... Th- this, this kind of blows my mind. So years and years ago oh my goodness, I don't even want to tell you how many years ago, my family had a computer that did not have Microsoft Word on it, it had Microsoft Works. And we'd save documents in Microsoft Works. And Microsoft Word will not open those documents, but OpenOffice, LibreOffice, and I'm assuming Collabora Office does. Indeed. And it is of endless delight and hilarity to me that Collabora, and of course the rest of them, are more compatible with Microsoft products than Microsoft products are. Yeah. And I think so that's hilarious on one hand, but there's also a, a deeper thought here about Making that information of those documents accessible in the future and even just like the inf- the idea of the information preservation that's going on. Yeah. And so like you think about this, it's 100 years from now, are we going to be able to open any of these documents? And I think part of the conversation that needs to be part of that is like open source allows you to have a chance at preserving- Yeah ... that information. Yeah ... who- Microsoft may not exist in 100 years. ... Obviously Microsoft Word is not going to exist in the same format in 100 years, right? Yeah. And so th- this is not... I think about this with a lot of things, right? Even like the the preponderance of YouTube videos that are out there right now. Like YouTube ev- is eventually going to go away, and are we just going to lose... Like YouTube has become culturally important to the world. Are we just going to lose all of that? And so that's- Sure ... that's obviously, that's outside of the conversation we're having right now. But I think it is very important to think about that just for documents that like- Michael: Yeah ... Jonathan: it's important to think about the preservation for the future. Michael: Yeah, absolutely. And I think the sort of digital archeology almost is something. And there's a couple of guys there. So there's a chap called Friedrich Stroeher and Valek Filippov, who calls himself Frob. And these guys are just passionate about that. That's what they love doing. And they've found a niche there that they can serve in a beautiful way- Yeah ... and created those document liberation filters, and all kudos to them. And there's a project there to liberate documents they call it. But yeah, so we can import those Works files. Now, there's a security question there because the more import surface area you have, the more risk you have. So we tag those with what we call an exotic filter. So probably you're gonna get a pop-up going, "You do know this is a file that last was used, 20 years ago. This is the last file that was written with this, this format," right? And so hopefully that, that defers the, makes it much harder for a zero day in, if- if there is such a thing there. But then again, these guys are really passionate, and they fuzz their codes very heavily. There, there's a whole, there's a whole big fuzzing thing going on there. And actually it's really cool. Let me just tell you about that. So there's a thing called OSS-Fuzz. I don't know if you've seen that. The Google project. Yeah, absolutely. And that's run in part by a U- Oxford based UK company, I think called duh. Nope. I can't remember their name. That's terrible, isn't it? I how rude, let me see if I can look it up. Jonathan: I Michael: know OS- Anyway, Ada- ... OSS- ... AdaLogix. Okay. AdaLogix. Yeah. Google fund it. So Google fund the hardware, but I think A- AdaLogix contributes very heavily there. One of the things that they have is these huge multi-thousand node clusters basically do using evolutionary algorithms to hunt down bad code paths that, that take you to nasty places. And that's extremely effective. Before that started running, we would routinely get what looked like nation state reports on, there is this remarkably well-formed bug report that w- with a sample document that takes you to a problem. And after we got the cluster, that didn't happen. We caught most of those during development- which is the way to do it. One of the interesting things we found from the Mythos stuff, so AdaLogix actually has c- the... So it's a code coverage informed tool. So it's actually watching what code is executed- Absolutely ... and then trying to enlarge the coverage. So let me give you an example. We had a bug in our HTML filter for copy and pasting into spreadsheets. So there's a little custom parser there of tables And there's a bug in it. And so not only did it find the bug by walking the code wider and wider until it, it got it- ... Michael: But then it reversed the process to minimize that down to the smallest piece of HTML that would walk all of those paths and go there. So first it colors it all in red, it finds the problem, and then it colors it all, in gray again until it's just got that. And so normally file formats are binary, but with HTML you can see, like these are the 63 characters that minimally walk down the stack tra- trace to find your crash. Absolutely. And that's just such a beautiful thing. Yeah, it's very useful. And so we really love that. And I guess that's almost automating QA. Or you could argue it's like a, a precursor to AI stuff. Jonathan: Yeah. Michael: But the problem comes when you hit a crypto wall, right? Like you can't evolutionarily work your way through a signing. That, that ain't gonna work, right? And so one of the wonderful things that Ada Logics is doing is then applying Mythos to audit the code past that. Yeah. The bits of code they couldn't get to with their fuzzing. And we're getting quite a number of interesting more interesting reports from them which is great. Jonathan: Have you- ... have you found that they're fairly high-quality reports? Michael: Yeah, definitely. I Jonathan: mean- So that's something that people have found. Like historically over the last, let's say six months, it's gotten a lot better. But, Yeah ... you would ask AI to find you a vulnerability, and AI has n- this is one of the problems with it, still is one of the problems with it, honestly. AI has no capability of saying no or I can't do that. And so it- ... will just, historically it has just hallucinated a vulnerability for you. And then- Yeah ... someone that doesn't know what's going on will report that to a project, Michael: In, and in great detail with this, this mass of text that then someone competent has to read, which is the people you wanna protect. Look, all tools, all new tools have this problem. So if you- Sure ... if you look at the sort of what we've done with security, we did like W error. So cr- s- stop compiling when there's a warning. And of course there are a whole load of warnings that are, you need to instrument your source code to turn these off, because actually- Right they're not really errors, but at least there's a manual process of doing that. And then you add more and more warnings, all of which fail, and you repeat this process and eventually your W error clean at the highest level the compiler can do. And then you put, maybe lint tools and you put Coverity Scan on it. And, Coverity Scan comes up with loads of, you know- False Jonathan: positives ... Michael: yeah, there's a drop through into a case statement, and of course that was intended that you did this case and then the default after it. And it's, but you then put an annotation in saying, "I meant to drop through here." I know that it drops through. "Please don't warn me." Yes. Jonathan: Yeah, Michael: absolutely. And, and so at least half of the, the warnings coming out there are false positives, maybe more in, ace human-made static checking tools. So we, you expect that. And there's always a peak and a trough. Every new tool comes with a new set of problems, and then they fade away. I actually have a OWASP talk when I show the graph of all the generations of these these checkers coming in and the impact that has. Jonathan: But Michael: yeah. Yeah ... so yeah, I think doing good stuff interesting security pieces and yeah, just great to have that OSS Fuzz back up that. Jonathan: Yeah, absolutely. I've thought for the longest time that it would be super in- and it sounds like this is what they're doing, but it would be really interesting to have a fuzzer that is guided by AI it wants to look in- So as Michael: is, lib- libFuzz is like that. Yeah. So it's, I- as I understand it, it's looking at the c- the code that's executed. It's then i- inferring how to perturb the input to- ... increase code coverage, and it's watching the code execute as the data flows through it, and then perturbing the data, and it's doing that incredibly quickly. Yeah. So it forks and checkpoints and comes back, and it's... yeah. It looks like magic. It looks like intelligence, but there's not a, there's not an AI in it, it's a- ... it's a, no. So it's really cool. Jonathan: Yeah. Super cool. Michael: I love that. Jonathan: One of, one of the note I've got here, we're running out of time, but one- Yeah ... other note I've got here is that you you also have, in all of your copious spare time, you have a comic strip. How did that come about? Michael: I do. I do. Oh I've been, in the open source industry now for a very long time, and there's a lot of funny situations and scenarios and problems that people come across. And as one gets much older and, it, it might be useful to to pass on some of the the fun stories and- in a friendly way. So explaining what open source is we do that with, Oh, actually, I've got... what do I have? If I reach over here. Yes, pictures of beavers. If you've got the camera on, you can see this. Yeah. Building roads instead of building software. I think one of the tragedies as a software engineer is that we make beautiful things. We make castles in the air poetry that you can't see. There's a whole aesthetic around taste and structure and architecture that were at a cathedral, people would want to walk into it and go, "Wow, that's really good," or, "Oh, that's a bit dingy." Like- ... why didn't we paint, repaint the basement?" And it's just a shame because software engineers can communicate that to each other and see it. And so how do you articulate and share that? Anyway, we decided to do road-building as a more concrete way of seeing a- actual- ... talking about construction and- right luckily I studied a bit of civil engineering as well, so there's some- Oh, okay ... there's a whole lot of sort of overlaps there. Jonathan: Pull- Michael: pulling from- And show me the road instead of show me the code. Ah. It's, Jonathan: Show me the road ... yeah. I gotcha. I gotcha. So speaking of these sort of in-jokes, and this crosses over to bugs as well, one of my favorite stories about a bug, and you may have been involved with this, was the can't print on Tuesdays bug. Michael: Oh. Oh yeah. I think Bjorn fixed that. Yeah. I don't remember all the details, but yeah, some horror in the, the generated, output that. Yeah Jonathan: It, when you went to print, it generated a file, and because of the way the timestamp and the date stamp lined up when the computer then looked at it to print it, it interpreted it- Yeah as a different kind of file. Michael: Yeah. It's really not good, isn't it? And you... Yes, you can't make this stuff up, but it, you ... And actually it's quite interesting. You think Linux is by now a very polished desktop. It looks beautiful, and, i- it beats many other things. My... Just recent files is such a useful thing in your file picker. That's the hot place everyone goes, right? And you just still not... I don't think you see that in Windows still, which is just nuts. It's such an obvious and simple innovation, right? It's a beautiful desktop. Underneath it, though, when you start to print things, there really is a shell that's forking LPR and piping stuff to it still. And that's that's all fun. And so yeah if the file doesn't look like the file they think it's gonna be, then yeah- ... that can cause some problems. And yeah. And then of course it goes to CUPS at the back end, and there's some kind of reason- reasonable, IPP thing going on hopefully. But, Jonathan: yeah. So it it thought it was an Erlang JAM file. I had to go and found it, and went through and- ... and pipe- clicked through it. Yeah, this was what? Back in Ubuntu 9.10. Almost 20 years ago, this was a thing. Yeah. It's still, it's my... It is absolutely my favorite bug story because it, it well captures the seeming inanity of some bugs. And you're like, "How in the... There's no way. There's no way that could be a bug. There's no way it could work like this." And then you step through the details like, "Oh, okay. Yeah, I see it. I understand now how, this, this random, seemingly random collection of of things came together," and you can't print on Tuesdays. Michael: Yeah. And sometimes you need an expert to help you find those things and fix it properly. And that's yeah, so we love to be that- Jonathan: Yeah ... for people. Absolutely. Have you guys started using AI to try to find and fix those sorts of things? Is, are you, are Michael: you- you know- ... on the AI Jonathan: programmer bandwagon? Michael: As I say, I think by far the most effective thing we can do to improve productivity is just train and help our young people get involved with the code base. So that I think is, that's the 10X. And I'm always intrigued to hear of these stories of huge productivity wins. And so actually we do a whole lot of training along those lines. So we do tee-time trainings, and videos, and we look at the code structure, and we try and- try and teach people that a- as we go along. But yeah. So come and get involved, and watch a tee-time training and get stuck into the code. Jonathan: If somebody wants to if somebody wants to come and actually hack on the Collabora source code, where's a- Yeah ... where's a good place to start? Michael: Yeah, so if you go to collaboraoffice.org then there's whole sort of pictures of beaver mascots that'll make roads and all sorts of nice instructions to help people, Get involved. And of course, there's so many different platforms now. So there's Collabora Online on Linux, and then there's iOS, and Android, and Mac, and Windows and Linux. And there's instructions for each of those, and simple setup steps. And then we have a Matrix channel, so you can just jump in there and say hi, and get, get stuck in. Or come to our... we have weekly meetings, so just come to the meeting if you got stuck. And there's a video chat there with a whole load of engineers and mentors. People love to help you get going, really. Jonathan: Oh, very cool. Yeah. Awesome. Okay I've gotta ask you a couple final questions before I let you go. Sure. I do get emails if I forget this. You personally, what is your favorite text editor and scripting language? Michael: Oh, dear. So I get... I use the editor for middle-aged computer scientists, Emacs. Jonathan: Okay. Michael: It's probably a mistake. I should probably enter this advanced modern world of, automated everythings. But yeah for me that's where the key bindings feel like home. Sure. And the nervous tic is to bookmark where I'm at, so I just do Control-X, RB, one, every, too often. And then you say... what was the other thing? Client choice? ... scripting, Jonathan: scripting language. Michael: Scripting language. Grief. So because I'm extremely old, if I need to do text processing, I would use Perl, just because I understand the stupid regular expression syntax- ... and whatever in it. I- if it's gonna be s- small and quick. If it's bigger than that, I go for Python, just because my colleagues tell me I'm old and crusty, and they can't understand my Perl scripts. Which is fair enough. I'm completely with that. So yeah Python, I think, is is probably the right answer. Jonathan: I shall have to ping Randall Schwartz and let him know that we got someone that said Emacs, and their first answer was Perl. He will have a little celebration for that. Michael: Excellent. Excellent. Jonathan: Michael, thank you so much for being here. It has been a blast to talk about Collabora and all the various things going on there. Thank you so much. Michael: Absolute pleasure. Thank you. Jonathan: All right. That was Michael Meeks of Collabora. And we've got some fun stuff coming up. Next week we're talking smoked meat, and that is actually about CI security and not losing your all of your secrets to a malicious pull request. And then after that, we have Jonathan Polant of Ferrous Systems. We're gonna talk Rust yet again. And then I know we have some things working in the pipeline, but we've got some openings after that. You can drop us an email, floss@hackaday.com, and we will get you on the schedule if you've got an open source project to talk about. Other than that, we just wanna say thank you to everyone that is here. We sure appreciate it. Whether you get us live or on the download, whether you listen or you watch we wanna say thank you. And we'll be back next week on Floss Weekly.
• 
  Episode 876 - There Is No Money Fairy 22.07.2026 1h 12m

  This week Jonathan chats with Michael Meeks about Collabora! What's the origin story in this consulting company, why do they have an outstanding office suite, and where is the world headed to accomplish digital sovereignty? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 875 transcript 15.07.2026

  FLOSS-875 Jonathan: This week we're talking with Neriman Gelva about Pewter. That is the operating system that runs literally in your browser. Maybe we're stretching the definition of operating system there just a little bit, but still. This is Floss Weekly, episode 875, recorded Tuesday, July the 14th. JavaScript as a systems language. It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we are talking about Pewter. That is the operating system, the desktop that lives on the cloud, on the internet, in JavaScript. I have questions about how this works. It's something like that. Can you locally host it? I think so, but I don't know for sure. I know that it's open source. That's why we have them on the show. And the guy behind it, the man himself, is Neriman Jelva. I met him at the Ubuntu Summit this year, and he did a couple of talks there. Actually, they had him do a lightning talk and a full-length talk all about the what you do when your open source project is a runaway hit. And I know a lot of people would say, "Oh, I wish I had that problem." It's actually a big deal. It's a tiring problem. It's a, there, there are things to deal with that maybe you don't think about. But Neriman is a programmer. He is an entrepreneur, and he's been doing this sort of stuff for about 30 years, so he knows what's going on and how it all works. But I think Pewter is a new ballgame even for him. It exploded in popularity. We even covered it. We talked about it over on the Untitled Linux Show as well as this cool thing that you may wanna check out. A lot of people thought it was cool. There's about 42,000 GitHub stars. I don't think he paid for any of those. 390 people have contributed. They've got over 400,000 downloads. And apparently, it's in 100,000 different web applications, which sort of talks to the the actual use case for Pewter. I could go on. I've played with it a little bit, but we do have Neriman here. And let's go ahead and bring him on, and we'll get the answers from him to all of these questions. So welcome to the show, sir. I'm so glad to have you here. Nariman: Thank you so much. Very excited to be here. Jonathan: Yeah, we were talking beforehand about... we were comparing notes about our travel schedule, and I was telling him about things that I've got coming up. But we met at the Ubuntu Summit in London. Are you based out of out of London? I'm in the middle of nowhere in the United States. I'm in southwest Oklahoma. I'm in flyover country. I wave at all the planes going by 'cause I know they got my buddies on them. Where are you based out of? Nariman: I'm in Vancouver, BC, Canada right now. Jonathan: Ah, our northern brethren, the Canadians. Nariman: Yes. Yes. And, but our team is from all over the place, basically. We have a couple people in the US. We have some people in Europe, Asia. Pretty much remote, like a true open source project. Even though we're a company, but... Jonathan: Hey it's allowed to be both. Open source developers- Yes ... need to pay rent and eat too it's okay to make money on these things. Okay, so let's start from the beginning. You- you've got a team now. You've got a business. You've got an open source project. That's not where any of this started, though. What was the very beginning? At what point did you go- A desktop on the internet would be cool. How did that come about? Nariman: Yeah, sure. So if you remember maybe you never had to pay attention to this part of the internet, but I would say between 10 and five years ago there was this period of time where everybody was proclaiming that the browser is the next operating system. It was like something people would say, and I really believe that because I remember the first days of the web. That's when I started writing my first line of code, and then, So I basically grew up with the web, and I saw the progression over time. So I really do believe that the browser is the next operating system, depending on how you define the operating system. So n- obviously not a Linux. That's not what we're talking about. Basically m- more about the user space. That's the discussion. We can talk about this more later on. But the real thing a- around this, the real thought around this was always that, hey, this is like... The browser is like the v- a virtual machine, that you can probably run any application in it. And more and more applications were moving to the web. But nobody really cracked the general purpose environment where you can just have any type of application, have an IPC, a desktop environment, and those kind of things. It's been tried many times before. In fact, I think the first try goes back to the late '90s. Imagine those browsers. Somebody tried to build a desktop environment. I think it m- it may have been literally called WebOS or something like that. So this has been tried many times in the past, where you have this operating system-looking or a desktop environment in the browser connected to the cloud. So there's been many attempts before, and I really wanted to take a stab at this, and that's why I was always passionate about it. And around five years ago peak COVID time, I was I was pretty disillusioned with building business sort of applications and that kind of industry, that kind of subsector of of the software industry. So I was just visiting my sister, and then I was on vacation, and then I just decided to do this. I was like, "I'm going to go after this." So started out as a hobby project, and then it took off later on. But the gist of it is people always wanted to do this. There's been many attempts at it, and then I think a lot of things came together a few years ago that made it actually possible. Jonathan: B- I jumped ahead by a question, and you answered the one that we need to ask. But let's make sure that we've gone over it. What is Pewter? What's the... what's the... is there a problem that it solves, or is it just a... s- start with this. Tell, tell us what exactly it is. What would you describe it as, someone that doesn't know, that's never touched it? Nariman: Yeah. So I would say Pewter is what we call the open source internet computer or internet operating system. If you go to Pewter... let's put the open source stuff aside for one second just- Sure just to explain how this thing works. If you go to pewter.com and create an account, you're basically given a very small cloud computer or a computer in the cloud- ... that you can use personally. Now- You might think, "Oh, we're giving you a server, or we're giving you a container." It's none of that. It's something that we built from scratch. It's a distributed file system with authentication database, everything attached to it. Anyway, you get this virtual computer, a cloud computer in the cloud that you can use for basically any purpose. You can put files in it. You can run applications on it. A lot of people use it to just r- use our office softwares. Just like a regular computer, it has very general purpose applications. Now, why would you want something like this? That, that goes back to the fact that, hey, operating systems in general I'm not talking about the kernel, but the user side of things, haven't really evolved in a very long time. We've been always device-bound, and anything with the cloud was always tacked onto the operating system. For example, operating systems don't support a cloud file system. It's always been, hey, I do something on my computer, and then it sync to the cloud. Or I'm using some website that does this for me. Or authentication is never universal. Maybe with Apple you have a little bit of that, with Google, but it's pretty fragment fragmented. So when you go into different applications, it's usually I create an account with this app, with that app. So the whole being sorry for the buzzwords, but cloud native or AI native thing- it, it never it doesn't really happen with the with the regular dominant operating systems. And so Pewter is an attempt at bringing that sort of modern paradigm to the basically desktop environment or Chrome-like interface, Chromebook-like interface. Now I could go on and talk about the use cases and what p- people are using it, but if you have any questions, I can answer those before. Jonathan: We will get into the use cases. I've, I've- Yeah ... gotta ask, though, and this maybe will lead on to that well. I'm going to intentionally undersell Pewter so that you can tell me where I'm wrong. It's just a, it's just a website that writes some JavaScript that looks like a desktop. That should be easy. Why did it take off? Nariman: That, that's a good question. So in the beginning, I think in the very beginning, Let's say four years ago when I put it on Reddit and it really blew up. I think at the time it was mostly because people were just shocked by the amount of effort h- that had gone into it, and the level of detail that was worked on. So ba- and back then, this is before AI. So I would say maybe five years ago. Jonathan: It's Nariman: in the dark times. But back then it was pretty difficult, right? It was really difficult to code like that. I had to sit down and manually code every single interaction. It was pretty difficult. So I think the very first time I put it out, people were just shocked by this, and then over time an application ecosystem formed around it. So we add applications. It started with very simple applications, like a Notepad application, and then a very simple camera application, and then all these APIs formed around it to make these applications better. So people started slowly, gradually using it as f- for file storage. For a file storage where you can edit text files. Believe it or not with something like a Dropbox or a Google Drive, you still can't edit plain text. So if you put the plain text in one of these services, you still can't edit it. You have to download it or install their local applications to do that. Yeah. So it started fill these small niches where you put your files there, and the interface is more advanced than a Dropbox or a Google Drive. You can double-click. You're familiar with this paradigm. I double-click on a text file, I can edit it, and then a photo editor was added so you could edit photos. But then the reason a few years ago it really took off was that we managed to port Visual Studio Code to Pewter. So we were- Interesting. Yeah ... basically the first yeah, the first company or project that managed to port it completely to the web and hook it up to the cloud. Now that's, that was a big hit, and it happened completely organically. But then again about a year ago coding basically died and nobody uses their code editor anymore. So that, that one- I- ... that one went down, but by the- Jonathan: I resent that. Nariman: You tell me, I've been coding since I was seven, so I... my identity is really tied to it. Jonathan: Yeah. Interesting. Nariman: Yeah. So it, it has started with basically a shock value sort of a thing- ... and then it, an ecosystem s- formed around it with applications and then with features. So now you can do a huge number of things with Pewter. It is really full of features. You can drop a directory on the desktop, you can right-click it and publish it as a website. It has zipping, compression. You basically get a lot of the things you get with a regular desktop, but it's happening in the cloud. So going from device to device, it's always there. Nothing changes. Your environment doesn't change. You can go from your- Laptop to your phone to your TV to your Tesla fridge. You, you name it. If you have a browser, you can use it basically. Jonathan: Yeah. So there's a ... To write all these apps, is there like an API that, that sort of resembles like a... Is it almost like a C standard lib or a, POSIX layer that you guys have written for JavaScript? Nariman: Yes and no. I, we were definitely inspired by it, but I think the JavaScript ecosystem in general, we had to move more towards maybe a Node.js-like API, but the concept is pretty much the same. We couldn't make it POSIX compliant. Maybe we'll end up doing it at some point. I did try it a few years ago, but it didn't really work out. We were almost there, but w- it didn't really work out. Now, we have our own basically standard, but it's close to typical APIs from Node.js or those kind of environments, more higher level. Jonathan: Yeah. So th- there's a, there's an obvious thought that I've had, and probably other people have asked you this too. But something that's really becoming popular and that people are doing a lot with is WebAssembly. And you've got some crazy things like taking existing apps and compiling them to WebAssembly. And so I'm, I've gotta ask, I'm so curious, has there been any work done with this in Pewter? Can I take an existing C++ app, compile it to WebAssembly, and somehow bootstrap that into this on the internet desktop interface? Nariman: Funny you ask, because we're doing a lot around that right now. In fact, tomorrow we're dropping a bomb when it comes to that. We're releasing a major port, a really major application. I don't wanna name it, but everybody knows it. We ported it to the web for the first time. Nice. And I think... And then we have two more, and they're progressively crazier. So yes and no again. The layer is not there yet, but we're working on it, and we have the proof of concept coming out. The first demo is coming out tomorrow. So very timely question actually. Nice. Jonathan: Yeah, so I was thinking about this at the Ubuntu Summit, and it's a potential end game of this, you could see there's several different ways this could go, but in the end, you could see Pewter being, like, almost a target that you could use CMake to compile something to. If you have a sort of a standard interface that would be, would map onto POSIX. You may not be POSIX compliant per se, but you can map it onto POSIX. And you could imagine just being able to write a prog- a cross-platform program, and one of the targets be Pewter via WebAssembly. And that's really interesting to think about. Nariman: Yes. Yes, for sure. And I think we're making incredible progress towards that. And this demo coming out tomorrow is a really great proof of concept. There, there's work to be done because there's a lot of file system calls and those sort of things- that we have to properly port and create a layer for. But what we're definitely making huge progress. Again, I think I keep, I I don't really bringing this up all the time, but I think AI is helping a lot with that as well. For sure. Otherwise, we would've needed s- we have a lot of contributors, but we would've needed way more than that to, to achieve some of this stuff. Jonathan: Yeah. So what what does the contributor base look like? There's hundreds, thousands of people working on it? What... How does that map? Nariman: No. No. We're at 390 right now, and there are ups and downs. So the way we typically do open source contributions, attract open source contributions, is that we go through cycles- where we create a lot of issues, we ask the community to help us out, and that's when things pick up, and then we have a merge cycle where we go through all of these and merge them back in and close all the issues. So right now we're not in one of those cycles, but when it happens we get a ton of contributions. I think by the end of this month we're gonna start another cycle like that. It's an odd, ... I would say it's an odd arrangement the way we do it, where we go, we call for contributions, and we create a lot of issues and we ask people to help out. But you didn't ask this question, but it's an answer to the burnout question, so that you w- you wanna manage your resources, especially you have limited resources, you wanna have focused times where you're only focused on the open source. Of course, we get contributions over time. We get contributions from some companies that are using Pewter, so there's ongoing things like that as well. But the bulk of contributions happen in cycles rather than consistently. Jonathan: Managed sprints with the community. Nariman: Yes. Yeah. I would call it that. That's a better way to call it actually. Jonathan: Yeah. Interesting. I'm looking at the App Store now and there's just a bunch. There's just a bunch of crazy stuff on here. There's games, productivity, lots of AI stuff of course, but has anybody ported has anybody ported Doom yet? Nariman: Oh, yeah. Oh, yeah. That, that, that was a thing. That was a thing back in the day. Yeah? Ev- everybody was obsessed with bringing Doom to it. But then again, Doom has been ported to the web, and Pewter is basically a web environment, so you don't have to do much to bring Doom to Pewter. Jonathan: Yeah, that's fair. That's fair. I remember you said that apparently there's been some interest in Pewter from the crypto community. Not the cryptography community, but the cryptocurrency guys. Nariman: Yes. So I think there is some parallels between Pewter and the ideas of a decentralized ownership and decentralized computing. So it's been it's being used by two crypto startups, and they've raised a bunch of money, and they're working... They basically forked Pewter, and they're building their own version of it, which is pretty awesome to see. So some of it comes back upstream obviously. And yeah, so they're building... One, one project which is interesting is that they're building a decentralized app store on top of Pewter. And so with NFTs, every time somebody clicks on an app or opens a file the creator of that app or file is supposed to get paid. I think it's a pretty cool idea. Jonathan: Interesting. Yeah. How does that work with the license? Isn't Pewter AGPL? Nariman: Yeah. It should be fine. As long as you open source it. Jonathan: Yeah. Nariman: Oh, Jonathan: that's true. Nariman: It should be fine. Jonathan: Yeah. A- AGPL is kryptonite to a lot of big companies like Google and Amazon. They're like- Nariman: Isn't that the point? Jonathan: I just find it interesting this startup, a crypto startup can could use AGPL and play nicely with it. I thought it would have been kryptonite to them too. Nariman: Yeah. But I really like it. I remember reading this article by Torvalds a long time ago, and he was saying GPL is the reason Linux is this big. I think he's being too humble, but it, it does make sense. It makes a lot of sense, right? We're going to give you a huge amount of resources we've put into R&D and IP creation. We're gonna give it to you. You can do whatever you want. Just- ... if you change it, just contribute it back and everybody wins. That's basically the idea. But AGPL as everybody probably knows, the audience being open source AGPL is basically the GPL of the cloud world, so that, the loopholes around cloud are basically closed up. Jonathan: Yeah, the GPL, if the code runs on somebody else's computer, you don't get any rights to it. The AGPL- Yeah ... basically says you still get rights to it, even if you use it on somebody else's computer. Nariman: Yeah. Jonathan: Yeah, it's, Now, does that mean that all of the apps have to be AGPL as well? Nariman: No. The SDK is Apache. Jonathan: Okay. Nariman: And the linking happens within an iframe, so it's not linked directly to our environment. No, you're safe. Your app is yours. Jonathan: You've done your homework. You understand how that works. Nariman: We have a, we have an incredible lawyer by the name of Heather Meeker, who's done all these licenses. She's very famous. She has written so many books about open source. In fact, she's the reason, she's the main reason I open source. She convinced me. So she's done she's done the open- the open core licenses for MongoDB, Elastic, if I'm not mistaken, all these big names. So she's incredible with this. And she guided us. Jonathan: Yeah. What so that was something I was gonna ask you about. When you first wrote this, it was all closed source, and you got talked into making it open source. I'm curious how that went, and also how it's going. What do you think of that decision now? Nariman: So ever since I put Pewter out, I would say one of the top three requests from the community was to open source it. So I was bombarded by it, comments, emails, DMs. Every- everybody just wanted this open source. I didn't, I- it wasn't a huge priority for me at the time in the beginning And because we ended up be- becoming a funded startup, there was also considerations around, hey, should we open source this as a business? And so I wanted to, even though, look, for many years I've been a programmer, ba- I basically grew up writing code, I still needed a proper, really logical business argument for it. I had to understand it really well- ... before open sourcing it. 'Cause I'd never done an open source company before, where the core product is literally open source, the entire thing. Yeah. So I wanted to understand this really well before I do it. I met, I ended up randomly meeting through friends, the founder of you might m- you may have heard of them, n8n the workflow automation software. So they are open core, but they're massive contributor base base. If you care about GitHub stars, I think they're around 200,000 now. So they're a really huge project. Yeah. Super close to the community, really doing it right. So the founder actually told me, "This is perfect for open sourcing." He gave me a number of reasons, and then he introduced me to the person I just named, Heather Meeker, whose whose entire career has been focused on open source. And then she's the one that gave me a bunch of really specific reasons why Pure should be open source. So for example, one argument is that, first of all, open source doesn't work for everything. For example, open source games, they're not a really huge thing, right? There are a number of open source games, but you never heard of a open source game that has, I don't know, 100 million users or something like that. Players. So some type of software is not really good for open source, even though I think everything should be open source. It's pretty awesome. But from a business perspective. But then again open source does really well with infrastructure or hyperhorizontal platforms, namely databases, programming languages, compilers- and and the best stuff of all, operating systems. You want that hyperhorizontal platform to be open source so that people can work easily on the long tail of applications of it. As an example- ... we would've never thought somebody would take Pure and extend it for crypto. These ideas wouldn't, would never come to us. So they took it, and they're improving Pure just because they have another type of use case for it. There's another really good reason for it. So in the beginning when we put Pure out and it went viral a couple of times, one of the biggest questions people had was around privacy. They're like- "oh, you're harvesting the data. You're trying to resell it." Because it was an open source and it was free, just, I just put it out. And because of the architecture, it was super cheap to run. So everybody was like, "What is this? If this is, if these are VMs or containers, how can he afford it if he's not selling the data," right? So it's, it makes sense. Even though I spent so much money on creating this incredible privacy policy, nobody cared. They were all suspicious. And then I remember Heather told me that the, your community is mostly developers, and you're basically front-loading the cost of building trust and branding with developers by releasing your IP to the world. So overnight, you're exchanging that with brand equity- ... and trust with developers. And she was right. The moment we open source it, when we open sourced it two years ago, never again did anyone ask me about privacy. It was the number one question before that, and the moment it became open source, nobody really worried about it. Jonathan: Yeah, absolutely. Nariman: So there were a couple other reasons, but I think these were really solid reasons. And there, there are some practical questions about how to open source. For example, you want to use common open source licenses. AGPL, MIT, BSD what have you. You don't wanna go out and create a custom license. You gotta be very brave to do that. The reason being people just don't know it, and then you add this thing called understanding tax to it, where now you have to understand what the license is. If it's a company, the lawyers are typically familiar with AGPL, GPL- ... all these famous ones. But then now you have a weird license that nobody understands, and everything just- Right screeches to a halt. Jonathan: If you're AGPL or GPL, there are literal cheat sheets where you just go, "Can I do this thing that I wanna do?" And it goes... Yes. Nariman: Yeah. True. True. Jonathan: Absolutely. So there's a- y- you were talking about trust a little bit, and I was gonna hammer on this, because that's the- that's the first thing that comes to my mind, is, somebody has a product, especially if it's free, and it's just out there and it does great things. I'm like, "Man, why would I trust this?" There's the old adage, right? It goes, "If something is free, you are the product." Nariman: Yeah. Jonathan: And that makes a lot of sen- especially the AI world that we live in now, that's also true, because usually if the thing is free, that means that your data that's going into it is being used to train an AI somewhere. And people might say, "Oh that's overly cynical." It- You may have seen Pokemon Go, right? Everybody was playing a few years ago, this was before COVID. I'm old enough to remember the Pokemon Go craze. Turns out that data has been used now to train AI of all those players' data. Oh, I didn't know anything. Nariman: Oh, Jonathan: wow. Yes. Yes. I forget- I never Nariman: played it, so Jonathan: I forget the exact details, but yeah, Poke- the Pokemon Go data has now been recaptured and where they had all that, the data downloaded, and it it's being used to, for training for AI now. Nariman: Oh, Jonathan: wow. Again the product was free, therefore the data that you generate is the product. And it's not that can't be a thing with open source. Obviously, there are companies that use data that... And sometimes that's not necessarily a problem, just so long as, everybody's upfront with it and you understand what's going on. But even just, would I be willing to put anything at all confidential into Pewter? If I can't see the source and I don't know what it's doing, then no, not at all. So I, I think just from the trustworthiness perspective open sourcing it was really the only option. I just, I can't see, I can't see the thing having a life outside of being open source just because of that. Nariman: You're right. You're right. And this answers the second part of your previous question- ... how is it going now after open sourcing? I would say it was an incredible decision. It, it couldn't be better than that. I think open sourcing Pewter was the exact right decision. I sh- I just should have done it earlier. So no I don't regret it even a bit. And I think it's given me so much personally and as a business. It's just Seriously, the best decision in the life of Puter, in the history of Puter. Jonathan: Yeah. Yeah. So I've... i'm real curious about a few things related to this. Do, does Puter have end-to-end encryption? Can... And again I'll be slightly cynical here. I'm doing it on purpose- Yeah ... since you can make the point. Can you look at my files if you want to? Nariman: Y- your files are encrypted at rest on S3. Okay. And if you really care about the privacy part, the open source Puter works really well, so you can put it on your own server and you can do that. But as much security and privacy as we get from AWS, and I think as an open source project and being a startup, we are pretty... I look at other projects and a lot of projects obviously have resource issues. You don't have enough volunteers and everything. But I think we- we've been pretty diligent when it comes to privacy and security. We have this really active bug bounty. We do, for example, buckets in different regions. We support this kind of stuff that is pretty much enterprise level, but we support it as an open source project. So we've gone out of our way to be privacy friendly. Now, end-to-end encryption, the way where your files are encrypted in the browser, we still don't support that, and it's mostly because we're not too good at it. We still have to figure out how to do it properly, but I think we're not too far from that. And our business model has nothing to do with people's data. That's one thing. So it's not oh, we're gonna sell ads to them based on their files or anything like that. So it's only in our interest to actually get deeper and deeper into this- Yeah because it's good for businesses and enterprise. Jonathan: Yeah. What is the business model? Nariman: Yeah, the business model is actually pretty simple. It's subscription based on the amount of resources that you have. So when you go create an account, you get a few megabytes of free storage, some AI credits. A lot... What we do, aside from AI and storage, we provide a lot of services under one API. There's a database there's hosting. There's, again, the AI and storage. So there's a lot happening. We have serverless workers, which is compute in the cloud. We have all of this as a simple API, and obviously if you consume, if you use a lot of apps, you're gonna keep consuming these resources, and at some point you'll, you're gonna end up upgrading. Jonathan: You're gonna grow out of the free tier. Nariman: Yes. So it has nothing to do with somebody analyzing somebody else's data to monetize this. It's none of that. It's basically pure cloud computing infrastructure for the regular user. Yeah. So it really does make sense for us to get deeper and deeper into privacy, security, and those type of things. Jonathan: Yeah, absolutely. Do you see a, Do you see a future where it might be, like, really useful for enterprise? Are you guys gonna have a, Oh, I for- I forget the terminology. There there's several different accreditations that you can get to where, you can use it with US government stuff and do you see that sort of thing in the future? Nariman: Oh, for sure, yes. Yes. And I think we're very well-positioned to do that because maybe out of luck in the very early days, I designed the architecture that is basically sharded to a to certain regions, meaning that even if you use pewter.com, which is the hosted version- ... you can, in theory, have all your data housed in a specific location. So that's really perfect, and it was baked into the architecture from five years ago- I see ... and it just grew with it. And that was by luck because I just saw the API and in the early days it was only S3. I saw the API that you can set the region. I'm like I'm gonna add this to our API." So that's basically what I did, and everything formed around it. So now that gives us so much advantage because this entire architecture and all these hundreds of thousands of lines of code has formed around that architecture that architecture choice. So we are incredibly well-positioned from that perspective. Also, being open source and having feature parity between o- the open source and the closed source version, proprietary version, the hosted version, all of these, and the fact that everything on Pewter is scoped to the user rather than being multi-tenant, that's also really incredible for the enterprise. It basically makes it enterprise ready right off the bat- ... from a technical perspective, but we still have to go get those certifications. I get that. Jonathan: Yeah. But that is something you guys are thinking about NIST certification and all that? Nariman: Yes. Jonathan: I s- definitely some potential there. Another hat that I wear, I've had to start thinking about these things, and some of that is a challenge to get right 'cause you've got to certify the entire desktop, the whole computer you're working on, and so- Yeah ... it seems like it could be a really interesting piece of the puzzle for businesses that need it, that are required to have it. Yeah. Nariman: Yeah. And I think in general, in the long term, it is, just from a business perspective, it might be a moat. I think some of these certifications actually enforce some good practice, so it's not just check- I see that- Jonathan: Some of them enforce some bad practice too. Nariman: Yes. True. Jonathan: Make it opt-in please. NIST certified you've got to use five-year-old encryption. You can't use the newest stuff, and there's- Nariman: Yes. Jonathan: There's some nasty stuff in there too. Oh, anyway so all of that is... Let's see. I asked you about the the WebAssembly, and you said you've got a f- you've got a fun announcement coming tomorrow. Hopefully, we can get that in the show notes, and people can go and check that out. It's pretty obvious why you used JavaScript, right? There wasn't a whole lot of other option outside of JavaScript. But you opted to use jQuery, and I'm curious about that. I have... So far I've managed to resist, and I've not written anything in jQuery. I don't know why I am stubborn and have chosen this hill to die on, but so far I have. Nothing particularly against it. I just, for whatever reason, it's no, I don't want to use jQuery in any of my stuff." What led to that decision to, to use it? Nariman: Yeah. So with the performance requirements of the desktop environment, we couldn't really use frameworks, front-end frameworks. So we had... Basically our choice was vanilla JavaScript meaning no li- no, no frameworks. And vanilla JavaScript is pretty nice. It's come a long way, it's... a long way. But I would say- It's still pretty difficult to work with vanilla Java scale- JavaScript at scale. So jQuery this library that was released, I think, in 2008, and it was built- That's a while ago for a completely different... a lot of stuff that jQuery did in the early days made it into the JavaScript, ... standard. Yeah. So we use jQuery as basically a helper library rather than a state management sort of a framework. But then again this sort of became a meme and I just ran with it. I'm like, "Okay, people make fun of it, but it gets so much attention." And some people actually defend it because it was such a wild choice building something this complicated in jQuery. But looking at other major very front-end heavy applications like OnlyOffice or Photopea, and I think even Google Docs they're- they don't really use any frameworks. They might use frameworks around the around the core for some of the basic UI stuff- Yeah ... but none of the major rendering or anything like that is happening using frameworks. It just, it's not performant enough for these purposes. So I use jQuery, and it- it's been incredible. We got really good feedback around the front-end code because it's easy to understand. Anyone who has any JavaScript experience can probably easily understand it, and you don't have to be tied to a certain framework to just read the code. So it did work out really well. And the desktop environment is very very stable, super stable. It never basically crashes. And it doesn't have performance issues. So far, we haven't heard complaints about it. It took many years to get here- ... but I think it was a right choice. But then again, something that has to be clarified, jQuery is a helper library that helps with with some of the stuff that we do with vanilla JavaScript. It's not, we're not managing the state using jQuery. That's not... We- we're managing the state using DOM, which is another crazy thing we do. But but jQuery is not, oh, jQuery versus React. It's not like that. Jonathan: Yeah. Interesting. You guys are using it more just to like backwards compatibility and to make sure things act the same on Firefox versus Chrome, that sort of thing, right? Nariman: That, that was the initial reason jQuery was made. To make make the code work consistently across different brow- browsers, because they weren't adhering to the standards perfectly- ... back in the day. But we use it because of SelectorsAPI. But that's not a problem anymore, Jonathan: right? Nariman: Yeah. You tell me. Have you written code for Safari? Jonathan: They still make Safari? Nariman: Oh my God. We have to. And- it's a, it's an incredible browser, but there is the stuff where- Yeah ... all of a sudden something breaks. And you go dig in, oh, it's not it's not... They haven't implemented the certain standards or whatever. Especially with the stuff that we push some of the really cutting edge that our team does, the ones that we are actually gonna start announcing, we do hit the wall pretty quickly with Safari a lot of times. Jonathan: Yeah. No, I'm not surprised. That makes sense. Yeah. I say that tongue in cheek, but Safari's dropped in popularity a lot. And then of course, if you're running iOS, your browser is Safari- Yeah ... whether you like it or not. Yes. I'm curious, have you tried any of the up and coming alternative browsers like Ladybird or what's the other one? Servo, I think is the other one. Nariman: Well, Ladybird I love Ladybird. I've been a sponsor for a very long time. Oh, very nice. Cool. I think it's an incredible project. I've talked to Andreas. Jonathan: Oh, Nariman: he's great. Yeah. And I once o- one of their team members used to work for Pewter as well. Jonathan: Okay. Nariman: So yes I do I do love them, and I think it's a great idea. The core of it is that Ladybird, I think could be the Linux of browsers, a really completely proper- open source license core that doesn't have any commercial interest. It's not tied to a certain company. And it's not corrupted. I love it. I love it, and I think it has incredible potential because if... Going back to what I said half an hour ago, if you go back to the idea that the browser is the new OS with some footnotes- Ladybird is very well positioned to be the kernel of that, basically. And in- incredible project. Jonathan: Yeah. Yeah. Super cool. Nariman: But does it work? We did port it to Wasm. It's a browser. It doesn't get more complicated than that. I think it's one of the most complicated pieces of software you can ever build. We did port it to Wasm just for fun a couple weeks ago, so you could run Ladybird in a browser. I mean- Oh ... just for fun, obviously. Not a lot of utility value, but we just did it for fun. Jonathan: That does get back to that thought, though, about WebAssembly as the new a new target to be able to compile anything to. I think that's a really intriguing idea. And- Yeah ... boy, I hope that can become a thing. There are so many things that would be really fascinating to be able to just run them inside of a browser instead of on the desktop. B- but does Pewter work inside Ladybird? Nariman: Yes. Really? To some degree. Yes. Surprisingly, yeah. Yes, Jonathan: asterisk. Nariman: We, we haven't tried it recently, but I remember Andreas actually did create a video about it. A lot of the things loaded, and I'm really surprised because we're doing some really crazy performance intensive stuff behind the scenes. And this was, I think, over a year ago, so I'm pretty sure it's way better now, way better. Yeah, I'm Jonathan: sure. The way Ladybird development works is one of the developers goes, "Hey, this website would be really cool to support." And then they go and they try to load it up in Ladybird, and they see all the things that are broken, and they'll just sort of- step down through the list of what they need to do to make that particular website work. And that's how the development works over on Ladybird right now. Yeah. And if you've got some people there that are interested in making Pewter work, then, yeah, sure. I'm sure they're working on the things that you need to bring it up. That's pretty intriguing that it works there on yet the, yet another browser. So is there better support for Pewter on Ladybird than there is on Safari? Nariman: No. No. Okay. I haven't been able to corrupt them. No. My sponsorship is not big enough to sway the interests of developers. Jonathan: Yeah. That's funny. That's funny. I- so interesting interesting segue here, talking about being corrupted. You've taken some venture capital money, right? How does that, how Nariman: does that work? Oh, boy. That's corruption now? Yes. We have. We've done we've done, I would say, two rounds of financing- Okay ... institutional investors. And yeah we were lucky to find investors that are super aligned with open source. They get it, they understand it. Open source com- it's called commercial open source software. So that's when you take an open source project and you have a commercial arm, it's called COSS or a commercial open source software. I think the understanding of it has been commoditized at this point. Investors get it. They understand it. They know how it works. They know what it means to be a commercial open source project. I think the success successes of the past the more recent ones, like MongoDB, Elastic- these sort of infrastructure offerings that became huge that really pushed investors to take open source more seriously. But historically- ... Nariman: I think open source was undervalued until we had these giant unicorns come out of it. So at this point, people just get it. A lot of VCs, they they are tracking every metric of repositories. They have indexes. They have they have software that tracks it to see how well they're doing and reach out to them. So it's become a very competitive s- space. So back in the day, I think it would've counted as semi corruption, but now it's just Jonathan: How is that process going though working with your in- your investors? Is it still going along pretty well? You guys are ma- making them happy and they're making you happy? Nariman: Yes. Yes. I think when it comes to open source, we w- again, we were just incredibly lucky to have investors that just get it. So being open source, I don't even remember being open source was a question at all in any maybe issue that we had with techno- whatever is happening on a day-to-day basis, it's never "Oh, maybe you shouldn't have open source." Like- ... not even once has that come up as far as I remember. So it's never about being open source is just the bedrock of everything we're doing, so we don't even go back to it, "Oh should we change the license or anything like that?" Yeah. No. Jonathan: Yeah. You've done quite a bit recently with with AI tooling, right? A bunch of the apps now have AI. You even mentioned that when someone signs up, they get some free AI credits to use. Nariman: Yep. Jonathan: What's been the what's been the special sauce that makes that work? Why is that so interesting inside of Pure? Nariman: Yeah. So if you go to our developer site the about s- SDK and everything, developer.pure.com or the API documentation, docs.pure.com we've explained this using illustrations and everything, but early 2025, we saw a surge in application. So first of all, when we open source in March 2024 application development really surged on Pure. We had issues at attracting developers, but open sourcing it really put us in a much better position, so we started getting developers organically. But I s- started seeing a surge in 2025, early 2025, and then we realized this is because of AI. So turns out- ... the architecture of Pure is really good for vibe coding and AI programming. Jonathan: I was just looking this up. So the term vibe coning, vibe coding was coined in February of 2025 by Andrej Karpathy. Nariman: Yes. That's when it started taking off, and I would say end of 2025 is when it just completely revolutionized- programming. Jonathan: JavaScript is probably really s- well-suited to the LLMs writing it, isn't it? Nariman: Yeah, I think not being typed or weakly typed, that contributes to it. And it's just, you don't need a very complicated build process- ... Nariman: To run a JavaScript program, basically, especially the browser-based ones. And again that's one reason Pure JS took off so much, because you add... lLMs, they don't need to MPM install anything, or they don't have- ... you just take that CDN URL, you put it in your HTML file, and then you see it in ChatGPT or or Claude. So that's why it took off. But yes, JavaScript is well-suited. Python, JavaScript. Yep. Jonathan: Yeah. You think about it, too. The entire training corpus of LLMs has JavaScript just sprinkled all throughout it because they're training on the internet. And so they're just seeing- ... JavaScript everywhere. Yes. And that's one, you might be able to make the you might be able to make the argument that JavaScript is the native language of LLMs just because there's so much of it on the internet. JavaScript and H- HTML, right? Nariman: Yeah, and I think there's another reason for this. So if you go to ChatGPT or Claude- ... but nowadays people use people use Call Code and Codex. They actually go through the installation process and download these tools. But a lot of people just use ChatGPT or Claude or Gemini, the web interface. Yeah. And if you ask the web interface to build an application for you, the only way it can actually show you something when they load the artifact or the preview panel the only way they can do it is by writing it in JavaScript. So they're biased, I think. They're biased towards building it in JavaScript so that the user can see it. Yeah. And I think that creates a vicious cycle or a flywheel. Jonathan: Yeah, for sure. One of, one of the guys I work with, he used, I think it was Claude, to put together like the b- a bare bones presentation, like PowerPoint presentation, and he got to looking around in the back end of that, and Claude built it in JavaScript first and then rendered it from JavaScript over to PowerPoint. Oh, wow. And did a really good job on it, too, which is fascinating. Like they're really good slides. Oh, wow. We've used them several times now. Just keep... we keep iterating on them. We'll go and we'll make changes by hand as needed. We'll have the LLM do changes, just depending upon what it is. But yeah, it's so- Was that Nariman: Claude Design, Jonathan: or? I think it was I think it was Claude. I don't for sure if it was Design or Chat. Probably Design. I don't know. It's been a little while ago that he did this, so I don't know if Design was a thing at the time. I think it probably was. It was probably in Design, but yeah. It's incredible. It's real fascinating to see. You've run multiple companies now. You said you're a serial entrepreneur. Is it particularly different doing an open source company? Nariman: Yeah, there, there are differences. We have a much larger community to deal with. And yeah, the, in terms of community and conflicting interests is a completely different ballgame. And from a business perspective also, the strategy is kinda interesting too. You have this hybrid strategy where, okay, you have the open source project and that has to stay open source and has has to have maximal feature parity with the proprietary version, and then you have this other product which is proprietary. You wanna make sure there's no conflict of interest between the two. Be very clear. So there, there is really differences. My other company, I did... I was pretty cl- pretty close to open source projects as well. For example I our company was doing very large scale data extraction from the web. So- ... Mixnode was a name, and it was massively scalable web crawling. So it was actually suited for AI and those kind of applications where you need billions of web pages of homogeneous data. So not scraping, nothing like that, just getting huge amounts of data to basically for machine learning for the most part. And I would, we were one of the seeders of Common Crawl, which all these LLMs are based on. So we would we would contribute to Common Crawl, and we had a lot of open source projects as well with Mixnode. But it was never the core strategy. W- looking back at it, we should have probably open sourced it. What were we thinking? But then this was a long time ago. But yeah, no, it is definitely different. There it's more complex to run an open source company, but I would say it's way more rewarding when it comes to software. Jonathan: Yeah. You get to see... I- if nothing else, you get to see your stuff used in places that you wouldn't have you wouldn't have imagined. Nariman: Yes, and I think that's incredibly important, not just because it's fun- ... which it is, but it's also very important that a lot of people take your creation, and they use it for things that you couldn't have imagined. So I've heard that somebody's using Pewter. They're trying to use it in a car in China. And so these are the types of things that I would never come up with them myself, right? Look at Linux. It's being used in the craziest places, and it makes sense 'cause you need that diversity of backgrounds of people who use it so that they can- ... take this, and they were like, "Oh, I have this idea. I could use this in my line of work for this purpose." And it wouldn't be possible without open source. Jonathan: Yeah. Have you, I don't mean this as a gotcha question, although it is. Have you seen Pewter pop up yet? Has it been used for things that you're like, "Oh, I really didn't wanna see our name there"? Nariman: Oh we are a very huge target for cyber attacks and those sort of things, and we have... I hope I'm not challenging people to attack us more, but we have pretty powerful anti-abuse systems in place. But there's been, people people were using it for phishing, that sort of stuff. Yeah. It's under control. We have a lot of systems in place right now. We are, again for a small company or with limited resources, I think we've done exceptionally well when it comes to that. We put a lot of effort and investment into it. But the really bad stuff, no. Thankfully, we've managed to prevent those. Jonathan: That's good. I I, so I'm involved in the Meshtastic project. Obviously you know that. That's what I was talking about at Avanti Summit. We did an interview one time with a reasonably well-known online magazine and, no, I think they have a print version too, and the the whole... It was like a 45-minute interview. The first 40 minutes were great, talking about the technology, and then they had the gotcha question at the end, which we knew it was coming. We figured that they would try to do this to us. But they had the gotcha question and they're like, "You know that Meshtastic is potentially being used for human trafficking across the Mexico border." And we're like, "Sure it is. Don't do things that are illegal with Meshtastic. Other than that, we're just like Signal. It's an open source project. You can use it for whatever you want." So we were ready for it. We knew there was gonna be a question like that coming, and there's some other things going on around the world that we keep, try to keep track of and, some of it's cool. Some of it, you're like, "Yeah I wouldn't, really didn't wanna see our name there." But anyway I think it's something that a lot of open source projects have to have to wrestle with and deal with, particularly ones, like Signal and Telegram, those sorts. They, they wrestle with this more than even than we do. But I'm always curious to hear, like, how that plays out. Nariman: Yeah. How can you even prevent it, right? If it's open source- ... Nariman: And you can Jonathan: take Nariman: it. Jonathan: Depending upon what it is, like what you're talking about, there's also the question of should you even try to prevent it? Or is this just... You know what I mean? There's a... And I guess when you're talking about a hosted platform like what you guys are doing, you do have some responsibility to prevent certain things. But there is, part of the open source definition, in fact, is, you don't make moral judgments with your licensing. Yeah. So like it's, it, you, doing open source you cannot say, "We consider this to be evil and therefore you're not allowed to do this with our software." And boy, that's unpopular with some people, but there's a really good reason that exists, and that is that, the world cannot agree perfectly on what it means to be evil or not. Nariman: Yeah. Jonathan: And that's really what that boils down to. Have you guys gotten- Nariman: haven't some of the craziest inventions come out of military, for example? Oh, sure. If there was a moral judgment against it, "Oh, you can't use this technology," or, you can do a certain thing," we wouldn't... Maybe we would have them anyway, but probably not as fast as coming from a military funding- basically, a research facility. Military funded. Jonathan: I'm trying to remember. I wanna say the microwave was actually invented because of radar, if I remember correctly. Nariman: Yeah. I Jonathan: think, Nariman: I think so. Jonathan: Do. Yeah, I'd have to, I'd have to look to find that out for sure, but I think that, I think it's all related to that. Have you guys... And you may not be able to comment on this and that's fine, but I'm just curious, have you guys gotten like subpoenas yet? "Turn over this person's data"? Nariman: I'd rather not comment on that. All right. Jonathan: That's fine. That's fine. Yeah, that's a, it's a, it's such a tricky thing to deal with, right? We've yeah we've seen a limited amount of, We have been contacted by law enforcement once, and that was very informal. But still we see that on the horizon of this could be a thing. Now the advantage over at Meshtastic is we don't have anything that we can give over. Nariman: Yeah, that's what I was wondering. What were they looking for? Jonathan: They were... I think in that case, they were just looking for information. We think this person may have a Meshtastic radio. What can you tell us about it?" We're like, "Not much, man. Here's the documentation." Nariman: Yeah. Yeah. They... they can ask the same thing from ChatGPT or something 'cause I know you're not tracking anything. Jonathan: Yeah, so the, Yeah, for sure. Apparently the the microwave oven. This sounds like it was written by probably Google's AI, but it may be accurate for sure. The microwave oven was invented in 1945 by Percy Spencer, an engineer at Raytheon who was working with electromagnetic waves. So yeah, Google- Google's AI overview. But still, that- that's- that matches the that matches the memory I have. He was testing out a radar magnetron and noticed that the candy bar in his pocket had melted. Nariman: Oh. Oh, wow, I didn't know this part. Yes. That's really cool. Jonathan: Yes. It's also terrifying. Nariman: It's yeah, it's like the story behind antibiotics where it was basically an accident. Jonathan: Basically an accident, yes. In the case of the microwave oven, though, he was microwaving himself accidentally, and that- Nariman: Yeah. ... Jonathan: that maybe was less good, but, I dunno. Oh, true. Fun stuff, though. Nariman: Things haven't improved much because- right now if I put something in the microwave, I have a cold basically outer layer with a super hot nuclear kernel of, ... spaghetti. Or the other way around. Actually it's really cold inside. Jonathan: Yeah, that's usually what happens. It's Nariman: raw outside. Jonathan: Yeah. So what, we talked a little bit about this, but what's the weirdest thing that you've seen somebody do with Pewter? Where has it surprised you the most to find a pop-up or to hear a, s- from a customer? Nariman: Ah, that's a good question. It's funny that I'm blanking because PeterJust RS CK has been used by 180,000 applications, and I'm blanking. I don't know which one is the craziest. I think in our showcase we have some really cool stuff. God. I'm blanking right now. I'll remember it though. Oh, Jonathan: you'll remember the- Nariman: I'll remember it Jonathan: in a few minutes ... yeah. Yeah. Yeah. If you think of it before, before the Hackaday article goes live tomorrow- Yeah ... you can send me a, send me an email. Okay. And we can include it in the show notes. That'll work. Sure ... the story of the Chinese car company sounds pretty fascinating. Nariman: Yes. If Jonathan: somebody wants to use it- Nariman: But it's not- ... for their self-driving car ... it's not really public, and I'm not sure when we're gonna hear back with a solid POC. But I think- Sure ... it should be soon. Jonathan: Yeah. Nariman: It was supposed to be more than six months ago, but then again, you know how- Oh, yes ... how scoping and... Jonathan: I know how it goes, yes. I- I've been catching up on some of my emails, and it's "Oh, this was sent two months ago. Sorry for the delay. Been really busy." Nariman: At some point you run out of excuses, yeah. There's nothing you can say. You can just start stop apologizing and just respond. Jonathan: Just do it. Yep. Just do it. And then I gotta ask you a couple of questions. First off, before I get to these, is there anything that we didn't talk about that we should have? Nariman: No, I think it's been pretty great. We touched upon the most important aspects of it. Jonathan: Okay. All right. Yeah I feel like we've done we've done a pretty good job taking a tour through the ecosystem. ... If somebody wants to get involved particularly host your own, where's the best place to go to? Nariman: Our GitHub page. And we've put a lot of effort into making it easy to run Puter. You can... As a de- for dev environment, you can just download it, clone it, and then npm install, npm start. There's nothing else you need. And then we have the curl to bash as for the- Oh, you remember that. Okay. Jonathan: Yes, I do. Nariman: Great. Curl to bash Jonathan: ometer. I... So I've complained about curl to bash for years. And when he started telling that story, I'm like, "Somebody else is on board." Bad. Don't do it. Nariman: Yeah. Yeah. But that's what people expect, right? There was so much... we internally, we argued too much about this, right? Because our better engineers meaning everybody aside from me, was against it, but I'm the one who was like, "No, we have to do it because everybody expects it." We ended up doing it because I pushed really hard, but I get it. I get it. It's not the most popular thing from, So- ... the engineering perspective ... Jonathan: I have thoughts here. And we probably ought to explain to folks what we're talking about, right? So- ... the question here is, like, how do you install software? And one of the popular ways to install software is that the software company just gives you a script. You download the script, you run the script locally, and it does all of the installation magic, right? And so you end up doing a curl piped into bash. And at the Ubuntu Summit we were talking about the, one of the keynote itself was from Shuttleworth, was, like... He had this, this- He called it to the curl to bash-ometer. And it was essentially like how many of the most popular applications is this the primary installation method? And, his chart basically w- y- you're going along and you have this, and it just goes up through the roof, right? It was it was pretty interesting to look at, and they were of course using that to pitch snaps as the alternative. And I have for the longest time made the argument that, unless you really trust the person publishing this script, you should not just run the script, and the fact that websites can get hacked means that you probably shouldn't just run the script anyway. You need to look at it and try to figure out what it's doing. And installing packages is better. But then of course there is the sort of the overhead of somebody like Pewter has to then package up the thing for RPMs, package it up for Debian, package... Make sure that package works on Ubuntu, package it up for this, package it up for that, and it's a lot of overhead work, right? And so there's this kind of back and forth of do you just write one script that does the install or do you do the packaging? And what I would actually say is you probably have to have a script that does the install, right? Like I don't think there's any getting away from that because that's how you would do... That's how you're gonna install it inside of a snap. That's how you're gonna do a Docker install, all of these things. But like it is also super useful to have then based on top of that, a snapshot that is installable without the script. And there's tooling to be able to do that so that you can not directly run the big ugly scary script on your production machine, which is really what we want to avoid, right? Nariman: Yeah. But it's the most convenient one, right? Absolutely. Sure. You're encapsulating the entire complexity of choosing what to download, what to run- ... and you just give them one script. So that's why I fought so hard for it. And I'm like, "Hey, it's our responsibility to make sure everything is safe." I know it's easier said than done. We have way too much experience with this security stuff to underestimate how much of an effort it is. Yeah. But at the end of the day I've always been har- fighting super hard to make it easy to run Pewter. And in the beginning, for the longest time, and I think one of the reasons Pewter used to go viral over and over again, is because for the longest time, if you went to pewter.com, you would just see the desktop. You didn't even have to create an account. It was just a desktop environment. It would do everything automatically for you. It would basically trust you, give you credits no matter what, just because I want it to be incredibly easy to use. The reason we ask for accounts now is actually over time, that became an issue where people got confused if they have an account or not. So it's not even because I want them to create an account. Anyway- ... but the idea, the core of the idea has always been it has to be incredibly easy. It has to be easy. Whether it's open source, whether it's hosted, you wanna make it, make this super convenient for the user. And I think curl to bash- with all its faults it is something people expect for the most part, and that it is a pretty convenient way to install. But then you might end up nuking your machine, so you gotta be careful. Jonathan: Yeah. Is there like a Docker image or a snap- Yeah ... or something like that people can use? Nariman: Yeah, we have a Docker image. Yeah. Jonathan: Okay. That's... See, that's perfect, right? It's the best of both worlds. You get both. Nariman: Yeah. Jonathan: There there's a... I think you probably... Most projects of this scale, you've gotta have an installer script. I think, like you said, that's just expected. Yeah. And then you also have a group of people, like me, that look at that script and go, "Psh, I'm not running that on my computer. I don't know what's in there." We're just a little bit paranoid. Yeah. For good reason, I would like to think, but still a little bit paranoid. And then, you've got the Docker image, you've got the other things to where, you can package it up, and people can do it without having to run the script. And so everybody's happy. Nariman: I think it's great. Being paranoid about this stuff is great actually. It keeps everything, th- these are the checks and balances around open source, right? And I think these sort of being "Oh, let me look at the script first," it actually, in a way, helps make the project better over the long term if the maintainers actually listen to the concerns. But the good thing is that now you can probably... you don't have to read the script. You can just drop it in chat GPT and be like- It's true ... "Hey, is this... How horrible is this?" Not if it's good or not, how horrible is it? It, Jonathan: it- Nariman: And then you can Jonathan: see- ... you know what, I'm about to run this script. Do you see any signs that it's gonna do anything malicious or otherwise cause problems? Yeah. Nariman: Yeah. Jonathan: That's actually a really good point. We have tooling now. Nariman: Yes. Jonathan: Yes, we have tooling, We have tools that make this easier. Yeah. Yeah, absolutely. I'm looking at your documentation to make sure that you have the the other stuff documented here. I don't see it. Maybe I'll make a pull request inside the self-hosting Puter. Oh, boy. Yeah. The one-liner recommended... I don't see the, I don't see the alternative. Nariman: Huh. Take a look. Jonathan: But I'm, I am of course very quickly skimming through this, so I may just be missing it. But yeah, I would love to see a a little alternative so that people... Oh, hey, look. Ha, even above the one-liner, there's the Docker Compose option. Okay. Yes. You're good. I just missed it. Nariman: Thank you. Jonathan: Very cool. All right. Again, two final questions I've gotta ask. People will send me emails if I forget this. F- what is your personal favorite text editor and scripting language? Nariman: Oh, boy. I wasn't... D- do I have to say Vim? Is that, You can Jonathan: say whatever you want to. I answered this with Nano most of the time. I'm not... You're not gonna lose geek points. We've had a few people tell Nariman: us- I do Vim I really do, but believe it or not I'm gonna sound horrible, but I do use the text editor on Puter a lot. Sure. So it was the first application I built on Puter- ... and it's been it's been improved over the years. I'm actually used to it. So- Jonathan: Yeah ... Nariman: but then again, I don't think that's what people wanted to hear. So I'm gonna say Puter. No, that's a Jonathan: perfectly valid answer. Okay ... I just I think yeah, Vim plus Puter, I think that's a perfectly valid answer. Can you run Vim inside of Puter? Nariman: Somebody did port it, actually. We actually took out the terminal because it was way too buggy, so it doesn't really... until we bring it back, I don't think- Or we can have Vim again I was Jonathan: g- I was getting ready to ask if there was Bash inside of Pewter Nariman: No we did, but it was way too buggy. We tried so hard. We tried to port Linux to Pewter. We got really far, but it was just not production ready. Jonathan: Right. Nariman: We're making more... As I said we- we're doing this release tomorrow. We're getting way more serious about porting things to Pewter really complicated software. And we're making progress. Again like I keep saying this over and over again, but I think AI is really helping us with this. Jonathan: Oh, sure. Nariman: So now with fewer resources, we actually have a chance at porting those really massive software to, or packages to Pewter. Jonathan: Yeah. That would be fun to see that come back. Oh, yes. Just as a command line purist, that would be fun to have it back. Yes. Nariman: Yes. Jonathan: And its scripting language. Nariman: Scripting language JavaScript. I was gonna say. Yeah, it has a special place in my heart. Jonathan: Yeah, of course. I suppose on Pewter it counts as a system language, right? Nariman: Yes. As crazy as it sounds when you say it out loud, it is true. Jonathan: Yeah, makes sense. All right. Hey, it's been a lot of fun. Thank you, man, for for coming on. And we... After after some big announcements, we will have you back here in a few months and talk about some changes- Sure ... that's changed. It sounds like things are moving pretty quick over in the Pewter world. Nariman: Yes. Thank you for having me on. Yeah. This was great. Jonathan: Yeah, it's been awesome. Thank you so much. It was good to see you Nariman: again. Yeah, likewise. Jonathan: Yeah. Nariman: Take care. Jonathan: All right. That is Neriman Selva talking about Pewter, a really fun project, and c- becoming quite the successful business, so good for them. Next week we have somebody else that I met at UbuNTU. We're talking Michael Meeks from Collabora, the open source office suite. And then after that, in two weeks, we are talking with Francois Proulx about Smoked Meat. That is all about security in places like GitHub, in your CI runs. And then we're having Jonathan Polance, a return from Ferrous Systems. We're gonna talk more about Rust and the things going on in that world. If you have a project or you know of a project that needs to be on the show, let us know. It's floss@hackaday.com. Give us an email there and we will get them scheduled. All right. Wanna say thank you to everyone that watches, that listens. Get, whether you get us live or on the download, and we will see you next week on Floss Weekly.
• 
  Episode 875 - JavaScript as a Systems Language 15.07.2026 1h 10m

  This week Jonathan chats with Nariman Jelveh about Puter! It's the project that takes the idea of the Browser-as-the-OS seriously. Why did a simulated desktop on the web take off, what the story of making it Open Source, and what's coming next? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 874 transcript 08.07.2026

  FLOSS-874 Jonathan: Hey folks. This week I'm talking with Andrea Gallo of RISC-V International. We talk all about everyone's favorite open ISA, the fact that there might be more RISC-V cores in the world than there are people, and lots more. You don't wanna miss it, so stay tuned. This is Floss Weekly, episode 874, recorded Tuesday, July the 7th. Really, we do PDFs It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we're gonna be a little risky today. We're talking about RISC-V. I, I had to make the pun. I know. I'm sorry. I'm a dad. I make dad jokes. Anyway, we're talking about RISC-V, the open ISA, the open ecosystem. And we've got somebody, we have somebody very special from RISC-V International and from various places. He does a lot of things. It's it's the man when you talk about RISC-V. We have we have Andrea Gallo. I think I got that did I pronounce that right, sir? Andrea: Perfect. Yeah, perfect. Jonathan: Okay. Excellent. Perfect. Excellent. And where I forget to ask this sometimes, but where are you speaking to us from? Where are you at in the world? Andrea: I'm just close to Milan, Italy. Jonathan: Oh. Oh, my goodness. You lucky dog. That's awesome. I'm in Southwest Oklahoma, which has its own charms. A business colleague of mine- Of course ... just the other day was driving through Oklahoma, and they stayed at a, he stayed at a little tiny town with his family in Oklahoma. And he's "Oklahoma's really nice." I tried to tell you it's really nice here. So Andrea, y- you are you're now the, let's see, the CEO at RISC-V International, and I have about three different questions that go along with that. And I think the first one is what's your background? Y- if I remember correctly, you're you've come from the technical side of things, not the- ... i, I don't mean to be derogatory here, but not the bean counter side of the house. What i- what has that been like to go from the very technical side of things to you're now CEO of this huge organization? Andrea: Oh it's a big honor and quite a challenge, of course. Yeah, I have a mixed background of hardware and software for- ... for, since the beginning I've always messed up because I always enjoyed mixing the hardware side and the software side- ... in many different roles. And I've al- always enjoyed facing customers as well. So I started as application engineer in my early professional days. But even before that I was writing little games that were published by one of those old style magazines where you had all the listings, and so I always enjoyed the public facing side as well. And I was also responsible for business development in a previous professional role. So I always enjoys the mix of both technical and customer facing. And, Jonathan: I was thinking as you were saying that, th- there, there's some combinations that c- give you superpowers. For example I've, one of the lawyers that I work with, she's also studying to be a CPA, and o- once you have both passed the bar and passed the CPA exam in the US you're a superhero in the legal world. And I think in the software hardware world, someone that has written firmware and dabbled with software, has done hardware designs being able to do both of those sort of gives you a superpower. And then on top of that, you also do the business side of things. No wonder they made you the CEO. Andrea: No, the real guys are the founders of RISC-V, and all the great minds on the technical steering committee that drive the work on RISC-V. Those are the real brains. I think at RISC-V International, we're just serving these big brains. That's our role, help them achieve what they are, they're really doing. Jonathan: So that's a, that's actually a really good segue. What is RISC-V International? How does it fit into the puzzle of what RISC-V is? Andrea: RISC-V International is the nonprofit association is the organization that hosts and coordinates the evolution of the RISC-V ISA. So we ensure that the evolution of the specifications is done in full coherence with the missions and standards set by the original founders of RISC-V and we ensure that there are no deviations along the path and that we ensure that the process follows the right policies and procedures, and it's transparent and collaborative. I think our role is to ensure that all the members that found the work at RISC-V, they collaborate in a constructive and positive environment. So we have RISC-V International to thank for things like RVA23? For example so we as a nonprofit organization, we are funded by all the commercial companies that design products based on RISC-V or IPs based on RISC-V. But we also benefit from the collaboration of nonprofit and academia institutions and some very skilled individuals. And we are uniformly distributed worldwide. I've been traveling a lot for the very r- RISC-V summits and for other industry events, and also visiting multiple academia. And I can tell you that it's it's so rewarding. It's such a pleasure to see that you can face the same level of technical competence and maturity in understanding and adopting RISC-V, the same level worldwide. From the US to Europe to Asia to Brazil all these students studying RISC-V at classes, at courses using RISC-V for their projects, for their master thesis. It's unbelievable. And even on the daily work at RISC-V in the technical working groups, when we... in the end we do PDF. If I oversimplify, what we do is that we deliver PDF. That's all we do. That's fair. That's what we do. PD- PDF. That's fair. So when we, on a daily basis on the technical working groups I see the conversations from all these great minds and I don't see one geography producing more or contributing more to the specifications and one geography consuming more. Not at all. I see the same level of interaction and collaboration across all. Do, Jonathan: do you think the open s- the open nature of the ISA is to to, to blame i- is what causes that? So many people know about it because it's open and it's taught in academia and it gets used everywhere. Andrea: First of all, just along the lines of I was saying we, we do PDF I would like to add that we focus on the standards. We like to make the reference to the USB or the Wi-Fi standards. Behind me I have some products that I designed many years ago that were, they were based on the USB specifications. And you can download the USB specifications from the website wherever you are, and the same is true for RISC-V. You can freely download the RISC-V ISA specifications wherever you are in the world, and then locally you would invest in designing your IPs or products locally where you are. So it's important because these, Reinforces the message that RISC-V is an industry standard. And this was also recognized by ISO last year. 2020- September 2025, the ISO/IEC, the joint technical committee recognized RISC-V International as a PAS submitter. PAS stands for publicly available specification. It was quite a long effort and helped us improve as well the way we work in tracking participation from all the members, in logging consensus from the members, in ratifying the specifications. And all this has been recognized as a transparent and and fully correct way of working, aligned to the ISO/IEC guidelines. So we got recognized as a, as somewhat as a standardization body. And the next step for us is to submit the RISC-V specification, the ISA manuals, to be evaluated and hopefully ratified as an ISO standard in the future. So I, I was just adding this as an important point for us that RISC-V is a standard. It is not an implementation. There are many implementations done by the members. Jonathan: I think that's a, I think that's an im- important point to make and to step into, and I was thinking about asking you the question, the leading question because the ISA is open, that means that all of the RISC-V hardware out there is open source hardware, right? Not quite. Andrea: I ... No, not quite, and I really like the question. Not quite. From the standard Then you can have all of business models. You can have a pure open source implementation done by academia or by companies that offer an open source implementation, and then offer commercial services to customize or productize it. Or you can have a fully closed commercial IP designs for licensing. All options are possible, from a pure open source academia to a fully closed commercial. But I also like the hybrid model. If you think about it companies in the open source space companies like Red Hat or Canonical or others, in a way they all invest in open source software. They sell you a subscription to their operating system distributions, and then they also offer you commercial services to integrate that into your s- software products. And this is this is an option for some of the RISC-V members, have an open source RISC-V implementation and commercial support services. Or fully commercial. So all the shades of colors are possible. Jonathan: And so when we talk about the RISC-V ISA that, that's essentially like the definition of the instruction set and like the definition of what each of those instructions are supposed to do, right? Andrea: Yes. Exactly. Okay. Exactly. Jonathan: Exactly. And so you can implement that. Someone can implement a f- fully to spec RISC-V implementation in an FPGA, in silicon. I'm sure- Yes ... someone has tried to do a minimum version of it in vacuum tubes or, people do crazy stuff just because they can. So there's a bunch of RISC-V all around us. Andrea: Yes. And even more on our website. We offer free training courses. One of them is one of my favorites. I took it during my interview process actually to get the job. And the train ... the course was Design your RISC-V CPU core, and it guides you from reading the ISA specifications- understanding them, designing the instruction decoder, designing the register file, and an example of a very simple single stage, single clock cycle e- execution unit. And you do everything in a browser. You get all the environment the system Verilog, the compiler, the emulators, and you can even run some simple assembly software to try it out. A- and this is a course that I recommend to all the hardware and software engineers, both. Jonathan: Yeah, that's very cool. I went through... In fact, we've interviewed him on the show. There's a similar idea called NAND to Tetris, and they start with the NAND gate, and it's "Okay, here's how you put two NAND gates together to get a NOT gate," and, all of those different things. Yes. And it gets all the way up to now program your Tetris game on your operating system that runs on the CPU that you built. Excellent. I don't remember if that uses the RISC-V ISA or not, it, but it's gonna be similar. I love those sorts of things, though, because you s- you start with the very basics and you build each layer up- Yes and so you get to the point to where you understand the whole stack. Andrea: Yes. W- And you appreciate the simplicity. I've, I worked on many different ISAs over, over the years, from CISC to RISC many different ones, and I really enjoyed looking at the hardware side of RISC-V, how simple it is from the op codes, the instructions, the bit fields, to write the instruction decoder. It's class. It's really classy. I think Krste Zaninović, Andrew Waterman, Yoon Soplee, Dave Patterson, they did something amazing with this RISC-V ISA. Jonathan: So it's RISC-V, which brings the question to mind, was there a RISC-IV? Andrea: Yes. Yes. Actually it all started with a RISC-I in the early '80s. Actually in the early '80s Dave Patterson wrote a paper making the case for RISC, reduced instruction set computing, and then the RISC-I was the very first chip as the very first proof of concept. And then there were a few other iterations done at UC Berkeley under the direction of Dave Patterson. And in 2010 the work started on the fifth generation, so the V in RISC-V, the V sta- is the Roman number for fifth generation. But we also joke that is also reminds V for vectors, because RISC-V has been designed since the beginning to have the most flexible vector extension design. Jonathan: Interesting. That and that's part of what RVA23 is about, isn't it? Is additional support for doing vector instructions? Andrea: RVA23 is a profile specification. Let's see. RISC-V is very modular. You have many extensions that add small features or security or performance, and they're all modular, and you can It's like ingredients in a recipe. And you can build your RISC-V microcontroller that is perfectly handcrafted for your specific workload or application or specific function. But then, yes, everyone may have a different recipe. RVA23 standardizes the non-differentiating bits- ... and sets the bar high for a high-performing application processor. So when you are when you own your design, you want to optimize it at best for your specific application that you fully own, hardware and software. Then RISC-V gives you this flexibility, and you can add your specific custom instructions for that specific function, the, and you will get the most optimized design. But if you want to run a standard operating system, like Linux or an RTOS, you are not going to port it Once for every single vendor. That would not be sustainable. That would lead to unnecessary fragmentation. And this is what RVA23 solves. RVA23 defines the minimum set of extensions that shall all be present as mandatory, as regional options, as growth options- that shall all be present in a design, in an RVA23 application processor. And this means that the operating system vendors can target RVA23 at the single build target, and you will get binary application compatibility. A- and this has been ... This has worked really well because Canonical immediately endorsed it. And Ubuntu- I was Jonathan: at, I was at the Ubuntu Summit at, the ... Where they, where they d- released that. They made that announcement, and then they- Yeah ... they had a f- ... Was it Space MIT? One of the, one of the companies had- Yes ... some hardware there. Yes. And I was like, "Ooh- Yes ... I can take this home with me, right?" And they're like, "No, you don't get to take that home." Andrea: No, but you will get access very soon- Jonathan: Yes ... to that one. Yes. Yes. I immediately sent- So it- I immediately sent a message to one of my developers. I'm like, "Hey, send an email to these guys and ask them for hardware." Yes. Andrea: No, th- that's exactly the point, and Ubuntu now runs on every RVA23 compliant design. And similarly, Nvidia disclosed ... Last year, they disclosed plans to port CUDA to RVA23. So you do the port once, you do the work once- ... and it works on all the RVA23 compliant application processors. Jonathan: Yeah, absolutely. What what are some of the challenges? You mentioned fragmentation, and that was something I was going to ask about. But i- and we can d- dive a little bit more into that if you want to, but I'm curious if there are some other challenges right now in the RISC-V ecosystem doing d- doing this development or even trying to work with the other companies out there. Andrea: I would say that we are working in expanding RISC-V from a horizontal technology. In deeply embedded microcontroller or security processors or AI specific schedulers, for example. We are expanding RISC-V into multiple different vertical industry verticals and so we, we introduced the concept of the special interest groups at RISC-V. So we started with the data center, with the automotive special interest group. End of 2025, we s- we launched a space interest group. The concept is that you invite the domain specific experts who come in with the domain specific requirements, and maybe they are new to RISC-V, so they need help. I remember in the early days two years ago, the questions was, for example if we want to design an ECU based on RISC-V, how do we run AUTOSAR on RISC-V? How do we achieve the right physical memory partitioning or protection? How do we run a trusted execution environment on RISC-V? And then we bring in the technical experts, and it's like matchmaking. And by mapping the requirements to the RISC-V technologies, then we come up with a gap analysis. We may identify low-hanging fruits for innovation. And we also write application notes that they explain how to configure RISC-V at best for each each vertical. So in the automotive space we're working on, first of all, functional safety. We have a white paper with a functional safety special interest group and experts in functional safety. Where we find that there are no gaps. And then in automotive in addition, how to configure RISC-V for an ECU microcontroller, or for a zonal controller, or for the central compute in automotive. Similarly in space I find it quite interesting and amusing. For me, SaaS means Software as a Service. But in the space world, for for our friends and colleagues and members i- working in space, SaaS means Satellite as a Service. Y- cloud in space. And you lease a an instance, and you run your workload on a satellite. In that case, you have requirements of software isolation, security, redundancy- Reliability, quality of service and you need to map them all. And the flexibility of RISC-V means that collectively, all together, we, we can ... Maybe we will do another RVA23 specific for space or for automotive or for Edge AI Jonathan: That, Andrea: that- Like specific configurations- ... for the various markets. Jonathan: That, that is actually a really interesting idea to try to tame some of that fragmentation, is to have y- a- additional specifications that say, "Okay, look- Andrea: Yeah ... Jonathan: to be able to be able to s- you know, for us to put our stamp of approval, RISC-V International says that this design is ready for space, you've gotta have this." And then, the next sort of logical connection on top of that is somebody like Canonical or whoever wants to says- ... " Jonathan: We can build a Linux image." We are going to port- we're gonna- Yes, exactly ... build an image for that. And so here is the- Yes ... Ubuntu space edition with the stamp- Exactly of approval on it. Yeah that's that's a fascinating idea. I look forward to that happening. Y- Andrea: The if I use I very much like the term by ETH in Zurich, the academia. And Professor Luca Benini. He gave an a very fascinating keynote at the last RISC-V Summit just a couple of weeks ago. His theme is focusing on domain-specific accelerators, DSA. I like the term domain-specific accelerators, and it means that they are building extensions or accelerators for specific workloads in specific domains. So in that case, i- it's not unness- unnecessary fragmentation. In that case, it is a real innovation. When you want to run standard operating system for a standard application in standard market, then you need the profile specifications like RVA23, and even more, we recently ratified the server platform. So going beyond. If I look at the hierarchy, you have the base ISA, you have all the extensions that are modular. Then you combine an, a subset of the extensions for an application processor profile like RVA23, or we're working on an RVM profile for microcontrollers. And then the next step in the hierarchy, which is a requirement from our members, for example, in the data center space our members realize that there are so many peripherals that are close to the CPU core. Timer, IOMMU the message system interrupts for the PCIe bus, all these core peripherals close to the core They need not be different. They shall be controlled, configured, accessed the same way. Then you can compete on the implementation, of course. Best performance better power management, et cetera, latency. That's the implementation side, but they all concur that they need the same specifications, not only for the ISA, but also for these core, core close peripherals. And so the server platform that we recently announced is based on RVA23- ... which means mandating support for hypervisor vectors and a few other critical extensions. And then the server platform adds specifications for ACPI tables, specifications for these peripherals that are non-differentiating, and then specifies the standard boot architecture for RISC-V. And all this leads to the data center distributions to run seamlessly on, on the RISC-V server platform chips. Y- Jonathan: you knew exactly where I was gonna go next. I was gonna ask about how you boot. I have... So I love embedded devices. I've got, probably 25 Raspberry Pis around the house, and I'm a sucker for buying new things. It's oh, it's an, it's a RISC-V board that you can run something on. Oh, sure, I'd love to having a RISC-V board I could run Linux on. Or, it's a, it's an Arm board that you can run Linux on. Oh, sure, that's different. I don't have one of those yet. And- ... the amount of time that I have spent fighting with trying to get those to boot, and going in and fiddling with the Linux kernel and, the the binary tables and all of that to try to get all the hardware to come up is just... And so when Arm had their system ServerReady, they called it first, and then SystemReady, and I think they used UEFI in that, Yes ... and things started to just work. It's oh, that... It's so nice. And, Yes ... so I was gonna ask, like on, on these s- server RISC-V boards, what does that look like? How do we just get something to boot without having to have a device specific image to flash on there? Andrea: Yes. That is exactly going back to a- adopting industry standards And at the same time, industry standards adding support for RISC-V. So the server platform specifies exactly the standard boot requirement architecture and boot requirement specifications. And it's, it leverages UEFI ACPI for servers. And this was possible because the latest ACPI specifications added support, official support for RISC-V ACPI tables. So y- you see it's coming from both ways meeting each other. Jonathan: Yeah, absolutely. I- is there some cross-pollination with the things that Arm is doing? I guess I should ask this first do you guys consider Arm to be one of your big competitors? It seems like that might be the case, but maybe not. Andrea: We are friendly to everyone. I think the market is growing so fast that there's room for everyone. And we are friendly and respectful to everyone as much as to all the regulations from every geography worldwide. I think that a healthy competition is is a great asset for everyone for the industry. Jonathan: A- and then is there with things like the, a- again, I mentioned Arm's ServerReady, and it sounds similar to the RISC-V project. Has there been some cross-pollination there? Have some of those additions to ACPI been helpful to, to, to both of those pro- projects? Andrea: I think we are all adopting the same industry standards coming from the UEFI Forum. So again- ... it's a great value of the open standards, 'cause you can you can download and study the ACPI specifications, the UEFI specifications and adopt them. And of course, those specifications, they receive feedback from all the implementations. So they receive also feedback from all the ISAs. So there's cross-pollination between the industry standards. Absolutely, yes. Yeah. Jonathan: One, one thing that I've seen that I find real fascinating is the ability of the RISC-V, the core ISA, to scale both up and down. And so we have essentially microcontrollers that are RISC-V all the way up to, now with, specifically with RV823, you've got eight core chips that are r- eight RISC-V cores in there, and you could legitimately run it as your primary computer. It might not be the most fun experience yet, but you could do it. It- Oh, it's coming ... it's, yeah. Yeah. And we're rapidly approaching that as a, a more reasonable thing to do. What's the secret sauce? What have you seen with the ability of RISC-V to scale up and down like that? How is that possible? Andrea: Because it's a single ISA and it's modular. If you, if we go back to the example of automotive. You have an ECU the elec- electric control unit. That must be a very tiny microcontroller, real-time, low power. Then you have the zonal controller where you may need some AI capabilities. And you have the central compute. You have AI extensions. You have GPUs. You have NPUs. So- Outside RISC-V, you have many different ISAs. You have a different 32-bit RISC ISA for the CPU, and a different 64-bit ISA for the central compute. And a different ISA maybe based on RISC-V for the GPU or the NPU. So you have many different ISAs RISC-V is a single one. The same single ISA, the same single instruction set can be used across all. You can configure RISC-V to be a 32-bit microcontroller for the ECU in a car- ... and focus on the real time extensions. It's the same ISA that you can configure to be 64 bit, but it's still the same ISA, the same instruction. It doesn't change- ... for the zonal controller, and you can add vector extensions, and in the future, matrix extensions to have AI inference inside the zonal controller pre- preprocessing data coming from the sensors. And the same ISA without change can be used in the central compute. Then you would add extensions like hypervisor support. So the same ISA, the same programming model, the same tooling can... The same expertise can be used across all. Jonathan: Yeah. That's... it's super cool to see. And so you see as a result of that, RISC-V pop up in maybe some unexpected places. I'm pretty sure Intel runs r- a little tiny RISC-V core as their a- as their sort of s- root of trust in most Intel processors. And I think I've read about hard drives that have RISC-V process. It's, we- we're part of Hackaday now, and so I remember there was a there was a particularly interesting story where someone says, "I installed Linux on my hard drive." He's no. I don't mean I wrote Linux to the hard drive. I meant, I hacked into it and I'm now running Linux on the RISC-V processor in my hard drive." Oh. Andrea: I love this. Thank you for sharing it. Jonathan: I'll see if I can dig that link up for you- Andrea: Yeah. Yes, please ... ' Jonathan: cause that one was particularly fun. Andrea: Yes, please. But- Yeah, RISC-V is popping up everywhere. As you say, it started in the deeply embedded microcontrollers from Western Digital from the early 2015 when the RISC-V Foundation was re- i- initially created. And Nvidia was also one of the first to adopt RISC-V in those years. And Nvidia has been shipping RISC-V in every single GPU between 10 and 40 RISC-V cores in a single GPU. In 2024 Nvidia were our guest speakers at the RISC-V Summit, and they publicly explained, and that's a very interesting keynote. They explained how they use RISC-V in many different ways inside a GPU for many different functions, and you can find that on YouTube, or we can provide the link later for- for your for your guest. And they evaluated that just in 2024, Nvidia shipped more than one billion RISC-V cores in the GPUs. Qualcomm in 2022 estimated that they had shipped about 600 million cores in wearables, and just a few weeks ago they updated the they've estimate to two billion cores in wearables. Infineon one year ago announced that their entire automotive microcontroller roadmap is based on RISC-V. Meta is using RISC-V in their Meta training and inference accelerator, the MTIA chip. Google is using RISC-V in the Coral NPU, which was also released as an open source project- ... last year. And then you have AI accelerators with hundreds of cores by Tenstorrent, Inspire Semi. You have the latest NPU by MIPS, all using RISC-V. And then we have the application processors coming this year Jonathan: There are d- do you think we've passed the point to where there are more RISC-V cores than there are people? This is my metric for when talking to a project that is really popular, and so I asked this, I asked Daniel Stenberg this, the guy behind Curl. I'm like, "Are there more Curl installs than there are people?" He's "Oh, definitely." Andrea: The SHD Group estimates that RISC-V will pass the 35 billion SOCs in 2031. And every SOC can have more, multiple RISC-V cores, multiple. Just as I said, the Nvidia GPU has between 10 and 40. So I think that we by large, we passed the number of humans on Earth already. Jonathan: That's, that is so wild to me. And of course with the with the AI boom and just the way all of that is growing I don't think any of this is slowing down. I think we're just continuing to accelerate. And you have you have serious contenders talking about, putting cores up in satellites and doing data centers up above us, and- Yes ... it's just it is incredible to watch. It, we are in a very interesting transformational moment in in human history right now. Andrea: Yeah. And you were also hinting at using it as a main computer- ... Andrea: Right? And yeah I'm happy to touch on that as well. Thanks to Deep Computing we have had RISC-V based laptops for the last few years. With the DC Roma laptop, the first generation, the second generation, and then Yuning from Deep Computing partnered with Framework the modular laptop company. Yes. The latest RISC-V based Framework m- motherboard, based on the SpaceMID K3- ... you sh- you should try it. Believe me, you should try it. You will get 4K YouTube video playback, very fast browsing office applications. So it's still a machine targeting developers in terms of price tag. It's m- but you should try it. We were really planning to purchase as soon as it's available in quantity, purchase a few for the RISC-V staff, and use it- ... at events, and eating our own dog food. Jonathan: Absolutely. I remember I reviewed, this has been a couple of years ago now, it was, Vision Five, I think Yes. I r- I reviewed that, and I set it up with a monitor and I s- I tried to use it as a desktop for just a little while, and that was not a great experience as a desktop user. But in my review, I made the statement that it's just good enough for developers to start actually doing development- and compiling their stuff on it. And so it's not gonna be great for running as your desktop, but it is gonna be transformational for the RISC-V ecosystem as a whole. And I think we're now seeing some of the results- Yes ... of that kind of work with, you know- Andrea: Yes ... Jonathan: players Andrea: Ubentu- We don't- ... and, Jonathan: and all Andrea: of that. Yes. And the latest RVA23 chips are bringing that level of performance. The SpaceMeet K3 is just the first one that has very good performance. It also running local LLM models. But there are so many other RVA23 silicon that, that are becoming available. This is the year of the RISC-V application processor, really. Jonathan: Maybe not the year of RISC-V desktops and laptops for everyone quite yet, but it ... do you ... W- when you allow yourself to dream, do you think about that one of these days Intel and AMD is gonna have to compete with RISC-V? Is that a future that you see? Andrea: Our goal is to ensure that RISC-V is the first or default choice for new designs Jonathan: That's fair Andrea: the default ISA for new designs. Jonathan: And that, that encompasses a lot. That is actually quite the quite the aspiration. Andrea: Yes. Jonathan: But on the other hand, so many of these devices have a RISC-V core in there. I suppose one might say that you're pretty close to that. Andrea: Thank you. Jonathan: That's fun. So what about ... And I touched on this briefly, the idea of developers getting ahold of RISC-V. Andrea: Yes. Jonathan: What does the sort of the ecosystem look like for the tool chain, the compiler, the developers, and I'm sure this is something that you guys have worked on quite a bit, is to- Yes to grow this ecosystem in the community. What does that look like? Andrea: Yes. We are focusing on the developers from multiple angles. On one side, we have been purchasing and shipping the latest development boards over the last few years. In 2024, we shipped more than 200 boards to key developers, including tool chain operating systems. In 2026 in collaboration with RISE, the RISC-V Software Ecosystem Project, we are setting up a build farm, a RISC-V based build farm with native GitHub runners in collaboration with the open source lab at the Oregon State University. We also have the first RISC-V based commercial cloud service provided by Scaleway in Europe so that's one, one axe is, one axis is hardware availability. The second one, as you mentioned, is the tool chain work. So we we are in close collaboration with the tool chain projects not only GCC and LLVM but of course, we also have many members who provide their own tool chains like Green Hills, IAR, TASKING, embikasm, they're all working on tool chains. So the point is that, for example, when we release the RVA... We ratified the RVA23 profile specification. Patches to add support for RVA23 in GCC and LLVM were already being reviewed by the maintainers. And then Rise is collecting funds to direct them to optimization work. So Rise has been funding work to optimize support for Python, LLVM, Go, and also projects like PyTorch and Iri or Llama CPP. We have been investing together with Rise to add RISC-V as an official ISA in the Yocto Project. In just f- we, we upgraded our membership to the top tier the platinum level in May 2025, and just five months later, RISC-V was the the third officially supported ISA. Today, Yocto supports x86 ARM, and RISC-V- as the three official architectures. So it means that for the developers, Yocto is not only an embedded Linux and a set of tools to optimize and build your embedded Linux distribution, but it also means that you have an LTS support for the file system that is using the kernel LTS, and most importantly, it's all using upstream Linux support. And this was possible thanks to the collaboration RISC-V International and Rise together. And it's a major achievement for us. And the third axe, so I mentioned hardware availability, the operating system, and tool chains, and the third one is the trainings and the developer experience. So continuously adding new training courses to our free catalog on our website. I mentioned the designer RISC-V CPU core, my favorite one, but we continuously add new trainings for real-time operating systems, for porting software from other ISA to RISC-V, or optimizing software for the RISC-V vector extensions Jonathan: Yeah, interesting. The the training that you guys provide, is that freely accessible? Can someone just- Andrea: Yes ... Jonathan: jump on the- Andrea: Just go on the riscv.org or RISC-V.org as you would spell it or type it. And then you have the community, and you have the resources and the full catalog over free online trainings. Jonathan: Wow. It sounds like that would actually be a, if someone was, say, in high school or starting college and really wants to break into the sort of MCU embedded environment- Yes that would be a really good place to get started. Yes. Put that on your resume that you've worked with RISC-V. Andrea: And we... we also have the mentorship program every year where we select multiple students and mentor them on… And they deliver real results. They do real work. Jonathan: Yeah, absolutely. What sorts of what sorts of work, what, where do they get em- embedded at when, in that internship program? What kind of work can someone expect to do? Andrea: It varies a lot. Some worked on enabling kernel CI on RISC-V and kernel testing. Others are working on tools that automate documentation and consolidate the, or get better organize the documentation for the various extensions, and generate programmers reference manual and so on. So it varies a lot. Jonathan: Yeah. Interesting. What what are some weird places or weird things that people have done with RISC-V that, that you're aware of? I feel like everyone that's in the tech industry has a story of some sort. I built this tool and someone used it for something completely different than what I intended it for. There, there's gotta be some oddball RISC-V use cases out there. Andrea: Oh, we in one of the last summits, we were offered one of those interactive teddy bears. That runs on RISC-V. That's pretty good. Yeah. That's pretty good. And more is the professional one is the humanoid that ran the marathon in China- Okay ... a couple of months ago. That was running on the Space Mid K3 chip. Oh, that's cool. RISC-V in space. RISC-V was on one of the last moon lunar lunar landers. Okay. And then today I was reading somebody designed a GPU using 8,000 RISC-V microcontrollers interconnected. Jonathan: Really? Andrea: Yes. Jonathan: That's impressive. Massively parallel indeed. Andrea: Oh, yes. Jonathan: A- and so l- what what do you see as you rub your crystal ball and think about what comes next for for RISC-V? What are some things that you're excited about for the future? What do you see coming down the pike? Andrea: In short term, all the RBA 23 application processors. SiFive, Andes, Tenstorrent, Akeana, Alibaba BOSC, Nuclei, Spacemi. There are so many that are coming up. At NextSilicon EPiC SEMI. I think at the last RISC-V Summit a few weeks ago, we announced more than 10 RISC-V application processors. Many of them have server-grade performance. So in short term, this is super exciting. Then ratification of the matrix extensions is something that is very important. The completing the work with ISO IEC I really want to get through and make sure that RISC-V, the RISC-V ISO manuals can be properly evaluated as a possible ISO standard. And the other thing I really want to continue investing in the idea of the developer workshops. W- this is something we started at the last RISC-V Summit North America in October twen- October last year, and then Summit Europe a few weeks ago. So at every RISC-V Summit, we're now dedicating one full day to hands-on labs. And just a couple of weeks ago the fee for this lab, the cost was, the price was 30 euro. So it's really just to ensure no-show, that's all. And we had a hardware track, a software track. We had labs running an open source RISC-V design where the authors broke the system very long. So while running some assembly test, suddenly you got an exception. And the developers had to find where in the system very long it was broken. That's alwa- Can you imagine that? That's always fun, to break your tooling. Can you imagine that? Yes. Lauterbach had something similar. They enabled 24 debug sessions, debug tools in parallel, independent. So we had multiple developers on each station, so 100 people. And Lauterbach demonstrated how you can do advanced trace and debugging on RISC-V. And then they ran a challenge. There was a software running with some memory leakage or some bug in it. And these 24 teams had to find it and fix it as quickly as possible. We had workshops on running ExecuTorch or Edge Impulse AI inference on RISC-V or running trusted execu- execution environments and mitigating risks from the threat models. So all these were hands-on labs. And this is something that we will continue investing, and just sharing ideas to organize the next workshop is super exciting. It's so much fun. So much fun when you see what can be done. Jonathan: Yeah. Have ... Some of my listeners hate it when I ask about this, but it is the world we live in. Ha- have the flavor and the way those workshops work, has that changed radically because of LLMs and AI in the last, say, six months to a year? Andrea: No. No, not yet. Oh. Not yet. Jonathan: Interesting. Andrea: Maybe in the future. Not yet. Jonathan: Oh. Andrea: Oh, one thing, yes. Do you- I think it may be used to generate more design verification tests from our members. But for these workshops we are really running them with a human brain. We want to challenge the developers. That's why they are joining these workshops. They want to learn hands-on. They don't want to just prompt the AI to do it for them. Correct. And I think it's super exciting. At times I would like to go back being a developer and being part of these workshops. Jonathan: Yeah. That, there, there's only so many things that we get to travel around and go to, and yeah. It's one of the downsides of being the CEO. You don't get to, you don't get to be as hands-on with some of the things that you really enjoy. I get that. I too am CEO of a company d- a small company, but still I have to balance those things too. I'm lucky that I get to write some code still, but I see a, maybe a time coming before too much longer that'll come to an end, and- Andrea: A- and that's why I enjoyed so much the interview process with Calista two years ago, when she interviewed me, and then the board interviewed me. And the founders of RISC-V. 'Cause that gave me the opportunity of studying all the ISA manuals, and even I had fun in taking some code in C and write my own RISC-V assembly. And I wa- I was good at arm assembly in the past. So I took this C code, I wrote it in arm assembly, then in RISC-V assembly, and I appreciated the differences. And then I took that course on designing a RISC-V core from the website. So both the software side of a RISC-V ISA and the hardware side, and I really enjoyed that, You- ... that interview process. Jonathan: You really understand RISC-V well. I don't know that I've ever talked to a CEO, somebody at the head of a company, that has the technical understanding of what he's working with that you do. That is quite impressive. Andrea: I better understand the rest then. Jonathan: Yeah, sometimes that is the challenge. All so I... Boy we've gone through a lot. Is there anything that we didn't cover that we should have? Is there anything that I didn't ask you about that you wanted to make sure and mention to folks? Andrea: The developer workshops and the laptop were really the things that I thought were maybe not on, on your mind, so I mentioned them. Jonathan: I must say, I have the I have the Deep Computing store page pulled up. Andrea: Oh, okay. Jonathan: I have Framework laptops. I really like the Framework laptops. And yes this particular main board looks very interesting. Don't tell my wife, I may be about to spend more money. The really funny thing about that is that she watches the show, so she's probably out there, muttering at me. Oh, so she knows already. She already knows. She's checking Andrea: your credit card statement now. Jonathan: Yes, of course. Don't look too close. So what, W- y- RISC-V is going to space. Do you foresee a big uptick there? I, so I went and looked f- for one thing. Yeah. I was interested. The the Ingenuity rover the helicopter running Linux on Mars. I, I thought "Was that RISC-V?" No, that one's running ARM. But, Yes Maybe the next helicopter will be RISC-V, right? Andrea: Oh, RISC-V was in the lunar lander last year And the interesting part is I attended a RISC-V in Space workshop organized by the European Space Agency and Geissler in 2025 last year. There were maybe 200 speakers in the room, 200 projects based on RISC-V. It was unbelievable. And we had another RISC-V workshop at the last summit just a few weeks ago. And we had representatives from the European Space Agency, from NASA, from Microchip, from Geissler E4 Computing, and they were all explaining why they need RISC-V, why they need an open standard, why Geissler moved from Spark to RISC-V now. Why they chose Spark in the first place, and now RISC-V. It's, it is so fascinating. Yeah. So fascinating. Jonathan: I I did have a thought earlier, and it just comes back to me. Has RISC-V had any problems with things like export controls? Andrea: No. Jonathan: I- is this because you're intentionally a European company, or is it because- Andrea: No everything is open? No. Jonathan: Or how have you- Because we- How have you avoided that? Andrea: Because we are extremely scrupulous and respectful of all the regulations worldwide. And because we are a standard. We are an industry standard. We do not produce any implementation, any reference implementation, any artifact that would be a technology that gets exported. We don't do that. We just do PDF. Remember? Remember? We just do PDF, like USB. Jonathan: I, I think this is going to be the show title, We Do PDF. We Andrea: Do PDF. Why not? No I'm s- I'm serious. We are laughing together, but I'm serious. Doing in- international open standards means that it's a standard. And the implementation, the technology gets developed locally where the members are. You can design a USB mouse or USB pen drive in Silicon Valley or in India or in Europe or in Brazil. And that will be a local design. But the standard is global, and the same is true for RISC-V. Yeah. No, that certainly makes sense. That- that's why this recognition from ISO/IEC is so important to us, that we got accepted as a PAS submitter, as a standardization body. It's very important that we are officially recognized as a standardization body. Jonathan: Have, has there been any has there been any patent problems with RISC-V over the years? Andrea: No, we don't do any patents. Jonathan: But I g- I guess the way that I would imagine that working is, you get an email from, and I'm just ... I'll pick one at random. You get an email from Spark, says, "Oh your ISA violates one of our software patents." Andrea: No. Jonathan: Nothing like that? Oh, lucky you guys. I think it probably helps that it's based on something that came out of academia back in the '80s. And so a lot of those things that someone would say, you could just point at and look, this guy came up with this in 1982. The patent on that's already expired." Yes. "Go away." Andrea: Actually, we are the fifth generation of the original one- That inspired all the other RISC-V designs. Jonathan: Yeah. That helps. That helps a lot, being, being the original. We did it first. Oh, that's great. All right. I do have a couple of final questions that I'm required to ask everybody, and I get in trouble if I don't. I get emails if I forget these. You personally, what's your favorite text editor and scripting language? Andrea: My favorite text editor was Zap Jonathan: Okay Andrea: That was a text editor that was, I was using when I was programming on an Acorn Archimedes. That was one of the first machines based on the ARM3 processor- ... and the RISC OS operating system. And at that time, I used the Zap editor. I loved it. I loved it. Jonathan: That's a throwback. Andrea: Yes. What do Jonathan: you use these days? Surely you don't run Zap in an emulator. Andrea: No. No, these days I just use a simple text editor available on on my laptop. Jonathan: Wh- which one, though? Do you go for VI or Nano or ... Andrea: ... w- Ah, let me check. No, the name is just Text Editor. Yes, Text Editor. Jonathan: Okay. Andrea: As simple as it is. Jonathan: As simple as it gets. Yes. A- and scripting language, if you have to, if you have to write some script code? Andrea: I was using the ... The one I used the most in the early days was the the MS-DOS Batch Jonathan: Yep, yep. I've have some experience with that from a long time ago. Andrea: I'm sorry i- I have some gray hair. Jonathan: Oh, don't look too close at me. I, I've- I- ... I am gaining some as well. I- It Andrea: may not... it may not sound that fascinating, Zap or Batch. Jonathan: I got my start programming with Microsoft QBasic. I'm old enough to say that. But I've have not done anything with the Acorn, with the Archimedes and or any of that, so I fit somewhere in there. Okay. All right. Andrea, it has been a blast to get to talk to you. Thank you so much for coming, and, Oh, my pleasure ... I'm glad we got to- Thank you so much Andrea: for inviting me. Thank Jonathan: you so much. Yeah, glad we got to talk about it. Really enjoyed it. Yeah, I did as well. Thank you so much, sir. Andrea: Thank you. Jonathan: All right. That was Andrea Gallo talking about RISC-V and RISC-V International, and boy, had a great conversation there. We've got some fun shows coming up. Next week we're talking with Neriman Jelva about Pewter. That is the desktop operating system that runs in your browser. I have questions about that. And then the week after that we're talking again with Michael Meeks of Collabora. And then the week after that we have a returning guest, Francois Proulx, about Smoked Meat, and that's all about security in your CI, like in places like GitHub. Very important. We do have some openings after that, so if you or your project you think should be on the show, you should let us know. Drop me an email. It's floss@hackaday.com, and we'll get you scheduled up. Other than that, just wanna say thank you to everybody that's here, whether you watch or listen live or on the download, and we'll be back next week on Floss Weekly.
• 
  Episode 874 - Really, We Do PDFs 08.07.2026 1h

  This week Jonathan chats with Andrea Gallo about RISC-V! What does it mean for RISC-V to be an Open ISA? Where is RISC-V popping up, and what's the new frontier?  Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 873 transcript 01.07.2026

  FLOSS-873 Jonathan: This week, Aaron and Andy join me to talk about QNX. It's the other operating system that runs in a whole bunch of different places. It really has safety and real-time nailed down. There's just one little problem. This is FLOSS Weekly, episode 873, recorded Tuesday, June the 30th. Wait, that's not open source It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we're talking about a Unix or a Unix derivative or a Unix-like. I'm not sure which of those terms is technically correct. We're talking about QNX. It runs a whole lot of places. It used to run on your BlackBerry, and it runs a lot of places that you don't think about. Maybe not on your toaster, but maybe on your car. Anyway, we've got we've got Andy Green and we've got Aaron Bassett to talk about QNX, where the company is, where they're going, what changes are there, and why, maybe you should think about doing some development on it. Let's go ahead and bring them on. Guys, welcome. Welcome to the show. It is great to have you both here. Andy: Thanks so much. Really appreciate it. Jonathan: Thanks for having us. Yeah. And now, we gotta start with this. We have to start with this, right? So this is the show about open source, Floss Weekly, free, libre, open source software. We're not... We don't do dentistry here. We do open source stuff. QNX, at least like the core of it, the kernel of it it's closed source. And what are we even doing here, guys? Get get out. What's Andy: the conversation? Yeah. Why are we on this show? Jonathan: Why does this Andy: make sense? Yeah, no that, that's very fair. So I, we have had a kind of a really interesting and long evolving story with open source, and I think for some of your audience who may remember, we did go fully open source at one point in our history. And and that was like right around the time when we were acquired by Research In Motion for use in the BlackBerry, and they were thinking, "Hey this stuff is special sauce. We really don't want this to be, like-" "... our... one of our defining characteristics. We don't want to be fully out there because we wanna actually have some of that proprietary goodness for ourselves, for our own phones." So we clawed that back. And so that that is something that I think, a lot of us in QNX were not necessarily... we understood the business rationale for that and all that. But I think that there was always this feeling that, there's a lot of good and value that we get with working in the open source community and with the open source community, and we wanted to bring some of that back. And so I think relatively recently we've had this, So I guess QNX Everywhere has been a program that, that both Aaron and I are part of both sides of the, on the technical side and on the s- sort of more on the business side. And that's been around for about a year and a half. And that's been our start back to recognizing, l- look we need to be more involved with the community. We need to be more transparent. We need to open ourselves up a little bit more. So I'm not gonna make a commitment and say, "Hey, yeah, we're going all open source," or anything like that. We're not gonna, make that same mistake of making commitments and then pulling back on them. But what we are doing is we are trying to open up a lot of the pieces that we see don't really have impact on our business. Because I'll be frank, there's a few different things that are factors as to where- whether or not we do things open source or not. And one of the big ones is that a majority of our business is based on regulated industries. Things that either have functional safety components or cybersecurity components or things like that. And in a lot of cases there's this thing called SOUP, which is software of unknown providence that is an acronym that's used within those sort of, certification bodies or regulatory communities. And having open source software in a lot of cases makes things extremely difficult to pass through certification. Not because of the testing or anything like that, it's because you can't really guarantee where that software has come from, or you have a very difficult time tracing the line through of all the edits that have been made to it- from different contributing parties. So we have to be super careful and super cautious about things that are gonna impact our business. So we don't we're not gonna go and say, "Oh, yeah, hey, you're, we're gonna open source the kernel," because then anybody can contribute to it, and then we turn around and find out, oh, somebody put this really super clever hack in that- wasn't really intended, all that kind of stuff, right? But to that extent, though, we are doing things like, making sure that we're open sourcing all of our board support packages, open sourcing our development, our driver development kits- ... open sourcing pieces of our software stack that actually originally may have come from open source and we've made changes to it or things that we don't think are, impacting, functional safety or any of those kind of things. So keep your eye on that space because that's the one aspect and, but the other aspect is us working with the community because we get a lot of value out of working with open source and making sure that, like open source software runs on our platform or that we can use it or deploy it in, for either hobbyists or academia or, companies that are doing prototyping or proof of concept or R&D or any of that kind of stuff, right? So there's a lot of people that are playing with different open source things, and we wanna make sure that all those people are enabled. So there's like a lot of things that are going on within the company right now around open source, and we think we're good open source supporters, and we do recognize that, yes, we haven't thrown the kimono open for everything, but- but we're doing that in an intelligent way, right? Jonathan: Yeah. I do wanna touch on one, one thing that you mentioned, and this is something that we've seen several projects make moves in this space. But the idea of when you're open source, that means that you accept pull requests. And that is-- we've seen several projects come to the point where they say, "Look, we're open source. All the source is out there, but we do not accept pull requests," or, "We only accept pull requests from people that are already in the project." And, I think AI is leading to some of that. In some cases it's because of just very high code standard quality, code quality standards. I think in other cases, though, it's because of that legal question. And so there's a relatively famous blog post from DHH that talks about this, and it's like open source is a license. It's not a community management style. And I just think that's interesting in this particular context to think about that. And it goes all the way back to this idea of the cathedral and the bazaar, right? From ESR way back in the day. And you can do things that are free software and open and still, the monks in the cathedral are the ones that are slowly working away on it, and it's not just a bazaar where everybody can push code in. Andy: No, I agree. And I, and, and certainly I've seen things where it's yeah, but people are gonna have to stop taking software because of all the AI slop that's being generated and I think, our approach to that might be, like, even more conservative than- ... is strictly necessary. But I think that we're approaching this as a, we, we learned a lesson from what we've done before. We don't wanna actually, go a step far and then take a step back, right? So- ... so we're gonna do that i- in an intelligent way. I, the CEO of our company John Wall, has made a lot of comments about, how he feels that we, we should, open source as much as we humanly can. But there are gonna be, w- we have to still do that with all of the processes that we adhere to, with all of the regulations that we adhere to, all the rest of that stuff. Honestly, yes, we could immediately pivot to saying, "Oh, yeah we're gonna just do an open source methodology and modality to manage all that stuff." But kind of it's, I'll just say it's the path of least resistance to not do that- ... and to just say, "Okay, for those things that actually have critical components that we're j- we're gonna manage them the way that we always have," because we have processes that we know work that have been vetted. We've actually been audited on all that stuff. We don't have to re-audit everything and re- like reinvent the world when it comes to that, Jonathan: yeah. I, th- there's a lot more to talk about with this, but I do wanna bring Aaron in on the conversation. And you are- Absolutely ... you're part of the open source team, Aaron. What, so what are the bits of, in the company that, that you work on specifically? What's the open source story there? Aaron: Yeah, so primarily for myself, I work on the QNX developer desktop. So that is the self-hosted developer system that we've been making to allow Basically anyone who wants to use QNX, whether it's hobbyists, students, to get easier access to QNX. Primarily it's been cross-compiled, embedded system, right? So you've done Arduino development that, you cross-compile your thing, you copy it there, you flash a board, you go there. A lot of people don't m- know how to do that or it's something they've never learned, so it's really hard to get into, especially at hackathons. 48 hours, let's learn how to cross-compile and flash- ... an SD card and everything, right? That's something you just don't do. But part of that is, is I maintain the package manager that we have for that. How many... couple hundred or thousands of packages that we have now that are in there that we've built and distributed across that. So we touch a lot of stuff. Jonathan: Have you guys found that that AI is making a big change with the way that people interact with this whole system? Andy: Yes. Yeah. Yeah. Yeah, so I started this is s- cr- this job like just about a year ago. And when I first started doing it... So I've been in the industry a long time, but I came back to QNX after running my own company for a while. But in any case when I first did it, we were doing, we'd go to a hackathon, we'd be seeing somebody's project, and they'd be like I guess we used AI to do part of this," right? And hackathons now, it's "Yeah, no, we're 100% full in A- AI. We're- we- there's no bones about it. We're not being bashful about it. Like-" "... we just had Cloud Code do the whole damn thing." And it's like in the course of a year, like that's the perception- ... change that we've had, at least, from students, and maybe they're on the leading edge of that kind of thing. Sure. But I don't think they're the only ones. Jonathan: Yeah. So it, this is obvious. This is one of those observations that I made that's obvious in retrospect, and maybe I was late to this game, but open source makes AI work. And if you're if all of the things that you're doing is closed and not documented publicly, then AI is not gonna be able to do much with it. But when you put it all out there- ... on the internet, and it's part of the training data, then AI knows exactly how to work with your stuff. I- is that s- s- part of the consideration that QNX makes with, like, how much of this to publish and how much to make open? Aaron: Andy, I have an interesting observation on that. So- Andy: Yeah, Aaron: please go for Andy: it. Aaron: Yeah, so I have been playing on on my own time, on my own hardware what does AI on QNX look like? 'Cause, again, you go to these hackathons, and we actually were just at an AI hackathon- ... at Berkeley, and very surprisingly, there's next to no code that knows how to write a QNX Resource Manager, which is how you do devices and drivers. And I was able to, in an afternoon, ask it, "Hey, let's make a temp- like a memory file system," and it spit out some of like perfect QNX-isms, I'll call them. It knew exactly APIs I was able to use, how to do it, found the stuff that you're missing. But I was like what, like where did it learn this all?" There's no code out there that it's open source, like lots of it that doesn't do this. And it may just, it's just all from our documentation. We have really detailed technical documentation that's all public and free. And just by scraping that, it's actually able to do- to do things like that. I saw a lot of that at the AI hackathon. It was able to spit out things that looked like a professional QX, someone who's in QX development for years would write. Andy: Yeah. Yeah. And actually that so that's a really good point and I'll come back to, I think, your opening comment, Jonathan, that led us into this, which was I don't know if it's Unix or this or that or whatever." Yeah. And technically we're none of those things, but we are POSIX. So we're POSIX compliant- ... meaning that we use the same APIs essentially that Unix and Linux do for the most part. There's a little bit of stuff at the f- three signal layer that we differ for and Linux does and Unix does. But for the most part, that means that a lot of stuff runs very easily or without, minor stuff. And when you start playing with the hardware or dealing with, some of the minutiae of device drivers and stuff like that, yes, then you can start running into some things where we're different because of the nature of a microkernel or because of the nature of real time or whatever it is. But a lot of stuff will just work out of the box, and that's one of the really interesting things that, that we've been, like leveraging in terms of a lot of work that, that Aaron and team have been doing to, create all those things that are open source software but that runs on QNX. Jonathan: What, what does that process look like? So if someone has, let's just say, an open source project oh, I don't even know what a good one would be. Nano. All right? Text editor Nano. They, I wanna compile Nano for this QNX board. What does that look like? Can we use GCC? Does CMake work under QNX? How alien is the landscape for someone that's used to like Linux tools? Aaron: I am confident enough to say that if I gave you an SSH connection to my RPi that has QNX on it, if you didn't run uname, you probably would not be able to tell the difference. I'll get rid of the uname thing, 'cause you're probably like, "Oh, uname, hey, oh, it's a QNX system." It looks and feels like a Unix system. I can say the same thing about FreeBSD nowadays. You do SSH to FreeBSD, it, it feels and looks like everything you're used to. And that's one of the things that we really strive to with QNX Everywhere, is make the development system as close to as possible as FreeBSD or Linux, right? So for QNX Everywhere, we have the, we use the Clang compiler for self-hosted development. Obviously for cross compilation, QNX uses GCC with our QCC wrapper, which is, has all the safety stuff for it. But yeah, you just have, you have Clang, you can run Make. We have Mason. Actually just recently, about a couple months ago, you can run oh, what's the Java one called? Bazel. Oh. You can run Bazel on it, right? So we can... You can run everything on it. It's very familiar. Yeah. So- So but go back to your thing for Nano though. Nine times... you grab Git, you do your Git clone or grab it for release tarball. Start off with your standard configuration just like how you'd do it on Linux, and then start following the errors. Most of the time it's just telling it, "Hey, QNX, we are a system, we are a Unix system. Please respect us." But past that, all our headers look the same. There may be the odd header that you have to like, oops, it's not at sys something, it's just in the regular location. Most of the time you're not massively touching the code base to fundamentally rewrite it, 'cause like Andy said, it's POSIX compliant. Yeah. Nano is a very simple example obviously, 'cause, Nano's been around for a long time. I'm fairly certain it's already been ported to QNX, at least QNX 6.0 and probably QNX 4.0. I've found lots of remnants of QNX 4.0 and QNX 6.0- ... throughout a lot of old software Net- the Netscape libraries that power the web. You'll find QNX 4.0 patches left in there. It's like, "Oh, wow, that's a history lesson for you." Jonathan: It... And so you mentioned a package manager. Is there a package manager built into QNX that probably already has some of this stuff where you can just, you know, It's obviously it's not apt, it's not DNF, but, can I just install Nano if I want it on a QNX system? Aaron: On QNX Everywhere, yeah, 100% that's what we started with, funny enough. We started with a package manager before we went self-hosted like compiling on QNX. We actually use Alpine, so if Alpine Linux, APK tools, their package manager. If you've done any Docker environments on Linux, it's like the thing you use to make- Yes Docker environments a lot smaller. Standard. Yep. We just ported APK tools. We use their able package package builder system, so you know, if you've grabbed an Alpine Linux system before, it looks the exact same. It's the same tool. Andy: So that is one place where I do, I wanna just i- insert my, my kinda legal tiny print quotes. Yeah. Aaron: The Andy: legal version. Because this, that's all true on QNX Everywhere. QNX Everywhere is like our non-commercial version that anybody can download. But the stuff that we have for our commercial version, which is our SDP8 we don't have a package manager yet. So there's a lot of things that we're able to do I would say on the bleeding edge- that's in QNX Everywhere because it doesn't have to go through all the safety certs, because it doesn't have to, meet, like cybersecurity process, blah, blah, blah. So there's a lot of things that we can do there that takes us a while to do, and, in terms of moving those things over to product haven't been done yet. So the package manager doesn't exist in a commercial product. So if you're actually doing the free version, there's a l- few things that you can do that you can't do on the other side. But that's just the tiny print. That's- Jonathan: Yeah, and that's interesting. I think that's something to dig into here. I went to the Ubuntu Summit recently- ... and there was a representative from Nvidia, actually, that was there, and was talking about their plan to try to make, in this case it was the Linux kernel actually compliant with some of the safety regulations. And safety with a capital S, you could say. ... Because there's, there's regulations and there's tooling around how this works. I- I assume because of the places that QNX runs, that's already something that's baked into it from the beginning. But what does that process look like both in the production versions, but also in trying to pull more of this bleeding edge and more of the open source world in to QNX? Andy: Yeah. Yeah, that's a really good question. Like N- NVIDIA's a good example. Like we, ... NVIDIA has, I don't know, probably almost as many QNX developers as we have maybe. Because they, they do all of their own BSPs. So normally for a board support package, we would do the board support package. NVIDIA doesn't ... So they're like, "No. No, we're doing all of our own." It'll ... And so they're very proprietary about their IP and who touches it and who knows what. Yeah. But as a result of that and because they're in all the automotive applications and stuff like that, we're on a bunch of their hardware. Not all of their hardware- ... but we're on a lot of their hardware. And I think that they, like we're like their known partner for most of those sorts of engagements. I think that they probably want to expand that as much as possible to add Linux into the portfolio and things. I know that there was a big exercise done by, some car makers and others to try to basically do a safety certified version of Linux, and they spent a lot of year like two years and a ton of money on- ... doing that. And I think they eventually came to the conclusion that they, it really, that they didn't wanna do it. So what's necessary to do that? Honestly, I don't know. I haven't lived through that exercise. And to be perfectly honest despite the fact that I work with the colleagues that do that stuff is so incredibly boring. I like, I ... As soon as you start talking about "Oh, the safety this and that," like my brain just tunes it all out. So I really can't like- You know, knowledgeably speak about what is required to do it. I just know it's difficult and we've already done it. So there you go. Jonathan: Yeah, ... QNX is a real-time operating system, I think. Andy: Yes. Jonathan: Yeah. Yeah. That, that's- That's right ... that's one of those, that's one of those boxes you have to check to be able to be in certain places the actual drivetrain loop of a car or some of those things. Yeah, interesting. Andy: Yeah, it... it's funny though, but real time is a very flexible kind of a thing. For example you can use Linux with preempt RT patches and get a certain level of real time. But it depends on what it is that you need to do. What real time means f- for you, right? I think for a lot of our stuff that needs super, super critical real time we have customers that use us because, they need to be able to respond to an interrupt within 10 microseconds or less, and we can give them that. But that's super, super unusual. And in fact most things that I think you'd call real time, as long as you respond within a couple milliseconds, it's probably fine. And, Linux in the right environments can do that- ... most of the time. The problem is when the unusual cases happen, how do you manage that and things like that? Yeah. I think the bigger thing that we contribute is the reliability aspect because we are a microkernel and because there's very little trusted code that runs in kernel mode, and pretty much all of the rest of the system runs in user mode. So all the device drivers, file system stacks, USB stacks, networking all of the other components that you might consider kernel mode drivers like in Linux, they all run as applications for us. And so that way if they crash, doesn't bring down the system, right? So that tends to be something that has a very strong need across a lot of domains as opposed to real time, which is, I would say a lot of things in medical and robotics need real time, but y- and cars I guess too. Yeah, cars too, de- depending on what it is. But a lot of the car stuff doesn't need it, right? Jonathan: Yeah, and that's something else that's interesting. U- QNX runs on the head unit, but also maybe the engine management unit, and also maybe on a self-driving car in the loop to do the driving, right? You've got- ... you've got a whole bunch of different places and different scenarios where you can run and with different requirements on each of those. Andy: Yeah. I would say honestly though, we're not usually in the drive loop stuff because those are sometimes just microcontrollers that are bare metal. They have no OS, no nothing. They've been fine-tuned by the, the OEMs for a long time, and people just don't wanna really touch them, right? So y- like when it's that critical a thing, you tend to not wanna play with it a lot. But to your point though, we're in a lot of different applications in the car, like domain or zonal controllers tends to be like the new thing where you're trying to like do ECU consolidation by taking a whole bunch of different modules in the car and saying, "Hey, look, we've got, 20 different 16-bit or 32-bit micros here. Why don't we just put it all into one beefy 64-bit micro and just have it do everything that's there? Throw in a couple hypervisors or multiple different things, and we can manage all that stuff." And I th- that's the way that a lot of the automotive industry is going, and that sort of is a continuation of that is, more consolidation. It's leading to autonomous cars and all the rest, right? So yes, you're right. There, there's a lot of different places that we can play. Jonathan: Yeah. So I think you've probably... L- let me put it this way. I know the answer to this based on some things you're saying. But I think it's an interesting question all the same. Can you do crazy things like take the QNX kernel and run a Linux user space underneath? Or do it the other way around, take your QNX user space and run it with a Linux or a BSD kernel? Andy: I know we can run things on top of QNX because we have a hypervisor, and we do that all the time. I don't know about the other way around. That might be an Aaron question as to whether that's technically possible or not. I, Aaron: I guess it depends on what do you mean by user space? That's an interesting question. What's, what is Linux user space on QNX to you? Jonathan: So I've ... I am thinking of user space being everything but the kernel, and I think the answer to that technically is no, because it's such a different kernel architecture, being a microkernel. But I think it's probably interest- So maybe let's break this down into sub-questions. So what does the QNX init system look like? Can you run system D with... I don't know why you would Aaron: want to, but- Oh, okay. Yeah. So yeah, that's an interesting question. So based off that, so system D, no, 'cause that is just it needs C groups and everything. But that being said we're actually working on OpenRC right now for the QNX Everywhere system anyway, right? Yeah. We're using OpenRC 'cause it's just more around. We do have other init systems for the productized one it's called SLM, which is like a similar idea to OpenRC and all that. It serves the same purpose, but and things like that yeah, you can usually just get any anything that is running Linux that is not using specific Linux sys call. So if you pick something and you say, "Does this run on Linux? Does this run on FreeBSD? Maybe macOS 'cause they're all Unix systems," then yeah, it should just more than likely run on QNX, unless there's a very specific thing it needs implemented like C groups. Yeah, we're not gonna support C groups anytime soon. But- Jonathan: i- is Aaron: there Jonathan: been some work done to emulate those things that you don't actually support? Aaron: Yeah. Epoll is a great example. We actually have, ... So let's think. Epoll, timerfd, signalfd. There's one more I'm forgetting, but those sort of family of things. We actually have user space implementations of those. It will run actually as part of your application. It will spit up another thread that actually handles all the epoll stuff for your program. Nice. But yeah, we do try to actually have compatibility layers. That was actually really important when I did when I ported Java. Trying to write a, the, like an epoll style, like just doing like poll and select in Java would've been super slow, and I was able to just utilize our epoll user space implementation to accelerate the porting of it, 'cause it's good enough. Yeah. Andy: Interesting. One of the- Yeah. One of the things- Kind of to your- Go ahead. Oh, sorry. Go ahead, Jonathan. No, I was just gonna- I was just gonna say d- to that question that you were asking about like what's the boot process like. It's probably worth spending just a second or two to explain how QNX does it, which ... 'Cause it's very different from what a Linux or Windows world is like. And so basically when we start up, the only thing that's really running is the kernel. And you, as the system architect, and because we're an embedded system, there, there's always this concept that there is going to be somebody who's like planning out what the system does. So we don't really have a, out-of-the-box desktop model, other than some of the, QDD stuff that, that Aaron works on. But in order to do that, like you start with absolutely nothing. So if you want a disk driver, like you wanna actually see files on a disk, you start up a disk driver. If you wanna actually talk to a terminal, you start up like, like a serial port and, so there's ... You actually literally start every service that you want- ... in the order that you want them, and that's like one of the things that's used to get us super fast boot times too, right? So like- Sure ... in cases where hey the car starts, you don't wanna wait for three seconds while, like the whole thing is initializing. It's like you wanna actually start the car and go. And so that gives us, like the ability to get enough of the system up and running in, however many, 10, 15 milliseconds or whatever is required to do that. But because of that, it is super different from, yeah init D, R- RCA, all those kind of things, right? Jonathan: Yeah. Y- so Aaron mentioned something a minute ago, and that was his his Raspberry Pi. And I am- ... I am, I'm a Raspberry Pi geek. I've been a fan of- Nice ... of their stuff just as long as it's been around. I added up one time how many Raspberry Pis I had in my house and, lost track after a couple of dozen. What is the, what does the process look like to run QNX Everywhere on a Pi? How difficult is that? Aaron: I'm gonna say it's dead simple 'cause we just directly support it. That's the cheap answer. From a process, I guess probably look at what a process of supporting it would be. The nice thing is Andy kinda started off saying, "Oh, the kernel is the starter thing." There actually is one more thing above the kernel that is our, was the BSP. It's our startup code. Okay. So right above the ker- the kernel is not the thing that actually initial- initializes the CPU cores. It does- it doesn't actually do that level of initialization, the startup code does. So the startup code is the first thing that hits. So when you're flashing an RPi you have- we have a little syntax on how to say I want this startup code and then this kernel with these arguments. The startup code will run, do a- it'll actually set up the hardware. For instance, on the R- Raspberry Pi, it'll talk to the RP1 chip, initialize that, initialize the graphics core. And then from there it'll ... There's a bit of a proto- I'll call it a protocol for conversation. It does a protocol in memory effectively, and then it runs the kernel. The nice thing about that is the kernel you run on my Abund- oh, sorry, my, I say Abundu. My QEMU machine or this board or that board, it's all the exact same kernel. It's just the startup code that changes. Jonathan: Okay. Aaron: So actually our Raspberry Pi startup code is all open source under the Apache 2 license. Jonathan: Very cool. Aaron: Let's see if I can remember this off the top of my head. Github.com/qnx, and then it's BCM, the chip, Raspberry Pi. And they are actually the Raspberry Pi 4 as well. So all that startup code- ... is all open source which is an interesting reference point if you're like, "How do I boot up a Raspberry Pi from scratch and then have a ch- have a core that's initialized?" It's quite interesting to read. Andy: Yeah. I can give you the TLDR version is we have a quick start image for a Pi 4 and a Pi 5. So you can go on and download them and burn them onto a card and then boom, you boot it up. It's pretty simple. Jonathan: Now, the question: Do you have to register an account somewhere to be able to download those? Andy: Yes, you do have to register an account. Oh. Honest- I- so if y- oh my gosh. Y- you, ... So as the ecosystem director of the ecosystem kind of development and trying to get that thing off the ground and make sure that we could, have it a- available in the Raspberry Pi flasher, d- this and this, I'm like, "Look, guys, I don't want any addresses anywhere. I just wanna provide an image you can download." And they're like, "Okay, but w- how do we do export controls?" I'm like, "Oh, God, why are you ... legal guys are killing me on this." And, so yes, you still have to do an email address. I'm trying to get that as simple as possible. I really do wanna get it to just be able to do that, but, just make a, make up a Gmail thing, whatever. I don't care. It's fine. But you need one. Jonathan: Yeah. Export controls, that that's, ... And I get that. I have had to work a little bit with that over the years. They just ruin all the fun. Andy: Y- it, it does ruin all the fun. The, and then, it gets into all these conversations that are really oh, is it a mass market thing? It's if it's ... I'm like, oh, God, y- again, the lawyers start going on about all the details of everything. I'm like, "Okay, can we make this work? Yes? No. And can we make it as simple as possible? Yes or no? Yes? Okay, good. Let's do that." Jonathan: Yep. I- when someone does jump through the hoops, are there some limitations on what you're allowed to do with QNX Everywhere? Andy: Yes. You're not allowed to use it for high-risk applications. You basically take all of the liability of what you're gonna do. Jonathan: Oh, okay. Andy: You're also not allowed to use it for commercial applications. You can't build a Kinix thing and then start selling it, because that's a commercial application. Now, to be fair, we... That- that's the guideline that's in the QDL, the Kinix developer license- ... that we provide. When it comes to startups and proof of concept and, prototyping work and stuff like that, we know that those are, like, sometimes paid engagements. We know that you're building something and you're trying to actually sell a handful of them to see if there's any viability in it and all that stuff. So we really don't care about that. What we care about is look, you're building a product, you're selling millions of dollars worth of it, then that has to become a conversation- Yeah ... because then that engages with the business part of the organization. But if you're just using it to play around with then that's fine. Yeah. Yeah. Jonathan: Yeah, y- and again I'm gonna beat the open source drum because that's what I do. Andy: Yeah. Do Jonathan: it. Companies have solved this problem rather than saying no commercial, because that's, it- this is a thing that's been, like, a conversation in open source for a long time. And so you've got things like the business source license, which technically makes things source available and not open source. But one of the other solutions that I think is better is people use the the Affero public license, the AGPL, which is, that, that sort of closes the, "Oh, I'm running your open source code, but it's on my server, so I don't have to share my changes," right? And so I think there's a, there's an interesting approach to this where as a business you say, "Look, we're gonna open up all the code, but the license that we give it to everyone with is the AGPL, which means if you put it in your commercial project, you also have to release everything as open source. And if you want to put it in a commercial project that you have some closed bits in there, then you have to have a business conversation with us." And that's a, an- another way to go about it. It's an interesting approach. And for those of us that are the real open source enthusiasts that's the way we like to see it done. Andy: Yeah. No, that's fair. And I think, if you're coming at it certainly from the approach of y- you know everything open source and the infectious model of trying to encourage that ecosystem- ... I think that does make sense. Again, I think, we are approaching it and we're trying to actually make sure that, we're true to our word, that we don't you know- go out over our skis, so to speak. Maybe that's a Canadian analogy. I don't know if they use that term in Oklahoma very well. It translates okay. It translates okay. Yeah. Yeah. So like we're not gonna, we don't wanna kinda go all, all out like that then just realize, okay, that's like a terrible thing from a business model point of view- Sure because a lot of those things are done for companies that are fundamentally not the same sort of business that we are. I'm not gonna say that we couldn't ever get there. That's not where we're at right now. That's not what the current thinking is. Jonathan: Yeah. No. That's fair. It is interesting though. So the, one of the other sort of big open source projects that that you guys have worked with is is Eclipse, isn't it? Andy: Yeah. Jonathan: What's the- Yeah, we- ... what's the story with Eclipse? Why did so- so- ... why did someone at QNX say, "Let's give a lot of money to the Eclipse Foundation. Let's work on this source code"? Andy: I think we... So when we had... so we, we support Momentics IDE, which is our, development environment, and that's all based on Eclipse. And I think when we were trying to figure out what are we gonna do from, a software development standpoint?" That looked like really the best choice, and we decided that, th- that relationship was a good one to do. Now, that's not to say that we're not working with Eclipse on other stuff. So like- ... there's the various different automotive consortiums, and Eclipse has one of those, and we're working with them on that. But we also have seen like that Microsoft has taken over the IDE environment, with Mi- and so we're- Yeah, that's true ... we're embracing both worlds on that front. I think, like we're still big fans of Eclipse, to be quite honest, like our tooling on that side is a little long in the tooth. And we had a bunch of Eclipse experts who left the company to go start their own company, and so they started their own, UI framework company. And we were like, "Okay, we still have people to work on that, but like it just hasn't been as high of a priority. And so now it's okay, now there's a shiny new kid in town, so people are chasing that, so you know. I don't know. I don't know where that's gonna be. But yeah. Jonathan: Yeah. Aaron, do you do work on the Eclipse side of things? Aaron: No, I don't actually. I know quite a few people though that do that via some of our... They're not part of the open source team funny enough, but they're part of more of some of the services teams. UProtocol I believe is part of, under the Eclipse Foundation for automotive, and that's a big thing. I know we ha- we have a full-time dev who does internal stuff, but he's also like one of the lead maintainers of uProtocol. His name is escaping me off the top of my head, so I, ugh, apologize to him. But yeah, no, he's a... He works for us and he does a lot of stuff with uProtocol. So that's a proto- that's an automotive protocol that allows you kinda to, I don't wanna say it's like gRPC, but if you're trying to imagine in your head, it solves like the service discovery and protocol between how to like between cars and services and offline support and multiple languages. Jonathan: Yeah. Interesting what does it look like trying to build a community around QNX without the fully open source part? That's gotta be a, that's gotta be a challenge. You can't accept pull requests from your community, and that's one of the ... That's one of the best things about having a community. What- Andy: Yeah ... Jonathan: how does that work? Andy: I think it's probably important to think about what it is we offer and what the community means, right? So it's if you're building Linux like some Linux system, like how much of the code is actually stuff that you're changing in the kernel versus stuff that you're creating on a application side? And, so do you really need to be submitting pull requests to like, tinker with the kernel s- task scheduling- ... priorities and stuff like that? Probably not. Probably that's never gonna happen, right? There's been, thousands of people beating on it, and we have the same sort of dynamic, right? There are very few people who are interested or care about, the really gooey inside, details of how things get done. They just know, oh, hey if I use this software package on a QNX system, I can get this thing done. And yeah, there might be bugs in that and I have to get that fixed, or I want a feature request on that, but it's mostly at the application level or the library level. And that's, that stuff is all open, and that stuff is things that ride on top of our OS. So I think for the most part our interaction with building a community, if you can... I- if you're the kind of open source person that kind of says, "Oh, you know what? I'm not gonna touch it if it's not open source," we're probably not for you. If, like it's the same thing with Qt, I know Qt has had a on-again, off-again relationship there because, they have a commercial side, and it's yes, and they're a business just like us. Like- ... they have to make money they have to figure out a way to monetize what they're doing. But if you're gonna do stuff and you don't need to use that stuff and you can use their free stuff, then great. And we basically are in that same model. We just don't have a open source component to it. If you wanna use it and you wanna do your own projects with it, and you can deal with the fact that we're not, 100% morally within your open source framework, and you can do it for free though, then yeah, then it's actually cool and you can do all these kind of cool real-time things or, experiments with, your Raspberry Pi and, and- all of that kind of stuff. So yeah it's actually been okay. Jonathan: Yeah. Is there at least an avenue where someone can say, "Hey I've messed around with all of your open source stuff. I'm running into this bug. I think it's upstream into the stuff that I can't touch"? Can I go and report a bug- Oh, yeah. For sure ... to QNX as just a regular- 100% ... individual person? Andy: 100%. W- we would probably do that through our Discord. So we have a very active Discord in terms of, we have y- our chief architect is on there. He- I think he's addicted to Discord. He's on there- ... responding to people. People put on questions like, "Oh, how about blah, blah?" It'll be some simple question, and he'll come back with this crazy answer. I'm like, "Okay, Elad, do you... it's okay." So he gives all this detail and all this crazy stuff. So but we have a very active development community. And yeah if you had bugs there, we would track 'em down, submit internal JIRAs, get 'em fixed. Yep. Jonathan: Yeah. Very interesting. So I do to ask, and the question I normally ask is, what's the weirdest thing that you've seen somebody from the community do? And maybe that makes sense here as well. You, you've got a similar question in the rundown here, where they gave me some questions to ask. And it's the other way around. What's the weirdest thing that you guys have done? So maybe both of these are good questions. What's some, what's some- I don't know ... oddball projects? Andy: Aaron, you go first. Aaron: I, I have a couple, 'cause I'm quite free in the Discord. I've... People 'cause kinda going back to your question of building the community, there's a lot of people really enthusiasts who basically we have un- I'll say uncharted water. Maybe the, maybe not the right word, but there's a lot of things that if you wanted, like when somebody ported Kerberos to QNX 'cause they needed the libraries for something. Like- ... that's an interesting thing. How the heck did you get that to work? And then you'll... the software is there, it's just, like, how do you make it work on a new operating system? And people like that discovery, having something existing, working, and then porting it. I am gonna forget the guy's name, but there is some, this is not for QNX 8, but just someone who's having fun in the community. He took the Black, old BlackBerry Passport, so it's running QNX 6, and he's been doing op- he been doing open source development on it, like AI work and drivers. And it's really cool to see people take even some of the older hardware that, that was more, a little more open source. But taking some modern stuff and porting it over, and updating it, Has Jonathan: anybody, has anybody tried to port like KDE Aaron: to QNX? I have had some requests that, to do that, 'cause we have Qt 6 all ported over for self-hosted, so- Oh, ... it's been on my radar personally. But hey, if someone wants to go port the libplasma and see what that looks like, be my guest. I'd be very happy personally. I Jonathan: mean, if you've got, if you've got the Qt libraries, it might not be as much of a lift as I expected So it's, but it's a good p- Yeah, I mean- ... good Andy: portion of the work done ... and it's similarly like we like what is it, XFC or something that we run that runs on- Yeah Wayland that's on top of, QNX. So there's definitely good chunks of things that- Oh, yeah ... already exist. So I would recommend to anybody who might be serious into looking at this kind of stuff, is go to oss.qnx.com. And that's that's our open source portal. So that shows every package that we've ported and/or tested, where it lives, if we're the ones maintaining it or if it's being maintained on another server somewhere in GitHub or GitLab or wherever else. And what the status of it, what architectures that we know it works on, all that stuff. And there's some, I don't know, 2,000 some different packages that are there or y- somewhere around there. So it's actually, it's got a lot of stuff in there that's interesting to see if you, for example, if you're doing AI stuff and you wanna use TensorFlow or MediaPipe or any of those kind of things, that those are all ported. And you can just find what stuff's been ported and what stuff you've got access to. And those are all things that you can either get and, clone 'em onto your machine or, use the APK tool like Aaron was talking about and get access to 'em. So- ... makes it pretty simple. Jonathan: Yeah. I will tell you my most interesting QNX story, my only really QNX story. Okay. I've got a I've got a dbx DriveRack. It's a speaker management platform for doing pro audio. And turns out that I'm pretty sure it actually runs QNX inside of it. And I know this because this version of QNX, somebody accidentally left the debug port open and you can- Ask it to debug bash for a shell. Ask it to de- debug shell for you, and it will gladly do that. Which, means that you send it a single packet and you pop a shell, and you've got root access to it, which is fun. So I've crawled around inside of that, inside the QNX system inside of it which is yeah, it was quite an experience. That's been around for a long time. That is some- Oh, yeah ... old software. Aaron: Yes, I'm really curious what version it's running, 'cause it must be six or four if it's been around for a long time. Jonathan: Yeah. I, at one point I had the the actual CVE. I don't remember the CVE number that it was. I could I could probably find it again. I'm not... My Google fu is not quite good enough to pull it up right now. Andy: No, Google Flu is actually getting ruined too, because it's like you get results that are like, what that means nothing. Like what year did Biden die? Oh, 2039? Wait, what are you talking about? Jonathan: Yes. Just all hallucinations. Absolutely. Andy: Yeah. E- everything pure hallucination. Yeah, the Google Flu d- means nothing. Yeah. So Jonathan: I will tell you- Yeah, my- ... speaking of hallucinations, when I Googled for that- ... Google's AI told me that both QNX and DBX are owned by Harman. I'm like, "I don't know that's true." Andy: Was true. It was true actually for, so I was here during that period. So 2005, I think we were acquired by Harman, and I think Harman sold in 2010 to RIM, to Research In Motion. So there was a period of time when we were doing a lot of stuff with Harman yeah- It was probably during Jonathan: that time. I would... That would be very likely this is during that time that, that, Andy: that change was made. Probably was very likely during that time. Yeah. Yep. Yeah. And we were in like some guitar pedals and yeah, a bunch of cool applications, you Jonathan: know? QNX on a guitar pedal. That's- Yeah. I can't say much. Linux runs in some really weird places too, and, Andy: Yeah, Linux runs in weird places. And we're in things like space arms and nuclear power plants, so we're in l- we're in some pretty niche applications. And my crazy stories tend to come from my first stint around when I was at QNX as a FAE, and, finding out that it's oh yeah, this whatever it was, a coal plant or something like that, hadn't rebooted their system for 10 years, and then they tried to reboot it and it was running off of a floppy. The entire system was running off of... And the floppy wouldn't work. Yeah it hadn't been used in 10 years. So I'm, like, helping this guy who's frantically panicking as you know- ... the entire plant is brought down because they haven't even tried to run it. I'm like, l- on eBay looking for floppy. Okay, what is it? It's a model, something that, and I'm like, "Okay, I think I found one." That there's one in Arizona somewhere. We can get them to overnight it, Jonathan: like- Andy: Yeah, so- Jonathan: I- is- Yeah ... is QNX still a part of Research In Motion? Is that still the sort of corporate overlord? Andy: Research in Motion renamed to BlackBerry, and we still are a division of BlackBerry. So there's basically, there's two main divisions. So one is QNX and the other is Secure Comms. So if you think about what, what made the BlackBerry phone kinda all, secure and, its own very highly protected ecosystem, all that software, they've taken all of that stuff and turned that into a business. So like governments large corporations, institutions, all those kind of big company players that really value corporate security and those kind of things, they're customers of Secure Comm, so our sister company. And then QNX, we- we're doing the basically the same thing that we've always done. We just don't do it on phones, Jonathan: so they- they've- they've forked it. They've split the streams. That's right. You guys are no longer doing the phone stuff. Ah that's good to know. Andy: We're not... Yeah, basically as of whenever BlackBerry kinda folded up shop on the hardware business, then, we stopped sup- supporting the phones. But yeah, like we, even through all that whole period, we were still doing tons of automotive work, lots of robotics, industrial, all that kind of stuff. Jonathan: Yeah. Super interesting. What as you look into your crystal ball of the future, what what interesting things do you see coming for specifically for the, like the QNX ecosystem? What are some things people should be watching out for? Andy: Yeah, that's- This- That's actually a really good question ... Jonathan: this is where you can make your earth-shattering announcement that you're gonna open source it- Andy: Yeah ... if you want to. Yeah it probably won't be that earth-shattering. I like I- I'll tell you honestly, I think that one of the biggest challenges that, that I am trying to struggle with is just what are we doing with AI? And there's a lot of smart people in the company that are trying to figure that out, and we're looking at lots of different approaches to that. One is that we think we're, like, the ideal platform for people implementing physical AI systems because we provide all the safety and the real time and all of these kind of characteristics that you need. But we're also POSIX compliant, and so therefore, all of these AI things that people are experimenting with will run on our system. And like, when, Aaron had mentioned we had just gone to the Berkeley Hackathon, and yeah, that it was a AI-based hackathon, and we had people using our stuff to, implement, tank detecting or, looking for patterns outside cars or, doing whatever, doing games, all kinds of interesting applications to AI. So one of them is enabling AI, and that I think we have a relatively good handle on. But what I don't have a good handle on is what are we gonna do from a tooling standpoint to enable people to be better using AI with our platforms? And it's a little tricky for us because our entire brand is all about safety and security and reliability and stuff like that. You don't put us in a wind turbine 'cause you think, so you vibe coded what that is, and then it kind of- ... starts feeding the wrong current back into the, thing at the wrong phase and totally blows out the system or whatever, right? But there's all these serious consequences to this stuff. So- I think that there, there's that, but then there's the realization that it's okay people are gonna be using AI and we need to figure that out. And we're kinda undergoing that exercise ourselves of trying to understand, okay, how do we take AI and merge that with a lot of the lessons that we know about- functional safety and certification and process and all of those ways that you can use to kinda get a handle on a system. Yeah. And in some ways, it's very similar to just kinda hey, what if you hired a bunch of interns that are really eager, and you just set 'em at a task. How do you- Absolutely control their energy, right? Like, how do you vet that they're doing something right? And I think I- we're gonna come around to that, and I think the industry will come around to figuring that out. But- ... that's the biggest challenge, and I think that's the biggest place for us to kinda make a splash. But we're not quite there just yet. Jonathan: Yeah. Aaron, can we run Python and PyTorch on inside a QNX Everywhere? Aaron: Oh, of course, yeah. PyTorch. Let's see here, what's on my list? Llama CPP. I know we have Stable Diffusion, the PR up for that coming out in a minute or two again, it's one of those things. You, you name it, either we got it or tell me I don't and I'll make sure it works. Somebody's Andy: working on it, yes. I'll send you... Yeah, I'll send you a blog afterwards that- Okay ... that has all of the things that, that are AI related that we've got ported, and it's a list of a, I don't know couple, three dozen different packages that are in that space that, that would be useful. Actually, there's a couple blogs that I could send you that, that might be useful for that. Jonathan: Yeah. And something we briefly touched on this earlier, but I think it's interesting. Can I take one of the QNX images for... Obviously I've gotta make an account somewhere, which I'm not super happy about, but I use a burner Gmail address. It's fine. Can I take one of those images and run it a inside a virtual machine on my Linux machine? Play with it- Yes ... that way I don't have to, I don't have to dedicate any hardware to it. Yeah. And then does it work the other way? Yeah. Can I run a Linux VM inside of a, inside a QNX Hypervisor? Andy: Yeah, you can. Yeah. And a QNX Hypervisor is also part of the QNX Everywhere. The stuff you get for free is part, the Hypervisor is part of it, so you can do that too. Jonathan: Okay. Andy: Honestly, there's also a bunch of stuff about Graviton and running an AWS and, running in Azure as well. I- ... I'm not as familiar with that. A lot of the stuff that, that I've and Aaron do are, like, a lot of university stuff, but I do know that, people use our stuff to run in cloud instances as well. I would not- But if you start asking me any kind of detailed questions there, I'm gonna get way out of my depth instantly, Jonathan: I would not have thought of running QNX in the cloud. That's actually really interesting. Andy: I- It is interesting. It's, people use it for a scalability thing. And if they wanna have a whole bunch of developers work on the same kind of system across time zones and across the world or be able to scale up how many instances are running or not, there's a lot of reasons to be able to say, "Oh, hey, yeah, you wanna actually use this particular..." Let's say you're developing a, a digital instrument cluster or something like that, and you've got developers all around the world that wanna be working on it at the same time. It's easy to do that in a cloud instance, whereas it's tricky to do that with a physical piece of hardware, right? And plus you can get out ahead of, the needs. Like we're very much an embedded company. Almost all of our applications are embedded in some way, but that also means that you're tied to a piece of hardware. So as soon as you break that you can say, "Okay. Let's get this working in the cloud first, and then as soon as we get the hardware shipping, then we can start tying it." But otherwise you're waiting for hardware to come and it's "Okay. We can't run anything, so what are we gonna do?" I... it's, it just helps accelerate that whole thing. Jonathan: Yeah. I've gotta, I've gotta ask, and this is a meme-y, troll-y question, but Andy, did you inspire any- Awesome ... did you inspire any of the characters in the Blackberry movie? Andy: No. Jonathan: Were you around for that time period? Andy: I was around for that time period, but I- I actually, I left after that and then I came back. So yeah, but I was around during the kind of the acquisition and the phone sort of thing. Yeah some interesting times, that's for sure. Y- honestly, because I, at the time I was on the automotive side- ... and they had this really l- like this firewall of stuff. So it's like we knew that half the company was being hived off for something. And for a long time we didn't know what it was. We were like, "What the heck is going on? Are we being sold? Are we being bought? Are we..." We had no idea. And then it was then there was the announcement like, "Oh, okay, now it all makes sense." And that's why I couldn't talk to this guy for four months, yeah. Jonathan: That's hilarious. Yeah. Andy: Yeah, it was pretty weird. Jonathan: Yeah. Fun. All right, so before I let you go, it's been a, it's been a fun conversation. I gotta ask you, each of you though what's your favorite text editor and programming language? Let's go, Aaron: I'll, Andy: I'll go Aaron: first here ... Aaron's geo first. So I used to say VS Code, but because VS Code does not run on QNX, I am now a Neovim user. Jonathan: Ah, there you go. Aaron: For favorite language, I've been a JavaScript fun- I've worked in a betting company. I've been a JavaScript developer for a long time, but I've really come to appreciate Rust and those newer languages like Zig have been very interesting to see what they're doing in the space and modernizing embedded development. Jonathan: Yeah. Have you guys done anything with WebAssembly? I, that, that's something that really blew me away- Yes ... is that people are doing WebAssembly in, in embedded. And we're actually- Yeah ... in, in my company, Mesherastic, we're actually doing sort of that in reverse. We're taking some embedded code, compiling it to WebAssembly, and then running it in the browser or, on Android. Yeah. All kinds of crazy stuff. I'm just blown away by- We- ... by what you can do with WebAssembly. Andy: WebAssembly is pretty cool. We were talking with some people at Carnegie Mellon, and they were super into WebAssembly. And they were like what do you guys have for WebAssembly? What can we do there? 'Cause we, we think you got a really cool product, and we want to do all this WebAssembly stuff." And then we do support it as a outshoot of the fact that we have a browser, but we don't have ... we're not natively doing anything in it. So it's when they're talking about "Oh, can we do research projects with it and stuff?" We're like you can, but we're not ... We don't have a vested interest in it because we don't have any customers using it." But, we'll keep our finger on the pulse there, but yeah, I don't know. It is pretty interesting. Okay. So I'll answer those two questions with regard to favorite editors and stuff like that. Favorite editor that's like a weird sort of a question in my mind. I've got two answers to that- ... and you'll ... You'll probably say, "Oh my God, this guy." So one of my answers is VI. And okay, now please hear me out on this one because it just works everywhere. I can go anywhere, and I can do what I need to do with VI. You can tell I'm not- So it's like I'm not, like- You could tell that into a Jonathan: router and it'll be there Andy: Ex- exactly. Like I, I can always depend on it being there. Yeah. So that's But like for day-to-day use, oh God no. Like I used to know a guy who I worked with and like he did that with his main editor. And I'm like, okay, he was astounding. Like he knew all these colon commands and he could do crazy stuff. I was like, I would just watch him going, "Wow, if I knew like half of what you knew like that would be amazing." But I can get by. But Xcode, and I'm sorry, yes I do Mac and I'm, my next thing is not gonna surprise you, Swift. I love Swift. It like, it's just so well-designed. Like I just, I can't get over how nicely designed of a language it is. And now it really irritates me that the interfaces change like every flipping release. But other than that- that's a very Apple thing to do, Jonathan: I think. Andy: Yeah. They're just trying to make it perfect. So then of course why not break everyone's code every single time that they release something. So- Of course ... yeah. Jonathan: C- can you run Swift on QNX? Andy: No, but I want to. Aaron: Oh man, that would be amazing. I know, but the thing is that Swift is finally getting Linux better Linux support. Relatively recently. I remember there's a whole thing with Ladybird was looking at using Swift, but- ... it just wasn't there yet, I believe is what it was, if they get a little support Aaron, I had a couple of- ... maybe I'd love to do it ... kids come Andy: up to me and say, "Can we get Swift on, on QNX?" I Aaron: know. I- Andy: And I was like, "Yes! I found my people." I, Aaron: I love, every language we add to QNX just makes the ecosystem better and better, right? P- pick a, I don't care what language it is. Totally Andy: agree. Odin, Aaron: right? I take Odin, it's for 3D graphics and everything, but hey, people, auto- auto manufacturers need a good 3D programming language and do stuff like that. So yeah, every language you put there, let's do it. Yeah. It's fun. That's Jonathan: fun. Yeah. That's great. All right. Thank you, guys. Yeah, that was really cool. Thank you, guys, for coming on the show. Thank you for being patient with my questions and my badgering. It's been a lot of fun, though. Andy: It has. It's been an absolute pleasure. Thank you. Thank you so much. Jonathan: All right. Good to have both of you. That is Andy Green and Aaron Bassett from QNX, and it's been a bit of an oddball show, but that that's fine. We've had a lot of fun with it. We've got some fun stuff coming up as well. Next week we are scheduled to have Andrea Gallo. We're doing the complete opposite next week. Not only is the operating system open source but all the way down to the ISA, we're talking about RISC V. And then the week after that we're gonna talk about JavaScript and hopefully some WebAssembly because we've got Neriman Jelva of Pewter, somebody else I met at the Ubuntu Conference. And then after that we're talking with Michael Meeks of Collabora, and then we're having Francois Pol again, talking about smoked meat. That is security, specifically security on GitHub. A fun schedule coming up. You don't wanna miss it. And we will see everybody next week. Just wanna say thank you to everybody that watches, and yeah, we'll see you then
• 
  Episode 873 - Wait, That's Not Open Source! 01.07.2026 1h 3m

  This week Jonathan chats with Andy Gryc and Aaron Basset about QNX, and the interesting Open Source history and future of that embedded OS. Why does QNX Everywhere feel more open, and why do you need to register an account to download images? All that and more -- Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 872 transcript 24.06.2026

  FLOSS-872 Jonathan: Hey folks, this week we're talking with Tris Willaker about open source and the law, including but not limited to topics about the GPL and legal cases, what AI means for lawyering, and the big court case over who the real Satoshi Nakamoto is. You don't wanna miss it, so stay tuned. This is Floss Weekly, episode 872, recorded Tuesday, June the 23rd. I'm not Satoshi It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today, oh, I've been looking forward to this one for a while. Today, we're gonna be talking with Tris Wilcher, and Tris is a friend of mine, a new friend of mine that I met at the Ubuntu Summit. And so you have to imagine that b- betw- at, they would do a couple of sessions, and then they would give us 15 or 30 minutes. And the, the tag on that was, "Go network. Go meet people." And I am an introvert at heart, so I have to work really hard during those times to go and meet people, but I was doing my best. So I start walking around and talking to people, and I walked into this conversation, and it's like there was one person there I was wanting to talk to, and that we all introduced ourselves, and Tris was the other person. I didn't know him yet. And Tris said, "Yes I'm a lawyer." And I went, "Wait a second. You're a lawyer, and you're at the Ubuntu Conference of your own volition? We need to talk. I need to know more about this." And so I got to talking to Tris about some of the things that he's done, some of the cases that he's been on that he is allowed to talk about. Sometimes in the legal world you're not allowed to talk about things, but there's some of these that happen in the open, and so you get to talk about them. And then just his other open source stuff. Tris was a real pleasure to get to talk to, and I said during that conversation, "I need to have you on Floss Weekly," and he agreed. So without any further ado, Tris, welcome to the show. Tris: Thank you so much for having me. It's a pleasure to be here. It's a sweltering day here in London- In the UK. We've been issued with a cataclysm warning of how hot it's gonna get, so you'll have to forgive me if I gasp. Jonathan: It was warm. It was warm when I was there s- surprisingly there, we Americans, we think of, sunny Britain. It's so far north, it's 70 degrees Fahrenheit all the time and just beautiful. It w- it was kinda warm while I was there. Tris: Yeah, it's a high fever s- temperature right now, so you'll have to forgive me if I gasp. Jonathan: Absolutely. All right. So Tris let's start with I think your background would be interesting. So how did you end up in this sort of interesting position where you're both a lawyer and a software engineer? Would you claim the title software engineer? At least canny. At least you understand what's going on. Tris: I do release my own open source software, so yeah, I think so. I've got no formal engineering background, so it might be a steep claim. But it's not like I'm vibe coding anything Sure So I'll accept it for the purposes of today. Jonathan: Sure. Tris: But yeah, my day job what I do day to day, I am a lawyer. Originally my background was in engineering. That's what I studied at university. It was not software engineering, but a m- mixed discipline. And then I converted to law and IP and tech law specifically, and that's where I've worked at the interface of law and tech for my entire career now. Jonathan: What, what led to that decision to, to switch from engineering to law, to legal? Tris: In fact, there's a picture just behind me on the wall which I drew when I was maybe four years old, and it says, "When I grow up, I wanna be a lawyer." And I fought against that for some time. But then I I couldn't resist the pull of it anymore, and I fell back into that. Jonathan: That's great. That is very cool. It is a, it is an interesting field, isn't it? The sort of... Because you've got over a thou- especially in England, a thousand years of history in the law, and then it's something that is at the same time so present and it affects every... Whether you realize it or not, it's got effects on every part of life, and- Yeah ... people that have devoted their careers to thinking about that. I definitely see the appeal. Tris: Yeah, you're absolutely right. A- and I think we're quite lucky as a generation to have liv- have lived through both the dawn of the internet and the dawn of AI- Yeah Whi- which have- Yes ... bookended my life to date. I don't think a lot of people get to live through two revolutions in one lifetime, right? So- ... being a lawyer in that context is very exciting because although our statutes our actual laws are old, our computer copyright law is from the '80s the law is all about keeping up. It's all about developing and reacting. And that's where I work. I work in disputes in courts in, in crisis situations, and we're always trying to think ahead, adapt to the changes. So yeah, I, I totally agree. Very interesting field and a wonderful partner for a tech interest. Jonathan: Yeah. V- not very many years ago I thought rather regularly that the internet is the next big revolution and it's equivalent to the printing press. And in my naivety I thought, "It'll be another thousand years before we see another revolution like that." Oh, how I was wrong. But we're accelerating. Tris: We're accelerating. Jonathan: Although I think probably, years from now looking back, people will probably see the internet and AI as basically being the same event. That- Oh, yeah ... being in the midst of it, they're very different, but history will probably see them as basically the same thing. Tris: One precipitated the other, for sure. Jonathan: Yeah, absolutely. All right. So you you, you do specifically copyright and IP, intellectual property law, that sort of thing? Tris: That's my specialty. And especially where that touches software and tech. It can be in the kind of case that we got chatting about when we were when we were- ... having our introduction at a printer conference or it can be somebody running off with source code and it's proliferated, it's been pushed out to I don't know, Git repos all over the place a- and controlling that, understanding the architecture. That sort of thing. And when there are new issues coming up, you get to learn a lot. I'm always learning from intelligent people who've come up with new concepts from businesses who've started new ways of doing things. A- and it's just a real joy to be in that world and, a- and help it progress if I can. Jonathan: I- is AI causing some ad- additional I don't wanna say headaches, but complexity in Tris: the legal world? Yeah. It's certainly causing a lot of complexity and headaches for for lawyers who don't understand it. I'm at Bird & Bird, a firm in London and EMEA w- where- Al- almost all we do is tech related in some way or another. And we act for a lot of AI companies, and we act in a lot of AI cases. There's been one big case which I can't talk too much about- Sure ... but is out there for and did catch some headlines. It was a case between Stability AI, w- who is well known for Stable Diffusion, and and Getty. And we acted for Stability, a- and we were victorious in that case. But it was a fascinating lesson in how your mental model of the law doesn't necessarily match up to reality and finding that out in real time. That was, I was lucky enough to be part of that team. It was run by my colleague Toby Bond who does a great deal of cutting edge AI work. Jonathan: I've seen some really interesting things with AI. I think I've mentioned this to you in another conversation, but there's a website. It's satire, but at the same time the point they raise is very interesting. It's Malus, M-A-L-U-S.S-H. And this website it's basically claiming to be clean room as a service. We will take an open source project, this is exactly what they claim to do. We'll take an open source project and we'll turn the AI loose on it, and we'll have one AI bot create a documentation for it. And then we'll take another AI bot and have it look at the documentation and create all new source code, and then you can put whatever license on it you want to. And they're basically making the claim that the, the modern rise of AI is going to... And I've heard people say this, "It's gonna kill open source." I'm... I think it's i- imminent demise is maybe a little overstated there. Have you had to look at things like this? Tris: Yeah. Yeah, absolutely. In fact, the Malus site we did chat about it. And the next, the very next day actually- ... a friend of mine sent it to me saying, "Oh, my God, have you seen this? It's crazy." And the reaction from the people who didn't take it as satire- ... was really alarmist. A- and I think what that tells you is that it is hitting a nerve and it is satire done as well as it can be, right? Yeah, absolutely. It does take a while for the concepts to land even once you process the joke. And I think that's for good reason because there's a philosophy about code and there's a philosophy about copyright. And where they interact it can be very difficult to draw the line. Now I don't think that the satirical recipe for a clean room is a complete recipe. There's a fair philosophy there. A- and I think probably done the idea of understanding a process and then clean room coding a process- Could work. But I'm not convinced that an AI pipeline is a foolproof method. Jonathan: That's probably fair. Yeah, I think I think there are some open questions just in general around what an AI does. Is it sufficiently transformative to break the copyright from the training data? And, there's some cases where you can get an AI to spit out, character for character some of its training data, and in that case, obviously it's not broken that, it's not transformative at all. And then w- the other really interesting thing is the output of AI is, at least in some jurisdictions, not considered copyrightable at all, which I don't know, that has some implications for businesses using it. One of, one of my trips recently, I sat beside a comic book artist, surprisingly. Really surprised me to have this conversation. I asked him "Are you guys using AI in your stuff?" He goes, "No, not at all, because we can't copyright it." Oh, that's interesting. Tris: That is interesting. I think if you're using code as a tool, then you probably don't care whether you can copyright it or not, if it's just an internal tool. But if you're if you're writing code as a product or as a service- ... then that is a pretty big issue. And it extends to all creative industries, right? Yeah. I don't think we should lose sight of the fact that software engineering is a creative industry. Jonathan: Absolutely. Speaking about people being creative in software engineering, There is one case in particular that you have been on top of. In fact, I think you were the technical lead there at Bird & Bird about this case. And that was when Craig Wright claimed to be Satoshi the mastermind of Bitcoin. And this was the story... the case that we got to talking about. And some of the things that, the stories you had from it were just so fascinating. I, I wanted to I wanted to go into this a little bit. Can you give us some background? I think most everybody knows what Bitcoin is, but starting from that point, what, what happened? What was this case about? Tris: Yeah, I'd be happy to set the scene. And before I do, I'll say that we're very lucky to be able to talk openly about this case. This was a case about showing the truth to the world. A- and everything was done, as you put it, in the open. We're also able to see the outcome of it from, in a very long and detailed judgment- ... resulting from the case. So with that in mind I think the best way to set the scene is probably to remind ourselves that open source is a community, and I feel every community has its myths and its folk heroes and even villains. And I think if Satoshi Nakamoto i- is the mythical hero of the story for sharing his invention with the world back in 2008- ... releasing it under the MIT license, and then instead of becoming its BDFL stewarding it for a couple of years and handing over to the community before vanishing into the background then, That Jonathan: vanish- that vanishing act has raised a lot of questions among the community. So I pitched the idea while we were talking about this, like I think Satoshi wasn't a real person, I think it was an intelligence agency, and I've heard other people say that. And as part of your work on this case you said probably not." Tris: I'm of the opinion that Satoshi was a single person. I was exposed to along with others in my team- ... we were lucky enough to be exposed to potentially as much information about Satoshi communications with Satoshi a- as anyone ever has been- ... who wasn't his closest confidant. A- and in order to be able to honestly ask to see that f- for the purposes of let's be honest, justice- I gave a commitment to each person that sent something to me to say that I would never use it to try and dox Satoshi, and I've kept true to that. I've never actually tried to investigate who the real Satoshi is. So while my opinion is that Satoshi's a real person, not an agency I have n- I have deliberately never indulged i- in the temptation to, to scrub about. Jonathan: Absolutely, and I wouldn't ask you to. Tris: No, of course ... but there is Jonathan: one person in particular that you can you can say with some confidence that is not Satoshi. Tris: So if w- there's actually only one person I'm aware of in the entire world who is by law proven to be not Satoshi. A- and that is, yeah, Craig Wright. So he's if anything- the villain of this story. He was certainly found by the court to be dishonest in many ways. But the core of that dishonesty was the methodical staking of his claim over the course of many years that he was Satoshi, that he was the creator of Bitcoin. A- and true to this myth he built up his persona in a lot of different ways. And to my mind, he was quite clever in that he focused on the narrative- ... on, on selling the dream. And I think from what I can tell he definitely convinced some people. He, he convinced people who funded him. He had a great deal of monetary backing. He started off by doing a kind of signing demonstration- ... to, to s- to journalists a- and even to one of the key Bitcoin maintainers where he dressed the whole i- idea of a, of a PGP signing process up into this dramatic event. A- and it was very theatrical, and I'm pretty sure it was faked because a- anyone who has used PGP will, will know that you can just sign something and it works or- ... or not sign something a- and you've got no proof, right? But he began to convince people which was fine until he escalated. And what he did was he moved from shelling a story fr- from the sort of reputational side of it to actively attacking others. A- and I don't mean verbally attacking or physically attacking. He, he did do certainly verbal attacking. But the really problematic ones were when he started to interfere with people's lives, with their rights by suing them personally. And he was suing people, right? Not just companies, but people, actual cryptographers, journalists, a- and open source developers who doubted him. He would take them to court to, to prove that they were wrong to do so or that they had libeled him. And with all of his financial backing it came to a head because he began to win, right? He began to- ... get judgments against people, that they had defamed him by the way they had called into doubt his claims. Jonathan: Yeah. I have a ... I wanna interrupt and ask something. He had financial backing. Tris: Yeah. Jonathan: Why would Satoshi need financial backing? Doesn't the man, whoever he is, or woman for that matter, I don't know own like 1,000 of the first minted bitcoins? Isn't Satoshi like unbelievably wealthy? Tris: Yeah. Why indeed? A- and there are many holes in the story. I think if, f- out of all of them, I feel like that is the most forgivable hole in the story, because you can imagine that i- if Satoshi who set up... Let's remember, Satoshi set up Bitcoin in its early days- where it was an unstable system and more vulnerable to a 51% attack. He did so by effectively sinking mining energy into it. A- and if he started to move those coins, which were effectively, as I see it, ballast in the system- ... Tris: Y- you can see that might be considered destabilizing. So perhaps of all of the of all of the questions, that is the most reasonable. A- and true to life those coins haven't moved, right? Jonathan: I remember at the time hearing critics of his say, "If the man wanted to prove he's Satoshi here's the real easy way to do it. Spend one of the first bitcoins, and then nobody would have anything to say about it." Tris: Just spend a sat. Jonathan: Yeah. Spend- Tris: A- and of course, there was this there was this principled response to that whi- which was enough. And this was really important, right? Because what Craig Wright was doing was selling the narrative. It was enough to sell the narrative- ... to say, "I, as a matter of principle, won't prove it cryptographically, because I want to prove it with different evidence." Okay he failed to do that but that was his story. Jonathan: That's a weird that's a weird place to stand on, but okay. Tris: Yeah. It... But if you've got no keys, then that's what you're left with. Jonathan: True, Tris: true. But you highlight a really good point, a- and the point you highlight I think implicitly is this: you don't need to prove you're Satoshi to benefit from this story. What you need to do is you need to prove that you're enough of a risk That people need to listen to you. So a developer may not believe you. Somebody who's got the technical savvy to understand how cryptography works may never believe you. But their boss may. The boardrooms of the companies which are handling these exchanges may. And it doesn't really matter whether you believe him or not because if he sues you, the judge may. And I see that as almost a form of social engineering, right? So social engineering works by taking power from a person at its core, right? By deceit. And I think if you can spin a yarn and get a judge to believe it, then you've effectively got the judge to award you that power just as in the same way an employee might give you a password into a corporate network. Jonathan: Yeah. Tris: It's an inter- it just resounds much more widely, right? Jonathan: Yeah. I suppose we see that, And of course this is my opinion on these, but we see that with things like junk software patents. And here in the US there was a problem for a while. One company was sending out letters to individual businesses over a patent that covered, I think it was scan-to-email. And so they were sending out- Oh, really? Yeah. It was terrible. They were sending out basically w- letters demanding a $10,000 patent license purchase for an- any company that they thought was using scan-to-email because you're infringing on our patent." And somebody finally- Tris: And let's say you receive one of those, right? What do you do as an open source developer? Y- you can choose to fight it on principle or you can choose to try and settle it or you can choose to pull your product. If you're a small business you might try and settle it. If you're a huge business you might fight it on principle. But if you're an open source developer you're gonna pull your product 'cause it is not worth it. The open source bargain is that you contribute and it doesn't come back on you, right? And whether that's more or less enforceable in different countries I'm not sure, but if you are faced with something that so dramatically changes that paradigm- ... then y- you're gonna just back out of maintaining. Pass it on to someone else, pull the software, archive it, whatever you do you're just gonna say, "No thanks. It's not worth it." And that's what was happening with Bitcoin. Some of the main contributors to Bitcoin were saying, "This guy is a litigation risk against us personally. It is too much. We are no longer going to be contributing to this system as much as we love it because we can't risk our livelihoods." So it's destabilizing and there's a chilling effect. Jonathan: Yeah. W- was there, and you may not be able to speculate on this, but like what do you think Wright's endgame was? Obviously he was trying to amass money and power, but like he he ran the risk of killing Bitcoin, and I ... it's just such a, it's such a weird, it's a weird thing for somebody to do to- Yeah ... to claim to be this and to, threaten all these people. I don- I don't understand it. Tris: I'm not sure there was an endgame. I'll never know, right? Yeah. And you're right to call it speculation but sitting here in my speculation chair I, I'm not sure that it ever would have ended. That there were many irons in the fire. I don't think you need to look far beyond power and money to get to the end of the story. But maybe it would have continued for a long time. A- and thankfully we'll never know, because I hope it has now ended if not in the way he predicted. Jonathan: So y- we talked about this idea of an open source contributor might just pack up shop and go home, and that, that was happening for some of the Bitcoin contributors. What what changed? Was there some big group that said, "On principle, we're going to stand up to this and stop it"? Tris: Yeah, and it is exactly that impact. So the Crypto Open Patent Alliance is an alliance of, B- blockchain ecosystem companies. N- not just in Bitcoin but Bitcoin is obviously a foundational technology. A- and these are by and large a group of competitors. Th- they're competitors in their industries in many ways- but they come together to form this alliance to share patents, which is why it's called the Open Patent Alliance- ... or although we always abbreviate it to COPA to share the patents among COPA members- ... a- and pr- promote stability in a system because this is a system built on open source- and these members are all about keeping the system going allowing everyone to benefit from it, a- and making it work i- in a way which it won't unless there is stabilizing a sort of stabilizing economy behind it. So these guys sat up and took notice because when you when you are starting to discourage people from contributing their intellectual effort to the foundational open source system, you're destabilizing the whole mountain that's built on top of it. And we've all seen that XKCD cartoon. We know what happens when that domino collapses. So the th- these guys stepped in, and they came and they spoke to us, and we put together a team at Bird & Bird, and together COPA and Bird & Bird we started this case to try and prove that Craig Wright's story was false- a- and to try and show the world that was the case. So it was actually led by my colleague Phil Sherrell, who heads up our our London headquarters a- and there were several of us. I will say that I will say that the guys on the other side had more than twice our number. So we often felt we often felt like we were underdogs in this fight. Jonathan: And Bird & Bird is not a small firm. That's probably not something you guys are used to, being the underdogs in a fight. Tris: Yeah. That's right. We are often pretty lean in how we handle things. So we like to put together a team where y- you know the ideal the dream team scenario, where everyone's got their place, and is an expert in their field, and we all integrate. Jonathan: Yep. Tris: And that worked really well in this case. It was an absolute delight. So I ended up leading on the technical side and the- ... forensic investigation, some of the technical factual investigations too, which we might come onto. My colleague Ning who was magic with the more open world factual i- investigations. And then before I joined the team Graham Smith who you may have heard of, he's been an internet lawyer for decades, since the dawn of the internet. A- and he actually created- And by, and Jonathan: by internet lawyer, you don't mean someone on the internet that thinks he's a lawyer. You mean an actual lawyer- I mean- that does internet law. Tris: The law of the internet. He's seen it all. Yes. ... and he's still very active. And he'd actually put together this amazing database, I think a SQL database- ... drawing all the threads of the factual stories together and cross-referencing to see where they didn't add up before the case. I- it was a really amazing team, and we were able to keep it quite lean by, by segregating everyone's specialties. Jonathan: Yeah that's super cool. What, what did the actual legal structure of the case look like? So like you can't just, from what I understand, you can't just sue someone because you think they said something that wasn't true. What was the actual- Yeah ... how did s- how did standing work? And and obviously for those of us in the US it's gonna be a little bit different because this is British law. It was in a, an English court. But how did that part of it work? Tris: It's a fascinating legal question, not one that most people usually touch on, so good one. A- and if I can try not to get too technical, you're absolutely right. You can't just sue someone because you don't like what they say. That's not how the law works. And the question of standing was very important. But in the end it came out all right because Craig Wright had embarked on a campaign of claiming copyright in the Bitcoin whitepaper itself. A- and as anyone in Bitcoin knows, it's a sort of mark of being part of the club that you host that whitepaper on your site- Jonathan: Ah ... Tris: to show that you're part of that ecosystem. So anyone who was hosting it, he was writing to them and complaining that they were infringing his copyright. So we checked with him, we wrote to him and we said "COPA's hosting the whitepaper. Are you saying they infringe your copyright?" And he said, "Yeah, I'm gonna sue you," at which point we, we were able to take it to court to prove- ... that he didn't own that copyright. A- and who owns the copyright on the whitepaper? Satoshi. A- and it boiled down to the same question. In fact it was a bitty, a bit of a tangled ball of wool. There were many threads altogether. He was suing open source developers and also the companies who happen to be COPA members. A- and it did coalesce into one huge case w- with one central factual issue. Wa- was, Jonathan: Was that a challenge to get the court to bring it all into one case rather than have to fight him 50 different times in 50 different cases? Tris: Yeah. I can't say exactly how the court would've approached it, but if I was the judge seeing this, I would've said, "That is a tangled mess. How can I shake it out?" A- and maybe that's what the judge did. It's certainly the effect of how the judge handled it. So we did coalesce the cases into one trial. Certain aspects were stayed because they were gonna fall away if the main case resolved the way we wanted it to. And it all came to a head after a few years in one big six-week trial which was an amazing trial for a few reasons. One is that it was held in the biggest courtroom in the building- With everyone piling in. And the seats at the back of the public gallery totally full. Two, because it was actually live-streamed, and i- in a, I, in a term which I've never heard of before or since, there were more than 1,000 people following the live stream daily. Which is pretty thick stuff- in a court of law if you're not a lawyer. Hats off to those guys and girls who dive in every day. Jonathan: Yeah, absolutely. Going into this, was there a part of you that thought maybe Craig Wright is Satoshi? Maybe he's Tris: Wright? Yeah. A- absolutely. In fact a- as a matter of principle. I'm a lawyer, so I don't have any skin in this game. I came to this totally cold. I don't have any Bitcoin. I don't know about the technology, and I'm learning from everybody around me. And I don't know the difference between Satoshi and not Satoshi at that point. You come to it with objectivity. You come to it open-minded. And you see the difficulties, but you can also see how they might be surmounted. And it's only as you begin to go through the evidence, the mountains of evidence that he dumped on us, a- and you start to see that everything that supports his claim turned out to be a forgery- that you start saying, "Yeah I'm beginning to make up my mind now." So it was a process. It was absolutely a process. Jonathan: How would you have handled it? So obviously you personally came to the conclusion, and I know this from talking to you, you personally came to the conclusion that Wright's not Satoshi and so that made this easier. How would you have handled it being on the side you were of the litigation if you had looked into the evidence and gone, "I think it's him. I think this is him"? Tris: Y- then you would have put the case as best you could, but you would never tell something to the court that was untrue, right? I- it's a bit like asking a criminal lawyer, how do you defend someone who you think is guilty? Yeah. You just have to be very careful. You give them the defense to which they're entitled- ... and and you do what is right and just, and you leave it for the judge to do the right thing. So luckily we weren't in that position. We didn't have to introspect too much. It was all outlooking because it became pretty clear, at least within our four walls- it became pretty clear to me who were the good guys pretty early on. Jonathan: Do you remember the moment or the piece of evidence that you looked at and went, "That's not him"? Tris: So we got his disclosure, I remember, and I did... The first thing I did was I just spun it all up on my Linux box- ... which I actually had to switch to j- just to get through the volumes of evidence that were being poured on us- and to be able to pull tools to analyze it. And I spun it up, and I just I just put some filters through it for the earliest documents that mentioned Satoshi. And I remember very clear, very clearly pulling up the first, I think it was a Microsoft Word document- ... opening up the internals of the document and seeing right then and there within a few moments that the internals of the document had all been sanitized away to appear from maybe 2007 But they had all of the editing artifacts, u- unfortunately for Craig, still compressed into the document itself. And it's all in the forensic reports. You can see the references to the Financial Times from 2016 and The Economist, and you're like, "Oh, yeah. That was prescient." Now, Satoshi was prescient, I think, but not that prescient. Jonathan: So when you found evidence that things were forged, that was pretty much the moment? Tris: That was an eye-opening moment. Very exciting. But actually the evidence came in many threads. It wasn't just forensic analysis. On that technical side, though, what really struck me was the extent to which everything was based on open source tooling. So we were there, and we were defending an open source industry against somebody who was suing open source developers o- on behalf of a patent openness client. So we're already into that field. But then we're receiving forgeries created with with LaTeX, with C++ with OpenOffice. A- and we're using Linux, and we're using open source tooling to be able to analyze them. And that was my moment when I ... when the whole open source bargain and community really clicked for me because you could just take these things, apply them, a- and get the result you needed. Maybe tweak them if you needed to 'cause of your specific use case. A- and I had to learn, I had to learn Python and I had to learn Bash scripting to get through all of this. So that was my learning curve. That's- And it was a delight. Yeah, Jonathan: that's great. One of, one of my ... Oh, I can't re- I can't think of his name, but one of my absolute heroes, there is a, there's a judge in the US that did the big Oracle Google case. Tris: Oh, yeah. Jonathan: ... Oh, what? I don't remember. S- Steven No. Oh, goodness, I can't remember his name I- Tris: is the name Alstrup or Alsop? Jonathan: That, that sounds right. Judge Alstrup. Yeah, anyway, he he presided over this big case about Android between, Google and Oracle, and at the end of it he's... One of the things that he said is, "I learned how to program to be able to understand what was going on in this case." And, he's... And then says the- there was, like, 11 s- lines of source code in question at the end, and he looks at it as part of his ruling, and he says, "There's only one way that you could write this. There's no other... you could ask a, a first-year engineering student to write it, and he would give you basically the exact same thing. So that's fair use." And I just... I love the fact that he learned to program to be able to try the case, and one of my, One, one of those stories about the law that I really enjoy, somebody Tris: doing it Jonathan: I think Tris: this is because we're in the information age and lawyers all around the world need to know that if you see things through the lens of the information age, if you start seeing your cases that way- then you become a better lawyer because you can use software tools to become a better lawyer. You can assist people who are using software more effectively. Yeah. And I th- I hope that everyone starts to do that. I once had a case a long time ago now- ... where somebody brought me in and they said, "This is a real problem. I know you've got... I know you've studied engineering. Maybe you can make a head, heads or tails of this because our client says that they never copied anything, but their code block is identical." And I looked at it and I was like, "This is just linear regression." "There is only one way that you can write this." It's your same point. I, and I do think that comes up again and again, actually. Jonathan: You actually... You see something similar not exactly the same thing, but something similar in music copyright. ... It's like the, "They copied this tune," and there's a, there's actually a project out there where a lawyer that understands music put it together and "Okay, six notes in a row. There's 12 notes. There's only, and I forget exactly how many, but it's there's only a few thousand ways that you can put this together. We're just going to write all of those tunes and copyright them all, and then put it in the public domain, and then we'll just be done with this idea of, "He stole my melody." It's six notes in a row. It's... You can't copyright that. Tris: I like that. I like that. Did you hear about how Ed Sheeran was in court and he brought his guitar with him, and he played through the various ways that you could put some chords together? It was a very effective- ... piece of personal advocacy. Jonathan: Yes. Yes, absolutely. Every l- how does that go? Every pop song for the last 30 years is the same four chords, something like that. Tris: Yeah. Jonathan: It's very much the same thing. It's the same idea in music theory as we would have in computer theory. "There's only one way to do this. Every song is basically the same chord progression." Tris: Yeah ... Jonathan: that's Tris: similar. It sounds good. Jonathan: Yeah. So one of the, one of the other neat stories that you had, and I'm probably jumping partway into this, but B- Bjorn Straustup, y- he was a, ... Y- you made personal communication with him. Yeah. And something about the story here was, like, you needed an expert witness, and you couldn't get one, and the only way- So it, it is the case- What's the What ... Remind me the story. Tris: So we ... W- Background is we'd gone through these 100 main documents that Wright said was ... his crown jewels, and we'd basically proven that every one of them was either unimportant or fake. A- and we'd served maybe 2,000 pages of expert evidence. There was a conspicuous absence of code in any of his documents. So he, as far as I can piece it together had read our manual of how to forge and how to be caught forging. A- and must have seen the whole seen the whole where there was no code, a- and tried to fill it, because the thing about source code is it's plain text. There's no metadata that goes with it. A- and that came very late in the case. We were already gearing up to trial. It was an extremely late disclosure, a- and I believe that it was late because he'd just finished forging it. In fact we were able to prove exactly the dates when it was all forged. But the the problem was that in order to prove that certain functions that were called on in that code- couldn't be correct to 2008 or 2007. It was necessary to un- understand things like what is the standard namespace? W- when were these things published? And it's very difficult to prove that without an expert who will educate the judge about what these things mean. So w- what do you do? You either go through a big procedural question of does everyone get evidence on both side? There wasn't time for that. Or you ask the man himself. So we decided we'll contact we'll contact the C++ developers. We contacted the person who wrote the library in question. And we contacted Professor Stroostrup and they very kindly responded, because open source is a community. And when people ask you about your software pe- people are generally in this community extremely helpful- ... at sharing knowledge, both technical and historical. And this happened again and again. So we had forgeries made in OpenOffice because that's the software Satoshi used. So we were able to contact the then maintainers of OpenOffice- ... who were able to give us their build logs and prove that the build that he'd used to forge it didn't exist at that time. And nor was it possible to predict the build hash o- of the software that, that was output by that compilation process. That was OpenOffice. The same happened in LaTeX because he then pivoted to, plain text LaTeX source. Which didn't end well for him either. But Professor Stroostrup a- and many others joined in to help us prove fact after fact. A- and these things were bricks in the wall that, that added up to a proof that each and every piece of evidence he relied on- Was false. Jonathan: I have a little note here that there, there was some interesting sleuthing done outside of the digital domain. Tris: Yeah, that's right. So what do you do if all of your digital forgeries are getting picked apart? Hand write some, right? W- we had this amazing ... A- and this c- this particular document actually went through three court cases I believe without being pulled apart, and it was very simple. It was a single sheet of A4 paper, and on it were written meeting minutes, handwritten, very terse, and they were on this form which was headed up minutes- ... prefilled table And with a single sheet of A4, handwritten, there's not much you can do- ... to prove that it's fake. So we thought about how do you go about this? A- and without going into the privilege detail the upshot was that as a result of an amazing teamwork of people all around the world y- across three or four continents, we were able to track down not just when this pad was printed and by whom, but we were able to follow the trail of companies that had been acquired one after another, and back up that stream, contacted the lady in Shenzhen, China, who had operated the printing press- which printed that particular pad of paper. Ah. And she'd kept her PDF proofs, and they still had all of their metadata intact, and we could show, As a group, we could show that this pad couldn't have existed before 2010, which was three or four years after he, he claimed it to be. It was the most magical moment when that landed. I think it was like 11 o'clock one night. I didn't sleep at all that night. It was so exciting. Jonathan: Yes. Oh, that's great. You do the little happy dance. Tris: Oh, yes. Jonathan: Is it... Because that's the sort of thing you don't know for sure until you get it in. This is a... i'm sure that was a long shot. You g- My guys must have considered that a long shot. Tris: It's totally a long shot ... maybe there's- But you're open-minded, right? You follow the truth because you'll either find out something which is true and is against you, in which case at least you found out the truth. Or you'll find out something which is true and for you. But if you're open-minded and follow the truth that is the right way to to approach these faction investigations. Jonathan: Absolutely. The the trial itself, you said it lasted six weeks? Tris: Yeah. Jonathan: How did that Tris: go? It was, it was pretty incredible. We started off with Craig Wright himself on the stand, as you say, for six days of cross-examination. And then that's a long time. That is a long time, yeah. Plenty of breaks. It wasn't inhumane. A- and he was- ... very good at telling his story. But I think it's fair to say that the people who went into that courtroom thinking that he was he was Satoshi, were able to convince themselves, "Wow, this guy s- has such a facility with answering questions, that he must be Satoshi." A- and the people who were maybe more open-minded could see the holes beginning to emerge at that point. And we went through all of the documents and every facet of his story and he clearly prepared. Excepting one case which I will dwell on because it makes a great story. So let's say that you... Just imagine that you are a person who's forged. W- whether you've done it with your own hands or you had help doing it, w- we'll never know but imagine you're that person a- and you've got an answer for everything, and then the one unexpected question comes, and you react on the fly, but you realize there's a hole in your story. What do you do? You go home that night and forge another document. And that is what seemed to happen during this trial. So a few days later, after that first week w- we had a situation where the lawyers on the other side said, "We... There's something we need to bring to the judge's attention." And they did and they did it in open court. They said, "These documents have just been disclosed to us." And they provided them, and of course we had to go through a whole another forensic investigation. By this time it was maybe our sixth. Because of the layered disclosures that we'd been having all the way around. A- and it was it was... The upshot of it was a document that he'd forged during trial So he had to be recalled once to answer for certain additional problems that had arisen, and then again right at the end, right at the end of the case, he was on a third time under oath to answer for this forgery that was made during trial. So it went right up to the wire. It was an incredible experience to be conducting a forensic a- examination back at the office while the trial was in progress. I hope I never find myself in that situation again. Jonathan: So it was established in a court of law that Craig Wright lied on the stand under oath? Tris: Yeah. Jonathan: Why is he not in a British jail right now? Tris: He, ... I don't know exactly where he is. I think he went to Thailand for a time. He was found to be in contempt of court and has not come back to the country since then. But I think the way the judge put it was like this. "If Dr. Wright's evidence was true, he would be a uniquely unfortunate individual, the victim of a very large number of unfortunate coincidences- ... all of which went against him- ... and/or the victim of a number of conspiracies against him." And then the judge took a break and says, "The true position is far simpler. It is, however, far from simple because Dr. Wright has lied so much over so many years that on certain points it can be difficult to pinpoint what actually happened." A- and he went on to find over the course of maybe 230 pages it's the length of a novella, Okay ... that Wright had lied and lied. A- and it was a very cathartic result. W- what had really happened was that all of Craig Wright's eggs were in this one basket by the end. He just had to convince this one person, this one man, that he was more likely, 51% likely- ... to be Satoshi. The archetypal 51% attack on Bitcoin you might say. And he picked the wrong mark because he was up before a judge who himself has an engineering degree and a long history of practicing in IP at the forefront of technology who really dived down into the detail, read every word- ... looked through, I'm certain looked through the source code line by line, really understood what was going on. A- and at the end of that six-week trial, the judge had been, He stands he-- I was gonna say sitting. You, we say a judge sits in court but this judge stands, and he'd been standing and listening to all the evidence silently throughout. And at the end of the case, it was his turn to speak. And the typical the typical conclusion of a case is that a judge will say, "Thank you for all of your submissions and evidence. I will consider it and deliver a written judgment." And that's what happened here. And then he paused, and he said, "But I have seen all of the evidence, and it is quite clear to me that I can make the following binding declarations." And he did. He declared right then and there, "Craig Wright is not Satoshi Nakamoto. He did not author the Bitcoin whitepaper. He did not author the Bitcoin software." A- and he delivered it, he brought it to an end right then and there with 1,000 people on the live stream, I think. A- and us all in the room. And I still get I still get goosebumps thinking about it now. Jonathan: That's great. That's a sort of once in a lifetime moment. Tris: Quite right. Jonathan: Yeah. That's awesome. Now, you you've done some other open source work that's in the sort of the legal sphere as well. So you do work at Bird & Bird in software. Yeah. Did they give their stamp of approval of, you did this on the clock, but you're allowed to open it, release it as open source? Tris: Oh, no my side project is my own and done off the clock. Okay. So I do make tooling internally- ... for n- neat jobs that need doing. I like to make self-contained tools- ... which can be run client side. A- and with a bit of JavaScript and just understanding how libraries work- ... combining your knowledge of how the legal process works with that technical capability allows you to make tools- a- and just deliver solutions, often very quickly. A- and it's often like document management or document manipulation. So I have a l- li- little library of functions that I can use. That's all great a- and helps us and is very very fun to, to work with my colleagues to to solve those problems. But one thing that I made for myself was a, o- originally Bash and then Python a- an implementation of a pipeline to get your documents from loose PDFs- ... to a bundle of documents that you can take to court. A- and what's what's great about this is most lawyers will give it to a trainee or a junior to do and they will deliver the documents- Okay ask for it to be done, and then receive it back the next day and never have to get their hands dirty. But that is the most soul-draining part of a young lawyer's job. It's awful. Ah. I- it is the bogeyman task that everyone tells stories about bundling late into the night. So I, I realized this was actually really useful. And I thought I've spent the last three years benefiting from open source tools myself. I've got the bug. Why don't I open source it?" So I did. I re-implemented it again in, in JavaScript- ... so that it runs client side. And taking advantage of some of the quite cool WASM r- recompilations that are available now. Oh, yeah. Especially for Move PDF. And with a bit of front-end magic, it allows people to create their court documents into a bundle which fulfills all of the court's requirements, and to do it in a- in about 30 seconds. And that saves time in two ways, right? So first way it saves time is if you're a lawyer and you know how to get it done, it's a quick process. The second way it makes a sa- a time saving for people and I find this very wholesome, is that if you don't have a lawyer, you can't afford reputat- representation- ... then you'd have to learn what the court requires first before you could start even doing your document preparation. And this pipeline does it for you. So all you do is throw your documents at it, get them in the right order, and it does all of the hyperlinking, cross-referencing, and all of that for you. So it is a time saver mainly for litigants in person. Also sometimes for lawyers, and I know some some major law firms in London definitely use it. So that, that's it. It's my way of giving back to the community because I don't have a great deal of time for pro bono work at the moment. And it's built on open source, so why not release it open source? Yeah. Jonathan: Absolutely. I imagine it also helps avoid those awkward conversations of I'm sorry, Your Honor, the table of contents says page five. That's actually on page seven." Tris: It's such a waste of time. And it's not just an admin problem, because it's an access to justice problem. Let's say you- ... haven't got a lawyer, and all you wanna do is go to this, possibly the most stressful place you'll ever find yourself. I can't think of many more. Perhaps hospital. But you're going to this stressful place, and you need to make your argument against a talented adversary. And if you can't communicate to the judge with the documents in front of you, if you're always losing your focus because you have to add three to the page numbers, you're not gonna be able to communicate effectively. A- and actually it should just help people not fall into that problem and I hope give them a better chance of conveying their case to the court. I can't improve their case for them, but hopefully it will help them do what they can do for themselves. Jonathan: Yeah. Yeah. That's, that's- Something I've thought about quite a bit is one of the problems with the legal system around the world, I think, is the barrier to entry. And a lot of times that's cost. But even just become- having your tooling together and your understanding together well enough to be able to present a case. We have this... The ideal is that anybody, no matter who you are, you can go to a court and you can present your case. And if you, if the, if right is on your side, you win. And unfortunately the reality is that oftentimes it's whoever has the most money that they could pay for fancy lawyers is the one that gets to win. And, Tris: Of course in the Copa case, if right's on your side, you're gonna lose. But the the barrier to entry problem is huge. Not just in terms of tooling, but in terms of c- conceptualizing. A- and actually the legal system is very similar to, to to software engineering. What it's attempting to do i- is create a highly precise form of words to convey a concept. And that is in essence w- what a coding language does. It, it applies a particular syntax to that problem. The language we use is English. But it's about conveying concepts with precision. A- and, and- ... if you squint at a contract next time you have to sign one and get your get your software engineer mindset on you'll notice that we have effectively imports function definitions in the main loop. I w- we don't call them that. We call them party names. We call them defined terms- Yeah ... and clauses. But they operate mechanically in exactly the same way. So I feel like there's a lot of shared mindset there. Yeah. And perhaps then of all people the people who, who suffer the least barrier to entry are software engineers. Jonathan: Yeah that's probably true if only because we are used to parsing through the difficult things and understanding how they work together. So one of the, one of the quotes that I like to use on the show and it's Simon Phipps, and usually he's talking about the importance of community when he says this. But he makes the statement that "Software licenses don't compile." And his point there is, you don't know exactly what the software license is going to mean until it be- appears before a judge. And, the judge has comments and some of it doesn't survive, some of it gets understood differently than you expected. Is that software license a contract or not? Some of those various questions that, that show up. It's interesting though to think about that in a way that is what this is an attempt to do. The law is a, an attempt to make s- these concepts almost compile. That's a rather intriguing thought. I wonder if someone could formalize that at some point and make it so that they do actually compile into a perfect meaning. Tris: Ethereum would say that smart contracts are the way. But the i- if licenses are your compiled programs, then lawyers are your CPUs. So what you're really trying to do is you- as I think out loud i- is you're trying to have something that reproduces no matter what processor it runs on in the same way. A- and a well-established license with well-established clauses- ... will be met with the same legal response, with the same legal advice, no matter which lawyer you go to, as long as- Right they know their job. I think that's the closest analog to compiling a license. I- if anything, the license is its own machine code i- in that paradigm. Jonathan: Tries to be. Do you follow some of the cases, like for example right now there's a quite notable case in the United States where Vizio is being sued by one of... Oh, I forget the exact name of the group, but one of the free software groups, basically saying, "The things in your TV are GPL. You need to release all of this source." And it's, it's notable because it's one of the first times that the GPL will go before a judge, and we get to see what the judge thinks of it. Tris: So that's, I think, a Software Freedom Conservancy. Jonathan: Yes. Yes. Tris: SFC. SFC. I had a chat with with those guys at one of the conferences. A- and, ... it's a fascinating debate. I am certainly not qualified to, to comment on the legal aspects of it. Sure. But I, I think I think there are not only the f- like, the fundamental questions of what does this license mean, but it also is... It's a big stand, isn't it, for the industry. We are going to enforce our licenses. We are not just gonna sit back a- and let you in, in the terms of SFC, abuse our license. And the outcome of that I potentially could change how people measure the risk balance. I don't know. E- especially since it's it's in an important court, isn't it? I don't know if it's federal court or Californian court, but it's a court that people are gonna be waking up and listening to. Jonathan: Hello Just going to look at what court that's actually in. I don't remember. Yeah, that's what I was gonna- I think it's, I think it's a California court for now. But the way these things tend to work though, US law is based on a lot of British law and the idea of common law, what the courts have decided over the years tends to become binding. And what a California court decides will get referenced by, a Florida court looking at it, or a federal court looking at it un- until someone else comes along, a higher court, and says no, you got it wrong. We're gonna change that." I, in a separate conversation I asked you something about whether you thought the the next version of the GPL was being worked on, and I'm reminded of that because depending upon how court cases like this turn out, that may be exactly how we get a GPL v3.1 or GPL v4. And I think- I Tris: actually have no insight into the question, but- Jonathan: I Tris: think- But I think you would... You're absolutely right that waiting for the outcome of a case like this would be a great thing to do if you were working on it. Jonathan: Absolutely. It may very well be one of the things that spurs on that, that next revision. Personally, I think AI is something that needs to be addressed in the next version of the GPL. Probably have a couple of different licenses, whether, you want people to be able to pull your work into an AI training des- set or not. Like how does that work with copyleft? How does AI training work with copyleft? That's an unanswered question in some ways. Tris: It's... In some ways I think it might be unanswerable. Be- because it, in many ways the core principle o- of open source is reproducibility, right? So does that mean that if you're gonna be looking for an AI model to be open source, you need to be able to reproduce, what, the weights? Or do you need to be able to reproduce the training from scratch? Where does that re- reproducibility line start and finish for something which is a totally different kind of software doing a totally different kind of task? Jonathan: Yeah. The OSI actually took that particular question on, and they now have definitions for what it means for an open sou- or a model a, an AI model to be considered an open source model. And they made some decisions about how exactly that would work, and people are not happy about it because it is, it's so new and it's so different from anything else that we've had to fight with before. So that, the, there are OSI definitions for what it means for a model's license to be, OSI approved. But people are not, people were not happy about Tris: it. It's a highly political question be- Sure ... because people have different opinions about what AI means- ... what it should be, and what openness means in that space. I thought it was a really good thing that they went ahead and published it. Yep. Be- because a lot of the questions were being asked in the- abstract. And this gives someone a framework against which to test their their thoughts and assumptions, right? I feel like it probably created as many opinions as it put to Jonathan: bed. Of course. Of course. I know, Tris, that you've got to hard out here in just a few minutes, so I think, and we've gone for over an hour. I think we can go ahead and wrap it up. It has been an absolute pleasure to have you on the show to talk about- Tris: My pleasure to join you ... Jonathan: things legal. We we'll have to do it again. Give it a few months, we'll have to have you back. It'll be a lot of fun. Love to. Maybe once the the Vizio and SFC case wraps up, we can have you back to talk about that. That could be interesting. But- Let's Tris: talk about it. Jonathan: Yeah, we'll do that. All right. Thank you so much for being here. I appreciate it so very much. Again, it was a real pleasure. Oh, I gotta ask. I gotta a- I will get emails if I don't ask. Okay. Favorite text editor and scripting language. Tris: Oh I love to script in Bash. That's my favorite that's my favorite scripting language, for sure. I have been experimenting in the last few weeks with Helix. I'm more of a Neovim kind of guy. And when I just wanna get a quick note together I find Kate, because I use Kubuntu. I find Kate to be extraordinarily fully featured. And very good for markdown with a preview. Jonathan: Helix is the post-modern editor and of course in their FAQ it's, "Why post-modern?" It's a joke. If Neovim- ... is the modern Vim, then Helix is post-modern. Tris: I have had trouble getting to grips with its reverse bindings- i'll be honest. But it does have a handy pop-up. Jonathan: Yeah. That's fun. All right, Tris. Again, thank you so much for being here. I appreciate it. It's been a blast. Tris: My pleasure. Thank you. Jonathan: All right. We've got some fun stuff coming up in the future. We've got Andy Rick and Aaron Bassett from QNX next week. We'll make sure and ask them why QNX is not open source. And then the week after that we've got Andrea Gallo of RISC-V. Looking forward to that one a lot as well. And then in a few weeks we're talking with Michael Meeks of Collabora. I believe I met him at the Ubuntu Summit. And then we have a returning guest, Francois Poulin of Smoked Meat, and that is yet another GitHub but related security tool. You don't wanna miss that one as well. We appreciate everybody that is here, whether you watch or listen, get us live, or on the download, and we will be back next week on Floss Weekly.
• 
  Episode 872 - I'm Not Satoshi 24.06.2026 1h 7m

  This week Jonathan chats with Tristan Sherliker about the Craig Wright case, Open Source and the law, and Tristan's own Open Source project, BunTool. How did Open Source help win the day at the Bitcoin trial? And why is right now such an interesting time to be in the legal field? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 871 transcript 17.06.2026

  FLOSS-871 Jonathan: This week we're talking with Florian Gilcher of Ferrous Systems about Rust, Rust in the business, Rust in the kernel, the history of Rust, and a whole lot more. You don't wanna miss it, so stay tuned. This is Floss Weekly, episode 871, recorded Tuesday, June the 16th. Rust won't save you It's time for FOSS Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we're gonna get rusty. And that's because we have Florian Gilcher, the the man, the managing director, the co-founder behind Ferrous Systems. Ferrous Systems, Rust, you see what they did there? Anyway, he has been doing Rust since 2013, which I think is before Rust existed as it exists today. Wanted to make sure and ask about that. He's a co-founder of the Rust Foundation, been a core team representative, and done a lot of other really interesting things. We're mainly gonna talk about Ferrous Systems, but you can believe we'll get a lot of Rust questions in. I'll probably tease him a bit about rewriting things in Rust and the way the internet seems to be crazy about that. Without any further ado, let's let's bring him on. Florian, welcome to the show. Florian: Yeah, hello. How are you doing? Jonathan: I am great. It is good to have you here today. So you are, let's see, you're a German citizen. You're from Germany. Yeah. But you're not in Germany right now, are you? Florian: I'm currently in the UK at ACCU on C, which is the merger of the ACCU Conference and the C++ on C conference. So I'm currently over at the other side, essentially. I've been giving a Rust workshop there, but I have very good conversations with C++ folks all day long. Jonathan: Okay. That's a, this is an interesting topic to to touch on first. How well or poorly do you get received doing a Rust conference at a C++ convention? Florian: Oh, very well. Okay ... it's actually in subject. The ACCU reached out in 2016. That was right after Rust 1.0. And the Russell Winder, who ran the UC- ACCU conference back then actually had the stance of ACCU, which is the Association of C and C++ Users, I think. That's what it stands for in the UK. Thought about, "Okay, how do we deal with the fact that there's new systems programming languages coming up?" And their choice was, "Oh, it increases our relevance. They should be here, and they should be talking at this conference." So there's people from Swift here, people from Zig, and I think that's very much true. And the reception in the C++ space was generally good. Whenever something like that comes up, there's people who, there's naysayers on both sides. I know a lot of people on Rust that say C- C++ is gone now. And I think nothing could be further from the truth. Still very relevant. And I think most people will just be like, "Oh, ah, it's good that you're here. Let's talk. Let's exchange opinions. Let's see where all of these things are going." At a- Sounds so cliche, but yeah ... Jonathan: at a conference like that, you already have the subset of, C++ and C users- ... that can be together at the same conference without killing each other. You know- Yeah You're halfway there already, Florian: I was sitting at a table yesterday with someone from the C Standardization Committee and someone from the C++ Standardization Committee. I won't say anything they talked about, but we had a lot of fun. That's Jonathan: fun. Yeah. All right. So Ferris Systems, obviously it's a play on Rust. What is Ferris Systems and what do you guys do there? Florian: Okay, so Ferris Systems is one of the first companies that provided services for Rust. It actually comes out of... I was, just to answer that question, I was part of the Rust community before 1.0. 1.0 was May 15th, 2015. And I was part of, I w- I was doing training for Rust back then. I w- I had an old company, another company that was dealing with Ruby, also doing a lot of open source there, mostly applying it, less building it. And at some point someone just came up and said "Hey, could you do a Rust training maybe?" And so I said I could probably do this. I'm equipped for that as good as anyone else." Started doing that, and in ... Slowly started also offering Rust as a service, and in 2018 a number of larger clients came around. And so things like, "Hey, we're build- we're using Rust. We're a hyperscaler. We're using Rust internally." And I was like, maybe a second company makes sense at that point. Jonathan: Yeah. Florian: And on the other hand, so what I always wanted back then was also for the spread of the language to make sure that, for example, all of our training material is open source, because I wanted people to apply that and to spread the language. And yeah, that has turned into now a major operation. We're more than 20 people serving customers from, I always say from Japan to California which is- Jonathan: Everywhere ... Florian: a blessing and a curse sometimes. Yeah. Yeah. But, The Jonathan: time zones. Time zones are interesting ... Florian: yeah, that's what we do. Yeah, it's mostly the time zones that are the blessing and the curse, yeah. Jonathan: Yeah. Yeah ... do you guys have people in the various time zones? So like you're not the only one answering emails in the middle of the night? Florian: Yeah, this the first growth phase of the company was during the pandemic, so we're people call it remote first. We're, ... This, it sometimes takes a year before I see a new colleague personally. So yeah we're distributed mostly time zone-wise Berlin to Californian time zones- ... Florian: Mostly. Jonathan: I've got work colleagues, I've got work partners that I haven't met yet, and we've been doing our business for over a year now. It'll be, ... i'll meet a good portion of them at DEF CON for the first time, and then later this year- i'm going to Europe and I'll meet a- another, at least one hopefully he'll make it, another one at a conference there. It is, it's fascinating to have, people that you know so well, you have business relationships with, and never meet 'em in, never met 'em in person. We just know, we're just- internet friends. It's such a, it's such a wild thing. I- Go ahead ... Florian: it's the same in open source projects. I find that quite interesting also when I was part of, of- Absolutely ... a team lead there. It's you have such a wide diversity also not just of where people live, but also some people are just there for an hour a week. For some people it's their day job. So it's completely different to manage and a lot of different concerns to just get get under one hat. Jonathan: Yeah, absolutely. So Rust, Rust was a very- ... different language, and I alluded to this in my in my opening intro. Were you around for some of this transition from, and this is my understanding. You obviously, you can tell me that I'm wrong, and I will not hate you for it. But my understanding is that Rust started out as an academic language and, was, was- ... almost like a toy academic language at the very beginning. And then people said, "Oh, this is really interesting." And then, you have folks like Mozilla picked it up and said, "We could actually take this and turn it into something really useful as a system language." And so like the Rust 1.0 is quite a bit different from the Rust that the the academics were using before that. Is that accurate, first off? And second, were you around for some of that transition? Florian: I was around for the end of it- ... basically. So but some of the pretty big changes. So I was around when it was still three days later, the language could be completely different, where some syntax has changed, was added, was removed. So it, it's between, it's that. It always came out of Mozilla. It was a plaything of a Mozilla engineer for a while. And at some point, Mozilla Research picked it up as a proper project. And the interesting thing I found about Mozilla Research, and this is from an outsider's perspective, I was never part of Mozilla Research was that they had a very good vibe of we are doing research, but we are mostly doing research application in the sense of that Rust is based on a lot of things that came out of programming language research in the '90s, in the up to 25 or something like this, or a little bit later, plus some things on top. So things like the borrow checker and things like linear types that are the b- foundation of the type system had already been researched. No one has put that into a language yet. And so there was, before the 1.0 release, a huge group of people that were also using this language to try out things. There's a really good talk by Marjin Haverbeke at my first Rust conference as the closing talk. The Rust That Could Have Been, where he talks about all the features he has tried out that were all removed. So there's none of the features he tried out on the language were put into the language. And he went on stage and said, "And it's actually a better language for that." Yes that time existed, but at some point It, y- people found, okay, there's something here. We really want to industrialize that. So there was this phase of, And that was the time where I was mostly around. And that more as a community member than an actual project member from 2013 to 2015, where they figured out, "Okay, let's actually make this a systems programing language. Let's make it play in the C and C++ space, and let's actually release that." So the 1.0 Rust release is very imperfect in hindsight, but it is a release. And, Jonathan: Every, every 1.0 release is imperfect in hindsight. Florian: Exactly. And that was the thing that drew me in. There's this, they, this vibe coming up of, oh, we could actually do something here. Also, just to be clear, I think there was, it was the vibe in the room back then. It was not that surprising that Apple shortly later came out with Swift. So a number of people... That, that's what I meant, is like the, this observation that I gave earlier systems programming was becoming relevant again was very much in the air. So I always say Rust's success is not only the success of Rust, but also the success of making the topic relevant again. So it's also the ocean in which you swim. Yeah. Jonathan: Yeah. That's an interesting point. You mentioned, so I'm gonna ask you some questions just because Rust- Sure ... and I have Rust questions. Y- or maybe I'm gonna rant about things. Y- you mentioned making Rust play with C and C++, and there's a... I see it as a problem. Maybe this is bec- because I come at it from a different background. I don't think that Cargo works very well with integrating with C and C++, and I feel like maybe the kernel has had that problem, too. I've mused before that the worst decision the Rust guys ever made was packaging a, putting a package manager as part of the language. Tell me how I'm wrong. I Florian: say, so first of all I have a very long rant about package managers for on, on Linux distributions because I'm from the Ruby space. And we suffered so much with package managers that had their own opinions that were like just aggressively against what their community was doing. So if you think Cargo looks like Bundler, that's because it's actually from the same author. And I think some of that- some of that actually shows up in Rust a little. I think the good decision that Rust has made is that Cargo drives the compiler through its public interface. Okay. So you could, you can replace Cargo, and this is frequently being done. Google does build their Rust stuff with Bazel. Facebook has written their own build tool called Buck2 actually in Rust. So this is it is not uncommon that people replace Cargo. I think Cargo wants to own the world and wants to own the project is very much true, and if that's not what you want then then th- those stresses show up. Yeah. I d- I don't think it's an... it's an okay reading. I think though it has made Rust successful in the sense of it brings this mode of development to the systems programming world built by people who have actually had experience. It's very much a, it's very much lean, like some- something we took from the NPM and Ruby ecosystems, just to be clear. And that may sometimes feel a little bit off. This is an, like just one more sentence. This is another observation that p- I have about the Rust community. R- the Rust community has an interesting mix of systems programmers, but also people who went from scripting languages towards a systems programming language, and that sometimes leads to an interesting mix of concerns. Jonathan: Sure. Yeah, absolutely. Has Rust experienced any of the security problems with malware packages that NPM is famously having just about every day? And, now we see it in other places like the Arch User Repository is having quite a moment- ... with malicious packages. Has that happened in Cargo and the Cargo package backend yet? Florian: Let me put... just to be clear FOS systems used to operate or would be the on-call service provider for Cargo, so I need to make sure that I'm know what can share. I can definitely just say the, first of all of the package managers actually have forums in which they talk about these things and are observing- those attacks. Most of that is not happening in public for obvious reasons- Sure ... because it gives the attackers the up. Cargo d- has experienced some of those attempts. We have definitely also seen people just taking other people's code, republishing it ripping the open source license off, putting their own name on it, and all of these things. All of these things happen on crates.io as well. It- I don't think the situation on Crates.io is in any way that bad. I also don't think the situation on npm is that bad. Most of the time those things are smelled out and removed relatively quickly. You just need to be aware that you are using a live service. This is actually one of the things that we're offering to customers, is we're saying if you don't want to rely on a live service where these things may happen all the time we can offer you, for example, managed crates hosting, some ways of ingesting the crates that you want that are relevant in a semi-automated fashion. Jonathan: That's interesting ... but of course, Like a cur- a curated list of safe crates- Yeah ... and not- Yeah ... immediate updates and- Florian: Yeah ... Jonathan: some probably a combination of automated tooling and some human eyeballs to look at things. It's, it's- Yeah ... it's pretty interesting. Florian: And but that is, for example, a place where the more slow-rolling model of distributing these things with, for example, your Linux distribution definitely is not that, that much subject of you... I cannot hack Debian by just uploading a package somewhere. Someone needs to make an active move of integrating it into Debian, and that's already a first line of defense. Yeah. Jonathan: Yeah. N- not to say that it is a perfect line of defense. Yeah. Because XZ did happen. Florian: Yeah. Jonathan: Now that's, that is- Yeah essentially the same sort of attack, and it's just, it's a demonstration of how much more complicated it is. Florian: Yeah. Yeah, but you mis- ... So one of the things that I have as an issue sometimes is w- it's so easy to point to XZ, making it into Debian and saying, "Hey, the Debian model is broken." No, it isn't. It's a human system. Failures happen. There's a, this... if this doesn't happen all the time, I- I... Jonathan: Yeah, absolutely. Yeah. I would say that the fact that it, that only one pack- that we know of, only one package made it in and it got- ... caught that quickly is, if nothing else- ... it's a vindication of the Debian system. That their- Florian: Yes ... Jonathan: it works and their paranoia has some basis in reality. Florian: Yeah, definitely. Definitely. Oh. Yeah. Jonathan: All right. Yeah. What... So going back to our road map here, and we just, we touched base on it just a moment ago, but when someone is looking at Rust, what are some of the things that they're gonna worry about? What are some of the potential problems with Rewrite It in Rust? Florian: I don't think most people take a problematic stance at adopting Rust. They usually adopt Rust because they already have a reason with what they currently have. Jonathan: Right. Florian: You rarely make, you rarely migrate away from something that works for you. So they're more looking at what are the benefits than they're looking at what are the costs. In- interestingly- ... having been both in the growth phase of Ruby and in the growth phase of Rust 15 years ago people would be more like, "Come on, it's a nicer language. It has nicer syntax. Let's go." Nowadays, the questions you need to answer is will I break bank by, in the migration? With that, we've seen multiple companies actually go down in- ... between 2010, '10 and 2020 by just dying in, in a rewrite. Like this, it, it even became a topic, should you even do rewrites or not? So what are the benefits? And then it's usually operational concerns. How do we actually integrate the technology? The question, do we rewrite or do we not? W- what's the benefit? What's the disadvantages? If we're not rewriting, how do we integrate our old code base? Is that possible or not? And there's always also the question, I think that's something even even though my business is obviously in, in the Rust side of things we've had a number of clients where in the end our advice was probably- Don't do it you should first get someone... Yeah, you should first get someone to look at your current code base and maybe fix the issues that are in there. It will probably be more effective, ... in, in the immediate, And that's the other problem. If you're, if you already lost track of your software product, to be honest, the rewrites become more hazardous because you don't even know what they should do. You're just adding another problem to... It's a problem. And just to be clear, if you're in the services consultancy business you're never called when everything's in order. Jonathan: That's Indeed. Florian: And that, it, that's okay. There's like very often you have very successful companies who are like, "Okay we got successful overnight and we need help." So it's that, it's- Yeah ... it's- Oh, that's... Yeah ... I always take the stance of very few companies ended at the place where they are out of incompetence. Mostly it's competence, but also y- you know, you can only do that much in a day. Jonathan: Yeah. It's that, it's Florian: that- You only know about so many things. Jonathan: It's that combination of like just the inertia of what you've done, the fact that- something has snuck up on you, whether it's your growth or just the passage- ... of time has snuck up on you. And then suddenly you look down in your code base, it's "How did this happen? How did we get to this point? We need some help." I, I find it interesting, and maybe something like this will be the show title, but Rust won't necessarily save you. It's not gonna- ... fix all of your problems. Florian: Of course it will, but, but yeah it is it is something definitely where there's also, there was also a hazard when we grown Rust, and that was actually something that we, I was not only part of the core team, I was part of the community team and very much in lead there. And community team sounds so... You're basically the outside representatives. It, it always sounds like these are not the real core engineers of the project. Jonathan: Yeah. Florian: Quite the other thing is true, because you need to be so up to speed with so many technologies- ... because someone comes at you and says "How does Rust work on a GPU?" And I'm like, "I know nothing about GPUs, but maybe I should read up quickly." But there's always... you develop this feeling for, are people just there for the why, but do they have an actual problem? The other thing is that we always told people is, "Don't speak to people who currently don't perceive any problem. It will not be a useful discussion." But if someone says, "My current software stack, I really don't like it," then there you can make an offer. It sounds like marketing. I'm actually... I would actually open s- have open source projects to be more okay with doing marketing, but doing marketing in the sense of a friend of mine once said, "If you're not talking about a good thing you're doing, you're depriving people of the ability to use it." Jonathan: Yeah. Yeah, absolutely. Florian: And marketing on that level, not marketing in the sense of advertisement tracking and so on and so forth. And I used to work in advertisement. I know where the bad rep of advertisement marketing comes from, and it's very- Yes ... I g- I very much understand why people hate it. It's more like the go to conferences, go to a C++ conference, talk to people and be serious about your project, but also be serious about what they're doing. Jonathan: Yep, yep. Absolutely. Have you found any places where there's a, I don't wanna overstate this, but let's say friction or problems trying to make the business case and the business side of things work with the community side of things? Florian: You mean yet now company-wise or, Jonathan: So i- I mean it as an open-ended question, but you talk about being- both a core engineer and a community engineer and the- ... the difference between those two. And, either in that, those roles or now- ... into the business side of things. Florian: I would even say these things are always conflicting. I cannot just personally I can't, I cannot spend as much time at community events as I want because I- kinda have to, Jonathan: Maybe a Florian: business to run ... to run my business. Jonathan: Yeah. I see Florian: that. Yeah, but also needing to represent a business. This is actually one of the reasons why I why I left the Rust Project was- ... quite simply because I was in calls where people would ask me "What's your stance on this?" And "What's your stance on this as the director of the foundation? What's your stance as this, as the director of Ferrous Systems? And what's your stance on this as a core team member?" Because I was representing the Rust Project at the foundation, not my business. And that's a constant friction. It's a it's a... it's not that much of a problem. It's a conflict of interest. It's an open conflict of interest. Jonathan: Yeah. Florian: I don't think conflicts of interest are in any way a problem. It is you need to be in a meeting, and you would then need to say, "This is a business meeting between you and Ferrous Systems. I don't represent the Rust Project there, so the only opinion you can get out of me is- ... mine. I can tell you the stance of the Rust Project. The stance of the Rust Project is this one." And they sometimes diverge, and that's fine. The other problem is obviously on any pro- project I was experiencing that hard when I was running the Rust Allhands. You have... So the Rust Allhands the first ones were a one-week meeting where people finally met. And for everyone who was d- at who was contributing to Rust as a hobby, that was a week of vacation. Qui- quite literally they could go to their boss, and sometimes their boss would be like, "Okay, go there. It gives you five days for going." So they were like, "Finally, I have one week where I can only do Rust," and it was their first week where they could do that. And on the other side, you have people who are full-time employees and who are like, "Oh my God, that's the 70-hour week," right? And- Yes ... th- these Those are problems, but they're inherent, and I think you need to be very open very open about them. But that, for example, also meant, and you do need to find your way around this. The community mailing list that I was managing at the Rust project frequently got requests for trainings, which was my business. So the community team internally had a rule of we have a list of training providers. Training providers who are in the community team will not reply, and we will send out a structured list as, "Hey, by the way, these are all the people to call," so that I, that we don't snatch business out of the project just because we are at the right, because we have access to the right email address. Jonathan: Yeah. Florian: And the Rust project is actually quite good about having these things sorted. Yeah. Jonathan: It sounds like, and maybe you have been a part of this, but it sounds like there's some people involved in that project that understand the business side of things and understand the- potential conflict of interest and the- ... sane ways to handle that. ... Jonathan: It's k- it's refreshing actually that an open source- ... project that is also a business is not a train wreck. Florian: Yes. I actually- Thank you for that. That comes from also you may have observed that the Rust project has absolutely no problem with being political. And I don't want to be divisive there, and that sometimes leads to conflict, just to be clear. But that also means you have people there who say these things, like these are in a way political problems. Like we often put political problems more in the social side of things. But this thing like corporate politics and all of these things, and you have a lot of people who are actually well-versed in that. You may agree or disagree, but yeah. Yeah. Jonathan: Yeah. I've taken the stance over the years that an open source project should try to stay out of the politics and culture wars as much as possible. And I know like even that in and of itself is a political/culture war statement, and I acknowledge the irony of that. But that's just where I've come to with my projects. Like I'm not going to- Yeah ... put a statement out on what I think of any politician or... And I don't think- Yeah ... in, in my opinion, it's not the project's place to do that. I think it's a distraction. But at the same time- Florian: Yeah, but no and I didn't even want to to go there. That's a whole other discussion for a whole other - Right ... I just think like that sometimes I think people would wish those things wouldn't exist, and we could run a successful open source project that runs a new programming language without having corporations involved. And if, I've literally been in discussions where people express that view. Yes. And the problem I see there is if you're not set up for these companies have goals, and they have set these goals internally, and they are on your project because they want to fulfill those goals. And like that level of, hey, they're, this person might make that statement because it's their goal setting internally at their company, even if they personally disagree, that these effects you can't argue those effects away, and you need to deal with them. That's all I'm saying there. Sure. And the Rust project has been, from the beginning, very much aware of these effects. That's and I personally, that, that's my view. I actually feel that under a political mindset in a, in the sense of how I operate in my brain. That's all I want to say. Jonathan: No, that's fair. That's fair. Yeah. Let's talk for a minute about safe Rust. Florian: Yes. Jonathan: And and there's this feeling among some people that, "Oh Rust is safe. It's a memory safe language." Which, yeah. That's great, unless of course you use the unsafe keyword. ... But then you have in the regulatory structure and the safety, like safety engineering structure, something being safe has a different meaning. Is there Rust in cars? Does Rust run airplanes? In those serious positions of safety engineering, has Rust made a foothold there, and what does that look like? Florian: Yes. Mostly in what's called quality managed or a little bit of mid-safety at the moment. The overloaded term, like, of safety is obviously a problem. Rust says it's memory safe, and there's a very clear definition in the language what it actually considers memory safe. Jonathan: Right. Florian: Rust on the other hand also has a culture of correctness, I would say. People go to great lengths to use other facilities of language to make sure something's correct. We also have a strong type system. We have a type system that does not have that much baggage, because w- it's a new language. That's that comes with that fact. And we see a lot of interest in regulated industries. Like a lot of the things we're doing is automotive, but also a lot of, industrial safety. Where the definition of what safe software is it's also different. It's usually called functional safety. So in that front, it's more it's less the... So memory safety is a part of functional safety because it could impact the functional safety of my tool, but it's so much more. It is- ... even things like this plane will not take off at under 230 miles. And your software needs to deal with that. So a lot of outside requirements actually being put into software. The speed is completely random. I don't know what the normal takeoff speed of a 767 is. We're Jonathan: not aeronautical engineers. Don't quote us on that. Florian: Yeah. Jonathan: I went to the I went to the most recent Ubuntu Summit, and one of the speakers there was talking about trying to use the Linux kernel in that sort of- ... safety engineering sense. And the I don't remember what company he was representing. I wish I could remember that. But anyway he was talking about the lengths that they were going to try to re-engineer the kernel in such a way- ... that it could be considered safe for, like- ... automotive use or some of those industrial uses. And it was- ... it was eye-opening, like the lengths- ... that they were talking about having to go to, to- and again, not just memory safety, but, you also have real-time guarantees and other things that- ... that have to be considered. It's a lot- ... for these really regulated industries, but also the ones where there's a, there's engineering definitions of what it means for something to be safe. Florian: Yeah. So the interesting thing I like I have not spent most of my career in safety critical. And one of the things I liked when moving over to it is first of all, one of the things I find interesting is that there's a lot of things that would've been done for high safety software that are now table stakes. Big open source projects are practicing methods that are definitely on the level for safety. But the one thing I like in safety engineering is that the currency is the argument. You can basically do anything, like obviously not anything, but they're open to quite a lot of approaches. What they're also interested in is you also bring me the argument why you think this is safe, and what the safety standards actually say is, here's a number of basic techniques that we consider helping you towards that goal." So standards are not just checklists, but they're also a source of inspiration. Actually the the avionics standard, the DO-332 has a very good list of this is what can go wrong in dynamic memory allocation and the things that you should look at. Sadly, that standard costs I think $300 or something like that, so most people haven't read it. I can actually recommend reading it. It should be ... we could teach this at university. It's also not super magical in that sense. Yeah. Jonathan: It I went and looked it up. It was actually an Nvidia employee talking about ASIL B qualifications for Linux. And yeah. Very interesting. If somebody wants a quick primer on like this safety engineering stuff, that's actually a really good talk. He did it in just over 20 minutes. It's actually a really good talk- ... to check out to get an idea of what all is involved with that. I found- Yeah ... I found myself sitting there listening to it going like I I'm a software rebel. I tend to not believe the different "This software is secure." E- I've heard that before. And when he's talking about, "This software is safe and this is how we prove it," I'm like, "Sure it is, bud." And also "To be safe you have to do this and this," and then it's built into the spec. And it's my rebellious side comes out again and I'm like, "No, surely not." Florian: On the other hand, I've heard w- when I started in s- safety people were like, "Open source and safety is incompatible." "There's the bazaar out there. We do structured requirements management," and so on and so forth. And so my joy over the last five years is seeing that move very drastically. The Eclipse Foundation is now one of the biggest foundations in in open source in, for example, automotive. And I'm like sud- suddenly everyone's around. Suddenly there's the BMW, the Vectors, and all of these companies are around and they're like, "Okay, let's consider, like, how many of those open source things can we reuse?" And actually the one thing I want them, I want to convince them of more and more is then engage in those projects. Like- ... our open source projects would do better if they have a... I don't know if they would do better, but if it's a s- if it's a project that you find so high value that you wanna put it into a car, but you have these f- five things that are missing- the usual mode in open source is go engage, talk to the maintainer, see if they accept it get involved in the project. And I know a lot of people in those industries who are extremely sharp thinkers about particularly the where is my software deployed and what can happen in the physical world around it. So they, they have such a big mindset of it's not just my TCP/IP connections, it's actually what does the camera see? What can ... w- what's all of the possible permutations that my camera could see, and how does it need to react in that case? So yeah. In the other thing that you're saying though is also very much true. There's this kind of like safety standards are a catechism and this is what you need to follow, and you can't go left and right. And this is where open source definitely doesn't fit in because someone at the Rust project will say, "Hey, I'm a student somewhere in Ottawa, I'll just do whatever." Jonathan: Yeah. There, there's another... I forget who originally made this observation. But open source is a license. Yeah. Open source is not necessarily how you run your community. So like you can run, and in fact we're seeing projects do this now for various reasons. You can run an open source project and not accept- any outside contributions. For various reasons there are several projects that are taking that stance, which is wild because it's so different from the way open source has been run. But that is perfectly acceptable with the license. Florian: Some of our projects, like our we have our own Rust compiler distribution, and that one does only accept contributions from a limited list of people because we're going into liability- For these things. So we're actually encouraging people, if you want to change the Rust compiler, go to the Rust project. We will vet it later. So it's also, you were talking about conflicts earlier. For us it's a way of staying outside of like not to be a drag on the open source project as well. Or no, we are an open source, we are an open source project. We don't wanna be a drag on our upstream. And projects I think that's a good thing that projects are thinking more and more about their policies. And yeah, as you say, open source is just a license in the end. Yeah. Jonathan: What's the intersection been between Rust and the current AI craze that we're in? I ca- I say that slightly derogatorily. I've actually pretty much become a convert because it's just so powerful these days. I've written multiple new features by just saying, "Hey, Claude, do this for me." And it's "Oh, that code's actually not bad. Here, let me fix it in a couple of places." It's- astoundingly useful. Florian: Yeah. I think the Rust community is a little bit tired of sometimes being in the middle of those hype cycles. It was already in the middle of the blockchain hype cycle, and we were- ... we know where that went. And some people were incredibly aggressive about this. So- Yeah ... if you see some pushback just be aware that we just went through a five-year phase of people telling us, "If you don't do blockchain, you'll be out of business in the next three years." Or you'll be out of job. And now people are coming on, "If you don't do AI, you'll be out of this, out of job for the next five years." So there's part of that the other thing though is Rust is actually quite benefiting from that. There's a-- Because it seems like AI models seem to be quite good at working with strongly typed languages. We're a strongly typed language. We're in the hype field. We know that Rust is being used in multiple companies internally quite heavily, so there's something coming out of that. So it's also a funding source. The Rust project is- that's nice. Yep. The Rust project right now has struggles or struggles in the sense of they have a debate about what their LLM policy is. I won't say much on that because I'm- Sure ... I have a number of people who are having those debates, and they don't need someone else going on a podcast I get that, yeah ... stating their opinions. But what I can say is I have seen multiple people who were rather aggressively either saying it needs to be a hard ban or it n- no one needs a policy at all, and that just isn't feasible for that project. Yeah ... a project of that size or a community of that size needs to position itself. Internally for our safety critical products, we're currently not using any LLMs at all just for the reason that they are still faulty in ways. And in my experience with them, I find, I regularly find pretty outrageous bugs in the things they generate. And I don't want to be the first person who signed off a piece of software that using an L- an LLM hallucination has run a car into a tree. Yep. That's- Because that will be on me. But it is pretty much in the middle of it just like any other community as well. Here over the C++ folks are discussing that thing left and right as well. And but one of the things it, again, coming, circling back to the beginning it makes systems programming languages more relevant again because one of the reasons why people are switching to Rust or switching back to systems programming languages is not even the whole safety bits and all of these things. It's power consumption. These things burn, like every cycle you burn if you scale it up by a million is, Yeah. Yeah. Jonathan: It really Florian: makes a difference. And performance engineering gets interesting again, and this is where LLMs are actually pretty bad. Performant code? Not so much. I have no int- Yeah, Jonathan: that's- Yeah, Florian: I don't... Jonathan: Yeah ... it's an interesting observation. Yeah. We talk about LLMs making lasagna sometimes. They wanna be so, ... verbose and easy to understand it ends up with this really weird layered code. I've also observed that languages that are, like, open source, well-documented, and have a big corpus- out that's easy to get ahold of on the internet, those are the ones that tend to do really well in the LLMs. 'Cause obviously- ... it's got a lot of sources to pull from, and that kind of- ... describes Rust pretty well. So I imagine that the the modern LLMs do fairly well in writing Rust code just because they were able to train on so much of it. Florian: Yeah. And I used to do, I had a phase where I worked as a, in consulting for a full-text search, particular Elasticsearch. And the... I think a lot of people are very focused on the generative side of the whole thing. Jonathan: Right. Florian: Which you can consider problematic, and a lot of people are seeing that as an offense of basically you're replacing the human act of writing, which is I think, very cult- this is culturally pretty a pretty big lift. On the other hand, on the data retrieval side, I used to it I find it quite impressive and a big step up. On the indexing side of things, it's not that much more than a more sophisticated index. Very regressive view of things, but yeah. Yeah. Yeah. So I'm actually way more interested in the in the data res- retrieval- Aspects than in the generative aspects. Jonathan: Yeah, that's, that, that is something that's easy to easy to forget, that there's more to the modern inference than just using an LLM for generation. ... The techniques that have been developed, that there's they have more uses than just a really good chatbot. Yeah. Yeah. Are you mentioned briefly Rust on the GPU. Is that a thing that people- ... are doing? Can we use Rust to write the LLM and run it right on the ker- on the I guess as the kernel on the GPU? That's how these things work these days. Florian: That's a long way to think. Rust actually had a pretty active machine learning community very much in the beginning. Until Google came along with TensorFlow, which just kills off all the startups- Yeah ... because everyone says "We can't compete with Google." We can't Jonathan: compete with Google. Florian: Which I actually don't think is true, but, No, that's not at Jonathan: all, but Florian: in the investment market it has that effect. And since then there's, there have always been people who have built Vulcan wGPU implementations and all of these things. The, there's a number of approaches to actually run Rust on GPUs most of them by using a custom compiler backend. The Rust compiler actually is very well suited for plugging in an, in multiple code generation backends because it used to actually carry multiple code generation backends for quite a while. And but most of... And so there's an active community out there, but they run very shallow compiler forks effectively. So they build these backends but only on, for example, very specific versions of of the Rust compiler's nightly. But those things are usable. So there's a, there's an ongoing small community forming. The Rust project itself, this is actually something which, but that was a decision that I was part of at the beginning, did not invest much into that just for reasons of you need to focus on something that you can actually achieve and not have you- Sure things in different pots. It's up and coming, I would be say, I would say, but it's up and coming since about six, seven years. Yeah. Yeah. Jonathan: Have you followed any, any of the sort of Rust experiments like, and I can't remember the name of it, but the let's rewrite a kernel in Rust. Oh, what is that called? I can't remember what they call that project. It's got a, it's got a- It's multiple ... it's got a metal themed name I think, the one that I'm thinking of. Florian: Redux. Jonathan: Yes, Redux. That's it. Of course. Florian: Yeah. I've not followed Redux that much. I'm casually aware of a couple of people using it. But writing operating systems in Rust has been quite common. There's a number of microcontroller OSs, DockOS, which even has a commercial counterpart now called Oxidos. And all these things are coming along quite nicely. I think for most people a C based OS is currently okay. People are s- first starting to, for example, write drivers in Rust and things like that. It's also, it's one of those levels where Rust still has to answer quite a few questions. So it's more you need to know the compiler behavior. You're not always sure if that, if it's going to stay this way. You m- mostly will be on the nightly compiler because you want some unstable features and things like that. Those gaps are actually pretty quickly closing, so I would say in two to three years I would not be surprised if the Rust compiler's fully capable of writing a kernel on stable. Yeah. And yeah Jonathan: I know- Florian: These projects are coming along late. Jonathan: Yeah. Yeah. Yeah. I know that in places like the Linux kernel famously- Florian: Yeah Jonathan: Down in the depths of that, there's some really ugly things g- go on, like inline assembly and- ... all sorts of sort of dark magic hacks. Because hardware is hard. And and so I whenever somebody talks about "Let's build a kernel in Rust, it'll be memory safe," that's not how that works. You have to do so many questionable memory unsafe things to be able to talk to real hardware. That like I I've asked the question is it really even worth it to do the rewrite in Rust? Are you getting enough out of it? But that's just me. Florian: There's the second question is should we still work with operating systems from the '90s? W- the answer may be yes and no, but I think that's a m- are there other designs out there? And then we may write one of these new designs in Rust. We're seeing that in the microcontroller space. So- Sure ... people are actually actively writing new operating systems. The the thing is, I think it's also a misunderstanding of Rust. Hardware is fun- from the perspective of a compiler, fundamentally unsafe. Memory mapped registers behave randomly over time from the perspective of a compiler. They do things. They don't behave randomly. They obviously behave like the hardware is specified, hopefully. Jonathan: Sometimes. Some, sometimes they do. Florian: Yeah. I I... and then Rust's game is much more encapsulating all of these things. So the language gives you facilities to say, "Hey, by the way, this is this is a UART. This is how a UART behaves," and build a safe wrapper around it that is hard to misuse. Eh I would say the thing that pe- people forget about Rust is it als- is also a very strong e- encapsulation language. And that bit obviously So at the very bottom of the stack, the first question is: how does that hardware actually behave? And it's being answered in C and in Rust in the same way. I am actually, on the other hand, surprised of how many people nowadays, even if it's a nascent community, actually go and say, "We'll write it all in Rust because then it's all in the same language." So it's more of a... So that reduces the tool set we're, that we're using. I would've expected that we are, we've spent way more time in mixed land between C and Rust. Yeah. But those are... This ends up, again, as I said earlier we're now in a place where organizations that do actually spend money on these things and time, or even open source projects spend quite some time thinking about these things, thinking, thinking these things through before they actually do it. Yeah. Jonathan: Yeah. Are there any complications with Rust handling different iterations of an ISA and the different instructions that are available? And I'm thinking of things like, AVX-512 and the... For the longest time people thought, "Oh yeah, that's not very interesting. That's only for data science." And then someone came along and said, "Hey, you can do string compares with AVX-512." And some of those other things We can make things so much faster now. And getting that, getting support for that in, the C standard libs and the kernel, and then- ... being able to have one binary that can on the fly detect, oh, this processor does AVX-512. I should use this new advanced string comparison routine. I- is that sort of stuff possible in Rust too? Florian: Yeah, that is possible. This is something where Rust came after these concerns and C came up, so we can learn a lot from that. The, on the other hand with everything that touches ISA, we're based on LLVM, so- everything we inherit a lot of problems and a lot of benefits from LLVM. And if, for example, LLVM is not as well, as good in using some of these instructions as, say, GCC, you're out of luck. Or y- you can always hand code things not only in assembly, but also, for example, by using compiler intrinsics by saying, "Hey, please insert these instructions here and there." But- ... it becomes very detailed. Yeah. Jonathan: Yeah. Florian: Yeah. Jonathan: I was intrigued to see that same process happening in, in RISC-V, which we're gonna- ... we're gonna talk to some RISC-V people here coming up in the next few weeks. Cool. They, I think RV23 is what they call it, and that's their essentially next version of the ISA that's got- a bunch of hardware accelerated things. And, so with- ... with Ubuntu they've just said, "We're only gonna support RVA23 going forwards because it's so much better and it's the n- ... the next thing." Yeah it's just, it's real interesting to see the way that the different projects, the different languages have dealt with that. Florian: The benefit you have in an early stage, and I would still consider RISC-V even though it's, I know we're working on this for a decade now, it's still early stage is that usually communities across the board, even corporate communities still have a habit of, yes, we're betting on a new technology. There will be two to three migrations- ... Florian: To, to a better thing. So you're still in an environment where things were not... Like where it's totally okay to go to a manager and say, "You will probably have to do a compiler update in two years, and you will probably have to re-engineer some of your software in two years." But it's manageable. It's these, as long as you can say it's these and these and these things. And can give people a rough feeling. I g- I, I always say that you're always dealing with humans. It's not there's always this picture being painted of industry always wants everything to be stable. No, they also know that they need to change and that new technologies come with that. The biggest the biggest fear I have for RISC-V is they're going up against some pretty big behemoth that can basically just undercut them at every at every corner. So let's see how that works. Jonathan: Yeah. I think the advantage RISC-V has is that those behemoths are already using RISC-V in their technology. Like Intel- Florian: Yeah ... Jonathan: famously, the Intel management agent is a little RISC-V core. Yeah. And it would not be... And now one could make the argument whether or not Intel is the behemoth anymore, but RIS- RI- I think RISC-V is here to stay just because it's- Yeah it's in so many things. Florian: And just to be clear that I don't wanna spread any kind of FUD. It's just like these things that grind my gears. It's how- Sure ... are they approaching this problem? As you said, they've already been around for a decade. They're, as you say, they're here to stay. Also this is something I think the time where people who are really into these technologies, like the, there was a time in Rust for about two years where I knew where Rust was being adopted, like a number of larger companies that would later come out. And most of the time, even everyone in the Rust project knew. And you're sitting there and you're like, "This isn't very public news yet that we basically being used at all FAANG companies." And you're still you're at conferences where people's "Yeah, this is still this nascent language that is used nowhere." And you're like- I've seen- ... just cannot tell. And I- Yeah ... I probably think the same thing at RIS- is going on at RISC-V, maybe. Jonathan: Yeah, th- that's probably true. I've gotten to look just a little bit behind the curtain there and I've seen some things where- ... it's, a lot of hard drives for instance are running RISC-V cores. ... Intel is running RISC-V cores inside of... I I joke, it's not really a joke, we don't have any x86-64 processors anymore. None of them are native, and it has not been for a very long time. None of them are- Yeah ... natively x86-64. They're all something else and they're emulating x86-64 instructions, which is a wild place to be, but that's, that's fairly accurate. All right. Let's z- let me think. You got a conference. Let's talk about the conference. What is Oxidise? What's the story with that? Florian: Oh. Oh I c- I ran, I usually joke I sometimes accidentally run conferences. Jonathan: Just fall right Florian: into it. So Jonathan: I- Stumble into it. Florian: Yeah. I I had a Ruby conference called EuroCamp, and then later when I was starting in Rust, people were like, "We need a European Rust conference." I got going to run one. And that was RustFest. Which actually ran for 10 editions before the whole pandemic basically killed it off. Yeah ... it's been replaced by other events in Europe. The European conferencing scene does well. But at some point I figured out I actually want to have an event. And that was in parallel to RustFest. I want to have an event where we're talking more about things that people are actually doing with Rust, not necessarily, but also that often happens to be like what do you do in your day job with Rust? So what is the thing you're building? Or if you are, for example, building a major open source project what are you doing there? And I wanted to have a conference where it's less about, okay, this is what the language can do, this is what the type system does, and he is, he has a plan for making the Boa checker more powerful or something like that, like all of these things that are flying around. But more like having a place where I can, where s- someone who comes to me and says "I'm putting Rust on microsatellites." I'm like, "Then go to this conference, talk about microsatellites, talk about it." And I'm trying, I always try to encourage speakers as "Actually, don't hold back. Don't do an advertisement talk for your product." So it's not about that, but, "Hey, please talk about the actual full thing but that you're building. If it's a piece of hardware, bring the hardware, put it on, put it there, and then say, 'This was my actual experience in good or bad,' like proper in a proper engineering fashion. That was my experience of putting Rust on this thing." But not speculative, but more as a you share your experience or your current struggles with w- with Rust deployment. Summed up as the conference for applied Rust- ... in that sense. And I wanted to have that, and the, decided to basically run my own conference cycle and starting it at that time. Actually that was also hit by the pandemic in a weird way. It had multiple experimental editions over with s- some online formats. And nowadays- Virtual Jonathan: conferences Florian: and hybrid. Yeah ... it's a yearly thing that happens in Berlin. Yep. Yeah. Yeah. Jonathan: Cool. It's in, September? Florian: It's in September in Berlin. Jonathan: Yeah. I probably will not be able to swing making it out there. It's too far away from too far away from day job stuff. Florian: If you're going to Embedded World, we meet at Embedded World then. Jonathan: Yeah, absolutely. I don't remember if I ... I don't think I talked to you at Embedded World, although I may have. I talked to a bunch of people at Embedded World this past year. But I- ... I'm pretty sure that's how this conversation came to be, I got somebody's business card at Embedded World. I've joked- Yeah I need to print business cards that have Meshtastic on one side and Floss Weekly on the other. Yeah ... that way I can give it out and people know who I am. Florian: But it's the you'll end up in the open source and Rust corner. So we've actually managed to merge all of our booth. We have a cluster of companies there who are actually all household names in the open source- Jonathan: Yeah Florian: Scene. And Embedded World is actually, from that perspective, always interesting because embedded used to be that space where, yes, open source exists, and there's maybe this little bit of Yocto and things like that. But most of it, it was you buy, you it was very much a culture of proprietary software, proprietary- Right tools. You buy your BSP and things like that. And over the last five years, that has changed so much. Yeah. Especially there in Hall 4. There's so many open source companies- ... running around. Jonathan: It was very cool to walk around and just "Oh, I know that name. I know that name. I know that name. I'm gonna go talk to all of them and get business cards from all of them." Yeah. It was a lot of fun. Yeah ... some interesting people there too. I talked to QNX and it's- Yeah ... like I would love to have them on this sh- in fact, we have the guy from QNX on the show, and he's "You realize QNX is closed source, right?" I'm like, yeah, absolutely. But you guys do ... " They're a big part of Eclipse and, ... some other things that they're really big into. We'll have them on and talk about that. It, yeah, but it is, it was really fascinating to Florian: see. Are you going to have them on? Jonathan: I'm gonna have QNX on the show. Florian: Oh, very good. Jonathan: Yeah. I will ask him. I'll be like, "Hey, what's up with this? Why isn't it open source?" And I told him when I talked to him about at the booth. I'm like, "I'd like to have you, and this is what I want to do." I'm like, "Let's just lampshade it. Why in the world is QNX not open source?" I'm like, "I'll tell you up front I'm gonna ask the question and you can have a good response for it." He's "Okay, that's fine. We can do that." So- Florian: Oh, yeah ... Jonathan: look forward to that. All right. So- Yeah, Florian: great ... Jonathan: what's some weird things that people have done with Rust or something that people have asked you? And I, obviously you can't share every story, but some stories you can share. What are some weird things that you've done with Rust or that you see people do with Rust? Florian: Oh weird things. As we, for example, write a part of a writing Rust analyzer, the IDE, we know a l- a lot of Rust code that's incredibly weird. Yeah. But I think then weird things with Rust, Or Jonathan: s- or surprising things Florian: I'm just trying to figure out what's Jonathan: What you're allowed to talk about, right? Florian: Yeah. No not allowed to talk about it, but just trying, it ... there's fun place things all of, all the time. There, there was someone who built a compiler back-end that compiles to Redstone, which is that Minecraft, that computer built in Minecraft. Jonathan: Oh oh, I know what you're talking about now. Yeah, that's pretty good. That's a pretty good Florian: example. Things like that. Jonathan: Rust Florian: in Minecraft. We have people building all kinds of CLI tools. Someone over the pandemic thought they're going to rewrite g- new core utils, and suddenly they're successful things like that. A lot of these things aren't actually planned. I have, I've personally, I've started very much at the beginning, I've rewritten SL, the I don't know if SL? Jonathan: It's, it's- Florian: Most important ... Jonathan: I don't know if that's a core util, but yeah, that's, Florian: It's a, it's a- I'm trying to remember exactly what that- If you mistype LS, it runs a steam locomotive. Jonathan: That's right. Okay. That is not what I was thinking of, but yes, I know what you're talking about now. Florian: Yeah. But I have a patch in SL, which is my claim to fame in the open source community. It's not that Rust stuff. I have a patch in SL. The Rust community in general is pretty playful, especially on the embedded side of things. G- the the Hulks Robots team which we're supporting th- they're doing robot football. Like the interesting football, not the thing that's currently on TV. And one of their robots is built, Jonathan: Wait ... PyTorch. Which kind of football is the interesting football and which is the one that's currently on TV? Now wait Florian: a second. Oh you called it soccer, right? Jonathan: So we call f- the European football we refer to as soccer, and then there's American football- Florian: Yeah ... which is gridiron. No, so yeah, so it's a robot soccer team. Sorry. Jonathan: Okay. Florian: Yeah. So they have tiny robots and- I'm still trying to work out whether I'm Jonathan: offended or not. Florian: Yeah. It's all good. So they have a robot where the firmware is written in Rust. Oh, Jonathan: that's good. Florian: Things like that. We... I there's a number of things where people do... So there's a whole subset of where people are using Rust in medical software. But because they've done it very early, they've basically used it for prototyping. Jonathan: Okay. Florian: And then the and then have rewritten everything in C at the end of it be- because that's safety qualified, so it's safer suddenly because yeah. You paid for the tool, all of these things. Jonathan: Checks out. Florian: Ah, weird. Jonathan: I know we got Linux- Yeah ... on Mars. Was Rust part of the part of any of the Mars missions? Do you know? Florian: Yeah. Yeah. Jonathan: Did the helicopter run Rust? Florian: Exactly. Jonathan: Yeah. Florian: It's going everywhere, but the thing is I don't, I wouldn't say that there's also a ton of weird things that people are doing with C. Sure. And it's I think both of the communities in that f- on that front have a pretty similar vibe going on. And I find- ... I find especially the hardware communities to be, i- in the positive sense very hacky, ... and stuff. We're seeing Rust in the demo scene now. Yeah. Jonathan: Yeah, that's an interesting, that's an interesting one. I'm trying to remember if I've seen anybody port Rust back to the Commodore 64. Florian: Oh, I can totally see that. Jonathan: Yeah, that'd be fun. Florian: Probably. Oh we had someone who we have someone who maintains something they call Visual Rust, and it's, Of course they do ... it's V- Visual Studio 90 ... So it, it's Rust compatible with Windows 95 and Windows 98. That's great. Okay. And I just needed to check whether that's actually public. The fusion reactor they're building out of Boston, Fusion Commonwealth? There's some of their simulation software is in Rust and beta and on their GitHub. Very cool. So there was like fusion reactors in Rust, why not? Why not? ... people because Rust has some influence from Python, people are using Rust as a fast Python in a way. Jonathan: I can see that. Yeah, I can see that. Yeah. Python s- surprisingly is used by a lot of academics and researchers- ... like that, and it's been impressive the performance that they've gotten out of it. But I can see someone wanting to go to something that feels similar but gets a little bit to the next step on the performance. Yeah. All right. So is there anything, and this is another one of these hard questions. You gotta do some set math in your head. Is there anything that we didn't talk about that we should have? Anything I didn't ask that you wanted to cover? Florian: No, I think, ... I think I'm actually good. All right. With a good conversation running through everything. Yeah. You didn't ask about the rewriting that much, though. Jonathan: Oh, the rewrite it in Ru- that's just a running joke. It's obviously it's happening a little bit, especially things like uudutils and the- ... we talked about the replacement kernel where they're trying to rewrite it in Rust. Florian: I actually have something like we, we were part of the Sudoers rewrite project. And the one thing I said is like these things aren't often that competitive as they are. So first of all, the, one of the reasons why these things are getting rewritten is that because there's actually people asking for it. But one of the good things particularly about the sudo rewrite was we, first of all, before we started with this, we got in touch with the sudo maintainer, the sudo, the, like singular. That, that singular. Oh, Jonathan: that's a little terrifying, but okay. Florian: Yeah. And we were like, "Hey there's we got a request to rewrite sudo." And the response was basically, "Oh, more eyeballs on sudo is always good." So both of these projects are actually talking. And he also advised us on a lot of things where we can... this is one of the benefits of a rewrite. Sudoers does not support everything that sudo re- supports, and that's actually a conscious decision. Jonathan: Get rid of some of the cruft- Florian: Because- ... Jonathan: from over the last 30 years. Florian: It's not only cruft. It's if there is a platform where there's two or three users that really need it, sudo can't kick that feature out. We can because those people just don't move to Sudo RS. Jonathan: Right. Florian: Those people just use old Su- Sudo. And the, I think the most important thing there is w- that these rewrites are done with the maintainers. And the other thing about UUtils, it was really just someone was searching for a thing to do while they, they were at home one time and were like- I know, we Jonathan: did. We've interviewed him. We- Yeah ... we've had both UUtils and CoreUtils on the show. It's been- Yeah ... it's been a lot of fun. The- Yeah ... with the, with Sudo RS, have you guys done the same thing where you share a a test suite and, go back and forth between the two to make sure that, you have your test coverage and you do the things that you intend to do the same way- you pass the test- Yeah suite in the same way? Yeah. Florian: Oh, that's a good question. Now you caught me. I currently don't think so. I need to check that, though. Jonathan: Okay. Yeah. That's one of the, that's one of the really interesting bits of synergy between the two Core Util- CoreUtils and UUtils- ... is that there's they share... I don't know if they literally share the same code, but they have this shared database of test suite tests. Yeah. And it's one project will implement a new test, and then go to the other project and go, "Do you have coverage on this?" And if no then let's add that test suite here." And then, you know- they find all these weird edge cases because you're- ... re-implementing it, you find an edge case, and then you go to the other project- Yeah ... and "Were you aware of this edge case?" And sometimes the answer is no. There's been bugs fixed over in CoreUtils because UUtils was implementing an edge case. And, Yeah, as it should be ... it's been pretty neat to see the two projects coexist and both get better because of it. Florian: As it should be. Jonathan: Yeah, absolutely. Yeah. Okay. I gotta ask- Cool ... two final questions. I will get emails if I forget this. What is your favorite text editor and scripting language? Florian: Oh, let's let's start in the back. I used to be in p- president of the German Ruby Association, so it's obviously Ruby. That, So I always joke all languages that start with R-U are good. I had a fun, fun thing. O- one fun thing. I was actually in a way recruited to the Rust project by Steve Klabnik, who was previously one of the biggest contributors to Rails. So I I'm very thankful to the Ruby community. So that will be my favorite. Problem is most of the people in my company don't agree, so we're producing very little Ruby at Ferret Systems. And my first business was a Ruby business. Yeah, Jonathan: yeah. Florian: Text editor, stock VI. I used to do- AV server ... a lot of systems administration work. Exactly. Yep. AV server has a stock VI, so I'm, Jonathan: Even your router ... Florian: I don't cons- I I don't configure my text editors. And Emacs is, I don't have an I don't, I haven't even spent enough time with Emacs to even know if I should be doing it or not. Jonathan: That's so true of so many of us. "Oh, I've heard great things about Emacs. I don't have time to l- dive into it and learn it." Florian: Yeah. Yeah. I had a well-running Spacemacs configuration at some point, and I actually quite liked it. It just, Spacemacs starts for s- 30 seconds, which is way too much if all you wanna do is change a YAML file. And I will probably get an email about, "Yeah you should've configured it right," and that person is probably also Jonathan: correct. You're pro- yes, you're probably correct. All right. Yeah. Awesome. Hey, it has been a blast to get to talk to you, Florian. I appreciate you so much coming on the show and talking about Rust and Ferrous Systems and all the things going on there. Thank you so much. If somebody wants to find, if somebody wants to find you, find Ferrous Systems, or learn more about Rust- where are some places they should go to? Florian: Obviously our website. Send me an email first name.last name@ferrous-systems.com. Meet me at a conference. We got a list of conference places. In Berlin there's the Rust Meetup, which is super active. Okay. Mostly because I'm not leading the Rust Meetup anymore. If anyone he- here's running a meetup, the first thing you should do is find 10 people who run the meetup and be the backup because then this thing will go on for 10 years. And I'm always around the Rust Meetup. There you go ... so I'm attending all events. Jonathan: All right. Florian: Yeah. Jonathan: Yeah. Very cool. Florian: In general, if you're running a meetup, you're the backbone of the community. Thank you very much. Jonathan: There you go. Florian: You too, those that listen. Yeah. Jonathan: Yeah. All right. Awesome. Okay. Appreciate it so much. Okay. That was indeed Florian Gilcher, as we say in English. I don't think we've made it during the show, but he was telling me that he has trouble saying his name with the German pronunciation when he's talking English. It's one of those, it's one of those quirks that was telling him about my experience talking with German friends and hearing them say English things, and it's just, it's the languages, it's such a fun experience. We do have some really cool stuff coming up on the show in the future. We talked a little bit about that. Next week we're gonna talk with a new friend of mine, Trish Shurlooker, about FLOSS and legal issues. He is a lawyer. He is a lawyer that came to the Ubuntu Summit of his own accord, and so that was that was pretty incredible to get to meet, and he's got some great stories. And then the week after that, we're talking with Andy and Aaron from QNX. We're gonna ask them why it's not open source, see what they have to say. And then the week after that, I am super excited, we're gonna talk with Andrea Gallo, the president of the RISC-V Foundation, I think it's called. But very technical guy, and we're gonna talk about RISC-V and what it means for the ISA to be open source. We've got a, we've got a blank after that, and then coming up after that, we're talking with Michael Meeks of Collabora. And then we've got w- a returning guest, Francois talking about smoked meat. We talked po- poutine last time. We're back with smoked meat. I love the meat theme that they've got going on. Anyway a the schedule is filling up, so be sure to catch all of those. We appreciate everybody that's here, whether you watch or listen, get us live or on the download. Thank you very much, and we'll see you next week on FLOSS Weekly
• 
  Episode 871 - Rust Won't Save You 17.06.2026 1h 15m

  This week Jonathan chats with Florian Gilcher about Rust and Ferrous Systems! How have we gotten here, what's coming next, and what's new in the Rust world? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 870 transcript 10.06.2026

  FLOSS-870 Jonathan: This week I talked with Alexander Neumann about Restic, the backup program written in Go that runs everywhere and most importantly will restore your backups. That's important. This is Floss Weekly, episode 870, recorded Tuesday, June the 9th, open source gardening It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett. Back after a break, after a respite, longer away than we really wanted to be because there was sickness in my family, that I had to travel all over the world. I've been to London, I've been all over the place, but we're back. And we've got a show for you, and we've got something really interesting. We're talking to Alexander Neumann about Restic. And I know that is a backup solution, and I know that it's not written in Rust or C. It's written in yet another interesting language, in Go, I think. And I've got questions. Like, why did we need yet another backup engine? What's wrong with the ones that are out there? And why should somebody use Restic? How does someone use Restic? I have questions, and I bet we have answers, 'cause we have Alexander right here. Welcome to the show, sir. Alexander: Yeah, thank you very much for having me. Jonathan: I am glad to finally get you. We, I talked about I was sick. My family was sick about a month ago. You, too, were on the schedule and then came down with a bug of some sort, so we had to reschedule it. It's good to finally have you here. Alexander: Yeah, it's an honor to be here. Jonathan: Yeah. So you are, first off, I forget this half of the time, but you're actually speaking to us from Germany, right? Alexander: Yes, that's right. Jonathan: All right. And I'm in- We say- ... southwestern Oklahoma, so we're, you know- ... it's almost nighttime there, and it's morning time here. Time zones are real. We don't believe in flat Earth around here. Alexander: Time zones. Time zones are my, my, my end boss. Ah. Because I always tend to mix up time zones, Jonathan: yes. And, the tools you would think that would make it so easy to get it right, Google Calendar and all those that do it all, those will get it wrong too sometimes. I found that out the hard way. No, I thought you were... No, I thought..." Yeah. And then but what really gets us is when I think the US does time change first, and then Europe does two weeks later. Sure. So we've got about two weeks there where nobody knows what time it is. It's the worst. Alexander: It's the same here in, in Europe because some countries do it earlier and some do it- ... later. It's not some like synchronized time zone all over Europe. We also have several ones because Europe as a whole is quite big. Jonathan: Yes. Understood. All so you do Restic and backups. Now, w- what's... I guess give me first the overview because there's a lot of things that can go into backups, and that can be handled- ... a lot of different ways. What's the overview of Restic? What does it do and how? Alexander: Yeah, may- maybe just a step back because at I'm working as a penetration tester in IT security as a day job. And in 2014, I was so fed up with all the different backup solutions that were out there because no- none of them seems to be suited to my use case. I wanted to make a backup of my home directory to a third-party server that I'm not the only root account at that server. So I wanted to have this backup location. It's not necessarily trustworthy. And so I looked at all the different backup solutions and all the bash scripts and Rsync and Rsnapshot scripts that I've written over the years- ... and decided that something new must be done about that. And so Restic was born, and at the time I didn't know that, but there was s- a software called Attic that worked in a similar way, but it, I did not discover that. And after starting the project, I also made a list of all the backup solutions that I stumbled across, and it turns out there are a lot of backup solutions. Oh, yeah. Yeah, quite a few. And I still get pull requests for including this or that backup program. And even after Restic was quite popular people started writing new backup solutions. But that's totally fine with me because it's not the backup that you want, it's the restore that you need. Restore is much, much more important than backup. Jonathan: Yeah, interesting. So you said you were in the you're in the security in security field, IT security field, and that brought to mind something funny I remember from several years ago. When ransomware really started to be a thing, and I saw advertisements a- and was it an advertisement or was it something in Windows saying you... essentially, "You need ransomware protection." And I remember thinking at the time "Isn't that just called a backup?" Alexander: Yeah, de- depending on some- sometimes it is. So in, in some engagements, we managed to compromise the backup service of some company, and then you could, in theory, delete the backups or tinker with the retention period so that they only have a backup for the last day or so. And if you then start deleting files and encrypting files, then usually the company has a very hard time restoring from that. Jonathan: Yeah that's something I've actually thought quite a bit about. I've done my own share of, writing rsync scripts and all of those different things. And I've thought about that quite a bit, because I have had a customer... i've, I wear many hats, and one of the hats that I've worn for a long time is a managed service provider. And I have had a customer get hit with ransomware on one desktop. Thankfully, we caught it before they were able to pivot into anything else and had a good backup solution. A couple of... Actually, we had two or three different backup solutions, and one of them was easy enough to be able to recover everything. But I've thought about that quite a bit. Like, how do you set a backup up so that even if an adversary gets into a machine, they can't, at least not trivially break your backup? And so there has to be an element of things have to... You either have to have another trusted machine that's gonna do this, or there has to be some sort of like you're, you pull instead of just push. And so we... This is a complete rabbit trail away from just talking about Restic. But it does bring you to the qu- lead into the question, does Restic have some of these protections? Can you set Restic up in such a way that it, it is indeed a ransomware protection? Alexander: Yes, you can do that. So what we started a few minutes ago with my special use case- ... that I would like to back up my files to a third-party server that is not necessarily trusted completely. At least I had to trust it to not lose my files. So this is the baseline. But apart from that, Restic is written in a specific way. We can get into the details in a bit, but you can set it up that you only... Restic is push, so you upload your new files that Restic creates to some third party storage provider, and there are providers out there, for example, Backblaze or sometimes S3 providers also provide this option that you cannot really delete files, but they are only hidden for some amount of time. For example, you can configure in your account that files are not deleted instantly, but instead they are just hidden and only deleted, really deleted after 60 days. So if you have this with ransomware scenario where your files are encrypted and then Restic only backups the encrypted files, and then the attackers even manage to gain access to the host running Restic and use the credentials stored there to delete the files. If you caught the adversary within these 60 days, then you can just easily restore the hidden or deleted files, and then restore from there. Jonathan: And if you didn't realize that your files were encrypted until 60 days, they probably weren't that important anyways. Alexander: Yes, and this is why we implemented our own storage backend that you can run locally yourself. And there you have the option to have append only backups, where you only can append new data, but you're not allowed and you cannot, from the protocol side of things, you cannot really delete files. Instead you would then do the backup pruning and removing of old backups from a different host that's, or even do it manually from time to time. Because at the end of the day, storage is very cheap these days. So then you can store 10 years of data there. That's not an issue. Jonathan: Storage is cheap until you have multiple terabytes, and then it gets really expensive. I know. I've paid the bills on S3 and various places. Yeah, that- Still have some of those I'm paying. Alexander: That, that reminds me of the of the time where we accidentally discovered a bug or a faulty hard drive within Backblaze's service because a user reported that his backup did not check out. So we have a check function within Restic- where it's, where it downloads some amount of data from the repository, and does some kind of self-check so you don't have to restore the whole backup in order to make sure that it's, will still work. Restic has the ability to do that itself. That's really neat. And it selects randomly some kind of files that it downloads and checks. And with Backblaze, there was always an error when some file was downloaded, and then we contacted Backblaze and they fixed the error in one of their storage pods. And yeah, that that was when we found a bug within Backblaze. Jonathan: Interesting. Yeah, that shouldn't be possible. That's the whole point of Backblaze, that's not possible. Okay boy, this is really fascinating. With Restic then, it, there's two, two components to it? Three components? So maybe- Yeah, yeah, that's, ... you've got one that's gonna run on some local machine, and then potentially another one that runs on a server somewhere. Is there a third component in between? What's the architecture look like? Alexander: Okay. The architecture is basically there is this single binary, it's a Go program, so it compiles to a statically linked single binary, and you can do 90% of ev- everything with it. You can do backups, you can do restore, you can do also like a fuse mount where you have run Restic mount, and then you get a file system simulated on your machine that is mounted in a directory. Perfect. And then you can use all your normal shell tools or whatever to explore the snapshots and the files that are in there. That's nice. And the data is only fetched on demand. That's interesting. So if you have several terabytes of data and only need a single text file from somewhere, like an SSH key or something like that- then it's really fast and really efficient. So this single binary does everything. It also does pruning and backup retention policy application and so on. Yeah. And there is a second component that can be either it's the Rest server, which is one of the storage backends that we support. And you can run it on some machine, and then do an HTP connection to this machine, and then have Restic push new data there and store it there. So you can run the service yourself on your local machine or on some Raspberry Pi in your basement or something like that, so that you can host it yourself. And the same protocol is also implemented by Rclone which is a very popular cloud s- R sync copy tool. It can also u- be used as a server for Restic, and in the backend it can store the data on all the backends that Rclone supports, and I think that's quite a lot. Jonathan: Yeah. It's gonna be a bunch. The server side, does it have does it have support for multiple tenants? Or does it have any, does it have any of the advanced tooling, to be able to go into a webpage and see this guy didn't back up for the last 30 days, I need to go to his office? I come at this from that sort of managed service provider mindset. Alexander: It has multi-user support, so you can have multiple users with different- ... access credentials, but you don't have some fancy web UI or so. We try very hard to limit the scope creep for Restic because this- Sure ... is a real issue once a project becomes popular. And yeah, I- For sure I, I must admit that Restic's popularity exceeded all my wildest expectations by several orders of magnitude because when I started in 2014, I was alone. I talked to my colleagues about the architecture, but I was implementing this thing. And then, yeah, several years later, we have almost 30,000 stars on Res- on GitHub. Oh, wow. So that's quite popular. And everybody would like to submit their feature, even with code, but we are the ones having to maintain this feature over time. So- Yes ... but the protocol that Restic uses for this REST server interface is is, it's documented in a separate document, and it's very easy to implement this. This is just a basic REST API get, put post, delete and that's it. And it's very easy to implement, and I'm aware that several other storage providers have implemented their own solutions that you can just use with Restic. Because at the end of the day the product goal is that more people have the option to restore when they need it. And backup should be as smooth as possible because backup is the chore that, what you do and so that you have it when you need it. But if backup is complicated or backup does take a long time, or it doesn't run every time, or you have to manually remember to do it, then you don't do it, and you don't have a backup at the end of the day. Jonathan: Yes. I- if it's a well-documented REST API, it sounds like it would be pretty easy to just vibe code a dashboard. And I say that- Oh, yeah ... I say that halfway jokingly, but I also acknowledge that we're at the point to where you could probably do that, and it would probably work and give you a reasonably good result, too Alexander: which is incredible. Yes, you could do that. A f- a friend of mine implemented the server side of this protocol, like 10 years ago or so in just under 100 lines of Ruby code. So the server side is not that complicated at all. Jonathan: And I assume it has it's credentialed, right? It's got some sort of login system or a token system to be able to even get in and upload- Yeah, it- download data and see things. Alexander: Yes, it's just username and password. But maybe we can start from the beginning at the end. At, in, in the beginning I thought about and discussed with my colleagues and friends, and what would we expect from such a personal backup tool to back up my home directory to some server? And we came to the conclusion that it must be secure. The backup s- storage location may not be trusted entirely, so I trust it enough to keep my files, but I don't trust it enough so that some other admin on that server decides may decide to have a look at the files that are yeah, lying there. Encryption is not optional, but- ... very integral part of what Restic is. And it must also be able to check whether the dep- backup has been tampered with, and th- that was also very important for me. And since restore is much more, yeah, much, much more important than backup itself, it mu- must be very easy to simulate a restore, because you don't really restore or test restore every week. But if you have an automated process, for example, if you every week, if you download 10% of the repository selected at random, and can have some kind of like self-check where it downloads the data and makes sure that it got the r- the right data, then you can be sufficiently sure that a restore would be possible. And this turned out to be a very good idea because over the years we have we have even have a label on, on GitHub issues for that. We uncovered a bunch of hardware issues with hardware doing wrong things. Usually the CPU is broken or the memory is bad from our users' machines, which they tend to not believe at first. But I've run into this myself twice already where I was wondering why a Restic check returned wrong data and returned error with a freshly bought machine. That was a very good idea. Jonathan: Interesting. Yeah. Time to go time to go pull out Memtest86 and make sure your memory's not- Exactly flipping bits and losing- Exactly ... losing bits. Yeah, no, I've, that, I've found that several times on fresh memory sticks. It seems like first it was a real problem there for a while. It may still be. Yeah. But there for a short period of time, it was surprisingly bad. I had a probably 25% fail rate on memory sticks I was buying. It was really- Alexander: Oh, Jonathan: That's Alexander: really high. Jonathan: Yeah. Okay I was gonna ask... Sometimes my brain wanders and I've, I completely forgot the thing that I was gonna ask. I really like the idea of doing this this random sampling pulling data back down. That's that's really fascinating to me. Oh, I know what I was gonna ask. So you have encryption is mandatory. How many emails have you gotten then from people out in the community saying, "Please help me find my encryption key"? Alexander: I think only once or one, one or two, something like that. Oh, Jonathan: wow. I was, I expected more than that. Alexander: Me, me too, but Rustik is very explicit. So the first step for, in order to be able to do backups with Rustik is you has to initialize a repository that's very similar to Git, because I like the way Git is built. And I took a lot of inspiration from that, and that turned out to be a very good idea. So you need to initialize the repository, and it asks for the password, and it very explicitly states that if you lose the password, you will not be able to access the data anymore. And this is coming from somebody who breaks systems and encryption protocols for a living, and I've been doing this for quite some time, and this is really, if you lose the password, then it's gone. ... What surprised me more was that there was a very very vocal group of people requesting that Rustik accepts an empty password, for example- ... or that there be an option for, to disable encryption because they don't need it, their hardware is trusted, and so on. But over time in my day job, I learned that it is very important to test all the code paths and be very sure in what you implement, because once there is a feature that is implemented, like having unencrypted repositories, maybe attackers can find a way to trick users into saving their backups without encryption. So in Rustik, there is no code path without a password. You'd have to supply one. And if you create a text file, a password.txt, and save it alongside the repository that's totally fine with me, but I would like to have Rustik not have a code path where it works without a password. Jonathan: Yeah, i- indeed. And I'm sure there's somewhere in the readme it says please take your phone out and take a snapshot of that .txt file. Don't send me an email. I won't be able to break into it." So you- Exactly ... you were inspired by Git. Does that mean- Not just ... I'm gonna be a little bit of a troll here. Does that mean that you're storing backups on blockchain? Alexander: No, but it's if you want to view a blockchain not as a concept to burn arbitrary amount of energy b- by using this, by computing the proof of work and- ... but rather see it as an append-only ledger, then some concepts within Rustik are like that. So once you have initialized the repository, then you can do the first backup, and there it deviates already from maybe existing backup software because the backup workflow is very highly optimized for that it's as smooth as possible. So you can just Rustik init, and you tell it where the repository is, which may be like a local directory somewhere. Yeah, you it asks for the password, and then you can directly start backuping things. It's just like Rustik backup, and then you pass a list of files or directories and and it will start saving the data. And in order to do that, and you can e- just after the backup has finished, you can just run it again, and it will take just a few seconds at most because it has a lot of optimization going on in detecting which files have changed. But also if you have the same files over and over again, or you have similar files with, for example an image of some some embedded device which is several gigabytes in size, and you have a second image of the same device after you've turned it on and configured it, then most of the data will be the same. And Restic will read the first file, split it into blocks, and only store these blocks. Then the blocks are identified by their SHA-256 sum. And this is similar to Git, but Git uses SHA-1 I chose the more modern hash algorithm. Jonathan: They're moving to SHA-256. That's a good thing. Alexander: And yeah, that's Then it stores the it stores the blocks identified by their by their hash, and then it stores the metadata in adjacent documents. So the repository format it uses it's very well documented. There is a separate design document that I wrote alongside the implementation, and there are even implementations that can write and read the same format because at the end of the day, you don't gain anything from being able to restore your backup if the backup program is not around anymore. That's a bit easier with Go binaries because they are statically linked, so you just need some Linux kernel or some- ... some Windows machine to, to run it. But the backup format also is quite well documented. Yeah. Yeah, so the first file is read. All the blobs are the file is split into blobs. The blobs are stored. Restic made a note which file consists of which blocks, and then it starts reading the the second file, and it sees almost the same blocks, and they are not stored again, so they are just stored once. And instead the second file will just reference the blobs from the first file. And yeah that's that What, what- The de-duplication ... Jonathan: that's cool. What size blocks do you use? Or is that c- probably user configurable? Alexander: No. Rustik has if I can if I can prevent it, Rustik does not have any configuration if it's not strictly necessary. For example, we have added compression. It was an, a feature added several years after Rustik's initial release, and we did it in a almost backwards compatible way so you can upgrade your repository and use all the data that's already there, because backwards compatibility is very important to us. But if we have the option to not make anything configurable and instead select a good default that works for most users or even, yeah, c- just have two or three configuration options that's a good idea. Because at the end of the day, most of our users will not be able to decide which encryption algorithm they want to use for their files, because they cannot really decide which one is best for their use case. Instead, we settled on just one encryption algorithm. And for the compression we use Zstandard. That's the one by Facebook. It was quite popular a few years ago. It's very fast and ha- has very good compression results, and this is the only compression algorithm that we support. It makes the software simpler. It does not have so many dependencies on different libraries. And for configurability we have three settings for compression. It's off, auto, and max, and that's it Because off when you have a very fast bandwidth to your storage location and don't care about compressing the data, you can just turn it off. Or auto is the default setting. It will compress data, and the Z standard is s- so optimized that you usually don't really yeah, don't really have any CP- additional CPU load by if you compress it. And then there's the max setting. That's for people who have very limited upload bandwidth, for example. If you have ... in a country where you have one or two megabits of upstream bandwidth but locally you have a lot of CPU power, then you can spend some time to really optimize the compression and really compress as much as possible. But that's it in terms of configuration options for compression. Jonathan: Yeah. And then I don't ... You may have said this and I just missed it. The actual block size when you f- when you carve files up, is that what, like 50 meg blocks and then it gets compressed or? Alexander: It's dynamic. So there are lower and upper bounds for blocks. Ah, okay. So the lower bound is obviously if you have a very small file of two or three bytes, then this is one block. But usually we try to have a block size between 512K and four megabytes, something around that. Small blocks, okay. And the algorithm that selects the block boundaries is a similar one to the one rsync uses. So if you have, for example, like a huge file and you have the same file with five bytes appended in front of it, then almost ... Then the ... If you have a static block size, then the first and second file will yield completely different blocks. And what Restic does instead is computes a sliding window over I think 64 bytes or something like that, and does a very fast hash computation that if the hash has some properties, then it will split the boundary the Split the file at that boundary and make a block. So it will recognize when the same block boundary is there again Jonathan: Oh, interesting. So it's almost got its own compression algorithm built into it. Alexander: Yes, the deduplication is hugely efficient. This is why we didn't at first have so much a focus on adding compression. But the compression itself also is, it's on a block level, so it's selects the blocks and only then compresses them. That's also quite efficient. But it was the most efficient thing from the compression that we got was that we, in the repository store, encrypted JSON documents because this was a serialization format that worked quite well for me when I started implementing it. And at the end of the day, these are text files, and text files are very well compressed with zstandard, for example. So this, Okay ... gave us a huge reduce in metadata s- size, for example. Jonathan: Yeah. Oh, that's clever. I like the sliding window. That's really clever. It's got its own built-in compression. That's- Alexander: Yeah, that was also- ... that's very neat ... i've studied computer science, and I was always fascinated from old papers from the '80s that are still relevant today. And there is the the content-defined chunking paper from 1983 or something like that from Rabin, and I managed to find a copy or a scan of the original paper, and it was all typewriter written and edits the small numbers edits with, by hand and so on. That was quite fun also implementing that. Jonathan: That's great. I like that. I like that a lot. So something you you referenced is the idea that you have people out there that, they have their pet feature that they would love to see in Rustic and you guys- ... you guys resist some of those. That's probably gotten a lot worse in the last six months, hasn't it? Alexander: Oh, yes. I'm a bit out of, I'm a bit out of the loop in recent developments because I don't have so much spare time anymore, and yeah. But this is, m- it was always the case that people tend to request features with very strong requests- because otherwise they will use some other backup program, and we just say, "Okay, then please do that. Just be sure to make backups." That's the important part. Jonathan: Exactly. Alexander: And the other thing is that we cannot, for example, we cannot really support all storage backends. And, ... people tend to like that their preferred storage backend is Build directly into Rustix, but then we defer them to just use our clone and use it as an adapter- which works quite well. But yeah, some features are, yeah, the discussions got a bit heated also because we as a project have limited resources in terms of engineering time and also in maintenance time. And keeping the project alive is the highest priority, and implementing features is somewhere below fixing bugs. So sometimes you just have to be very upfront about it and tell people, "Okay, this is a nice feature. I'm leaving the issue open so that other people will not request the same, but rather find this issue." But it is unlikely that it will be implemented in the next couple of years, for example. Jonathan: Yeah. Yeah, that makes sense. Have you gotten a lot of AI generated pull requests recently? Alexander: I've looked that up. There are sometimes people submitting pull requests that I, that are as I, I assisted and that's quite fine with us. So we will review them as other pull requests also. And usually people at the moment at least tend to be able to yeah, respond to questions about these pull requests. And I don't think we have some- sometimes we have pull requests that are obviously AI, and then sometimes re- reject them, and even when especially when people are not able to answer questions about that. Because then I could easily use Claude myself to, to code this pull request. So I don't need any people for that. Jonathan: I have my own Claude tokens, thank you. Alexander: Yeah. Jonathan: Thanks. Have you had any of the... This is something that blows me away every time it happens. Have you had any of them where you ask questions and the person on the other end is obviously just copying your question and pasting it into the AI, and then copying its response and pasting it back into the issue? Alexander: I've seen the reports by from the cURL project from Daniel. At this at... I don't think it has happened to us yet at least. But this is something f- for each feature that we get or for each bug fix that we get we need to do a thorough review. Because at the end of the day with the features that we integrate into Rustix, we have to maintain them indefinitely. Because Rustix as a project is only can only work if it's exists long time long term. And if people are even able to restore their backups that they did 10 years ago. So we'd be, usually we are very conservative in adding features or, for example, changing or extending the repository format. We had to do that when we added compression, and we did it in a backwards compatible way, which was it's technically not the most elegant solution. It's some kind of a hack. Sure ... but I liked it a lot because it lets people enable compression for newly uploaded data, but all the old data that is already stored in the repository m- remains valid, so they don't need to re-upload the data. And I think that was the right choice, but it was a very unpopular choice because the code looked not so elegant. Jonathan: I understand. The it's the e- the eternal the eternal war that happens within each of us as programmers. The, th- there, there's some things that are just you just have to do it. But it's like it, oh, it hurts to do it, but you have to do it. Alexander: And by the way, we have two more compression levels in the release that I published just a few minutes ago. There is also the fastest and better compression levels, so you have some n- not only these three, but now five compression levels. But apart from that most of Rustik is not configurable at all. Jonathan: So what about what about retention, data retention? Is that configurable, or is that just- Yes. Okay. Alexander: This was, that, that was a huge surprise for me because I thought when I started the project that getting the chunking algorithm and the crypto and the security right, this was the hardest thing. But it turned out it wasn't. Retention policy application is the worst because we would like to... what I had in mind when I started was, like you had, if you do one backup each day- ... and then after a while, you thin out these backups. So for las- last four weeks, for example, I have daily backups of my data, but for the backups that I did more than one month ago, I think just one backup per week is sufficient. I don't need any backup. And for six months ago, I maybe I only need one backup per month. And for more than a year ago, I'd only need one backup per year so that I can, I have a chance to discover very old data that has been deleted in the meantime. But apart from that, usually the more recent backups are the most important ones. Jonathan: Right. Alexander: So we implemented a retention policy that you can configure, and it turned out that people have very strong opinions about backup retention policies. And basically you can tell Rustik to keep the last 14 daily backups, the last six monthly backups, and the last f- like 10 yearly backups or something like that, and it will remove all the snapshots, all the points in time for which these policies do not apply anymore. So if you have yesterday I did two backups, and then I run the forget command and tell him, okay, to keep only one daily backup, it will remove the oldest daily backup for yesterday and keep the most recent one for yesterday. And then in a second step, it's called Rustik prune it will go through the repository and identify all data that is not referenced anymore, so it does some kind of garbage collection step. And there are also some heuristics that, for example, if you have several small blob small data blobs from small files bundled up together in a larger file, and if only one of them is not in use anymore, it will just keep the file as it is. So because it is not not a good idea to download all the files and re-upload them if only it will save five bytes or so. So this is something that you need to regularly do. And but you can also keep all snapshots forever. That's also an option if you have enough storage space. Jonathan: If you have infinite storage space, yes. Has there ever been like a serious bug in Rustik where data was lost? Alexander: I think at one point we had some kind of this bug, but we caught it early because we have a lot of unit tests- and a lot of integration tests. This is something that Rustik is written in Go, and Go makes it really easy to write unit tests, and it is also very fast, so we caught that. Usually what we have is people coming to us in the issues and say that, "Okay, Rustik reports that there is some kind of hash mish- mismatch." And usually it's some kind of hardware defect like memory or CPU or even storage in terms of the hard disk is dying and returning invalid data, for example. This is the most severe thing that I can remember, at least. Jonathan: Yeah. Interesting. And then you have unit tests. Do you do, fuzzing. Do you do fuzzing and coverage tests? I'm just curious. Alexander: We do, yeah we do coverage tests, and the Go test suite is already coverage-driven, so it tells you when there is new area that it uncovers. And Go, m- Go some, for some years, Go has had fuzzing testing integrated into the Go test command. I don't think we've... we've not enabled that for Rust yet. Jonathan: Okay. Interesting. Something that, and I don't know that this makes a whole lot of sense for Rustic just because of the kind of data that it's d- that it's processing and the way that it processes it, but I've thought for the longest time that AI-driven coverage testing would be really interesting. So y- you have a fuzzing harness, and then you watch the coverage on the back end, and basically you tell an you tell an LLM, "Come up with new, based on the corpus, come up with new data patterns to throw at this to try to exercise more of these portions of code." I've always thought that... I don't know of a project that's doing that. I'm sure somebody is. But I've, I don't know of a project that's doing that. But I've always thought that would be a really interesting approach to, to- Yeah, for, for- ... find bugs and security problems. Alexander: It is, but for Go you don't even need that. You don't need an LLM to do that. The Go test command has this integrated, and it also has means to modify the input, and it will automatically discover when new area of code has been covered. So this is really efficient. Especially for kinds like image parsers and so on, Jonathan: Yeah. Yeah. Super interesting. Now I haven't asked you this, but why Go? Sure- surely you could have rewritten that in Rust by now, or, pick your favorite language. Alexander: Yeah, sure. At the time of when I started Rustic it was 2014 and Go was out for some years then, and there was the Go 1.0 compatibility promise, so every Go code that you write will still be, yeah, can, you can still build it and run it with Go 1.X. And this was very appealing to me, and I wanted to start learning the language a bit more. I've used it for some toy projects before, but nothing serious. And then I started implementing it and came to like it very much because Go in, in contrast to other great languages such as Rust is optimized for readability. So it's very simple language. It does not have so many different im- language constructs. For example, you have to explicitly check every error, which becomes a bit repetitive, but yeah, you get used to it. And it's very easy to read, and also people who have never written Go before can grasp what the code is doing just by looking at it and not having, they, you don't have to study it for one or two years. And I've never really, unfortunately, I've never really got into programming with Rust because at the time when I started Rustic, then Rust was not really stable yet. And then I didn't have the time to really pick it up. Jonathan: Yeah, that's interesting you talk about being e- easy to understand, because I, for the longest time, and I still believe this to some extent, that, one programming language is going to be roughly equivalent to another in that I've learned how to program in C, I can go apply that to C++, and there's just a few things you gotta figure out with C++. Y- and, you can go apply that to PHP or Go or whatever, and the central concepts are gonna be the same. You just have to learn, the syntax that's different. And then I went and I looked at somebody's Rust code. I had the same idea. I went and look at Rust, and some of the things that Rust will let you do, I forget what it was but it's one of their, It's like where you do, it's almost like a ca- switch case, but it's- done in a very Rust sort of way. And I just, I remember- I think it's, Alexander: it's called pattern matching- Jonathan: Yes, pattern matching ... what you're talking about. That's what it is, yes. And I remember looking at that the first time and "I have no idea what this is doing. I'm gonna back away slowly and go back to my C++ code." Alexander: Yeah, what's also interesting for me at least, is that I, when I got more comfortable with Go, and that you have these concurrency primitives with channels and Go routines and so on. And it really gets you thinking about states and state machines. And having a few years of Go under my belt, I now write better Python code, for example. Sure. Because some concepts translate over to different programming languages. For example, the, in, in Go you have, most people write the Go code with a early out. So you have some- something that must be done five steps, and do step one if there's an error return. Do step two if there's an error return. And this is in contrast to Python code, which usually looks like a staircase. If that, and only then something happens. And I find Go code much easier to read because the happy path is on in, is left-aligned and you can follow it a lot better in a function. And you can also do that in Python, you just have to do it. Jonathan: Yeah you get into what Torvalds talks about with good taste in your code. And sometimes we get into bad habits and we write things that, from an objective standpoint are just terrible to look at. Yeah, it works, but you gotta sit there and stare at it for 20 minutes to understand what it's doing and all of the different, all of the different w- weird ways that it can fail or succeed. Yeah, absolutely. Alexander: Yeah, at the end of the day, a programming language is a tool to get a job done. In this case, make a backup and be able to restore. And and I'm not a fan of these culture wars of this programming language is better than the other. Just use the one that most suited to the job and that and it will be fine. Jonathan: Yeah. Yeah I would tend to agree with that. So w- I find it fascinating that you're thinking about such long-term backup and restore with Restic. It's Alexander: backup after all. Jonathan: And so I think about this sometimes with just I have floppy drives around. I do not have a floppy drive reader. I Alexander: have a a DVD burner lying around, but I've not read any CDs or DVDs in 15 years or so. Jonathan: Yeah there's that too. VHS, I have VHS tapes. I don't know that I have a working VCR. And you start thinking about these things, and it does, it brings to mind this idea of what, what of our media is going to become inaccessible here after a while? Sure. And so we've talked with we talked with some p- some folks, some projects over the years that are doing archiving work. One in particular was a group that was archiving source code, like off of GitHub, with the idea being that GitHub may not be around for- forever. There's another group that's archiving YouTube videos, because YouTube probably won't be around forever. I find that real fascinating to think about sort of those long-term archiving bits of work. And you mentioned 10 years, but in the back of your mind, how, like how long would you like this project to the project to be around, but also, like how old of data would you like to be able, for people to be able to restore with it? Alexander: That's an excellent question. For my personal data, which this all started with I'd like to be able to, like f- four or five years is usually sufficient. For the project that I do and the source code and so on. But I have an, a digital archive of all my paperwork with finances and so on at the moment stored in a paperless instance, and for this I'd like to be able to read for 10 years at least. But for Rustik, I think it sh- would be a good idea to be as backwards compatible as possible. So at the moment for Rustik is at the release 0.19.0, which has been released today. But it can still read all the repository formats that have been written by any released version of Rustik. So even the ones made in 2014. And this means that have, we have a bit of maintenance to do in terms of being able to read these old s- storage formats. But we haven't really changed the storage format since then. It's turned out to be quite robust, so the introduction of compression was- the biggest change overall. Jonathan: Do you have some of those old repositories snapshotted to be able to pull into your test suite? Alexander: Yes, in the test data. Okay. There is there, there are some old repositories archived as a tar. So we and we have like unit tests which try to restore data and exit it, and so on, so that we don't accidentally break backwards compatibility. Jonathan: That's excellent. I figured you would. You guys seem to be careful enough that would be definitely something you would do. Thinking about that idea of the age of backups, like- I've got digital photographs that are over 20 years old that I would hate to lose. And there are some things that I, I don't know exactly if I have a copy of because my backup system when I was a teenager was burning things to CDs and DVDs, and those don't last forever. Somewhere around here I've got some very old DVDs that every once in a while I look at. I'm like, "I wonder if there's any of those old photographs on there that I don't have anywhere else." Some recordings I made in college of classes and just all kinds of different stuff. So i- it's real fascinating to me to think about that idea of like how long is this going to exist and I could see this being useful to kids and grandkids in- Alexander: Oh, yes. Yeah ... Jonathan: 60 years, 80 years, 100 years. Who knows? But what is, what are computers even going to look like in 100 years? I- is there- Yeah ... going to be, like how many generations back are you gonna have to go like we do now. "Oh yeah, I've got I've got the big five and a half floppies." You have to have one computer to move it from a five and a half floppy to a three and a quarter, and then a second computer to be able to go from three and a quarter onto a USB drive. And then, then you can finally take that USB drive and plug it into a modern machine. So y- you, you're gonna have to have a 20-year-old computer to run Go, and a 40-year-old computer to run a Go 1.X. It's just it's wild to think about what things are gonna look like and- Yeah, definitely ... I guess also like the tail that our work is going to have and just that idea of, maybe my kids and grandkids are gonna run my code. That's just such a fascinating thought to me. Alexander: Yeah. I would like to, the, for the Rust project, I would like to, it, it to outlive me. That would be excellent. And I from the start I created an organization on GitHub so that it's not really tied to my person. I personally sign all the releases, but I'm not necessarily one who does the m- most of the work nowadays. And this is something that I would like to talk to about a bit like project organization. N- not meaning like how, who does the releases and how is code reviewed and so on, but I think it's very important to have a very positive attitude in the project and set the tone right from the beginning. Because there are these... i've been around for the late '90s, early 2000s when there was these huge flame wars on the Linux kernel mailing list and- ... sometimes, sometime later, years later, they realized that it's a bad idea and it drives people away, and this contempt culture is not something that, yeah, should, you should allow in a project. And then this was very important for me from the beginning, that it's the Rust project and the GitHub issues and also the Rust forum this is my small bit of the internet. This is my garden. And you either you behave or you're shown the way out. And I've, yeah, over, over time people from time to time people get into very heated discussions and sometimes I step in and say, "Okay, this is stop." And then I'm the bad guy saying, "Okay, stop. I will lock this issue now, and if you ever talk about it again, then please do it in a way so that you don't y- you don't drive other people away." And this is something that stuck with the project. Especially in the forum there are people- who stuck around for several years who- doing not s- not any coding, but who are helping other people with their backup issues. So somebody reports something that it's that their backup is strange or their retention policy does not work, and people in their own spare time step in and answer these questions, and I don't have to do that anymore personally, which is very great. Jonathan: Yeah, no that's interesting. I've seen projects take different shades of that. But I think some sort of, w- community management is really what it is. There has to be- ... some sort of community management because somehow the internet brings out the worst in people, and I think it's the pseudo-anonymous nature of it. There's not a, there's not a person in front of you that you're yelling at, so it's very easy to be mean when typing comments. Alexander: Yes. Jonathan: And yeah. Alexander: I also think some- sometimes people don't realize how their text is read by somebody else. And then I step in and point out that, "Okay, this can be read as very aggressive. Can you please tone it down, or can you formulate it in a, in the, in another way and not assume a bad intent but assume a positive intent?" And usually one or two remarks of these sorts is sufficient to bring the discussion back on track. Jonathan: Yeah. Yeah. That's good. Do you guys have a, do you guys have a policy on bringing culture war and politics into the project? Alexander: Not really, but I, th- this did not really come up. If I like Trump or not, that's shouldn't affect how the backup software is used, and Restic has a very liberal license. It's licensed as a two-clause BSD license, which is the most copyleft license I could think of. And I know that it's used in several different areas in, in, in Germany and all over the world where I don't have any control how they use it. But at the end of the day, if people can restore their backups, then I'm happy. So we don't have an explicit policy, but I Jonathan: don't Alexander: think- I actually- Jonathan: Yeah ... I actually think that is the way to go, and also what is intended by what it means for something to be open source. I've seen projects take very weird political stances before, and I always find it very weird. You end up chasing about half of your user base away just to, to wave whichever flag you wanna wave. So I am on Team It's an Open Source Project, and that's what it should be about. Alexander: Yeah. There, there are yeah, there are borders, of course. At the end of the day, as I already said it's our little garden- our little b- place on the internet, and if we don't like somebody or we, if we don't like something somebody said, then I, we will state it very clearly, and then maybe also hide posts on GitHub or hide posts in the forum because, yeah, that's not productive, and I- Absolutely ... like to be productive and help people. This is what it's all about. Th- this r- this reminds me of the the, some- someday somebody wrote a post on the forum and said, "Hey, did you know that CERN, the the nuclear research facility in, in, in Switzerland they use Restic, and they have the big instance, a- and I've discovered some slides here," and yeah, and then- this was very interesting, and d- and a day later somebody else showed up and they were, "Oh, yeah, I was the guy creating the slides, and we have a very big installation." And they have some 60K users or something like that. Oh, wow. And they have a, an automated Restic backup for each of them. So that's, that was really nice. Jonathan: Yeah, that's cool. That's always fun to see your work show up in, in unexpected places, but big places. Yeah. Alexander: Probably the most unexpected place where Restic was in the Arctic because I got a bug report. It was, like, a v- re- very long time ago. I got a bug report, and then at that time I still had time in the evenings and started fixing this bug, and said, "Okay, you can download the beta build from our beta builder software." And th- they responded back and said something like, "Okay, can you send me the patch, and can you show me the c- the commit? Then I can use the patch later on because our ship is leaving." And I said "Okay, but what ship? What are you from?" And then was from some American university. I forgot which one. And they had an Arctic expedition, and they used for saving their data they used a MinIO cluster, and they backed it up with Restic. And th- they said, "Okay I will have internet, but it will be very slow. It will be satellite internet over the Arctic. The bandwidth is very limited. Please, can you give me the patch and not the binary?" So that was really cool. Jonathan: Yeah. No, that's neat. That's really cool. Where is Restic at on the progress to being done? And so the... I g- I get the joke, and I say that on purpose, but at the same time there are some pieces of software out there, like- There's a there's a spreadsheet program that runs on the terminal. And it's not seen an update, like they've not written any new code for it for 10 years because it's like it's done. It does what it's supposed to do. The- there's no outstanding bugs. It's, it's done. It... Do you foresee a moment like that in Restic's future where it's like, "Okay we've fixed all of the problems. It does everything we want it to do. We... This is all of the backup, backends that it's gonna support. We're now in maintenance mode. It's pretty much done." Are you there? Do you see that coming? Alexander: Not really, to be honest. At some points we might be, we might declare that this is what version 1.0 and that's it. But usually with backup programs and computers in general, it's always more complicated, a lot more complicated than you think. So what's usually what I expect that we need to do and continue to do is fixing bugs and fixing weird issues because you cannot really, with such complex program, you cannot rule out that there are no, no bugs anymore, and people use it in unexpected ways. And this, I think, will continue for s- for quite some time. We have a lot of ideas what could be improved. For example, at the moment you can use Restic Restore to restore some some directory to some file path. You could also use diffuse mount, for example, to interactively browse within the in the snapshots and use your normal shell tools, for example. You can also... it would be very nice if Restic had a web UI, for example. If you can s- use like Restic Serve, and then it s- spins up an HDP server, and you can use your browser to browse on that because then you could run Restic where it's much closer to the data and just use HDP to access this data. And I s- I've started implementing this some, a few years ago, but it turns out that this is also quite complex, and this is something that would be nice to have. But it's not on the roadmap for the next releases. Jonathan: Yeah. Alexander: There is also something else that's is also very important. If you have like this, you have the backup step, which adds new data. You have the forget step, which applies the retention policy, and then we have the prune step where data is repacked in a safe way and re-uploaded and so that at the end of the day you can remove a few files from the backup storage location. And this last step needs an exclusive lock on the repository, which is not a really great design because all other clients that may access the repository at the same time, because you can backup from multiple machines at the same time into the same repository. They cannot really do a backup while the prune step is running. There are constructions which allow lock-less pruning, and this would be something that would be very nice to have for a 1.0. Jonathan: Yeah. I- is Restic, does Restic have a 2038 problem? Alexander: I don't think so. Because I was w- I was I was there when the year 2000 problem happened. So for example, for the timestamps, we used the ISO something string representation in adjacent documents, so there is no Unix timestamps. Jonathan: Yeah. For those that are not aware, 2038 is when we're gonna run out of y- seconds or mic- I think it's seconds in a 32 bits int seconds since the Unix epoch. Sometime in 2038 some subset of machines are gonna roll back around and think that it's 1970 again. Alexander: That'll be fun. Jonathan: Oh, yeah. That'll be great. Probably more There's a decent chance that we'll have more problems in 2038 than we did in 2020. Alexander: I also think so because at the 20 at the 2000, year 2000 there was a huge uproar and panic and people think the world will end. But it also led to a lot of people fixing stuff before the year 2000 showed up. So we got there yeah, without any m- major harm. Jonathan: Hey, we have still, the better part of a, more than a decade to panic. So yeah, we'll probably manage it. Probably manage some panic. Goodness. If you had to go over and do it again, or say Restic didn't exist, and yes, this is one of the, this is one of the pre-questions that he fed me, but it's an interesting one. Would you do it again? If Restic didn't exist, would you start the project? Alexander: To be honest, I would like to be in a situation where I don't have to do that again because there are so many great alternatives to Restic right now. There is Borg, for example, which is the fork of Attic, and there are other backup programs that also took quite a lot of inspiration from how Restic was built, and I think that's great. I think it's a very good idea to have options for people because not, nobody sometimes people don't like Go and then they use something written in Rust or Python or whatever. But if I were in the same situation 12 years ago I would start again, I think. Because at the end of the day, if nobody steps up and said, "Okay, I'll do it now," then nothing happens. Jonathan: Yeah, absolutely. I- is Restic anyone's day job? Is there like a business built around it? Alexander: I don't think so. Jonathan: Okay. Alexander: Because at the... We- we've started s- to get more requests for f- how can I donate to the project, but I'd rather spend time doing actual stuff than managing funds. So there is no as far as I know, there is nobody working on Restic full-time. We had this yeah, we have basically the project at the moment is mostly run by Michael, and Michael did a Ph- PhD. But unfortunately for us as a project, unfortunately he finished his PhD and also had a day job now. But there are people working on integrations for Restic. For example, there was Relica. Which was a, it was or is, I'm not really sure, a backup service that offers Restic in a nicer packaging. They added a web UI and so on around Restic, so they don't have to interact with the CLI binary at all. Because Restic explicitly supports that. There is a JSON output mode, so you can call Restic and read the JSON data back as JSON, and then use some kind of other tool to read that, and it was quite a nice service. And I suppose that there are more services like that, that we don't necessarily know. Jonathan: That, that seems like a natural evolution for that level of support and that long-term, Long-term maintenance for the project is, if companies, Backblaze obviously is one of the big ones that comes to mind, but in some of these smaller providers, they're using it for backups. It, they're, they are they're obviously interested in keeping it working and keeping things working well. And so you could imagine a few of those different companies putting a engineer in the, in the open source pot to make sure things are doing what they're supposed to do. Alexander: Yeah, but then you have to manage this this engineer. And so somebody has to guide them into implementing features and reviewing stuff. And this was also very hard for me, to be honest, to transition from somebody who writes code in the evening to somebody who's managing a project and doing code reviews. And oftentimes I'm, got very frustrated with reviewing code because it's so different than actually writing code. And sometimes it's even easier to just scrap everything that somebody spent a lot of time and submitted and- ... took the plunge and submitted code. It was, sometimes it's easier to just scrap it and write it myself. Yes. And this is the maintainer's conundrums. It's, there, there's no solution for it. You, at first you're a developer and then you get a manager and gardener and that, that's something different. Jonathan: In, yeah, indeed. It is. It is indeed. You've got the, you've got the interesting piece there that if as the gardener, if you do your job well enough, then your your sproutlings, those people sending you pull requests, they'll get better at it, and then eventually it'll be easier, you hope. Alexander: Yeah. Somebody, some- some time ago we had somebody who, yeah, submitted a lot of pull requests and had strong opinions and, but the code was good. And we didn't have the bandwidth to review everything. And at the end of the day, they got frustrated and then started a different project. And that was very unfortunate, but- Yeah yeah, that's the reality. We're all in doing this in our spare time for fun. If it stops being fun and becoming a chore, then yeah, I'd rather do something else. Jonathan: Sure. Ha- have you guys started using obviously Copilot has a reasonably good code review. Th- there's some other ones. CodeRabbit I think is another popular one for open source projects. Have you guys started using any of those for helping with code reviews? Alexander: No. Not yet. Because at the end of the day, the most hard the hardest things to review is not let, that the code is secure. This is very hard to do this with Go code, for example. But to make sure on an architectural level that it's a good idea- to actually do that. Also from a, like strategic standpoint, do we add another storage backend or not? This is not something that an AI could decide, and it's also not something that an AI could argue with somebody who submits this code. And there is, it's very important to have a human in the loop because this vision for the project, also this long-term vision, is not necessarily what people see when they want their storage backend to support, to be supported. Jonathan: Yeah. We have found in the projects that I'm in that those AI code reviews, they're really good at catching the dumb mistakes that a human might look over. And so sometimes that's just, a misspelling in a comment but also sometimes a little bit more subtle things. But a logic inversion where it's in, in this function you checked for this, but in this function you checked for something slightly different and you might wanna be aware of that. It is reasonably good at catching some of those things. Yes, indeed ... we found it to be pretty useful. All right. What what's the most surprising thing that you've heard of that someone has backed up with Restic? Is it, was it the Arctic expedition, or? Alexander: I think it was the Arctic expedition, because usually people don't tell me what they back up, which is totally fine. I don't really wanna know. But the amount of different, or the d- different people using Restic for so many different things. From time to time I get emails from people personal emails telling me that what's that Restic has saved their day or something like that. And this is really rewarding. But this Arctic expedition thing, that was quite something. Jonathan: Yep. You have a note here about an absurd interaction with someone you had concerning Restic. Alexander: Yes. This was one of the things that I mentioned earlier, where somebody was very sure that their hardware is not at fault. Jonathan: Oh, yes. Alexander: Because they tested Restic on a different machine and it has the same error, and at the end of the day it turned out that both machines were faulty in different ways. So sometimes Restic makes people very unhappy because they discover things that they- ... d- did not want to hear. Jonathan: Oh. Alexander: But it's a good idea to, to know that hardware is faulty even if they at first don't want to accept that. Jonathan: Yes. There are fun stories about the the various hardware bugs found in CPUs over the years. Alexander: Yeah. This is even a layer deeper, when the CPU itself is functioning correctly, but correctly means not really correctly. Jonathan: Yep. No I've heard of some of those. One of the, one of the very very early mini computer CPUs, you could put it into a loop that should last forever, and a couple of- ... a couple of engineers tried it. And, after about an hour it kicked itself out of the loop. Wasn't supposed to do it, but it did. And, they wrote in to the company that, who, DEC or whoever it was that made it, and it's "Ah, somebody found it." And the next, the next time they published the manual, there's a little asterisk there, "This command is not guaranteed to return." And then, you, we've had that in more modern processors too. There was a floating point error in one of the early Intel processors, and- Alexander: Oh, yeah, I remember. Yep. Jonathan: Yeah. All kind, all kinds of weird stuff that, we expect our computers to do what they say they're gonna do. That's the whole basis of this thing. And when it doesn't, it causes problems. Alexander: Yeah, this is something that, that engineering and computer science students at some point discovered, that floating point is not exact. But in terms of ab- absurd uses of Restic, there's also this ... I know of a company who has a lot of raw video material, and they use Restic to not only archive this, but also transfer it from location to a different location, because then they- they use a Restic repository, write all data there, and then transfer the files, because transferring a lot of smaller files that you can checksum before and after is much better than transferring a big file and only then discovering that something went wrong. Jonathan: Restic as a BitTorrent replacement. Alexander: Even that, because at the storage location, the repository is structured in a way that every file the file name for each file is the SHA-2 hash of its contents of the- ... encrypted version. So you can just use very simple tools to check at the storage location without knowing the password for the repository. You can check if bit rot has occurred and if something has changed. Jonathan: Oh, interesting. Yeah. That's a neat idea. All right. Is there anything that we didn't talk about that we should have? That's a Alexander: good guess. Oh, maybe a lot. One, one thing that also surprised me is that it's really important for a project to have good change logs. Because sometimes projects don't have any change logs or they paste the output of Git log into some GitHub release form or something like that. What we do instead is I've written a small tool to collect text files in a very simple format from a bunch of directories. And for each for each pull request that does something non-trivial, we require that there is a description within the change logs directory. And at first it's collected in the changelogs/unreleased directory. And whenever we do a release, we rename this directory so that then we can automatically generate well-written change logs from that. And this is it's not a Git log on the one hand, but it's not a complete explanation of what's going on. It's always a description of what is important for end users. So if I renamed variables or if I upgraded some library, that's not relevant. So it doesn't get a change log entry, but instead if some bug was discovered and fixed, or if compression was introduced, this is something that we want our users to to read about. And a- after using it for several years, I discovered that there are quite a lot of projects who use this simple program that I wrote some time ago to manage their change log. I think ownCloud is probably the most known one from that. Jonathan: Oh, cool. Alexander: But I think that's a really impor- important part of a project also. Jonathan: Yeah. I don't think I asked you what the what the encryption standard is that you use, actually. Alexander: Yeah. That's, it's AES 256. And the signature is made with Poly1305. And these were two of the encryption primitives that were available at the time, and that w- were sufficiently fast in Go. And so I had to make a trade-off between using something, a more standard construction, for example, AES GCM which is encryption and signature in, in one. Or building something myself. But since then, one of the Go crypto maintainers has had a look at the encryption protocol and did not find any issues with it. This is also something that I'd like to change and I would do different if I would, if I were to implement it again today. But now it's there and we have to support it. And it's fine as it is. Much more important is probably that we use a very intense key derivation function to get from the passwords to the encryption keys. And there are, like, different encryption keys for encryption and signing and so on. Jonathan: Yeah. AES 256 has really stood up the test of time in an impressive way. Yes. It's ... Yeah no, no real weaknesses in it have been found, and it's considered, quantum secure for the future and- Alexander: It's also very it's also much easier to do this to have this just attribute for a symmetric algorithm. The asymmetric algorithms always rely on some problem being mathematically hard. Yes. And if you discover either a new algorithm or if you have a different architecture, like quantum versus classic CPUs- which were not there when the protocol or the algorithm was designed, that's very hard to do. But for a symmetric algorithm that's much easier. Jonathan: Yes. Yes. I am wrestling with that right now- ... trying to figure out how to make an X25519 scheme quantum resistant at least, and not easy. Alexander: Indeed, yes. Jonathan: All right. I've got a couple questions that I've got to ask before we let you go. I think we have, I think we've gone through our hour already. And I will get emails if I don't ask you these. That is, what's your favorite text editor and scripting language? Alexander: My favorite text editor is Neovim. Jonathan: Okay. Alexander: Although I must admit I do use VS Code a lot these days. Sure. But with the Neovim extension. Jonathan: Okay. Alexander: And my favorite scripting language is probably Go, although that's technically not a scripting language, but you can, like- You can script with it ... use Go run and then- ... have a small file there. And it's usually I tend to ... Yeah if it's not Go, it's Python. Jonathan: No, that's fair. That's fair. All right. It has been an absolute blast to be able to talk with you, Alexander, to learn all about Restic. I'm ... Oh, I didn't ask. I'm gonna, I'm gonna cheat. I'm gonna throw one more question in here at the end because it was on my mind- Shoot and then ... Windows. Can you back up a Windows- Yes ... machine with Restic? Sure. How does Alexander: that work? We compile the binary for Windows and it just works. We don't have in the repository format, we have a adjacent document describing a folder of files, and if you have a sub-folder, then a different adjacent document is referenced. And so you don't have the slash versus backslash differentiation on the repository format. It's not ideal if you back up something on Windows and restore it on Linux or macOS, but it basically the basic functions work. You will get the content of the files. And some te- some things do not really translate well from one operating system to the other. ACLs, for example- ... on Windows. I'm not sure what the current implementation for that is, but there are some issues on macOS where file attributes are not really cannot be restored on other systems and so on. Interesting ... Windows was supported I think from one of the early releases. Somebody stepped up and did a port of Rustic to Windows, which is not so hard to do because Go just compiles- Yeah ... on Windows. Jonathan: Yeah. Very cool. All right. Appreciate the the last second answer there. Sure. Alexander, it's been a blast to get to talk to you. I appreciate it very much. Thank you for being on the show. Alexander: Yeah. Thank you for having me. Jonathan: Yeah, absolutely. All so that was Rustic, and was a lot of fun to get to talk about, and I've got some systems that maybe I need to start backing up with Rustic. Alexander: Please do. So we- You don't want back up, you want restore. Jonathan: There you go. Coming up in the the next few weeks, we've got Florian Gilcher of Ferris Systems talking, I think about something about Rust. And then we've got Tris Scherlicher talking about the legal system and Floss. Tris is a lawyer that I met at the Ubuntu Summit, and immediately he had great stories, and I immediately said, "Ah, we need to have you on the podcast." And then we're talking with a couple of guys from QNX, which that's gonna be interesting. And then we have at the 1st of July, the the first Tuesday in July, it's actually July 7th Andrea Gallo of RISC-V. He's actually the president of the RISC-V Foundation. And very much looking forward to that. And so a really neat lineup coming up over the next few weeks. I am very much looking forward to it. We appreciate everybody that's here, whether you get us live or on the download, those that watch and those that listen. And we'll be back next week for another Floss Weekly.
• 
  Open Source Gardening 10.06.2026 1h 12m

  This week Jonathan chats with Alexander Neumann about Restic, a particularly compelling backup and restore solution written in Go. Why did the world need one more backup program? And what's Alexander's personal take on transitioning from programmer to maintainer? Listen to find out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 869 - Linux on Your Toaster 29.04.2026 34m

  This week Jonathan chats with Andrei, Mahir, and Praneeth, live on location at Texas Instruments! The team at TI has been working hard to provide really good Open Source support for Sitara processors, including upstreaming support to the mainline Linux kernel. We talk about the CI pipeline for these devices, the challenges of doing Open Source at a big company, and more. Check it out! You can join the conversation in the Hackaday Discord, watch live or get the video version of the show on Youtube, as well as getting the full story and show links from Hackaday. Oh, and follow the official Mastadon account! Theme music: "Newer Wave" Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 License http://creativecommons.org/licenses/by/4.0/
• 
  Episode 868 transcript 22.04.2026

  FLOSS-868 Jonathan: This is Floss Weekly, episode 868. Recorded Tuesday, April the 21st. Remove the Noodles. Hey folks, it's time for Floss Weekly. That's the show about Free Libre and open source software. I am your host, Jonathan Bennet. And today we're gonna talk about programming. We're gonna talk about, is it fair to call it vibe coding? See, that's become a prerogative. Pejorative, excuse me, and I don't know that it's entirely accurate. Hopefully not accurate for what people are doing in open source projects, right? Anyway, the tooling is getting better. The tooling is getting better around this. And one of the fun things is that there's a lot of open source tooling around this, and that is what we're talking about today. I'm chatting with Johannes Millan about parallel code. And something he calls super productivity. Let's go ahead and bring him on and we'll dive into it. Johannes, welcome to the show. Johannes: Yeah, hi Jonathan. Thank you very much for having me. Jonathan: I am it's good to finally have you, this has been scheduled for a month and a half now. Quite a while. Yeah. And yeah, it's good. It's good to have you here. And so your project is parallel code? Johannes: Exactly. Yeah. I've two big projects. I say one Super Productivity, which is an source product project I've been working on for I think, nine years now. And just recently I think two months ago I started to work on parallel code yeah, which is by comparing much smaller. But yeah, also at the moment I'm dividing my time equally for both projects. And so Jonathan: These are two very different projects, aren't they? Johannes: Yes, they are. Jonathan: Let's talk about first the older one which is, because there are some people that are gonna go AI and just check out. So we'll talk about the not AI stuff first and then yeah. And then we'll dive into the LLM coding and maybe how those two go together more than you'd think. So what is super productivity? Johannes: Yeah, so productivity is an open source to do Time Tracker app which I started many years ago because I'm working as a freelance programmer for, yeah, I dunno, 15 years now or something like that. And and for some project there was required to do jra time tracking. And yeah, as as most programmers I don't like to do repeatable stuff. And so I thought, oh, there must be a smarter way to use this, and this is how it all started. And yeah, I don't know, some somehow I stuck with it. It's like sometimes became a little bit of an obsession of mine and yeah, it was, even though there was no, never making money with it or something like that was never yeah, a focus of it. Yeah, somehow. I dunno. I use it myself every day. That's probably a big part. And yeah, I really enjoy tinkering with my tools, like to yeah to make my day a little easier for myself. And so that's probably the reason why I couldn't just drop it and yeah, I don't know. And then in, in the last year, that's probably worth mentioning it grew quite a lot. I think maybe also a little bit there also many new people on the project who contribute stuff who yeah, do testing, write back reports, and it's. It's really interesting how this changed. And but for the most part of the seven years, I think or for the nine years yeah, it has been mostly a solar project. Not totally, there were always seven people, but yeah, Jonathan: yeah. I've not met many open source developers that wouldn't say. They like to tinker with tools. That sort of seems to be something that's true of all of us. That's why we're here. We like tinkering with the tools. Johannes: Yeah. Jonathan: Okay, so this is a setup for you to answer this question because it's gonna sound a little mean, and I don't mean it that way, but it's just a time tracker. How hard can it be? Surely that was like, you programmed it in a day and it was done, right? Johannes: Yeah it's a good question because I think the first prototype back then was much worse tools and without AI and everything, it was done, I think in maybe a week or something like that. And I dunno, I can't really say what happened, but there is there, the complexity grew. I think one part of it is the integrations like that it's connected to, yeah. To other tools. GitHub, like Jira GitLab and many more. That's one part of the complexity. And the other part is like the more yeah, also maybe a little bit typical for sources. If people keep requesting stuff and then there gets added more and more. In hindsight, I'm not sure that was always a good decision, I have to say, because it's much harder on many levels to remove stuff than to edit. Like you Absolutely, Jonathan: yes. Johannes: You also, people get angry sometimes, if you remove stuff, yeah, it's there's definitely some tension there. Jonathan: I've found myself using, I'm sure you're familiar with XKCD. I think a lot of our audience is there. There. One of the XKCD comics is entitled Workflow, and it's a bug that was fixed in an imaginary program that holding the space bar was causing, unlimited CPU cycles of just spin and spin. And the guy's I finally fixed the bug and you had multiple users writing in and going, no, that broke my workflow. I've used that, I've used that comic multiple times in the last few days. Johannes: Yeah. Yeah. That's, yeah, there's something to it. I think for the most part, like it's really a great community and like people are really friendly and there are many people that are very understanding also of the limitations of the project. Sometimes I maybe I or there have been occasions like when I broke stuff, and it's usually people yes, stay friendly through it. Some maybe panic a little bit. But yeah it's really surprising in a way to see like that there, people coming together and to, yeah, to work on something which is not yet their own, so to say. And then, yeah, being just friendly and, good. How do you say? Good, good. Good welding or well intentioned about it. It's a really, was really a nice surprise again and again. Jonathan: Yeah. So using super productivity I'm seeing a term here that I'm not super familiar with. Pomodoro it, it works as a Pomodoro timer. Johannes: Yeah. That's what, is that one of the many features like Yeah. Pomodoro is a technique for for focus work or also an anti procrastination technique. It's pretty simple. Like you take like these red Pomodoro timers, this kitchen clock you have sometimes and then you set it to 25 minutes and then you try to focus as much as possible for those 25 minutes. And, yeah. And this helps some people. And this also was I think important feature for many, I don't use it as much myself, but this, yeah. Also I have always been a part of the project, like the two not only be a to do app, but to bring in productivity methods to play with them and to, yeah, to really make it easier to work and more, also more enjoyable. I think that's also a part of it. And yeah, now it's still there, the feature, it's a little bit extended. There's also like a flow time timer and you can yeah. Switch between these different methods. Yeah. It's also maybe explanation like why it grew recently to pay a little respect to it. Like last year, I think it was I also added a plugin system the idea is to have a strong core, but that people, because I think productivity is very much about experimenting with stuff and things, everyone is different and for everyone there different methods that work better and also it changes for everyone individually. Like maybe for some specific task or for some time in your life. I dunno, the pulmonary method works very well, but at another time, maybe you are facing different problems. And so I always think it's good to experiment with the staff and like with the plugin system it's, and also with wipe coding which is now pretty open to everyone everyone can adjust this tool. Yeah, at a very deep level. Which I think yeah, it could be an interesting direction like where the project is heading. So far, I think there are some 20 plugins or something like this. There's stuff like how to put it like yeah, it, it's a gamification of of you to do. Oh, Jonathan: right. Yes. Johannes: It's an experience system and yeah, I think it's really cool. I honestly, I don't use it myself because I don't need at the moment, but I think it's really cool, to have the option to do that. Jonathan: Yeah. Yeah. Interesting. I'm poking at the the web version right now which is pretty fascinating. I found a coffee counter. I may have to get that going. Johannes: Yes. Jonathan: This, so there's versions of super productivity for all of the desktops and also Android and iOS and running on the web. Is that all one code base? Johannes: Yes, it's electron based. Jonathan: Oh, okay. Johannes: Okay, Jonathan: gotcha. Johannes: Electron. And for mobile platforms, it's capacitor js and yeah. That's what makes it possible. It's still quite the hassle sometimes, I have to say absolutely. Especially on, on Linux, unfortunately. It's really, and it's also not the fun part, I have to say, like this very special knowledge involved and knowledge. You don't. Use for anything else. And then it just, someday it breaks again. It's, yeah. What Jonathan: I will, I'll tell you the secret that I have found to making this work is find somebody in your community that actually uses it on Linux and hand the packaging duties over to them. Johannes: Yeah. Yeah. That's probably, that would be a smart idea. Jonathan: That's the way to do it. Yeah. Not everybody can pull that off, but when you can, man, it's great. Johannes: Yeah. Yeah. I have to think about that. It's really a great idea. Yeah. I think actually, yeah, Jonathan: find somebody that actually uses it on that platform. All right. So we support on all these different platforms and this immediately brings to mind, and I know it's in here 'cause I've seen it being able to sync between those different platforms. 'cause it's not very useful to have a to-do list on your phone and have a different one on your desktop and a different one on your Windows desktop specifically if those don't talk to each other. So what's the answer here? Johannes: There's also a long story to it. For some reason many years back I decided I don't want to have a backend because honestly, also because I didn't want to deal with illegal stuff. I didn't want to do, deal with the risks of losing the data of other people. And so I decided to go with the how it's called. There's a specific term for it, I forgot. It's not self-hosted. Jonathan: Yeah, Johannes: I don't remember that one. But there's a specific term for these kind of apps. It's not many who do it like this, but so I decided people have Google Drive. It's not that complicated of it. The data's not so complicated. And why not just, make sync, file based. And yeah, and that's what I did. I think I started with Google Drive and then I added Dropbox and also file based sync. And in, in hindsight also, maybe that was not the best decision because also likewise with integrations and the many many channels where there is available this yeah, it was a third party getting involved and there were problems. And also using services which are not really meant for it cause problems. And also I had to cancel Google Drive at some point because they changed their policies and and also then I added web. D FFR support which opened, no, a whole other bunch of problems. But yeah, it's still, I think it works well for most people. But all these ongoing problems led to the decision to, to work on a dedicated backend service, which you can self host if you want, but that yeah, that's specialized on, on, on syncing the kind of data this application produces. It's called super sync just because of the, yeah. Super in it. Actually I'm not so happy about the name if I'm honest, but it is what it is and it doesn't make sense to change. It probably would've flagged something, some, something clever, more like maybe banana tasks or whatever. But yeah, here we are. It's Google Bill. So that's what it has going for it, I think. Jonathan: Yeah. There you go. So you're hosting super sync for right now. Johannes: Yeah. Jonathan: And I see a note here. This service is free for now, but will likely cost money in the future, Johannes: which Jonathan: having servers is not free. Somebody's gotta pay for it. So Johannes: it's unfortunately true. And also it's yeah, personally I would really like to work on the app full time. I'm doing it now, but I'm doing it from my savings. So this won't work forever, especially now that I have a family to sustain. This was like yeah, an idea, like how it could work. Like it's really tricky subject, I think like monetization for source projects especially especially if you want to be as ethical about it as possible. And but I think like providing this, the service which costs money, like I, as you mentioned, like the service are not for free. The legal risks you enter by having that are not free if you if you want. And so I think it's yeah, it's a fair. Fair compromise to go about it and nobody has to use it. I won't remove like the other thing options that are already there. But yeah I'm hoping that it's that it's, that it will become a source of income which is enough to sustain myself, but at this point, it's not yeah, it's not clear if it will happen. Jonathan: Yeah. Johannes: Yeah. Jonathan: No, that's understandable. All right. Super interesting. And then there's this looks like there's password based encryption. I am, as we talk, I am sitting here poking at this, trying to set up a little account for myself. So Johannes: Yeah it, like hands on it's a good thing. Yeah, exactly. Yeah, that's, it's another thing we see all sync providers, but with a new one it's mandatory. You do like end-to-end encryption, which means like you enter password locally and the data gets encrypted on your machine. And so I can't I, and also nobody who maybe, I dunno, HEXA server or something like that will be able to read the data. Given you don't have a, I don't know, a two letter password or something or someone use a quantum computer. I don't know how this will play out honestly. But yeah, it's, so it's should be pretty secure so you don't have to worry about yeah. Other people doing stuff you don't want with your data, which I think is a good thing. Also, I think yeah. Important point actually for many people. That's that you don't just send all your data to the cloud and that you Yeah. That, what happened with it. It's you don't have to be dogmatic about it, but I think yeah, at least having an idea what's happening with it is probably a good thing. Yeah. Jonathan: Yeah. Absolutely. And as I'm poking around at this, I'm seeing that, you have just your simple task list. You have a time tracker, but you can also set it up as a productivity suite. Johannes: Yes. Jonathan: And so what's the, what all do you get with the productivity suite that's, beyond just time tracking? Johannes: Yeah, there is there is a habit tracker, which is pulled in. You, I have to look at actually at the, yeah. Okay. Now you have you have a notes feature. You have all the integrations you have you have a planner, which is which is yeah, for planning obviously over time. And then you have a schedule which is like a timetable. So super predictivity you wouldn't build in feature time boxing, which you also don't have to use. But it's there. And if you time box your tasks, if you assign a certain estimation to them the schedule will automatically generate an overview, like a timetable how it would play out. That's also another thing. And you get a boards feature, which allows you to yeah, configure all kinds of boards like Kanban Eisenhower Matrix yeah, basically whatever you want. It's pretty flexible and yeah, lots of stuff. And like coming back to it like it's with productivity, I think you, you don't, I often use this thing, oh I don't think I'm using it to its full potential and maybe it gives me a little bit of anxiety or something like that. But it's really about having the options for experiment if, when you need it. So I think for many people, probably, like who never had a to-do list app before, like just doing this is probably. Will work very well. I think, like it will really change how you how you go about your day. But if you need something more specific or for example, you feel like I don't know where my time went that's, you also have options for that. And yeah, that's that's what I personally, or maybe also what explains this obsession a little bit or why I didn't get bored of the app is how I use it evolved a lot through the years and changed a lot. And yeah. And I'm, I am hoping to strike a balance that it's still like very accessible for, I don't know, non-power users or first time users. But there, yeah, there really are a lot of options if you wanna experiment. Jonathan: Yeah. I'm finding more and more, the ability to like connect it to a Google calendar. Yeah. The ability to pull tasks from GitHub and a bunch of other Yeah. Integrations here. But GitHub is the one that really is most interesting to me. I'm trying to, I'm trying to get some of these things set up because it looks really cool. I probably have to use this Johannes: yeah, try it out. I recommend it. Jonathan: My, my current time tracking solution is unfortunately just a website that I think is probably entirely closed source. Maybe not. Maybe they have some open source stuff. I don't know. I won't mention the name of it then, because I don't remember. But this looks like it's more featured than that is. Johannes: What's your reason for for doing tap tracking Jonathan: How you do it? So I have gone full time with the Mesic project as a programmer, but also a business person and just trying to. Trying to understand how much time I'm putting into that. Yeah is fairly important as some things develop and trying to track, which pieces of hardware I'm working on. And so that's what I'm working on. But the ability to it would be really interesting to be able to say, okay, let's pull in the GitHub issues that are assigned to me, and then, okay, I'm working on this one right now, I'm working on that one. And to automatically get that time tracking, like that's a cool idea. I could, yeah, I could get behind that. Johannes: Yeah, Jonathan: it'd be very neat. Like I've spent. I mentioned it before the show, I spent some time today. This is Lily Go T watch, S3 plus which is a cool little piece of hardware. It's, so it's got a lo radio in it and SP 32 S3 GPS IMU, all the normal stuff that you would expect out of a mastic device. And someone sent in a pull request, 'cause one of the buttons is wired up directly to a power control unit inside of there. And so we didn't have any, we couldn't observe the button and somebody sent in a pull request and said, Hey, here's how you observe the buttons that you can turn the screen on and off. And it's oh cool. So I pulled the pull request down and started testing it and the device crashed. It's oh, that's not cool. Come to find out the device crash was even without the pull request and we just hadn't noticed it. So this is why we have Alpha Software. So I've been, most of the morning I've been playing with that, trying to understand why and how we are crashing. I think I'm just about there, making progress on it, but yeah, that's been fun. Oh, anyway, let's see. It looks like things like the Google Calendar sync don't necessarily work from the web app. Johannes: I know. Yeah. With Google, the problem is yeah, it's something I started before my little break and it's actually a Google thing. You, I, it's a legal thing. I need to I dunno. I need to send them a video about the permissions and these kind of thing. And and yeah, hopefully I get to tomorrow and then it probably will take another week or two for them to check it, and then hopefully it'll work for everyone again. Jonathan: Yeah. Johannes: Yeah. Jonathan: Gotcha. Yeah. I'm I could share my screen and show you, I'm getting the message access blocked. This app's request is invalid. Yeah, Johannes: yeah. Yeah. It looks scary. Jonathan: It does. If you're not familiar with what's going on under the hood, alright, so that is super productivity, which is a very cool project in and of itself. And now I wanna pivot for a minute and talk about the new project, and that's parallel code and what is parallel code? Let's start there and then we'll talk about why it exists and what you're using it for. Johannes: Yeah. Probably code is wrapper for agent cl coding clients, like code codex, like Gemini, CI. And yeah, it also came to life because I saw a gap like when working with these tools more and more myself. I think. At some point I started was using them within my IDE and then they got more powerful and you could trust them a little bit more to run on. Yeah. To let them run on their own. And so it developed that I just had multiple terminals side by side to each other. And that Jonathan: describes the Linux experience for me, multiple terminals side by side. Johannes: Yeah. Which is a wonderful thing. There, I don't have anything bad to say about it, but I thought it's repetitive and as I said being a programmer, that's doesn't feel good. Sure. Jonathan: No, I gotcha. I gotcha. Johannes: So I thought yeah let's, it'll add, it's make this easier, like also this using GI workflows for it because it just makes sense to have isolated context when you're working on stuff in parallel and to add information, which I need all the time. Also not like with teamworks, you can also do all the stuff but you have to configure it again every time, or I dunno, maybe maybe you don't. But that's as far as I got with teamworks. And so I thought, yeah, it's I like the terminal, but there's also something to be said about the good user interface. I thought, yeah, it's not too complicated and I, I need it for myself. And so it started and yeah, and then I thought, yeah, maybe this is useful for other people too because I really thought it's pushing my, my, my workflow to another level really because yeah, with, I think with agent decoding now it's still, for some people it's still very controversial. Subject and rightfully but I think for most programmers it's already in reality and it's very exhausting way of coding, especially if you're doing stuff if you're doing multitasking it's not really what the human brain is built for and it's really exhausting. Like after you, you have to have a high concentration. And what connects to this or what is the reason for that is that you yeah, that these context switches are of hard. For example, like in my previous workflow, I was switching I had the terminals running side by side, and then maybe I had another bunch of terminals open side by side. And then I wanted to check the changes. And then I opened my IDE again. Then I wanted to check in the browser and open that. And, while there still is like this you're still doing this multitasking. It's a little bit softer because you don't have as harsh of a change with this parallel code because you have it on one screen and it's doesn't feel as disconnected as Yeah. Switching to another app does. And yeah, and yeah, basically I build around my own use case. There is there is like a changed file view there. You can fire up sub terminals like for example, to run your test suite, or you to just quickly check something with the command line. You can yes. Not it's not too crazy of a feature, but there's a notes panel, which is surprisingly useful because Jonathan: I find myself opening a a dedicated application just to make notes all the time. Like Kwright is usually I have usually multiple instances of Kwright to either, pop a quick note in or, to, I've got, I think I have two of them open right now, or one K right, with two tabs with a couple of crash outputs. I have no little, throwaway notes open all the time yeah. Is parallel code and IDE then, Johannes: oh, I dunno what the correct term is actually, it's, the soft base is so new in a way. I, yeah I dunno I dunno what the correct technical term is. Maybe I would call the wrap like, very pretty generic term, but it's rep for different, c Eyes. That's maybe, and it, Jonathan: so it, lets, I wanna make sure I have my understanding of this is right it's almost just a container for terminals, although there's obviously some more sparks in it than that. But as you first look at it, like looking at the screenshots here, it's just multiple terminal instances side by side. Yeah. But what this is doing is it's letting you load up your different coding agents, so you know, whether you're working with Gemini or Claude, or whichever ones you have access to. And so is the idea here that you would have, you have one task that you're trying to get done, and you want to give that same task to both Claude and Gemini to see how they approach it differently? Is that sort of, that also, do you use what I, I'm, I've only begun dipping my toes into the water of letting the AI right code. I've had good success with it so far. But I am not the I'm not the power user for this yet, so I'm still trying to understand like, what all can you do with it? What, so what does your workflow look like when you have a programming task to get done, or multiples, let's say? How do you use parallel code to make that happen? Johannes: Yeah. A reason for why multitasking is becoming a, has become a reality for me is because you wait for for the agents to finish, they're doing that thing, whatever they're doing. And so for me, like I'm an impatient person and I don't know, maybe I watched many YouTube videos or something like that, but I get distracted really fast. If I have this downtime in between I, oh, maybe I start surfing this and that and it's. Always requires like willpower to go back from that. So it's also another thing which exhausting. And so I started like opening up these terminals on different tasks at the same time. Sometimes you can do at work at, on the same stuff at the same time, but it's tricky because it gets messy. For example, you have two agents writing the same file and and then you can get a new weird, it could possibly Jonathan: go wrong. Johannes: Oh, the file change. Let me change it back. Yes. Oh, the file change, let me change. And, so it's for me personally, you can use it. There's also maybe I can come back to this later. There's also a feature for for letting agents do the same thing. But for me personally, it's mostly about this doing the same thing doing different things in Lia. So for example, there's it's also GitHub integration. So I just drop GitHub link from Backport, I drop it onto the interface and then and then it spins up and it analyzes. And so then, I dunno, this maybe takes five minutes and I can maybe open up five sessions and then I can check okay. It's set in that output and okay. We probably should proceed like this. And, that's how I use it and what's, what it, it doesn't do something completely new, but it does all those things I would usually do, do having another good work tree and branch for that. And all this is yeah, much. There's much less friction and and so you can work step by step at these things without hopefully getting yeah, losing track of what you're doing less because the UI is built around specific specifically that there's yeah, Jonathan: super interesting. And so I, one of the, one of the features that I saw here is that. Parallel code actually does isolation between agents so that you can have multiple agents working on the same code base at the same time without doing exactly what you described. Oh, something changed this file, lemme change it back. Johannes: Yeah. Jonathan: How does that work? What's the approach? Johannes: You have good work trees, so this what I usually do, but you can also, there's also a feature which if you need more isolation and also more security you can also use a docker container to have a completely isolated session that are the two options. You can also work on the main branch if you want. But like the how I use it the most often during the day is just having this good work trees alongside each other. Jonathan: I've never used get work tree, and I immediately feel that I need to look into this. I Google it more. Me neither. Johannes: Before Jonathan: I Google it, one of the first things I see is I was doing this and normally I would just stash it, but that gets really obtuse. Yes. Yeah. Yes. I have multiple STEs and sometimes you lose track of what's which one is which. Johannes: Yeah. Yeah. For good work trees, it's basically get copy copies, like your whole code base into another folder. So like you Yeah. Also like with your third party dependencies and everything. So you really have no conflict there if you're working on that. Nice. Jonathan: I'll have to look into this in and of itself, but back to the ais. So I'm still, I'm curious about the actual editing step. So I've, again I've dipped my toes into working with ais and letting them write code. I still find that I have to go in to some extent and clean it up. We talk about the facts that the AI likes to make lasagna. It makes too many layers. Like it'll do functions with, single line functions just to be able to get a prettier name on it. And it's no, I understand what you're trying to do, but man, that makes it difficult to actually read the code. I talked about needing to go back in after the AI and remove some of the noodles out of the lasagna to make the code better. How does that part work? Johannes: Yeah. That's it's an important thing and to still check the output. Probably people, my assumption would be that people don't do it as thoroughly anymore because of the sheer amount of code you, you have to deal with. It's for example, like before when working with a team and someone wrote a pull request and let's say there may be, there are 10 changes, the 10 lines changed I review. So really I can say something about that, but if if I see like he has written my coworker has written like a thousand lines of code, then maybe I'm maybe I'm looking at this like really quickly or Jonathan: sure, Johannes: yeah. Looking at this differently. But yeah, I think it's still important to do and what code adds. Also not like a brand new crazy concept, but it adapts the changed file tree you have. Which you would have also, like for example, if you do a pull request on GitHub and you can quickly go into this files. I dunno, or maybe a side question does, would it make sense to share my screen to show stuff hands on or, Jonathan: If you can do it inside of inside a video ninja, I can probably capture that. Yes. Johannes: Yeah. I hope this is. Is it big enough? I don't know. Can you see something? It's not too Jonathan: bad. Yeah, I'll get it pulled Johannes: up. I can make it a little bit bigger. Yeah, there. Like for, from the UI level, what parallel code does, there's just like this diff you. You can select stuff here and you can comment in. For example this is stupid. Oh, Jonathan: wow. Johannes: No, this is Jonathan: great. I, so the integration with all this stuff is a lot slicker than I expected it to be. To be able to just click on something and immediately go to the code and be able to write it feels a lot like working inside of the GitHub web actually. Johannes: Yeah. Yeah. That's the inspiration because like they're wisely the the professionals about it. Yeah, that's an adaption basically of what they do. You can also open like your local editor, if you click here you can also for example, you cannot just comment, but you can also ask please explain this to me. I don. No anything. Okay. And then it's syncing and and hopefully it at some point it, it gives you like these Oh. Jonathan: So when you do an ask it, it is an immediate query, but when you do comments Yes. You then gotta hit the button to send to agent. Okay. Johannes: Yes. What it does, it's just execute. Oh yeah, there you go. Oh these freelance, yeah. Jonathan: Small unit test for the scheduled component. Here's what Johannes: they do. Yeah. Who would've thought? Yeah. And yeah that I find it pretty handy feature. But as I said, like it's not something super innovative or something innovative. Yeah. And what I also like maybe can also share this. Sure. So I can. Just drop in GitHub links. Jonathan: Yeah, I did wanna see this. This sounded pretty cool. Johannes: Unfortunately. Maybe less Texas. This is a super productivity park. And then yeah, it prefilled, it, it auto to detect the project. It yeah, pre, so it figures out the branch, the issue. Yeah. Then you can choose which tool you want to use. Then you can decide if you want to use work tree. And you can also run it in a docker container. And yeah, Jonathan: you can skip all confirms if you really Johannes: feel Jonathan: brave. Johannes: Yeah. Then it does, its sing. And also another layer, like for this getting more better feel, or maybe let me start like this. Another thing I currently dislike about coding with AI is the feeling of losing control because you, you are at least, maybe there are people better than me who don't do it as much, but I am I'm not as thorough and as I was before ai. Because as I said, there is so much code you have and you delegate. You, you have to delegate. Also like part parts of the responsibility. But there layers to, to ate this and of course like 1, 1, 1 is testing and so what parly code does you can like create terminal shortcuts for, which could be basically everything. But here for this, what makes sense is like to have like the test command run, which runs the unit tests or the end-to-end tests. And so you have this all clustered in this block, which belongs to this task. So you don't have a terminal here, which belongs to this task and another terminal there, which also belongs to the same task. But you have it like a cluster that UI wise which hopefully makes it or which think does make it easier for the brain to yeah. To down feel as lost when doing these kind of things. Yeah, Jonathan: this is very interesting. What have you found is the most helpful for you? You talked about the difficulty in reviewing all of this code. What have you found that really helps to try to make sure that not only your work, but also pull requests that are vibe coded? How do you best make sure that the quality stays high? Johannes: Yeah. Yeah. There are multiple layers. I say the stuff or it's good to have layers which don't require your own mental capacity. I think like you can you have unit tests, you have end-to-end tests. You also have from the AI workflows there, I, for example, I have different skills for cloud code I'm using like, which. Then in turn fire up multiple agents and different models like my, my multiview task I, I'm using for most stuff is like, fire up five regular agents for different aspects. And then it it fires up also Codex and copilot. And so you have hopefully they're producing something useful. But usually I think it it helps to identify problems before, before I have to step in. So the idea is you try to reduce like the amount of stuff you need to think about yourself as much as possible. And so this means I hopefully when I start reviewing I has done the best it. So I, I just have to deal with the stuff which is left over after these obvious yeah. These layers beforehand. Jonathan: Do you ever do things like let one AI review another? This is something that we do quite a bit actually in the mesh astic project. 'cause Johannes: Yeah, Jonathan: all of that is run through GitHub and in GitHub there's the copilot reviews you can ask copilot for review. Johannes: Yeah. Jonathan: And we found that to be useful actually. Yeah. Like it, Johannes: yeah, Jonathan: it's it sometimes will be a little too nitpicky, but it usually finds really good stuff. I wrote a I wrote a pull request just the other day. I was doing notifications. We wanted, in the desktop app, we wanted notifications and I was feeling lazy that day. And so instead of using Lib notify, I just used the binary that comes with lib notify and just built the string and pushed it all out as a command. And, then did the poll request. I'm like, yeah, it works. And then we asked co-pilot to review it and co-pilot's you probably shouldn't use untrusted user strings on the command line like this. Yeah. And I went, oh, of course I can't do it that way. Johannes: Yeah, no, definitely. But I have to say, it's something there, there already people like on, on GitHub for the project who think who are thinking further about this and how to integrate it within the application. But for me at this point, this is something I try to figure out on the model level which means like for me most of the time using or creating cloud code skills for that, maybe I can quickly show like how this looks multi review. That's it. Yeah. Hope this is readable somehow. Yeah. Basically what I have is a file which defines like which other agents to, to start for this kind of review. So there's one specifically checking for correctness. Like a change really does what it's supposed to do. Then there's one for security, then there's one for architecture and design. Then there's one for exploring alternative approaches, and then the current one. Then there's one for performance. Then there's one for simplicity, and then there's one codex reviewer, like using another model, which just like a general review of everything and yeah. That's basically like how I, at the current time find it best to work because also yeah Phil philosophy of of the of the app is like not to not to be its own ecosystem. Like open how it's called open code is like where you have another layer, like its own model of how do have to structure all this AI stuff, but just to be a wrapper about the workflows you already have. For example, if this means if maybe in a two months time or something like I dunno, jet Brains maybe brings out a totally crazy new IDE just for agent decoding. Then you don't lose anything by working with power Layer code now, and this is also important I think for me at least, because the space is moving so crazy fast or has been moving crazy fast for the last year that it for me doesn't make much sense. Yeah. To redevelop the wheel and yeah. Jonathan: Yeah, absolutely. Let's see. I was gonna go a certain direction and of course I something dinged on my computer and I've completely squirreled and forgotten the the direction that I was going to go. Let's see. Oh, pushback from the community. I am super curious. Yeah. Like in, so you've got the existing project. Have you gotten pushback from the, have people said, ah, none of this AI on get off my lawn with your AI tooling? Johannes: Yeah. Jonathan: Yeah, because you've seen projects out there that are I've seen one project that have completely closed poll requests to anyone outside the organization. Yeah. I've seen other projects that are talking about going closed source because of the AI thing. Just some, some of it I think is really a step too far in, in some of those cases. But what's your experience been with that? Johannes: I, it's really hard to say I, from a, from moral standpoint, I. Don't have a, I don't have a clear opinion about it. Like there, but from how I personally work, like I was skeptical also in the beginning, like when the whole AI wave started. But it is currently, like for me, there's no doubt that it's more efficient at the moment. It's the most efficient tool to use while coding is using ai. And to come back to your question, like I, I don't have any, i've emotional connection to my code, of course, but I don't feel like I, I don't also, the license super productivity has, for example, is MIT I'm really okay with people doing what they want with it because yeah if somebody takes it and does something better with it I'm fine with it. I know that's probably not doing the subject justice because there are other aspects to it, but yeah. I Jonathan: Have Johannes: you resident, yeah. Have Jonathan: you, have you Johannes: resident think it makes much of a difference in a total, like if the models or new AI overloads have access to the code of super productivity or not, it doesn't make, probably want Sure. Make a big difference. Jonathan: Have you seen an uptick in low quality pull requests because of ai? Yeah. Johannes: Yeah. Jonathan: Is it pretty bad or just a little bit? Johannes: Just a little bit. Okay. I think, yeah, I think on the other hand, the stuff is better documented. It's more about also there's sometimes it's too much information because AI to produce so much text, like if you have a poor request, like with a long technical specification. And to be honest, I don't always read it very carefully. So Jonathan: yeah. Johannes: But yeah, on the other hand, like it's two to two sides to, on the other hand, like it enables people like, who are not programmers or who would never have the time to submit code to do it. Like to say, oh, there's a problem I'm seeing. And the very least it does for me is to. Yeah. To to get attention like that, oh, there is this problem. And even with the code, like it's horrible. I know about this. And and especially the part, like getting more people involved I think is a, it's generally a great thing even if it comes with some challenges. Jonathan: Yeah. The, some of the most interesting and terrible, but also interesting AI poll requests is where people ask the AI to do something that we didn't think was possible or we thought would be extremely difficult. And sometimes you've got people, I'm thinking of, I'm thinking of one poll request in particular where, Johannes: yeah, Jonathan: We had told people for the longest time, there's really not a way to do this. It would be such a huge lift, and somebody apparently had the tokens to burn on it and has what may or may not be working probably. Honestly, probably working at this point a poll request to make the thing happen. And in this case, it's running mesh tastic as a part of Zephyr, which, if you're in the embedded space, you know what I just said. And if you're not, then you know, that doesn't make any sense to you. But it was it's something a lot of people have asked for and it was gonna be a hairy a hairy thing. And it's like someone ignored the fact that we said that this would be really hard and just went out and said, I'm gonna do it. And made the AI do tortured the AI into doing it for them. And now you look at it, it's okay, first off, I had told someone offhandedly a couple of months ago. I'm like, if you actually wanna do this, here's the way to do it. And that's the way the AI approached it as well. And so that was interesting to me. But just like the fact that it I, it lets people challenge these, pronouncements is interesting. Sometimes it's a good thing, maybe sometimes not such a good thing because there, in some cases there were reasons the project didn't wanna do that. I don't know, it's just, it, it does, it's really, it's changed the game. It has, it's changed. Open source AI has changed open source in ways that I don't think we fully understand yet, but there's no putting that genie back in the bottle. Johannes: Yeah. Yeah, definitely. I totally agree that there, it changed a lot. Like for example, for before a year ago I really put a lot of effort into merging every pull request because I wanted. To get people involved. And like there, there haven't been so many contributions by other people. And these days I find myself more and more having to say no to stuff to say oh yeah, it's well done. It's totally, it's, yeah, it's great code. You did really well. And, but it's it's it's too much. It's it's a feature. It's too many features. Jonathan: You did a great job prompting your AI to write this code. Good Johannes: work. Yeah. But yeah, it's like, for me personally I still feel ownership for these things I'm doing. Of course it's losing something I dunno, back then when you wrote code by yourself it's it's satisfying. Like it's, there's a lot of beauty and well written code and a lot of satis, or at least I got a lot of satisfaction out of just key beautiful code, like absolutely clear concept behind it. And you can tap yourself on the back. Well done. Jonathan: Sure. Johannes: And this is different now. This is yeah, you, it's more, it's on a, more, on an architectural level maybe. And more on a, it's for super productivity, it's more on a product level that, that the decisions or the stuff I do this more about the decisions I you feel ownership to, and I think rightfully than for the Yeah. Intricate details of the code itself because you might not always operate on the level anymore. Jonathan: Yeah. Do you think there's a danger that so one of, one of the. One of the problems people suggested is that you have to treat, and I don't know if this is true anymore, things have developed so, so rapidly, but people said you have to treat the AI as though it's a junior program. Junior programmer may be an intern, and you have to be the senior programmer. And then there's this question that comes about, then how do we get new senior programmers? Yeah. This is a reasonable concern, but I think the question is, will we run out of senior programmers first or will AI get good to the point to where it can then also be the senior programmer? Johannes: Yeah. Jonathan: I'm not sure which of those is worse. Johannes: I also don't know, like for the time being I've settled for I just don't know because it's, yeah, it's all involving so fast and I think there are points to be made for, yeah, for things just change, but there's still need for programmers. I don't know. Yeah, I think a couple of days ago I read like that there may be I'm probably quoting the number wrong, but that there may be one, 1% of the global population or 1.3% of the global workforce for our programmers. And actually to me, if I think about that, I think that sounds like quite a lot actually. Do we need this much software? For so maybe there will be some sort of adjustment. There maybe will be a little bit less programmers. But at the moment, like from my own experience and how the past year working with AI has been, I still think like the technical decisions behind that's yeah, I may, people smarter than me said like that, that AI is not yet a good architect. And I hope it stays this way. Jonathan: Job security. Johannes: Yeah. Jonathan: Alright, so the big question I've gotta ask is, do you use, do dog food, this, do you use parallel code to work on parallel code and super productivity? And while you do do you use super productivity to track your time doing both of those things? Johannes: I do. I do. I do. That's the beauty of it. Jonathan: Yes. It's, Johannes: I've worked very many years as a programmer and yeah, to be honest, I. Apart from the code itself? For most, to most products I was involved with, I didn't feel a strong connection. I didn't, to be honest, I didn't for much of the stuff, I don't think it's, has been necessary to do at all. It's really, for me, it's really nice to work on, yeah. On, on, on my tools on things I'm using daily. And that's it's, I think it's also an effort for the project. I am, I'm thinking much differently about super productivity then I would for about some yeah. Freelance gig. Project. Jonathan: Sure. Are you still freelancing? Johannes: Yes. Jonathan: Yes. That makes sense. Yeah. Alright, we have we have hit the bottom of the hour. I've got two questions, two final questions I've gotta ask, I gotta ask this to everybody. What is your favorite text editor and scripting language? Johannes: Yeah, as my, it's, I don't know if it's the right answer because it's an IDE, but I have used JetBrains. Yeah, Intelli J Idea for many years and I really liked it. But to be honest, I haven't used it as much in the past months. Jonathan: I parallel code is almost an IDE. It blurs the line into being Johannes: it's my favorite and it's the best one. Jonathan: There you go. And then the scripting language. Johannes: Scripting language. Yeah, I think I still prefer type script just because I'm most comfortable with the language. There's some stuff I'm doing with Spanish and some stuff I'm doing with Python, and Python is very elegant. It's a beautiful language. But I'm I like to work with tools I'm comfortable with. So it's Cript. Jonathan: Yeah. Yeah. Very cool. Alright. If people wanna find out more about these two projects, where is the place to go? Johannes: Yeah. Probably the easiest way to find both is to go on, on, on my GitHub profile. There are both projects linked. Jonathan: Okay. Johannes: And yeah. And you find all the other stuff from there. Jonathan: Okay. We'll make sure and get some of those links in the show notes as well. Cool. Thank you Johannes, for being here. It's been a, it's been a delight to get to chat with you. Johannes: Yeah, likewise. Thank you. Thank you very much for having me as it has been an honor. Jonathan: Yeah, it's been a lot of fun. And I've got a couple of projects to check out here. One that I'm in the middle of checking out, and the other one I'm gonna have, I might talk myself into, doing some more AI work. I'm still tipping, dipping my toes into it, but it's fun. Alright that has been Johannes Milan talking about parallel code and super productivity, which I already have a super productivity account spun up here and got it installed. I'm gonna play with that some next week. If everything goes according to plan, we will not be live. We will instead be recording a show on location at a a particular place down in Texas. Hopefully we can make that work. And then it'll come to you, not those that watch live, but those that that get the download at the normal time, we hope. But it's been great to be back after missing a couple of weeks and yeah, we'll be back next week. All things all things go well. After that, the schedule is still looking a little bare. So if you wanna be on the show or know somebody that needs to be hit us up. Let us know. You can email us@flossathackaday.com or you can get ahold of me at the various places where I hang out. Appreciate everybody that's here. Those will get us live and on the download, and we will see you next week on Floss Weekly.