The Dark Side of Open Source With Dotan Horovits.

Transcript

Mandi Walls (00:09): Welcome to Page It to The Limit, a podcast where we explore what it takes to run software in production successfully. We cover leading practices used in the software industry to improve the system reliability and the lives of the people supporting those systems. I’m your host, Mandi Walls. Find me at LNXCHK on Twitter. Alright folks, welcome back to Page To The Limit. I’m Mandi Walls. I’m your host today with me, I have a very special guest. We’re going to be running a mini series throughout this year about topics in open source. So we’re hoping to chat with some folks across the open source ecosystem about not really how to contribute to open source, what open source is, but some advanced topics around open source. And our first guest here is Dotan and he’s going to talk about all kinds of wild stuff that’s going on in the open source community right now. And if you haven’t seen him live somewhere, I’m surprised because the man is all over the world all the time. So welcome to the show. Tell us a bit about yourself and what you’re doing these days.

Dotan Horovits (01:13): Yeah, thanks for having me on the show and so great to be here after following the show in your works for so long. And I’m actually A CNCF ambassador talking about the open source. So the CNCF is the Cloud Native Computing Foundation. I’ve been around open source for I guess over a decade now and primarily in the recent years around the cloud native stack and DevOps observability, platform engineering. These are the areas of passion projects such as Open Telemetry and Prometheus, Jaeger. Those are familiar with the observability space. And recently more focused around open search, the OpenSearch project. And actually just very recently, I guess I am glad to say that I joined AWS as the, I guess chief evangelist for the OpenSearch open source project. So going to delve into that much, much more in the coming months and years.

Mandi Walls (02:14): Excellent. That’s fantastic. So well let’s start with OpenSearch. Thinking about projects that are part of different foundations and other things that you’ve sort of represented over the years, tell us about OpenSearch as a project. It had a big milestone recently. What’s going on with that particular project right now?

Dotan Horovits (02:36): Yeah, so it’s been already over three years now. It started as a fork of Elasticsearch and Kibana, the popular ELK stack that we all know and love from the observability and logo analytics and search space. But I guess the big news that just came out is the fact that OpenSearch joined the Linux Foundation. So it’s now officially a foundational project. It’s something that has been led by AWS so far, but now it’s open for everyone. We have quite a few already members that are active in contributing like SAP and others, the major one’s Uber. So now it’s open for all for those who are deliberating whether to join OpenSearch because it’s under a vendor now it’s foundational open source.

Mandi Walls (03:32): What’s that process look like for folks who have a project that might want to get spun out or donated to a foundation? What’s that process actually? What’s the mechanics there for that?

Dotan Horovits (03:44): So first of all, the question of which foundation is best fit, obviously Linux Foundation and the CNCF that I mentioned, I’m an ambassador of the CNCF and the CNCF is also affiliated with the Linux Foundation. And there are others like the CDF, the Continuous Delivery Foundation, Apache Software Foundation, Eclipse Foundation, so many. So obviously it’s relevant. What’s the natural home if you wish for your project synergy to the other projects in the stack, the governance policy and other aspects that are relevant for your project. And then depending on the project, the size, the maturity, then there’s an evaluation and fit. I can talk for example in the CNCF that have been an ambassador for a good few years, there’s very clear maturity ladder and projects go through from sandbox to incubation to graduated project. Obviously also some deprecated at the end of life.

Dotan Horovits (04:43): And then when a project wants to join based on the evaluation, it’s determined whether the it’s sandbox level project as the entry point or for obviously many less projects are fit. But those that can jump straight into get straight into the incubation phase. And it’s all based on the, I guess the open source maturity signals, like the cadence of releases, the diversity of the foundation, sorry, the members that are active there, vendors and individuals, the regularity of releasing the drops and all the other aspects that you’d expect. Obviously how many deployed in production, all these aspects that will give you the certainty that this is a sustainable open source that will not disappear in a year or two. And then it’s very important to say, because this is different than the software maturities, people tend to confuse that with having a software piece, whether open source or not, being generally available or beta or alpha. This is something else. This is this softer level of maturity or feature maturity level and so on. This what was talking about is the open source project’s vital signs, if you like.

Mandi Walls (06:01): Yeah, so subtly different from the actual process of do you have automated builds and stuff like that. The other aspect of the project itself, so you mentioned sustainability as something that plays into this and as we look at what’s been going on across the ecosystem, and it’s almost hard to think about open source as an ecosystem, it’s almost its own industry within our industry. There’s just, there’s open source everywhere, but there’s been a lot of maybe disruption or sometimes it’s just a consternation what is going on with some of these projects. But from a sustainability standpoint, something that we think about with commercial vendors, are they going to cancel this project? But those kinds of things. But with open source, the other concern is are the folks behind this going to keep doing it? Are there enough of them to keep it going? Especially if it’s not something that is financially backed, they’re donating their time or those kinds of things. So some of these forks and things, and we’ll get into some of those in a minute, but as these projects go out into the world, what kind of sustainability concerns are there for some of these projects over time?

Dotan Horovits (07:32): So yeah, is becoming increasingly relevant with all sorts of incidents that we’ve seen. But I guess the most common thing that we see is when you have a single point of failure, when you have essentially a project owned or essentially backed by a single vendor, and obviously when this is the case, on the one hand, a single vendor can boost and put a lot of resources when this is aligned with the business strategy and push the project forward. And it could be a startup, it could be a multinational public company, but then again the question arises when the business does not align so well with the open source and with the community needs. And then this is where the friction starts happening. And it could be like decision to change the license to relicense the project, whether taking it entirely off the open source grid or to make it more constrained, I dunno, copyleft from Apache to copyleft A GPL or things like that or other things, changing terms of use or terms of service, access to the resources and so on.

Mandi Walls (08:55): Yeah, for sure. So you had written an article on Medium that you posted on LinkedIn. I’m going to link it in the show notes because there’s just a lot in it about things that have happened over the past, over the course of 2024 that are concerning I think for consumers of open source as projects shift and change or the corporate sponsors change their, like you mentioned licensing components and things like that. It seemed like 2024 was a busier year for that than we’ve seen maybe in the past then what do you see that’s kind of going on across the industry? There’s just been so many projects that have been taking closed source or their open source is one version back or all these licensing changes and things like that that have happened. It just seems

Dotan Horovits (09:50): Yeah, for sure. I actually, I just came back from a State of Open Con, conference in London and that was a hot topic there. Everyone has been discussing this and I also gave a talk about this year that has happened and also some best practices that maybe to stay safe, to keep safe in these situations, but it’s definitely top of mind. When I wrote an article, I called it the Dark Side of When Your Open Source turns to the Dark Side. A few years ago it was funny, people were giggling saying You’re being overly dramatic on this one. And then last year at Open Source Summit, suddenly I see the same blog post when your open source turns to dark side on the keynote there at the open source summit. So you see that it’s becoming top of mind for everyone. And that’s why I wrote this one, which is like the community strikes back if you wish, because there is an upside, there is a positive angle, which is again, the resilience of communities.

Dotan Horovits (10:48): But before we talk about that, you talk about what are the types of things that we’ve seen. And just the past year alone, if you look at it, we saw things such as big noise around Redis, the very popular caching key value store that has been around for ages and suddenly goes off open source. And this is a critical component for so many communities out there. If you look at the Jango and Python and so many others that use it as an internal component and Linux distros and others. So this has definitely shook the community quite a bit. And we all remember all the things that happened with Terraform and other projects by HashiCorp, the change license. This is something that is obviously very, it makes the community think, okay, how are we going forward with that? Especially ones that count on this being open source.

Dotan Horovits (11:47): For example, if I’m an open source project, my dependencies need to be open source. This is part of my definition. So if one of my dependencies, whether a library or a tool or something else suddenly becomes a non-open source license or even an open source license, but not the license that I’m okay with for my project, for example, CNCF projects align with Apache v2 and the dependencies need to be similarly aligned. And what happens if I started with a dependency that was such, but suddenly it’s not, it’s another type of open source license that’s a problem. So this is why it’s becoming really problematic to lots of consumers, whether open source or closed source. And this creates lots of dynamics such as forking and others. So it’s very interesting to see.

Mandi Walls (12:37): It’s not just like if you have an enterprise project and you’re paying for a requirement, it’s not like they’ve increased the cost. They’ve actually made it illegal for you to include that based on what you are doing with the software. And it’s not that this often goes to court, but it’s not like you want to think about that or risk it on your project.

Dotan Horovits (13:06): Exactly, exactly. And when you choose any tool, open source or not, you have your set of criteria for choosing, right, based on performance, based on cost, based on others. And if one of the criteria is a certain licensing model or licensing scheme, that is something you bank on, you count on. And then if this changes, this could even become a business risk for your company. So this is something that is definitely top of mind. By the way. It could also happen with individual contributors that are not a vendor. It could be the vast majority of projects out there on GitHub are actually maintained by a single maintainer, maybe two maintainers. And just imagine if that single maintainer suddenly disappears for all sorts of reasons.

Mandi Walls (13:51): Yeah, well, so that happened to Vim, right? The main maintainer of Vim passed away at the end of 2023. And there was just recently I think a talk about that maybe late 2024, about what that project went through to honestly recover the project. You need a succession plan for some of these projects that you wouldn’t necessarily be thinking about with a commercial component. There’s going to be other employees most of the time of some of these companies. But yeah, there was a really interesting discussion of what happened to Vim when, he passed.

Dotan Horovits (14:34): And it’s crazy how popular it is as a tool, how vastly used it is. And by the way, the example that I love most is actually Log4j and that everyone came to realize because of the CVE with the Log4Shell, and then people realized how vastly popular it is and used throughout, and this was backed by two maintainers at the time or something like that and everything, literally everything was using it. So it goes to show it doesn’t necessarily need to be something negligible. Even the most popular things that you have installed in your dishwasher and your car could be based on open source pieces that are maintained by a single maintainer. That’s also a risk. So a single point of failure, whichever type one source is an issue.

Mandi Walls (15:22): Yeah. Oh, for sure. Yeah. So you mentioned the community strikes back. So there’s been a few interesting ones that were included. Obviously OpenTofu was sort of an interesting project. I felt like that popped up within about 45 minutes of the announcement by HashiCorp that they were changing their licenses. Project was almost ready to go as soon as that was there. And there’s been others, I was part of Chef when our licenses kind of got changed and some forks popped up like the cinc project and some other things. And now in that same field, Puppet has changed theirs as well.

Dotan Horovits (16:05): Actually they haven’t changed the license, just to be fair with ‘em. Not that they changed the license, but they announced that they’re going to essentially, I guess reduce their investment in the open source and instead divert a lot of their investment into a sort of closed version that is like, that’s where, and it was an official blog post that came out and said our open source 2025 plans. So in this case the project is still Apache two, but the question begs itself if the resources are shifted into a closed repo that you need to even sign if you want, even as an external developer want to have a look at this repo, have access and engage with that one, you’ll still need to sign some special agreements and things like that that doesn’t feel like it. And that’s what drove the community, that community to fork and to establish another, I guess a fork of that. And it’s now merged into a VoxPupuli. So there is an established entity there of Puppet, I guess enthusiast and professionals that is willing. So it’s not starting from scratch, which is a good thing for the community there

Mandi Walls (17:25): For sure. The VoxPupuli has been, they’ve always been active and there’s been other things that got rolled into the Puppet ecosystem that I think started out as Vox projects rather than actually stuff from Puppet. It’s been a while since I’ve dug into their structures. But yeah, they’re an active bunch. We’ll say they’re very active, but for things like OpenTofu, that kind of spawned out of thin air, I wasn’t watching close enough to know that there were that many folks that were, I guess really concentrated in that part of the Terraform ecosystem that they would’ve been ready to go to put that together. But then you turn around and they had a list of grievances already that they were like, we want this, this, and this. So we’re going to fork your stuff and off we go and we’re going to put these things in because you didn’t want to do it. So from the user standpoint, there might’ve been some things in there. You’re like, Hey, OpenTofu is going to give me this, so I’m headed that direction regardless of what Hashi wants to do. Was super interesting at the birth of that project.

Dotan Horovits (18:39): Yeah, in this case, this is really the dynamics, as you said, it’s actually vendors from the Terraform ecosystem that just joined forces when they realized that this is the move. I actually even had a dedicated episode in my podcast, by the way. Everyone’s invited the Open Observability Talks if you like podcasts and you like the topic, but I remember that was very briefly after the announced that it was with one of the co-founders of env0, which is part of the founder members of OpenTofu. And it was amazing to see how they competitors out there in the commercial were able to gather around this joint mission of keeping Terraform open source. And yeah, it is very, very decisive. It took time. It was like a manifesto that tried to change the verdict by HashiCorp and they went ahead with the fork and so on.

Dotan Horovits (19:29): I think actually by the way, the fastest that I’ve seen is Valkey, which is the fork of Redis that happened this year. That was within a month. It was already GAed. That was I think, record breaking thing. That is crazy. At the moment that Redis announced, I think Redis Labs announced the relicensing of Redis. I think within days there was the announcement that fork is happening. And here the dynamics was slightly different because you actually had members of the core, the technical steering committee, if you will, of the leadership of Redis ones obviously that were not Redis Labs employees that actively drove this. So it was really driven by members, core members of the community. And it was joined, it was immersed into the foundation from day one. So it wasn’t done and then contributed. It was part of the initiation as well. And within I think a month, if I’m not mistaken, they already released a generally available version of Valkey. So that’s astonishing, I think.

Mandi Walls (20:34): Yeah. Oh, that’s a lot of work. That’s a lot of enthusiasm or just need for the project to put that kind of time into it right off the bat to get what they want out of it.

Dotan Horovits (20:49): Yeah, definitely.

Mandi Walls (20:50): Do we ever see these the other direction? Do the companies ever change their minds? I can’t recall any offhand. It was like, oh well that was the wrong direction to go. So we’re going to change back to our original setup.

Dotan Horovits (21:11): I know that the Corporate Elastic added an additional license that is open source just before the announcement of contributing OpenSearch to the Linux Foundation. So you could say today that Elastic has, I guess a third license in addition to the dual license they had before that now has AGPL license. But whether it’s understanding that it was the wrong move, it’s not back to the Apache two license, so it’s not really reverting back. So I dunno, I don’t really know if this is the thing that will regain the confidence of the users because it’s not just about the licenses. We all understand that now these just go to show us that licenses not something you can count on. We need to look beyond the license. Open source is more than a license. This is something that I keep on telling people that tend to stick to this part and why, because first license can change as we see.

Dotan Horovits (22:12): And secondly, there are very other critical components that people need to look into. What’s the governance model? Who backs the project? Is it backed by a single vendor or a single entity or a single contributor? Is there a clear governance policy that shows, so going back by the way to what we talked about at the beginning, the vital signs of a mature open source, these are the exact vital signs that show the sustainability of the open source and the resilience from such cases. So I guess the same vital signs that I guess the foundations look into when we talked about how foundations evaluate projects, it’s the same that we as individuals, we as companies adopting projects and tools and libraries, we should evaluate the projects, the dependencies that we take. Definitely it’s mission critical for us. That should be the process.

Mandi Walls (23:06): Things about conflict resolution and good participation. And we get so many projects to start off without even a code of conduct for contributors. And you’re just like,

Mandi Walls (23:19): This is level set, man. This is the first thing you want to check in after your README, is that particular file, get that stuff started right off the bat. But there was one more story you had in your article that I followed a little bit, but I admit I do not understand the ecosystem at all. And that is the Automattic and WordPress thing that was going on and it reached the news for, right, I saw it in places I wasn’t expecting to see this kind of rigamarole, I guess, and I’m like, what is even going on here? There’s some kind of copyright thing and there’s an actual lawsuit and there’s all this crazy business going on. What happened there? What was really going on the WordPress side of the house?

Dotan Horovits (24:10): So I think this was a very important lesson for us all in the industry, in the open source in particular, that people were, I guess not very clear on that open source relates to the source, namely the source code and other pieces are not necessarily open. And the classic is the brand, the branding elements, the logo and even the name. So the fact that you can use can access the code and maybe even modify it and then redistribute. It doesn’t mean that you can call it the same name as the open source, the original open source. And this is something that people got confused about. And in this case, I think educated everyone about that because in this case, the WordPress brand belongs to, or if you owns owned by Automattic and then it’s even more confusing because there is a foundation, there’s the WordPress Foundation, which is sort of an open source foundation.

Dotan Horovits (25:09): But then again, there is, and I’m not a lawyer and I won’t even try and explain that, but essentially a way that the brand is being sublet or subleased, I dunno, whatever to Automattic. So in a way, the bottom line is that they are the ones who can decide who can use that brand. So when another vendor came and they felt that they don’t contribute enough to the open source, then they said, okay, if you don’t contribute more to justify, then you can’t use this and that or you’ll be exposed to in terms of the branding elements, otherwise you’ll be exposed to a lawsuit. And that started rolling again, I’m not going into the legal aspect. I think the relevant part for us as a community is understanding that there’s a very, very solid open source here that runs what 60% if not more of our web that is now at jeopardy because of this fracture in the community.

Dotan Horovits (26:13): And the fact of sharing the load in the open source is definitely a valid topic to discuss. But the impact on, and then following that also automatic, decided to shift away some of the resources and then divest in the open source and other things, the impact is enormous and it’s still ongoing. So I can share the bottom line I guess we are all looking to see what happens, but it’s such a shame for such a successful open source that drives so much of everything that we see and do and use on the web every day to be at jeopardy because of these things. So

Mandi Walls (26:55): Yeah, so should all these things make users more cautious?

Dotan Horovits (27:01): I think we need to be more mature in the way that we use, consume, obviously contribute and build open source, that’s for sure. I think we definitely need to understand, as I said, as users that open source is more than just a license. And look beyond that into the aspects of, we talked about. As builders of open source, we need to understand that open source is not a business model. So if you do decide to take your project and put it out there under an open source license, you need to understand if you’re a vendor and need to really make sure that you have a sustainable business model around that, that you understand how you’re going to make money and how that making money is not going to conflict with the open source and its value to the community. So on top of that, and not jeopardizing that or even it as an individual, if you decide as an individual working out of your, I dunno, basement release an open source, don’t expect material compensation on that.

Dotan Horovits (28:02): There’s a ton of opportunities out there to get paid for software development and so on. But if you decide to take your development, your baby and release it as open source, don’t expect material composition. And we saw great examples of very successful projects that are being used by all the Fortune five hundreds and everywhere and are still maintained by a single maintainer. The fact that all the Fortune five hundreds use your project still doesn’t mean that you’ll get paid for that. So I think these are basic assumptions need to be understood. People need to understand that these are not good assumptions to have or realistic at least. And then when you use the open source, you can also take some precautions to when you use it. Very basic example, obviously beyond just checking the license is to take care with automation. If you have dependencies and we all have automations, we’re all engineers.

Dotan Horovits (29:02): But when you do that, be very careful with auto updates of the third party dependencies without having guardrails on the licensing. Because imagine the ones that moved from, I dunno, one version to another version. Sometimes even minor release in terms of the numbering conventions that might be then in different license. If you haven’t caught it, you’re now in trouble if you just went with the automation. So things like that that make sure that it’s still part of top of mind for you and the day-to-day operation of your system, I guess system architecture and system operations. This is how we can still use and I believe open source and I think that, but just do it in a more mature manner and more seeing the risks, understanding them and make sure that you protect yourself when choosing the open source, when using the open source, when building the open source, even redistributing some of our listeners probably are also that take and repackage and deliver also in these aspects how to do that safely and not expose yourself to the risks.

Mandi Walls (30:11): Excellent. There are so many things to consider now. They’re just so much more than, oh, I’m just going to pull this thing down from GitHub and use for a while and see what happens. There’s just stuff. Open source has eaten the world. It absolutely has basically taken over and like you say, the practices haven’t matured in a way that reflects that ubiquity I think yet. So

Dotan Horovits (30:39): Yeah,

Mandi Walls (30:39): It’s been amazing to watch, but my goodness, all these things going on.

Dotan Horovits (30:45): But then again, remember that, let’s look at the upside. Let’s finish with the upside, which is the vitality of the communities that were able to recover when there was a vibrant community around these projects. Even if the vendor decided to change a license, the community responded. I gave the example of Valkey that within a month we able look for can release, we talked about Puppet and now the new fork under VoxPupuli and others. So you see that when there’s a vibrant community, they can join together to keep that project open source. They can act swiftly, decisively and keep the momentum going. And also looking a year forward or two years forward, which is always the question about sustainability. It’s easy to hit the four button, but will it exist within one year or two years from now when the main backer, which is the vendor, is no longer in the picture?

Dotan Horovits (31:36): That’s a very valid question. And I think the past year also showed us in this aspect also very good examples. We saw value reaching, general availability. We saw OpenBao reaching, general availability. So we saw projects that have been forked and then fast forward a year or more and we see that there’s not only still there, we see that they’re releasing, they’re stable, they provide the generally available software drops and also the ecosystem looking at more vendors joining, more vendors, contributing more use cases of adopters running it in production. So I think there are good signs, as I said, for the community strikes back.

Mandi Walls (32:22): Yes. Oh for sure. Yeah, 100%. That has been great. Yeah, so thank you for joining me. Thank you for illuminating all of these amazing stories. Like you say, the vibrancy of the community right now is really just kind of amazing. There’s just everywhere you look, there’s someone who contributes to open source in some way and in a way that I think 10 years ago we just weren’t seeing the explosion of participation and the enthusiasm for it has just grown immensely and it’s been really something to see. So thank you from reporting from the ground and keeping us all informed of what’s going on there and it’s great to talk to you. So thank you for joining me today.

Dotan Horovits (33:08): Yeah, pleasure being here. Thanks for inviting me.

Mandi Walls (33:11): Excellent. So for everybody else out there, we’ll have another episode for you in a couple of weeks and we will wish you an uneventful day. That does it for another installment of Page it to the Limit. We’d like to thank our sponsor, PagerDuty for making this podcast possible. Remember to subscribe to this podcast. If you like what you’ve heard, you can find our show notes at page it to the limit.com and you can reach us on Twitter at page it to the limit using the number two. Thank you so much for joining us and remember, uneventful days are beautiful days.