Open Source and Communities With Heitor Lessa

Transcript

Tiago Barbosa - PagerDuty: Welcome to Page It To the Limit, a podcast where we explore what it takes to run software in production successfully. We cover leading practices used in the software industry to improve both systems reliability and the lives of people supporting those systems. I’m your Host Tiago Barbosa, and you can find me on Twitter at t1agob. Welcome Heitor! Nice to see you again, and yeah, welcome to the show! For everyone that is listening to the podcast episode, we are going to discuss open source communities and open source with Heitor Lessa from AWS. Welcome, Heitor.

Heitor Lessa - AWS: Thanks for having me. Hey, good to see you again.

Tiago Barbosa - PagerDuty: Yeah, same. So we worked previously at AWS few years ago, same team. You were doing a slightly different thing, but already working on AWS Powertools. So please give us a quick introduction on what you have been doing with AWS in the last few years and well basically…your background.

Heitor Lessa - AWS: Sure, absolutely. Well, so I’m Heitor Lessa, for everyone who I haven’t met before. I’m a chief architect in a product called Powertools for AWS. It’s a fancy job title, just to basically say that I do the execution, the planning, working with communities, a bunch of different roles, tech writers and so forth. This product Powertools for AWS is a developer toolkit. It helps customers implement serverless best practices, which are defined in a project called AWS Well Architected, which I was luckily part of it to help alter the serverless best practices as well, so if they go hand in hand. But eventually over the years, this project, which I thought it would be a nice way to showcase it, could be easier to implement. These practices, blew up in the sense that it became way, way too popular from a project that was literally trying to do this on a weekend and as anything else that you don’t really expect much of an adoption. It went from simple downloads like 10, 100 downloads a month to now 44, 45 million loads a year to thousands and thousands of customers. I can share the specifics, but we were aggregating actually just last week when we were doing operational planning and we handled something like trillions of API integrations with this project as well. So it’s definitely no longer a toy and I’m more than happy to share all the challenges and how this came from a very little tiny project to having a full dedicated team now.

Tiago Barbosa - PagerDuty: Yeah, that’s pretty cool. And so with AWS Lambda Powertools, you have support for different languages. I know that Python is your main one, right? But you also have support for other languages, right?

Heitor Lessa - AWS: Yeah, absolutely. So Powertools is in four languages today, Python, TypeScript, .net, and Java. Python leads the way largely because Serverless is huge in Python, but also because Python has a quite diverse set of people using the programming language for all kinds of tasks. Powertools is definitely initially focused on developers, but eventually we found out that a lot of data engineers, data scientists, platform engineers and DevOps be focusing and even AppSec related recently starting to use power tools. But TypeScript is now the fastest growing in terms of adoption numbers.

Tiago Barbosa - PagerDuty: Cool. And do you have an idea, so I know that you have a bunch of different features. So you have these core capabilities related to metrics, logging and tracing, but then you have a lot more, and you already mentioned that Python leads the way from that and now other languages are keeping up. Do you have an idea of how many do you have, how many features or capabilities you have right now?

Heitor Lessa - AWS: Yeah, I was trying to count it now because we are adding more next month. I think features wise, I think it was 56+ if I’m not mistaken, but what we call utilities, which is more of a major feature like Tracer. Then we have around 15 of those because of the nature of open source. We started with observability, but it ended up becoming more of someone called it a last week actually a customer said this is no longer the observability toolkit or serverless developer toolkit. This is kind of a modern application toolkit framework type. So it goes from observability, batch processing, idempotency, feature flags and whatnot.

Tiago Barbosa - PagerDuty: Yeah, so as you said, it’s part of bringing best practices and improving developer experience. So observability is one of the main problems or major problems that you started to try to solve initially, but then it grew to something else and it’s a very complete tool right now. So shifting a little bit into the community side of things, because this is a tool that, as you said, started as a hobby that you were doing during the weekends and free time, and now it has its own team contributing to these tools and you also have people from the community, people that don’t necessarily or that don’t work for AWS also contributing to the tools. Was this planned? How did you manage to grow the team internally and how you managed to grow the community and keep it active?

Heitor Lessa - AWS: So it was planned but not at this proportion. What do I mean by that? So when I wrote with other folks as well, I wasn’t the only one. I was just leading the write up and all the processes to the serverless best practices for AWS. There was a clear need in the market for frameworks that would adapt the likes of SpringBoot and Flask, Django and Node.js Express and whatnot to something that would fit Lambda. But that means a lot of institutional knowledge in how developers love writing APIs and authorization should basically build applications into something that it would work for Lambda, that you don’t even have to make a judgment call anymore of no longer using the tools you love just to use Lambda. It shouldn’t be a question of a choice anymore. So that was kind of a vision I want people to have no longer that discussion of, oh, I have to only use Node.js if I want to use Lambda because the tools are out there, the community’s all behind there. So I had that clear in my head on what I wanted to do. And the other piece that was clear, but I had to obviously evolved, the clarity of thinking was I knew I wouldn’t be able to do it alone because it’s too big. I don’t come from a development background. I had to learn how-to, and I still learn a lot every day. And what I planned was I am going to try this thing out, but I will primarily work with the community because I have no guarantees that the company would actually fund this and I would start as an experiment. That’s why we started entering into the AWS labs and it was clearly called out. And so everything I did, which was super hard if I’m honest, was forcing myself to work in public. And by working in public, I was always trying to work building this community aspect externally and internally. But nowadays, I was trying to count as we were speaking, just last year we had 1,300 pull requests and changes as a whole and roughly 67% came from the community.

Tiago Barbosa - PagerDuty: Okay, yeah, that’s a lot. And that’s really good to see because it’s very difficult to grow a community that stays there and keeps contributing to the products for a long time. And so it is really good to see that you get those numbers and you have a lot of contributions from the community. It’s definitely relevant. And one point that I wanted to dig a little bit deeper on is you said that you weren’t sure that the company would invest in this kind of component that you were building initially, like a small component that you were building initially that is now much larger. So was it very difficult to sell this and how did you do it?

Heitor Lessa - AWS: Oh yeah.

Tiago Barbosa - PagerDuty: I’m kind of trying to fish for some tips.

Heitor Lessa - AWS: Yeah, absolutely. Anyone working in open source, you get that thrill of building a community and especially if you come from a startup background where you wear multiple hats, open source is amazing for that because it passes your SKUs in every single level. I read a quote somewhere, I can’t remember who said it, but that’s something like maintaining an open source project is like running a business you didn’t know you had. And when I looked into the open source and everything kind of became my life post business hours, I’ve noticed that, hang on, this is actually a business without actually making money. It would be very, very difficult to get buy-in an investment, especially when we have funded places like AWS SDKs and others, which they do a stellar job and they’re way more effective than me just building the SDKs, which is why Powertools builds on top of these things. So when I was questioning myself on that one, I was thinking what is it that resonates with the company? But obviously I can’t knock on every single door asking for investments. This is a business that has to run at the end of the day. I also had my day-to-day job. So what I did was trying to map out the leadership in the organization that was in inside AWS and then try to work out their incentives and the goals that I was also aligned because at the end of the day, it has to be achieved a kind of a two-way street where doing something like this also helps their goals and whatnot. So it wasn’t so much about let’s build power tools and get investment because power tools is awesome. It was more how do I make power tools crucial to the business and to the community without losing the integrity and the vision, the tenants, et cetera. So that was a very hard one. After they figured that out on what was important and such, I had to group the challenges I had in front of me to customer adoption, the popularity, how many downloads you have and who’s actually using by afar because you can only tell by people making comments. And then you go there from the GitHub and then you see which company they work for, but that’s not really sustainable, is it? And then the second one was, okay, downloads is great when you start and GitHub starts like, yay, it’s fun but what does it actually mean at the end of the day? So I needed to find out the commercial impact when power tools was used and what exactly was contributing to for the other organizations consume it and the plethora of personas that I mentioned earlier so that it become trying to discover solutions for that. How do I track without getting in the way because especially Lambda, because you pay every millisecond of it. So adding something intrusive, it would be against our tenants first off. And second, we’re not going to add something that we force customers to be tracked for the sake of getting investment. So it took four years of trial and error until we got it. If I were to summarize was trying to explain the customer adoption angle, the commercial impact to the business and how he worked with the rest of the organization and the goals that the organization had explaining that this was aligned, this was a boost and augmented their commercial initiatives, et cetera, then it became an easier to palette or to digest.

Tiago Barbosa - PagerDuty: Yeah, that’s pretty interesting. Another thing that you mentioned already I think twice is one is you decided to build this in public and this is something that you struggled a little bit with because it was not natural from what I understood. The second thing is that it took four years to get to this point. One of the things that we see in social media these days is that there are a lot of people trying to build in public and expect success in one year. And everyone is, I’m going to build these six different SaaS services and products this year, and yeah, I’m going to reach, I don’t know, X amount of money and this is my goal for the year. And it’s not always like that. And actually there are very few cases where you can actually do this. So it takes a long time, even if you just start something today, there’s a lot of work that you did in the past, a lot of failures and challenges that you’ve been through in the past. So my question is, there are many challenges I bet that you faced during these last four years in growing Powertools to the scale that it has today. So can you share with us some of the challenges that you faced, some of the main difficulties that you faced, I don’t know, growing the community, interacting with the community or even internally some of the challenges that you had that you would like to give us some advices on?

Heitor Lessa - AWS: Yeah, if I think from a timeline, the first challenge of working in public is that imposter syndrome is actually augmented in a way that you are at the spotlight now. Everyone sees every code change you make, every rationale you put out there, the way you write and everything else. So it definitely took me a lot of time to get used to that. Only now I know that it’s not natural. Even onboarding someone to the team now can take four months, eight months to get to that written to get comfortable with that piece. I guess the first challenge, it came from a book that’s called Working in Public. I forgot the whole cover, but I can send you the links of people listening later. There’s an issue that’s called Maintainers Attention Depletion. When you’re building a community from marketing purposes, it’s amazing that you have people reacting and contributing and working together and speaking about these things, which is what you want. You want these reactions, you want this viral effect In open source, it could be life or death in the figurative way that a project could be very, very popular, can go to the hacker news page or you name it. But it could also mean that this now person who’s working by himself, by herself could be now out of the blue, completely devoid of time to do anything. Everything else is responding to people answering everything and the internet is not kind. You also have the opposite side of the fence, which you’ve barely talked about on how toxic it can become when people start demanding things or when they start contributing and then they think you own them something as well in return. So there’s so many tracks that it takes a while to navigate. So soft skills is something that I had to learn a lot, especially writing and learning the hardest for me, which was - let me close the laptop. I won’t argue today. This is not the time. And it’s different from the day-to-day work because you’re always receiving messages on your phone community and you kind of want people to work and contribute. So that was definitely the challenge, number one. I think the challenge number two was getting into that phase where as you mentioned as well, I had this idea that you have to be a 10x engineer, you have to be one of those big brains, should build something super clever engineering wise. And what I noticed was actually those big brains are already doing some of the work for me, the AWS SDK, and some of those amazing people. All I need to do now is the more human side of user experience, the attention to details, a great documentation. So what I learned on that challenge was it wasn’t so much about building something on the weekend that will be so popular and so forth, this rarely lasts. What I learned was consistency was key as more on growing the same thing for a while and planning and adjusting your plan, sticking with our vision, but adjusting the details and bringing people along with the journey and trying to be vulnerable because that’s what bonds us as human beings. And this was super great because I went from, oh my God, everyone will see how a fraud I am to, oh, I love this thing. I’m working with so many people, I’m learning so much. And now I’m learning from people who are excellent at APIs, others are excellence at cryptography, others are fantastic at data pipelines. So I became a sponge and trying to learn from those things. So those were the two major challenges if I will. And then lastly as we already talked to you was this is becoming too big. How do I stop it from dying and how do I get investments, but in a way that doesn’t become an empire otherwise you would change the heart of it.

Tiago Barbosa - PagerDuty: Yeah, no, it definitely makes sense. And I think you touched a really important point there, and one of the things that I always use as a reference in my projects is the way that you do release notes. So in every release you have, so typically in the projects you just have all the commits that go in that specific release, but you don’t, right? So in AWS Lambda Powertools, basically you have a summary of the changes with screenshots and all that stuff. So it’s really well documented so people don’t need to actually go into the commits to understand what was added on that specific release. You have more humanly or human readable information on your release notes and that’s perfect. And there’s a lot of investments as well, if I’m correct, a lot of investment as well in automating that, right? Because as you said, you need to scale. You’re just one man doing that job or now you have a team. But if different people do the same job, we would do it differently. So there’s a lot of investment in automation, correct?

Heitor Lessa - AWS: Oh, absolutely. In our documentation, which again, we can send the link, there’s a page that says processes and it has some of the automation in CI/CD, our whole release processes, even a single release, it goes with something like nearly 200 deployments in a single release. All the checks and everything else is automated. I also, another challenge and also learning was all these amazing projects that I saw and trying to learn from, these super clever people and whatnot, is I noticed that there was always so much pressure or rush to make a release because they spent so much time releasing something, working on something. And for me it was how are my customers going to know how much effort I put in into that release? What if they’re not developers, which is basically our secret sauce. What if I’m actually going for something that everyone should understand? And if they’re developers they can get extra details, extra sauce if you like. So we automate nearly everything except one, two things that we will never automate and that’s our decision, unless I guess it never issue much, but you get the point. Release notes is one release notes for me as a philosophy is this is your time as an open source maintainer to go out there and tell them, consumers, everyone, the whole community who actually contributed to this release, what exactly is in there for them, why should they upgrade in the first place? Even upgrade guides, major releases. We spent months actually writing these things as well. And the second is documentation people over time, over the years, especially after we went way above the 20 million downloads and such and grew fantastically well was a lot of people came because people recommended we didn’t do any marketing, which was amazing as well. And it was only the word of mouth. And they came in, tried the feature, they liked the experience, but they stayed because of the documentation, how we interacted the community, et cetera. So this human element is like I would never try to automate. I tried a few things once and it was always too much of a gut feeling. Now my biggest challenge for the next few years is to document an operational model publicly on the different activities and roles. Someone plays in Powertools, but it could equally be applied elsewhere and create a training on how to write these things in public, how to write documents with these notes.

Tiago Barbosa - PagerDuty: Yeah, I would pay for that. That’s really cool. And as you said, this is something that you’ve learned also from others projects that you were using and you learn from the community and you are one of the mechanisms that you are using to give back to the community as well is just giving them visibility when they are part of this new release, they have contributed with something not necessarily code, so it might be just contributing to docs or something else, but basically you are giving them visibility and that’s really important for people that are contributing to open source projects to be recognized.

Heitor Lessa - AWS: A hundred percent. Yeah. There’s definitely this missconception that a contribution is code when the hardest thing is documentation, which is why we spend so much time, we know how hard it is, but there’s also even contributions like making a comment with a detailed explanation on your reasoning why something is good or something is bad, or even something like, I wrote a blog post, let me contribute by saying, I wrote this blog post, would you like to feature this? Or you name it, in our Powertools we have even documented now for the first time, if you never contributed to the project, here’s the types of contributions you can make. Even after all this, you don’t even know where to begin. Here’s some ideas and whatnot.

Tiago Barbosa - PagerDuty: Yeah, that’s pretty cool. Do you also run Discord channel or Slack channel or something like that for Powertools?

Heitor Lessa - AWS: We do, yeah. So we had Slack for a long time and then we had to migrate to Discord and now it’s roughly 600+ people. The only thing we were not doing yet, which maybe this year possibly definitely next year, this year, there’s a lot of operational focus is to start running public office hours. We don’t do that yet.

Tiago Barbosa - PagerDuty: Yeah, okay. Would you say that that’s one of the main priorities for what’s coming for Powertools and how you interact and manage the community?

Heitor Lessa - AWS: I think community wise, definitely. Yeah, I think office hours is one of those great aspects of being there and listening to everyone. But I also know many people feel that they’re also going to be on the spotlight. I want to do a different office hours after watching and listening to many office hours from different open source projects, including some of those very big projects, I want to do it differently. Maybe you’ll got wrong, we’ll never know. This year we will focus a lot more on operational excellence, how do we do features, how do we plan things, how do we do marketing, how do we go back to the community and such is to try and do our product planning in public, open up a line, Discord, Chime, I don’t know, you name it, and allow people to join. I’m not necessarily looking at input, but at least tell them, here’s how we’re going to plan those features, here’s how your input matters. And then eventually we can open up for people so they can see that there’s a lot more than just making a pull request quest.

Tiago Barbosa - PagerDuty: Yeah, yeah, totally. Okay, Heitor, just so we close this one, I have a challenge for you. So based on your experience with Powertools and other open source projects that you are contributing to, what would be the one advice that you would give to people and what would be one thing that you would change if you could do something different?

Heitor Lessa - AWS: I think I will focus more now on people who are at companies. They have a day job and they love their companies and they might have an open source project that they would like to get funding or whatnot. I think my advice for you, which it took me a while to learn, is don’t worry too much about downloads. This is a popularity metric. This might not resonate well with your leadership. Try to understand from their side what matters to them and see if you can work backwards from that or map out what your project could also do or potentially adjust so then you can speak the same language and now you can work as a team and go beyond. One thing, I would change one thing that we’re doing now this year, which is going to be a bit taxing, and I wish I had done it before, but we also didn’t have that much of investment. So I can’t necessarily blame me or anyone in the team. Nobody. It’s always like a team effort takes a village is I couldn’t see an open source and now they were writing it. It was very difficult to explain to anyone in leadership. And if I were to hire someone to be in an open source position, an open source maintainer, what does this person do? How do I measure, how do I track, how do I promote someone? So I’m now writing all these things now and I wish I had spent more effort on that before we assembled the whole team, et cetera. We’re obviously going, well, we’re getting by, but if we had this before, it would have been less chaotic. Let’s put this way.

Tiago Barbosa - PagerDuty: Yeah. Oh cool. Well, Heitor, thank you for sharing all these insights with us, with me personally in the audience. It was a pleasure to have you on the show. Is there anything else that you want to share with the audience or…

Heitor Lessa - AWS: I think I’ll share some links, that’s for sure. The book, I mentioned that some of the links from Power two so people can join and also our Discord so people can join and have this conversation as well.

Tiago Barbosa - PagerDuty: Perfect. We’ll have these links to the podcast information and thank you so much once again for joining us today. It was a really good to have you here. Thank you so much.

Heitor Lessa - AWS: Pleasure as always!

Tiago Barbosa - PagerDuty: That does for another installment of Page It To the Limit, we’d like to thank our sponsor PagerDuty for making this podcast possible. Remember to subscribe to this podcast if you like what you’ve heard. You can find our show notes at pager two, the limit.com, and you can reach us on Twitter at Page It To the Limit using the number two. That’s @pageit2thelimit. Let us know what you think of the show and thank you so much for joining us. And remember Uneventful Days are Beautiful Days.