There will be bleeps

Well, we are here with a breaking Changelog - Justin asked me to do that pun - a crossover episode. We are publishing shows to both Changelogging & Friends and to Justin’s Breaking Change/Hot fix/Merge Conflict… I don’t know what this is on his pod, but it’ll be there. The explicit version will be over on Justin’s side. On our side, there will be bleeps, because we also have not just Justin, but also Mike McQuaid with us. What’s up, Mike?

Thanks for having me. I hope to make heavy use of your bleep count today, as it’s my Scottish self-employed tradition.

Well, Mike’s only requirement was that there would become a non-bleeped version of his voice out there on the internet talking about this, and so Justin will happily oblige.

Yes, and I’m not going to make it a contest or anything, but I’ve got a feeling I’m not going to go bleep-free for what we’re about to talk about.

And the reason for the bleeps is because we’ve got trouble right here at Ruby Central… Yes, that’s an old music man?

What is that? I don’t know…

…for a new problem – maybe not a new problem, an old problem; an issue that’s been going on with Ruby Gems, with Ruby Central, with Ruby Together… With Ruby, the community, more so than the programming language. The language is doing just fine, isn’t it, Mike?

It seems to be. I wrote some today, it still works…

[laughs] Did you install any gems?

I did. They installed okay. It seems to be fine.

Yeah, I was actually doing an iOS project when all this stuff broke, but then I was worried that Ruby would stop working… So I dropped that, switched gears, and now I’ve been working on my posse party project in earnest, to try to get it done before the servers turn off…

Just in case, just in case. Well, there’s a lot of ins, a lot of outs, a lot of what have yous, as the dude would say, to this particular story… More probably than I can summarize, which is why I’ve mostly ignored this in Changelog News, because there’s just so much going on, and every once in a while I just link over to Justin, who’s been commentating and color commentating… But we’re going to let Mike try to set the table for us, just some of the events that’s going on, for those who are uninitiated with some of the Ruby drama that’s been percolating and coming to a head recently with an actual root access event published on RubyGems.org. So Mike, help everybody understand exactly what’s been going down.

Yeah, so I guess things kicked off probably – where we are… We’re October 15th at the time of recording. So this time a month ago, things seemed to all be fairly normal, and stable, and whatever. No one seemed to really know much.

I guess my first personal involvement was there was a governance PR on Ruby Gems that was based on the Homebrew one, I was pulled in and asked to kind of give my thoughts on that… And then a few people started messaging me, and whatever. But essentially what went down is Ruby Central, for the main parties involved here, is the non-profit organization that controls RubyGems.org… And they had as employees and contractors at various points various maintainers of Ruby Gems, the open source project, and those people were involved with RubyGems.org kind of on-call rotations, and whatever. So essentially, kind of, I guess, last month, we’re talking I think September the 18th or whatever - from then onwards, over the kind of following few weeks, Ruby Central notified Ruby Gems maintainers, including I guess Andre Arko… I guess if you want to read the two differences of accounts, I guess the starkest extremes here are Andrei’s written a few things on his blog, Ruby Gems have written a few things on their blog… And basically, from September the 18th onwards, Andrei and some other Ruby Gems maintainers were removed from the on-call rotation, they were removed from their GitHub access, and various bits and pieces went down.

There is kind of back and forward and arguments and disputes about what was communicated exactly by who, and when, and what happened, and what didn’t happen, and whatever… But essentially, we’re at the point today where almost no one who is involved with Ruby Gems the open source project has access to be involved with it today. Andrei and a bunch of the other Ruby Gems maintainers have created their own thing called gem.coop, which right now is essentially a modified version/fork, whatever you want to call it, of RubyGems.org. It’s run as separate infrastructure.

Andrei has personally been involved with kind of some competitors, Bundler, and Ruby Gems, and whatever, I think there’s a tool called Rv… And what seems to be now public knowledge is that both parties are writing various blog posts, [unintelligible 00:06:51.05] and it sounds there’s some kind of lawsuits in action between various parties as well. Does that provide the overview you’re looking for, Jerod, or do you want a bit more color on particular bits?

I think that’s a good overview. I think that brings us to what seems to be the biggest milestone, or moment, which was published just last week by Shan Cureton, executive director at Ruby Central, of this AWS root access event that happened in September. Justin, do you want to hop in here on that? …or do you want Mike to continue?

Yeah, so if you had been following along, the thing that everybody had been clamoring for, kind of regardless – like, people are taking sides. There’s a lot of – even though we’re not public figures, we’re not famous people, Ruby’s been a smallish pond for a long time. A lot of these people have been there for 20 plus years, and everybody kind of knows everybody if you go to the conferences, and we’ve all seen each other, and people talk, and there’s different clicks and there’s different groups. And so regardless of where your allegiances fall in terms of what friend group is sort of thinking this way or that way, and where things line up, in general, I’m talking about a universe of two hundred people max. And way more people in the world use Ruby, and also read the Internet. And so they’ve been operating under this complete lack of complete just information of “What’s the whole story? Something isn’t adding up here.”

Some people have been happy to fill the void with sort of conspiratorial thinking throughout, “This is all a takeover from Shopify, because they’re trying to get after this one guy.” And it’s “Okay, so why?” and then no one’s got an answer for that. And other people are just very honestly and earnestly being “Ruby Central’s saying that they just removed everyone for supply chain reasons. Why?” And Ruby Central is not talking, right? And all you get is a kind of hand-wavy, “Oh, well, because the lawyer is telling us we can’t”, or something that, if you ask people. This blog post, which - what was it? Was it the 30th? No, it was more recently. It was about the event on the third.

It was published October 9th, 2025.

October 9th. Thanks.

But the event might have been the 30th… No, on the 30th was the blog post that raised concerns.

That’s So part of why this is confusing is the post is a timeline, but the timeline is three timelines in reverse order, to first talk about the last thing that happened, and in what order things happened when the last thing happened… And then the next section explains the why behind that, and then the next last section is the why behind that… And if you were to – and this is why as soon as I read it, I put up my own blog post, I kind of tried to unspool it, to explain what are the stakes.

What this reads to me when you go through it – so I’ll first try to summarize. It’s characterized as a post-mortem of a security incident response, where on September 30th a person named Joel Draper - or Draper - posts a blog post that says “Yo, Andre Arko still has all these systems accesses. He’s the only owner of a particular GitHub organization, he’s still got these AWS accesses…” And I think that the if not expressly stated implication of the post is “This is how incompetent Ruby Central is, that here we are, weeks after supposedly having these accesses removed, he actually still has this access, and so look at how insecure this is. This is Ruby Central not having their act together.”

Because they had taken access away from Andre as well as other rubygems.org maintainers prior. However, Andre still had access, according to Joel Draper’s – Drapper? Draper?

It’s two p’s.

It’s two p’s, but Joel Draper just sounds more natural, so I don’t know…

Yeah. And my wife and I are rewatching Mad Men, coincidentally…

Don Draper.

Yes. And so it’s just really hard to separate.

I agree. I haven’t seen it for years, but I’m still saying Draper. Anywho… This post that he says on September 30th is that they’re so – I mean, the implication is that the incompetence of Ruby Central, who is the… Is it a foundation? Is it a nonprofit? Help us understand some of the entities here. Ruby Central - is it a for profit?

My understanding is it’s a 501(c)3, which frankly, I really wish I didn’t know about US nonprofit organizations, considering I don’t live there and I never will… But yet, I’ve been involved with open source nonprofit stuff long enough that I’m, sadly, intimately aware. So a 501(c)3, they’re somewhat hard to establish nowadays, because various government agencies have decided that open source software is a bit too easy to look businessy.

Basically, it’s an organization that exists to own the assets of Ruby Gems. It merged previously with Ruby Together, which was started by Andre. I don’t know who started Ruby Central personally, but basically, it exists as an entity to provide legal ownership of the service, to provide the ability to receive tax-free donations from individuals and companies, and then redistribute those to whatever nonprofit appropriate areas that they do… Which has been Ruby Gems maintenance, Ruby Gems on call, some conferences etc.

And there’s a lot of moving parts, which makes this hard to track. There’s the repos on GitHub, there’s the servers that are actually running said code, there’s the access to the servers, there’s databases, there’s other things that make this just really hard to track. So that’s Ruby Central, the entity. Back to where Justin was - they published this post about Joel Drapper’s post, saying that Andre still had access, or implying that there was still access…

Yeah, so before the post-post - don’t worry, they actually were notified of this permissions exposure.

So it wasn’t a zero day announcement.

No. Yeah, it looks like they had seven minutes, where Andre emailed them that he still had these accesses, and that this was the only disclosure.

Gotcha.

Joel Post goes up at 5:30 UTC. And now this is Justin, the guy – I’m just reading a blog post; forgive me if my characterizations are at all inaccurate. At that point, Ruby Central has to treat it like a security incident, so they go into emergency mode, try to lock down all these systems, initiate password reset, and then begin a relatively long investigation of all these other knock on systems, first over the next few hours, and them the next few days.

When they backtrack – so that’s September 30th. Then there’s an analysis of events that goes back and says “Hey, look, on September 18th Ruby Central notified Mr. Arko via email that he was going to have his access removed, or that it had been removed”, and while they removed his particular IAM, I assume, AWS account that presumably would be tied to his email address, they did not rotate the password on the AWS root account. So if you’re familiar with AWS, there’s typically an email address and a root password – or an email address that is effectively the root account. And it’s bad practice to use that thing and log in as it, because you don’t have any of the sort of policies and procedures available to you. But because Andre was one of, if not the core operator of RubyGems.org for so long, it appears that even though that he was removed from whatever their password vault system was - presumably a 1Password - he had a separate copy of that password or that login item somewhere… Because even though his individual AWS account was apparently according to Ruby Central disabled, it looks like roughly eight hours after that notice sent they state that an unauthorized actor from San Francisco logged into that AWS account, into the root AWS account, and then proceeded to change the password. And as far as they know, didn’t do anything else. I’m not an expert in cloud forensics, so I have no idea if that’s a thing that Ruby Central – it’s just the absence of evidence that’s leading them to say that, or whether they have any sort of just positive proof that nothing bad happened…

I think I read somewhere that there’s some sort of immutable time-based log, and they confirmed from the log what had happened, what hadn’t happened.

Yeah… I read both sides of that, and I think what Mike just stated seemed like it was more informed than the ulterior… Which is once you have root access, you can change everything. I don’t know AWS well enough to confirm or deny a side, but that did at the time of my reading seem to be the most reasonable stance, that they could confirm it. That being said, I also don’t know for sure.

Yeah, I pride myself in my ignorance when it comes to DevOps stuff, so I’ll take your word for it.

Same. Especially AWS stuff.

I think the thing that jumps out at me with a lot of this stuff is you can plausibly see why both sides thought they were doing the right thing a lot of times in this. From, I guess, Andre’s perspective, what’s been published is he says “Well, I didn’t have enough information to go on that this wasn’t was a legitimate event, and it wasn’t someone at Ruby Central’s email, or GitHub account, or whatever had been hacked… So I was trying to do what I could to preserve the integrity of the service.”

But I think what is hard for me with the communication of both sides, I think particularly unsurprisingly now that maybe some lawyers are involved, is that I would love to – maybe it’s just because I’m British… I would love a bit more from both sides, and saying “Hey, it turns out in hindsight I didn’t do the best thing here. And going forward, if you find yourself in the situation I was in, my advice to you is to do X instead of Y, and I did Y.” I think that’s the hard thing with all this, is that we’re now at a point where – there seems to be some degree of stability, and it doesn’t seem like Ruby Central is going to be inviting the folks who have left, including Andre, back into the fold anytime soon. It doesn’t seem that Andre & Co are going to take control over Ruby Gems GitHub.org or whatever, anytime soon. But I feel like the main parties who suffered through this are people in the Ruby community, of which I include myself, who just have a lot of uncertainty of what’s going on.

I can’t remember if I said this publicly or privately, but essentially, the person I feel the most for is anyone in the last month who is trying to pitch a new Ruby project at work to a management or leadership team who looks at Hacker News even once a week. Good luck, because a whole lot of fear, uncertainty and doubt has come into this. And also, I think both sides, I see – I’m a strong proponent of… You know, I wrote a post a few years ago, which some people hate, some people love, called “Open Source Maintainers Owe You Nothing”, which points out from a legal liability perspective, essentially every open source license says “If you don’t like the terms that I’m providing you here, you can go f*** yourself, and just take what you’re given, and like it.” And from that perspective, I am sympathetic, and I tend towards my sympathy being towards the maintainers. But at the same time, we have a critical part of the Ruby ecosystem which essentially had no governance process, no public governance process whatsoever, that had and still has to some degree, very little beyond the legal required levels of financial transparency, required of like a 501(c)3, and a lot of figures making decisions and making statements where most people I don’t know who this person is. I don’t know who this person is, how they got access, who’s right, who’s wrong… Are my Ruby gems safe, or unsafe, or whatever? And I think that’s the part about this, what I find really frustrating, is that a whole lot of people in the Ruby ecosystem are still being disrupted by this, and it doesn’t seem like it’s going to get solved anytime soon, maybe ever. Because right now, both parties are now just in damage control. And again, both sides on – I had someone that was on BlueSky, or Mastodon, or whatever, who was basically “Oh, blah, blah, blah, you seem to have switched sides on this.” And I’m like “Well, I don’t think –” I mean, rarely in life is there a situation where side A is a hundred percent right, side B is a hundred percent wrong… But this is definitely a situation where that’s not the case. Both sides have made mistakes. You may well be inclined more towards one side than the other, but both sides have to do things to repair trust, and fix things, and improve things moving forward.

Right.

I would just jump in real quick - sorry, Mike - because I wanted to point out two of the things from this timeline, and then we can move on to the bigger conversation. Open source maintainers who have an MIT license indeed owe you nothing. However, part of the complexity here is what’s in their dispute. And this is a natural – and I wrote about this in my first post on the topic… The fact that Ruby itself is 30 years old, mostly maintained by a committer group that’s mostly based in Japan… Ruby gems as a tool was created in America, by Americans initially, and then hosted in America, and has this separate lineage… And it’s a three-legged stool between custody of the code for Ruby Gems, and then later Bundler, and then later they merge custody and management of Ruby the language… And then rubygems.org, which is like a going concern and operational system that is running in the cloud. So from just an accesses perspective, we’re not talking about who’s got commit bit necessarily…

So when they say that he logged in with the root email eight hours after getting noticed that his personal access had been revoked, and then he changes the password - and that’s on September 18th; you know, you scroll down and it also says on the 28th he logged in again while he was in Tokyo, at Kaigi on Rails, another conference event. And then it’s only on September 30th, seven minutes before I blog post, that there’s any disclosure whatsoever.

So if he was concerned about their implication in this post, so if he was supposedly concerned about the security, had this exposure been leaked out to other maintainers, or if it was out in the wild or something, 12 days is an awful long *bleep* time to sit on that information and not disclose it.

Additionally, when you go back and ask “So why did we get to that point? What actually happened?”, you scroll down and the precipitating event to “Why are we getting serious about supply chain security?” was apparently - and this is an event that predates August 3rd… So presumably, when - and I haven’t met her - Shan, the new executive director at Ruby Central, doesn’t have a lot of technology experience, but does have nonprofit experience. I imagine, Mike, if it’s as dire as you’re saying in terms of lack of governance, lack of policies and procedures, my understanding is they didn’t even have terms of service and privacy policy up at the website until earlier this year… I suspect she probably came in and she’s like “We’ve got to get serious. We’ve got to run this thing better, we’ve got to introduce the standard operating procedure, make sure that we’re buttoned up from a regulatory perspective…” And then we’ve got to understand Ruby Central has been extremely budget-constrained since Rails Conf and Rails World turned into this schism - and also get the budget under control.

So I knew, and I’d talked to Marty about this I think maybe late last year, early this year, when he was taking on the role at Ruby Central, that they were trying to get more serious about controlling the finances, and getting the budget managed well. And so when you go through the timeline, apparently they cut the budget for secondary on-call rotation for rubygems.org, which had previously apparently been $50,000 annually, and that all of that had – that amount anyway had gone to Andre Arko’s consultancy to provide that service, even though it was rarely invoked. When he was informed that they were removing that budget, he sent an email to Marty that was kind of spit-balling an idea, it looks like, to say “Well, in lieu of getting paid in dollars, if I could be permitted to have access to the HTTP logs”, that could then be used presumably by a company to do some sort of analysis for some sort of marketable purpose… And you can debate whether there’s any sort of PII implication, or if that could be discerned from the logs… And Mike, my understanding is you probably, being the Homebrew guy, probably been approached by similar companies in the past… Regardless of the actual mechanics of that would look like, it was pretty clearly not something in the privacy policy currently, or the terms of service.

So that email is received on August 3rd, and then it looks like the board and leadership team… And I’m imagining now - and this is just speculation on my part - Shan as executive director, who’s still trying to get her bearings and trying to button this stuff up, probably sees that as… And this is the person who has all of the access to all these systems, and could just take it anyway, and is upset that we’re cutting the budget… That’s probably the precipitating event, and that’s how they characterize it in this email, of the thing that leads to “We’ve got to tighten up the security, and these accesses. And we don’t have an operator agreement signed for all these people who have the operational access, and we don’t have committer license agreements for all the people who are committing to this codebase, and it could cause disputes later. And so we’ve got to get those in place. So first, cut everyone’s access, get the agreements in place, and then we can start to rebuild on a firmer footing”, is my understanding of the timeline.

Now, that’s what I read reading this. Sure, it boils up to unauthorized access ultimately alleged against Andre, and changing the password and not disclosing it… But do I have anything there based on your – because Mike, you’ve been in a lot of the same discussions as I have, and with some of the principles… Is anything that I just characterized wrong, or is there anything you’d want to add to that?

Yeah, I don’t think there’s anything massively incorrect there. I guess for me it’s kind of an emphasis thing. Just to jump back, I guess maybe a bit of context that might be helpful for folks of where I fit into this party. So I’m a Homebrew maintainer, Homebrew being a macOS package manager, if you’ve not used it before. You can use it on Linux now as well. Fun fact. Yeah, I’ve worked on that for 16 years. I’ve essentially been probably the main person since the creator left, who’s kind of stepped into leadership stuff, and have kind of led us through various levels of financing, and fiscal hosting, and a lot of the kind of boring nonprofit-esque stuff… But notably, Homebrew does not have a dedicated nonprofit. We do not have any dedicated employees or anything like that… But we have an Open Collective, which essentially, if you’ve never used Open Collective before, is like an online banking app, but is in the spirit of open source public. So you can go and see “Two Homebrew maintainers went out for lunch about a month and a half ago in Singapore together” as our expenses; public docs say that they can do. And they had lunch, they talked about Homebrew, and they expensed that to the project, and I approved that expense.

So essentially, all the money coming in and out of the project, you can just go and look, without even logging in, and see what’s going on here. You can’t see people’s specific receipts with their credit card numbers for hopefully obvious reasons. But the thing I struggle with with all of this stuff is like - okay, Homebrew and Ruby Gems have been going for about the same amount of time. Ruby Gems has definitely received dramatically more money in that time, I would bet, than Homebrew. It would surprise me if it was as little as 10X more than Homebrew. But yet we, a group of volunteers scattered around the world, have been able to have transparent governance, transparent finances for five plus years. And I appreciate this is all part of a kind of tightening process and whatever, but the thing to me that all of these kind of blog posts and a lot of the discussion is missing, it’s like “Okay, who got what money, from who, and when?” That goes as far as Ruby Central employees, Ruby Central board members, Ruby Central contractors… There’s been a lot of conspiracy about how much Mike Parham has provided or removed funding, DHH has provided or removed funding, Shopify has provided or removed funding… And I don’t even want to speculate on one of those, because I don’t know what’s true or not true. But the thing I find slightly depressing about it all is it would be very easy to have all that information be at least semi-public, right? But it’s not, so it becomes an exercise… And I do think – again, with the what access Andre had to what, and when, and how, and whatever, I also think… Again, what I said earlier about the “Open source maintainers owe you nothing” thing - it’s like, at that point, is he an unpaid volunteer working on a service, and providing that to the best of his abilities?

I certainly think from the accounts we’re looking at, had Andre never received a cent from Ruby Central ever, I think he would read his narrative and be like “That is a hundred percent defensible, what he did, and Ruby Central are a hundred percent wrong, because they are a well-funded organization with full-time employees”, who – sorry, the bar is just much, much higher for them than it is for unpaid volunteers. But if volunteers are paid, how much – are they employees? Are they contractors? What’s their contract say or not say, or whatever? And I think that stuff is where it just all gets very murky. And that stuff is where personally, it makes me not happy that this is happening, but… I’ve been sort of saying, sometimes privately, sometimes a bit more diplomatically publicly for years, that “Hey, look, it seems like a lot of people have decided that open source sustainability is a problem we solve by just throwing more money at things, and this is the type of thing that happens when we do that.” It’s not to say that we shouldn’t, no one should have been getting paid, or we shouldn’t have had money or whatever, but once you start getting a lot of money involved in these things, things get very complicated. And you need to have significant levels of maturity and governance and transparency and experience in open source and experience in nonprofits to not f*** it up. And then maybe even if you have all those things, you still f*** it up.

But I think that’s where this stuff gets interesting for me, is I’m like, well, in some ways, if you were to look at Homebrew and look at Ruby Gems, you would say “Well, Homebrew is in this precarious, silly situation because they don’t have a dedicated nonprofit, they don’t have significant corporate backing”, whatever. But in a funny way, we are immune to a lot of the problems that have happened here because we have not gone in so hard on we now have significant dependencies on paying significant numbers of people their monthly wage.

And again, not to say we’re doing it better, they’re doing it worse, whatever, but I think there’s a lot of the open source - I sometimes call it big ‘open source’ - that is trying to push a lot of people in this direction, that we all need to just get maintainers to be paid full-time employees, and contract everything out, and whatever… And I think what gets lost is like “Well, what happens if we do that? What are the pros and cons, and to what extent that events like this happen, or at least get a lot more messy and complicated than it could have been otherwise if there was not the same degree of money involved?”

Break: [00:31:55.06]

So if I were to just jump to the end of this - and I’m going to jump right back where you are, Mike… But if I were to just jump in, as a guy who just types gem install every once in a while, or bundle… I know you two, I’ve interviewed DHH, I’ve interviewed Shopify people; I’ve never met Andre myself… I’m tangentially related to the Ruby community as a regular, old Ruby user. And to speak to Mike’s point from earlier, at the end of this is a rubygems.org AWS root access event. If that’s what I hear, and I find out it was days, or hours, or however long it was, and was it Andre, was it somebody else, were they malicious, were they not? Can we verify it? Like, that’s a five alarm fire, isn’t it? I mean, I install my gems, just like anybody else does - unless you’ve switched over to gem.coop - from rubygems.org… And somebody had root access to their AWS account for some amount of time, and that’s probably all that I’m hearing; maybe a little bit more about other things. And so the end result is a disaster. I mean, this has been disastrous. And now we’re in, like you said, Mike, my damage control… And so that’s just incredibly unfortunate, and a fact of history now that I think comes from, whether he said/she said, they did this/they did that, who’s to blame etc. it comes down to like money has just muddied and created no end of trouble… And it’s just really, really murky now, how you navigate money and open source. I mean, just combining those two things together… Which has been a desire and something I’ve preached for many years, is like “We need more money in open source. We need to fund these people, we need to sustain these people. We need to help these people, because they’re giving things away for free, and then other people are using them, and taking advantage, and applying pressure” etc, etc. But we’ve gotten some money now… I mean, I’m not talking about now this year, but over the last 10, 15 years - money’s come in, in certain amounts; you know, it’s not evenly distributed by any means… And it seems like – I don’t know if I can say it’s caused more trouble than it’s solved problems… Maybe it’s been better than – maybe it’s a net positive… But man, it’s sure made things even more complicated. And I’m not sure how we navigate this. I’m not sure how we navigate this going forward. Maybe the answer is complete transparency, and maybe the answer is… I don’t know, I can’t even imagine an answer that makes sense. But Mike, your stance is you can’t do it. You can’t do full-time open source maintainer as free to work on the project however they like… Like, whatever we all imagine would be the perfect life of an open source maintainer - there’s no such thing. Is that your stance?

Pretty much. I mean, Justin’s Hotfix podcast thing was like “Okay, come with a pithy quote that’s like a controversial statement.” And I think the shortest, pithiest version I got was “Open source is not a career.” And what I mean from that is that I think there’s plenty of ways to be an open source maintainer, and make a load of money. I’ve done fairly well for myself financially, I have had some doors open for me that would not have been opened otherwise were it not for my open source work, for sure. I mean, arguably, maybe bar my first job out of university, college, whatever, I think every other job my open source maintenance has influenced me getting that job… But also, every other job has – never has my paid main work being working on open source. And particularly working on Homebrew.

When I was at GitHub, I was at GitHub for 10 years, and I had a couple of months in which I was given permission to help migrate us urgently from bin tray to GitHub packages. And I worked on a bunch of internal code for that, I worked on a bunch of external code for that in Homebrew… But bar that, that was not my job. I did Homebrew stuff in bits of spare time and evenings and weekends, and time between meetings, whatever it may be, and that was not what I was paid for, or promoted for, or whatever. I built stuff internally. I guess not unrelatedly, I was one of the first four engineers to build GitHub Sponsors. So that was an interesting thing for me because it was being on the front line of like “Well, what happens if we put a bunch more money into open source?’

And I think GitHub Sponsors, again, it was telling, because if you looked – and a lot of this is all public knowledge. If you look at the people who have made the most money out of GitHub Sponsors, they are not the best open source maintainers, and they would not be offended in me stating that they are not the best open source maintainers. They are the people who have done the best job selling themselves, and selling something which other people value, through GitHub Sponsors as like a payment provider, essentially. And good on them. That’s great that they do that. But there’s a lot of people, myself included, who just slap up a GitHub Sponsors… Like, I do a lot of Homebrew stuff, and I had done for 16 years… My monthly GitHub Sponsors payment is $22 a month. That’s not me going and saying “I want to get a bunch more.” I get a bit more than that, I get $300 a month from Homebrew as our kind of maintainer stipend… But you know, the reason why I don’t get a load of money that way is because I have not dedicated time and energy into building essentially a sales process for my sponsorship pipeline. And that’s how it works. If you want to be an open source maintainer that gets funded primarily through sponsorships and money and whatever, it doesn’t look like a tip jar. It looks like you are now running essentially - you know, in the same way that other influencers might have an influencer economy or whatever… And there’s various routes of making that money, and paying those bills, and getting that open source work, but there’s not a single, easily-treaded path that is not without compromises… And it makes me very cross to see a bunch of particularly younger maintainers being told “No, this is the way. If you follow this, you will both be rich and you can work on whatever open source you ever want, whenever you want.” That’s just a lie. That’s just *bleep*.

Yeah. Well, hold on, Justin. Let me just say this… For those who are putting them through the pain of watching us on video, the fringe benefit of doing the video version of this is for the last 10 minutes you can watch Justin sit there and chomp at the bit for his opportunity. Because we’ve said so many things that I’m sure he wants to address, so much so that at one point he was literally holding his mouth shut, because he has so many things to say right now. This guy talks for three hours uninterrupted, and he hasn’t had a chance to say anything for the last 10 minutes. So Justin, just turning the floor over to you, my friend… Which of these many points would you like to address first?

Gotta admit to anyone watching the video, I did get distracted at one point because my pen ran out of ink, and I had to write down –

He literally left – at one point he literally left and then came back. [laughs]

So that was distraction number two… Distraction number one was I saw – somebody texted me that the Vision Pro with M5 was announced.

Oh… Well, there’s a fire alarm fire right there. [laughs]

Just to tell you my priorities… As one as one of the five remaining daily driver Vision Pro users who – I use the Mac virtual display every day, like we’ve discussed several times on the show even… So I will not be buying it, because it’s just – basically, I’m using it as a monitor… And I’d much rather just talk about that now, but… To try to honor what Mike just said, when we say “Open source is not a career”, it’s kind of – running a consultancy, so helping co-found, help start, whatever, a consulting company… Speaking at conferences, doing some amount of open source, none of – I was very fortunate in life in that none of my open source was super-successful. It seems like a huge pain in the ass when that happens, for the most part, in terms of the burden of issues, and pull requests, and yadda-yadda. But I was visible enough that people would come up to me, younger folks, less experienced people, people trying to succeed in this industry, and they would – whether it was about speaking, or whether it was about blogging, or whether it was about doing open source, a lot of the activities that people saw me do… They’d look at that and they’d confuse the means with the ends. So they’d say “Hey, how do I break into open source?” And so I’m like “That’s a real weird goal”, because I do open source because I’m trying to get something done that makes me money, like working for a client… And the first real project I did was a Java client, and we had a whole lot of JavaScript and no way to run tests against it in CI. And I was like “That won’t do.” And so I started unit testing our JavaScript with Jasmine, locally. Well, how do you get that running in CI? I had to figure out a – this was back in 2010, or whatever… Run a sudo HTML and JavaScript fake environment in a Java runtime, and then make a Maven plugin that we could incorporate into the build. And I did that on my own time, on a weekend, or late at night or something. And then I just plugged it into the client code the next day… And like “Here you go, a gift, free of charge, because it makes my life easier at work, and we’re going to ship your code better.” Almost every one of my open source projects was that. It was “I have a job, it pays me money, and that job would be massively improved in terms of the outcomes or my life at that job by writing some open source… So I’m going to solve my problem, and I’m going to make it open because that’s easier than getting approval to make it closed internally.” If I have some asinine idea for how I can make things a little bit better, forgiveness is way better than permission. Maybe it won’t work. Why would I fight for budget and time to go on some escapade to like “Hey, I can vaguely improve things in a way that you don’t understand”, when I could just go on my own GitHub, publish something publicly, and then go and consume that at work? And if it turns out that it’s useful, then work is happy to let me spend some hours building it, because then I’ve got a virtuous – I’ve got at least one person depending on it, and that’s the person paying me. So then I can keep working on it a little bit and make it a little bit better, respond to issues and feedback… And that’s typically how I’ve done open source. That’s one way to do it, and in a very, very small sense, that at least has incentives aligned, where it’s like “I make this thing better because it’s built for a purpose, for somebody who needs it.” And that’s not breaking into open source. That’s not saying “Okay, so what I want to do is I want to work on this open source project…” And I’ve seen this play out lots of times, whether it’s through the sponsorship deal, or Ruby Together and later Ruby Central did this - they actually paid hourly for people to work on Bundler and Ruby Gems codebases… And when you’re talking about a high $150 an hour or whatever it is hourly rate to work on an open source tool, then a lot of questions arise. Like, what do you work on? Who chooses the priority of that work? If you’re talking about 150 bucks an hour, or zero dollars an hour, with most of the people contributing making zero dollars an hour, suddenly there’s a perverse incentive there to be like “Alright, well –” What if there’s nothing else to do? What if Bundler is a solved problem, and basically does everything it needs? Now you’ve got a perverse incentive to create makework, to justify more hours, to get more money.

And if the goal at the end is “I want to make a replacement-level tech career doing open source activities”, I think the places that leads you in terms of the amount of complexity and machinations in terms of how that funding gets to you, and the amount of singing and dancing, and glad handing and arm twisting that you need to do to extract those dollars from people who might not necessarily see a direct value or benefit or ROI for giving you that money - it leads to places like this that are as God damn confusing as the situation that we’re in now.

That’s I guess maybe a twisted knife version of the picture that Mike’s painting - this is just all about, at the end of the day - incentives were not aligned because people wanted to get paid. And this is where this leads you. And yes, it’s really sad that a lot of people do open source for free, but a lot of people have hobbies for free, and if they choose to keep doing them, it’s kind of their task. I don’t know what to tell you.

And that’s the thing - for me, genuinely, open source for free as a hobby is a beautiful thing. And we should think very carefully before we decide to kill that, by either paying everyone, which won’t happen, or by making people who aren’t being paid feel like they are being exploited. Again, it really *bleep* me off no end when you have people often with minimal involvement in open source themselves going around telling open source maintainers “Big companies are exploiting your labor.” And I said “Well, if only there was a way to get a big company to pay me to write software for them. Oh wait, we have lots of that.” But the reason why lots of people, including me, enjoy open source software and have continued to work on it for a very long time is it’s like, I don’t have an engineering manager, or product manager, or technical project manager being like “Please sign your TPS reports. When will this be done? Is this a yellow T-shirt or a green T-shirt size project?” Being able to just opt out from all of that bureaucracy is – you know, that bureaucracy is important and necessary sometimes, but it’s also nice to not have to do that. And when you don’t have to do that, you can operate in a different way and work in a different way, and that’s fine.

And again, we go back to the “Open source maintainers owe you nothing”, where it’s like if someone doesn’t it – I mean, there’s legitimately issues on Homebrew that people file, and it’s like “This is a legit issue, but you’re being a ****head about it, so I’m just going to close your issue.” And when they’re like “Oh, [unintelligible 00:48:17.12] then I’m like “Well, **** you. Tough luck. I can do what I want.” But if I’m a paid employee, and if that person is a paid customer, in some companies you can do that, sure… But that’s not terribly advisable. And I think this goes back to what we were saying, about perverse incentives, and who gets paid and who doesn’t get paid. It’s like, well, open source maintainers owe you nothing, sure. People who are being paid to do something owe someone something.

Right.

And again, in Homebrew’s case, we specifically do… We have all open tooling, and open documentation, and blah, blah, blah, and we pay maintainers that hit a certain threshold $300 a month. We call it a stipend. There are people for whom, I’m sure, the average American child delivering newspapers, if that’s still a thing that happens, is making significantly better hourly wage than some of these open source maintainers are… But it’s fine, because that’s just a token amount for appreciation for them. And even then, we’ve not done anything publicly or whatever, because we try and in Homebrew do a reasonable job at keeping some of our drama behind closed doors… But even in that case, for a very small amount of money, we had a person with a very well-paid day job, who was doing the wrong thing, essentially because they wanted to hit a threshold to get an extra three hundred bucks a month, right? And that made us reconsider how we do this stuff.

But to go way back to something that Justin said right at the beginning, about he did open source to solve his own problems. Back when I got involved first with open source, in the heady days of the Linux desktop back in 2007, when I did my first Google Summer of Code, and met a lot of these KDE people, and I was a Linux guy… The thing that people used to say all the time then, that I never hear anyone saying now, is “Open source is about scratching your own itch.”

And what people meant by that back then was open source is primarily about solving your own problems, and then releasing it in a way that other people can benefit from you doing that. And that’s why I did it, and it’s why I do it. Because when I work on Homebrew, and I make Homebrew a little bit better, it’s usually because something in Homebrew annoys me, and I use Homebrew, therefore I make it better. And that’s completely the opposite from the attitude Justin was saying, of “How do I break it into open source?” Whenever anyone’s asked me that, I’m like “Well, find a tool that you use, find something annoying with that tool, and then go fix it.” And they’re like “Oh, but I don’t know JavaScript yet” or “I don’t know this” or “I haven’t contributed to that codebase” or “I can’t find a mentor” or whatever… And again, as we’re being a bit more spicy on this podcast, it’s like “Well, congratulations. You lose. You’re never going to be a successful open source maintainer.” I’ve never seen someone who’s been a good open source maintainer who has been taught into being so, or encouraged, or given enough money that they kind of finally get over a line, or whatever. That helps people contribute, and it’s great, and we shouldn’t discount that as being a thing that we do for some people sometimes… But most of the time, most of those people, it’s intrinsic, internal motivation that gets them to do this stuff. And that doesn’t come from “I want money. I want clout. I want my resume or CV to look better.” It has to come from “I just want to do it because it’s interesting and fun for me.”

I think this goes beyond open source software, because the pattern is very generic. It’s like “I love doing X, so I do it for fun. I would love to do X more. If I could only make money doing X, I could do it more.” And then X becomes not fun anymore, if that’s successful. Basically, that’s the pattern. So it’s not just software, but creation of all kinds. Anytimes you can parlay your hobby or your passion into a job, your passion is now your job. And your job’s a job, and jobs aren’t fun. Even if it’s the exact same –

You can add to that, Jerod, because then you can take all your friends you have good relationships with - and I tried this one, it was a lot of fun - and then you hire your friends, and then your friends aren’t your friends either.

That’s right. [laughs]

And you live in a basement in Ohio and you’re like “Oh my God, what have I done with my life?”

I’ve destroyed my hobby and my friends. But I got money in exchange. We’re just trading it all in for money. Yeah, that’s the unsolvable problem I think right there. And for years and years we’ve been looking for more paths to funding, more ways, because there’s so many different kinds of open source… And there’s ones that have an easy time making money, because they are right there staring you in the face; they’re like end user applications. And then there’s the dependency of the dependency of the dependency, who only has a GitHub account and is not on any social networks, and is never going to make a dollar on their GitHub Sponsors, because nobody even knows that they’re transitively installing… Xz, for instance; just naming one that was exploited.

And so maybe – I just thought there was different models, and we need to just explore all these different models. And I remember, going back years, to Nadia Eghbal’s Lemonade Stand repo, which she published back when she was doing her open source funding writing… And she documented, “Here’s all the different ways: bounty programs, open core”, blah, blah, blah, blah. And it’s like “Pick one that helps you.”

And we’ve explored on this show, over the course of a decade now, different people doing these different models, and with more or less success. And at the end of the day, the happiest people that we’ve ever talked to about their work are the ones that are like “Yeah, I do this for fun. I’m not going to monetize.” And it’s like, that ends up being the healthiest relationship to your open source code, is “I do this for fun. I’m not trying to monetize it.” Because every one of these models - we can go through them all and they all produce at some point the perverse incentives that have happened in the Ruby community.

To me it’s not even monetization versus not. It’s like direct monetization versus indirect monetization.

Sure.

And I can’t remember what it was, I’m sure it’s some much wisened, old philosopher or whatever who said this - and I’m going to paraphrase it and butcher it horribly… But basically, the idea of like the best things in life are achieved indirectly. So let’s not be overly crude. I’ll try and be relatively diplomatic, but basically, for example if your goal is romantic companionship, there’s ways of getting elements of that very quickly in exchange for money. But most people would say that actually, that’s probably not the most fulfilling long-term option. And the most fulfilling long-term option - you know, I’ve been with my now wife for 23 years. We have a perfect marriage and relationship. I’m very, very happy. That takes a lot of time and effort. And again, I have almost a pending blog post brewing about just being patient. It feels like it pays a lot of dividends on this stuff. I was at GitHub for 10 years, I’ve been with my wife for 23, my best friend, who comes around to my house, we’re currently watching Alien Earth. It’s very good. We’ve been friends for –

Right now you’re watching it?

Well, I mean, not right now, because I’m on a podcast…

[laughs] Okay. Well, you said you have your best friend coming over and you’re currently watching Alien Earth. I’m like “Dang, this guy can multitask.”

I’ll do my best to multitask. But yeah, I mean, we’ve been friends for 30 years. And Homebrew for 16 years. Most of the good things in my life are things that I have done for more than 10 years. And there has been ups and downs in those times. And there’s been times where if you were to look at some tiny microcosm of like the first months or weeks or whatever, or a particular segment, you’d be like “Oh, well, this isn’t worth it. I’m going to bail. I’m going to quit. I’m going to move on.” But again, all of these things are things that I’ve got, and have made me very, very happy, and I’m delighted with, not because I wanted to make as much money as possible, or have as much whatever as possible - human relationship, interaction, whatever - but just because I enjoy the process going along, and I still enjoy the process going along… And that tends to get you to a very good place. If every day you enjoy your life, and you keep doing that and keep enjoying your life, and at the same time stuff like money, you’re like “Okay, well, I need to pay the bills. I need to make sure that I make enough money to provide for me and my family” and whatever, but you’re also really excited about what you’re doing, then that tends to work pretty well. And if you go and chase maximum financial return for anything, on the short term basis, like - okay, you might make a bunch of money, often you don’t, but you’re probably going to be miserable in the longer term. And you’re going to be like “Why am I doing this?” and “I need to quit” and whatever.

I mean, almost that’s the definition of sustainability. When we’re not talking about open source sustainability, when we talk about anything being sustainable, the idea is “How do we get people to be able to do this for a long period of time, and without being a cheeseball?” Open source sustainability comes from within. The reason why I have been able to do it for a long time and not burn out - I mean, literally, in that 16-year window, I don’t think I’ve gone a month without working on Homebrew. I’ve maybe not even gone three weeks without working on Homebrew. And the reason why is because I enjoyed it then, and I enjoy it now. And I know what I’m willing to do and what I’m not willing to do. And for me, it has to stay enjoyable, or I’m going to quit. And I built a group of maintainers where I’m very protective of them, because I know that’s the same for them. And to me, just almost looking at yourself in the mirror and being like “Am I enjoying this? Is this good?” And if it is - great, do more of that. And if it’s not, then just stop. And that terrifies a lot of people, because it’s like “Oh, well, open source will collapse, and we have all these maintainers who walk away.” And I say, “Well, actually, no, because someone will step up and do what needs to be done if what you’re doing is really important.” And that’s maybe the hardest part of it all. It’s like, maybe that open source project you’ve spent a huge amount of time and energy into, maybe it’s not that important. Maybe it is replaceable, and maybe your role is replaceable. And that sucks, to look in the mirror and think that… But maybe that’s true, and maybe you need to do that.

Look… If you’re watching this video, you’re going to be probably noticing that we’re three white men… The collective noun for white middle aged men is podcast. I get it. But it’s true. And a piece of context about that 2015 to 2021 era, especially in the U.S. and politically, is like, this conversation about just being tutted by older guys who are already successful in their careers - “Oh, just do it as a hobby. Just do open source in your free time”, or something. When that comes into contact with this motivation, that open source is an ends, not a means, and it’s a way to get – because highly visible people are doing it, people rightly assume, for whatever reason you’re doing it, “If I were to do that at the same stage or at the same level of impact or in the same high profile project, then I too would be highly visible, and therefore I could parlay that into more marketability as an employee. I’d be hired in at the staff level, or at the principal level, or I’d have better employment prospects.”

Even though I started doing open source to scratch my own itch, to Mike’s point, and that’s still the reason I’m ever motivated to do it, it created a virtuous cycle, not just for my employer to be able to benefit from my hobby work, but for me to be able to parlay those projects into new relationships and credibility, because your GitHub profile kind of redounds to at least some kind of proof that this guy can write working software, and this is what it solves.

And mixed in with all of that is like “Well, what if you don’t have the free time, or if you’ve got family responsibilities, or there’s some other systemic reason why you aren’t able to just work for free and make this time possible?” It’s creating an avenue for more privileged people who do have that luxury of time to pursue that hobby. And then even though it’s just a hobby, it’s not like – I definitely made more money out of my programming habit than out of my Japanese language acquisition habit in terms of a hobby. So that’s not lost on me here, it’s just kind of a fact of life… And I don’t know how to solve it, because when you try to solve this through the lens of… Like, when the conclusion - and I heard this a lot in the late 2010, was “And that’s why we have to pay people to write open source.” I was like “The number of circles and loops necessary to kind of connect these dots together in a way that’s actually going to get the outcome that you want is so convoluted that this is probably just going to make things worse.”

I can’t say we’re in a much better place in all of these experiments where people are – and again, to Mike’s point, where people are being directly compensated for their open source efforts, especially in an ad hoc or a semi-directed pseudo employment manner, through contracting and through kind of an amalgamation of funds and sponsors and donors. I don’t know what to do – I just want to put it out there so we don’t get emails about privilege, I’ll be honest. That’s why I said those things.

[laughs]

I think that’s a good point. I think it’s well-made, and I’m glad you made that. I think for me, my take is like “Should we try and improve the diversity of open source etc.?” Yeah, we should. I do. Hopefully, all of us would agree with that, to some degree. But I also think not everyone having the free time… Again, it’s one of those things where if – sometimes I think the people pushing a certain narrative haven’t done the five why’s on – the reason why they’re often doing that is because they themselves come from a position of like “You need to do open source to have a really great career.” And I’m like “Well, no.” I’ve worked with plenty of star plus engineers at GitHub who are phenomenal engineers, who’ve literally never done a single open source commit ever. And I would not tell those people that they should or shouldn’t. Considering the number of open source related emails I’ve had that have fascinating things to say about the size or presence of my *bleep* and whatever may relate to that, and various interactions with my mother etc. I don’t encourage anyone who doesn’t have a current interest in open source to sign up to receive those types of emails. And I don’t think we’re going to solve that problem anytime soon… But I think that’s the thing, it’s like, do we see open source as an essential on-ramp that we have to use to get people to be successful in the software industry? And if you start from that prior, then it’s like, yeah, of course you’re going to start thinking we have to improve diversity and pay people to encourage people in, because otherwise you’re just going to really **** up the software industry by not solving that. But I don’t think that is a problem. And ironically, I think that is a problem that is exacerbated by people saying “We have to pay, and we need to put more money in”, rather than it being solved by that. Because no one has to write open source. I think that’s the fundamental thing.

If I can say anything to any person listening to this, it’s literally, if you have a job where you write open source even, even if you’re a paid employee, or contractor, or whatever, I’m sure you can find something else where you don’t have to do that. Everyone writing open source in some degree has some element of choice. Maybe not short term - maybe this month’s mortgage payment relies on you writing open source and whatever - but long-term in the career.

And also something - Justin, you pointed this out on your podcast a lot, about how there was this golden age of… You know, the three of us, again, are probably similar sort of age. There was this golden age of programming, the early 2000s, where if you came out and you were interested in open source, chances are that was going to help you into a reasonable tech job, and you could probably make a decent amount of money. And then now we’re seeing things are completely horrific for a lot of juniors trying to get into the industry… And I think that’s the thing - I don’t like that. I don’t think anyone on this call likes that. But we can’t just magically go backwards and undo that and fix that. And I think often the people I hear advocating this type of stuff around open source just have wishful thinking of like “No, if we can go back, we’ll do it right this time, and we’ll reinvent it”, and whatever. Well, that’s not how it’s been.

And I loved your point, Jerod, about how this is just universal for hobbies. We could probably have had exactly the same conversation we had right now about music, or whatever. And money, and whatever. I remember when I was at GitHub, and there was – I can’t remember the name of the band. It was some top 10 indie rock band, and one GitHub conference GitHub had paid for this band to come and play. And the band came out on stage –

Was that when Cold War Kids played at the summit?

Yeah, that’s the name of the band. But having been there in the room - and I think there was a Silicon Valley episode that was loosely based on this… They came out and started playing –

Hold on, I’ve got to interrupt you. Like if you don’t know this, most Silicon Valley episodes were based on interviews with Chris Wanstrath and people…

I’m sure. I’m sure. I rewatched it all recently with my wife, and it’s aged well. But anyway, so they came on stage, they started playing, and 75% of the people immediately left the room, because they were like “I don’t want to be here.” And I’m going to confess something to the two of you here; hopefully no one else hears this… But I used to be an aspiring musician. Anyone on video can see my mostly now unplayed guitars behind me. And I immediately thought to myself “What a bunch of *bleep* sellouts.” I would hope to myself that I would never be at a level where I would take any amount of money to go play to a room of nerds - myself included - who don’t want you to be here, and would just immediately leave the room as soon as you start playing.

Right.

And again, we can have the same conversation - money and music. Was that bad, that they had that money, or took that money, or whatever? I’m not a musician. I’m sure lots of musicians are very angry at me based on what I’ve said, and I’m being a massive hypocrite… But yeah, I’m sure there’s another podcast and an identical conversation about this, and I think we’re not special in software /open source, and this is not the first industry to have this problem, nor will it be the last.

No, I think that’s a great point. And I’ve said this before, that by and large, people don’t make music in order to make money. They make money so that they can make music. And I think that there are people who’ve done both, and they’re called rich and famous rock stars, or whatever. And of course, people idolize those, because wouldn’t it be the dream, to be able to be rich and famous, and make music? That’s a lot of people’s dream, which is why it’s really hard to do. And we draw that across to open source… And we have had some people who’ve made livings, who’ve made really good livings publishing open source code, and creating a following, and becoming whatever you have to be in order to get that done… You can go to the top of GitHub Sponsors and see a few of those people there. It’s an entirely different skillset, just like being a rich and famous rock star requires more than being able to play the guitar. Because lots of people can play the guitar, but there’s more to it than just that. One of those major factors, in fact, is timing and luck, and has nothing to do with who you are, your skillset. Another major factor is what you look like, unfortunately, but that’s the case.

So open source as career might follow the exact same trajectory as music as career. It’s like, yeah, a few lucky, very privileged people find a way to make that work, and they live a great life. And the rest of us - it’s like, you’ve got to decide if you want to do it or not. At the end of the day, open source is a gift to the world. It’s you giving back. A lot of the motivations for doing it is people saying “Well, I got so much for free that I felt I should just give back. And so that’s what I do.” And so that’s a gift. And a gift comes from a place of privilege. You have excess, and so you give it. And one of the beautiful things about the digital economy is that we can give it to everybody, whereas if I was going to go out and buy a new vision pro M5, I could just give it to Justin. I couldn’t give it to the entire world. It’s an amazing thing to be able to do your work once and gift it to the entire world… And sometimes it just has to stay that.

You’ve got my address, righ? For that –

I figured you already have it on order, so I’ll probably just…

No, no, no, not yet. You need the phone. You’ve got to scan your face.

I’d buy you the new one so you could send me your original. Or maybe sign it.

You got it. Deal.

I got Justin Searl’s original Vision Pro…

There’s a thing here, right? …because we’re kind of conflating programming and making a living programming and open source as a hobby activity, or whatever. And just like music, there’s a difference between – being able to play music is a skill. Being able to program a computer is a skill. And being a songwriter who puts love and craft and passion and creativity and something of themselves into the music that they create is a passion, a craft, it’s an individual pursuit… And yes, if you do that, you are lucky to get paid. And if it does all work out and the stars align, then that’s *bleep* miracle and good for you. But at the same time, getting paid as a musician with that skill to go play gigs - like, there’s a guy who sings at the lobby bar at a hotel next to my house every Tuesday… And he’s mostly playing Wonder Wall, and Ol’ Country Road…

The hits, baby.

Yeah. And he’s not there for his health, but there’s a transaction happening, and I don’t begrudge him at all. There’s ambience and there’s music. And so – hell, the Cold War Kids made me laugh, Mike, because across the street there’s a new hotel that opened. I live in Orlando, and it’s just all resorts and stuff, and pools, and whatnot. A new hotel opened earlier this year, or late last year, and they had the Goo Goo Dolls play. And I was like “Hell yeah I’m going to go over for a free Goo Goo Dolls concert, and free food and drinks…”

Why not?

Yeah. And it was great. I don’t begrudge them at all, because that was them applying their skill for money, in that sense. I mean, the songs were written 20 years ago, and everyone’s really old, and they look suspiciously good, and that made me wonder about how they’re maintaining that… But when a programmer is applying that skill to make money, at the end of the day somebody, whoever’s paying that money, is going to be looking for some kind of value out of it.

Sure.

And whether you’re working for somebody and they’re – why do we have planning sessions? Why do we have product owners? Why do we have requirements handed to us? It’s because the things that we’re typically being asked to build as programmers are at the behest of somebody else who wants to see that software built, or continue to operate, or to be refined or whatever, in order for them to extract some kind of value out of it. And just like with the singer/songwriter who can eventually get to the point where they get to open new hotels and get paid to sing their own songs, every now and then you get really, really lucky. I worked on Ruby and Rails for years and years and years, and now there’s a Ruby and rails team at a company like Shopify, and they hired me, and I get to continue doing the stuff…

Aaron Patterson’s a good friend of mine - and yes, that’s part of my bias and my allegiances, is I talked to that crowd of people more than the maintainer side of people who are involved in this particular dispute… And I fully own that. But I’ll talk to him about his job, and it’s confusing sometimes. I’m like [unintelligible 01:12:30.22] here and there and stuff, but at the end of the day he makes good money doing that for Shopify. Shopify gets a lot of benefit out of that. But if he were to quit, he would basically be doing the same thing every day, for free. So that is the stars completely aligning. And you know what? I know half a dozen people for whom that is true, out of how many thousands of people that I’ve interacted with.

So it’s just an unrealistic thing - and now this is back to Mike and our original kind of Hotfix premise… It’s an unrealistic thing to just plan on your career being that. It’s way too high of a bar, it’s way too skinny of a bottleneck to hope that you’re going to, with any level of confidence, squeeze through. Because so few people do, and not for a lack of trying.

Well, it’s like all the youth right now, they all want to be YouTubers or TikTokkers. It’s the new version of “I want to be a rock star.” It’s like, do you know how many rock stars there are? There’s like seven of them.

Yeah. But the influence your economy, I think that’s similar.

It’s been democratized to a certain degree, yeah.

Yeah. And we’re, I guess, again, three white men in our forties… Probably if we had some Gen Zed guest on – yes, it’s Gen Zed, because I’m British. Then I’m sure they would have a very different perspective here. But again, I sort of wonder whether some of this comes from the concept of like the side hustle, where the side hustle is often the “I have a hobby, and on top of my job, I am going to monetize my hobby, and hopefully I can get to a point”, like you said earlier, Jerod, “where my hobby can become my job, because I fully monetize it and I can pay the bills”, or whatever. And I think, again, music is an obvious example, where there are people for whom music is their career, and I know some of them for whom they then go home and they are not interested in playing or producing music in their spare time at all anymore… Like, “I will just do it for money.” There’s people for whom they are not interested in ever making any money from their career, ever, which if we wanted to have a horrific open source metaphor, some – I was going to say man or woman; it’s probably like a dude with long hair, playing their guitar around a fire while their friends, some of them might want to listen, probably most of them don’t… Someone could go up to them and say “You’re being exploited for your labor. You should be paid the same as Taylor Swift for this.” And it’s like, well, actually, no one wants to pay that person, because they’re not very good, and they don’t want that job. That person is doing it entirely as a hobby. And then there’s the people where there’s a blend between the two. Maybe it’s their hobby and it’s their job.

And I think that’s the thing… It’s like career, hobby, both. And all of those are acceptable options, but it feels like this stuff often gets conflated. And even someone like Aaron Patterson, right? Good example. He’s written an absolute boatload of open source code over the years. It would be interesting if you went back and somehow did accounting of like “Okay, well, let’s take the amount of money you’ve been paid to write open source, and then the amount of hours you have spent doing open source and hobby-related things, say any work on a public repo on GitHub in your entire history as a programmer, and let’s figure out what your hourly rate is.” And my guess would be – it’s obviously getting better each year that he is employed by Shopify… But my guess would be it’s actually a lot worse than you think it is, because he - again, like musicians; musicians are not getting paid to play their scales. People are just sitting – when I used to actually be a half decent musician, a lot of it was just sitting and spending two hours playing the same riff again and again and again, slightly faster, slightly faster, slightly faster, slightly faster… And no one’s going to pay you to do that really boring stuff. And I don’t think open source is dissimilar.

What I worry about is, again, I’m from an era of which it wasn’t clear that open source was going to win. When I was at university, there was still all the kind of Linux, and - I can’t remember the name of the company, that basically there was this big lawsuit, and Steve Ballmer, “Linux is a cancer”, all this type of stuff. And it was like a battle between proprietary software and open source software. And for some reason, the only reason we’re even having this conversation is because open source software so clearly and unambiguously won. And maybe I’m being paranoid here, but I worry for a world in which we say “Okay, any big company that uses any open source software in any capacity is extractive, and unless you are paying all those maintainers a Bay Area living wage, then you’re an evil company.” What happens if we actually go all-in on that viewpoint? Do we make those companies be like “Well, you know what? I’m actually not going to touch open source anymore. I’m just going to pay a team internally to build this stuff.” And for a lot of people like me, that would be a very sad thing. That would be the end, in some ways, of a lot of the non-hobbyists’ open source.

Right.

And I just think, where does the money come from to pay every single open source maintainer with 10 stars on GitHub a living Bay Area salary? Particularly if, as Justin said, we say to them “Hey, we just give you that money unconditionally. You don’t need to have a product manager, or whatever. You just build whatever you think is best, however you think is best.” I mean, it seems ridiculous to me. Maybe that’s just me, but… I think that’s the logical conclusion we get, of the peak “We should pay everyone, because otherwise it’s unethical” argument here.

Right. To that point, we just did a show recently with Feross Aboukhadijeh about Npm security in light of just the onslaught of Npm hacks, which have been going on and continue. I think after he published that there’s even some newer ones… And one of the things – and he owns and operates a Socket Security company, and so he’s very much in the info sec world, and talking with CTOs and CEOs of larger corporations… And they’re saying things like “Well, this is not because of open source sustainability, but it’s because of open source security, and this “You can’t trust a network thing.” They’re starting to have those conversations, especially because the - first time to say it on the episode; I’m glad we’re over an hour in… Because of the new enthusiasm around AI code gen tools, they’re saying things like “Do we need Npm?” This is not “Should we contribute?”, this is “Should we even use Ruby Gems?” Why don’t we just write everything in house?” And that is starting to percolate amongst leadership and Silicon Valley companies, and you add to that an open source tax, so to speak, whatever it would be, Mike, when you talk about “Every maintainer has to get this much money, therefore every user has to pay in according to their dependencies”, or however you’d actually work out the impossible logistics of all that… It would just be one more reason why people will start to opt out of the economy altogether and say “Yeah…” Because I look at a world - I don’t know if we’re going to get there, guys - where my code gen tools can reproduce for me; instead of vendoring a gem, why don’t you just reproduce a gem? And now, I don’t think we’re going to be there myself anytime real soon, especially with gems such as Rails, for instance, which is not a gem, it’s a meta gem of many gems… But the smaller ones. It’s like, why would I even care about open source? I can just generate everything.

Jerod, it’s funny you mentioned this, because it’s not just your fever dream, but it’s increasingly a thing that I am also hearing.

You want this world.

Oh, I’m living it.

[laughs] Okay.

But my brother, Jeremy, he works for cars.com. He’s a an Elixir programmer. And he gave a talk at ElixirConf last month, which - I don’t know if it was the title, but basically, the thrust is zero dependency software. Instead of pulling in these dependencies – and it’s not from a security perspective, although that’s… When you’re talking about an open source tax, most people aren’t thinking about how this stuff gets funded. They’re thinking in terms of the “Yes, maybe the security concerns. I don’t trust the software necessarily”, especially if you’re in a regulated industry, and you have lawyers go and check licenses, and verify all that stuff… There’s that tax, for sure. But more so, I now – I am running code that I don’t control or understand, and it’s going to have its own update schedule, and I have to keep all of these dependencies up to date. And of course, the Npm community is famous for really, really tiny modules, and that’s lots and lots of things that you have to keep up to date, and so that upgrade burden is really high. And right now, you take Claude Code, you take Codex CLI, you point it at a readme and you say “Alright, go clone this repo, and I just want this section from the readme. I just need this kind of couple of features right here. Just go clean room implement that for me, if you don’t mind.” And it’ll do it. And to your point, you can just vendor that into your project…

And so I’m working on this Posse Party Rails app, and since going with agents - I’m trying to think now… I started working with agents in April, started with the GitHub Copilot stuff, moved on to Cursor, moved on to Claude Code< and now I’m into harder drugs…

[laughs]

…I think that I may have added zero new gems. Zero new dependencies. Definitely zero JavaScript.

And you’re doing like OAuth stuff, you’re talking to APIs…

It’s almost all integration work, yes.

For me, as a starting point, it’d be all gems. I’d be like “I get the BlueSky gem, I get the Instagram gem, OAuth gem…” And then I just tie those things together. Because it’s just posting, across different social networks. I don’t want to say “just” to belittle it, but my point is I would expect that to be mostly third-party code.

And 10 years ago that’s how I would have done it, too. But it’s not just because of AI enabling this. I learned the hard way that if you’re writing code that’s basically glue code of eight other different things, it’s kind of like you’re on one train and trying to jump onto another moving train in real time; you try to keep everything in line, and you’re just holding the stuff together…

And so no, I’ve written – in fact, this week I’m rewriting my LinkedIn integration, and it’s not just a whole bunch of HTTP requests to post stuff. You’ve got to wait for it to download, so you need another whole series of self-enqueuing tasks to go check to see if the download’s done. And then my wife Becky is like “Well, if it doesn’t support stories, I’m not going to use it.” So I’m adding a story support, which I think is supported by, as far as I know, no – definitely no Ruby gems, but I don’t know if any dependencies are doing this much… And so the nice thing is that when you own the code, you can build on top of it… And when it’s inside the codebase, it’s part of the context of whatever your agent’s doing… There’s a lot to be said for as the cost of code - and this is how we tie it back to this conversation. As the cost of code basically craters to zero asymptotically here… The writing of the code is not the part that is worth money anymore. So if I’m a maintainer and I want to get paid to write open source code, the market dynamics for that are also flatlining right now.

So what we could be experiencing - this conflict being a flashpoint… And where did, where did it all start, according to Ruby Central? It started with them cutting budgets. And why were they cutting budgets? Because sponsorship’s declining. Why is sponsorship declining? Apart from the macro stuff and apart from the conference stuff that Ruby Central also runs, I think a part of it is a general sense that we’ve already hit peak open source… In the sense that people’s unpaid labor is the way that we’re going to get all of this stuff built. Not in the sense that things are going to be necessarily open, or built on open protocols and platforms; if anything, that’s probably going in the opposite direction. But the value of an individual programmer going and hacking out a particular issue or a PR is going to go down if now you can just @codex something, or @claude something. And Mike, you’ve been working with this too, using Copilot agent and finding it – tell me how you’ve found it.

Yeah, it’s funny, because there’s a lot of Copilot-related stuff… I was one of the first people to test an internal beta of that. It’s alpha, technically… And the reason why lots of people inside of GitHub would me to test things is because I seem to be allergic to telling people that things were good when I thought they were a heap of ****, even when it was strategically and politically advisable for me to say “This person’s pet project is wonderful.” And so yeah, my feedback on Copilot pretty early on was like “Wow, this is surprisingly not a piece of ****.”

You have to remember, when Copilot came out before ChatGPT, and stuff - or at least it was being alpha’d before that…

So this sounded ridiculous, that we would use AI, whatever. And I was like “Wow, this is a better auto completion for Ruby than I’ve ever seen, using any IDE, or indexing, or whatever.” And it immediately made me more productive.

So fast forward to the kind of Copilot coding agent stuff - again, I saw the kind of means on Hacker News of like Microsoft employees being encouraged or forced to use this publicly, and it just being an absolute disaster… And I was like “Well, I’m going to try this out.” And again, maybe because Homebrew has borderline fascistic issue templates, demands everything of you, and essentially force you into Mike McQuaid’s opinionated way of how you file a bug… But if you follow that template properly, it’s actually really great context for an LLM, particularly when it has all the comments of us discussing it.

And I basically just I think a couple of months ago assigned every bug in the Homebrew package manager to Copilot agents… And then it took a couple of weeks. Some of them got there 99% their first time, some of them got 50% of the way and I finished off a lot of the stuff… But essentially, in two weeks those were all fixed. And you can go and look at the public record and see how well that went or didn’t go, or whatever. But anyone who says this stuff is not productivity boost in any circumstance is massively kidding themselves. And the awkward reality of it - again, back to the conversation - is I would say “Is Copilot agent great? Does it avoid handholding?” No, definitely not. Is it better than the standard drive-by Homebrew contributor? Jury’s out. It’s certainly comparable, and I definitely think at this point the first iteration of PR I would struggle to tell the difference between Copilot and a first time contributor. And a Homebrew maintainer, and indeed myself, who’s worked on Homebrew longer than anyone else, will do a much better job… But even then, I find myself leaning on it, because sometimes it’s like there’s 10 ways of fixing this. “Copilot, go *bleep* take one.” And whatever mess you make, I will clean that up and get it over the line… But it’s not an empty page problem.

And yeah, I think this – I’m probably not quite as… Not pro AI, but I don’t maybe buy stuff quite as heavily as Justin says, about the cost going to zero. But I definitely think it’s impacting stuff pretty heavily, and I think we’re in some ways seeing a reversion in the tech industry of back to what skills are valuable, who’s valuable, whatever. And back perhaps to being like “Maybe tech is just a slightly less fun place to work”, right?

Going way back for me, back in 2009, I got a job on the third application or whatever, to a company called KDAB, who I knew about them because they employed more than anyone else of KDE maintainers in the KDE ecosystem. And I was a big Linux on the desktop guy, and I had been contributing to KDE. I was like “Woo-hoo, I’m going to go work for this company, and I’ll get paid to write open source.” I discovered pretty quickly, on my first 100 percent open source project, that it’s like “Oh, actually, getting paid to work on open source as a consultancy looks like very tedious, boring work.” And I can probably say specifically now, considering it’s been that long ago… So K-Office, which was like KDE’s open office alternative, essentially a corporate sponsor, was like “We want K-Office to be good. Here’s a spreadsheet with 5000 regressions compared to Microsoft Word in K-Office. For these particular important documents, go through and just fix these line by line, one by one. Incredibly tedious work. No volunteer wanted to spend their spare time doing it, so a company paid people to do it. Right.

And again, this kind of comes back to a lot of the stuff in the early conversation. All the fun, enjoyable parts of open source - I don’t think you’re ever going to struggle to find people to do those. At least not until you tell them all they’re being horribly exploited by capitalism by doing so. But there’s going to always be a bunch of really boring, *bleep* tedious work in open source. Maybe it’s supply chain security stuff, maybe it’s whatever… That’s the stuff we need to get the money towards and fund. The stuff that people don’t want to do. And I think that’s going to be there… And again, it’s one of those things where the jury’s out with all the supply chain security stuff, whether the costs of that balloon beyond what people are willing to pay. If we have a team internally, or we have LLMs that mean that we don’t have to pay this tax, then maybe that’s what we do now.

I’ve seen some people on the internet be critical of – way back to Ruby Central, that organization we were talking about at the beginning of this conversation, it kind of incited this back and forth… That Shan, the new executive director is not technical and not from technical communities, but rather is more there for her nonprofit experience. You know, I remember when Ruby Central was just Ben and Evan, and then they added Marty in 2012, or whatever. And it was just the three of them as the chair people, I guess. And all rubyists, all programmers, and that felt right. And back then, in that era that we kind of keep hearkening back to, the writing of the code, the building of the tool, the brilliant API and just the blood, sweat and tears to get it over the finish line at a high enough level of quality, whether it’s a CLI or whatever it is, a library, and then to publish it - that was where all the action was; that was the work that was the thing that required brilliance, and that’s what the market was rewarding so handsomely in terms of great tech salaries.

If it’s the case that it is now no longer an incredibly rare thing to have capacity for writing replacement level, decent code, that means everything else now is more valuable. And so to Mike’s point, if I’m operating a service and Jerod was scared *bleep*, he said - he said *bleep*, he bleeped it out… Nah, I’m kidding.

I must’ve bleeped the whole sentence out, because I don’t remember saying that.

I wanted more bleeps for a minute. I want to beat Mike. I was lying earlier, I do want to beat Mike.

Oh, gosh… He’s got you down.

He does. But if I’m running a service and I’m going to be scared bleepless to have a root-level security event occur at the place that I’m downloading all of my gems, which could then be root kidding my computer for, I don’t know what it was, 12 days… Like, I’m actually really glad that the executive director has the hat on now to finally shore up the governance, make sure that their regulatory and their finances and their tax situation is all buttoned up… And probably, I think at the end of the day we’ll probably end up being much more transparent than they’ve been in the past, which was just the result of negligence, I’m sure, and not malevolence, up to this point… Those are actually the skills that are more valuable because you can’t just easily replace them. It requires good judgment, and diligence, and experience, and oversight.

So part of this is a story of just an organization maturing, but I think part of it is also a reflection of the change in what’s valuable and important right now. And as you look at open source, your conversation with Feross, which I thought was excellent - you know, Feross is a really smart guy - this is where the new locus of control goes, when the writing of the code is no longer the most important thing.

I’m out of things to say, so I’ll just say a cliché: the times they are changing, are they not. That was a musical cliché on Mike’s behalf? Liked that one, Mike? Bob Dylan, right?

Yeah. I love it. I don’t know that Bob Dylan song. Sorry.

[laughs] You don’t know Bob Dylan? Come on.

Unless it’s like Northern European power metal, I’m pretty lacking right now…

Yeah, I don’t know if they’ve got Dylan in Scotland. They still have newspaper boys, and children – do you have milkmen still up there, Mike?

We until recently had our milk delivered to our house by –

Jesus Christ… That’s so idyllic.

We’re living the dream over here.

Well, you knew so much about American culture I figured Bob Dylan would be right in your wheelhouse… But it only extends so far, I guess.

Yeah, I only do the parts of American culture - I guess to echo this conversation - that I’m paid in order to…

You’re forced to, yeah. [laughs]

…forced to learn in order to maintain and sustain employment. And that hasn’t yet gone as far as Bob Dylan.

Well, I’ll send you a $10 donation and a Bob Dylan album. You can just get to work on that.

The moment when the American should make some sort of sports metaphor and I nod along as if I understand what baseballs fourth strike, third innings catch MLB championship Fridays means…

Well, this is a nerdier podcast, but the joke I was going to make, I was going to say “No wonder you like Alien Earth so much, because it takes place in a world where America doesn’t exist anymore.” You can finally relax… It’s just five corporations, each get a continent… Just like how it should be, right?

Alright, Justin, this is as much your podcast as it is mine. If it were mine, I would say thanks and end it. But do you have more things you want to talk about?

Well, since this is going to go show up on my feet as a Hotfix, we try to end every single interview with some sort of pithy one-liner that represents “what is the fix to the problem” statement. And it’s the guest, who I guess in this case is Mike, being double hosted by us…

[unintelligible 01:35:15.02]

Hey, I’m just trying to –

Can I bleep that?

I’m trying to beat the bleep filter. You can’t – you’ve got to keep it in. That stays in. Mike, it’s your job now to help us land the plane and find a title for this thing, at least on my feed. So if the problem statement is “Open source is not a career”, what’s the fix? What do you tell people instead? How should they – what’s the takeaway for them in terms of where should their attention be? You tell me.

Well, I guess in some ways I would divide the statement in two. So from the open source side of things, do open source if you want to do open source. If you don’t want to do it anymore, don’t do it anymore. This is when this podcast gets published and 99% of open source main things quit, and we have a world crisis, and I’m like “Oh, well…” But genuinely – and I very strongly encourage people in Homebrew to be like “Hey, the day you’re not using Homebrew anymore, thanks for all your work, stop using it, move on, get on with your life.” There’s been people kicked out of Homebrew who are doing a good job, but clearly, Homebrew has been very bad for their mental health… So we kicked them out of Homebrew.

So ultimately, this stuff needs to be fun, and it needs to be something that you enjoy. And if you care about sustainability and burnout in open source, you want that stuff to be better, if you’re an open source maintainer, go get some therapy. You probably have some *bleep* you need to deal with, chances are.

And then the career side of things is like, again, same deal. If you work in tech, chances are you have a career, which is nice. It’s harder right now than it used to be. I’m lucky enough to have a lot of friends with careers outside of tech People who are personal trainers, who are opening up the gym at like five o’clock in the morning, working a 60-hour week with a split shift… That’s *bleep* hard. Those people do not get paid a lot of money, they work really hard, and they make a really big difference in people’s lives. If you’re in a situation where you’re getting paid good money to sit on your ass in your home office, nine to five, Monday to Friday - again, maybe it’s about figuring out a way to be happy with that for your life. And if you don’t like that, then quit, and go do something else. But same deal - find a career that has some happy medium point between paying your bills, getting what you need out of life, and that you can actually enjoy and not hate… And do that.

Just, in both camps, do what makes you happy, because at the end of the day that’s probably what is going to give you a better life and also give those around you a better life… And even give other people that work with you at your work, with your open source or whatever. If you’re miserable doing what you’re doing, you’re probably not very nice to work with, so… Let’s just all be happy, kids.

That’s going to make some people cross, I think. Alright, so I started with “Don’t do open source if you don’t want to”, which is a little bit long for a podcast title. Then I said “Do open source for you.” And then I just flipped to your problem statement and I said “Open source is a hobby.”

Yeah, that works.

Works for you? Alright.

Or just “Unhappy maintainers should quit.”

If you don’t like it, quit.

Yeah, pretty much.

Alright. Well, on that note, I’ll pretend I’m going to hit my button that plays my music now, and then I’ll have my callout and I’ll say “Hey, Jerod, thanks a lot for–” We’re playing at Jerod’s house right now, because we’re in his Riverside account, instead of Mike’s Riverside account, or my SquadCast account. Just, really, we’re all broken people here… But we’re glad that you’re listening. Hello. Hi.

But it makes us happy, right?

Especially listening – if you’re still listening at this point, especially.

Nobody’s paying me. I’ve got no sponsors. That’s why I get to skip the bleeps.

That’s right. Our sponsors pay us extra to bleep things out, you know? So I’m going to make lots of money off this episode.

Yes. I hope it was sufficiently brand-safe for you.

That’s a new business model… Pay per bleep.

Outro: [01:39:28.06]

There’s some words I decided in advance I wasn’t going to say, so you should be glad for those… [laughs]

I appreciate it. That’s the sign of a good friend, Mike. I appreciate that. Alright. Well, we’ll just hit the button on both sides. I’m not going to end it, I’m going to let Justin’s be the end, and I’m going to hit the music when he said he’s going to hit his music. So the show’s over at this point. Justin got the outro.

Well, alright.

Okay.

Okay, I’m going to keep this stuff in.

So am I then…

Alright…

If you’re going to ruin your show, I’m going to ruin mine.

Yeah, that’s mutually assured content…