Potpourri: Product, Platform, Paravirtualization
So, happy Labor Day. How are you are you, staying cool over there? I know it's
Speaker 2:Well, I mean, relative to some, I mean, man, it is oppressive.
Speaker 1:It's hot. It's a lot hotter it's a lot hotter inland, but it is Yeah. Definitely warm. So okay. I well, not to be too I mean, well, I guess we're already on brand.
Speaker 1:We're working on Twitter Spaces. But to continue being on brand, I realized that I have been mispronouncing this word my entire life, I think. So first of all, how do you pronounce this word? Potpourri. Status.
Speaker 1:Potpourri potpourri. Potpourri.
Speaker 2:Potpourri. Potpourri.
Speaker 1:I know
Speaker 3:where I'm from
Speaker 1:Paree, but if you say it quickly, it just sounds like I'm pronouncing it like everybody else.
Speaker 4:If you say Paree saying it really quickly.
Speaker 1:Potpourri, though. I say potpourri. It sounds like I'm saying it quick. It sounds like I'm not pronouncing the t, but I know in my heart I am pronouncing the t. That's
Speaker 2:You know, Dan is just unmuting himself just so he can shut this down. You know that right?
Speaker 1:Yes. Well, I I listen. Dan is gonna write my geography. That's pretty.
Speaker 2:Everyone's gotta roll.
Speaker 1:Everyone's gotta roll. Alright. So I the a word that was introduced to me by Alex Trebek, I feel.
Speaker 2:Yes. Potpourri. Yes. For sure.
Speaker 1:Right? But
Speaker 2:how but how what did you get, like, the spoken did you get, like, the written version of trip of Jeopardy at home? Like, how do you how did you not hear it pronounced?
Speaker 1:I was here. I was hearing it pronounced. I just thought I was hearing a t. Like, this one, I've I've got, like, less of an excuse. I'm not not saying potpourri.
Speaker 1:I've never said that. Yeah. But I I felt it was pronounced pooper. Like
Speaker 2:like like poda like podacram kind of?
Speaker 1:Like poyal. Like
Speaker 2:in that direction?
Speaker 1:Yeah. Like like a POTUS a POTUS kind of
Speaker 5:a thing.
Speaker 1:I see. Okay. Or f POTUS as we've been reading a lot of.
Speaker 2:There we go.
Speaker 1:POTUS. So I've been mispronouncing this word, I realize. But I I I think we've been, we've been a bit overdue to just kind of have a structuralist random I wish you know what the history of potpourri on jeopardy is. I mean, that's a goofy name for a category. You would think, like, miscellaneous would have been more accessible.
Speaker 1:I mean, there's gotta be a story there, don't
Speaker 2:you think? Exactly. Miscellaneous, I'm sure, was more accessible. Like, why would they do that? I my my I mean, I can I know we gotta get down to whatever the business is, but I would say that my favorite moment in Jeopardy, and I watched a lot of it, I used to pause it on TiVo so we could, you know, make sure we got the answers or whatever?
Speaker 2:But was that when contestants all you know, after either after or before Jeopardy, you know what showed? After or before the game show.
Speaker 1:Wheel of fortune.
Speaker 2:Wheel of fortune, precisely. Okay. Easy question. All the contestants were stumped. Time was ticking down.
Speaker 2:Finally, someone buzzed and got it right. And Trebek goes, woo. I thought we're gonna have to kick you all down to Wheel of Grace.
Speaker 1:That's great.
Speaker 2:And it was the most wonderful trivia show burn that I've ever seen.
Speaker 1:That is great. And, like, in an era before social media. So, like, unless that's gonna be, like, an evening news story, like, that that that that's kind of, like, a private moment. Like, it was live TV more or less. You had to kinda be there to observe it.
Speaker 6:Right.
Speaker 1:That's great. And and Pat's name
Speaker 2:It's all like Vanna is gonna tweet at Trebek or something. Do you think that that starts a beat.
Speaker 1:Did you know that Richard Scary at Doctor Seuss hated one of his guts? I did
Speaker 2:it, and I love it.
Speaker 1:And the in in particular, I think Richard Scarry didn't really care about he just didn't care one way or the other about doctor Seuss. But Theodore Geisel, like, held Richard Scary just viewed Richard Scary as, I I mean, just beneath the world. The fact that anyone did, like, very quotidian, like, who And I'm, like, personally, I'm I I I love Richard Scarry. So I'm I guess I'm on Richard Scarry side of this. But I
Speaker 2:Oh, really? I I was gonna say, dark Ted Geisel, you got me because, man, fucking busy town. Like, and What's wrong with Busy town? Times have I gone what's wrong with Busy town is Busy town is interminable.
Speaker 1:Everyone's gotta everyone's gotta work at Busytown. Like, it's everyone has a critical role to play in Busytown.
Speaker 2:Yeah. Well, Ian, you got, like, some 2 year old
Speaker 3:with a No one's drawn funny eyebrows or or any kind of racial stereotyping in busy town.
Speaker 1:Like, you're like, oh, no.
Speaker 2:A 2 year old with you got a 2 year old with photographic memory. So if you try to skip a page to get to the end, he's like, daddy, hold on. Time out. Skip the page. Like, oh, did I?
Speaker 2:Jeez. I guess. But, like,
Speaker 1:I mean, you're finding Gold Bug? Like, it's it's so much fun to find Gold Bug in Busy Town.
Speaker 2:Yeah. No. I guess the memory of build Busy Town is better than the actual business.
Speaker 1:I just think this is where I realized that, like, in even though I tried to deny it, in my heart of hearts, I I just am a capitalist. I I love, like, it just, like, the the beating heart of busy town. I I mean, it it's very name is so Calvinist. It's got the Protestant work at it's like, I love it. It's like
Speaker 3:It's right there. It's right next to Taylorville.
Speaker 1:Exactly. Right next to Taylorville where we all we're doing time measurement studies in busy town, making sure that that also, you gotta love the fact that the butchers are all pigs in busy town.
Speaker 2:It it is macabre. Yeah.
Speaker 1:It's genius. I think. But so you know, what I'm wondering is, like, did out did Alex Trebek and Pat Sajak have some, like, fierce hidden rivalry?
Speaker 2:I choose to believe that.
Speaker 1:Or is it like a Ruth Bader Ginsburg Clarence Thomas thing? They're, like, going golfing or whatever those people were doing. Secretly Dutch. Yeah.
Speaker 2:It's just like I
Speaker 7:Wasn't that wasn't that Alito?
Speaker 1:They they're all,
Speaker 2:like I think it was Alito. I think it was Alito, but who knows? Maybe Thomas liked to go golfing with RPG as well.
Speaker 1:I would read a book on this is my point. Which brings me to my but the the next question has been plaguing me all day. The book losing the signal. Did we talk about did I get what how do I have this book in my possession? Did we talk about this here?
Speaker 2:I don't remember talking about that.
Speaker 1:So this book's extraordinary. It's on the rise and fall of Rym. It is so, oh, so good.
Speaker 2:Maybe maybe we did talk. We definitely did talk about RIM, you know, on and off over the over the decades.
Speaker 1:We certainly have talked about RIM. But so and I I mean, I even went I actually wish that Amazon time stamped your order history because I so I know I ordered it on, like, a 3rd I ordered it before we did our books the box reduct, so it's not there. I don't know where I got this from, but it's really good. It's so, losing the signal, and they're turning into a movie, apparently.
Speaker 8:Wow. Yeah.
Speaker 1:I Okay. I A biopic? A yes. A bio
Speaker 2:You did. Yes. You did. In in disparagement of, of another particular company, which may have been more successful.
Speaker 1:Yes. Did we talk about I can't remember if we talked about that here or not. No. No. Definitely
Speaker 2:not. That's that's right, Patrick.
Speaker 1:That's right, Patrick. So is it so the the the context is and then we'll get to we'll we'll get back on topic, whatever the topic is. The on potpourri. The, so this is in 2011, and all of a sudden, Rim is very interested in acquiring Joyant, supposedly. And this is at a time when the Joyant CEO was, at all times, very interested in selling Joyant.
Speaker 1:So whenever there was this kind of interest, it was immediately, like, scramble. And one of the I mean, Adam, how many times in your life have you, like, needed to be on the next flight to international destination?
Speaker 2:Literally never. Okay. Yeah.
Speaker 1:Never happened to me. It's it's happened to me very rarely.
Speaker 2:And Yeah. You worked for Crazy.
Speaker 1:I definitely worked for Crazy. Oh my god. That is yes. I did. I definitely did.
Speaker 1:So we were we need to be on the next flight to Waterloo. It's like 6 PM, and, like, we need to be in Waterloo tomorrow morning. It's like, okay. Oh, okay.
Speaker 2:It's like, well, then we should
Speaker 1:have left yesterday. I don't know. Exactly. That's it. So we buy last minute red eye tickets to go out to Waterloo, and we have this this these series of meetings in Waterloo and, like, why are we this doesn't make any sense.
Speaker 1:Why are we here? Why would you why would RIM acquire Joyant? And it's very clear over the course of talking them over the course of the day that Rim is a total tire fire. This thing is an absolute mess. It's 2011.
Speaker 1:It is, like, June of 2011. The company is burning. And unfortunately, they don't know why they're interested in buying Joyant. And I realized that I am errantly suggesting good reasons that they could buy Joyant. And they're like,
Speaker 2:oh, yeah.
Speaker 1:I know. That that seems really interesting. Like, yeah. We're like I like that. I'm like, wait.
Speaker 1:What am I doing? I do not wanna be acquired by this company. This is like the the the this is the the Titanic in golfing for me.
Speaker 2:It's like the the it's like they were talking about that Titanic being insufficiently buoyed and someone heard it.
Speaker 1:Totally. Totally. And so it and as it is going on, it they they also keep talking about the toy company. Again, there's, like, a queer I don't know if it's, like, it's, like, a Canadian thing, but it's a nemesis.
Speaker 2:Toy
Speaker 1:The toy company.
Speaker 2:The toy company?
Speaker 1:I thought it was the fruit. No. It's the toy company. And it took me and it took me hours to figure out what the fuck they were talking about. Because I didn't wanna be, like, who is is there, like, something like has Toys R Us declared some sort of unconditional war on Waterloo?
Speaker 1:I mean, like, what's I don't I'm missing some very important piece of context. And then at some point, maybe, like, the second or third meeting this came out. So we're meeting with kind of all these underlings through the course of the day. And maybe sometime, like, just before lunch, someone mentions the Cupertino toy company. And I'm like, sorry.
Speaker 1:Is the toy company Apple? I mean, I was just,
Speaker 2:like, gobsmacked. It's like It's like, do you mean the most valuable company in the world probably at the time? If not, like, top 5.
Speaker 1:And when this is, like, the company that's devouring you right now. Like, the company that has
Speaker 3:People people really love toys, man.
Speaker 4:I mean
Speaker 1:Well, and it just it it it should kind of their their whole, I mean, perspective. I mean, it was just always super, super dangerous whenever you have this kind of, this broad it it within a culture when you're kinda denigrating, not just a competitor, but someone that's actually, like, beating you in the marketplace. And, then we finally had this meeting with them at the end of the day, and it was so weird in so many ways. One, the things that we had heard through the day had been kinda conditioning us for this meeting, which now make a lot more sense reading the book. And then we kinda the the meeting was kinda performative, And, what was crazy so the, this is Lasaridis Mike Lasaridis, and, Jim Balsalay are the co CEOs, like, that ever works.
Speaker 2:All all ready recipes. Exactly.
Speaker 1:Red flags everywhere. And when Lazaridis was talking, Bosley was rolling his eyes. It's like, hey. I think you're a teenager. Oh, absolutely.
Speaker 1:I can see you guys. Oh, get a load of this guy. Yeah. Sure. Alright.
Speaker 1:I'll be home by 11 PM. Yeah.
Speaker 2:Jerk.
Speaker 1:Wow. Jeez.
Speaker 7:You broke just the death of him.
Speaker 1:Oh, well, no. And it it really I felt like after that whole, like, insane episode and which, you know, supposedly, it it I mean, I have I have the avarice of joining CEO at the time to thank because they appear so supposedly we're interested in buying the company, but not at a dollar figure that he was interested in selling at, which is god blesses. Again, god bless the greed. So did not get a chance to, to get taken on board with the curse there. But it was you know, I Dan, I think it was exactly right.
Speaker 1:I was like, man, this is how lucky was this to have been invited briefly into the wheelhouse of the Titanic? It was It was wild.
Speaker 2:A real moment in time.
Speaker 1:Moment in time, but the book is, is extraordinary. Losing the cycle. But we we have not talked about it here. So anyway.
Speaker 2:On on topic of books and that is actually something I wanna bounce off you. I've been reading for for far too long because I'm a slow reader. The, masters of doom
Speaker 1:Yeah. Yeah.
Speaker 2:About id software, John Carmack, John Romero and terrific. I really enjoy the story. Book is a little inconsistently written. There's some, chivalents in there. Like, for example, when when someone says, I could care less, it, it really undermines, I think, the craft of the book, for me as a fussy person.
Speaker 2:But, but some, but there was this moment that it was describing, and I wanted to see if this resonated with you, but, you know, it was talking about, some of the first network gameplay, you know, chat IRC, and, you know, some of this stuff of this era, sort of like the the mid nineties, and describing these clans in in Doom and and Doom 2 and Quake and Quake 2 and so forth, that had only communicated online, hadn't even spoken on the phone or whatever. It brought me back to this moment where I realized that there was this moment as like a kind of young teenager or so where these online conversations were really this moment of privacy that I had not had before. Where, you know, the the phone was not a private thing. Right? The phone was constantly in peril of, like, someone else picking up because it was not my phone.
Speaker 2:It was like a phone shared with a a house full of 4 people and in some cases more. But but it was it was sort of I was thinking back to this specific moment as, like, the Internet and BBS is not kind of stuff unlocking privacy that I had not experienced before. The does this this might have been a very narrow age band for which this resonated. Because it may be that by the time you were on BBS's, you're already, like, in college and had, you know, all the privacy you needed.
Speaker 1:I was on BBS's as a teenager. I feel like I mean, a true, I was so off the leash at that point. I don't recall really yearning for privacy. I just didn't have, like, my parents were just
Speaker 2:fair enough. Like, off I could
Speaker 1:pretty much, like I don't know. You know? Be be home before dark. I I I just, but that is interesting about, and I mean, certainly, I I think it explains the enduring popularity of Snapchat. And the reason that Facebook, I think, has actually failed to ignite, with teenagers is is because of privacy.
Speaker 1:I mean, I do think that the teenagers today for sure want privacy, and they're much more mindful too of the fact that the things that they share. Like, they've got you you got a kind of privacy. You you also have things that can be screenshotted that can come back to haunt you. So you do definitely need the the mics are all hot for for teenagers. Yeah.
Speaker 1:It's interesting. And Max of Doom is, I I've not met it, but I've heard great things.
Speaker 2:Yeah. I'm I'm into it. Like, I I mean, again, a little uneven, but the story is great. Just like, you know, John Carmack as this legendary programmer slash abysmal coworker. It it it's it's pretty awesome.
Speaker 2:Just the kinds of stuff that he was getting, these early PCs to do. It it it got me to, you know, one of the first things they did was or one of the early things they did at id was port or make, like, a, you know, pirated port effectively of of Super Mario Brothers 3 for the PC, pitched it to Nintendo. And and I ended up, like, pulling up a video that Carmack released or Romero released, like, 10 years ago. And it's pretty wild to just see what they were doing on a PC at a time when people did not think those things were really possible.
Speaker 1:Not only did they think that they were not possible, and I don't know if they get this into this book, but graphics researchers were very upset because it's not possible. So so graphics researchers were like, you can't this is cheating. Like, what you're doing is not 3 d. And I think they were like, who cares? It's fine.
Speaker 1:I can close the door. Exactly. Alright.
Speaker 2:Yeah. Matt, you were you were getting in?
Speaker 9:Yes. So I've I've read Masters of Doom or rather listened to the audiobook narrated by Wil Wheaton. And, I I think that, while while the the story is good, it's it's pretty light on on the technical detail or technical rigor. In particular, the the the description of the, of the the of Carmack's first major breakthrough where he got side scrolling working smoothly on a PC was just really hand wavy. And I guess maybe that maybe that comes with writing for a general audience.
Speaker 9:But, if you want a, a more condensed history of of that period that, paradoxically, don't know if any of you guys have ever read the blog, The Digital Antiquarian by Jimmy Maurer. It's an ongoing history of computer games. And, so so, like, he started with I think he started with Oregon Trail in the early seventies, and he's he's been going, he's up to 1996 now. He's been doing this for, like, 10 years. And, he wrote a series of articles called the Shareware Scene, which started by talking about shareware in general and then focused on id Software with the, the their the their meeting at soft disk and then and then Commander Keen and Wolfenstein 3 d and concluding with doom.
Speaker 9:And his description of side of of how Carmack implemented side scrolling is more, more succinct, but also more, technically meaty than what's in the book.
Speaker 2:So Yeah. No. Fair enough. I mean, it it did get definitely get me wondering what the actual technical innovation was, because, like, I was writing graphics code not that long after that. So could could sort of see through to to some interesting technical innovation there.
Speaker 2:The other thing that book called up called to mind that I totally forgotten about was the first three d accelerators, like the 3 d effects card.
Speaker 7:Oh, yeah.
Speaker 2:We're really pining after that, in the early days.
Speaker 1:And have you listened to the oral history with the 3 d effects creators?
Speaker 2:Oh, no. I gotta check the is this computer history museum?
Speaker 1:Computer history museum. Nice. I gotta check that out. Man, that stuff is so good. The computer history museum content is so I wish they would make it available not only via YouTube, but it there is so much good stuff out there.
Speaker 3:Yeah.
Speaker 8:Yeah. Yeah.
Speaker 1:I Simeon. Yeah. Simeon. Yeah. On the on
Speaker 5:the topic of, id Software and John Carmack, it's it's worth mentioning the, the his, his dot plan files. That would have been something that you would have found lying on a random BBS at some point. I guess it's pre blog.
Speaker 2:Yeah. The and the book, yeah, the book talks about that quite a bit about how you're right. Like, his dot plan was, like, his Twitter or his blog or his medium or whatever of, you know, updates for the day, bugs for the day, taking big shits on, John Romero when when he deemed them appropriate, that kind of stuff.
Speaker 9:I remember reading one of Carmack's dot plans, in 95 or 96 when I was when I had just recently gotten on the Internet and for for just because of my the way I started with BBSs and progressed to the internet, I learned about old internet kinds of things like like finger. And I I think I actually read one of John Carmack's plan files back then using finger.
Speaker 1:There you go. I I think there too, which to me, is always gonna be most famous for the Internet worm. The the the hole that Morris found in figurative. Yeah. Ian.
Speaker 8:Yeah. I just wanted to say that John Romero has a book coming out next year, an autobiography called Doom Guy Life in First Person. And I'm kinda looking forward to reading that to be able to see his perspective on the early software years. Because he it was very clear that the you know, there were some big personalities present in that room. I'm kinda curious to see his perspective on on those things.
Speaker 2:Yeah. Absolutely. Amazon definitely got me with the same algorithm. Masters of doom is, you know, Romero, being right at the front of the parade, taking credit for lots of stuff. So not surprised at all that, he entitled it Doom guy.
Speaker 2:And we will definitely, it'll be very interesting to contrast these 2 to see, know, if Carmack gets mentioned and and what his role was.
Speaker 1:Right. Right. So you're in the index. No Carmack in the index. Okay.
Speaker 1:Well, alright. I guess they did. The actually, Ian, while we got you, you mentioned when we did our Books in the Box episode, you'd mentioned Cyberville by Stacy Horn, which which I don't know if you read that. I actually I I read I got it on your recommendation and read it. Yeah.
Speaker 8:I I just finished it, last weekend on a on a flight to San Francisco. So, yeah, I have read the whole thing now.
Speaker 1:I thought it was really interesting. So this is about I mean, talk about, like, Adam, because this is kind of this interesting bridge between this is effectively an East Coast version of the well, that she was running called Echo, and this is all about kind of her I mean, it's not like it's a it's a memoir and it's pretty sloppy, but it is I I don't know. You know, I found it to be really interesting. I do that kind of like early era this is like earliest era social networking.
Speaker 9:And Yeah.
Speaker 8:Definitely interesting. It was definitely it was like part a New York in the nineties book and part, computer. It's like a soft early social networking book. Learned through reading that book instead of learning through mistakes again for the, you know, the era of social networking around the mid 2000s to to, like, early 2010s. There's, you know, definitely definitely some really difficult problems that they face.
Speaker 8:Yeah. And and no easy answers to them. But, you know, they they could have learned from that, I think.
Speaker 1:Well, and I I think it does show, like, just how endemic some of the challenges of social media are in terms like, the the the challenges that we have are not new challenges, but the, it was I mean, she had all of the same kinds of not all, but had many of the same kinds of things that, would would come to be really define the kind of the struggles that that Facebook and others have had. That's too recent.
Speaker 8:I mean, it was absent the the, like, advertising network and data privacy problems that some of the, social networks have have experienced more recently. But, that it was very much a, you know, humans on the Internet are still humans and then all the problems are, at this going to be the same whether it's now or 20 years ago. Humans haven't changed that dramatically over the past 20 years.
Speaker 1:Yeah. Humans on the Internet are definitely problematic is the the conclusion. But, anyway, thank you very much for that recommendation. So I think it's, I mean, pretty obscure. It's not it was, I had not heard of echo at all.
Speaker 1:Was definitely definitely interesting.
Speaker 3:It was it
Speaker 8:was only because of that, broadband book Yeah. Right. About about, the contributions of of women to to technology over time. That was the only way that I found it.
Speaker 1:The other thing that's super interesting, the reason that, like, this book, like, you absolutely could not publish this book today. It actually kinda reminds me of of Paul's boutique, Adam the Album, you know, which is so like, samples of the Beatles all over the place. Yeah. Yeah. You could not make Paul's Boutique today because it was kind of like that was the medium was new.
Speaker 1:In particular, she everybody on their social network on echo uses their real name. And so she uses real names, like, throughout, like, throughout throughout. First name, last name. And then along with, like, a, you know, a dramatispersonae at the end, just in case you means you could, like these people are all readily Googleable Googleable. And I mean, it would it's just for I don't know.
Speaker 1:Ian, that must have struck you as well because she's like, wow. Okay. We're gonna, like, talk about a post that this person had from 1995. Okay.
Speaker 8:Yeah. There was definitely very, very much a, like yeah. As I said, it's like New York in the nineties, book where it's talking about, like, how these people's lives intersect with the social network rather than the social network itself. So it's, like, leans on that social aspect and has a lot of very, explicit stories in in the book about these individuals, that I don't think that you would publish this book today with with real name real names attached. But it's also just an interesting, like, experiment on, what a real name social network looks like.
Speaker 1:Totally.
Speaker 8:Which, you know, we do have I mean, Facebook people mostly use their real names. But, yeah. It's very, very, very different to say, you know, Reddit of today order.
Speaker 1:But I just found myself yearning for, like, a symposium of these folks because these are all, like, a half a generation older than I am. I think these are all I think, actually, most of these I think Horn would probably consider herself a young boomer, not an old ex. I consider this is, like, another 10 plus years older than certainly I am. And I would love to get this group of people together to reflect on what it was like then versus, you know, what social networking has become.
Speaker 8:Yeah. It's very interesting. I mean, the it it seems that the majority of contributors were in that, mid thirties to early forties range. It seemed like that was the bulk of their people in the early nineties. So there's, they definitely would have had some very interesting experiences then and to be able to reflect on I mean, a number of them are on Twitter today, so it's clear that they continue to use social media in some form or or another.
Speaker 8:So it's like it would be interesting to to hear their perspectives on, you know, social media over the past 25 odd years, and how it's changed and what things that they think, you know, they saw and they saw other companies making the same mistakes mistakes and be like, oh, okay. We went through that 15 years ago or whatever or where they, you know, where there's new things have come along and they're like, oh, okay. That's interesting. It's kind of like this that we did 15, 20 years ago, like that kind of stuff. Yeah.
Speaker 8:The
Speaker 1:I I would I would love to get that crew together. That's gonna be Adam, I wanna host a picnic for everyone who's on the boat
Speaker 2:in the next Perfect. And we'll record it. Sounds great.
Speaker 1:That's what I'm trying to say. Alright. So we should probably get some of the a couple of folks did actually ask us, some pretty good questions. 1, Matt, you'd asked us a question about, network storage. Thought it was pretty good.
Speaker 1:Do you wanna go ahead and reask it and we can take a swing at it? If Matt's still here?
Speaker 9:Sure. Sorry. Just had to unlock my phone and find the unmute button. So the, the question is, I I was well, I was looking around the oxide website a little bit today because I I knew you were gonna do some oxide q and a. And I I had I had 1 or 2 other things in mind to ask to to possibly ask about.
Speaker 9:But when I came across the storage page and I read that oxide is using, distributed block storage. I thought, well, wait a minute. What about, back at Joyant when Brian and a couple of other people blogged in in the wake of a big Amazon EBS outage in 2011, blogged about, the perils of network block storage and and, a magical block storage abstraction and then more generally about cascading attached, storage on the compute nodes, using ZFS, of course. And, and not the the Amazon EBS type, networked block storage. So I'm I'm just wondering.
Speaker 9:And so I asked Brian if you had changed if you had changed your position in the intervening years, which, of course, is a is a fine thing to do, or if if there's something different about, about oxide storage implementation that might resolve some or or eliminate some of the old failure modes?
Speaker 1:Yeah. It's a a terrific question. I would like to say that in my blog entry, entry, talking about the peril of network storage, I did make a Simpsons reference at network storage delicious but deadly, which is a, that's a Troy McClure lead paint reference. That's right. I think and lead paint, let's face it, is delicious but deadly.
Speaker 1:Yeah. And I think a little bit of both. The I think that, it is definitely true that and I I don't think, you know, we're we are we have not, found a way to square the cap theorem, and there there are still trade offs to be made, and it definitely it is there is parallel there for sure. Problem is and, actually, this is what would be interesting to kind of expand on. The all local model has problems too.
Speaker 1:Right? And we the the, data durability is not so much of a problem, but availability, obviously, is a problem. That box bounces. Your data is sitting there. It's nowhere else.
Speaker 1:And then the thing that was really kind of brutal that we found and that we I think and and, Adam, you should obviously, you know, take the wheel here in terms of something that you and Josh thought a lot about and Alan. But the the the thing that was really tough at Joiant was doing kind of asset management with all local storage and, actually being able to, be migrating an instance is really, really painful. And when migrating an instance is painful because you gotta move all the data, obviously, when migrating an instance is painful, you don't do it very frequently, and you end we ended up with a lot of problems that kinda cascaded from that. So that were kind of operational challenges. Now the flip side of that is we there you know, you don't have a network block store that a that everyone is leaning on.
Speaker 1:So there are, like, lots of problems that we didn't have. So, yeah, we're gonna we're we're gonna we're we're flip the script a little bit, and we're gonna get rid of the problems that we did have and have a bunch of the problems that we didn't have.
Speaker 4:Yeah. We'll try some try some different problems out.
Speaker 1:Exactly. We can't hog the same problem.
Speaker 3:I think we I think our position is broadly that it is still difficult. And if you can avoid it, you still should. But that we can't avoid it in order to deliver the features that we want. In particular, live migration of VMs is challenging when you can't just point the storage point at the storage from both the source and the target of a live migration. So
Speaker 2:Well well, not just like with migration. I mean, even resiliency against a node failing or resiliency against a single,
Speaker 3:you know Yeah. Any kind of motion of of VM without needing to move the storage then.
Speaker 2:Yes. Yeah. Right. But whether live or not live. Exactly right.
Speaker 5:Right.
Speaker 1:Spontaneous migration.
Speaker 2:Surprise migration.
Speaker 1:Surprise migration. Exactly.
Speaker 3:All planned, but, you know, I mean, it's not expensive. Like, it's not expensive to move it because you don't move it. That's the
Speaker 1:Yeah. You know? And
Speaker 3:and then yeah.
Speaker 1:And, I mean, I think there are also there are certain aspects of the problem that make it easier. One thing that's actually important, Matt, and another kind of, I I don't know if this is a, you view this as a switch or a refinement, but definitely the hardware virtual machine has become the abstraction. And it's when we are not the the when you're build deploying on an oxide rack, you are always deploying on virtual machines, hardware virtual machines, which means you've got a a cleaning block interface coming out of that virtual machine.
Speaker 9:But wait a sec.
Speaker 1:You I know. Right. Yeah. I know. Go ahead.
Speaker 9:So many talks about running containers without VMs.
Speaker 1:I did. I well did I I and I mean I still like I stand by my past selves.
Speaker 2:This is a left for the age is and Matt has the receipts.
Speaker 1:Exactly. Well, I well, no. I think it's great. I honor the receipts. I mean, it's like I look I've left a lot of receipts over plan.
Speaker 2:I I I There's a paper trail. Right.
Speaker 1:There there is a paper trail. And so there no. This is always on my it's probably more of my I mean, this is always on my mind of, like, am I to what degree am I contracting my past self? I mean, you also don't wanna be kind of, like, foolishly consistent with your past self. I do find it, like, you know, aesthetically at some level, unfortunate, I think.
Speaker 1:More so than displeasing. I think it's aesthetically unfortunate that we are recreating all of these kind of confining hardware abstractions, but I've also come to accept that that's the only the the only real way. That layer, that that that hardware virtual machine abstraction layer is actually our most important layer because it allows us to run effectively arbitrary software.
Speaker 2:You you you mean the fact that the industry has standardized as like on the X86 instruction set as the lingua franca like that, that, that is the unit of of sort of compatibility across these different things. And we just sort of need to make it work on both sides.
Speaker 3:Yeah. I don't think it's quite even x86. Like, I think that there's room for ARM in the future, but it will still be partitions that look a lot like virtual machines we're delivering now. It'll just be, you know, with an ARM instruction set in the future that have existed.
Speaker 7:That's what we need. Bring their own operating system image. I mean, that's really what it boils down to.
Speaker 1:That's what it boils down to. And it's the thing that kind of is is rotten about that is that there's at some level like, in particular, the use of DRAM. This is not a a most efficient use of DRAM in that you've got the guest is gonna ask for, you know, 16 gigs of memory or whatever, 8 gigs of memory. Any of its memory that it doesn't use, the hypervisor doesn't really have good visibility into that. So you end up with kind of isolated memory.
Speaker 1:The problem is that the alternative, which and I think feel Josh, is what you and I certainly live is where we make better use of of available hardware resources also leads to less consistency. And when that clips out, it gets really ugly. So someone has gotten used to the fact that, like, oh, actually, my application actually is leveraging this Deepgram cache that I'm effectively sitting on that I can't see. And when I'm the only tenant here, it performs really well. And then another tenant lands on this compute node, and all of a sudden, it performs very poorly, and I'm upset.
Speaker 1:And that's that's that sucks. So it's a long way of me trying to square my past self. How am I doing? I I
Speaker 2:mean, I just the the the Docker alternative, I mean, to me is is even less appealing that it standardizes on the Linux system call, you know, table as the standard unit, which is I don't know. Oh, he's more frustrating.
Speaker 9:Did all that work to make it work, dude.
Speaker 1:I like
Speaker 9:to brand it. And I'm I'm I'm not trying to, like, give give you a hard time about flip flopping for its own sake. It's just that I bought into what you said before. No.
Speaker 1:I bought into it too. Oh, yeah. I listen. I bought into it too. The so I'll tell you actually, like, the true Waterloo on that, and I mean, Waterloo in the, like, the
Speaker 2:Not not the RIM headquarters.
Speaker 1:No. No. Not not the RIM headquarters. Exactly. But that in the the Napoleonic sense, the true Waterloo was when we had a regression, you know, this application kinda had work yesterday, didn't work today kind of a thing.
Speaker 1:And it's, like, alright. Well, that's and it had was reproducible enough. The actual code was terrible. Some of the worst code I have ever seen as a customer's code. I mean, it wasn't they didn't write it.
Speaker 1:They were using an open source package. That was just, like, I mean, I don't know. How do you express, like, memory and safety? I you know, I've actually I want to, I actually can't I've actually tried to look for it. It was it's a very non script name.
Speaker 1:I swear to Steve that I wouldn't name and shame at the time. And now that that time has long since passed, I wanted to go back and, like, now is the time of name and shame and I can't find it. But it is, like, memory violent, this thing. It, like, is you it's the kind of thing I don't usually look at, like, code c port c port plus code where you're like, that's a bug. That's a bug.
Speaker 3:That's a bug.
Speaker 1:That's a bug. That's but you know what I mean?
Speaker 2:Yeah.
Speaker 1:And in particular, what this thing was doing had, like just it was amazing this thing even compiled, let alone did anything. And in particular, it had so many rampant memory violations that were happening over accident. And it was phishing effectively, not deliberately, accidentally, deep in its own stack and relying on the fact this uninitialized value was 0. And as it turns out, like, we had made a fix that we needed to make that effectively pushed the stack up by one word, something that the operating system has an inalienable right to do, and it broke the application because now it wasn't getting 0. Now it was getting this, like, the it was going from a fixed offset, so it was getting this kind of other different value.
Speaker 1:And we and the app was dying, and it was one of these things where it's like, this is not resolvable. This is we we are having to run-in it is really hard to run incorrect software with a porous abstraction layer. That's the problem.
Speaker 2:So And and you're saying VMs are a much tighter abstraction layer.
Speaker 1:VMs are a much tighter abstraction layer. And the thing
Speaker 3:that sucks with that monitored system calls or something now. And and, like, some of those system calls open up fractally into its own base of system call that all like like the because Linux doesn't really do character device things. They do magic new system calls that effectively open what other operating systems might do with a character device. So like epol and and event FD and signal FD and all these things. Yeah.
Speaker 3:It's one system call, but then you get a device that has its own entire behavior that you don't have to emulate. And I think I think when we finally flew off the rails with namespaces, specifically like mount namespaces, where, like, your temp and my temp and his temp are all different temps. Every process has a totally different VFS, basically. It's like, well, let's, stop the world. I would like to get off.
Speaker 7:Yeah. It just it's So the the the myriad contents of anything under /sys or /proc or whatever. You have all these sort of, you know, obscure files that have contents that are in some format, which may or may not change. You may or may not emulate faithfully, and god knows what it's supposed to be in those things.
Speaker 2:Yeah. And this is gonna be a
Speaker 7:cap wider than just system calls.
Speaker 2:That's right. If you echo some random bits into some random file, it changes some random behavior. Yeah.
Speaker 1:Yes.
Speaker 3:But then What is the swappiness anyway?
Speaker 1:Truly, what is swappiness? Me and the virtual machine doesn't have these problems. Right? The virtual machine is an it it for better and for ill, is a much less porous abstraction. And I and I'm with Josh.
Speaker 1:I don't know if that that that's necessarily gonna be x86 forever. I think that could be ARM, maybe even with 5, some kinda graceful in the future. But I think it's probably gonna be a first No.
Speaker 2:That's right. I mean, it's
Speaker 3:I'm a gene politician of some kind like that. Yeah.
Speaker 2:Yeah. No. Virtio is is more significant than the particular ISO for for sure. I I guess I just mean, kind of this low level.
Speaker 6:Yeah. Yeah.
Speaker 2:I mean I'll tell you what, I mean, uniting these last two things. One of the things that really irritates me is that we've got a a guest operating system creating a file system on top of block storage interface, which talks to some other file system, which is, you know, chunking things up and and spraying it all over, you know, some some NAND, which also isn't really speaking block. So we've got like these, these at least like 3 to 5 different div re divvying up information all to emulate this interface that has really existed for 10 years and and probably was antiquated. And, like, the the biggest innovation we've seen in it is going from 512 bytes to to 4 k bytes.
Speaker 1:Right.
Speaker 7:I mean, you're describing modern hardware in general in many respects, but also it it might be important to note that that is an implementation artifact of of our current implementation. Nothing prevents the guest from paravirtualizing and coordinating with the hypervisor and coordinating with the whole software stack and coming up with a different abstraction.
Speaker 2:I I totally agree, Dan. And I think, like, some of those exist, but there's also a chicken or the egg, you know, conundrum of who is going to, you know, make who's gonna popularize what first and and, you know, how are we gonna jump together?
Speaker 3:Well and ironically, if you do make the abstractions better, each time you do that, they become more complicated and more nuanced and much more like the Linux system called plane that we were trying to avoid emulating. Like, Vodafest stuff that they're driving over in Linux now to replace the 9pfs stuff to try and present a file system, like a hyper virtual, sorry, a parent like a power virtual interface to a host file system without an intermediate like block fiction, ultimately is increasingly like, well, let's just do whatever, like, the vNode subsystem replacement in Linux does, and we'll just expose that as a direct call, like, hyper call to the guest. The guest is Linux. Right? It'll be fine.
Speaker 3:Like, I mean, it just it's you see you gotta now you're emulating a really specific file system thing about which there are probably not that many stability guarantees long term anyway, as as effectively like a a whole new driver and hot like emulated hardware. It's just it seems
Speaker 1:blocked. Drive Block is
Speaker 3:at least just a big array of blocks. Like, it's pretty hard to screw that up. I don't know.
Speaker 1:Yeah. I say, Tom, I know you I was gonna Tom, go ahead, actually. I I obviously, you've, lived the history of NFS and all of this.
Speaker 4:Yeah. I I was just gonna mention that I've done some research into the origin of the 512 by block block if you're interested.
Speaker 1:Yeah. Always.
Speaker 4:Yeah. So I think it comes from the r k o three disk drive for the PDP 11, but it was also also available for other PDPs. So there's one particular mode of it and support for 5 12 byte blocks. And it was physically identical cartridge to what IBM used on the IBM 11:30, but for some reason, they used 320 16 bit words. It wasn't a power of 2.
Speaker 4:So we have it all to thank, the PDP 11 4.
Speaker 1:And you said it's the r k I o? What was it?
Speaker 4:R r k o 3 is the model drive, which was predecessor of the much more popular r k o 5.
Speaker 1:Yeah. Interesting. And and so you say, like, y 5 12 versus a instead of a larger size, presumably.
Speaker 4:Yeah. They could have done I mean, obviously, it's a power of 2 as convenient. And I I thank them for that. But, other people were squeezing more more data out of the same drive.
Speaker 1:I wish they had picked a power of 2 that was nowhere near a tempting power of 10 to avoid the in terms of, like, lies that the industry tells, I do feel that the, like, what is a megabyte versus a, mebibyte?
Speaker 3:It's pretty hard to say mebibyte as an adult. And, like, in a professional set, I just it's I feel like it's a made up word.
Speaker 1:There you said it. That's yeah. I I feel like that doesn't really come up
Speaker 8:challenges of running a large number of virtual machines, is the management of security patches because you have, you know, a number of different, OSes running in those in those different, virtual machines, and you need to be able to make sure that, they are keeping up to date on their security patches, and you understand you can identify the nodes that, you know, are running a a known bad version of the software that needs to be updated and that kind of stuff. Is there plans around building tooling to assist customers to be able to to manage those security patches and manage the metadata around what, OSs are running inside those VMs and what sets of softwares to be able to manage those security patches? Or is that going to be something that you continue to outsource to, you know, other software providers?
Speaker 2:You know, I think that that's something that you know, we're gonna be in heterogeneous environments where people have VMware or GCP or AWS or whatever. And I don't think they want, like, a solution per cloud or solution per, way of deploying instances. You know? And we didn't need another big problem to take on, between us and the initial ship. So I'd I'd say that if customers know, maybe there's ways we can support that better.
Speaker 3:I think something that we learned the hard way a lot at Giant, which is partly informed. Our choice to drive really hard on on virtual machines is trying to meet the customer where they are without needing to educate them. I think and
Speaker 1:We're gonna experiment with that?
Speaker 3:We're gonna give that one a try.
Speaker 1:Like that one you want.
Speaker 3:Do you wanna buy this and only this? Alright. Well, then but but, like, critically, I think customers that have 100 and thousands of VMs probably already are managing their software estate in some way. And I think that as much as possible, VMware stack or something.
Speaker 1:Well, I do think it's and may Adam, maybe it's worth talking a little bit about. I mean, this is not that we would have done it any other way. We definitely had done it this way in previous lives, but this part of it is more API driven for everything. So it is actually easy to go build the customization on top of it.
Speaker 2:Yeah. I mean, for sure. Like, anything that that our customers can do through the console or through the CLI is also something they can automate, so, you know, they can have at it. And I think we'd envision integrations the other way too, where, you know, our console can drive other types of APIs in the fullness of time. If there's, like, some, you know, 3rd party software like what you're describing, Ian, that folks want to automate in some way as part of, like, a standard practice, like, for all their instances deployed in oxide.
Speaker 8:Yeah. I guess the the building block here that I I guess that I would make the most sense here is, like, is there opportunity to store, I mean, metadata is a bit of a dirty words to me, but data associated with, particular VMs, so that you can potentially build this this tooling on top of the oxide platform rather than, you know, having to manage, the VMs in oxide, but then you have to manage your own database to be able to then say, this VM is running this particular piece of software or that kind of.
Speaker 2:You know, it's interesting, Andrew, because because lots of APIs, in in lots of different domains give you, like, a little bit of, you know, space to store some of your own metadata like you're describing. And in almost all of those cases, you run out of gas, you know, whether it's they don't give you enough space or you need some additional, level of consistency between objects. So I think even, even these APIs that provide a little bit of glue like you're describing, you know, a lot of automation ends up needing their own key value store, their own database of some description.
Speaker 3:You know? Yeah. You can definitely upgrade that you can
Speaker 8:definitely upgrade that kind of stuff. But I guess the like, is there, like, a, you know, a power up or a plug in platform or The other part of it is, like extend extend the the that. Like, is that is that kind of extensive? We tend to
Speaker 2:be pretty, like, problem driven. Like, customers say, we need to solve this particular problem, and then let's go build facilities for that. We tend to not build support for things that are that are not concrete because I think we've seen too many times that that support can end up being insufficient even for the kinds of things that we sort of imagineered people into using.
Speaker 1:Do you think of kiosk mode when you think of this, Adam?
Speaker 2:I did, but that's a great example. You wanna talk about that?
Speaker 1:Yeah. So kiosk mode, because I do feel that and as an engineer, you always have to resist the temptation of building something, especially if it's easy. This is where engineers get real if it's easy to build, like, oh, I can build that quickly. So why not? And so we had, in particular, a customer when we we were building a storage appliance and the, they described, like, you know what I would love to be able to have is I wanna be able to give the kind of this this analytics display out to people on my team, but I don't want them to be able to do anything else.
Speaker 1:So I want like a kiosk, like, of just analytics. Like, okay. So that sounds great. And it was super easy to go do, in that way. But, of course, like, nothing is actually easy.
Speaker 1:Right? So it was, like, you know, it was, like, maybe I think Bill was working on it for, like, a weekend. And then it was, like, maybe maybe drag in the Monday or Tuesday or Wednesday, maybe a little bit. Then by the time you get kind of, you know, everything integrated, it's probably a week of effort by the time you get it all done. And I don't know that anyone ever used it ever.
Speaker 1:We definitely talked to the customer, like, who became a Big Fish Truck customer. We were the big appliance customer. We asked him, like, how's kiosk mode? They're like, what? Like, we'll do it.
Speaker 1:Do you like kiosk mode? Like, do you not know what you're talking about? And then we kind of explained it, and they're like, oh, okay. That's interesting. Why did you guys do that?
Speaker 1:It's like, because we had a conversation with you and you said that you'd like this. Like, I okay. Yeah. Boy. I guess I say a lot of things.
Speaker 1:I don't know.
Speaker 2:It was definitely an early lesson for me in product management, which is, like, customers do say lots of things. And sometimes they even mean them. And, like, other times less so.
Speaker 1:And it is really hard. It is you kinda have to go to the real so I think the and kind of a long way of saying that we try to go to this really acute pain points that, hey. If you guys do if you do this, I can that locks me. If you don't do this, I've got no way of solving this versus, like, I've got a way of solving this without this, but it's kinda ugly. It's like, okay.
Speaker 1:Well then, like, go solve it. Let's go solve it the ugly way. And once you've done a couple of times, let's maybe invent a better abstraction. But I mean, there's no pat answer for this. Right?
Speaker 1:Adam, I mean, it's kind of an it's an art for sure.
Speaker 2:Exactly. And and some of it is the, you you know, we want to enable people to do as much as possible with our product. You know, and then when once we see where those things go, then making them easier. Like, as a concrete example, one of the things we've talked about recently is, when the, you know, inevitably something will fail on the rack, it will need to scream for help. And what's the mechanism by which it will scream And lots of customers And
Speaker 1:we and we've settled on a speaker. We're actually gonna put on the speaker on the rack. Actually, it's the
Speaker 2:other way around. The alarm is gonna go off every 10
Speaker 1:seconds. It's a problem.
Speaker 8:Instead of you screaming, it's the service in the data center.
Speaker 1:That's right. That's right.
Speaker 2:That's right. That's right. A little little turn about a spare notification platforms. And we will probably build integrations into all of those in the fullness of time. And in the meantime, we're gonna have some webhook, and we're gonna call that webhook.
Speaker 2:And we will probably, you know, you know, Brian, me, Josh, our service, personnel folks we haven't even hired yet. And maybe the customers themselves will build these integrations and some of them might be amazing and some of them might be a little bit janky. And once we see that, you know, people need this and people have done 2 or 3 or 4 of them, them, then we'll build into the product. But then it becomes really clear that if we build this, this, this meets a real need rather than building some sort of bridge to nowhere.
Speaker 1:There's gotta be are there I mean, I know there's, like, a bunch that's been written on product management, but I feel a lot of it tries to be a little too pat and kind of, it kinda skirts this fundamental ambiguity and that, like, there's not an easy answer to this stuff.
Speaker 2:Yeah. I mean, I'm embarrassed how much I've read about product management, and I have not read a book I would recommend on it. And I've read like 4 or 5. No, just because like yeah. Because like it's hard as you say, it's hard.
Speaker 2:And I think that, you know, we talked about agile too much on this, on this Twitter space at times. But I think that it's the same thing where if you get too focused on the ceremony, then you lose all of the texture of what you're actually doing and only go through the ceremony. And I think that, product management is really susceptible to that.
Speaker 1:Oh, I kinda feel like it it it's a kind of thing where I just wanna read, like, case studies. Like, I just, like, give me 20 case studies that show, you know, show some good examples, show some bad examples. I mean, definitely feel like it with with the rim. I'm reading a big one where, you know, not really paying attention to what people want and then doing it in the wrong way. It's and it's super hard.
Speaker 1:Matt, you got your hand up. I I what piece of my past is about to be dug up right now? We're gonna we're gonna spin the wheel. You know, I wanted to do a that's actually I believe it is still the most Oh, I
Speaker 9:I just realized I was muted.
Speaker 1:Sorry. Go ahead, Ben.
Speaker 9:No. No. Not not nothing on your past this time. But, were you serious about having
Speaker 1:That's why
Speaker 2:we're getting so lucky.
Speaker 9:Were you serious about having a speaker in the rack?
Speaker 1:No. No. No. We we probably don't. Yeah.
Speaker 2:No. No. We have.
Speaker 3:It's a question, though, because, I mean, you know, we we we do some things.
Speaker 2:We we have toyed with the idea of, playing music through fan oscillation. But that that that may be a v two feature because we've got some other higher priorities if if I understand correctly.
Speaker 1:It is also the fan controller is really a spoilsport on that because it it, it's a very fancy fan controller, that actually limits the rate at which you can accelerate the fans. So it's actually hard, I think. But why am I putting this challenge out? I'm sure we could probably
Speaker 9:Well, I'm I'm relieved that you weren't serious about a speaker in the rack because I thought I was gonna have to point out an accessibility issue for a different disability this time.
Speaker 1:Yeah. Exactly. Do not worry. No speakers at the rack. Only bullhorns.
Speaker 1:No. Not kidding. Ben, your hands up.
Speaker 6:Yeah. Oh my goodness. Everything that was said about project management is, like, in my life for the past couple weeks. So that was that was interesting to hear echoes. But, a long time ago on the website, you mentioned something about, like network monitoring of virtual machines inside the rack, which seemed very interesting for me.
Speaker 6:So I currently work at Redjack, which is a cyber resiliency company, and the whole business is, collecting network information in order to build up to date constantly up to date asset inventories. And what you were teasing and maybe aren't teasing anymore seemed really useful. So is that still in the works or what?
Speaker 1:Sure. Yeah. Adam, you wanna talk about what kind I I mean, I think teasing is probably the right verb for it, but, we definitely, are what we're really excited about. We could because we have an integrated switch and because we control I mean, with it's a p 4 based based silicon. So we actually control both ends of it, and there's all sorts of stuff we can go do, of which we are only gonna scratch the surface, I think, in the immediate future.
Speaker 1:I think there's a lot of things we can go do. So, yeah, I would be excited to hear the kinds of things that that, that it unlocks. I to me, I it feels like and I always felt that I you remember the the company Boundary?
Speaker 2:Yeah. Yeah. Yeah.
Speaker 1:And I always felt like Boundary was actually really neat and didn't really get traction for a variety of reasons, and not with not we switched it. Some some boundary problems, but the but the technology itself was it was they did really powerful things with very basic information. So I think by having real I mean, certainly, I you know, I've got my own kind of thinking on it in terms of, like, the just the real time information. I think it's very helpful to know what's talking to what in real time. I feel that there is just a lot that you can go do with that.
Speaker 1:Not Yeah. Go
Speaker 6:ahead. What's interesting that at least I've learned is that a a lot of companies in this space, there's a a sort of there's sort of 2 interests. 1 is on up to date, and the other is, like, not up to date, but accurate. Right? And so that's that's what I'm I'm mostly focusing on.
Speaker 6:That's what I'm I'm interested in here. Right? So our our biggest problem that we have is basically if you put multiple things behind one IP address, we rapidly result to a bunch of guesswork to figure out what's actually going on and then, you know, you you end up with one asset in your inventory, which is actually multiple things or maybe you have the opposite problem. And this this, getting information, at a different sort of layer of the networking stack seems really appealing to get that sort of, accurate information.
Speaker 2:Yeah. I mean, it it it blows the mind how, you know, with so much control and observability to so many different points in the stack, the kinds of questions we're going to answer is just wild. In version 1, I think we're going to see something fairly pedestrian that we're going to put the basic pieces in place. I think over time, what we'll see is, is that same product management, philosophy that we're describing, which is working with customers to solve their immediate, you know, extrapolating from that to say, how could we have enabled customers to solve those on their own? And how can we take those scripts that we've built and additional instrumentation we've done or just analysis or visualization, and then wrap a product around that.
Speaker 2:So, again, it may it will definitely be a little bit late at times, you know, for the first customer to maybe hit a unique problem. But those will spawn, you know, tools and visualizations and utilities that we know are gonna be incredibly valuable.
Speaker 1:And we have found this is the kind of thing that actually because we had great reticence about integrating our own switch because, you know, it's we're trying not to actually be 9 startups in 1 startup. We actually we actually are trying to have a minimum viable product, but it's really hard. And on the the switch, when it it we felt that we that we for a bunch of technical reasons, we really need to integrate the switch. And the thing that was so interesting is I assumed that we were doing that, like, okay. Well, this is gonna be tough because now this is much harder sale for people bluntly because now, like, the networking group has to be bought in and, you know, what are they gonna think?
Speaker 1:And when we talk to folks about the approach we're taking and why, and especially the fact it does provide this foundation for network observability, it it has a lot of resonance. So, Ben, there's no question that, like, people are really interested in the problem that is being solved. And I I think people are frustrated out there, and they're frustrated by this kind of switch layer. I'm sure you're frustrated by it as well where you don't have the visibility that you want to. You know, Arista kind of Arista's original, what it made hay on was its programmability, and I'm not sure how much of that has actually really come to a fruition, versus kinda remained aspirational.
Speaker 1:Certainly, we remained aspirational for us at Troy. We're never able to really use it in that regard.
Speaker 6:But we The worst thing that I see is someone will will try to stick our network sensor onto a packet aggregator that promptly rips all the VLANs off.
Speaker 2:Yeah. Thanks very much. Right.
Speaker 1:Right. Thank you. Exactly. Right? But, yeah, exciting stuff.
Speaker 1:We'd be, definitely excited to get your thoughts on that for sure. So we we we definitely need to do we've gotta get Ryan here, Adam, into
Speaker 2:this. Absolutely. Every time he gives a demo, we say we gotta get on here. We gotta get them on here. This is
Speaker 1:our colleague.
Speaker 2:Yeah. We're working on networking stuff that each demo blows my mind just a little bit more.
Speaker 1:But that's the fact that he's
Speaker 3:not distracted by being on the Twitter space, I think.
Speaker 1:Interesting. I think
Speaker 3:that's why his demos are the best demos. He's not finished.
Speaker 1:I've Ryze is terrific, and I would love to yeah. We gotta get him out of here, because he's that's really interesting stuff, with and he and he can kind of expand at length. So we definitely need to Simeon, your, your hand is up.
Speaker 5:Yeah. I, I've been going to Rust meetups here in Brisbane, and I I ran into someone who I probably just won't get into specific names, but it's somebody who's working on, OpenTitan, Rust stuff, on behalf of of a vendor. And and I and I asked them, you know, the talk came up, and and I asked them, so these oxide guys have gone and they've done their own thing, and, and why do you think that is? And and and their answer was, well, you know, I don't really know what their what their drivers are, what their requirements are, but I get the impression that it's hard for a start up to invest a lot of time in these large sort of multi stakeholder open source, communities. And and, you know, you, Brian, have spoken a lot about open source and how that's, you know, changed the world and, you know, specifically in oxide and and, you know, your approach to open source.
Speaker 5:But you guys do invent a lot of your own open source. And and I wanted to get your take Yeah. On, like like, to what extent do you need to, like, have this, by necessity, have it not invented here syndrome kind of thing because you just need to be able to drive towards your objectives. And you cannot actually afford to to cooperate in in in these larger communities where, like, you know, there are competing objectives and you have to negotiate when maybe you just don't have time to do
Speaker 2:that. Brian, I I was gonna offer to be crazy cop if you wanna merely be bad cop.
Speaker 1:Oh, that's oh, oh, well, thank you.
Speaker 2:That's so much. What I would say, Simeon, is I think that's categorically false. Right? That that we we participate in lots of different open source communities. Brian can speak specifically to Talk.
Speaker 2:But, when when we build our own thing, it's because we weren't able to find the thing that we needed to fit these the the role. And I'd say I mean, Josh can also speak to this. He invests a huge amount of time in open source communities to make it work. In some cases, in hindsight, it feels like it might have been easier, an easier path to go our own way. So we do lots of stuff with the open open firmware.
Speaker 2:We're working with lots of different folks in lots different communities. I I think there may have been a little bit of FUD associated with that that response in that, like, as a startup, we we can't afford not to participate and use and collaborate in the open.
Speaker 3:We put a lot of effort into getting, like, 1st class Rust support for a Lumos, for instance. And then, like, I mean, a handful of us have wandered the countryside for 2 years getting small patches into crates all over the ecosystem that we didn't write the crates, just, you know, adding support for things that we need. It's not particularly glamorous, but it's also not like something that makes the papers. I I we we participate in a lot of existing projects. Okay.
Speaker 1:Yeah. I and so I think first of all, boy, that Adam is real hot head. Well, he's he's gonna he's gonna take a walk, and now you're, the no. So, I think that I mean, Adam is right. The and and there's an entire spectrum of activity that we have.
Speaker 1:And I think Josh is right too that, like, the the community because, like, so for example, we're using Cockroach. Right? We are not a 100% not reinventing Cockroach. Dave Chico's got a great kind of document at documenting that whole exploration. And, you know, you'll see us just, like, participate in that community where we, you know, definitely hit issues, and we collaborate
Speaker 3:with that. We diagnose issues and and we deploy the software. Like And
Speaker 1:and that is kinda yes. So we, you know, Cockroach and ClickHouse are probably kind of one end of, like, we're pretty much just using these things, participating a little bit. Things like Rust, I would say we are using and we are we we do participate in a decent amount. I mean, obviously, we, you know, Steve Klavnick was was a Rust core contributor. We've got Rust is extremely important to us and really try we are very, I would say, active in the community.
Speaker 1:We find progressions. I would say movement finds for regressions. Movement is like a, always seems to to have a a an exact bead on what kinda what broke where. So I you know, that that's a kind of a community that we're I would say we're very active and although we had a broad first one. Things like ProBRS that we use for which is super important for us.
Speaker 1:And, you know, that's, like, that's one where we've sometimes had frustration with and sometimes been like, is this consistent where we wanna go? Is it not? And, you know, we come to the conclusion that this is consistent where we wanna go, but it's something we constantly reevaluate. I think in talk's case, is kind of at the talk, I I would actually say that, and, Adam, I mean, correct crazy cop. Correct me if you disagree, but I actually think that we arguably erred by being too invested in talk for too long.
Speaker 1:I was really reluctant to walk away from talk.
Speaker 2:We really wanted to make it work. I mean, we we really, really did. I mean, we we I think we saw, you you know, early on I mean, it's one of the first things I did at Oxide. I think probably one of the first things you kicked the tires on, Brian. But, like, we wanted to make it work.
Speaker 2:It really seemed like the stars that aligned. Phil Levis, who's the professor at Stanford, who did a bunch of the, you know, organized a bunch of work there is a guy I went to school with. I think, Brian, you overlapped with him as well. Like, I'm I'm kind of buddies with Phil. We we wanted to make that work.
Speaker 1:We we went to the deck of our system.
Speaker 3:We went
Speaker 1:to go visit. No, we, we, we brought still offerings. We brought still
Speaker 3:a dead road or whatever.
Speaker 2:That's right. So, I mean, we wanted to make it work. We were, we were almost desperate to make it work. And it was not the right fit and not the right fit for a bunch of different reasons. Some of them having to do with, you know, hardware and the kinds of things that, that just the exigencies of our timelines, and some of them having more to do with, some of the discussions, some of the multi party discussion that you're alluding to actually where, you know, not everyone was aligned on things where we had absolute clarity, and I think it was gonna take a long time for them to get to clarity.
Speaker 1:Yeah. And I think that, Sumit, if you haven't seen it, I would really strongly recommend actually the talk that I gave at NodeConf in 2017, a talk I did not wanna give. And the, but the the the person
Speaker 5:This is the breakup. Right?
Speaker 1:This is the the breakup.
Speaker 5:I love you guys, but no.
Speaker 1:Thanks. Well and and, actually, the thing that's so funny is that so Charles Bior, who was organizing NodeComp, really wanted me to keynote Node comp. And I'm like, Charles, like, we kind of broke up with Node. Like, I don't know that I got anything nice to say. And, of course, Charles is like, oh, then you definitely have to talk.
Speaker 1:Like, oh, that would be great. It's like, oh, come on. I do and it was actually a really I I'm very grateful for him kinda pushing on it because it kind of forced me to be like, alright. Wait a minute. What actually happened here?
Speaker 1:What did actually happen with Node, and why did we break up? Because, you know, we were so in loved ones. And what what what happened to us anyway, node? And kind of reflecting on that and realizing that, like, actually, there was a there were there's a values mismatch, and the values for node were not the values that we had for node. And when there's a values mismatch, it's really tough because values are often expressed as positives and no one wants to say, like, oh, by the way, like, we're not interested in rigor.
Speaker 1:You know? Like, that's we're not interested. Talk is not gonna say, you know, we are not interested in your ability to debug the system. Like, that's not something that talk is gonna say. But as a practical matter, when we were building infrastructure that we needed to be able to go to bug talk, There is was zero interest on that.
Speaker 1:And that was, like, that was a huge problem. A really big problem with talk in particular. And I think, you know, there's things I like about Talk for sure, but Talk is designed to be a teaching operating system, and it is designed to in fact, it's, like, almost its first class operation is the ability to load foreign binaries. And it, therefore, is agnostic about the language in which those binaries were written, And we neither of those things apply to oxide. We are not interested in running foreign binaries on a service processor.
Speaker 1:We are only interested in running as we know about, and we are not agnostic with respect to language. And in particular, an irony with talk is, like, the Rust use level support is really not very good.
Speaker 2:Or at least it wasn't 2 years ago. I mean, maybe it's
Speaker 1:something that way. I don't know. Yes. Oh, why look look who's crazy trapped now. I guess that's just the ROAS.
Speaker 1:Get off the phone. Who
Speaker 3:okay. Okay.
Speaker 1:Stop that. Don't worry. That's right.
Speaker 10:Well, anyway, thanks for
Speaker 5:humoring my question. I mean, it, that's very interesting. I would take that then as your position is that a startup doesn't necessarily have to step away from open source communities as much as, you know, I was suggesting.
Speaker 1:Oh, for sure not. And I think that we try I think that it is always This is exactly and I would say Rust is probably a a really good example of that. We share values with Rust, and it and so, I mean, in particular, like, I'll give you something that is super weird that's important to us is the dwarf support in Rust that other people aren't depending on the way we're depending on it. When that has broken, it has been really revealed that, like, this is something that matters to the Rust community even if they're not necessarily using it, which is really great. And so, like, constantly evaluating I think, actually, this is important for, like, every because open source is just another variant of a technological partnership decision, and you're building this thing out of all these other things.
Speaker 1:Some of them are open. Hopefully, a lot of them are open. Some of them, in terms of physical components, are are you know, at some level, it's like it's being manufactured. Right? And one thing that we look for for all of those is to to what degree do we share values?
Speaker 1:To what degree is the values of this thing represent our values? And sometimes we get that right out of the shoot. We try to get that right out of the shoot, and there have been plenty of times where we have oh, we think this is a fit, and then we get in and discover, like, shoot. This is actually not the fit we thought it was. That fortunately has only happened a small handful of times, but it has definitely happened.
Speaker 1:So I think that's assuming, I think it's a great question. That's it that would be my my elaborate
Speaker 2:too long answer to it. One more good cop answer is that I I think it it is a common pit pitfall for startups to say we don't have time to participate. We don't have time to give back. We don't have time to like show up to these committee meetings every 2 weeks or whatever. And I think that's, that can broadly be true, but there's still some, some key areas, key technologies, key standards, or whatever that it's worth that time investment, that it's worth kind of guiding getting involved in those communities and steering them in, you know, wise directions.
Speaker 4:Especially if you That
Speaker 5:seems similar to the tools. I sorry. Yeah. I I just wanted to say, I mean, that seems quite similar to the time to build tools thing. Like, there's no time to build tools.
Speaker 5:No. No. No. Don't get distracted.
Speaker 1:Yeah. And I think it's also, like, the the the thing that's very important to to take the time for is for setting expectations, I think. And and I noticed that Rick is here. And, Rick, I don't know if you wanted to talk a little bit about what we did when we open sourced Hubris. Because one of the things that we really try to be very deliberate about is what our what people should expect of oxide.
Speaker 1:Because I think if you set those the more clearly you set those expectations I mean, you projects should should be very overt about what's important to them because it allows people to quickly make these decisions about whether that project is a Fed or not.
Speaker 10:Yeah. I mean, coming back to, like, the general, like, why talk, I mean, Hubris is kind of the the follow on to that. And, you know, part of the the decision making there was actually that we actually had multiple people on staff who had written bare metal real time operating systems for that class of devices before, and we didn't wanna do it if we didn't have to. You know, there's a whole lot there. But, certainly, when we got to talking about open sourcing it, it was, well, why are we open sourcing it?
Speaker 10:Like, is this just to show off? Is this to actively encourage people to come in and and, you know, build a foster community around it? Is it to, you know, like, what is our top ideal? And how do we communicate that? Because otherwise, you end up with the, oh, look.
Speaker 10:It's a company that threw this over the fence. And now that you can see it, but what can I do with it? Are they actually gonna respond to to, pull requests? Requests? Right?
Speaker 10:And so it's like setting up the expectations of here's why we're doing it. Here's what we expect in terms of and here's what you can expect from us, and here's what we expect of other folks, who wanna participate with it. And it's it's not a perfect process. Like, you know, people are gonna see the projects and and make some of their own conclusions regardless of what you write. But, on the whole, we, you know, tried to set some expectations.
Speaker 10:No. We we aren't going like, we are building this to build our product right now, and we do want to see a community build around it, but we're not there yet. Like, that's we wanted to set that up front.
Speaker 1:And I think and this is very much a consequence of the kind of the previous lives we'd all lived, Rick. But in particular, I think that one of our problems with OpenTitan was there was lots to love about OpenTitan, but we as just kind of a community member couldn't get any visibility into when the ASIC was actually gonna be available. I mean, that is ultimately, you know, the we had a lot of issues with Talk. I mean, in the case of the Talk and OpenTitan decisions were arguably could be made differently. But our problem with OpenTitan in particular was really just, like, when is all we gonna have an ASIC?
Speaker 1:And
Speaker 5:because that is very interesting.
Speaker 2:And and worth noting, OpenTitan is the hardware root of trust that came out of Titan from Google. Open, Titan is, the security chip that's in Chromebooks and other stuff.
Speaker 1:And Google has been the the lead leader on this. The Google and Apple have been really the leaders on this. So we were very excited and wanted to for community members in terms of, like, okay. How can we participate? How can we learn more?
Speaker 1:And how can we and how can we build a product from this? And it was just that was, it was pretty clear that was not gonna be a bug.
Speaker 2:And just getting crazy cut back on the phone, I think, you know, sometimes you have you have bigger companies, open sourcing technologies, and then steamrolling the little ones. So I think in, in this case, you know, as, as we try to engage with this community, you know, we we certainly did not have equal voice, you know, nor necessarily should we have, but that became very evident.
Speaker 1:Yeah. Absolutely. I actually that that's a dovetail to another there's another good question that was asked online about, like, what were the big technical challenges that we had at Oxide, which I thought was a a really good one. Be interesting to get everyone else's answer to this. I mean, we've got some, like, immediate, like, super scary technical challenges that we've talked about.
Speaker 1:The, certainly, the, the 499 ohm resistor that we needed on the Telcia part, then the later 100 ohm resistor that we needed on the Telcia part. Each of those resistors was the difference between life and death. The firmware issue, on the renaissance firmware issue where we were not implementing SBI 2 properly. That was losing life and death. But I think the things we haven't talked quite as much about are some of the challenges that we had that were slightly more architectural.
Speaker 1:And, you know, Adam, I don't know if you I was Ari was in the office last week, and he had an interesting line about, you know, everyone thinks how great it would be to have a totally clean sheet of paper. But in many ways, those days were really, really hard because the problem was so unbounded. And there were so many degrees of freedom. And I feel like there were a bunch of really tough things that we had to grapple with during during that period.
Speaker 2:Yeah. I I totally agree with that. That that I mean, on one hand, clean sheet of paper has a a lot of affordances, but it's overwhelming. Right? And and and at some point, you end up almost putting arbitrary constraints on yourself just to make some of these things tractable.
Speaker 1:What I I mean, the one that, that always comes to mind particularly is is NCSI. And the, which the the the network channel side bay interface. Right? I can the which is the ability to effectively control a service processor, over your single high speed link or dual high speed links, which feels like a that like, boy, that's the way you wanna do it because the alternative is to have a a second network for just dedicated for service processors, and that was a really tough issue. Rick, do you wanna talk about some of the the the kind of the the gnashing of teeth we had about 1?
Speaker 1:Because that was in some ways the the toughest technical issue I feel we've had are from an architectural perspective.
Speaker 10:Yeah. I mean, as as you said, it's it's kind of a it seems like the right thing you wanna do. It, effectively, it lets you NCSI is to let your BMC or or service processor, connect to your primary NIC in the machine, your Ethernet network card. And the network card then acts like a switch, a 3 port switch. So you have the outbound interface, and then you have the port that goes up to the host machine, and then you have a port that goes out to this, management processor.
Speaker 10:And so you're like, well, this saves me a cable, and it saves me a whole switch, and it saves me a port. And there's there's a lot of good things out of it. But then you start digging into it, and and, like, I've had I had had a lot of experience with it beforehand from from prior research work and and deployments. And so we were talking about this, and it's like, wow, we could save a lot of cabling this way, all this stuff. And I'm like, yeah.
Speaker 10:But, you know, you really have to dig into how this is implemented because here's this Broadcom gigabit network card over here that I know you can compromise the firmware from the host and then intercept all the management traffic that's supposed to be isolated from the host because you actually get to man in the middle right in between in the in the pipe. And so you get into these, like this this technology that seems great wasn't really architected thinking through security models, and they weren't designing for the types of isolation and and and functionality that you want. You know, what happens in a in certain power modes. You know, there's all these different edge cases, and you start working through it. And you just you you start kind of looping back to, well, maybe I don't want this.
Speaker 10:Right? And and I think that's kind of you see this even today that NCSI is not, you know, a big box feature that you're gonna see on on the side of a a new machine. But it's also one of those where you can still see in in designs today where the OEMs oscillate between, is this a good idea or not? And then and a lot of times, they actually offer sort of a a halfway where there's, like, a config option that leaves it up to the customer to decide what to do, which
Speaker 1:is actually kind of the worst possible. It is. You know, I used to say at some work in fuse, so you don't have to be. And I feel like this is like, we have outsourced our confusion to you, the customer. Customer's like, oh, what?
Speaker 1:And then you're, like, even worse because it's like, okay. So if I turn this thing on and it doesn't work, like, how supported am I? And the answer is kinda like, well, not very. I think that and, Rick, I don't know. As I was trying to think about, like, how we so this was a really thorny issue because both alternatives were so bad.
Speaker 1:It is like there's not there's not an easy path here. And what is kind of like the easier path from a component perspective is a much darker path from certainly from a firmware perspective and vulnerability perspective. And what is an easier path, what is kind of easy to wrap your head around is super complicated to have a second switch. And, you know, Rick, it'd be interesting to know kinda your perspective on it. I felt like it was when we went.
Speaker 1:I thought we did a good job of, like, alright. We need to, like, make forward progress without knowing like, we all kind of agree that NCSI is problematic, but it also feels like, can this work? Because I thought it was, like, when we were asking vendors some pretty pointed questions about NCSI, that's when it really all came crumbling down. Like, this is not something that any of the vendors
Speaker 3:really not really good.
Speaker 1:It was terrible. The answers were terrible. And I think it was, like, it was it kind of began to break under its own weight. It's kinda my recollection of that rec. I remember I also remember it was super hot that summer.
Speaker 1:I but I I can't think Adam, were you in any of these discussions? Were like some we had I mean, it was like Some noise.
Speaker 2:I steer clear of this.
Speaker 1:Oh, man. They're brutal. Just because it's like it's brutal because, like, there's no good alternative and, like, you know, it it was just tough. And,
Speaker 3:the the characteristic that I think is really important between the two parts though was that I think we looked at them and, yes, both both parts in the end would be a lot of work and that the solution in either way neither part is a particular slam dunk. But the one that we picked, I think, highlights the sort of decision we make often, which is, that we picked the way that is hard, but when a lot of the work is sort of up to us. So at least it'll be out
Speaker 1:control our own destiny.
Speaker 3:Right. That's we can control as much of the firmware as possible. Yes. We have to put more switches in there, but we can pick the switches and we can write the software and we can debug the software. And we're kinda not beholden to the firmware in the nick doing the right thing and then not being able to fix it when it doesn't?
Speaker 1:Well, we're beholden to different firmware. I mean, this is the there Yeah. There is the 8051 that is in the the the the VSC 7448 that, we've gotta take this opaque blob and upgrade it. But,
Speaker 4:well a good theory.
Speaker 3:The if
Speaker 10:if I remember timing wise, this was also we were having the discussions about NCSI, and we were really caught up on you know, and we wanted to go with NCSI to reduce cabling challenges. And that was around the same time where we came up with the idea of, well, what if we actually just made it entirely blind mate with a cabled backplane? And that that, combined with the discussions we were having about the NCSI implementations in the NCS and what their firmware look like, it was kinda like, wait a minute. If we just sort of pump this whole problem back and take a wildly different approach, suddenly having that extra cable isn't really a problem. Right?
Speaker 10:There's
Speaker 1:Yeah. That's a good point. I hadn't really those are correlated in time, aren't they? I hadn't really put 22 together on that one. But, yeah, you're right.
Speaker 1:That is about the same timing. Because that that did because I remember, like, we had a lot of challenges with having separate service processor network, but doubling the cabling was certainly one of them. And with the cable backplane, it's like, well, kinda taking that one away.
Speaker 3:But I was a I I I
Speaker 1:don't know. The the the the other problems you could think of that are kind of of that ill. But that was what I thought one would that was really tough. No easy answer, both paths hard, and that was definitely, I felt like it was, like, a month to get that all resolved, maybe more. I mean, it was a long time of, like, researching both of these paths.
Speaker 10:One other one that came up, that I that I remember was when I first came in, when I first joined, there was this discussion about from a serviceability standpoint, you want to know which machine is plugged into which slot in the rack. And how do you identify it? Like, how do you know which which of these interchangeable machines is there? And we there were all these discussions about different communication, schemes, different, you know, physical layers, different software concepts and protocols. And I sat down and I'm like, well, from here, would it help if I just go through the list of things that I know has been tried and doesn't work and why?
Speaker 1:Right. It was helpful, as it turns out.
Speaker 10:Yeah. And and and that ultimately you know, that that continued to progress for a very long time. And, like, it it still became this problem of it. It's It's still a hard problem of, you know assume you have a machine that doesn't power on. How do you know that there's actually a machine plugged in that slot?
Speaker 10:How do you relay that information to the operator? And, ultimately, it came back to things like, well, the cable backplane gives you some opportunities, and you can do, you know, some careful design and and sort of solve it in our own unique way. But I I'd remember that just being a very long drawn out thing of, like, well, do you do CAN? Do you do LIN? Do you do you know, is like, what can you actually stick in here as a low speed bus that's that's gonna work, that's gonna offer you the ability to find this out?
Speaker 1:You know, I had forgotten about CAN that we were, and this were I mean, I I want to look with a lot of these, Rick, when you were talking about the various things people had tried.
Speaker 3:I was like, alright. Let me go Google that one.
Speaker 1:Let me go Google that one. Let me go learn about this one. Let me go learn about this is where I just this is one of those early conversations where I'm like, I just do not know how anything works, actually. I don't know any of the stuff. It was very educational.
Speaker 1:And then we and what we ended up landing on on in terms of LVDS, I also knew nothing about. And it was was interesting to go learn about about that, and then doing our own FPGA on top of that was super educational. The, the the question was asked online was, was the issue with the the reason that it kicked off so much the switch hardware development, or is that unrelated, in terms of, like, why the switch? The reason we did this oh, that it it again, this is where we're kind of, you know, victims of our previous scar tissue, but we had just found over and over and over again that the, commodity switches in particular, and the inability to control that layer of the stack had really serious consequences for, for availability and so on. And we had all of these same problems that we've had with the compute slide we've had with these historic switches.
Speaker 1:Adam, did I did I ever tell you about the Quanta switches that we had to use that they did that?
Speaker 2:No. No. It's just this is a joint problem, man.
Speaker 1:So okay. So so, Samsung is doing due diligence on joint. Samsung is gonna buy joint, and they wanna, like, we wanna do some real stress testing. Okay. That's great.
Speaker 1:We the the, unfortunately, the only equipment that we have to do real stress testing on is this, like, total garbage over here. So you need to, like, stress test on this garbage. And so we had and there were so many problems. I'm sure I'm Josh is I'm sure twitching right now just remembering all the the so we had these, all of these Dell machines that had their, their batteries on the Perks had all long since died. So the perks the greatest of
Speaker 3:since, by the way, when you were all still pretending that we didn't know we were gonna get bought by them, that they were gonna be a customer.
Speaker 1:Look. We this is a challenge for any so that's true.
Speaker 3:I mean, you did what you had to do, but it just I mean, the some for some context of this shitty data center that we ended up in, Which which, by the way, now that we've, like, all moved on, I feel like the next, pre acquisition cloud is gonna get to use our the data center that's left behind?
Speaker 1:Yeah. We're great. Yeah. Yeah. So the what what Josh is alluding to is Samsung, it wanted to do this all the tech due diligence, but we, couldn't reveal to the company that they wanted to buy the company.
Speaker 1:So, it but we're also a a transparent company, and so we we did message this, which is really a euphemism for stretching the truth, that this is a customer, not a not a potential acquirer. So, Josh, I'm sorry if I'm sorry for having misled you.
Speaker 3:The it was it was it was pretty silly, but
Speaker 1:Thanks. Thank you. Always good to know that I'm I actually am an incompetent liar. I actually do feel it's one of my more charming attributes.
Speaker 3:Secretly, accidentally honest.
Speaker 1:Exactly. The but so they had us on this, like, fleet of garbage with all of these bleeding perks that were that that were complaining about how they were need to be replaced. And then do you remember the Quanta switch that we had on there, Josh? That would that had this tendency to drop all routes to everybody for 30 seconds.
Speaker 3:Yeah. But then it would be fine after that.
Speaker 1:So this thing would go, like there's split brain, and then there is brain split into, like, its atomic particles. So literally, like, every compute node can communicate no computer can communicate with any other compute node. And this is a storage service.
Speaker 3:Putting the p in CAP,
Speaker 1:I think.
Speaker 3:It's not just a theorem, it's a promise.
Speaker 1:It it is it is a promise. And so and it was a I mean, you know, our software did relatively, you know, okay, given
Speaker 2:the circumstances. Given the but,
Speaker 1:of course, they're just like, why is it like, why are we seeing these latency bubbles? And, of course, like, when you drop and everything begins to arp everything else to kinda figure out who they are, and and there are also, like, knock on effects of doing that. And it's like, why is the switch doing this? Like, this is actually not on the one hand, yes, you wanna be resilient with respect to kind of these some of these issues. But the other hand, like, this is a this is a Byzantine failure mode.
Speaker 1:Like, this is not something, you should be able to engineer this out of the system. And so we and we'd had so many issues. I mean, kinda with lack p Josh, I'm sorry. I should be doing this. I'll update the the the the the link aggregation protocol, remember, Adam and the
Speaker 3:Not just that, but the the other thing. There was the, MLAG?
Speaker 1:Yeah. MLAG. Yes. Yes. The the distributed
Speaker 3:made up black p, it doesn't work. It just doesn't work.
Speaker 1:Don't do it. Do not do it. Do not do it. And so we just found that, like, integrating and trying to integrate even when you're using what is what these are standard protocols or the you've got issues with them, they are on a kind of a per switch basis. The vendor is not very interested in helping, and it's, ultimately, it's the end customer that's holding the bag of, like, well, I've got, like, infrastructure that doesn't work, and it's your software that's running on it.
Speaker 1:So it's kinda your problem, and it's, like, hard to argue with that. Like, yeah, it is kind of our problem. And as long as it's gonna be our problem, I think we wanna have agency over and control our own fate. And, I mean, I remember agonizing about that quite a bit, Adam, about whether to integrate a switch or not. And that is talk about a a decision that just have not looked back from.
Speaker 2:Yeah. Nailed it. I mean Yeah. In particular with the with the cable backplane, like, maybe that made it easy.
Speaker 1:You know, and it's funny because, like, the the integrated switch definitely came before the cable backplane. I very much assumed we were gonna do a traditional cabling in the cold aisle kind of a thing for months long after we'd made that decision, long after we burned our boats on that decision. And it's kind of a good example of how taking a clean sheet of paper avails you of things that you don't have a way of getting otherwise, which I feel we've done a couple of times, which has been fun. Well, this has been Do you guys Yeah. Yeah.
Speaker 1:Sorry. Go ahead. Do you
Speaker 5:guys do you guys ever have, like,
Speaker 4:a fear that oh, I'm I'm
Speaker 5:pretty sure you have a fear that sometimes that you're bidding off too much. But, like, like, you you you have this promise, like, servers as it should be. Do you ever think that, like, okay. You're doing a great job at building a fantastic foundation for servers as it should be, but there's, like, but that's, like, 5% of the total problem. You know, just thinking about security as one aspect.
Speaker 5:Like, so you're doing this great root of trust, but there's, like, so many other aspects. Like, do you see yourself thinking now about, you know, the oxide bug bounty program or things like that? And does does the the full scope of everything, like, sometimes give you paralysis?
Speaker 1:For sure. Just the the full scope definitely gives us paralysis. I think that the thing that we all struggle with, I I and other folks definitely chime in. But I think part of what we all struggle with is we know where we want to take this thing and the kinds of things we want to be able to go build on it, and it's just gonna take time to get actually get there. And the product that we're gonna have I mean, the the challenge of this company has been defining what that minimum product is and then getting that thing actually in the market and being able to iterate on it and improve on it.
Speaker 1:And that I feel is a is like the the anxiety is not for me anyway, is not can we go build all the stuff because I know we can and will. And not have we bitten off more than we could chew because, yes, we have, but we also we we deliberately bit off, I think, the least that we could. It's more It brings
Speaker 3:it back to product management. Right? Like, the what like, one aspect of product management is putting in order all the things that you need to do and then drawing a line between, like, between like, somewhere down that list as to, like, when you can stop to ship something. Like like, you know, how far down this list do I have to get before I can sell this to our customer?
Speaker 1:Oh, right. Or as as a software that Josh and I are currently dealing with, like, where on the list is the software that you need to be able to manufacture it? It's like, yeah, that's high on the list as it turns out.
Speaker 3:It's really important not to look at the whole scope at once.
Speaker 5:Yes. I'm guessing I'm guessing you also need to figure out, at what point can you, like, look other people in the face and say, this is some minimal minimum viable version of servers as they should be versus, this is just a server, and it kinda doesn't entirely deliver on that promise.
Speaker 1:Oh, I think we're gonna what the what we're bringing in I mean, I think when we bring this thing into the market at the end of the year, I think we're gonna have something that is that it is a pretty clear and concrete embodiment of a very different way of doing a bunch of different things. So I'm not I I'm definitely gonna be able to people of the eye know that we've taken a, I think the challenge is more gonna be as we go there's a lot to go build, on top of that. That is not about, I mean, about functionality too, about big kind of basic functionality. It's obviously the priority. But it's more the than realizing the ultimate potential of this thing is what's that's gonna be, you know
Speaker 2:That's right. I mean, the the first product is going to be coherent on its own. And there's lots of other stuff that we can do. Some of it we've even talked about on this Twitter space. But some of those things like, you know, the depth of network observability that we can build or enabling other types of automation, those are kind of easy things to excise, harder things to excise have been, you know, like, or harder decisions have been like the one that Brian just described, or, you know, should we do hyperconverged storage or disaggregated hyperconverged storage?
Speaker 2:Like, those have been some of the tougher ones. And in the future, like, yeah, we can deliver on this vision of what servers can be with what should function as a service look like, what should, you know, container orchestration look like enabled by these lower layers? But, you know, what we've got in the first cut is is like an sort of an easy place to stop.
Speaker 1:Well, I think that also getting these lower layers right. You gotta get the layers right that are gonna be immutable in the field. So the hardware, you gotta get right, and you gotta get the, like, gotta get the switch right, the backplane right. You gotta get so that way I mean, for that reason, these issues around bring up have been kind of the the most anxiety bruising. But and, actually, there've been some really thorny issues there too, architecturally too.
Speaker 1:I think, actually, one the biggest, I think, gaps in where I we felt that we could go and where we were gonna be was around the mechanism by which how can the root of trust prevent the service processor from actually, running if it's if it is not if it is not a test itself at some level. And, we the way we're originally doing that, it's like there was a window where the service processor was basically gonna be off to the races because we're not interposing on spy. Service processor is gonna be off the the the races, and there'll be a window in which the service processor you know, a window that you'd wanna minimize. But where the service processor could there would be a gap in there, of, you know, 100 of milliseconds, seconds, And before the root of trust would be able to say, hey. Wait a minute.
Speaker 1:It's not running the right thing and bring it down. And that was really, really dissatisfying. And then and, fortunately, Rick and Laura and Matt, a bunch of folks, kind of been noodling on this for a while and came up with, I think, is a absolutely beautiful solution whereby we have the root of trust, treat the service processor as a debug target, using SWD interface, which is the single wire debug interface in ARM, which is ends up being super, I think, elegant. So we effectively bit bang the SWD interface, and we emulate that. And the s because that that is a really, really robust interface, and that actually allows us to completely close this gap, so the root of trust truly controls the service processor, but without using spy interposition.
Speaker 10:So I do wanna chime in because, you you know, there was a mention of like, on the security side specifically of, are we shooting too high or are we gonna you know, how how close to our goal of servers as it should be will we get? And, like, security is one of those really, really difficult topics of there is no way of achieving perfect security, and so you're always having to make that decision about where are you gonna draw the line. But with a system like this where we're really reimagining things at very low levels of the system, a lot of it is, how do we make sure that we get the right groundwork in so that we can if we find that our thoughts about where we're gonna go aren't spot on, like, do we leave ourselves in a solid enough foundation that leaves us enough room to migrate in different ways and and implement things later? It it's there's definitely a challenge of how much do you have to build into the hardware itself. And, you know, when you fuse parts in the factory, that's your last chance.
Speaker 10:And so there's there's careful thought in those regards. But then there's high level things, like, are how do you do a bug bounty program? Well, that's that's a problem for later. Right? Like, we don't have to do
Speaker 1:a bug bounty program.
Speaker 10:And is it even appropriate to do a bug bounty program for a product like this? It's it's hard to say. And it's things that we will have to figure out, but not yet.
Speaker 1:Matt, last last question. I think we probably wanna wrap up because, get Adam back to his his toddler on a very hot labor day. Yeah.
Speaker 9:So, one of you guys mentioned, storage again and and hyper converged versus something else. So, get I I forget what the other thing was.
Speaker 2:Oh, it's disaggregated hyperconverged. Yeah. Yeah. Don't don't Google it.
Speaker 1:Right? Okay.
Speaker 9:So, back to Brian's old, post about, network storage in the cloud, delicious but deadly. One of the things, Brian, you mentioned in that post was the problem of running lots of VMs on a a very small number of centralized, well, disks at the time, actual disks at the time. So are are you get so so so now with oxide, are you guys, using like, is does every SLED have an NVMe have have have NVMe storage, and and is every SLED serving as a storage node amongst among its other duties? Or how how are you doing at this time?
Speaker 2:Yeah. That's right. So we we have one kind of sled. It's the kind of sled that has 10 NVMe devices, and it hosts both VMs and storage software, providing, you know, block storage for, it might be for some of the VMs that are running locally, or it might be VMs that are running on some other side.
Speaker 9:So then it's it's it's, network block storage, but it's widely distributed.
Speaker 1:That's right. And then also with the I mean, a 100 gig backplane, so to have a little more room there, from a performance perspective, but we're we're not and then I think also another big difference, honestly, between, when I wrote that, I think in 2011, and today is we are we're not actually hitting spindles for any of this stuff. So this is all flash. And those absolute numbers actually do make a difference. I mean, I feel like that one of my big pieces of education from having built a a storage appliance with rotating media and then doing an object storage service again when we did it together at at Joynt.
Speaker 1:The amount, the blast radius of a single bad drive is really impressive.
Speaker 9:Well, Amazon EBS even doesn't even have, at least in it to my knowledge, the kind of big failures that it used to.
Speaker 1:It right. It does not. And they and, you know, for a bunch of reasons, a bunch of engineering reasons, I think that they, you know, spent a lot of time improving that. And then it also I'm sure they've improved the substrate, with a focus on not having those kind of arbitrary queuing delays. And then you also have to be careful about kind of when the system begins to reconstruct itself when you can get into real trouble is when you the the system is reconstructing itself automatically and where latency outliers can cause the system to start reconstructing itself, you can create a positive feedback loop where the system is creating traffic because of the amount of traffic.
Speaker 1:So there's a bunch of, like, just gnarly implementation issues that you've gotta square, to get the thing to work at scale. For us, the the storage is all gonna come locally out of that rack. So every compute slide, as as Adam mentioned, every compute slot's got the 7 VME drives. The it is spread across the rack. But does that storage not going across the rack?
Speaker 10:And every time that we we feel like we're getting set in it, I just bring up some experience of running my ceph cluster at home using spindles on a 10 gig network and just, you know, remind us of what the pain points actually are.
Speaker 1:Exactly. The you know, a a huge amount of venturing at Oxide is just being able is everyone pointing to their own scar tissue. Unfortunately, we've got a good union of scar tissue, I feel like. We've got a lot of, it's actually always really I actually love when and I feel like this has happened a couple of times, when there's a technology area that people have explored from, like, 7 totally different engineering cultures and all have drawn the same conclusion. It's like a 9 o Supreme Court decision.
Speaker 1:It's like, this is just the truth about this thing. Like, everybody is drawing the same conclusion. We've got a couple of those, I feel.
Speaker 9:Oh, speaking of blast from the past, when when, I think it was Rick mentioned the Ceph cluster just now, I was reminded of the, the blog post that, Faithlife did, who I I believe was a you know, ended up using, Smart Data Center at the time Yes. Several years ago about their their their nightmare with with OpenStack and, and Ceph storage.
Speaker 1:That is the Satapocalypse is the title of that blog entry, which is out of a few episodes.
Speaker 2:No. I gotta check it out.
Speaker 1:Yeah. I gotta go check that. That's a good one. Actually, good note to end on too. That's a good one to go check out.
Speaker 1:But it is the attributes are really hard and thorny, and I'm sure we've got plenty of failure modes ahead of us. But, the, the flexibility that you get from it is just too compelling is the is the short answer there. Alright, Adam. This has been fun potpourri. It's been, this Awesome.
Speaker 1:Like,
Speaker 8:quite a grab bag.
Speaker 1:Yeah. Yeah. Quite quite a grab bag. And I, I I am gonna gonna tease the future a little bit. I I may have heard back from the one of the co authors of Losing the Signal, and I think we're gonna be able to get them on the Twitter space.
Speaker 1:Awesome. That's gonna
Speaker 2:be great.
Speaker 1:So, prepare for a rim themed Twitter space coming at some point to, near you. Alright, everybody. Stay cool, and see you next time.
Creators and Guests
