Scaling Manufacturing
Bryan.
Bryan Cantrill:Adam, how are you?
Adam Leventhal:I'm doing well. How about you?
Bryan Cantrill:I'm doing well. Yeah.
Bryan Cantrill:I'm I'm excited for this episode, I gotta tell you. Yeah. Long time coming. A long time coming. A long time coming.
Bryan Cantrill:Did you did you go back and listen to our episode from two years ago?
Adam Leventhal:I did not. Take it you did.
Bryan Cantrill:I did. And it which it was great, but it was really very much about manufacturing the first rack. And that was for I mean, which is like, hey. There's plenty to focus on there.
Adam Leventhal:Oh, actually, you know what? I did. I did actually listen to this episode. I but I listened to it, like, two or three weeks ago when we were supposed to have this episode. Yeah.
Adam Leventhal:And we talked we we we talked about the trait, and we talked about,
Bryan Cantrill:yeah, the whole thing. It's crazy. No. I did. Wait.
Bryan Cantrill:I accidentally did the homework. I did the I did the homework when I I thought it was assigned, and it wasn't. And now I and then I forgot that I did it.
Adam Leventhal:Yeah. Thought it was due, but yeah. Then I forgot that I did it. Exactly.
Bryan Cantrill:But no, it was great. But it, but there's been a lot that has happened in meantime. Yes, I am, I'm I'm really happy to get everyone here. So we've got CJ here and Doug and Aaron, I think, and Kirsten and Eric presumably, and Root. Yes.
Bryan Cantrill:If CJ can find the mute button. I gotta say CJ is Hey. I I CJ is having some generational struggles today. Sorry, CJ. I'm gonna call you out.
Bryan Cantrill:I I really it it really kills me to do this. I it kills me to do this. CJ's like, doesn't kill you to do it because you seem to be still doing it. CJ was talking about something that that we need to get done. He's like, hey, no.
Bryan Cantrill:I owe Megan a couple of those and I wanna be at $6.06 to seven by the end of the week. And I'm like, you do not hang out with any teenagers. Like you are nowhere near you don't live next to a teenager, you have got zero contact with teenagers. Adam, maybe you got surely Adam.
CJ Mendes:Peripherally aware. Peripherally aware, I would say at best.
Bryan Cantrill:You like if I were I mean, I guess I CJ, I am also so trained to now avoid that in any conversation because it just derails it into brain rot memes. It was really very troubling what is happening to our to our youth.
CJ Mendes:I'll make it seven to eight. It's seven to eight from now on.
Adam Leventhal:There you go.
Bryan Cantrill:You know, it it sadly like you you can make it seven to eight and seven to eight is too close to six to just just like seven. I just like you're gonna have to like you just have to be further away. I I tend to avoid any kind of numeracy actually. I just, you know, really hate
Adam Leventhal:All numbers below 10 are really unsafe.
Bryan Cantrill:All numbers below 10 are I just think you're playing with fire. But then also like, was walked through, like what became a thing and then what is in and what is out. It's just very, the culture is moving too quickly. That's what I like to say. I think
Adam Leventhal:we can all agree on that.
Bryan Cantrill:There we go. But great to have you all here. This is this is terrific. So I so let's kind of kick it off because we in terms of like where we were when we did this two years ago, we'd really just started manufacturing the thing. And we kind of had the idea that at some point we would need to scale manufacturing.
Bryan Cantrill:And we kind of spent like the last mean, CJ, I don't think it's too much of an exaggeration to say that we really kind of have been spending the intervening time doing a bunch of things to get ahead of the need to scale. And then, you know, in the last end months, we've need up and now like we it's it's on. And I think it's fair to say that demand is outstripping supply and we've got some exciting problems. So do you want to walk us through some of the kind of the excitement there and what that's meant?
CJ Mendes:Well, certainly I was just thinking to myself, was like, wow, was it really only two years ago? But yeah, I mean, time is flying. And it's something that we've kind of always thought about in the abstract, right? Where we are today versus where we want to be eventually. And so common supply chain things, process definition, documentation, staffing.
CJ Mendes:These things all feel kind of abstract when you're dealing with the immediate, you know, gotta shift the next one and slowly getting better along the way. I think we've done a lot of things, maybe kind of below the waterline in operations and partnership with the engineering team to get to where we are today that actually put us in a pretty good position to go scale. We've got crew of folks that have been at Oxide for, on the supply chain side, four years or thereabouts. And I was just thinking to myself how grateful I am that we've got this core group of folks that have been involved for several years now because we have a really good feel for the product, Everything from the bill of materials and how it's structured, how it gets built. Eric Anderson will be joining.
CJ Mendes:I think he's doing a drop off for one of his kids right now or a pickup rather. But Eric has been with us for about four years and I often joke that my single greatest contribution to Oxide so far has been introducing Eric to Steve and Brian and Kate. He sets the pace for the operations team and he's our boots on the ground with our contract manufacturer. And Kirsten has also been here for almost four years or over four years now. And on the procurement side, just the complexity of roughly a thousand items on the bill of materials to go buy in smaller volumes and now in much higher volumes.
CJ Mendes:And there's been some additional complexity here recently with buying things. We source globally. So it's been exciting to think about how to get all these parts to show up on time. But but, yeah, I would say in general, going from shipping, you know, one and then the next one or the next two to now we're looking at, you know, 10 x that volume is exciting and I am so, so grateful that the folks that we have working on this at Oxide have been doing it for a while and I feel very well placed to go pull this off.
Bryan Cantrill:Yeah. So let's talk a little bit and and in particular, because I think that I maybe frequent listeners to the pod will realize this, but but folks that are kinda new to hardware may not realize that we, Oxide, we do not have our own manufacturing facility. We use a contract manufacturer. Right. Do you want do you want to just talk a little bit about why and I think that that's I mean, I think it's probably true.
Bryan Cantrill:Virtually any certainly any start start up doing hardware, but most companies doing hardware use contract manufacturers. Do you wanna describe a little bit about why we why we use contract contract manufacturer in terms of Benchmark Electronics?
CJ Mendes:Yeah. Certainly. So we you know, we've partnered with Benchmark since the beginning. We're proud to build our product or assemble our product in Minnesota. And the advantage of contract manufacturing is manufacturing is a very capital intensive business.
CJ Mendes:Really it comes down to a couple of things inside the four walls of Benchmark. In Rochester, it's you have the SMT side, which is taking raw circuit boards and converting them into printed circuit board assemblies. So those are automated production lines that do all the SMT things.
Bryan Cantrill:SMT is the is service technology.
CJ Mendes:Yeah. Service mount technology. The all the pick and place machines and, you know, mesmerizing to watch for sure.
Bryan Cantrill:Mesmerizing to watch. Oh, so mesmerizing to watch. Even in the home they're the it's Nathaniel's Home Lab pick and place is mesmerizing to watch, like, let alone when you guys when you actually watch, like, an industrial pick and place is really mesmerizing.
CJ Mendes:Yeah. You know, but but those are very capable lines that that Benchmark has. And and then the other side of the house, it's assembly. And so thinking about how do you put together the work instructions, the workstations, the staffing, the training, all of the compliance things you got to think about, safety. And so the advantage you get with a contract manufacturer is you can partner with these firms who do nothing but this all day, every day and take advantage of some of that investment and expertise that they have.
CJ Mendes:And it's been, I think absolutely the right decision from the get go in terms of what we've done here at Oxide partnering with Benchmark. Really, really high end Centimeters that that we're proud to work with.
Bryan Cantrill:Well, and one that understands NPI. Right? NPI is new product introduction and there are I mean, NPI poses different problems to scaling up manufacturing. Do wanna elaborate on what some of the kind of the different challenges are at those different scales?
CJ Mendes:Oh gosh. NPI, you know, think back to our to Gimlet. And we were working closely with the NPI team. I think when you go over to the production side of the house, let me start by production side first before I talk about NPI. Production side, things need to be very, very well defined.
CJ Mendes:The build materials needs to be largely locked. You would hopefully accomplish some low rate initial production of the builds to help dial in your processes. On the NPI side, it's much more flexible. I recall Is Robert Keith on? I think for the Gimlet builds, I think RFK literally went out to benchmark with a bag of parts and was like, hey, we're changing the bomb.
CJ Mendes:We need to put these on there. These ports are out ports. I mean, it was so to give you a sense of the flexibility that you can pull off an NPI, that's kind of the level that that you need versus production where things need to be much more dialed in because you're operating at some kind of volume and you start worrying more about getting line time, getting allocated because we're not the only customer at Benchmark. I think in general, we've punched way above our weight at Benchmark, which has been great and largely Way above our weight. Yeah.
CJ Mendes:The relationships that Eric's built over there. But yeah, I mean, NPI is really key towards flexibility. You don't have a lot of documentation. You don't have a lot of things dialed in just yet. By the time you get to production, all that needs to be figured out.
Bryan Cantrill:And but and so you need that flexibility on NPI and you and it's I don't know how common is it to have NPI and production kind of from the same house? Is that Certainly that that was a strength of benchmark from our perspectives that they could do both. That that we would that that they could work with us on NPI and then we also knew that we had a real path to to scale up.
CJ Mendes:Yeah. Certainly in the same facility because you'll see a lot of CMs will do like an NPI focused facility. But what Benchmark has there is they can do NPI and then volume. And so the advantage that Benchmark has is they've got a couple of facilities in the Minnesota division. We're primarily in Rochester now, but they've got another facility down the road in Winona that we've built that before.
CJ Mendes:And so the ability to go from NPI to volume, whether it's the same building or just within the same division is pretty unique and has been very, very helpful for us. So, yeah, it's I I will say, like, we tried doing MPI on the production side, and I don't think we'll do that again. We we like the flexibility. It turns out we really need that flexibility before we kind of hand it over to production.
Bryan Cantrill:Yeah. Yeah. Yeah. And then okay. So then what are some of the things that we learned in manufacturing?
Bryan Cantrill:Because we and most of this experience is with Gimlet, but of course we're we're we're shifting our focus to Cosmo here. Gimlet being our Andy Milan based sled, Cosmo being our Andy Turrin based sled. What are some of the things we learned with the manufacturing of Gimlet? Like what what I guess we talked about what what worked, but of course what didn't work is always more interesting. What were some of the kind of the manufacturing things we learned about Gim?
Bryan Cantrill:What some of the mishaps, some of the opportunities for improvement?
CJ Mendes:Oh gosh. I would say there were a lot of design things that we benefited from low rate initial production on, You know? Yeah. Think the one that I the one that I think Can
Bryan Cantrill:you elaborate on that? That's actually a very that's a very good point actually about benefiting from low rate production. Yeah.
CJ Mendes:I I mean, because you get to tweak, you know, anything from like the SMT stencil, right, to like optimize your SMT workflow. But I think the one that I always think of, and I know Doug is on, the cable backplane kind of cartridges that we redesigned.
Adam Leventhal:Yes.
CJ Mendes:I mean, people don't realize some of these high speed cables, apparently a feature of super high speed cables could sometimes be fragility. And so we were just going through cables like crazy trying to put together the cable backplane. And Doug and Brooks and our, you know, our mechanical engineering team redesigned the cartridges. And I I'll stop talking here, Doug. I don't if you wanna chime in on what we optimized there, but what I know now is knock on wood, we're not going through as many cables in assembly there.
Doug Wibben:Yeah. I can go through that, and and Eric probably has some feedback here as well. It it was certainly not a single solution here. Part of it was redesign redesigning bits of the the interface to the cable backplane to minimize stress on these cables. Because for all the, you know, warning we got up front about not bending these the wrong way, they were even more fragile than we thought they would be.
Doug Wibben:So Sure. Going through, you know, our our various early builds and finding we were replacing two to three or more of our 16 cables on every rack build, that was that was not a scalable method. So, you know, in addition to making changes to the way in which they were mounted, the way in which, the operators handled them, that was a small part of of the overall improvement. And then our our supplier interaction with, our partner there, Semtech, I think proved to add a lot of value there as well in how they're packaging them. There's there's been a lot of improvements there.
Doug Wibben:They've they've been, you know, going through manufacturing process updates as well, based on feedback from us and and other customers. So, yeah, there's it's it's been a a whole team effort to finally reduce that to zero as it should have been from the start, but it took a bit to get there for sure, longer than I would
Bryan Cantrill:have liked.
CJ Mendes:Yeah. Well, I mean, Doug, that's the other thing too is, like, you know, thank god that happened during low rate production because, I mean, getting these cables I Kirsten, what's the lead time on those? It's like twelve or sixteen weeks sometimes.
Kirstin Neira:Yeah. Could definitely be long.
CJ Mendes:Yeah. And so not great. If you were doing this at volume, that would be a much scarier scenario to not have that dialed in.
Bryan Cantrill:I mean, yeah. It shows like so when you damage the cables as part of I mean, you have I mean, there's a COGS impact obviously, but you also I mean, much more acute is the labor impact, the lead time impact. I mean, just like you don't wanna have to be redoing parts of it in order to just Doug, as you say, it's a real real impediment to scalability when you're having to put together the rack two or three times in order to get it. And then we also have the problem that we it was just very hard to test the whole thing.
Aaron Hartwig:Yes.
Bryan Cantrill:You wanna speak a little bit to to kind of like what we learned about about our unit of testability and the need for improvement there?
CJ Mendes:Gosh. I you know, those first few racks, you know, they they got tested at the very end. Right? Which was out of necessity. We have rack tests, but that's a bad time to find problems.
CJ Mendes:And so I would say a lot of the work over the last eighteen, twenty four months has been in upstreaming testing and quality checks. And Aaron, we've got several testers that our mechanical and electrical engineering teams have designed and our software team has written software for. That's really just it's been a long time coming and we're finally getting there and it's just so exciting to see. And I know we've had the episode before on minibar, but there's a handful of these that are gonna allow us to move testing upstream, which is to say way before it ships so that we can find problems early and sort of disambiguate, is it the sidecar or is it the cabled backplane? Is it the scrimlet?
CJ Mendes:Help us figure out that we've got no good units going into the rack. I don't know, Aaron, if you want
Bryan Cantrill:to talk
CJ Mendes:to anyone been super involved.
Aaron Hartwig:Yeah, was going to rewind for a second and just think about how for those first few racks, we didn't invest in that unit level test like you had mentioned upfront, and we were kind of letting the failures guide where we needed to add test. And how, one of the biggest pain points and something that still crops up is the fact that the DDR sockets that we use are press fit.
Bryan Cantrill:Yeah, was gonna say, yeah, we got to press fit connectors.
Aaron Hartwig:Yeah. And it's like, when you're trying to press fit 16 connectors that have two eighty eight pins, there's a pretty substantial force involved there and not much margin for error. And it's a surprisingly manual operation at very low volume. And that was actually one of the first tests that we developed was a kind of an early screen for DRAM. And did we press all of those pins in and did nothing get bent over and short out and things like that?
Aaron Hartwig:Because it's very tough to actually see that. And that's something we're looking forward to a whole new class of problems Cosmo and DDR5 being surface mount.
Bryan Cantrill:But yeah, it was definitely a problem and we because I mean, early on we were getting it was not uncommon to have dim fallout where you would have, you know, one or two of the dims that were that you couldn't actually speak to on on and so yeah, getting it getting on that and understanding kind of what was going on with the press fit. I had Aaron, do you recall that I had one of these where the the the press fit connector came off in my hand? And I of course, I got accused by everyone of ripping it off. I'm like, I don't have the physical strength to rip this thing off actually. Trust me, it's not.
Bryan Cantrill:But it would and I also and I still actually, you know, I I still have it on my desk when one of the the press fit connectors that showed the the actual the two of the pens were just bent burbent. I mean, that they cause I mean, a press fit connector is super violent. I mean, it's gonna you are basically I mean, Adam, does remind me of Adam Leventhal, manufacturing engineer. I mean, this thing definitely requires a running start. It is basically like we are just gonna absolutely pound this thing in.
Adam Leventhal:I mean, the directions are right there in the name. Right? Exactly. Not more complicated than that.
Bryan Cantrill:Right. Exactly. Like press fit, it really is not more complicated. Like, don't overthink it. But it was definitely And also we also had problems on pin issues with the socket.
Bryan Cantrill:Am I remembering that correctly? Didn't we That
CJ Mendes:was like a whole science project to figure out because we messed with tooling, like, adapter to put the sockets in. I mean, that was we worked on that for a while just to get the right amount of force distribution across the entire sockets to get it into the board and try to minimize some of these bent pins.
Bryan Cantrill:Eric, am I remembering that am I remembering correctly that we discovered that there was a height issue with one of the operators? And we realized that we need I I I can't.
Adam Leventhal:When you say a height issue with one of the operators, I'm sorry. Does that mean
Erik Anderson:Does it mean
Adam Leventhal:of the people building the thing was short or too tall? Okay.
CJ Mendes:Eric's six foot eight. So
Adam Leventhal:Yeah. I was like, if it's Eric, I know it's not too short.
Bryan Cantrill:That's right. Everyone's Yeah.
Erik Anderson:I think we're touching on something. So a common denominator here with the sidecar backplane, sidecar cable or cable backplane, sidecar cables, best fit is that, I mean, as you scale, things you may need to be looking out for, is quite obvious, I think, potentially. But you need to make things that are processes that are independent of the operator. You have processes that are robust enough where they will survive beyond impact. So when we're in low rate production, yeah, we have a chance to maybe create ad hoc processes, like Aaron talked about, or tests.
Erik Anderson:But as you scale, that's something that is really not an option. Or you make the decision of, all right, this is a design that is not good enough that we can live with for a certain amount of time before we can phase in a certain design change or a test. Or you make the decision to stop. And that is also very impactful as you try to consolidate consolidate to meet customer delivery. So, yeah, we've been very fortunate to be able to have that that flexibility.
Erik Anderson:But yeah, as we look forward, as we need to scale, those are things that we need to be very intentional about. When do we phase things in? You can't drop everything most immediately because then you are causing mass chaos as well. So you have to be intentional about triaging what fixes are most important and incorporating those in the manufacturing line. Because everything once we cut an engineering change order or an ECO, it's done in our hands.
Erik Anderson:I didn't say we pushed it, but we need to be very aware of the latency, if you will, to get something implemented on the line. So we push an ECO while there's training, there's work instructions and things that need to be updated. You almost need to back plan just like you're buying material. Right. Long time part.
Erik Anderson:Yeah. Absolutely. So those are all things that we consider.
Bryan Cantrill:Well, then I think also, I mean, is where people will encounter the acronym DFM design for manufacturing where you realize like, this design works, but actually we could change the design and make this more readily manufacturable. And I feel we had a we had a couple certainly we had we there's points of education on Gimwat. I I for sure think the press connecters were I think we had all kind of like, alright, we've done our experiment with the press connecters, and I think we're looking forward to Aaron, I like the way you phrase it. Looking forward to new problems with with with SMT because that was definitely a challenge. What were some of the the the I mean, I because I don't know that we we've spoke have we not spoken about Medusa here, Aaron?
Bryan Cantrill:I'm not sure. We I feel like we have some Probably not. Yeah. Okay. So like there's no time to ring, I think, on this one.
Bryan Cantrill:Can you just grind like Medusa and why we because I I think that I think people don't necessarily appreciate the degree to which scaling manufacturing is this kind of all encompassing technical problem that has operational their operational components, process components, but also a lot of like engineering components where it's like, okay, we could actually build a new kind of thing here. We can actually help ourselves out. And we've done that a couple of times certainly with Medusa. Do you wanna talk a little bit about what Medusa is?
Aaron Hartwig:Yeah. I mean, so in the spirit of PressFit is hard, we have a our kind of our front plane to our networking switch is 32 QSFP ports, and each of those is also press fit in. We were seeing similar rates of fallout. And when our unit of test was like an assembled switch that we would combine or put into kind of a full loopback configuration and run all the interfaces at speed, And then you find out something is wrong and you have to spend two hours tearing it back down to get at the thing you need to fix. And it's all very I mean, like we we touched on already, these these high performance cables are very ginger.
Bryan Cantrill:Yeah.
Aaron Hartwig:The Just minimizing touch there, both on the assembly and disassembly and rework is critical. So we came up with this Medusa, which kind of takes our front plane board and can loop back the high speed signals back to modules and the higher performance modules that have some of
Erik Anderson:them
Aaron Hartwig:have some self diagnostic capabilities where they can generate and check a pattern. So we use the Medusa to loop them back. We also double check all of the slow speed signals that are actually pins that get pressed into the board and make sure that every, whatever it is, six signals per cage actually is fully mated and making electrical contact. Because we did have a problem where actually our holes on some earlier revs were too big. And you could just, Brian Cantrell, Hulk strength, one of these out extremely easily.
Aaron Hartwig:So it's just in the spirit of being able to keep pushing tests upstream and test as earliest and as cheaply as possible, both time wise and dollars wise. So the farther you get into the process, the more confidence you have you're working with good material.
Bryan Cantrill:Yeah, I mean, God, I feel I've learned obviously so much in all this, man, QSFP connectors are so complicated. I mean, I feel it's like they are their own like tiny compute sled that we are kind of press fitting in. I mean, there's just like this they draw a lot of power. There is a lot of software complexity in them. There's just a lot to get it's just easy to get that wrong.
Bryan Cantrill:And being able to test that as early as possible is helpful. And Medusa is a board that allows us to do that quickly, allows us to test that before we get the whole thing assembled.
Aaron Hartwig:Right. Yeah. And and it's it's amazing the amount of functionality you can fit into a 256 byte memory map when you can just keep infinitely adding pages and banks of pages and your three dimensions removed from whatever you have paged in at at any given time.
Bryan Cantrill:Totally. I just had a remarkable amount of complexity. And I think that like in also that front IO board on Sidecar, and that is a wildly complicated board that I think we sometimes, I'm sure I heard you believe that we certainly don't give it its due because that was you spent a lot of time on that board. That is a complicated board, and there's a lot that can go wrong there.
Aaron Hartwig:Yeah. Yeah. It it's complicated in a in a really interesting way compared to one of our compute sleds in that it actually is kind of like the same design for the most part, copy pasted a whole bunch because it's a bunch of cages that we have individual control over and all these things. But it's all the same thing that's going on. So it's very important to get the couple of things that it is doing right.
Aaron Hartwig:And then, whereas on the sleds, there's just like, the problem is just fractal. Like there's so many, so many various problems. And not that it's not as important to get it right, but it's much more difficult, think, to keep them in your head. I don't want to make it seem like that front board was too all that heroic. And that was something that RFK had had done a lot on, and and I just kinda helped him get over the line.
Bryan Cantrill:And then the so I did describe some of other boards that we've used to test. So we've also we also have a board that actually I learned about for the first time that we did apparently, like, two years ago, Reverso, which I'm like I I actually there was a bit where I'm like, is that an actual board? So can you describe Reverso a little bit? And I think, Adam, you're learning about Reverso for the first time you mentioned. Yes.
Aaron Hartwig:Yeah. Yeah. So Reverso was something that actually Tom and RFK I think we had brought this up kind of off the cuff in a meeting. And all of a sudden, like, some number of months later, they shut these boards showed up. And we just really we didn't take the time to fully put into use, but now they're kind of becoming it's a necessity.
Aaron Hartwig:So what it is is it's just a backplane loopback. So rather than needing to put a into one of our cubbies, you can just put in a Reverso and it's called a Reverso because it just connects a sidecar to another sidecar kind of directly.
Adam Leventhal:Oh, so it connects, like, the two mega connectors. Like, all all the networks just get connected from, you know, from one to the other.
Aaron Hartwig:Exactly. So it, yeah, so it it it's really confusing to Sidecar at first before we started rolling out, like, the extra software here because because when you use things like
Adam Leventhal:pilot things I shouldn't be able to see.
Aaron Hartwig:Right. Like, when you're listing hosts, it's like, well, the sidecar in Covey 17 says it's and you're just like, wait. Wait. Wait.
Adam Leventhal:And are we also, like, wiring up ignition to ignition? Because I imagine that could be confusing as well. Ignition being the low level kind of power control.
Aaron Hartwig:Yeah. So so none of this expects to happen. So it is some custom software work kind of at on all three networks. Yeah. And but what this gets us is a a pretty lightweight way to test backplane connectivity.
Aaron Hartwig:So basically, can kind of decouple our ability to assemble racks from our ability to assemble sleds.
Adam Leventhal:No. That's really neat. And is it mechanically does it look like a sled? Is it look just like the world's simplest sled till you just slide it in?
Aaron Hartwig:It it looks like a sled and there's like, you know, a one third size PCB at the back with these connectors on it. And that's it. And that's part of reason it's a lot cheaper and these connectors do wear. It's like
Bryan Cantrill:These sleds are too expensive. I'll get you a much cheaper sled, sir. Yeah.
CJ Mendes:Brian, this might be too insider baseball, but I mean, I don't know, Eric, if you wanna talk about flight test gimblets that we were using before, which definitely is not going to scale. We have this pool of walking wounded gimlets that we're not shipping for whatever reason. And so the way we were testing out racks and the cable backplane before is we would take these and I think we're still doing this today. We take these walking wounded gimlets which we've taken out of process and labeled as flight test gimlets and that's what we put those in the rack to check out the backplane.
Bryan Cantrill:They've been designated reversos.
CJ Mendes:That's right. Super reversos. Very expensive reversos. But yeah, that's that's something not gonna scale. So
Bryan Cantrill:What And what smarts need to be on the Reverso board other than the connector? Is it just inert?
Aaron Hartwig:This or It is. Yeah. It's a it's a passive board right now.
Bryan Cantrill:It's a passive board. Yeah. Yeah. Yeah. So there's no SP on there or anything?
Bryan Cantrill:Nope. And
Aaron Hartwig:so it actually So the software
Bryan Cantrill:support Reverso is to is for the software not to the software elsewhere in the rack not to lose its mind when it starts seeing things, I should say.
Aaron Hartwig:Yeah. Exactly. Yeah. That's very cool.
Erik Anderson:And I agree with it. Like, it's kinda like our fiber loop back. Like, it says passive. We're just ensuring connectivity, albeit at speed. But essentially, we're making sure there's nothing broken before we go on to the next higher level assembly when we can actually test that speed with a sled.
Erik Anderson:Totally. This
Bryan Cantrill:is kind of the backplane equivalent of minibar. This is a minibar allows us to test a compute sled without needing the cable backplane. Reversal allows us to test the cable backplane without needing a compute sled. Or at least test the compass.
Aaron Hartwig:Yes. That's right. Yep.
Bryan Cantrill:That's very cool. And then what is the so the sidecar loopback, describe that a little bit.
Aaron Hartwig:Yeah. So the sidecar loopback is that, you know, what comes after Medusa, and that is we have a full sidecar chassis. So it's all cabled together and all closed up in sheet metal. And we loop back the backplane connectors to themselves and also the front plane with DACs and basically run everything at speed and make sure that all the interfaces can link up and don't have any errors. And when we find errors, then we can correlate that two ports that are failure and, you know, and for more specific components.
Aaron Hartwig:So basically, that gives us the confidence that, you know, a sidecar that we put on the shelf is is functional. And one of the, you know, 40 cables inside it isn't misconfigured or damaged and and things like that or a capacitor to get knocked off and and all that fun stuff.
Bryan Cantrill:Right. That's cool. So this is all to serve. This is all like engineering work we've done. These are this is mechanical engineering in some cases.
Bryan Cantrill:It's electrical engineering in some cases. It's it's industrial engineering, Eric, in terms of like process engineering in some cases. But we're these are all to serve getting us to get as close to a failure as possible. We don't wanna discover a failure when we get the whole rack together. We wanna only discover those failures that we could not have discovered earlier in the process.
CJ Mendes:Yeah. That's right. It's a much better time to find it than, you know, we're 36 from trying to put this in
Aaron Hartwig:a crate and ship it.
Bryan Cantrill:Totally. So okay. So that that is all very helpful for just getting us to be able to manufacture this thing more reliably. And and we you know, CJ, think what you said that we kind of benefited from a a slower ramp, which I definitely appreciate it. Adam, I feel like that's what we did not get at Fishworks where
Aaron Hartwig:Oh, yeah.
Bryan Cantrill:We we as I tell people, like we did too much business in our first year. And because there are a whole bunch of these kinds of problems that we only discovered after we had way too many units in the field. And you just don't need like you don't need that many customers to hit the problem. You actually the there's a there's a point where it's like, well, this excess customer pain was just needless. You know, it's interesting, Brian.
Bryan Cantrill:I hadn't
Adam Leventhal:thought about that lens of like being inside of a large company, starting a new product is a blessing and a curse in that regard. Right? Being a startup, you know, having a few customers, that's terrific.
Aaron Hartwig:And if you're if you're at a
Adam Leventhal:big company rolling out something new, having a few customers is impossible. Right? Like, as soon as it's out there, it's out there, and it's very hard to, like, hold back the floodgates.
Bryan Cantrill:Totally. And I think I mean and CJ, I mean, obviously, with your experience coming from Apple, like, you are coming from that, you know, big company world where, like, we put this thing out there and there's gonna just explosive demand. And boy is that I mean, that's hard that's a hard it's a harder problem. And that's a problem that is like we will con we will increasingly have as we get more and more mature. But we kinda there there's a you know, there's a some kind of a fortunate blessing.
Bryan Cantrill:And people are like, god, know, guys are a startup. Like, I don't know if I can I don't if I wanna buy a 100 of them? It's like, well, don't worry. Like, we're a startup, so we're not sure we can actually we we might have that issue making that many, but that was, you know, increasingly, like, that's not where we are. And we're like, no.
Bryan Cantrill:We can like, let's go. You need let's order ready to take your order. But that's only because we've got we had that slow ramp where we were able to to get that stuff figured out.
CJ Mendes:Yeah. I think a lot of us have laid that really sound solid foundation, right, for for where we need to go because we've had the time and then importantly, seen enough cycles to where we really knew where to focus our energy to kind of get these things in place. Like I said, it's been a long time coming. I know everyone on this call from Oxide has been talking about this for a year and we're finally getting to the point where it's coming to fruition, which is fantastic.
Bryan Cantrill:Yeah, it's fun. Okay, so let's talk about another really challenging thing when it comes to like, okay, so you got like the actual, like you you need to be able to physically make these things. You gotta be able to physically make them in order to be able to scale that up. Like that's gotta be reliable. You just need some maturity there.
Bryan Cantrill:You need to have it really, I mean, and obviously we've said before that we can kind of control all of our manufacturing software, which really helps. But you need to actually like have the parts. You got to go procure the parts. You got to go buy the parts. And procurement is its own total science.
Bryan Cantrill:I mean, is it is not easy. I would say like art and science because I think that
CJ Mendes:Oh, yeah.
Bryan Cantrill:Mean, Kirsten watching you, you know, I feel like there is this, know, getting into larger volumes has been, again, both both a blessing and a curse. Do you wanna describe some of the kind of the challenges there and opportunities?
Kirstin Neira:Yeah. Definitely. You know, I've been here for four and a half years, so I was here when we were buying parts for inlet EBT. So, you know, cut strips of a 100 resistors and capacitors. And now we're talking buying hundreds of thousands.
Kirstin Neira:So it's it's quite a difference. It's a much different process. It is a blessing and a curse, a lot of prayers, and it's a lot of relationships too. We've got about a thousand or so parts across all of our bill materials. So we've got the two main boards.
Kirstin Neira:There are actually three now that we're bringing on Cosmo, and that has actually a good number of new parts on it. And then all of the smaller boards as well. So keeping everything in check, making sure that I'm not the one that's holding up something from being built is my main goal every day. You know, the good news is we do have design stability, you know, at least on a Gimlet and pretty much a sidecar bomb. So those are are little, I don't know if I say, easier to procure.
Kirstin Neira:You never know when you're you're gonna find an out of soccer, a park goes end of life, and there's just no supply anywhere. But when you're looking at ways to kind of like we have a really small team, the ops team at Oxide, so trying to optimize the number of people we have and all that we have to do, looking at our contract manufacturer to kind of help with the procurement activities is one of the things on the table. They've got a team of people that does procurement. They do this for other customers. So really looking at those stable BOMs, looking at the parts that we could easily hand over, more of the off the shelf parts, kind of keeping anything custom for now.
Kirstin Neira:There are still some tweaks being made to some things here and there. Some of them are a lot harder to procure. We've got what we call the arts and crafts parts, which I honestly have been cutting at home for quite some time now. I I do finally have a supplier for them. But some of this stuff is just challenging when you're talking EMI gasketing with twenty week lead times and finding someone who will cut you 500 of them.
Kirstin Neira:Minimum order quantity is like 5,000. You're like, I just don't need 5,000 of them. I'll cut 500 on my kitchen table. So those are still some of the design or some of the issues with procurement. But as we scale, we're able to find suppliers now to do some of that stuff.
Kirstin Neira:But I think one of the biggest things, especially right now as we're looking to scale, is the relationships with our suppliers and distributors. So
Bryan Cantrill:Yeah. Yeah. Yeah. Yeah. Elaborate on that.
Kirstin Neira:Yeah. So while we're we're looking at handing over some of the more off the shelf parts, keeping those relationships with the, you know, the DRAM SSD CPU suppliers to be able to negotiate those prices and really bring them on board to sell them on oxide. Our suppliers, they love oxide, they love our vision. They petition for us within their companies to get us supply. With all of the AI boom right now, it's not the easiest to get supply, especially on DRAM.
Kirstin Neira:We are not a big AI company. So having people go to bat for us with the suppliers has been a blessing. We would not have been able to do this without those relationships.
Bryan Cantrill:So I think that this is a really important point, Kirsten, because and this is something we've talked about in the past, but I think that this is a really important difference about the way because and I we've spoken bluntly about this that when we the joint was bought by Samsung and we assumed like, oh, great. This is gonna be like, we're gonna have Samsung's purchasing power. And this means that we're gonna have like, we're gonna have like newfound relationships with our suppliers. And we did have a newfound relationship with our suppliers. Our suppliers hated our guts.
Bryan Cantrill:And Yeah. We're like, what? And they're like, look, we love you at Joynt and we're gonna continue to give you the kind of support that you've had, but you should just know that like I'm doing this over what my executives want me to do because they hate Samsung. And I'm like, why do they hate Samsung? Doesn't Samsung like buy a lot of parts and they aren't they a huge customer?
Bryan Cantrill:So yeah, but Samsung wants a dual source absolutely everything all the time. They're constantly pitting their suppliers against one another. And the thing that was kind of the moment is like, you think that you'd like, oh, that makes sense like dual source everything. Like, that's the way to get the best thing is to have everybody fight all the time. And it's like, no, everybody hates your guts.
Bryan Cantrill:And what it means is that you if you need anything, nobody cares because you've shown that you don't care about the relationship. So they don't care about the relationship. And you know, Kirsten, you you know, when I it was really interesting to hear, have you come aboard? You're You're coming from Lenovo and you know, I I CJ, I know that you had this is position to come from Apple because I kind of assumed that like, oh, well, you know, these folks are coming from, know, other companies and then maybe like this is just a Samsung thing. But you're like, no, I love the fact that I can actually build relationship here because when you are constantly pitting suppliers against one another, one you're saying like, don't actually care about your differentiator.
Bryan Cantrill:And the like, we've always said like, no, we wanna pick the right part and then really go deep with you. And then like, yeah, we we do expect that you're gonna give us competitive pricing, but you should know that like we're all in on you. So we're not you're giving us competitive pricing because you want us to succeed as a business and not because like I I don't need to to threaten you to do that. And in fact, we think that it's it's much more. And so it's, you know, it's always interesting to me when I, you know, hearing it that when you're, you know, boots on ground describe I boy, I feel we've seen that like over and over and over and over again and the magic that you all were able to pull off during the supply crunch.
Bryan Cantrill:And then as you say, like, with the AI boom and us needing to fight for parts, it's really important to have those relationships.
Kirstin Neira:Yeah. It definitely is. And one of the things that I I communicate with my suppliers, and we've got really great relationships. You know, I would say we're friends outside of just the work the work atmosphere, is that I I don't have the time nor the desire to sit there and negotiate back and forth. Like, I just expect you give me your best price first go around.
Kirstin Neira:I'm not gonna go pass your price to somebody else. I'm not I don't play that game. And I know that Right. Maybe some other companies do is I just let's trust each other, and let's have an actual, like, working relationship, open, transparency. It's it's not one of my favorite words, but but it's true.
Kirstin Neira:It's I don't have the time to haggle back and forth. I like you guys. I wanna use you. We'd love your product. You give us great support.
Kirstin Neira:Let's just give us a price.
Aaron Hartwig:Yeah.
Kirstin Neira:And that's different. We still have some vendors that'll come back with a form to fill out and they're like, who's the competitor? Who are we going up against? What's their price? What's your target price?
Kirstin Neira:I'm like, my target price is zero. Like, I'd love it for free. But, yeah, I think, you know, the relationships we have, have been great to date. It's been really great working with our suppliers and they really are oxide fans, which is nice. And I think we are probably a bit of fresh air for them as well.
Bryan Cantrill:Totally. Yes.
Kirstin Neira:Of those other companies.
Bryan Cantrill:Yeah. And I just think that, like, they are relieved. They're like, oh my god. Like, you actually want to build a relationship with us. And and then I think that, you know, and and I think we we talked about this in the past, but you know, you pulled off a a miracle with Senor Denki when they had a supply chain issue.
Bryan Cantrill:They let you get into their bomb, figure out the part that they were actually corked on, which is if I recall correctly was a TI FET because it was like, it's always a TI FET at the end of the day. Feel like the Yeah.
Kirstin Neira:Yeah. It was I remember that.
Bryan Cantrill:And you then like, look, I can go find that fat elsewhere and broker it to you, but I need you to support it under warranty. And the and because if I'm remembering that story correctly, they're like, yeah, we were willing to do that. We have never done that before. Like, this is the we have never even contemplated that before, but they were willing to do it for us.
Kirstin Neira:And that was watch. Yeah. I know. Well, we had issues with Sander Denke and Murata. And I know that I think it might've been Murata that we ended up getting finally a list of, like, 28 parts that they couldn't find.
Kirstin Neira:And I just kept finding them and sending them to them. And some we had to buy, some they could buy. And I was like, listen. And they're like, well, you know, I have to buy, a thousand. We only needed, 200.
Kirstin Neira:I'm like, use them for your other customers, but, know, just make sure they know where they came from. Nice. So we were helping out their other customers too because Totally. Yeah. Not.
Kirstin Neira:But
Bryan Cantrill:yeah. Well, that is so collaborative. I mean, I think it's a I mean, obviously, you win the you know, you boy, you build a relationship with you know, when you're solving this other problem I've got. I've got another customer that's breathing down my neck and now you're able to that is just extraordinary. And really, I mean, speaks of the relationship.
Bryan Cantrill:The other thing is like, I think it's really important is the relationship between procurement and engineering. Because I think one of the ask us like, how do you like this like power observability that you have in in in the machine. Why why don't I have that everywhere? And it's like, because there you get these part swaps that happen and the different organizations aren't connected and people are like well that's like this other part is like a little bit cheaper and easier to get and it looks like it's the same so I'm gonna swap it in. And I think that you I I love the collaboration that we've got between engineering and procurement, which sometimes means that CJ, I do recall you asking once about the TPS 546B24A, the the the beloved DC DC giver.
Bryan Cantrill:I love this part, Adam. Have looked at this part at all? Mhmm. No. Okay.
Bryan Cantrill:This is a TI regulator that is just like a this thing is a little champ. It's got like a 1,200 page data sheet. I may be exaggerating, but not by much.
Adam Leventhal:Don't let the other regulators know that you feel this way, though.
Bryan Cantrill:I want the know, I actually wanna hold this up. I want the other regulators to see it. I'm gonna hold the hands of this regulator in public. No. We are we are we're we're You're
Adam Leventhal:going steady.
Bryan Cantrill:Public or whatever. Exactly. We're going steady. We're steady. Yeah.
Bryan Cantrill:The but the no. The this is a little regulator that does its job. We use it everywhere. It's got really rich PMBus support, which is the reason that that data sheet is so long because like the PMBus support is good. It is it's a part that we really trust.
Bryan Cantrill:And so at some point, CJ was like, ah, these are really hard to get. How about this other one? And CJ, can't even remember what that suggestion did not last very long. I feel it was like I kinda had to dig you out of the bottom of a dog pile that maybe I was on on terms of the you're like, okay, I get it. Like everybody loves this regulator.
Bryan Cantrill:I've never okay. I will just go fine. I'll just go make some. I don't know. I'll go find some.
CJ Mendes:Well, Kirsten Kirsten probably found it.
Bryan Cantrill:Yeah. The
CJ Mendes:only thing I would add is like Kirsten's done an amazing job, again, kind of back to the low rate production to ramp. It's really hard to build the relationship you need when you're ramping while you're ramping. And so very grateful for all the work that Kirsten's done in procurement with building these relationships. Eric also does a fair bit of procurement work with sheet metal and circuit board suppliers that have come in very, very handy in our moment of need here. Just how we started, not just what we're doing, but how we're doing it, I think is really paying dividends for us now.
CJ Mendes:Really, really great job for those folks.
Bryan Cantrill:Totally. Also, got to think, Kirsten, I'm sure you have felt this too. Eric, know you felt the CJ, the the terms of like we have these advocates at these partners and we really appreciate them, but like we're also like little and weird. And for us to like really have explosive growth, it just feels good to like go back to them and be like, oh my gosh, like you guys are actually gonna make it. I don't know.
Bryan Cantrill:I I think it's really
CJ Mendes:fun. Very satisfying. Very satisfying all around.
Kirstin Neira:Yeah. It it has been very satisfying. They're very excited for me. Yeah. It it's been a fun job.
Bryan Cantrill:That is awesome. That is awesome. And so, know, the one question is that the I mean, there are a bunch of disadvantages to being low volume, but there's some advantages too in that, like, we're just not looking for as many parts. And we can you know, when we get to higher volume Kirsten, what have been some of your the the the challenges at those higher volumes? Because the the numbers get they're really big.
Bryan Cantrill:I also got to say, you all have got guts because like you gotta like walk in a price and these are not small first orders. These are like kind of stomach churning purchase orders and you gotta be like, nope, like we go now. Like I know that this price is only going up. I know this isn't the best price. We've gotta commit.
Bryan Cantrill:I mean, I just really admire that, like, knowing when you got a good price. Yeah. Totally.
Kirstin Neira:I mean, you know, I, you know, I I put a PO together. It the largest PO I ever put together. And I I sat on it for about an hour after I did it because I'm like, I just I don't I can't send this yet. Like, I'm sending this to Steve and asking to sign a lot of money away. I mean, growth is great, but when you're all of a sudden looking for 100, 200, 900,000 of this one part number, It does get a little challenging to go find all that on a whim.
Kirstin Neira:We're also, as we scale and demand increases, there were parts that we looked at back in 2022, 2023. We knew they were going end of life. Thought we had enough, right, until we got to a board spin, and they're just popping up in the last couple of weeks that, oh, that supply that we bought extra of in 2023 to take us all the way through 2026, we're gonna run out of in couple months. Yeah. And that would end of life three years ago.
Kirstin Neira:So thanks to the engineering team, a couple of you are on this call. They've gotten a lot of requests in GitHub in the last week or so. We go, you know, we have to find drop in replacements because we've got Cosmo boards, you know, out at the manufacturer right now about to start building. So there's definitely been some pains along with it.
Bryan Cantrill:Aaron, are we looking at the ADM 12 the ADM twelve seventy two when we is that is that what this is all is is what this is about?
Aaron Hartwig:It's still fighting. It is not that fighting.
Bryan Cantrill:Okay. Good. Because the we've had some I some of these things have been surprise EOLs a little bit where we've had some EOLs have gone and I know with the with the $12.72 apparently that's we're hoping to get a that thing's banging on the coffin lid, I think. Hopefully, we're gonna get a revival there. That's another regulator that I'm less fond of, Adam.
Kirstin Neira:We had another one come up that was one of the LEDs. So, you know, we've got, I don't know, thousand of stock, but we need more. And it's like, you don't wanna you don't wanna change an LED, like, mid production. They they all need to look alike. So some small things like that.
Kirstin Neira:We'll get through it. But yeah. I mean, good growing pains, but just some things are popping up that maybe weren't on the horizon right now.
Bryan Cantrill:Well, I I just think and and as Eric is saying in the chat, there's a lot of kinda natural empathy from ensuring for operations. And I just think it's been very collaborative. Like when you've had one of these problems, Kirsten, it's like, okay, let's go brainstorm, you know, the what can we find? What are some alternatives here? What can we how can we, put some things on the we just don't view it as we wanna get creative about how we can solve these problems.
Bryan Cantrill:But it does feel like there's new problems every day.
Kirstin Neira:There are. Yeah. But and one of the other good things that we've done is, we probably did this actually a couple years ago is, you know, when the when the engineers are adding a new part or changing a part, that I get notified right before the ECO goes out so I can make sure that I, you know, pay you guys. So it's like no supplies available. Okay.
Kirstin Neira:This is fine. I approve it, and then I can go procure ahead of time to make sure that we have the supply when that ACO rolls out. So there've been some really good collaboration and just some changes to the process to improve things all around over the past couple of years.
Bryan Cantrill:Yeah. That's
Aaron Hartwig:that's part of, like, the the engineering culture to try to build to is you can't just pick whatever you want and and design it in. It also has to be procurable at whatever scale that thing needs to be built at. And sometimes, like, ability to mark something as problematic for procurement in, you know, whatever database, part database we're using is really critical.
Bryan Cantrill:Yeah. And we also I mean, it's also helped as we have like, now, we've got experience with these various parts, and it is helpful to like we're you know, if we can use some of our existing I mean, this what we kind of talked about earlier in terms of design stability. We we we're we're not gonna just change things to for the purpose of changing them. If we've got a part that we know works, we've got a regulator that we like or a service processor that we like, we're gonna really weigh any change there very seriously. And certainly procurement is gonna be part of that part of that calculus.
CJ Mendes:But Brian, the other thing that we're not really talking about, maybe we shouldn't talk about, but I'll just say it, is heroic effort by Kirsty. Because we're doing this, we have some tooling, but it's a lot of spreadsheets, right? Totally. Eric and I talk with this all the time, it's not going to be CPUs that stops us in our tracks, right? It's going to be a fastener or something.
CJ Mendes:And so we're in Twitch town trying to just double check, triple check the entire bill of materials and make sure we have coverage. And Kirsten lives this every day of just making sure that everything that we need to order is on order and getting delivered on time. And it's a ton of manual effort. And so we're certainly looking to make investments there to lighten that load, but it's very startup y, I would say, in terms of how we've been kind of planning all this to date. And again, amazed that we're as well positioned as we are to go scale but plenty of work to do on that front too to help Kirsten out with some of the, what do you need to focus on today and how much and how many, those kinds of things.
Bryan Cantrill:Totally. And and of course like the the the kind of the body of software that you're referring to but but not naming is is ERP.
CJ Mendes:Yes.
Bryan Cantrill:And the and I think you know a startup would be kind of foolish to do a big complicated ERP build out before they have a product market fit. Then you're kind of like I mean, you like, you're you can only deploy an ERP system, like, too early or too late, I guess. I
CJ Mendes:mean, look, for all its flaws, like if Kirsten needed to order a part right now, she could cut a PO in two minutes. And so like Right. Can you do that in an ERP? It's not it's not always so easy. So, I I think, you know, it's served us well but it's time to make some investments there and we will.
CJ Mendes:Yeah, just heroic levels of attention to detail and I don't know, small dose of paranoia I think to make sure that we've got all this covered down on.
Bryan Cantrill:Totally. Yeah. ERP is enterprise resource planning and this is kind of software that is, I would say, infamous in terms of asking businesses to change for the software software is kind of my perception, CJ, of of ERP that the where you have this is very enterprise software. I mean, guess it says it right there in the tin. I guess you should really be like any enterprise software that has enterprise in its name, you know, is like extra enterprise But these are companies like SAP.
Bryan Cantrill:And you know, I think that we also want to be kind of careful about how we we move into that. Certainly with PLM software, you know, disposition has been like that the you need so much customization. And, you know, I do think that, like, we're in a world where it's a lot easier to write one's own software. And I'm not saying we're gonna write our own ERP, but, you know, I guess yeah.
CJ Mendes:Thank God. Thank God
Bryan Cantrill:you guys. But you're thinking it. Yeah. The you know, what's someone else say?
CJ Mendes:Yeah. No, you're absolutely right. I mean, the good news is is you know, Todd Welding on our team is an ERP expert and has used them all and just like Oxide does with everything else, they'll be an RFP you know, with the things that we care about and we'll be careful about the selection process but.
Bryan Cantrill:Well, and the conversion. And so Tom Lyon is in the chat making reference to something that was a that was recent when I mean, it was actually wasn't recent. Was in 1989, I think, Tom. Right? But when no.
Bryan Cantrill:When I arrived in '96, like this was still like legendary. So there was an ERP deployment at Sun that broke Sun. And they couldn't ship. And and this is like when ERP deployments go wrong, it's like customers are I mean, it is bad. It is This was like
Adam Leventhal:they had no visibility into the quarter, like didn't know how much business had been booked, couldn't book new business. It was a a disaster at Wall Street. Right?
Bryan Cantrill:A disaster. It's a disaster. And obviously, like, it's not there are, lots of blame on that one. That's not the like, the software didn't didn't make the decision to deploy itself poorly.
CJ Mendes:You know, it's hard to find is a story of an ERP implementation that went well. It's actually Interesting.
Aaron Hartwig:It's it's all
CJ Mendes:nightmare stories. So, yeah, we'll certainly be very careful. But
Bryan Cantrill:Yeah. And it it is and we'll be, you know, and we will presumably also, take advantage of the fact that we do have like different muscle at oxide than other companies might have in terms of like there are certain software problems that we're just not afraid of and not that we would we just are gonna be, you know, we're gonna go into it I think with maybe a different perspective. But oh, yeah, Adam, your the the ERP thing is in High Noon. Think, yeah, with the because it was definitely infamous.
Aaron Hartwig:Yeah. Yeah. I'm pretty sure it
Adam Leventhal:was I I read about it in High Noon, but I can't check because I gave you my copy.
Bryan Cantrill:You know, and I did not re gift it. I I would like you to know.
Doug Wibben:There you go.
Bryan Cantrill:That's why I don't give away my books. You know, I I I don't mean to punish you for having given your book to me. I feel that this was although, you know, I say that with an asterisk because I gave my copy
CJ Mendes:of
Bryan Cantrill:Hypergrowth. My copy of Hypergrowth has definitely gone around like like the clap at oxide. PJ, I
Aaron Hartwig:The clap has not
Bryan Cantrill:got around at oxide.
Adam Leventhal:I just wanna just just just for the record.
Bryan Cantrill:I get yeah. You know, I yeah. Yeah. That's kind of I guess that's that's oh, there we go. Yes.
Bryan Cantrill:Steve. Steve's in the chat knowing knowing where the copy of hyper growth has stopped. That has not made has not been did you read hyper growth Adam? That is it is really No. I haven't read it.
Bryan Cantrill:So Waiting
Adam Leventhal:for Steve to finish with it. No.
Bryan Cantrill:I'm just kidding. I know we all are. Let's we don't go out of business because of an operational mishap. Well, see well, the thing is that is in and obviously with hyper growth, you are very much getting like the one perspective of and you're you're getting Adam Osborne's perspective. But there are companies that get into a lot of trouble and I guess we talked about it with Sun in this ERP transition because of operational mishaps.
Bryan Cantrill:Like operational mishaps, I would say it is less likely to damage a company than not having product market fit, which I think is overwhelmingly. I mean, Adam, feel free to disagree.
Aaron Hartwig:It feels like
Adam Leventhal:Feels like yeah. I mean, no matter how operationally savvy you are, like having the wrong products in the wrong market seems like an impediment.
Bryan Cantrill:It it seems like an impediment. And but you there are counter examples where companies have product market fit and then they make operational mistakes. And and obviously, CJ, that keeps you and team I think I think it's also just in general like again high stakes. And Kirsten, I love you. You're like, you know, you're sitting on the PO for like an hour.
Bryan Cantrill:I would be sitting on that. I mean, I would I I I'm just amazed that you can like that that an hour is enough because it is just nerve wracking. You know and I I admire because you get in and I I've seen I mean you all there also like this is where you get to kind of the craft of it where I remember like a couple of years ago when maybe a year and a half ago, Kirsten, where we're like, we believe that DRAM prices are gonna go up. We believe that we are in a lull for DRAM and we need to back up the truck and buy more DRAM than we need. And you know, there's a little bit of like, you're you're kind of making a trade there.
Bryan Cantrill:You're kind of like day trading DRAM, but that was indisputably. Yeah. We actually
Kirstin Neira:make a lot of money off that DRAM right now, actually.
Bryan Cantrill:Yeah. That's right. Yeah.
Kirstin Neira:I mean That was act yeah. That was we actually ended up with almost twice what we needed at the time just because miscommunication, lack of communication between a disty and the actual manufacturer. They went ahead and shipped an order that we had asked to have canceled.
Bryan Cantrill:Oh, that's funny. I didn't know that.
Kirstin Neira:But, yeah.
Bryan Cantrill:Thank you, Erin, in your favor.
Kirstin Neira:It ended up in our favor. Yeah. That DRAM yeah. I won't get into, like, what we bought it for, but, it's selling for about $500 each right now. DDR four sixty four gigs.
Kirstin Neira:So, yeah, that one worked well in our favor, but there there's been a lot where it's like, yeah, we should look. We really, we probably have like a week. Then it's like, no, like Steve, you have to sign this PO today. We have to get this supply. We have to lock it up.
Bryan Cantrill:Well, I love have you have you ever been, Adam, in the CJ's chain of command for a a PO where it's like, okay. Steve has to sign this PO tonight. And, like, the, like, the plan is, like, if Steve is like unavailable, there's CJ has like a whole like chain of Yeah. Exactly. And
CJ Mendes:we got matrix and then straight to the phones.
Aaron Hartwig:That's right.
Bryan Cantrill:And next thing you know, like Al Haig is signing a PO. That's a great great one.
Adam Leventhal:I love it.
Bryan Cantrill:You know, give the kids something to go. Go go ask your go ask ChatGPT. ChatGPT is what I'm talking about. ChatGPT got the reference. Yeah.
Bryan Cantrill:You millennials. You know what I love about that is that Elle Haig said that that would be the fourth sentence in his obituary. And the New York Times made his quote about that, the fourth sentence obituary. It was very meta. I I just really the yeah.
Bryan Cantrill:I'm not explaining LHAG to you. I'm sorry. I'm just there's certain things just kinda go Google. So the so the procurement has been an adventure and it's gonna be an exciting adventure. Now we've got a like a lot we need to go do there.
Bryan Cantrill:You know, as we we kinda get into new markets, because people are like, when are gonna sell in Europe? Or you know, when are you gonna there are like lots of questions like that. Eric, when are we gonna sell in Europe?
CJ Mendes:I'm not sure if Eric's I don't know, maybe been disconnected, but So basically with selling to Europe, there's more compliance things to go after both on the technical electromagnetic type safety certifications, and then also some environmental like REACH and ROHAS. And so the EMC type stuff is we go back to the lab, the the national recognized testing laboratory, which we use Intertech. So we go back to the lab and do a bunch more testing. Region ROHAS is mostly a bill of materials and procurement exercise to make sure that all the components you're procuring have those certifications or for all the custom items that we have, like sheet metal and stuff like that, that they conform to those environmental standards. And so Kirsten has done a ton of work in this space, again, on procurement, part of scale, getting ready to go into new markets to go get that data point from all these suppliers and all of these components that we have on the bill of materials.
CJ Mendes:Now, most of these are pretty common for regionals. Think we found one resistor out of countless that had lead in it. And so we've got that in the queue to get that replaced.
Kirstin Neira:Oh, yeah. That was the only one.
CJ Mendes:Yeah. The only one out of
Kirstin Neira:And and and the easy drop in alternate. Like, it it's not anything that had to be changed. Be drilling. It was like, oh, they they have a lead free version. Here.
Kirstin Neira:Just use us.
Bryan Cantrill:Oh, okay. You know, I got this question once. I did not know how to answer it about what about the leaded components. The that's interesting. So we only have one that's amazing.
CJ Mendes:Is that is that super? Yeah. And I think it reflects the experience of our engineering team. Think, you know, there were a lot of things that they were looking out for, right? When we were
Bryan Cantrill:Yeah. Interesting.
CJ Mendes:Putting together the bill of materials. So but still amazing that there was, you know, one. I mean We were racing far worse.
Bryan Cantrill:So I was never gonna lick the board to begin with. It's just it's just helpful to know that if I were to lick the board, I would not be it would not be led that I'd be getting
Kirstin Neira:the Yeah. I was gonna mention, one of the things one of the things with with the reach that I've learned is that reach changes every six months. So they're always adding new components or reviewing the component list for substances you can't have or can be limited. So we're going through the exercise to get the REACH and REACH documentation for every single part that we've got on our bill of materials. And it it took several months.
Kirstin Neira:And during that time, like, was a new reach directive that had come out and it's like, oh, are you kidding me? Like, we really have to go back.
Bryan Cantrill:Are we gonna have to engineer the fentanyl out of the board? Because that's gonna be really
Kirstin Neira:We might have to. Yeah. A that's
Bryan Cantrill:gonna be a challenge. I did I did did drop a link to the chat, by way. I I encountered this over the weekend that I wanted to ask you about because the in terms of like our concerns about regulatory compliance, there was this admittedly in the $5 electronic pile at Target a as a a Bluetooth speaker for $5, which feels like, what god. What corners do you cut there? Apparently, you cut corners around the the FCC compliance.
Bryan Cantrill:So the the the or at least the the it was this rating was not tested under the FTC standard. And I wasn't sure, like, are you allowed to do that? Are you maybe this is maybe we can get I wouldn't get into new markets. We can just say, no, we just didn't do it.
CJ Mendes:Certainly not the approach we're taking.
Bryan Cantrill:That's not the approach I it's
CJ Mendes:But you know, and then there's you know, the other thing about exporting new markets is the trade compliance piece which we are figuring out. You know, we we're not ITAR, so we're not gonna have to worry about all of those compliance requirements, but we are gonna follow-up under the export administration regulations either due to some of the technology that's embodied in our product. So
Bryan Cantrill:Oh, interesting.
CJ Mendes:Work to do there. Yeah. A lot of particularly around networking, a lot of those ICs that we use have export controls on. Yeah. Interesting.
CJ Mendes:Pretty much every QSFP module is export controlled. So work to do there to get through the appropriate licenses. But yeah, fun new things to learn in service of growing the business.
Bryan Cantrill:Well, and then as long as we're on that, just because it is a question that people have asked me for sure is, you know because I think it's it's also the kind of the thing that people see is like, wow, the the tariffs must be really awful for you and like they're obviously not great. And in particular because things keep changing. It's very hard when things are changing. I think it's hard when anything is changing. I do I am like, you know, that there are so many other challenges.
Bryan Cantrill:It's not like they're not a like they definitely are a challenge, but I think there are lots of other challenges and then for lots of the things that we're buying, it's like they often already I mean, many of the components we're consuming from Taiwan, for example, fall under existing trade agreements. So when that has not been I I I I guess I should knock on wood when I say this, but it's not been absolutely brutal for us. It's just been kind of yet another thing we've had to go deal with.
CJ Mendes:Yeah. I think when this all started with all the uncertainty, Kirsten and I and Eric got with our suppliers and it's really just about mapping. There's a thing called the harmonized tariff schedule, the HTS code, which is going to dictate that when that thing shows up and gets received and what tariff is applied. And so there was a scramble for data gathering both on what is the HTS code or what is the HTS for this component and then what countries is it sourced from? Because sometimes there are country specific interpretations of those HTS codes.
CJ Mendes:But yeah, I think we at least have the data mapped for the top 90% of the bill of materials for cost. And I haven't checked Twitter, but that could change tonight. But at least we know now how things are are being processed through.
Bryan Cantrill:Well, and the the exposure is always kind of is like it's kind of funny. Like, we've got more exposure, I think, Malaysia than we've got to I mean, which is like, oh, okay. That's good. Like, let's key?
CJ Mendes:Yes. We source globally. I mean, we talked about Sengo Denki, those come from The Philippines. Source all over the place and there's really not any getting around that in this industry. Even if a supplier wanted to stand up a factory in another country to optimize for tariffs that's measured in years.
Bryan Cantrill:It's certainly like, these parts start in Taiwan as I mean, we've got, I mean clearly like TSMC fabbed parts so.
CJ Mendes:So it's it's definitely been a bonus bonus challenge for us to go work through.
Bryan Cantrill:It's a bonus challenge. Yeah.
CJ Mendes:But I think we're we're better for it I guess.
Bryan Cantrill:Well and I would say mean, correct me if I'm wrong, but my impression is that the challenges of scale are far greater, far far far greater than these kind of external the external challenges are like our challenges. We've gotta go knock them down. But they I feel feels like the challenges of scale but much much much greater.
CJ Mendes:For sure. Yeah. I think, know, we've we've got that stuff quantified and you know, like we talked about earlier at the top, we've been laying the foundation here for a little while but, you know, there's still plenty plenty of work to go execute on to to make it a reality.
Bryan Cantrill:And so I think people are always surprised about how teensy tiny our team I mean, the whole team is, you know, I mean, I think we've maybe we just crossed 80, but we've got like, you know, all of oxide is actually quite I actually really appreciate it. When we announced our series b, people are like, you guys have built everything you've built on $89,000,000. Like, yeah, that's so we've been we've really made things go far. But now that we're kind of in this new era of scale, I know we're really we're scaling across the team, but really focused on operations and the engineering required for manufacturing. Do you wanna describe a little bit, CJ, about kind of what
CJ Mendes:Yeah. I think we've been very well served with the group of folks that joined us here at Oxide with like a lot of domain expertise and versatility, folks that really understand all aspects of supply chain, kind of worked in various functions in supply chain and understanding how all that ties together. And then we were actually very fortunate on the engineering side, on the hardware team to have kind of supply chain savvy folks who are kind of at the top, really thinking through the implications of the parts we choose. And so I would say those two things are going remain critical. We want to hire for that domain expertise, but also folks who can be versatile and either go deep on something new as Kirsten and Eric and I have been doing for the last four years but really bring a core expertise to the table and then can can help wear another hat as needed.
CJ Mendes:It it
Bryan Cantrill:should be said that we are that we are using the oxide hiring process for these roles as well, which I I mean, you know, our hiring process is is like it's it I mean, it's it's normal nowhere. So there's no domain in which it's not idiosyncratic. And then I think that like there but then there are domains where you're just like, what? Okay. And I I mean, Kirsten, remember I remember vividly where I was when reading your first material, when your materials when you first applied to oxide.
Bryan Cantrill:I mean, is a this is an unusual hiring process by any standards, but certainly in operations. But CJ, we we still ask the same questions, same questionnaire. We look for a different work sample though. Can you describe a little bit kinda what goes into the into the work sample for some of these roles?
CJ Mendes:Yeah. I think as as we grow the team, we're looking to hire, you know, more focused roles a little And so what's actually been challenging is like what questions do we wanna ask in like the work sample or the analysis sample that's gonna help us identify candidates that have what we're looking for. And you have to really be thoughtful there in terms of what kind of work sample or whatever that you ask for. Because we're getting all these applicants and we're reviewing them and to sort of triage the candidate pool. You know, those questions for role specifically, the ones that are specific to that role are critical to get right in terms of asking the right question in the first place.
CJ Mendes:And then, you you have the standout applicants that really just nail it, which we've been very fortunate to have a lot of good candidates applying.
Bryan Cantrill:Yeah. And I would say to anyone applying to Oxide just because we are expanding a bunch and we're spending a lot of time reviewing materials It for all of our roles, definitely in operations and engineering, but across the company, like please read the job description very carefully and please read the materials very carefully. I I CJ, gotta say one of the things I really love about this latest crop materials is on the prompts. You really are very explicit about what we're looking for. I kind of like what our rubric is as we are what we're reading for.
Bryan Cantrill:Because we're, you know, we're not trying to like we're not we're we're not trying to play gotcha. We're we're we're trying to actually get terrific people to explain what they've done and why they want more peroxide.
CJ Mendes:Yeah. I hope so. I think it's all he'd pretty much take those questions and track back to something that we're working very specifically on right now. Yeah. And in a hurry.
CJ Mendes:So, but yeah, you know, there was a question in the chat about folks whose work is largely private or proprietary, we encounter that quite a bit. And what we found is that most candidates are able to describe the work adequately, even if they don't get into the specifics of like company names or dollar values or what have you. You may not be able to share the the planning workbook you used to plan, you know, whatever supply chain, but describing it in a thoughtful way and adding appropriate context, we've got folks that have submitted great materials, you know, even though some of their prior work was proprietary.
Erik Anderson:No. One of the just more broadly, that we're looking for is it'd be silly to say that there's anyone that fits every single box or rubric that you were talking about there, Brian. One of of the things that we're looking at, especially in our company, is where a solution doesn't exist. What do you do where there's not a system or a thing that tells you what to do? A lot of you know, like, CJ has worked at, you know, Apple and Pearson at Lenovo.
Erik Anderson:At some of larger companies, you have the benefit of a system or a process that exists. We have the benefit of working with companies that have processes, but broadly, they look to us to tell us how to do things, how to build. So I think what's really important is being able to have the, I guess, the humility to ask questions. But where something doesn't exist, you look to your left and you look to your right and say, it doesn't exist, you need to build it. Whether it's like a test fixture, for example, or a test itself on the software side or whether it's documentation or something to tell them how to build build it the best so we can build that quality into our product at an early stage as possible.
Erik Anderson:So I think that's what we're looking for. Initially when we looked at this, one of our roles that we're advertising for the manufacturing project manager role, initially it was aligned a little bit more closely to like a manufacturing engineer. And while that's a very important skill set and something that is important to the job, it is not the only thing that you need that person to be able to tackle. I think what's really important is being able to understand that there's, like we've talked about, the person in this role would have to be, like we use the term athlete quite a bit, but obviously have a strong manufacturing engineer skill set, but also someone that can reach out to and have the ability to speak to how something is scheduled on the floor, help to provide some insight on our manufacturing test fixtures, work with our engineering team, work with our software team. So there's not one person that is able to do that maybe out of the box, but someone that's willing to recognize where those gaps are and pull the right team together so we can be successful.
Bryan Cantrill:And yeah, and this is a challenge because I think as you say, Eric, you know, for a lot of folks that are engaged in manufacturing or or spot chain management procurement at a large company, you're part of this large system that doesn't change very much. And we while that that experience is very valuable, we really need folks that can be really I mean, CJ, you spoke to the the the need for versatility and just being able to be I mean, I just think the number of times we've been just creative about finding solutions to problems. And really, you know, we also you all do a very good job of not really taking no for an answer in a very polite way of like, okay, like let me understand, you know, Kirsten, this is what you're describing out like with the Marauder problem. Like, okay, so you say I can't have this for nineteen weeks. Let's understand why.
Bryan Cantrill:Let's get into there. Let's let's actually let's work kind of collaboratively, figure out how we can solve this problem. Work collaboratively with our contract manufacturer solve this problem. Think Eric, one of things for that role in particular, and I I think we are broadly a remote company or we are a remote company. There are a handful of roles where we actually we don't need you to be on-site in Emeryville, but on-site actually at our contract manufacturers is extremely helpful.
Bryan Cantrill:So in this case it's lovely Rochester, Minnesota. But the I think as we expand, we can probably expect a couple more of those. So, you know, we will be fortunately and it's CJ. I'm amazed at how much we're able to. I think there's there is a bit of a of a myth I think that you can't do hard tech as a remote company.
Bryan Cantrill:Think we've proved that you can.
CJ Mendes:Here we are.
Bryan Cantrill:Here we are. Exactly. That's right. We lived or we're alive anyway. But the there are some roles that that do actually need to be boots on ground.
Bryan Cantrill:And Eric, we've really appreciated you being your physical presence. You are not based out of Rochester. You are you are a drive away from Rochester, but your physical presence has been absolutely clutch over and over and over again to help us navigate problems by actually being there on the ground with folks.
Erik Anderson:Yeah, I think that's been a delicate balance. I think we've been in a low rate production, but we've been in some state of a new product introduction NPI for roughly two to four years, depending on where you draw that line. And I think it's really important to understand, one is building relationships at the manufacturer, but also we don't have everything built real time. A lot of the times we'll have to deploy something, then work through that process with our team. So I am a little bit more local, but we also have our Wisconsin contingent, which has been huge help as we've tried to deploy.
Erik Anderson:So if we have short notice, things that pop up where we need to either do some troubleshooting or work through a process, they're there and available. And it's building that rapport. I mean, we're going back to some of the conversation I think we had maybe two or three years ago when we last had our first customer ship. I think that's been absolutely invaluable. So in a world of remote work, sometimes you just can't displace or replace, I'm sorry, being on-site and just seeing something happen real time and answering those questions, especially when you're trying to move fast.
Erik Anderson:So even though we've been in a state of low rate production, we've still been throwing a lot at our contract manufacturer. And so being there to work through those problems real time is extremely helpful. So then you can bring back those issues to our engineering team and we can improve on that feedback loop much better than we could over email or even a video call because it's hard to bring your computer out and hang it vertically over a sled and say, hey, here's the problem we're having and here are the issues we're having. You just need to touch and feel and do that needful work.
Bryan Cantrill:Or or over an ocean, Eric. Mean, the the think this is like one of the the I mean, there are many reasons why we have a domestic manufacturer, but first among them honestly is the speed of that feedback loop and us being able to be on-site and being able to just quickly resolve problems. And Kirsten, I remember vividly when we had a really acute problem with a manufacturer that was an ocean away and it was brutal in all the ways that these things are like kind of comically brutal where you are taking these long laps of like you can only kind of have one conversation a day because of the time zone overlap and it was just it's tough. There's there's a lot to be said to just being that that time zone locality. But this has been terrific.
Bryan Cantrill:I think that, you know, we have got we're we're and all of you, I know CJ, you were you were praising the team, but all of you have done such an extraordinary job. I think the collaboration among the team has been essential for this. And I think the fact that, know, people have felt that they can always go solve whatever problem we have in front of us. Right? People don't need to worry about like well you know, okay.
Bryan Cantrill:Our job collectively is to go make this whole thing fly. And it's been really exciting to see that we do not get siloed and to that kind of cross team collaboration. But then also Kirsten, the thing you're mentioning about the relationships with our suppliers and treating those as real partnerships has been huge. You dropped RFD 60 8 into the chat, but that has just been, that's been really important for us and really grateful for those suppliers who've gone to bat for us. And it's great that we can actually, you know, make good on their belief in weird old oxide.
Bryan Cantrill:We can actually actually pull this thing off. Well, again, congratulations everyone. Really exciting stuff. A lot of excitement in the future. CJ, a lot of problems ahead for sure.
Bryan Cantrill:A lot of exciting problems ahead.
CJ Mendes:Good problems to solve.
Bryan Cantrill:Good problems to solve. Yeah. New problems as Aaron says, I can't wait to see the new the new problems. But we've, it's been learned a lot and we're we, we, all of this, the timing is perfect. Like we are, we are flexing this thing up at exactly the right moment.
Bryan Cantrill:We're ready for it. So ready for those new challenges. Thank you everyone. I think we're, I'm gonna set it. I'm gonna put out a teaser, Adam.
Bryan Cantrill:We just had the whole company out last week. It was amazing. The baller scheme was great. Wasn't that fun? Super fun.
Kirstin Neira:Yeah.
Adam Leventhal:And they're in the next round.
Bryan Cantrill:They're in the next round. I was I was I was actually visiting my my older guy in college with my daughter and we were listening to it on the radio and just it was the the the call as they won one zero. I don't know if you saw that. No, didn't say. Oh, no.
Bryan Cantrill:That was a that was a one zero victory with a with a Chabot product on the bump. Yeah, no, it was it was really, really fun. Gabe Tanner, nice going, fired up, pitched eight, very nice. So we got a my plug one is for the Bowers Championship is coming up. Tickets are going fast Friday night for will be game three of the championship series.
Bryan Cantrill:But I think I wanted to tease Adam was not a baseball game. Although you guess you wouldn't know it from what I just did, but the we saw some great talks at OxCon in particular, our colleague, Dave Pacheco gave an absolute banger of a talk on what he on just on the journey of software update. We're going to have to turn We got to have him on. We gotta get that thing out there somehow. Don't you think?
Bryan Cantrill:And
Adam Leventhal:Sure. Have you told Dave that?
Bryan Cantrill:You know, I mean Or
Adam Leventhal:is this you telling Dave?
Bryan Cantrill:Dave's a listener to the podcast. Right? I mean, I think we we we've established that this is the way to find out. No. I've no.
Bryan Cantrill:Of course. Definitely told Dave that. But the well, I've told Dave some of that anyway. I've I feel I've told Dave much of that. I've told Dave much of that and what I didn't tell, he could reasonably infer and that he didn't reasonably infer, he now knows.
Bryan Cantrill:So there we go. Yeah. It's all out there now.
Adam Leventhal:Good. I will hustle on to get this recording today as soon as possible.
Bryan Cantrill:Yeah. And I'm loud and proud of the TPS 546B20 546A24B. I'm I'm I'm out there. I'm holding hands.
Adam Leventhal:Good. Good. Good.
Bryan Cantrill:With with my favorite regulator. I'm just like, I'm I'm proud out here, man. Good. Good stuff. Alright.
Bryan Cantrill:Well, so that'll be added to future podcast episode, but looking forward to that.
Adam Leventhal:Should be fun.
Bryan Cantrill:And go Bowers.
Adam Leventhal:Yeah. Go Bowers.
Bryan Cantrill:Alright. Thanks everybody. See you next time.
Creators and Guests
