The Return to the Chamber of Mysteries
And there we go. New land speed record.
Bryan Cantrill:It's a way of insulting me? Is that what that is?
Adam Leventhal:No. No.
Bryan Cantrill:Land speed record is fine. It's fine.
Adam Leventhal:Getting the recording going.
Bryan Cantrill:Oh, there we go.
Adam Leventhal:Oh, no.
Bryan Cantrill:Not you at all. No. How how could perish the thought? How are you?
Adam Leventhal:I'm doing great. This weekend, I played adult baseball and I caught seven innings.
Bryan Cantrill:Oh, that's exciting.
Adam Leventhal:And my knees killed me.
Bryan Cantrill:So Oh, it is such a hard position.
Adam Leventhal:I felt great. I felt great the whole game. I felt great afterwards. And then going to bed that night, I was like, I was in active pain. So
Bryan Cantrill:Yeah. Great time. Do you that's that's great. I'm I'm very, very happy for you. I we gotta we gotta you need to fulfill our our dream of you catching a Oh.
Bryan Cantrill:Tobin went healthy.
Adam Leventhal:Oh, heck. Yeah. That'd be that'd be amazing.
Bryan Cantrill:Yeah. It's terrifying. I'm not sure if you do you do you have guys that were throwing sliders or?
Adam Leventhal:Yeah. Yeah. Yeah. So, I mean, it Yeah.
Bryan Cantrill:Yeah. We have been
Adam Leventhal:throwing, like, lots of junk, which is which is tough as a hitter because I I had not played in thirty years, but also tough as a catcher.
Bryan Cantrill:It's very tough as a catcher. So as it turns out, you learn why they it's very important for the catcher to know what pitch is coming.
Adam Leventhal:Yes. Yeah. Yeah. Keeps them busy.
Bryan Cantrill:And a little quite literal inside baseball that, Adam, I'm sure you are aware of, but I was not aware of until I had a pitcher as a kid. When a pitcher's throwing their warm ups on the mound in the inning break before they're gonna pitch, You will see the pitcher gesturing with their glove before they throw a warm up pitch. And that gesture is to tell the catcher what's coming.
Adam Leventhal:That's true.
Bryan Cantrill:So that is to tell them whether it's a curveball or a slider or a change up or a fastball.
Adam Leventhal:So I had seen that for years. Never really knew what it was. Then I went to warm up some guy, you know, the first time I was playing adult baseball a few weeks ago. And, you know, they started making motions, and I had no idea what it meant until the pitch started coming at me, I was like, I've I I put
Bryan Cantrill:it together. What the hell is that? Well, yeah, exactly. What did that you know what I would be very curious about is what is the gesture for a knuckleball? Or maybe when you've a knuckleball pitcher, they just don't perform up with it.
Adam Leventhal:No. No. No. I I I know this one.
Bryan Cantrill:Oh, there you go. My Dave. Boy He wiggles his glove. Oh, little wiggle. Yeah.
Bryan Cantrill:You know, I should know that like if there's one place you wanna find a knuckleball, it's adult baseball, which I'm sure like
Adam Leventhal:a third of the pitches are knuckleballs. 100%. 100%. I mean, there's a lot of junk and definitely plenty of knuckleballs.
Bryan Cantrill:That is great. Well, I'm sorry we've been away for a couple of weeks, but it is very it's good to be back. Yeah. I you know, I'm I'm battling myself because I I in terms of you know, we were at some summer vacations on. And then we had the my necessary fighting my neighbor at the planning commission.
Bryan Cantrill:I shouldn't say neighbor. It's with the owner of the house next door flipper that has got a I want more people with it. I I I it has that written down right in this index card that I am holding in front of myself to not
Adam Leventhal:I also would make clear
Bryan Cantrill:With your neighbor, we just but yeah.
Adam Leventhal:Left for too long, like, not fighting physically. I just you said fighting and then you didn't explain. So now please explain.
Bryan Cantrill:Yeah. Yeah. Exactly. Well, no. No.
Bryan Cantrill:I I am gonna I'm gonna continue to to resist the temptation. I am only gonna say this. The truly the role that LLMs are born to serve is, disputes among neighbors at the planning commission. Is the and I can only I would I think I would read a compendium. I mean, because you know silently or maybe not so silently.
Bryan Cantrill:LLMs are revolutionizing so much, but you know that they must like, HOAs is there a podcast that we can listen to about LLM drama at HOAs? Like, think that that doesn't that just write itself, I feel? A 100%. Yeah. Because I feel like especially you go to an HOA where it's like, some are going to be aware of it and some are There's going to be an asymmetry
Adam Leventhal:in Yeah. But it also brings even more confidence to an already confident conversation.
Bryan Cantrill:Yes. Yeah, absolutely. And well, and also, mean, this was amazing. I mean, not to again drag anyone into it, but let's just say the plans that these jokers have are riddled with very basic errors. And I just like to I had kind of worked through them with with ChatGPT and found all sorts of like really glaring stuff.
Bryan Cantrill:But I didn't like feel like, you know how you feel like, alright, I've just kind done this back and forth with one of them. I just don't feel like doing it with the other one. So the other one, just like chucked it like the plans as submitted to the city and be like, hey, like, are there any issues with these plans? And it just like, it's amazing. Just bullseye it.
Bryan Cantrill:I mean, I'm like, things that are very, very visual, you know. It's just like, anyway, it's amazing stuff. We live in a whole new world where we this is not just about writing trolling posts on Nextdoor. It's much more than that. It's about it's about battling our it's about battling our neighbors in our HOAs and planning commissions.
Adam Leventhal:Rubbing to the teeth with battles with our neighbors. Yes.
Bryan Cantrill:Yeah. And I really feel like this is the most immediate and profound impact of of LLMs. And I feel like we're not talking about it enough. That's what I have to say. There we go.
Bryan Cantrill:That's fair. We are we are only five minutes in. So I feel like we've managed to that's not what we're talking about today. It's not weird.
Adam Leventhal:You're weird. Also not very small, but you know, got that in
Bryan Cantrill:as well for your finger. Right. Anyone listening to this is like, are you sure you're not talking about this today? Okay. So I look forward to listening to the edit where this has all been cut out.
Bryan Cantrill:No such edit will be forthcoming. Okay. So we and Adam, I don't if you got a chance to to if you dipped back in to our Oxide and the Chamber of Mysteries from over to three years ago, like a long time ago.
Adam Leventhal:That has been a minute.
Bryan Cantrill:Yeah. It has been a minute. And Robert, were we on Discord at that point? Are we on Robert Keith, were we on Discord or were we Do you remember the Yes. Last time we did I think we just started on Discord.
Robert "RFK" Keith:Yeah, early.
Bryan Cantrill:Early Discord. Yeah, I think it was early Discord that we But it was a while ago. So I am convinced Oh, rather, I would love to be proven wrong. If there is any other company that is talking about their engineering endeavor in compliance, in EMC compliance. So EMC is electromagnetic compatibility.
Bryan Cantrill:We'll give people a quick speed run on on what that means. But I don't think people talk about this. I mean, we obviously were over shares to begin with. But I mean, Robert Stu, Dan, have you ever I just don't think companies because I and I think there are reasons for that. But I think we are the only company talking about our adventures in compliance.
Robert "RFK" Keith:In how you actually do it, yes. Mean, within engineering circles, obviously, we talk about it because we wanna share how we solve problems that we all hate. But no one at least not that I've been around outside of in more public setting, talk about exactly what they were doing and how it went. Because it's kinda always, I mean, there's really
Bryan Cantrill:Go on.
Robert "RFK" Keith:Say it. The Black Magic Handbook, which is a really good book. But it's not. It's not black magic. It's just like it's just different.
Robert "RFK" Keith:It's the things that you don't normally encounter or think about in your when you would go to a conference on design.
Bryan Cantrill:And I think it's because it feels kludgy, even though it isn't.
Robert "RFK" Keith:It does. And Yeah. It also feels like somewhat there's a nonsensical nature to it because it's almost like the things that occur to you are like, it was not intentional, and so you don't plan for it. And so you have to come up with some way to deal with it in
Adam Leventhal:the moment.
Bryan Cantrill:Yeah. That's that's that's another good point. And in other words, like, artifacts are always, accidental. No one is going in there with the intent. So that's the other reason it feels kind of like, I'm kind walking you through my dirty underwear.
Bryan Cantrill:But so, Robert, could you just give people context for what EMC compatibility is and why it's important? Just kind of catch people up on that.
Robert "RFK" Keith:Yeah, sure. So whenever you build anything in electronics and mechanical designs or whatever, you have your signals that go on the boards. As we know, that's how you communicate across different things. And those signals operate at different frequencies, and they can potentially become radiators and cause interference in in some cases with other things around you or other things you're plugged into if it's over like for conductive immunity or conductive emissions, something like that. So when you plug a bunch of things in together, they could possibly have modes that are introduced by your power supplies that could radiate back into the line and maybe mess up something that's close by you or connected to you or something like that.
Robert "RFK" Keith:And you don't really realize that until later on in your life in electrical engineering and you have to go do this kind of stuff or you're a radio person or an SI person and then it's a little bit more obvious to you because you study it in-depth. But then you have to go and learn about the standards that the government has set in order to make everybody play well together. So
Bryan Cantrill:Okay. So elaborate a little bit on this. And when you say a radiator, to be clear, you're not talking about like a radiator in the heat sense.
Robert "RFK" Keith:No. No. Well, I mean
Bryan Cantrill:Well, mean, okay. Yeah. Yeah. Fine. Yeah.
Bryan Cantrill:I mean, but in terms of like, you're talking about radiator in the Electromagnetic radiation. Electromagnetic So
Robert "RFK" Keith:the So at a frequency, you
Robert "RFK" Keith:will have a signal source that is escaping into free space and that is bad at a certain level. There are certain amounts of power at a distance that are acceptable, and you want to be below them, well below them, so that you're not interfering with things that are in a similar space. You say if I for ex a good example of this is your classic radio. It's on your phone. If these things were really, really close together, like two phones very close together and they didn't know how to not interfere with each other, you would be talking on a similar frequency because you need to in order to send and receive.
Robert "RFK" Keith:But if you send at the same place, at the same time, at the same frequency, you will stomp on each other and nobody will be able to talk. It will fail.
Bryan Cantrill:That's right.
Robert "RFK" Keith:Right? So but you don't want to do that. And so with
Bryan Cantrill:And things will just, like, not work mysteriously. Will end up with the will because you can do all with that, that electromagnetic radiation could do all sorts of things to devices that are around it, and it can induce all sorts of madness.
Robert "RFK" Keith:And so with a radio, like, you do want to radiate, so you have things that do radiate on purpose. With us, we don't build those things. We have things that radiate and we don't like that at all. Right. We've made a mistake and we want it to stop.
Bryan Cantrill:And so you'll hear us refer to this as like no accidental antennas, no unintended antennas. Yes. And I do think it's it it is maybe important to differentiate a slot antenna from a dipole antenna for purpose of this of this discussion because I think when you think like no accidental antennas, alright, isn't like an antenna like an aerial? Like, how hard like, really? Do you have, like, aerials lying around?
Bryan Cantrill:But that's actually not the kind of antenna. I think that's not the antenna. I mean, I'm sure we we could have a situation. But but I think the slot antennas are the much more likely antenna.
Robert "RFK" Keith:Yeah. Or cavities. What happens Right. Is
Robert "RFK" Keith:Yeah. With
Robert "RFK" Keith:the if you're thinking about an antenna that goes on your old school mobile phone, it's that little thing that you pull out and it's a certain length. And that length of the antenna actually corresponds to the frequency that your phone is going to transmit at. So you're looking at things between like you're looking at 400, 600, 900 megahertz. Like so that's gonna be like three inches long. And lo and behold, your antenna works because it needs to work in a bunch of these ranges, is about a three inch long quarter wave radiator.
Robert "RFK" Keith:Okay. Cool. Great. For the frequencies that we are working at, you will get things that are like and the one that we were looking at is up in the, you know, very, very high gigahertz, 26.565 gigahertz. You'll get things down sub gigahertz in the, you know, 625 megahertz range, which is kinda like what we're talking about with your cell phone, and that's gonna be your gigabit Ethernet.
Robert "RFK" Keith:And you'll I mean, you'll get things all over the place. But you'll so you'll be looking at shapes that will be a quarter wave, half wave of that. So that is a particular length. And if you have a cavity, which is like, call it a metal box with an opening on it, and one of the sides of that cavity happens to be five, six millimeters. Oh, that's gonna start looking like a nice order wave for your 26.565 gigahertz.
Robert "RFK" Keith:Right? So these cavities that you're if you have anything that radiates and it's passing through that structure, it will amplify it in a way that you kinda don't understand because we didn't build an intentional antenna. It doesn't have like a particular structure that, like, directs it in a way that you intended. Those are real antennas, and those are expensive and cool. And the ones that we have are, like, sad and stupid, but they still are effective.
Robert "RFK" Keith:And that's inside your design, like a mechanical the mechanical structure of your design can be like that. You can have traces on the top side of a board that don't have good cavities around them. For example, like, you have a topside trace, in typical PCB layers, there's a signal layer and a ground layer and a signal layer and a ground layer. Well, say the topside is a signal layer. There's nothing above it.
Robert "RFK" Keith:You have a ground layer underneath it, and you don't encapsulate it well is an example of a structure called the grounded coplanar waveguide, whereas where you put vias along the topside trace, and that provides a place for the radiation to go. Instead of radiating it out into free space, it goes to the ground vias that are right next to it, instead creating this tight coupling. Right? So that's a very, very common structure in, well, radio because you're routing these antennas on the top side of a PCB, but it also happens in other places. And if you don't do that, you and you have a trace along the top side of something that has a stub on it, maybe.
Robert "RFK" Keith:Somehow it's like a connector, and that stub is just long enough, well, it can become a radiator. And then so that's gonna try to go somewhere. It needs to find its return path, the other end of your signal. And if it can't easily, then it's going into free space.
Bryan Cantrill:Yeah. And this can be, as you say, these are not deliberately designed. So I think part of the reason that this can feel mean, when we talk about when people talk about this, I I don't know that there's an engineering domain in which the occult comes up so frequently. It is it's witchcraft. It's voodoo.
Bryan Cantrill:I mean, it's like people view it as as as supernatural because to a certain degree, like, is very hard. Like, you know, you have this emission at this particular wavelength. And then it is wild. We're like, okay, what is that wavelength physically? And then we're looking in here for an antenna of that length effectively, which I mean, I know there's more process to it than that, but it can Once you're in the chamber, it can feel like finding these accidental emissions, It's very much this kind of based on the geometry of the thing.
Bryan Cantrill:So one question I have for you. So the on the traces, the PCB traces, do we, I mean, because it feels like we have not had, those are not the problems we've had, knocking on wood. Is presumably, there is
Robert "RFK" Keith:By design.
Bryan Cantrill:Two okay. Right. By design. I was gonna say, so presumably, that's because we I I would assume that we get it, whether it's from ANSYS or others, we get a lot of tooling support to be able to simulate that and be able to because that feels like something that we can actually simulate pretty reliably. Is that right?
Robert "RFK" Keith:You can simulate it, but I think that a lot of what we have done for reasons other than EMC, grounded coplanar waveguide also gives you a very stable impedance. Impedance imbalances can create radiating signals as well. But we have a very stable impedance with a ground copator waveguide because of the, well, the coupling to ground. You also have high insertion loss as a result of that, which is kind of like a negative to it. For us, we design all of our PCBs with the intention is that our sensitive signals are routed on internal layers with ground encapsulation, which will pretty much necessarily prevent this kind of like free space radiating problem.
Bryan Cantrill:Yeah. Interesting. And the so then we're kinda like left with these others. Left with
Robert "RFK" Keith:Oh, yeah. So the connection points and these like, for they go out where the vias come out to connectors and where, you know, via structures, if you have good ground you need to have good ground return paths on those or you'll have an impedance imbalance, like I'm talking about where that can cause common modes to get into something else, the shield of a cable, for example, because it's trying to return, right? It's trying to balance itself off. And so you have a little bit of energy at the termination point of your load line, and that energy is going to get reflected back, but part of it is going to continue to pass through, and that part that passes through will radiate.
Bryan Cantrill:Yeah, and I love that kind of like intuition that you obviously have to have for this as you're looking for these sources. I think once you know that, and I should also add that the compliance think we talked about this last time, but the regulatory body here is actually the FCC. Is the In The US, it's the Federal Communications Commission. And it is any And I assume that this is like similar abroad as well, but it's any clock rate more than nine kilohertz. So Yeah.
Bryan Cantrill:That's like anything, basically. I mean, I'd you'd be hard pressed to, not have a clock rate above nine kilohertz. So that's lots and lots of stuff, obviously. Okay. So then the other thing I always point out to this, Adam, I know I talked about this last time with you, but for folks that are kind of new to this idea of EMC compliance and don't understand that everything you have around you has gone through this process.
Bryan Cantrill:If you take my kind of favorite example of this that I think Robert was either you or Nathaniel educated me about or Eric, drive caddy, if you take out a drive caddy, on the side of the drive caddy are these springy fingers that I always assumed was to like secure the drive. And I think I again, I can't even remember who this discussion was with, but the way it was kind of like posed in this kind of Socratic style of like, why do you think those are there? And I'm like, to secure the drive and whoever and I don't know if I I actually don't even know if I'm inventing this conversation or if it actually happened. But in the conversation in my mind, one of you was like, does that make sense to you? Really?
Bryan Cantrill:Do do NVMe drives become loose all of a sudden? Is that like does it they need additional, like, additional rigging to make sure that they stay stable?
Adam Leventhal:But, Bryan, in your defense, like, if the other guess, to me, sounds just as ridiculous. Like, if if they're like, what do you think you need the fingers for? And I'm like, to keep the electric magnetic radiation from leaking out. Like, that's
Bryan Cantrill:And you're like, no. No. No. Stupid. No.
Bryan Cantrill:Stupid. Don't say that. They're gonna make fun of you now.
Adam Leventhal:What? You think it's like water and it's like drip out if it's not like got a tight seal or something?
Bryan Cantrill:Oh. Oh, because like you think there's like a slot here and you think it's like because of the geometry of the slot that's gonna be like decide the it turns out that's exactly what it So great. Yeah. Exactly. That that is exactly.
Bryan Cantrill:But the the those fingers are breaking up the the length of that slot between the the the caddy, the drive caddy, and the chassis, and you are turning it in. Robert did it. Correct me. It's too correct me if I I'm getting this wrong, but you are turning that slot into several smaller slots. I think it's like usually like four or five of these little fingers.
Bryan Cantrill:And you're still gonna have that potential emission, but it is an emission that will be at a higher frequency with less energy.
Robert "RFK" Keith:Yeah. Yeah. So you're you're then depending on how you want to break it up. Right. And so and you just add a lot of that destructiveness in the way.
Robert "RFK" Keith:Right. And that's like the cheapest way to be like, well, I can't make it stop at this connector because, well, it's a
Adam Leventhal:disconnector. It does feel like It's
Bryan Cantrill:so funny because it's like, it's the physics of the thing. And yet it doesn't feel like first principles engineering. It feels like last principles engineering. Like, this is like, you know, shit, this thing's gonna ship and now we gotta like And But it's really not, you know, think it's It ends up being Because it is very kind of core. Anyway, this is like all of why people don't talk about it.
Bryan Cantrill:But I think they should talk about it. That's what I got to say. If you're a company doing MC compliance, like, come on, we've shown you ours, you show us yours.
Robert "RFK" Keith:Yeah. I mean, you might sometimes like, oh, I don't know. You see some people that are pretty frustrated in the chamber sometimes where they just I mean, we were very frustrated in the chamber. This but how many times? I don't know.
Robert "RFK" Keith:It's two. We probably went in there four times for so many so many hours, and then you just can't squash it. You're like, I what? Am I dumb? What's better?
Robert "RFK" Keith:Like, I know what I'm doing.
Dan Chirpich:It doesn't help that there's some time pressure too, because it's very difficult to get time scheduled to do these tests in the chamber and you need very specific equipment to do it. So you wanna make the most of the time that you have there, and you go through your three good ideas, and then you start looking at your bad ideas, and yeah, you don't feel very good about it.
Bryan Cantrill:Right. I think that's right. And I think it's also like, because it mean it is quite literally a check the box item and that you're to satisfy regulatory compliance. And so I think that's the other reason where it's like, there's no like good news that can come out of the chamber. It's only bad news.
Adam Leventhal:You're not like distinguishing ourselves by how you're secretly
Bryan Cantrill:we are. It's like, are you or aren't you? Right. Are you or aren't you? Alright.
Bryan Cantrill:Okay. So this is all a very good lead in. So we The other thing you've gotta go do is as the product updates over time, so like you've kinda done compliance once, you kinda You need to go back and you need to It's certainly if you change the CPU or change the sled or we you change the PSU or we change these parts, we we need to go do another compliance run. So, maybe that's a good context for, yeah. So describe our return to the chamber here.
Robert "RFK" Keith:Yeah. So we are updating our products, as you do. And so we have Cosmo, which we have already been in there with with in a limited but due to power constraints oh my god. I just recalled the entire shipping fiasco.
Adam Leventhal:I think we'll just
Bryan Cantrill:The shipping which shipping fiasco? I mean, have been several. So
Robert "RFK" Keith:Well, Bryan, you're so right. Just that the one like when they were dropping off both of the racks that for this for the first time that we went in there and it didn't work. Well, that's another story for another day.
Bryan Cantrill:Are we talking Gimwit or Cosmo?
Robert "RFK" Keith:Cosmos. The the most recent two racks that we shipped there. So we had a rack model zero, the one that we go with wherever. And we had the rack model point five, which is supposed to be mechanically representative, sorta kinda, you know, we're doing our best job here of rack model one, which is what we wanna ship. So we're still gonna have to go back, but doesn't matter.
Robert "RFK" Keith:That's not important right now. So you have to drop both of these off. And I just traumatized myself about the memory. Where are we going with this? Right.
Bryan Cantrill:And I I this is we're we're here for the trauma. This is the I mean, what what you call trauma, we call content around here, the way.
Adam Leventhal:While you're past now's the the trauma, I just want to say, Cosmo is our current generation AMD Turin based which Arke mentioned is our previous generation.
Bryan Cantrill:AMD M1. Yeah. Yeah. That's right.
Robert "RFK" Keith:So Cosmo, we have a we had a configuration of the original generation of a rack with 24 sleds, not 32, which is the full complement that you could put into a rack due to power constraints for only 24. Okay.
Bryan Cantrill:Right. Right.
Robert "RFK" Keith:Now we want to do 32, which we can do now because power constraints are no longer constraints. Great. You need to test the product with this higher concentration of sleds because it's all Gen five PCIe, and that's a higher frequency that we haven't really tested in that concentration yet. And we're worried maybe it'll radiate more than we thought.
Bryan Cantrill:This is the thing that's so the thing that definitely sucks about this is that like just because you've done it with 24, actually, like the 32 makes a difference. Like you actually have
Robert "RFK" Keith:a number makes it up. You're like, damn, I have to I have to look at that.
Bryan Cantrill:Right. Because you're kind of you got what might be the radiated emission from one sled would be okay, but when you multiply it by 32, it's actually not okay.
Robert "RFK" Keith:Yeah. And for us, because we sell this thing as one unit, we have to test it, like, altogether as one. And that makes it more challenging than other per like, server designers because they're just gonna go in there with one sled, plug it into 12 volt or, you they're going to plug the little power supply in, 12 volt power, and just call it cool, or 120, and it's got 12 volt power on the power supply. So they really don't have to worry about this huge chassis and like, all the different sleds and the different in like, the switches and the sleds together and then then, like, the cabling that runs through the back plane. And we we we just got all these places.
Robert "RFK" Keith:We have so many places where things can go bad and be sad and cause problems. But we do it. And so we've just decided that's fine. And so we go in there with the whole thing. And this time we had to go in with 32 to check.
Robert "RFK" Keith:Rack model zero is still good. Thirty two, nothing radiates ridiculously. Sure enough, fine. We're not plugging in a bunch of stuff in order to keep the draw down because the chamber at the time could not support over a the listen, which is the the filter for the EMC chamber could only handle a 60 amp load. Okay.
Robert "RFK" Keith:We will break we will blow that with the next model of rack. So they had another listen being shipped to them. Can do a 100 amps, that would have been good. There's another story here about that not working out and us having to wait and life being difficult. That's a story for another time.
Robert "RFK" Keith:Also traumatizing. Two had to live that one more up.
Bryan Cantrill:I mean, so is it okay. Is it a little bit surprising about the the 60 amp limitation? I mean, feels like wait. Or maybe that just highlights that how rare it is for something this big to need compliance as a unit.
Robert "RFK" Keith:I think that we will see more and more of things that require a higher amperage, but like
Bryan Cantrill:Oh, interesting.
Robert "RFK" Keith:It's you know, there are places that you could go. Like, if that's your specification, which it wasn't before, it is now, then you locate a lab that has something that is rated higher. We would like them to continue, we just want to test there. And they said they could support it. They're going to update the chamber anyway, so they'll just buy the new listen.
Robert "RFK" Keith:And we're like, great. Sounds good. You do that. We'll keep testing with you. And so ultimately, after a couple of fiascos and some scheduling sadness and a lot of false starts, they did get the 100 amp listen.
Robert "RFK" Keith:And we were able to bring in the rack model 0.5 into the chamber. And I left after rack model zero, and then Stu arrived to pick up where I left off because By the way, our time in the chamber, we are going from 8AM, really nine because it takes time to get started, to 1AM. So you're there like day after day, like you're back to back. It sucks.
Bryan Cantrill:And you are in something that I mean, you are you're in a chamber that is designed to be insulated from electromagnetic interference. It is also insulated from all other aspects of the physical world. You are in a sensory.
Robert "RFK" Keith:Is like convenient place to do things.
Bryan Cantrill:You're not gonna see the sun streaming in.
Robert "RFK" Keith:You have no idea of like, because there's no windows or not like, you just kind of you walk. There are times when it's winter, can go in there and it's dark. You walk in, it's dark. And you leave, and it's pitch black. And you're like, where the hell am I?
Robert "RFK" Keith:It's a jarring experience. And Stu, you started. You picked it up where I left off. And you're actually turning everything on in this new rack configuration for the first time was supposed to be a chill experience. Turned out to be un chill Right.
Robert "RFK" Keith:Kind of the whole time.
Stu Donnan:So yeah, the context there was we're trying to get two batches of testing done in one visit to the chamber. So we were testing rack model zero in a configuration that we were gonna be selling with 32 Cosmos, and then we're gonna switch gears and get rack model point five in there. So the first thing I had to do is take 32 computers out of rack model zero and then I had to install eight additional SSDs into each of these sleds before I installed them into rack model point five. So that took a little while. We got it all set up.
Stu Donnan:We powered it on, and I think this was the first time we had actually powered on a rack with our new power shelves that have the higher power limit. So we got to experience the new version of our rectifiers which have a much louder and higher pitched fan. So you have that screaming in your ears. And then the goal was to kind of sit back in the control room and check the box, right? We know this system passes, no problem.
Stu Donnan:It's just going to be a relaxing two days back to back of our double shifts, but we just need to babysit it through that. Turns out that was not the case.
Bryan Cantrill:What went wrong?
Stu Donnan:So, had an exceedance at 26.562 GHz. In the past, we've not seen much radiation at this frequency. We had seen a little bit of a peak, and we knew that was associated with some of our high speed network links. But in the past, that was well below the threshold. And so, now we had this higher radiated output at that frequency and that was very unexpected.
Stu Donnan:So network links all come out of our sidecar, which is the switch for the rack. And that is part of the rack model point five design that has not changed at all. So we didn't expect to see anything new here, and it was it was quite a surprise.
Bryan Cantrill:And did we get so then you're trying to debug that. Any luck on debugging that in terms of figuring out like it'd be what wavelength does that what physical length does that correspond to?
Stu Donnan:I'd have to go back and look at the math, but we're looking at something around like 10 to 12 millimeters, I think.
Adam Leventhal:Yeah. 12 millimeters is about
Robert "RFK" Keith:a quarter wave. So like you're and like 10 millimeters, that so like if you
Bryan Cantrill:There are yeah.
Robert "RFK" Keith:Looks like that there are not we have gaps of that size in this place that we believe to have appropriately accounted for in their shielding. Or so similar to the drive, the QSFP cages have little lance bridges, bridge lances, mechanical engineers, tell me which one's right. But they stick out inside of the cage and they press into the sheet metal of the sidecar. And that should connect you to the chassis, which is like completing your your big shield. Right?
Robert "RFK" Keith:And that that is the path by which chassis currents return to the the source, the buzz bar. Yeah. So technically, that should work. You would think. Like, we're like, oh, this works.
Bryan Cantrill:You think?
Robert "RFK" Keith:Before. We had not seen this happen quite as violently as as it did this time. And so you don't really expect this to come up.
Stu Donnan:Was actually one of the first things we checked when we saw the succeedance. And so I went and I disassembled a sidecar from a different rack so that I could take a look at it while we were proceeding through some of the other frequencies we had to check. Each of these tests takes a long time. They have to set up all the antennas, have to set up the equipment, and then they have to run the measurements. So each of these tests can take an hour or two depending on the frequency and the bandwidth they want to inspect.
Stu Donnan:So I was trying to make the most of our time, took apart another sidecar, and I looked at our sheet metal, front plate for the sidecar. And I found a few areas where the masking for the paint, which should keep the paint off of these important interfaces where the spring fingers from the QSFP cages and the circuit board were pressing against the sheet metal to make electrical contact. So the sheet metal is steel, that's conductive, but if you paint over it then you're not getting a contact there. So our manufacturer actually masks those off so that no paint is applied on those interfaces. We found a few where the paint was a little bit off from what we had in the spec.
Stu Donnan:So we thought, oh, this must be it. So then we get the test article out of the chamber. Oh, go ahead.
Bryan Cantrill:Oh, yeah. Is it angle grinder time?
Stu Donnan:Almost. I I actually left the Dremel in the box. I I cleaned up all of the cutouts using, sandpaper and, that unfortunately made no improvement. So that was that was the first thing we tried.
Bryan Cantrill:Okay. So the the so the first round of crops has failed again. And when you are so actually, just a naive question. When you are looking for that, you got this source of emissions somewhere in the rack. You don't really have a way.
Bryan Cantrill:There's not like a Geiger counter. I mean, you can't take an antenna into or or or to what degree can you localize where this thing is coming from? Because in my experience, least when I was in the chamber for Gimlet, like, didn't really have a way of of really honing in, certainly not for something that's like 10 millimeters long where this thing is coming from. Is that is that right?
Robert "RFK" Keith:Or is that
Robert "RFK" Keith:are with see, there's no, like, near field probe like you can with, lower frequency stuff for this. But because it's the you you can get the angle and the height from the from the antenna that is in the chamber.
Robert "RFK" Keith:So it's moving up and down on
Robert "RFK" Keith:this boom. And the the rack or whatever you're putting in there is on a turntable. So it's rotating around three sixty degrees. And so you can determine kind of what it's looking at from like, what the angle is and what what what angle the machine is at. So it the the antenna is at, like, two meters, and it's kinda, like, angled down a little bit, the antenna.
Robert "RFK" Keith:So the angle is at about two meters and it's pointing directly at the front of the rack. You're like, that seems a whole lot like right in the freaking middle. And I don't know what else is there. And what else is that? Like 26 gigahertz.
Robert "RFK" Keith:Kind of narrows down the pool of suspects pretty far. But in another case where it may not be so specific, you can do a couple of things to eliminate local where the source is locally, a lot of shielding. If so, like, if you wanna throw, say, where in the sidecar is this coming out of, and we did a lot of this to minimal effect. But if you put some tinfoil, literally, you just get out there. Then you start to look
Bryan Cantrill:like a
Robert "RFK" Keith:crazy person. You're like, I
Adam Leventhal:don't to
Robert "RFK" Keith:make this tinfoil half.
Bryan Cantrill:Excuse me. If you get the engineering coming through, you need just this tinfoil.
Adam Leventhal:We're out of tinfoil.
Bryan Cantrill:That's right.
Robert "RFK" Keith:Literally, you just get make a shield and cover areas and look at the spectrum analyzer and you can see if you dampen or not dampen a signal by blocking things off in the antennas, well, direction of the path. That's kind of a way to go about it because there's other problems where you get multipathing and you'll see the signal in a different way. You just moved it around. But trying to bisect areas. That's one way to go about localizing something.
Stu Donnan:So we were actually able to localize the switch fairly quickly because we were able to go in and just turn off the transceivers and prove to ourselves that the peak went away. So we knew
Bryan Cantrill:Oh, okay. That's good.
Stu Donnan:Roughly where it was.
Stu Donnan:And then Okay. Yeah. We inspected the front panel for any obvious defects, didn't find any. So then we started pulling on this other thread. Is it one of the transceivers or is it all of them?
Stu Donnan:So we started like a binary search where we turn off half of them and then turn off half of those, and then kind of move through turning them on one at a time. And unfortunately with that approach, we were not able to isolate it to a single defect. It just seemed to be the, you know, the total of all of the transceivers on
Stu Donnan:at the same time was what was pushing us over
Stu Donnan:the edge.
Robert "RFK" Keith:That was the Yeah. Really, that's the problem. We were.
Bryan Cantrill:Sad.
Robert "RFK" Keith:Well, sad. And it was late, I think. We had another day. And I so then we started we started taping things to cover up gaps where we thought that, like, maybe like, Steve had done a valiant job of shaving down perhaps rusted plots for the QSFP cages that probably took so much time and it's really just not kind of fair for anyone to have to go to. And so we were taping those down to make better contacts, not a whole lot of effect there.
Robert "RFK" Keith:Just try to get better return create a better return path for the for the QSFP cage, and that didn't really work. And then we started looking for things that weren't well grounded in the inside the sidecars themselves. That didn't really work. We tried to create some baffling because we thought maybe the flyover cables were poorly dressed internally. Maybe the flyover cables were radiating.
Robert "RFK" Keith:That was incorrect because the baffling didn't work. Should have resolved that. You would
Stu Donnan:have We're really
Stu Donnan:racking our brains for what types of manufacturing variations could have caused this because previously this exact same equipment had passed the test. So we were really kind of focusing on, well, okay, what's changed over time? Maybe there's been some configuration drift, and the cables jumped out at us because in some of the earlier models, when you look through the gaps or through the vent holes in the front, you can't actually see any of the cables. They're tightly packed in the center to stay out of the airflow. And in this unit, some of them were poking out a little bit, and these were the same cables that we knew were carrying the high frequency signal that we were seeing on the outside.
Stu Donnan:So we went in and tried to repackage those to be very nice and clean. Unfortunately, again, no impact.
Bryan Cantrill:When you're doing something like that, what level of what's your balance of optimism and desperation when you're doing that? Are you thinking like, I think this feels like it. Are you thinking like, God, I don't know. Let's try that. That's that's that's a difference.
Adam Leventhal:Right. It's it's it's already one in the morning.
Bryan Cantrill:It's already one in the morning.
Robert "RFK" Keith:Exactly. Those two where you have It's very different. Was good.
Stu Donnan:There was I felt like the amount of the amount that I cared about or thought that an option would work ended up being the opposite outcome. So, if I was really thinking that I was gonna make a huge impact by changing the cables, course, that did nothing. So then I just had to kinda convince myself internally to not get excited about any option. You had to remain stoic. Otherwise, you're gonna influence the outcome.
Stu Donnan:And this is where the religion part of EMIEMC comes in.
Bryan Cantrill:Oh, absolutely. Absolutely. The the gods are listening in. I mean, we know that the gods listen to the podcast. We've already established that.
Adam Leventhal:Right. There's no no atheist in the EFC chamber. There's no atheist in the chamber. That's right.
Bryan Cantrill:Okay. So, well, this is why people yeah. This is why people do appeals with the occult when doing this because it because it because gets results, frankly. That's why. Because it actually alright.
Bryan Cantrill:Did did did but we do we ultimately end up leaving the chamber with that unresolved. Is that right?
Robert "RFK" Keith:You left twice with it unresolved.
Bryan Cantrill:Woof.
Robert "RFK" Keith:Yeah. Woof.
Bryan Cantrill:Was out of stuff.
Stu Donnan:Yeah. And not only unresolved, but like no path forward. Like none of the changes that we made actually changed the measurement that we were seeing when we went back and did a full compliance sweep. So that was really demoralizing, just not even have a little bit of an inch of progress.
Robert "RFK" Keith:Yeah. And then I think after the second round, we were just kind of poking at things in chat over there because what it there was a huge amount of time block where we couldn't go back to the chamber for a while. And so it's like you get a lot of time to reflect on your sins. And he's like, what the hell is happening now? And I think Joe, he had wondered if these were the same transceivers that we had used last time.
Robert "RFK" Keith:And the answer is no. No, they're not.
Bryan Cantrill:Oh, wow.
Robert "RFK" Keith:Well, so they'll rev these transceivers like every year or something. Because they change the DSPs in them, change a little bit of They're spilled to a spec, but you can make some changes. And like, okay, maybe, I don't know, I'm not feeling very strong about this. That can't be right. These have got to be really similar.
Robert "RFK" Keith:And so before we got into this, we went back to this last time. I went to the office. And even if it is a different transceiver, it doesn't matter. You have to be able to operate with it. Right?
Bryan Cantrill:And when you say so these are the QSFP transceivers. And when you say a different transceiver, different transceiver make, model, or just physical specimen?
Robert "RFK" Keith:It is the same spec new year version.
Bryan Cantrill:Okay, so really so your skepticism is like, look, this is the same.
Robert "RFK" Keith:It will
Bryan Cantrill:This is like the same thing. It's a different, like, maybe different part number or different rev.
Robert "RFK" Keith:Yeah. Like, it shouldn't it doesn't matter. Like, if we're gonna we need to be able to use these transceivers. Right? Like, that
Bryan Cantrill:Right.
Robert "RFK" Keith:Right. Right. Doesn't matter. So it's great. Even if that is the problem, that's not an actionable solution.
Robert "RFK" Keith:Right.
Bryan Cantrill:It's still our problem.
Robert "RFK" Keith:Yeah, it doesn't matter. Whatever we talk about, the speculative why or how, it's important to know. But now that we're at this stage of something and we can't really go back and make dramatic changes to a design, That's just completely impractical. We've got to come up with a solve. And so it's like, okay, these things are radiating.
Robert "RFK" Keith:It's coming out of the cage. We're going to get some different transceivers just to see if life changes when you, well, change the transceiver, move the signal around. And the answer is yes. And I'm going to do the band aid that every great EMC sufferer goes through and you buy gasketing or absorbing material and and find your source and slap that stuff on there and be like, please go away. So we ordered all that stuff, and I was headed out in Stu.
Robert "RFK" Keith:Each time, it's like, I'm not letting you do it alone. Was like, oh, thank God. Thank God.
Bryan Cantrill:Good on you, Stu. You know, and I would like, you know, you know, I was like to say, you know, you do not to reveal too much of your personal lives, but you do both spend a lot of time in the outdoors. And, you know, I'd like to think that like, I'm not leaving you out here to die alone in the wilderness. I mean, it just feels like there's a very apt wilderness metaphor here where, listen, we can't let you die alone.
Stu Donnan:Yeah, think it's a do unto others type situation and you really wanna go in with high karma to get through this
Adam Leventhal:stuff. Right. Yeah. Right. Any rigorous scientist will say the same thing.
Bryan Cantrill:Absolutely. Absolutely. Listen, this is the spiritual aspect of the karma, the the the karma aspects of solving our problems. Okay. So we, so Stu's not gonna let you die alone, which is great.
Bryan Cantrill:Guys get to die together. The, and and how does it go in the in the chamber? And this is now, somewhat recently. Right?
Robert "RFK" Keith:Yeah. This is like a a week or week and a half ago.
Stu Donnan:Three weeks ago, I think.
Bryan Cantrill:Wow. Yeah. Just No. I was tell you that too. Moving on.
Bryan Cantrill:Moving yes.
Stu Donnan:Then, yeah, while so while we were getting set up in the chamber, so it takes a little bit of work to roll the rack in, get it plugged into power. You have to figure out which of the network cables that run from the chamber out of the chamber actually work. So we're getting everything set up, and then Robert received a very important email.
Robert "RFK" Keith:Yeah, so Dan is, by chance, at an SI conference in Texas, And is that a talk that at the end of the talk that is talking about well, there's lot in talk, but they're discussing operating these. They're not they weren't I think they're OSFP cages.
Bryan Cantrill:So Dan, what was the what was the reveal? What would tell me about the talk.
Dan Chirpich:Yeah. This is this is crazy. So the talk was completely unrelated, in the sense that the fundamental topic was about the finite finite diff finite difference time domain, as a method for solving three d electromagnetics three d electromagnetic analysis problems. And in the second half to two thirds of the talk, they went into a couple different applications of using this method for electromagnetic analysis. And one of them just happened to be optical transceivers.
Dan Chirpich:And it was in in that section of the talk where they show
Bryan Cantrill:And, Dan, could you could you just can I just just because I'd like to understand some of the nouns in the title? I mean, I know, Adam, I know you understand the title, obviously. But
Adam Leventhal:Yeah. Go ahead and explain
Bryan Cantrill:it to Bryan. I mean, I I I'd be happy to, but I
Adam Leventhal:have something to do.
Dan Chirpich:That's right. You know, the whole the whole purpose the whole purpose of the session is to give you an understanding for for what this method of solving Maxwell's equations is supposed to be about. And it didn't really give me a great background in
Stu Donnan:it for me to be able to relay that to you right now, actually.
Adam Leventhal:So, you know, the the irony
Dan Chirpich:of it is is I went to this talk going to try and learn about this method of of solving physics physics equations for electric and magnetic fields. And in the end, all I came out with was one example that happened to be very relevant at the time, and the solution in that case involved placing absorbing materials around the connector. And I was like, this actually makes a fair amount of sense because, like, part of the way these electromagnetic fields can can go is is, in general, they stay really well contained when you have, you know, a trace like on a PCB that's where the signal's supposed to propagate along this trace. And then the problem happens when that trace goes from running alongside of, like, a giant sheet of metal inside of a circuit board, and we'd call that giant sheet of metal, a return plane. But, really, it just needs to be a giant sheet of metal.
Dan Chirpich:Once that signal trace goes from running along next to that giant sheet of metal to going through a connector, or in the connector, you no longer have that giant sheet of metal. Instead, you just have some other pins, and and we call those those pins ground, signal and ground. But in the end, the basically, the pins don't control they don't contain the electromagnetic energy nearly as well as as inside a circuit board. And Oh goodness. Yeah.
Dan Chirpich:So it it you know, in the in the presentation, they said, hey. We added some, well, they said we actually ran a simulation, and we could see radiation happening from the connector, and we could see radiation happening from, a I need a visual to really have it make sense, but, basically, a a a circular slot in the front of the module. And in this case, you know, know, neither neither Robert nor Stu could do anything about the circular slot in the front of the module, but they had access to connector. And so so yeah. No.
Dan Chirpich:I just I forwarded the content onto to Robert and said, hey. Check this out.
Bryan Cantrill:And and I mean, you must have been sitting up in your chair where you're like, this is exactly the problem I think they're struggling with.
Dan Chirpich:It was crazy.
Robert "RFK" Keith:Don't Steve and I were sitting there. Yeah, well, what's so wild? Because I got your email and we were about to start cutting up, we were about to start cutting up the same, I bought the same material that they used in the paper. Everything was lining up. And you were like, none of this paper, none of the slides that you sent, none of this is very relevant except for these last 10.
Robert "RFK" Keith:And it seems like you might be interested in that. It's like, you know what? I am. This seems very
Adam Leventhal:interesting. I know. Right? I know. Right?
Adam Leventhal:Right.
Dan Chirpich:What was even for me is, like, this was the second session of this whole symposium that I went to, and the one before it just completely fried my brain. And so I kinda went into the second session with low expectations. And and in the end, it just it came back with this one data point relevant to, you know, what's going on in the chamber at the moment. And I was like it it was just a shocking experience for me, honestly.
Bryan Cantrill:It was That is amazing. I mean and I mean, as Adam is saying in the chat, mean, I think what this is Stuart, this is all thanks to your karma. You know? You knew you knew you were doing this for a good reason that, you know, keeping accumulating that good karma. Who knew it was gonna pay off through Dan?
Bryan Cantrill:You know? It's like it it's like EMI itself. You never know where it's gonna come from. So that was that is wild. And so okay.
Bryan Cantrill:So Robert and they've got a paper on this that that you're able to look at and see the exact technique. Okay. That's really cool.
Robert "RFK" Keith:Yeah, so we saw what they did, and Dan and I, we exchanged a couple emails about as we applied. I told them we did what we were going to do originally first, which is try to apply the material to the cage itself because the way the materials work, they they adhere to whatever your radiating source is, as an absorber. And you really want them of them wanna be facing the absorber. Some of them wanna be, like, adhered to
Robert "RFK" Keith:it. So the
Robert "RFK" Keith:we have two different types, and so we're like, okay. I'm gonna do
Adam Leventhal:the ones that you wanna adhere to the the cage itself. Like we tried that. And I you know, Dan and I we we exchanged some emails too,
Robert "RFK" Keith:I was like, oh, this is kind of the results of this thing. And then so at the end of this paper, saw exactly how they had done it. And we can't replicate that exactly because they have full access to the connector. And our connectors are a little different because we get them extruded into flyovers. So there's this other back half that we can't really get to.
Robert "RFK" Keith:And Stu and I get in there are poking around at the fully complete built up and populated QSFP cages. Okay, how do we we gotta figure out how to pop these heat sinks off and, like, how are we gonna place these, like, 10 millimeter by five mill we first off, we have to cut, you know, thirty ten millimeter by five millimeter strips out of
Adam Leventhal:this, like, silicon material. And then
Bryan Cantrill:I mean, sorry. Just you gotta I just paused right there. Just I mean, and I you I mean, I have you've got good hands. 10 millimeter by five. That is small.
Bryan Cantrill:I mean, you're doing that under the microscope. So
Robert "RFK" Keith:we could get it sort of.
Bryan Cantrill:Yeah. On what, sorry?
Robert "RFK" Keith:We had a ruler that which was-
Bryan Cantrill:Oh, had a ruler?
Adam Leventhal:Okay. It was a metric ruler. It was a metric ruler.
Bryan Cantrill:So, you know. Wow. No big deal. Right. Yeah.
Bryan Cantrill:So That's it's small. But this you you're in the the handicraft project from hell as far
Robert "RFK" Keith:as I can see. Now now it's
Robert "RFK" Keith:Arts and Crafts. And then you have to, like, slide little pieces of freshly cut gasketing material. Well, it's not gasketing material, it's absorbent material underneath the QSFP cage on top of the connector in the way that they had applied it. So we're in there with tweezers. And I mean, it's like an exercise and patience the entire time, like trying to get it to fit.
Robert "RFK" Keith:I don't know.
Adam Leventhal:Was Did you
Bryan Cantrill:make model airplanes as a kid? It feels like this is the model airplanes even a thing anymore. Adam, it's just like, oh my god, I should have
Stu Donnan:Nope.
Bryan Cantrill:Can we cut this?
Robert "RFK" Keith:Ever in my life. But
Bryan Cantrill:they were they were a thing when we were kids. Right?
Adam Leventhal:Model airplane. Yes. Yes. Yes. Yes.
Bryan Cantrill:Just help me on
Robert "RFK" Keith:that one.
Dan Chirpich:Yes. They're bottle Our backup. I I folded paper and made it fly.
Adam Leventhal:I I don't think you're helping. Bryan, you're thinking No, but like people nerds around here still fly model airplanes at the airfield. That's what you're talking about, right?
Bryan Cantrill:But just like this this pay I I never had a bike I didn't have I mean, I was missing many components that one would need for this activity. The the the dexterity, the patience.
Adam Leventhal:Right. I'm not sure familiar. Right. No. Because you're talking about, like, the the the, like, intricate balsa lattices that people would build.
Bryan Cantrill:Either balsa or even or or plastic too where you got the I mean, I was mean, I needed to always like the snap together models were always like that was kind of at the outer brink for me. But I the the yeah. Like the the the glue together model jets. Yeah. And I don't even know if that's like Again, I don't know if that's a thing anymore or if just like Minecraft ate all of that.
Bryan Cantrill:That kind of But it feels like you need that kind of like, you need that real dexterity and patience. And anyway, it's just it's Sorry, Robert. Let me cut you off.
Robert "RFK" Keith:Oh, no. It's just amazing.
Robert "RFK" Keith:You're fine. The it seems like most EMC adventures like come down to arts and crafts at some point. This is the at the beginning of it. Now we've arrived at arts and crafts. And so we get all this done, and it's you know, now because the dampening material did work, like, to a degree on the outside of the cages, I'm starting to feel pretty optimistic about this.
Robert "RFK" Keith:And that's, like, sad that you should that's just like as Stuart said before, like, you just kinda gotta keep that yourself. Right.
Stu Donnan:It was the first thing that actually made an impact. So that was Right.
Bryan Cantrill:Very hard. Okay. So that's great. Yeah. It feels like okay.
Bryan Cantrill:We've got it. And did it was it enough of an impact?
Robert "RFK" Keith:Yeah. Yeah. Oh, so we put it all back together, and, like, all the mechanical clearances are, you know, good. I mean, obviously, they had done it in this this paper. And so we we got to the point where put it all back together, and you put all the transceivers back in that you'd applied this to, and you're looking for, like, god, I need I need enough of because you can't get to the bottom cages.
Robert "RFK" Keith:Right? You can't get to the bottom QSFP cages because of the way that we've designed it. So we're doing top half and then doing the math backwards to approximate, okay, if we only run all of these top half transceivers that we can look at it without the material and we know what that should be, so we should see a reduction that if we do if we basically add four times the power, so it's gonna be double power and double again because you have four rows. We're only doing one row. Are we gonna see enough reduction in this application in order to that when we do do this to every single cage that it will still add up to something under the limit with some margin with some error margin for error.
Robert "RFK" Keith:And so it's kind of like you're watching this thing go. You're running the test in the chamber, and and you're just, like, staring at the signal that is for the time being. You're like, okay. This looks good. This looks good.
Robert "RFK" Keith:And it did dampen it enough to where if you do the math on it, we would we're gonna when we go we have to go back. So when we go back, I'm expecting to see, like, two dB of margin with this applied. And if we apply it better in the factory than we did with our tweezers and other and pliers and crap in the chamber. I'm expecting it to I think that they'll do a better job on the assembly line than we did. Let's put it that way.
Bryan Cantrill:Right. And so how do you work that in to the like how, how did we change the manufacturing process for that?
Robert "RFK" Keith:Okay. So what we're gonna do, so we can get this material in sheets and die cut it, which, so this is like, when you you're like, this is the proposal that you're bringing back to your factory. And you just like, look at all the people at the factory and they want to, like, murder you in your sleep. Like, this is like, I I fucking hate you, like, for this solution. But it's I'm sorry.
Robert "RFK" Keith:Like, I I don't know. Like, what are we gonna do? We're revving this board anyway. It's gonna die, and you'll never have to think about it again, I hope, unless it's the same problem next time, and then you will have to think about it. But, you know, we're gonna try not to do that.
Robert "RFK" Keith:So, like, you can die cut this stuff into, you know, preformed little tiny squares so that they basically are just, like, you know, doing your little Legos and you're placing the sticker on the connector. So before these get press fit, the cages, you have just the connectors bear on the boards after they're assembled in in the factory. And at that point, you have really easy access to, like, the whole connector. Right? You you don't there's no cage in the way.
Robert "RFK" Keith:You're not, like, trying to slide these things under little metal little metal, you know, features. So it's gonna be really in comparison to what we had to do easy for an operator on the line to take these, you know, 32 rectangles of gasket or absorbent material and just, like, stick them onto the connector. And fortunately, the area where it needs to be stuck on the connector is, like, pretty obvious where it goes. It's, like, gonna be exactly the size of the space that is on the top of the connector. So
Bryan Cantrill:Yeah. And does that go on the does that go on the QSFP cage or on the transceiver? It go
Robert "RFK" Keith:go Go to the cage. So there's the connector that gets pressed into the board that is attached to the flyover, and then the QSFP cage goes on top of that. And then the heat sink for the QSFPK on top of that. This is underneath all of it. So on the gray connector that the transceiver plugs into because what we're trying to do is cover we're trying to put the absorbent material as close to the pins of the transceiver that that plug into this as possible.
Robert "RFK" Keith:Right? Because that's where it's coming from. And so this goes right on top of it. And the transceiver plugs in and slides right over it.
Bryan Cantrill:And why did we not see this with other transceivers that were do we know that?
Robert "RFK" Keith:Well.
Bryan Cantrill:Why wouldn't we oh.
Robert "RFK" Keith:I so those are a different model of transceiver.
Robert "RFK" Keith:And in
Bryan Cantrill:And it's just fast. They just behave differently.
Robert "RFK" Keith:No. And we're trying to test to the worst case. Right? So we're trying to devise a worst case scenario when we go into testing. So we're using a different concentration.
Robert "RFK" Keith:We're doing a different transceiver, same concentration, one sidecar. Other sidecar, we have different transceivers that we're trying to sell, and so we want to plug in a bunch of different ones to be like, do any of these radiate in some weird way that, like, we don't know. But so but and we can't. It's complete it's so impractical. Like, there's so many transceivers.
Robert "RFK" Keith:Right? Like, you can't plug in, like, a full comp of 64 of, like, one type. Like, there might not even like, how are you gonna buy those? How are gonna test all those? And so you you do a you do your best estimation of a worst case scenario.
Robert "RFK" Keith:Right? So we have one that has one of all of them. Cool. And you have a different sidecar that has different ones. Great.
Robert "RFK" Keith:That doesn't radiate in the concentration of this disc. This does. Reason why we see it here is because we use 64 of all the same type running at 200 gig, which we can't even do in our real product because we can't saturate out that the the sidecars anyway Right. Because we don't have a 200 gigabit backplane. So it's like, this is no customer will ever have this.
Robert "RFK" Keith:Like, they they won't run it this way. Like, that doesn't that doesn't work. Like, we can't do that.
Adam Leventhal:We will be your faith keeps you from being bitter. Adam,
Robert "RFK" Keith:oh, man. The, so because of the concentration of this type, we see it. In the past
Bryan Cantrill:Interesting.
Robert "RFK" Keith:We haven't do it this way because we're trying to do a different idea of what the worst case could be and it doesn't show up. Yeah. Right.
Robert "RFK" Keith:So when you come
Robert "RFK" Keith:in the chamber and you see this, you're like, oh, shit. Like, I'm we've got is, like,
Adam Leventhal:and you're like, alarm. Five five alarm fire is going off. Like,
Bryan Cantrill:a Right. Right. Right. Right. Have we created a really serious problem out there?
Bryan Cantrill:But
Robert "RFK" Keith:Right. And the answer is, well, no. No. No. We we can't even do it because, like, we can't the the power draw like, it's not possible to be running this configuration is what I've what I've figured out.
Robert "RFK" Keith:But in the future
Bryan Cantrill:It's complicated, though. Yeah. It it shows you how complicated it is that, like because it's all these things adding up that add up to that that where you just get, like, it's, like, literal it's multiplying the energy.
Robert "RFK" Keith:Yes. So Yeah. Yeah. And we have the mitigation now for when when it does come to this. Now we have a mitigation for it even though our raters on the line will be very sad that this is the mitigation.
Robert "RFK" Keith:I apologize. One day, maybe we will not have to do this. And I think in the inner intervening time, I'm gonna, now that we have it.
Bryan Cantrill:No, I don't think you should apologize for it. Don't think you should apologize for it. No, no, no. I don't think, see, this is, this is the bit where it's like, this is not a mitigation. Like this is a fix.
Bryan Cantrill:I say, I'm beaten. Don't I. The But I don't apologize. Yeah. Don't apologize.
Bryan Cantrill:Take that apology right back.
Dan Chirpich:Yeah. I don't think I actually don't think it's appropriate to apologize as well because some of the root cause of this can be in the driver design coming from the transmitter on those optical Yeah. And so you can expect to have some level of this common mode energy that's just inherent in the circuit design. And, you know, to apologize for kind of a an intrinsic property of transmitter design is is kinda like trying to apologize for defeating not defeating the laws of physics almost. So, you know, it's I don't know if apology makes sense in this case, but adding absorber at the point where the energy really wants to begin to depart away from its intended path, which is around the connector, It it it sucks to have to do, but it but it does make sense, and it is a good thing to do.
Dan Chirpich:I mean, the alternative is is it's like it's like get rid of the connector and replace it with more cables or something like that. Like, you know, you can't really readily get rid of the connector. It's gonna be there. And so it's gonna be prone to emitting some radiation.
Bryan Cantrill:So Yeah. But this is this also tacks into I mean, Robert, you know, Paul Erdush famously called God the supreme fascist. And I feel like you've got, like, kind of have a kindred spirit there in old Paul Erdish. Think you you kind of like shake your fist at the gods for for creating such a a thorny problem that requires such a physical solution.
Robert "RFK" Keith:Yeah.
Bryan Cantrill:So when was the moment of kind of great relief on this? It, I mean, obviously like seeing Dan's result at the same, the kind of the result of the talk that Dan's in at the same time that you're grappling at that same problem, Was that, I mean, and I know, Stu, you're trying very hard to suppress optimism, but that must have been, mean, suppressing the optimism may have been even more challenging than me trying to suppress my grievance with my neighbors. That a fair read? Did you get accidentally optimistic?
Stu Donnan:Oh, many times. The measurement itself takes about twenty minutes. So you're sitting there watching, like you're just locked into this plot that keeps updating with each, because they rotate the test article three sixty degrees, and then they measure it with the antenna at three different heights. And so you're just watching this thing rotate, and you're seeing the antenna move, and the plot just keeps updating, and you're watching that peak like a hawk. You're just holding your breath hoping that like this is the one that's gonna keep it under the line.
Bryan Cantrill:Oh man. So, I mean, it must have felt good, honestly. I mean, had to have felt good to to get all the way through that and realize that we actually have a real solve a real solve here.
Robert "RFK" Keith:Oh, totally. Immediately were
Robert "RFK" Keith:like yeah.
Robert "RFK" Keith:As soon as we were like, I think this is good enough. We got a beer. Like, we're we're good. Thank god. We don't have to come back and well, we do have to go back, but we don't have to come back for this sort of.
Robert "RFK" Keith:We we do have to come back for this, but not the same way.
Bryan Cantrill:Yeah. We'll have some new addition to the team that we'll be telling. It's like, no. No. This is just a perfunctory run-in the chamber.
Bryan Cantrill:Everything is we don't expect anything in this. And then we can, get more content. That's the goal here. Is Man, this is an odyssey. And I think that we I've quoted it elsewhere, but Piet Hein has got the late Danish mathematician and and poet has a a great line of problems worthy of attack prove their worth by fighting back.
Bryan Cantrill:And this is a problem that fights back a lot. This is a problem that and I I think that, you know, to get that that feeling of real satisfaction. Unfortunately, you had to go through this the the valley of darkness here to to get there, but that that it's it's really terrific that we've got that nailed and, know, Stu, good on you for accumulating the karma. If you wanna accumulate some extra karma along the way, feel like we can use some more. Just, you know, don't hesitate to accumulate that karma as we go here.
Stu Donnan:See what I could do. I think I have a sheep or something around here I can sacrifice.
Bryan Cantrill:There you go. Exactly. And and Dan, has there ever been I mean, I'm very glad you chose to attend the second workshop on the
Dan Chirpich:end. Right?
Bryan Cantrill:I and I do love the someone Stu dropped the logo of the conference in the chat, the EMC plus. I assume you pronounce it SIPI because it's like integrity and power integrity. Or do people say SIPI, Dan? Okay. Yeah.
Bryan Cantrill:I said SIPI. Some
Dan Chirpich:people at the conference say SIPI, and I've never heard anyone else outside of the conference say SIPI.
Bryan Cantrill:So Well, I just I I I told and then you got, like, the the the rodeo rider with the the waveform and the lasso. It's just, I mean, it's just amazing. Yeah.
Dan Chirpich:The tech team there. Was in Dallas, Texas.
Bryan Cantrill:It is it is truly perfect. Just like the yeehaw on SIPI. It does remind me when I, you know what, Adam and I, when I first moved out to Silicon Valley, there was a billboard and what was it for? I can't remember who it was for, but it was a California license plate. The entire billboard was California license plate and the license plate was FPGAASIC.
Bryan Cantrill:And I remember thinking like, man, what is I mean, how amazing is it that I'm I'm in such a, like, a a nerd epicenter. I mean, what fraction of the of of the the broader population knows what those two things even stand for? And yet, there are enough people out here that I can actually go put it on a on a billboard. So I I I feel the same way about It's even nerdier. But it is really great stuff.
Bryan Cantrill:Awesome. Well, thanks so much you three, and thanks for the hard work on this. Roberts too, thanks for all of the suffering in the chamber and look forward to, yeah, exactly as Erica saying in the chat, you guys are SIPI cowboys. So really terrific stuff. And if folks do encounter other, this compliance is it is really, really thorny.
Bryan Cantrill:I think that everyone has some similar experience. I think someone had said earlier in the chat that you really have to have a growth mindset going into the chamber. I remember Nathaniel told me very early. It's like, will have something. So, like, you can't go in there with the expectation you're not gonna have.
Bryan Cantrill:You have to go in there with the just excited to find whatever it is. So I you all had the the the right demeanor, disposition. Stu, thanks for keeping your optimism yourself. That really was a winner. And the gods have rewarded us at least for now or failed to punish us, I guess.
Stu Donnan:I'll take it.
Bryan Cantrill:Yeah, exactly. Awesome. Alright. Well, we've got some we're I know we've been kinda
Adam Leventhal:In the saddle.
Bryan Cantrill:Off and on through the summer. Yeah. We're back in the saddle. Got the kids are back in school, Adam. We got we got a bunch of content coming up.
Bryan Cantrill:So Yeah. I hope I hope folks and at some point, we gotta have a Books in the Box sometime in the fall. So now is your time to read a book that you want to talk about in Books in the Box. So awesome. All right.
Bryan Cantrill:Thanks, everyone. Talk to you next time.
Creators and Guests
