Ha ha -- there are definitely voices that I know I have been implicitly inspired by over the years, but I can safely say that Chad the Bird is new to me! I'm enjoying listening to some of Chad the Bird's work now; my apologies in advance if it rubs off on me!
I meant it in a very positive way. Chad rocks! And so do you and the 0xide team -- what you guys are building is amazing to watch from the sidelines and I wish you and the team great success.
You may disagree with our rationale, but it is absolutely absurd to complain that that RFD 26[0] does not have "any meat." This is in fact dense technical content (10,000+ words!), for which I would expect a thorough read to take on the order of an hour. Not that I think you read it thoroughly: you skimmed to parts, perhaps -- but certainly glossed over aspects that are assuredly not your domain of expertise (or, to be fair, of interest to you): postmortem debuggability, service management, fault management, etc. These things don't matter to you, but they matter to us quite a bit -- and they are absolutely meaty topics.
Now, in your defense, an update on RFD 26 is likely merited: the document itself is five years old, and in the interim we built the whole thing, shipped to customers, are supporting it, etc. In short, we have learned a lot and it merits elucidating it. Of course, given the non-attention you gave to the document, it's unlikely you would read any update either, so let me give you the tl;dr: in addition to the motivation outlined in RFD 26, there are quite a few reasons -- meaty ones! -- that we didn't anticipate that give us even greater resolve in the decision that we made.
I did indeed read your document (twice, as I explicitly reported). I didn't address those parts because I found them better-supported. Instead, I addressed the parts I found confusing, and since your rebuttal here is just whining about what you think my behavior is, I continue to be mystified. That's okay; nobody expects you to explain yourself to me. If I thought it would help, I would suggest that perhaps a more effective defense would involve answering literally any of the questions I already asked. However, I don't appreciate accusations of bad faith based on your unwarranted assumptions about what I did or did not do and, bizarrely, what you imagine my motivations are. I'll just assume that the answers to the "why" questions I asked are rooted in similar wild-ass speculation.
There is a reasonable explanation for the "foregone conclusion" flavour of the RFD that doesn't cast aspersions (quite as much as you are) on the authors:
It is simultaneously an assertion of the culturally determined preferences of a group of people steeped in Sun Microsystems engineering culture (and Joyent trauma?), and a clinical assessment of the technology. The key is that technology options are evaluated against values of that culture (hence the outcome seems predictable).
For example, if you value safety over performance, you'll prioritise the safety of the DTrace interpreter over "performance at all costs" JIT of eBPF. This and many other value judgements form the "meat" of the document.
The ultimate judge is the market. Does open firmware written in Rust result in higher CSAT? This is one of the many bets Oxide is making.
Frankly, I don't think Oxide would capture so much interest among technical folks if it was just the combination of bcantrill fandom + radically open engineering. The constant stream of non-conformist/NIH technology bets is why everyone is gripping their popcorn. I get to shout "Ooooooh, nooo! Tofino is a mistake!" into my podcast app, while I'm feeding the dog, and that makes my life just a little bit richer.
i'm not sure if Dtrace interpreter was safer than EBPF. I guess in theory it should be because a JIT is just extra surface area but I'm not sure in practice. Both EBPF and DTrace had bugs. Also, I always thought EBPF JIT was just a translation to machine code and it didn't do any kind of optimization pass so should be very similar to how DTrace works. They both ship byte code to the kernel. But I guess the big difference is EBPF relies more on a verification pass while I think most of DTrace safety verification was performed while executing the bytecode. I remember there was a lot of stuff in EBPF where the verifier was meant to be able statically determine you were only accessing memory you were able to. I think there was a lot of bugs around this because the verifier would assume slightly different behaviour than what the runtime was producing. But this is also not necessarily a JIT problem you could have an interpreter that relied on a static safety pass as well.
...but your top post didn't ask any questions; certainly not ones that would justify a detailed answer.
It was several assertions, plus your admission of confusion. I mean, there are no stupid questions, but there wasn't even a question there, so I don't blame anyone for thinking you're communicating poorly.
I wasn't accused of communicating poorly. I was accused of lying about reading a text file and having some kind of ulterior motive for my own opinions.
Furthermore, advanced readers are generally able to infer from "I am not sure why x" that a similar flow of discussion might be as feasible as if it were phrased "why x?".
As though that were necessary. There's plenty of room in this comment box for questions better fleshed out than "why x?". Are you expecting advanced readers, or clairvoyant ones?
One point of clarification: the C version does not have (and never had) a hash table; the C version had a BST (an AVL tree). Moreover, the "Rust hash table implementation" is in fact still B-tree based; the hash table described in the post is a much more nuanced implementation detail. The hash table implementation has really nothing to do with the C/Rust delta -- which is entirely a BST/B-tree delta. As I described in the post, implementing a B-tree in C is arduous -- and implementing a B-tree in C as a library would be absolutely brutal (because a B-tree relies on moving data). As I said in the piece, the memory safety of Rust is very much affecting performance here: it allows for the much more efficient data structure implementation.
I wouldn't consider implementing a B-tree in C any more "arduous" than implementing any other notable container/algorithm in C, nor would making a library be "brutal" as moving data really isn't an issue. Libraries are available if you need them.
Quite frankly, writing the same in Rust seems far, far more "arduous", and you'd only realistically be writing something using BTreeMap because someone else did the work for you.
However, being right there in std makes use much easier than searching around for an equivalent library to pull into your C codebase. That's the benefit.
I don't often do this, but I'm sorry, you don't know what you're talking about. If you bother to try looking for B-tree libraries in C, you will quickly find that they are either (1) the equivalent of undergraduate projects that are not used in production systems or (2) woven pretty deeply into a database implementation. This is because the memory model of C makes a B-tree library nasty: it will either be low performance or a very complicated interface -- and it is because moving data is emphatically an issue.
Yes, the point I was making (and as you point out, have been making for the last quarter century) is that we err when not making this realization -- and indeed, I think the linked piece is exactly backwards because it doesn't understand this. That is, the piece views a world of LLM-authored/-assisted software as "industrialized" when I view it as the opposite of this: because software costs nothing to replicate (because the blueprints are the machine!), pre-LLM ("handcrafted") software is already tautologically industrialized. Lowering the barrier to entry of software with LLMs serves to allow for more bespoke software -- and it is, if anything, a kind of machine-assisted de-industrialization of software.
> Lowering the barrier to entry of software with LLMs serves to allow for more bespoke software -- and it is, if anything, a kind of machine-assisted de-industrialization of software.
Instead of people downloading / purchasing the same bits for a particular piece of software which is cookie cutter like a two-piece from Men's Suite Warehouse, we can ask LLM for custom bit of code: everyone getting a garment from Savile Row.
Appreciate the love, but I think you are drawing the wrong conclusion here: Sun failed to endure not because it loved its customers, but to the contrary because it lost track of them: the company was disinterested in the mechanics of running a business.[0]
As for Oracle and its putative endurance, I would liken it to the Berlin Wall: despite the seeming permanence, it is in fact an artifact that history will be eager to forget when given the opportunity.
Per above, it seems like Sun had a core architectural, technical, and engineering failure with SPARC failing to keep pace with x86, and a business failure dealing with the fallout:
> x86 boxes were starting to smoke the hell out of UltraSPARC.)
> we spent too much time trying to help save microprocessor management from an unmitigated disaster of their own creation (UltraSPARC-III, cruelly code named "Cheetah"
In contrast, HP mostly (though it eventually split into two companies) managed to survive Itanium and compete with Dell. IBM continued to evolve Power and its other architectures and still sells AIX as well as Linux systems. Cray still exists as part of HPE. Apple migrated from PowerPC to x86 before hitting a home run with their own version of ARM.
In an alternate timeline, I imagine Sun still existing as an independent company and being the leader in RISC-V systems. But I guess Oxide is something of a successor?
If Sun couldn't design a good SPARC processor then they couldn't design a good RISC-V processor either. x86 was really their only hope but they didn't succeed there either, maybe because of the same old over-engineering.
Well, RISC-V is likely easier to implement efficiently than SPARC (no register windows, etc.) and has many of the same RISC-derived advantages.
Sun had some very smart people (what is Marc Tremblay doing at Microsoft btw?), and they could also have acquired more of them, perhaps like Apple acquiring PA Semi or Qualcomm acquiring Nuvia.
Also I wonder what might have happened if OpenSPARC had happened earlier, been more open, etc. (Indeed, a main reason why RISC-V exists is that there were IP issues with other architectures.)
Given how Fujitsu was able to make competitive SPPARC64 models for Oracle, it was unlikely to be an ISA issue, and more a design and manufacturing issue....
> As for Oracle and its putative endurance, I would liken it to the Berlin Wall: despite the seeming permanence, it is in fact an artifact that history will be eager to forget when given the opportunity.
Sparkling wine bottles are sometimes popped when the last installation of Oracle gets retired in an organization.
The name is beside the point -- and their character outs them anyway. To be clear, this was a conversation I didn't initiate (they came into my DMs, going off half-cocked about several technical aspects of Oxide that they did not understand), and they made no effort to hide their disposition. We probably disagree on this, but I don't believe that there's a basis for an assumption of privacy here (I'm not your priest, rabbi, lawyer, spouse, etc.) -- and anyone who knows me would know that I'm not the person to be confessing these kinds of sins to anyway.
Bryan, if I may ask, what was the purpose of this LinkedIn / blog post?
Acknowledging that I don’t know you, and that I haven’t seen this private conversation, this post definitely reads like you just wanted to put a former colleague on blast semi-anonymously. I came to these comments to see if anyone felt the same.
> We probably disagree on this, but I don’t believe that there’s a basis for an assumption of privacy here
In my view you crossed the line when you included his gender, his current role and employer, and two former employers.
My intent at first was not to write about this, but I couldn't stop thinking about it: I was not only profoundly disappointed in my former colleague, but disgusted by the disdain towards VMware customers at Broadcom. I was earnest in that it brought a flood of memories back for me about how ashamed I was to (briefly) work at Oracle, and I did what I have always done when something is burning inside me: I spoke my heart.
I understand that you are concerned about my former colleague (though again, a little hard to say that I'm putting them "on blast" when they are unnamed!), but my sympathies lie not with Broadcom but with the customers that they are screwing over: I have heard many, many stories from VMware customers being taken aback by the audacious things that Broadcom has told them -- the kinds of things that even Oracle has the decency to not say out loud. These customers don't speak publicly (for understandable reasons!), leaving no one to speak for them.
So yes, a Broadcom employee shooting their mouth off in an unsolicited conversation with me about their contempt for their own customers shouldn't assume that their disposition will be kept in confidence -- especially when it tracks with so much bad behavior out there!
If you have a field engineering org that can move these players off of VMWare into Oxide (the software and monitoring, etc.), you could be looking at easy picking of VMWare customers.
you act like he violated some high moral commandment by badmouthing a customer. Wasn’t Reuters a sun customer you badmouthed? Wasn’t oracle a sun customer?
Wasn’t this guy you are badmouthing, asking you questions about Oxide tech, also arguably a customer, looked at more soberly?
I just don’t get the high horse. You’re going to defend llnl and Sandia and the nnsa no matter what, since they’re customers? Not badmouthing a customer is the eleventh commandment? It’s something folksy and nice scott McNealy said. It starts losing its charm when you bash people over the head with it in public humiliation rituals like you’re in the red guard or Khmer Rouge.
Well, this wasn't badmouthing a customer -- it was showing contempt for customers, full stop. As for the accusations of hypocrisy: the example you picked (a deep cut!) was a Sun customer who insisted that we disable DTrace for their application so their customers (who were also Sun customers!) wouldn't be able to instrument the software that they had paid for. So ironically, I was in fact operating in defense of their customers. (I have generally told that story with the ISV anonymized -- but you clearly found an example where I named them.)
Broadcom is definitely not an Oxide customer -- and the (misguided) questions that were being asked were not about them becoming an Oxide customer.
Finally: isn't it a little hard to argue that I'm public humiliating someone who I am not naming?
We are back to where we started. When I said in my first message that you basically outed this person, you said it didn’t matter as the tone of the (hitherto private) exchange gave them away anyway. Now you move the goalposts back to the original position and claim you kept them anonymous.
> I have generally told that story with the ISV anonymized -- but you clearly found an example where I named them.
It was on one of the OaF podcasts about dtrace. I worked for Reuters at the time and contempt for their customers was definitely a thread that ran through some parts of that org, even as it made a bunch of us feel very icky.
(I still have a side quest to find / talk to some of the people involved on 'our' side of the fence about this!)
I know from the outside this seems very simple, but it's more complicated than that. Certainly, if the objective is (merely) security for one's children, that can be secured with much (much) less money (and likely was secured in the secondary that the author makes reference to); having nine figures of wealth is not an unvarnished good, and in particular makes raising grounded, self-reliant kids pretty complicated. To appreciate this dynamic, read Graeme Wood's outstanding 2011 piece in The Atlantic, "The Secret Fears of the Super-Rich"[0].
> having nine figures of wealth is not an unvarnished good, and in particular makes raising grounded, self-reliant kids pretty complicated
Sure, but I’m pretty sure if you asked those parents if they’d rather lose all their money to make parenting easier their answer would be a resounding “no”.
Those aren't the choices. You don't understand how the poster passed on nine figures -- but if the secondary sale netted 7 figures (likely), the choice is in fact between having enough wealth to have total security for one's family versus having so much wealth that the wealth itself creates anxiety.
Then have someone manage the money away from you. Put it in a lifetime trust, whatever. The idea that you’d turn down that sum of money because of the anxiety it would cause you is simply not logical.
> Certainly, if the objective is (merely) security for one's children, that can be secured with much (much) less money (and likely was secured in the secondary that the author makes reference to); […]
See perhaps Nick Maggiulli's post "The Ideal Level of Wealth":
> Financial Independence (28.6x Your Annual Spending): $3.5M. Assuming you never wanted to work again, you would need about 28.6x your annual spending to cover your costs indefinitely [$120,000 * 28.6 ~ $3,500,000]. This 28.6 comes from the Kitces research[1] showing that the 3.5% Rule[2] is the safe withdrawal rate for a 40-year time horizon and beyond. This research suggests that if you can make it 40 years while withdrawing 3.5% per year, then you’ll likely make it 50 years (or more).
[…]
> Whether your goal is Coast FIRE or full financial independence, the ideal level of wealth in the U.S. is in the low-to-mid range of Level 4 ($1M-$10M), or $2M-$5M. I know this is a lot of money and many people will never reach it, but that’s why it’s an ideal. It’s something to strive for. It’s enough where you don’t have to worry about money anymore, but not so much that it becomes a burden or warps your identity.
Adjust the $120k annual spend for your own lifestyle and cost of living.
You're not going to fly private, but it will take most of the worry out of life. Morgan Housel, author of the recently release The Art of Spending Money (and previously The Psychology of Money):
> 00:50:16 […] You have the independence to be who you are and wake up every morning and say, I can do whatever I want today. That’s wealth.
I won’t disclose details out of respect for the other party, but no, not necessarily. As I wrote, it was a good deal for the founders and some investors, but not for everyone, including employees. There are many ways to structure a sale, and unfortunately not all of them split the cake equally.
Ha ha -- thanks, I think? For whatever it's worth, the most important advice I have for a speaker is: speak from the heart, not from the book. That is, don't tell people what you think they want; be true to yourself and speak your own truth, in a manner that is true to who you are.
To that end: different styles work for different people. Yes, I speak quickly (or can!), but there's a method to the madness: when I am speaking fastest (and... tripping over my words, I guess?), it is likely something that -- while interesting/weird -- is in fact only tangentially related to my main point. For me, it's really important to have my actual points written on my slide: my actual decks[0] are really important to me, and serve to make my main points -- albeit devoid of the visceral metaphors for which I've become (in)famous.
I'm glad you caught on it was supposed to be praise haha. If someone threw my speech patterns on the table and started dissecting them, I'd probably stop speaking entirely ;)
It is praise; you are my favourite speaker. A lot of what you cover in talks or on the podcast is (or was) unknown to me, but I listen because it's never made to seem uninteresting, there's a passion that comes from the heart (true of the other Oxide podcast staff too). I learn a lot of information, and the anecdotes actually stick it into my brain like thumbtacks.
As for tripping: I mean to say that in natural [non-verbetim] speech, there's natural moments where we find our footing. In a speech/talk setting, it's easy to feel like you lose that footing and fall into the volcano, if you're unsteady. You show that speed itself isn't the issue and you can actually run around the volcano without falling in (–this is still praise, I swear).
Also, I must ask: (when) will the people see the release of Netris?
I don't agree that we have been "bashing" Postgres. As far as I can tell, Postgres has come up a very small number of times over the years: certainly on the CockroachDB episode[0] (where our experience with Postgres is germane, as it was very much guiding our process for finding a database for Oxide) and then again this year when we talked about our use of statemaps on a Rust async issue[1] (where our experience with Postgres was again relevant because it in part motivated the work that we had used to develop the tooling that we again used on the Rust issue).
I (we?) think Postgres is incredibly important, and I think we have properly contextualized our use of it. Moreover, I think it is unfair to simply deny us our significant experience with Postgres because it was not unequivocally positive -- or to dismiss us recounting some really difficult times with the system as "bashing" it. Part of being a consequential system is that people will have experience with it; if one views recounting that experience as showing insufficient "respect" to its developers, it will have the effect of discouraging transparency rather than learning from it.
I'm certainly very biased (having worked on postgres for way too long), so it's entirely plausible that I've over-observed and over-analyzed the criticism, leading to my description.
> I (we?) think Postgres is incredibly important, and I think we have properly contextualized our use of it. Moreover, I think it is unfair to simply deny us our significant experience with Postgres because it was not unequivocally positive -- or to dismiss us recounting some really difficult times with the system as "bashing" it. Part of being a consequential system is that people will have experience with it; if one views recounting that experience as showing insufficient "respect" to its developers, it will have the effect of discouraging transparency rather than learning from it.
I agree that criticism is important and worthwhile! It's helpful though if it's at least somewhat actionable. We can't travel back in time to fix the problems you had in the early 2010s... My experience of the criticism of the last years from the "oxide corner" was that it sometimes felt somewhat unrelated to the context and to today's postgres.
> if one views recounting that experience as showing insufficient "respect" to its developers
I should really have come up with a better word, but I'm still blanking on choosing a really apt word, even though I know it exists. I could try to blame ESL for it, but I can't come up with a good German word for it either... Maybe "goodwill". Basically believing that the other party is trying to do the right thing.
For those looking for context, this is a regrettable response of mine from nearly three decades ago, resurrected because people disagreed with the way my handling bad community behavior well over a decade ago. And for whatever it's worth, my explanation of all of this from a decade ago still stands[0].
reply