While I don't disagree, memory certainly was more of a restrictions on us humans than it is on llms. Therefore, the answer may not be as obvious as it seems. We build abstractions to reduce (memory) footprint of features, right?
I think the trick here is plural; I guarantee no single human knows all 1 million lines. Note this is different than knowing how to orient yourself in a million line codebase quickly.
The limit here I think the ancestor comments are getting at is cognitive load, which is real and measured. We only have so much memory to devote to a "stack" when executing, and it's usually quite constrained.
Note this is different than knowing how to orient yourself in a million line codebase quickly.
Hence my library mention. Humans have been doing this for millennia: orienting ourselves within a library (the physical kind, full of books) and calling upon its information resources as needed to accomplish tasks (research). Ultimately, it's all just one big cache hierarchy. Your short term memory, your long term memory, the book in your hands, the desk at the library, the nearby shelves, the card catalogue, the stacks, the inter-library loan system.
To manage it all, we humans have developed our abilities for abstraction. When we build clean, tight abstractions we reduce our cognitive load. Perhaps the best abstraction we've built so far is the TCP/IP and web stack. We don't need to care at all about the hardware details of a server in order to talk to it. It's such a powerful and airtight abstraction that we take it for granted.
I'd like to hear from more people who have spent a lot of time building with LLMs, because so far what people are saying is that these models do not have the ability to reason about and build the kind of marvellous abstractions us humans have built.
I've built a lot with LLM's, my experience sort of but not really tracks that. I've had to course correct a few bad abstractions but the larger the code base becomes the better it seems to be at reusing things. Maybe this is because of types, or spec-first development (with OpenAPI), or black box integration testing - but also maybe not. But generally I have to think about the abstractions and let the LLM fill in the details with rare exception.
I built a web-OS, a graphical IDE, and a version control system to replace git, all in about 40,000 lines of highly abstracted Javascript. If you're thinking about how important it is to be able to maintain million-line codebases, I suspect you might have substituted a metric for the actual end goal.
That looks like a nice feat, can you share a repo?
That said, reality at scale always come with details that will break the model, and the main roads when it happens are to ignore/reject any change proposal in the model, go in the mystic quest to reach a model that will fit it all including these new cases with an elegant simple solution, or accommodate special cases on the side until it grows too big or just percolate too fast in the main part to let it be sustainable.
Yes, I believe it stems from the tankards having lids back in the day, which is due to the belief that plague-ridden flies could fly into your beer, and also against "night air". Interestingly some Germans still believe "moving air" (well, draft) is unhealthy, especially from an AC, and the cold air is what makes you sick.
Which, to be fair, overall probably still saved quite some time. The compile times alone would've meant they wasted so much more time by repeated earlier checks.
Whether a looming deadline changed the perception about that, we don't know ;-P
'liquid metal' sounds cool. It's probably a metallic glass. I super dislike that it seemingly will be synonymous with the brand name by Apple even though that stuff has been around for decades.
Not that there are particularly many places where this is used - mostly because it really is just very expensive. In the awesome position that Apple is in, economic feasibility is so much easier to achieve, with like tens of millions of guaranteed parts to be preduced.
Are you sure? Liquid metal was the name of a bulk metallic glass. There were usb flash drives using it as a case https://en.wikipedia.org/wiki/Liquidmetal. Wikipedia lists apple licensing this technology.
Metal injection molding is also a thing but I haven't heard it called liquid metal. Usually its MIM.
Honestly, I didn't know that "amorphous metal alloy" is also called metallic glass. I computed it to something else entirely. So you're right on that front.
MIM is something else, that's right, but properties of Liquid Glass allows it to be injection molded AFAIK.
MIM process is completely different from casting Liquid Metal. MIM generally starts as a powder and heated and molded, Liquid Metal can be just "melted and molded".
I have a stainless steel razor built with MIM. Has no resemblence to SanDisk Titanium's feel (which I also have).
I did my Ph.D. by developing BEM evaluators for working on metals, but glasses (as in class of materials) were not in my domain, so I'm thick as a brick on that part of the materials science.
Edit: BEM methods is as fun as USB buses and PSU units.
You can buy shares in the company that makes and holds the patents on liquid metal they are currently a penny stock and have been for several years and at one time the shares sold for $23 a share at its highest point many years ago..
I don’t expect anything out of it, but I own 12,000 shares just for fun.
Well, there is a financial 'sink' - stockpiles and ammunition or other non-reusable military gear are basically the definition of money 'destroyed'. Their political value is almost non-existent actual money. If any, at all.
> stockpiles and ammunition or other non-reusable military gear are basically the definition of money 'destroyed'
Goods like longer-lasting food, medical supplies or a strategic oil reserve are not wasted. The money that went into supplying them has gone back into the economy, and they serve a more strategic purpose than the market participants could have borne (i.e. societal insurance policies). The same could also be said of military stockpiles, and continuing to buy them sustains a capability that is hard to get back once lost.
Those stockpiles weren’t created by putting money into a shredder and getting ammunition out. They were created by paying for the materials and labor. At that point the government’s money is frozen and stockpiled, but the economy still has the money that was spent.
Sorry but isn't the bottleneck then simply to do even relevant things? Like how much of a qualified backlog do you have that your pipeline does not run dry?
I think the reason the pelican example is great is because it's bizarre enough that it's unlikely that to appear in the training as one unified picture.
If we picked something more common, like say, a hot dog with toppings, then the training contamination is much harder to control.
reply