Does this mean that it should be possible to load up a system with ~10 (seems to me at least the number of active experts) SSDs to get 40 tok/s even on truly gigantic models?
I'm not sure how you're going to get 10 SSDs to fit inside a laptop. And if you're not going to use a laptop, then you might as well get an expensive machine with plenty of RAM, no? Even with RAM prices as crazy as they are these days, that's probably still cheaper than a machine with enough PCIe bandwidth/lanes/ports for 10 SSDs, not to mention the cost of the SSDs themselves.
Though I guess this could be an interesting area of research, regardless of the cost.
SSD bandwidth will ultimately be limited by the amount of PCIe lanes you have available (for something other than the Apple Silicon internal storage). So the approach has inherent limitations. You can of course scale out to multiple systems to get more throughput.
You can use this approach with Intel Optane, which is wearout-resistant unlike NAND and can thus substitute for RAM. Last I checked, it was available quite cheap on the secondary market, ~$1/GB as opposed to ~$15/GB or more for DRAM. (Of course that's nowhere near as cheap as NAND, which is around ~$0.1/GB but quite wearout-prone with heavy writes.)
Yeah, PCIe is the bottleneck. The point being that whether the data originates from RAM or from NVME or Optane, you cannot get data to the GPU faster with RAM than with SSDs.
Meanwhile PCIe switches exist. So why not build:
1 CPU + memory + ...
N PCIe switch with each 1 low-memory GPU + 6 NVME drives (in theory 5 can saturate the GPU)
Each of those should only bother the CPU when they have some tokens produced and have plenty of PCIe lanes to get at their data.
Such a setup should be able to get a 6 to 8 times speedup from the solution detailed here, and a model compute increase should make relatively little difference in performance.
You aren't pointing to any data at all. Oh and the big deal with vaccines is of course that they prevent issues that happen in 1~4 per 100000 births, BECAUSE everyone is vaccinated. Vaccinations dropped that number, and are dropping that number, but if vaccinations stop, that number will again rapidly increase. And since you can bet your firstborn that if that number starts rising it will be an epidemic, those numbers will at minimum rise to their historic values, about 10% of the population. This is why there is such a strong reaction from others, when you choose to not vaccinate. One case does not matter, one too many and ...
Btw: if the number drops to actual zero, we do halt vaccinations, like we did with smallpox. There are zero patients, the last few patients were research lab accidents. Nobody is vaccinated against smallpox these days.
> "how many of the bad outcomes bieng "prevented", are actualy caused by mothers bad habbits and poor health"
Bad habits and poor health only matter, and only a bit, if the number of infections is ridiculously low, and while I get that during your entire life this has been the case, please look at some old films and get yourself informed about the bloody handkerchiefs.
Right now catching tuberculosis, you could say things like that it's not going to happen unless you spend a lot of time with an infected person in Africa, or in some "unsanitary part of" the third world. Otherwise ... it won't happen. True.
BUT if Tuberculosis starts spreading we'll be back to the situation 100 years ago in no time: spending even short amounts of time in a place where a number of infected stayed for whatever reason and you'll be infected ... and die in 2 years. We'll be back to the situation where caring for Tuberculosis patients is essentially a death sentence (meaning if you care for a lot of Tuberculosis patients, it does not matter how careful you are. You are absurdly careful and clean and ... after 5 years you'll find yourself infected. If not? 2 years. So who's going to care for those patients?)
It sounds like you're trying to save a very small amount of money, and for those tiny savings you're running the risk of creating not just a nationwide disaster, but a global disaster for everyone. The logic behind vaccines is just that: if enough people stop taking vaccines, millions will die, NOT just the ones that stop taking vaccines. And that's in the US alone.
If you must know, if vaccines cause an issue, it's called a "cytokine storm" and it happens in the single digits per million infections.
This is the central thing that changes in a person with age. When you are born, the only thing you do is pick up new things. Literally nothing else. When you're young, picking up new things is how you improve your social position. It's what you do to even be talked to in the first place. It's what you do to get a girl/boyfriend, or be the best student in class, or to be the best (or worst even) employee at your first job ...
Once you have a good social position, or at least one you're happy with, you stop doing this, and you grow ever more irritated at others doing it ... because it's your social position that they're coming after. And they're younger, more motivated and hungrier. More than that, a decent chunk of these people want a better social position, even if that means taking yours.
Sure, but this is mostly irrelevant because software dev is one of the youngest fields. Something like 70% of professional devs are under 35 depending on which survey you look at. And the number of CS graduates exploded over the last decade.
The only reason the numbers aren’t even more tilted is because people stopped hiring juniors 2 years ago. But they’re out there, and if there’s a new technology around that makes them vastly more productive than the seniors today, there’s nothing stopping companies from hiring them.
Jokes are only funny when they have an ounce of truth to them.
Oracle was the one who open-sourced the whole of the JDK, and is the main contributor to OpenJDK by far, which is completely open-source with the same license as the Linux kernel. It's fine to criticize them on e.g. Oracle db licenses and stuff like that, but they have been excellent stewards of Java and all the bad language around this java lawyering stuff is just FUD.
What? No requirement to personally bring in a form in triplicate to the Google office in Siberia, of course notarized by the Pope and Zendaya, and simply prove it was signed on the moon.
In a way it's just Maslow's pyramid, no? People who can't get housing, or good enough housing, don't care much about the image of the US abroad and may even support foreign aggression if they believe it'll help their situation.
Only once you're very secure and comfy in your little corner does "your image abroad" even have any meaning at all.
1) It's the current Iranian state has always been at war. They are at war with "the little Satan" (Israel) and "the Big Satan" (US). I know this sounds like a joke but they're absolutely 100% serious.
Serious enough to organize massacres ("terrorism") on unrelated individuals, like the Argentina bombing in the beginning and the Hezbollah massacres in Syria just last year.
And yes, there was a time of military cooperation between Iran and Israel (just after the beginning. Israel was instrumental and helped Iran avoid getting conquered by Iraq/Saddam Hussein and got intelligence on the Iraqi nuclear program in return). This has not changed anything for the ayatollahs.
And btw: the Soviet Union has had a much similar experience. They helped the mullahs get to power, helped defend against Iraq, and got an enemy in return, because the mullahs ... well presumably they saw what was happening in Afghanistan.
2) Blocking of the strait of Hormuz by Iran did not start when the current war started. Iran has always created problems for ships' passage through the strait. It has obviously been 10x'ed as a result of the recent aggression, but it's not like they left it alone before.
1) Yes, clearly Iran is run by extremist zealots who are horrifyingly committed to their cause. I don't think any reasonable person disagrees with this. To invert my original comment, 'Eurasia has always been at war with Oceania,' right? But that doesn't mean that indiscriminately bombing them is a coherent military strategy. Nor is it leading to the mass civilian uprising that was clearly hoped for.
2) I think the more important difference is that a large amount of gulf state energy infrastructure is being literally blown up. That has not happened before. We also don't know where it will end.
1) you implied that the US is perpetually at war and Iran is not, my argument was that it's literally the exact opposite. In fact, let me point out, it's even worse than that. Of the 2 or 3 superpowers that exist, the US is by far the most peaceful one, in fact it's the only one that ISN'T perpetually at war.
For the last 10 years, Russia has been at war with Ukraine, and China has been at war, meaning regular kinetic exchanges with ... everyone really. Especially Taiwan, but also Korea, the Philippines, Japan, Vietnam, and, perhaps surprisingly, Russia. In the last 10 years the US has not had any war, and I'm betting everyone kind of understands that compared to Russia's war, the US war with Iran is not even at the level Russia would call a "war". Not even remotely close. If exchanging missiles and long-range drones to and from ships counts, Russia has been at war with the northern and southeastern countries of the EU for pretty much a decade (meaning both the East Sea with Finland/Denmark/Poland/Sweden... and Turkey + Mediterranean countries on the other side)
And yet, somehow people seem to be convinced of almost the exact opposite of the facts.
Oh and the propaganda both the extreme right and the extreme left in the EU, as well as state media in Russia, are spreading is that the EU is attacking Russia (not the US, that is not a typo, that the EU is attacking Russia), even separately from the Ukraine war. That is the main factor why EU countries are so convinced Russia is close to a large scale attack on the EU. And it also of course means that at least Russia does not care in the slightest who is a threat to them and who isn't. They only care who they perceive as weak. Frankly, the same goes for Iran. Iran is not remotely threatened by the UAE, Saudi Arabia, Yemen, Kuwait or Qatar, and yet they're attacking all of them (they're even attacking Saudi Arabia with land forces by the way)
2) As to your second point, Iran was constantly attacking oil facilities pretty much their entire existence, if you include tankers, for at least 40 years, it was just constant. It was "normal", and barely reported. When it comes to land-based infrastructure attacked things heated up and cooled down, but never cooled down fully.
The US defended everyone, and the tankers, and by defended I mean we blew the hell out of Iranians, usually sinking boats, occasionally going after fixed installations. I don't think you're going to find many US Navy sailors surprised at what's happened.
Outside of that here is a list of attacks on gulf state energy infrastructure, just the last 6 years. You may notice that either by the location or the associations (e.g. Houthi attacks) it's always been Iran doing the attacks since Saddam Hussein left, with the occasional shot in the other direction. I couldn't find a good recent example, but actually it's not just Israel shooting and fighting back, but also Syria, Saudi Arabia, and Iraq. Iran isn't even at peace with the Taliban.
May 2019: sabotage of oil tankers near Fujairah in the UAE.
May 2019: drone strike on Saudi Arabia’s East-West pipeline pumping stations.
September 2019: the huge strike on Abqaiq and Khurais in Saudi Arabia, which temporarily knocked out more than half of Saudi output.
November 2020: attack on Aramco’s North Jeddah Bulk Plant.
March 2021: attempted strike on Ras Tanura and related Saudi energy targets.
March 2021: attack on the Jeddah petroleum products distribution plant.
November 2021: another Houthi claim of attacks on Aramco facilities in Jeddah.
March 2022: attacks on Saudi energy facilities, including Jeddah storage tanks.
March 25, 2022: Saudi Jeddah petroleum products distribution/storage site attacked.
June 14, 2025: Israeli strike on a Tehran fuel depot and refinery.
June 16, 2025: Iranian attack shut down Israel’s Haifa refinery.
March 2, 2026: drone attack on Saudi Ras Tanura refinery.
March 18, 2026: strike on Iran’s South Pars / Asaluyeh gas facilities.
March 18–19, 2026: Iranian retaliatory attacks on Ras Laffan in Qatar, SAMREF/Yanbu in Saudi Arabia, Mina al-Ahmadi and Mina Abdullah in Kuwait, and UAE energy sites including Habshan/Bab.
March 19, 2026: another strike affecting Israel’s Haifa refinery.
You're really playing fast and loose with your definition of war here, you change it depending on the country, but aren't clearly delineating which you're using. So let me ask a question to try to clear this up:
By the US's standard of what is considered a war, is China at war, and have they been at war in the last 10 years? If so, where and when?
Russia obviously has been at war with Ukraine, no question there.
China has been conquering territory from other nations, making it their own, always advancing (even in cases where it makes no sense whatsoever, read some articles about the Vietnamese border). If that is not war, then what is?
My understanding is that its Israel that is in endless war. Their state was birthed in 1948 by war and violence and they have been at war with their neighbors ever since. First it was the Arab states. When they made peace, they then pivoted to Iran and made them their enemy. Before this, Iran was a very pro western country (and we all know the historical lead up to this).
The question is, how do we get Israel out of this forever war state?
Can I just be cynical about such a stupid question? Obviously you're ignoring which side is attacking. But due to muslim tactics, it is easy. UN has rules about declaring war and peace that all countries (including Iran) have agreed to uphold, and muslims have a tradition of lying with peace treaties (you see, apparently some prophet did that, so it's a bit of a religious tradition). And I mean actually using peace treaties as tools for deception, ie. lying. I'm not saying nobody else does that, Russia and China certainly have the same habit, but they do too.
So because they lie anyway, muslims will very quickly declare peace ... and then attack again. As I said, not exactly a new thing, that. Iran falsely declared peace with Israel (due to the Iraq war) before the Arab nations did. Iran even betrayed the Palestinians and the Soviets for that peace treaty. They never had any intention on following through on the agreement to Israel either, of course.
So can I ask: do you believe in international law? Because either international law works and Israel is at peace. In fact Israel is currently at peace. Iran has not declared war on them, and Palestinians have declared peace and signed a peace treaty ...
Or international law doesn't really exist, and any kind of diplomacy with Palestinians has no point. Because any agreement from their side, whether signed or not ... isn't worth the paper it's written on. Same with Iran's regime.
Of course Palestinians have already violated the treaties where they declared that peace. Oh and not just violated it. Because there's violating a peace treaty and then there is putting out press releases saying you're proudly violating the peace treaty, while committing a massacre on Palestinians. Hamas likes to take such a drastically stupid approach, then wonder why Israel keeps reading their exact intentions so very, very, very well.
Well, it is (highly) illegal for them to do this. So they presumably lie about everything, like name, location, ...
Perhaps fake is not the correct word, but the actual individuals are likely to have more than a few faked details. They do exist, of course.
It's also very dubious becuase, well, would you really hire a worker from an organization that also does things like hack hospitals and then hold systems hostage for bitcoin?
Obviously, when working you have to follow the law both in the country where you live and the country where you work. Even in the case of remote work. Sadly, even if you just consult. So you can be pretty sure: highly illegal.
reply