I am still a little skeptical about utilisation rates. If demand is so extreme, wouldn't we see rental prices for H100/A100 prices go up or maintain? Wouldn't the cost for such a gpu still be high (you can get em 3k used).
On "runpod community cloud" renting a 5090 costs $0.69/hour [1] and it consumes about $0.10/hour electricity, if running at full power and paying $0.20/kWh.
On Amazon, buying a 5090 costs $3000 [2]
That's a payback time of 212 days. And Runpod is one of the cheaper cloud providers; for the GPUs I compared, EC2 was twice the price for an on-demand instance.
That ASUS motherboard is far from the cheapest available. If using it makes the user liable for failure, a large part of the market is unsuitable.
For both the cooler and the motherboard, AMD have too much control to look the other way. The chip can measure its own temperature and the conceit of undermining partners by moving things on chip and controlling more of the ecosystem is that things perform better. They should at least perform.
The biggest discussion I have been on having this is the implications on Deepseek for say the RoI H100. Will a sudden spike in available GPUs and reduction in demand (from efficient GPU usage) dramatically shock the cost per hour to rent a GPU. This I think is the critical value for measuring the investment value for Blackwell now.
The price for a H100 per hour has gone from the peak of $8.42 to about $1.80.
A H100 consumes 700W, lets say $0.10 per kwh?
A H100 costs around $30000.
Given deepseek, can the price of this drop further given a much larger supply of available GPUs can now be proven to be unlocked (Mi300x, H200s, H800s etc...).
Now that LLMs have effectively become commodity, with a significant price floor, is this new value ahead of what is profitable for the card.
Given the new Blackwell is $70000, is there sufficient applications that enable customers to get a RoI on the new card?
Am curious about this as I think I am currently ignorant of the types of applications that businesses can use to outweigh the costs. I predict that the cost per hour of the GPU dropping such that it isn't such a no-brainer investment compared to previously. Especially if it is now possible to unlock potential from much older platforms running at lower electricity rates.
Why is there this implicit assumption that more efficient training/inference will reduce GPU demand? It seems more likely - based on historical precedent in the computing industry - that demand will expand to fill the available hardware.
We can do more inference and more training on fewer GPUs. That doesn’t mean we need to stop buying GPUs. Unless people think we’re already doing the most training/inference we’ll ever need to do…
Historically most compute went to run games in peoples homes, because companies didn't see a need to run that much analytics. I don't see why that wouldn't happen now as well, there is a limit to how much value you can get out of this, since they aren't AGI yet.
This just seems like a very bold statement to make in the first two years of LLMs. There are so many workflows where they are either not yet embedded at all, or only involved in a limited capacity. It doesn’t take much imagination to see the areas for growth. And that’s before even considering the growth in adoption. I think it’s a safe bet that LLM usage will proliferate in terms of both number of users, and number of inferences per user. And I wouldn’t be surprised if that growth is exponential on both those dimensions.
> This just seems like a very bold statement to make in the first two years of LLMs
GPT-3 is 5 years old, this tech has been looking for a problem to solve for a really long time now. Many billions has already been burned trying to find a viable business model for these, and so far nothing has been found that warrants anything even close to multi trillion dollar valuations.
Even when the product is free people don't use ChatGPT that much, making things cheaper will just reduce the demand for compute then.
> It has basically replaced search for most people.
Not because it's better than search was, though.
They lost the spam battle, and internally lost the "ads should be distinct" battle, and now search sucks. It'll happen to the AI models soon enough; I fully expect to be able to buy responses for questions like "what's the best 27" monitor?" via Google AdWords.
Over the long run maybe, but for the next 2 years the market will struggle to find a use for all this possible extra gpus. There is no real consumer demand for AI products and lots of backlash whenever implemented eg: that Coca Cola ad. It's going to be a big hit to demand in the short to medium term as the hyperscalers cut back/reasses.
In a thread full of people who have no idea what they're talking about either from the ML side or the finance side, this is the worst take here.
OpenAI alone reports hundreds of millions of MAU. That's before we talk about all of the other players. Before we talk about the immense demand in media like Hollywood and games.
Heck there's an entire new entertainment industry forming with things like character ai having more than 20M MAU. Midjourney has about the same.
Definitely. An industry in its infancy that already has hundreds of millions of MAU across of it shows that there's zero demand because of some ad no one has seen.
Seems like your reasoning for how the next 2 years will go is a little slanted. And everyone in this thread is neglecting any demand issues stemming from market cycles.
It should be trivially easy to reproduce the results no? Just need to wait for one of the giant companies with many times the GPUs to reproduce the results.
I don't expect a #180 AUM hedgefund to have as many GPUs than meta, msft or Google.
AUM isn't a good proxy for quantitative hedge fund performance, many strategies are quite profitable and don't scale with AUM. For what it's worth, they seemed to have some excellent returns for many years for any market, let alone the difficult Chinese markets.
Amplifiers are quoted in peak output, not average (and play some games with other parameters e.g. resistance) to capture bigger-number-better sales. A 750w system will consume nowhere near 750w at typical listening volumes (just like your 750w PC doesn't use 18 kWh every day.)
Yup. Newer products use various tricks to try to fill in the gaps that their physical reality can't overcome, but ultimately there's no getting around that reality.
I will say that the Sony upright boom boxes aren't to be slept on (and, if one is active, fat chance). They're quite good for their intended use cases (parties, and closed Best Buys during clean-up/inventory).
A 500W amp is probably a class A and can't really be made more efficient. It would still be 500W in 2024. Decades ago there were more efficient setups too, though of course now they sound better and also have lots more features and connectivity.
it is my understanding that io_uring is the generalized open source implementation of this, although i do not think it bypasses the kernel fib trie like openonload does...
Aside for onload being open source, not really. AF_XDP is the generalized, hardware agnostic, version of kernel bypass.
In addition to bypass onload also provides a full IP/TCP user space stack and non-intrusive support for existing binaries using the standard BSD socket interface (incidentally onload also supports XDP now).
io_uring is really for asynchronous communication with the kernel.
interesting, didn't know that the networking stack had ring buffer infrastructure as well. (i don't think this af_xdp stuff existed when i was in this world)
the fib trie is the core of the ip stack - i was using it as proxy for total ip stack bypass.
