It all depends on the size of the model you are running, if it cannot fit in GPU memory, then it has to go back and forth with the host (cpu memory or even disk) and the GPU. This is extremely slow. This is why some people are running LLMs on macs, as they can have a large amount of memory shared between the GPU and CPU, making it viable to fit some larger models in memory.
- 0 Posts
- 3 Comments
Joined 3 months ago
Cake day: February 17th, 2025
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
0·2 months agoI thought nftables where replacing iptables?
Oh, I thought you could get 128gb ram or more, but I can see it does not make sense with the <24gb… sorry for spreading misinformation, I guess, in this case a GPU of the same GB ram would probably be better