I want to host some LLM’s locally and use more advanced models. Since new hardware is out of the question, I think I should be able to pull something off buying some yesteryear equipment on ebay etc. Did anybody attempt such a project? Does it scale horizontally? (I.e. can I connext two boxes to overcome single box slowness?)
I’ve got some circa 2010 cards laying about with a 32Gb server that already has 8Gb carved out for TrueNAS, so essentially I could squeeze 16-24Gb out of it, but it’s an older i5 Intel CPU
Your biggest issue with 2010 cards will be software (inference engine) support, I assume.