minus-squareIrdial@lemmy.mltoSelfhosted@lemmy.world•[AIT] what is your selfhosted local LLM stack?linkfedilinkEnglisharrow-up2arrow-down1·1 day agoI’ve got an M4 Mac mini with 32 GB of unified memory, so I can run MoE models in the <40B parameter range quite comfortably. I use LM Studio as the backend with Hermes as the frontend. Works great for everyday chat, web search, and RAG. linkfedilink
I’ve got an M4 Mac mini with 32 GB of unified memory, so I can run MoE models in the <40B parameter range quite comfortably. I use LM Studio as the backend with Hermes as the frontend. Works great for everyday chat, web search, and RAG.