Does anyone run their own AI service?
If so,
Which one?
What hardware?
Do you have a way to access it from outside or from a phone?
And, if you’re open to sharing, what do you do with it?
I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes.
I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.
You must log in or register to comment.
Not only do I run and share it. I’ve built an entire service to allow everyone else to do so safely. https://aihorde.net/
Yes, I have a desktop with a radeon 9700 ai pro with 32GB of VRAM. Its more than doubled since I bought it but its basically a down clocked 9070xt with double the VRAM. I also have a little arc b50 in my server.
Mostly yeah use it for privacy reasons. I run llama.CPP directly and access via open webui. I host a searx instance for it to use as web search. Mostly run qwen3.6/3.8 or gemma4 on the radeon and almost exclusively gptoss 20b on the B50 along with a second small model for tags and titles in open webui. Yeah I just use it mostly as a basic chatbot or as an agent in obsidian using the engram plugin. Helps search and summarize years of notes and works pretty well.
I’ve tried to use it as a coding assistant in zed without much luck so usually just post my code into open webui for feedback and it works alot better for me.
Yeah I can access it with other machines on my network including phone with conduit app. I can get to it on the go by connecting to home network with wireguard and sending a magic packet to wakeup desktop.
I havent done any image gen nor have much use for it outside of maybe using it to make app icons in the style of papirus icons for programs they don’t and won’t support.
It definitely feels more comprable to first ai wave than current frontier models but the privacy and price trade off is worth it to me. I’d say using it in obsidian has been the best use of it so far and one I certainly wouldn’t trust to an external AI.
Hardware:
2 x ASUS GX10 128GB(each)
2 x Mac Mini M4 Pro 24GB(each)
The GX10s are clustered and run DeepSeek-V4-Flash-0731-Abliterated. The M4 Pros run various utility LLMs such as Qwen 3.8
Software:
OpenWebUI
Hermes Agent
LiteLLM
Llama.cpp
vLLM
StoryBinder - custom web app that I use for story/script writing when in the mood
LASM - Local AI Stack Manger - custom web interface to manage pretty much all aspects of my local stack. Agents run on the 4 nodes as well as my NAS(Download GGUF/HF snapshots once and copy out to the nodes)
Have yet to connect my Home Assistant into the stack.
For mobile access I use Conduit https://github.com/cogwheel0/conduit
Yes, I run Ollama on a NVIDIA 1080 Ti at home. I run granite4:3b for my Home Assistant assistant for my voice assistant, paperless-ngx, OpenWebUI for chat, and a few other things. For speech to text, I run a custom-built hybrid speech-to-phrase + parakeetTTS (requests that match speech-to-phrase take priority over parakeet.) I run GOModel to route requests to my local Ollama instance, or DeepInfra (cloud-based for larger models), or Tinfoil (cloud-based for encrypted LLM chats.)
As much as I love to use local-everything, only my smarthome+basic chats go local. Trying to fit both the STT and LLM models on the same GPU at the same time while also using this computer as a media console is challenging. I’ve recently started looking at the Hailo m.2 accelerator board to see if I can shift the STT to it.
Yes, I installed an LLM at home. I use it for many useful things:
- reading novels by translating them from English to my language on the fly via machine
- for communicating on lemmy by translating posts and comments
- For HomeAssistant
- And very rarely for development and searching the internet
I have a RX9070XT and llama-swap to switch models on the fly. I’m thinking about installing a second RX9070XT so that I don’t take away the translation capability from my lemmy instance while I stress the other neural network with other tasks.
Yeah, I have a fairly useable setup although it’s only accessible on my lan. I have a web server but it doesn’t host the llm as it’s not powerful enough.
Rx7600 8gb vram GPU 5700x CPU 64gb 3600mts ram
Hosting ornith 35b a3b iq4_xs Llama.cpp server Pi harness or llama.cpp webui Searxng web search Coding sandbox Restricted filesystem access
Get around 40 tok / sec and it’s really useable. Not as good as Claude sonnet, but good enough for most things.
You can setup a reverse proxy on the web server to direct a subdomain over to the ai server if you’d like to have it accessible outside your lan
Ollama, openwebui, running the ~20GB Qwen models on a 3090
Epyc 7551? 256gb ddr4, r9700 + mi50. Running two models depending on what I’m doing . Llama.cpp + comfyUI or two llama.cpp for model and advisor in OMP. Main model is Qwen 3.8 27b abliterated
What a timely comment.
I recently gave my kid a gaming PC for his birthday…which I used as an excuse to upgrade my GPU from a 1660 Super to a 5060ti 16GB
Combined with a 5600x3d and 32gb of system memory.
I just started messing with local LLMs yesterday. “Getting started” is rather easy.
pacman -S lm-studio(I use arch, btw), launch it, download a model that your hardware supports, and start talking.Right now I’m working on getting it to talk to vscode.
I self-host my own neural network, all within the confines of my own skull.
Me also, and I can barely afford to run that
That 33% downtime though
The downtime isn’t the biggest issue, you can plan around that. The real problem is the random lag on some queries and volatile memory being lost (usually when there’s a context change).
It’s scheduled maintenance so it doesn’t count toward downtime. Besides, I’m not trying to use it then anyway.
It’s better than github
Only 33% downtime? Damn, that must be nice. I really need to get a hardware update.
How much RAM is it using?
Made me think of Johnny Mnemonic. https://www.youtube.com/watch?v=jS2GFqT5ycU
Iirc in a documentary, the brain runs on just 13w of electricity, so the more advanced and efficient AI gets, the less power and therefore resources it’ll need. Maybe Johnny’s world is just that advanced lol
I’m just glad that this is both Top & Hot comment. 🧠✊
24GB GPU
Qwen 3.8 ( 2 variants )
Pangolin>OpenWebUI>lama.cpp for general use and available outside the homelab
Zoo Code>OpenWebUI>lama.cpp for coding , mostly scripts and html
Qwen 3.8 was a game changer for me.
I’m self-hosting immich (with ML-based facial recognition, semantic search and OCR) and psiTTSa, an offline text-to-speech engine.
(Prob not the answer you were looking for, but a reminder that “AI” is more than just LLMs.)
Does Immich sell/use/keep your ML-based FR? Or is this totally local a never leaves your instance?
That was always the question that bordered me… I run Leeche its a very simple and minimal photo album service and I really like it ! But my family want something that’s more like immich but I’m very cautious about what i’m selfhosting and something that collects data points on my behalf is a no-go !
Immich is entirely free and open-source software and got a grant from the FUTO Foundation, which is fiercely pro privacy. As such, it does not engage in any data sharing. Development is financed through the grant and purchases of the server licence (I got one; it’s ridiculously good software).
I do not run Immich but their privacy policy states:
Locally Stored Data: Immich stores all your photos, albums, settings, and other metadata locally on your device. We do not have access to this data, nor do we transmit or store it on any of our servers.
https://immich.app/privacy-policy
Additionally in their docs I can find no mention of connecting to outside services. I do see references to downloading hugging face models.
https://docs.immich.app/features/facial-recognition#how-face-detection-works
Thank you very much ! I didn’t expected someone doing my part to RTFM o_O !
Without doing again the heavy reading part, what should I expect from hugging face models?
If my reading skills are correct it should be local, but I’m really paranoid about this stuff ^^ Even though I read your posted links, I’m still not sure if I should trust them…
Something written in a manual and what the code realy does are sometimes far appart ^^’
Not me. Call me old-fashioned, but I prefer organic.
Running Qwen 3.8 27B dense on some R9700s, and Qwen 3.8 flash next on dgx sparks. I use it for code gen outside work, including an automated Dev pipeline that rubs on its own. When I did the math comparing to token API pricing of Opus 4.8, it covered its own costs in a few months. With the sparks I get about 40tg/s, 1500pp/s, but can get concurrency up to 5 stably. With the R9700s, I get ~100tg/s, 3000pp/s and can also get up to 5 concurrency.
I access it over wireguard to the network that it is on, and use termux to ssh into a build machine for dev over CLI using https://pi.dev/
I also started building https://fcli.dev/ to make integrations easier; the idea being that LLMs should just implement simple reusable functions instead of repeatedly making complex curl requests to integrate with a service.
I also use LLMs heavily for work; Qwen 3.8 seems almost on par to Opus 4.8 IMO. I also don’t have to worry about token costs, and it often responds faster than Claude would because there usually isn’t a queue on my machines. At this point, I wouldn’t complain if my work provided me 2x B6000 instead of a Claude subscription; I’d still be able to get all my work done.
Would be interested to know your experience with fcli if you’d share more. Just hearing about it.
I mean, I haven’t really started discussing it outside IRL friends, so not surprised you haven’t heard of it 😅
In essence, the problem is twofold:
- LLMs tend to use CLIs more effectively than MCP, at least in the domains I use. However, installing all the CLI tools and configuring them can be cumbersome and not easily repeated. CLIs also cannot be modified, and are often quite verbose.
- LLMs will repeat very similar commands over and over with just a few tweaks. This often happens when there is a gap in a tool or poor documentation.
So fcli is intended to be an easily extended, configured, and authenticated CLI. The “one CLI” to do everything your LLM needs. Ideally, you set up a personal repo to store ‘ucks’ (f packages) in, and you set up credentials for that repo so your LLM can push any new ucks to it. You can then share that repo with other users if they could also benefit from any of that functionality.
As you go along in a session, when you notice the LLM repeatedly running similar verbose commands, you can have it create an uck for it instead and save those tokens.
Likewise, if, say, an MCP server doesn’t have full functionality, you can have your LLM import the SDK from npm into an uck and write a basic fcli wrapper around it to start using.
The other nice thing is that it makes your tool configuration repeatable between LLMs and harnesses. You don’t need to set up your MCP servers in both Claude and Codex and Pi, just in f and tell them to use f.
There’s also a creds uck to make your secrets more portable and (a bit) safer.
In general, I’m trying to model it a bit like pi.dev, but as part of the tooling instead of the harness itself. I think they work quite well together :)
- Qwen3.6 35b a3b Q4_K_M running on llama.cpp
- Open-WebUI as a frontend
- integrations with an embedding model and searxng & crawl4ai
- doing lots of different stuff with it, asking basic stupid questions, recipes, tech stuff, etc
Very similar, but I’m using Gemma4 12b. I could run 26b but that would exhaust my vram after a short interaction. And ollama. Open WebUI + Home Assistant hooked up with it, a couple very basic scripts I threw together, and connected to Kagi.
I’m trying to get my folks off the Gemini/Google bandwagon for the privacy aspect, but also because it’s just cool from a tech geek perspective. For me, any questions throughout the day that I want info on, but not enough to do a deep-dive for an hour+, gets fed to Steve, my models name. Or I’ll use it as a springboard and then properly read the citations for more info. And I feed it HA automations when I get stuck scratching my head on an edge case or whatever - it’s alright but helps me explain what I want from the automation, and when it understands my goal, great, and when it doesn’t I reflect on what I have setup and can usually go ‘ah ha’ and fix the issue. It’s like a rubber ducky/pair programmer that you can both think is an idiot sometimes, but it doesn’t judge you when you’re the idiot.
That’s a pretty good way to describe those kinds of models. Somewhere between rubber duck debugging and a really lazy intern. Better than an algorithm, faster than a human, but less trustworthy than an intern. It’s great if you know how and when to use it. Maybe it’ll become an intern soon enough, but for now let’s focus on decompiling adobe into bankruptcy.
Random Question, I’ve been trying to offload some of my stuff handled by claud to basically this exact setup. However one of the things I manage has a pair of MCP servers that are required. I might not be smart enough, but how do you get Qwen to use an MCP?
MCP is a really dumb server that just takes requests and sends back data.
Somewhere in your frontend settings (Open-WebUI has it under “integrations” I believe, some backends can set these things too) you see a section called “MCP server”. You need to keep your MCP server running somewhere (it’s very slim, so not a huge deal usually) and then in those settings you put your IP and port for the server, for example “http://192.168.1.15:8080/”. Usually if you did it right (sometimes you can check if it works in the settings), you can ask the model if it has access to the tools and it will tell you.
What’s happening in the background is that every conversation the AI does, it gets a short preamble with some context about what it can do, and if you connect web search, custom skills or an MCP server (it’s just skills but they are run through their own server basically), the AI sees all that in this preamble if it’s set up correctly. And then when it feels like it needs to use those tools, it will check the description on how to do that, and then it uses them.
What’s your rig?
1070ti and some hexacore something cpu that’s alright with around 24gb ddr4, I think one of the slower speeds. I also got a faster PC with another llama.cpp instance and I vibe coded a quick and dirty proxy which I can use to switch between the different backends. It’s genuinely useful because I got code editors but also my Open-WebUI using it.
Sounds elaborate but generally it’s all pretty primitive.
Also fyi long preloading steps because my vram is obv not good enough to hold the full model and kv cache and stuff, generation at about 15t/s tops.
My experience is right now we’re at the stage where it’s a good assistant but not great for actually taking over bigger tasks completely.
Am looking forward to:
- experimenting with existing non-uniform quants; I believe those quants are the next step to more bang for your buck on those low-grade consumer rigs
- hoping for new kinds of quants to come out and be more widely adopted
- maybe a more universal or consistent harness that is less prone to broken tool-calls etc.
- better models, specifically a new version for 35b a3b.
Pretty much my setup as well. Though I’m using qwen3.8 27b-Q4_K_M under ollama. Everything running with docker. Add nginx and either a domain+port forwarding or zerotier/tailscale you’ve got an app you can get to from outside your house.
Right now not sure how I’m going to leverage it not probably to streamline my self hosted stuff, help with config mgmt and documentation, web page summarization, etc. we’ll see how it does, just got it setup yesterday
In case you are unaware, friends don’t let friends use ollama. The llama.cpp CLI has become pretty much equally easy to use, plus since you use Docker there’s no need to compile anything.
Edit: typos and missing words
Oh damn what a useful link. Gonna check out those orchestrators, maybe I can replace my janky ass proxy.
Oh shit I wasn’t aware! Thanks for the heads up, I’ll swap to llama.cpp
I’ve got similar running on my RX 7600 XT. It is not a great experience for interactive use, but fine for async tasks. I’m also running nemo-speak for live local dictation and meeting transcription.
Yeah running in the background type beat. Not gonna trust this with anything important but it’s great to have someone just do shit for you or get a second opinion.
Yah doing similar on 5060ti.
That’s gonna be quick af with the same models.
It is, i also use the gemma4b model on it for description generation for my immich photos and its super fast.
